From 1e9e8716ccef13e3512f377a35c5fe5ca987024c Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Sun, 15 Mar 2026 21:02:44 -0700 Subject: [PATCH 01/92] docs: add project roadmap with five development workstreams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Public-facing roadmap organized into five independent workstreams: 1. Continuous Operators & Agents — hardening long-horizon autonomy 2. Mobile & Messaging Clients — iMessage, WhatsApp, Slack, SMS 3. Secure Cloud Collaboration — Minions-style hybrid inference, TEE 4. Tutorials & Documentation — filling gaps in continuous agents, tools, LM eval 5. Hardware Breadth — Intel Arc, Jetson Orin, Qualcomm, AMD Ryzen AI, RPi Each item tagged with maturity level (Ready/Design Needed/Research-Stage) and time horizon (near/mid/long-term) to guide contributors. Co-Authored-By: Claude Opus 4.6 (1M context) --- ROADMAP.md | 209 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 ROADMAP.md diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 00000000..07ec11ec --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,209 @@ +# OpenJarvis Roadmap — Personal AI on Personal Devices + +## Overview + +OpenJarvis studies personal AI through five core primitives — **Intelligence**, **Engine**, **Agents**, **Tools** (memory, retrieval, tool APIs), and **Learning** — running on personal hardware. This roadmap organizes development into five independent workstreams, each with items flowing from immediately actionable to exploratory research. + +### How to Read This Roadmap + +**Workstreams are independent.** Contributors can pick any track without waiting on others. Within each track, items are organized by time horizon: + +- **Near-term** — foundations and hardening of what exists +- **Mid-term** — significant new capabilities +- **Long-term** — frontier work requiring design exploration or research + +Every item carries a **maturity tag**: + +| Tag | Meaning | Contributor guidance | +|-----|---------|---------------------| +| **Ready** | Well-scoped, implementation path is clear | Pick it up — check issues for a spec or write one | +| **Design Needed** | Concept is clear but needs a spec before code | Start a design discussion or draft an RFC | +| **Research-Stage** | Exploratory, needs investigation before designing | Read the relevant papers, prototype, share findings | + +Items marked with **"good first issue"** are especially suited for new contributors. + +--- + +## Workstream 1: Continuous Operators & Agents + +Operators are OpenJarvis's key differentiator — persistent, scheduled, stateful agents that run autonomously on personal devices. The current tick-based architecture (OperatorManager → TaskScheduler → AgentExecutor → OperativeAgent) is solid but needs hardening for truly long-horizon autonomy. + +### Near-term + +| Item | Maturity | Details | +|------|----------|---------| +| Operator health checks & heartbeat monitoring | **Ready** | Add liveness probes to OperatorManager; surface in `jarvis operators status`. Detect stalled operators beyond the existing reconciliation loop. | +| Metrics collection for operator manifests | **Ready** | The `metrics` field exists in `OperatorManifest` but is not collected. Wire it to telemetry. Good first issue. | +| Capability policy enforcement | **Ready** | `required_capabilities` field exists in manifests but is not enforced. Connect to the existing RBAC `CapabilityPolicy` system. Good first issue. | +| Rate limiting per operator | **Ready** | Prevent runaway operators from hammering inference. Add configurable rate limits to OperatorManager. | +| Operator composition / chaining | **Design Needed** | Express dependencies between operators (operator A feeds results to operator B). Requires design for data passing and scheduling semantics. | + +### Mid-term + +| Item | Maturity | Details | +|------|----------|---------| +| Event-driven operators | **Design Needed** | Operators that trigger on EventBus events (e.g., new file indexed, channel message received) rather than only cron/interval schedules. | +| Operator versioning & rollback | **Design Needed** | Run v2 of an operator alongside v1. Roll back automatically on repeated failures. | +| Multi-device operator coordination | **Design Needed** | Operators spanning laptop + workstation + datacenter node. Requires device discovery and task delegation protocols. | +| Dynamic tool loading per operator | **Design Needed** | Runtime tool discovery rather than static string lists in TOML manifests. | + +### Long-term + +| Item | Maturity | Details | +|------|----------|---------| +| Self-improving operators via Learning | **Research-Stage** | Operators that use trace feedback to tune their own prompts, tool selection, and routing policies through the Learning primitive. | +| Distributed operator mesh | **Research-Stage** | Peer-to-peer operator coordination across devices without a central orchestrator. | + +--- + +## Workstream 2: Mobile & Messaging Clients + +Personal AI must be accessible from the devices people actually carry. OpenJarvis runs on laptops, workstations, and servers — users interact via their phones. Channels bridge that gap. Today, WhatsApp (Baileys), Slack, and Telegram are bidirectional; iMessage is send-only; Android SMS does not exist. Beyond the channels listed below, OpenJarvis already supports 20+ additional channels (Discord, Matrix, Mastodon, Nostr, IRC, Line, Viber, Email, etc.) — see `src/openjarvis/channels/` for the full list. + +### Near-term + +| Item | Maturity | Details | +|------|----------|---------| +| iMessage bidirectional via BlueBubbles | **Ready** | Current implementation is send-only. Add webhook/polling listener for incoming messages using the BlueBubbles API. Good first issue. | +| WhatsApp Baileys media support | **Ready** | Currently text-only. Add image, audio, and file handling to the Node.js bridge and Python channel. Good first issue. | +| Slack rich messages | **Ready** | Current implementation is plain text. Add Slack Block Kit support for formatted responses, buttons, and attachments. | +| Android SMS via Twilio/Vonage | **Design Needed** | No SMS implementation exists. Requires provider selection, two-way webhook architecture, and phone number provisioning flow. | + +### Mid-term + +| Item | Maturity | Details | +|------|----------|---------| +| Unified notification system | **Design Needed** | Push notifications when operators complete tasks or need user attention. Requires per-channel notification adapters. | +| Offline message queue with retry | **Design Needed** | Handle mobile unreliability — queue outbound messages, retry on reconnect, deduplicate. | +| Channel media pipeline | **Design Needed** | Unified image/audio/file handling across all channels with consistent metadata and storage. | +| Signal bidirectional | **Design Needed** | Currently send-only via signal-cli REST API. Add incoming message listener with background polling. | + +### Long-term + +| Item | Maturity | Details | +|------|----------|---------| +| Voice interface | **Research-Stage** | Speech-to-text (Whisper) → agent → text-to-speech loop over phone channels. Existing `speech/` module provides a foundation. | +| Cross-channel session continuity UX | **Design Needed** | Start a conversation on Slack, continue on WhatsApp seamlessly. The `SessionStore` already supports multi-channel identity linking — this needs UX and channel-level plumbing. | + +--- + +## Workstream 3: Secure Cloud Collaboration + +Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through three complementary approaches: **Minions-style collaborative inference** (local handles context, cloud handles reasoning), **TEE-based confidential computing** (cloud cannot see your data even during inference, inspired by [Tinfoil](https://tinfoil.sh)), and **secure multi-device coordination**. + +Key references: +- [Minions: Cost-Efficient Local-Cloud LLM Collaboration](https://github.com/HazyResearch/minions) +- [TEE for Confidential AI Inference](https://openreview.net/forum?id=ey87M5iKcX) +- [Tinfoil: Verifiably Private AI](https://tinfoil.sh) + +### Near-term + +| Item | Maturity | Details | +|------|----------|---------| +| Query complexity analyzer | **Ready** | Classify incoming queries by difficulty (token count, entity density, reasoning depth) to decide local vs. cloud routing. Extends the existing `MultiEngine` routing logic. | +| Cost tracking per-query | **Ready** | `CloudEngine` already has pricing data. Surface per-query cost in traces and telemetry dashboards. Good first issue. | +| Redaction-before-cloud pipeline | **Ready** | Wire the existing `GuardrailsEngine` in REDACT mode as a mandatory pre-step before any cloud transmission. The security primitives exist — this is integration work. | +| Minion protocol (sequential) | **Design Needed** | Local model extracts and summarizes long context → cloud model reasons over the compressed result. Native reimplementation of the core [Minions](https://github.com/HazyResearch/minions) idea in the `engine/` layer. | + +### Mid-term + +| Item | Maturity | Details | +|------|----------|---------| +| Minions protocol (parallel) | **Design Needed** | Local and cloud models work simultaneously on different aspects of a query; results are merged. Requires a new `HybridInferenceEngine` abstraction. Depends on: Minion protocol (sequential) from near-term. | +| Adaptive routing with learning | **Design Needed** | Router learns which queries need cloud vs. local based on trace feedback (accuracy, cost, latency). Connects the Engine and Learning primitives. | +| TEE attestation verification | **Design Needed** | Verify that cloud inference ran inside a trusted execution environment via cryptographic attestation. Add attestation checking to `CloudEngine` response handling. | +| Confidential inference provider support | **Design Needed** | Add Tinfoil (or similar TEE-backed providers) as a first-class engine backend. OpenAI-compatible API with attestation verification built in. | +| Taint tracking across local/cloud boundary | **Design Needed** | The `TaintSet` (Python: `security/taint.py`, Rust: `crates/openjarvis-security/src/taint.rs`) already tracks PII/Secret labels. Add routing enforcement at the `engine/` layer so tainted data only routes to attested TEE endpoints, never to unattested cloud APIs. | + +### Long-term + +| Item | Maturity | Details | +|------|----------|---------| +| Speculative decoding (local draft + cloud verify) | **Research-Stage** | Local model generates candidate tokens speculatively; cloud model validates in parallel for latency reduction. | +| KV cache sharing between local and cloud | **Research-Stage** | Transfer attention state between engines to avoid recomputation. Requires a shared cache serialization format and encrypted transport. | +| Secure multi-device federation | **Research-Stage** | Multiple personal devices collaborate on inference with end-to-end encryption. Extends operator mesh from Workstream 1. | +| Early exit detection | **Research-Stage** | When local model confidence is high, skip cloud entirely. Dynamic cost/quality tradeoff learned from traces. | + +--- + +## Workstream 4: Tutorials & Documentation + +OpenJarvis has strong reference docs and four tutorials (deep research, scheduled ops, messaging hub, code companion), but critical gaps remain in continuous agents, LM evaluation, learning approaches, and custom tools. Video tutorials are scoped as a contributor opportunity — written tutorials come first, with video scripts included so anyone can record. + +### Near-term + +| Item | Maturity | Details | +|------|----------|---------| +| "Building Continuous Agents" tutorial | **Ready** | Writing an operator TOML manifest, activating it, session persistence across ticks, state management, daemon mode. Example: a research operator that monitors arxiv daily. Follows the existing tutorial template (Python script + TOML recipe + markdown walkthrough). | +| "Adding Custom Tools" tutorial | **Ready** | Implementing `BaseTool`, registering via `ToolRegistry`, wiring into agents. Example: a weather API tool. `docs/development/extending.md` covers engine extensions but there is no standalone tools tutorial with a runnable end-to-end example. Good first issue. | +| "Testing & Comparing LMs" tutorial | **Ready** | Running benchmarks, comparing local vs. cloud models, interpreting telemetry (latency, cost, energy per token). Uses the existing `bench/` framework. | +| Per-platform installation guides | **Ready** | Expand `installation.md` with platform-specific walkthroughs: macOS Apple Silicon + Ollama, Ubuntu + NVIDIA + vLLM, Windows + Ollama, Raspberry Pi. Good first issue. | + +### Mid-term + +| Item | Maturity | Details | +|------|----------|---------| +| "Learning & Model Selection" tutorial | **Design Needed** | Router policies (heuristic, learned, GRPO), proposed approaches like Thompson Sampling, trace-based reward signals. This is the least-documented of the five primitives. | +| "Multi-Channel Deployment" tutorial | **Design Needed** | Deploying one agent across Slack + WhatsApp + iMessage simultaneously. Cross-channel session continuity. | +| Video tutorial infrastructure | **Design Needed** | Establish recording workflow, hosting (YouTube), MkDocs embedding. Write video scripts alongside written tutorials so contributors can record independently. | +| Interactive Jupyter notebook tutorials | **Design Needed** | Notebook versions of key tutorials for exploratory, cell-by-cell learning. | + +### Long-term + +| Item | Maturity | Details | +|------|----------|---------| +| Contributor tutorial program | **Research-Stage** | Templates and guidelines for community members to submit their own tutorials. Review process, quality bar, and integration with the docs site. | +| Tutorial localization (i18n) | **Research-Stage** | Translate core tutorials into major languages. | + +--- + +## Workstream 5: Hardware Breadth + +Personal AI means running on the hardware people actually own. Each new hardware target expands who can use OpenJarvis and generates data for the research agenda (energy, cost, latency tradeoffs across silicon). + +Adding a new hardware target involves up to four components: hardware detection in `core/config.py`, an inference engine adapter in `engine/`, an energy monitor in `telemetry/`, and an entry in the GPU specs database in `telemetry/gpu_monitor.py`. + +### Near-term + +| Item | Maturity | Details | +|------|----------|---------| +| Intel Arc GPU (B580/B570) | **Design Needed** | 12GB VRAM, ~$250 consumer GPU. Viable for 7-8B models. Engine path: IPEX-LLM or llama.cpp SYCL backend. Needs `_detect_intel_arc_gpu()`, energy monitor via RAPL/sysfs, engine adapter. | +| NVIDIA Jetson Orin | **Design Needed** | Best-in-class edge device. Orin NX 16GB handles 7-8B models at 15-25 tok/s. Ollama/llama.cpp already work; needs hardware detection, energy monitor (tegrastats), deployment guide. | +| Qualcomm Snapdragon X Elite NPU | **Design Needed** | 45 TOPS, Windows Arm laptops. ONNX Runtime + QNN Execution Provider is the viable path. Needs new engine adapter, hardware detection, energy monitor. | +| AMD Ryzen AI iGPU path | **Ready** | Strix Point RDNA 3.5 iGPU handles 7-8B via Vulkan. llama.cpp Vulkan backend works today. Needs hardware detection and energy monitor. Good first issue. | +| GPU specs database expansion | **Ready** | Add Intel Arc, Jetson Orin, Snapdragon specs to `GPU_SPECS` in `telemetry/gpu_monitor.py` (TFLOPS, bandwidth, TDP). Good first issue. | + +### Mid-term + +| Item | Maturity | Details | +|------|----------|---------| +| Intel Lunar Lake NPU via OpenVINO | **Design Needed** | 48 TOPS — most mature NPU software stack for x86 laptops. New engine wrapping OpenVINO GenAI. Good for offloading 1-3B models while GPU handles larger ones. | +| Qualcomm mobile (Snapdragon 8 Gen 3/4) | **Design Needed** | 1-7B on phones via QNN SDK. Ties into Workstream 2 — inference on the phone itself rather than relaying to a server. | +| Raspberry Pi 5 | **Design Needed** | CPU-only via llama.cpp ARM NEON for 1-3B models. $100 entry point for hobbyists. Hailo-8L NPU is not viable for LLMs (vision-only architecture). | +| Unified hardware benchmark suite | **Design Needed** | Standardized benchmark that runs the same workloads across all supported hardware, producing comparable energy/latency/throughput/cost numbers. | + +### Long-term + +| Item | Maturity | Details | +|------|----------|---------| +| MediaTek Dimensity NPU | **Research-Stage** | Most aggressive mobile LLM push (45-50 TOPS) but closed NeuroPilot SDK. Monitor for SDK openness or third-party framework support. | +| Hailo-10 | **Research-Stage** | Hailo announced generative AI targeting for next-gen hardware. Watch for availability and transformer support. Current Hailo-8/8L is vision-only. | +| AMD XDNA2/3 NPU | **Research-Stage** | 50 TOPS but software stack (Ryzen AI SDK) is immature for LLMs. Revisit as AMD improves tooling. The AMD Ryzen AI iGPU item above is the practical AMD target today. | +| Intel Gaudi 3 / Falcon Shores | **Research-Stage** | 128GB HBM, datacenter-class. Gaudi product line being discontinued in favor of Falcon Shores architecture. Wait for clarity before investing. | +| RISC-V + NPU SoCs | **Research-Stage** | 3-5 years behind ARM/x86. Watch Sophgo SG2380 and similar. Not actionable for production use yet. | + +--- + +## Contributing + +Each workstream is independent — pick the one that matches your skills and interests: + +| Workstream | Skills needed | +|------------|--------------| +| 1. Continuous Operators | Python, async/scheduling, agent systems | +| 2. Mobile & Messaging | Node.js (Baileys bridge), Python, messaging platform APIs | +| 3. Secure Cloud Collaboration | Cryptography, TEE/confidential computing, distributed systems, ML | +| 4. Tutorials & Documentation | Technical writing, MkDocs, video production | +| 5. Hardware Breadth | Systems programming, hardware-specific SDKs (IPEX-LLM, OpenVINO, QNN), Rust | + +**To get started:** Look for items tagged **Ready** — these have clear scope and are ready for implementation. Items tagged **good first issue** are especially approachable. Open an issue or discussion on the repo to claim work or propose a design for **Design Needed** items. From 3ab601c717d29586454f4a48d07d4cabc9aee7b2 Mon Sep 17 00:00:00 2001 From: Micah Date: Sun, 15 Mar 2026 22:58:38 -0700 Subject: [PATCH 02/92] Fix additional memory retrieval issues beyond PR #74 - Add porter stemming to FTS5 tokenizer (tokenize='porter unicode61') so plurals like 'medications' match 'Medication' - Strip punctuation from FTS5 queries so question marks don't break MATCH - Expand tilde (~) in db_path before opening SQLite connection - Remove 'id' from FTS5 columns (UUIDs shouldn't be full-text indexed) - Add memory context injection to server chat_completions route (routes.py had no memory integration - UI chat was never grounded) - Fix agent name-to-ID resolution in agent_cmd.py (foreign key error) - Align Python SQLiteMemory schema with Rust backend --- .gitignore | 2 + .../openjarvis-tools/src/storage/sqlite.rs | 60 ++++++++++++++++--- src/openjarvis/cli/agent_cmd.py | 22 +++++++ src/openjarvis/server/routes.py | 45 ++++++++++++++ src/openjarvis/tools/storage/sqlite.py | 27 +-------- 5 files changed, 121 insertions(+), 35 deletions(-) diff --git a/.gitignore b/.gitignore index dcdb7447..8843988e 100644 --- a/.gitignore +++ b/.gitignore @@ -88,3 +88,5 @@ CLAUDE.md # Research output research_mining_* +.python-version +openjarvis-bugfix-spec-v2.md diff --git a/rust/crates/openjarvis-tools/src/storage/sqlite.rs b/rust/crates/openjarvis-tools/src/storage/sqlite.rs index bfa9f97c..abe70eee 100644 --- a/rust/crates/openjarvis-tools/src/storage/sqlite.rs +++ b/rust/crates/openjarvis-tools/src/storage/sqlite.rs @@ -15,6 +15,19 @@ pub struct SQLiteMemory { impl SQLiteMemory { pub fn new(db_path: &Path) -> Result { + // Expand leading ~ to the user's home directory + let db_path = if db_path.starts_with("~") { + let home = std::env::var("HOME").map_err(|_| { + OpenJarvisError::Io(std::io::Error::other( + "HOME environment variable not set", + )) + })?; + PathBuf::from(home).join(db_path.strip_prefix("~").unwrap()) + } else { + db_path.to_path_buf() + }; + let db_path = db_path.as_path(); + if let Some(parent) = db_path.parent() { std::fs::create_dir_all(parent).map_err(|e| { OpenJarvisError::Io(std::io::Error::other(e)) @@ -36,7 +49,7 @@ impl SQLiteMemory { created_at REAL DEFAULT (julianday('now')) ); CREATE VIRTUAL TABLE IF NOT EXISTS documents_fts USING fts5( - id, content, source, tokenize='unicode61' + content, source, tokenize='porter unicode61' );", ) .map_err(|e| { @@ -110,9 +123,10 @@ impl MemoryBackend for SQLiteMemory { )) })?; + let rowid = conn.last_insert_rowid(); conn.execute( - "INSERT INTO documents_fts (id, content, source) VALUES (?1, ?2, ?3)", - rusqlite::params![doc_id, content, source], + "INSERT INTO documents_fts (rowid, content, source) VALUES (?1, ?2, ?3)", + rusqlite::params![rowid, content, source], ) .map_err(|e| { OpenJarvisError::Io(std::io::Error::other( @@ -130,9 +144,13 @@ impl MemoryBackend for SQLiteMemory { ) -> Result, OpenJarvisError> { let conn = self.conn.lock(); - let words: Vec<&str> = query.split_whitespace().collect(); + let words: Vec = query + .split_whitespace() + .map(|w| w.trim_matches(|c: char| "?.,!;:'\"()[]{}/ ".contains(c)).to_string()) + .filter(|w| !w.is_empty()) + .collect(); let fts_query = if words.len() == 1 { - words[0].to_string() + words[0].clone() } else { words.join(" OR ") }; @@ -140,11 +158,11 @@ impl MemoryBackend for SQLiteMemory { let mut stmt = conn .prepare( "SELECT d.content, d.source, d.metadata, - bm25(documents_fts, 0.0, 1.0, 0.5) * -1 as score + bm25(documents_fts, 1.0, 0.5) * -1 as score FROM documents_fts f - JOIN documents d ON d.id = f.id + JOIN documents d ON d.rowid = f.rowid WHERE documents_fts MATCH ?1 - ORDER BY bm25(documents_fts, 0.0, 1.0, 0.5) + ORDER BY bm25(documents_fts, 1.0, 0.5) LIMIT ?2", ) .map_err(|e| { @@ -179,8 +197,9 @@ impl MemoryBackend for SQLiteMemory { fn delete(&self, doc_id: &str) -> Result { let conn = self.conn.lock(); + // Delete from FTS5 using the rowid from the documents table conn.execute( - "DELETE FROM documents_fts WHERE id = ?1", + "DELETE FROM documents_fts WHERE rowid = (SELECT rowid FROM documents WHERE id = ?1)", rusqlite::params![doc_id], ) .map_err(|e| { @@ -238,6 +257,29 @@ mod tests { let results = mem.retrieve("Rust programming", 5).unwrap(); assert!(!results.is_empty()); assert!(results[0].content.contains("Rust")); + assert!(results[0].score > 0.0, "score should be positive, got {}", results[0].score); + } + + #[test] + fn test_sqlite_porter_stemming() { + let mem = SQLiteMemory::in_memory().unwrap(); + mem.store("Medication list for patient", "health", None).unwrap(); + + // Plural form should match via porter stemming + let results = mem.retrieve("medications", 5).unwrap(); + assert!(!results.is_empty(), "porter stemming should match 'medications' to 'Medication'"); + assert!(results[0].score > 0.0); + } + + #[test] + fn test_sqlite_punctuation_stripping() { + let mem = SQLiteMemory::in_memory().unwrap(); + mem.store("Medication list for patient Micah", "health", None).unwrap(); + + // Natural language query with trailing punctuation should not break FTS5 + let results = mem.retrieve("What medications does Micah take?", 5).unwrap(); + assert!(!results.is_empty(), "query with punctuation should still return results"); + assert!(results[0].score > 0.0); } #[test] diff --git a/src/openjarvis/cli/agent_cmd.py b/src/openjarvis/cli/agent_cmd.py index 47c3ba94..0958ca4e 100644 --- a/src/openjarvis/cli/agent_cmd.py +++ b/src/openjarvis/cli/agent_cmd.py @@ -24,6 +24,26 @@ def _get_manager(): return AgentManager(db_path=db_path) +def _resolve_agent_id(manager, agent_id_or_name: str) -> str: + """Resolve an agent name or ID to the actual agent ID. + + Accepts either the hex ID or the agent name. Raises SystemExit if + no matching agent is found. + """ + # Try direct ID lookup first + agent = manager.get_agent(agent_id_or_name) + if agent is not None: + return agent["id"] + + # Fall back to name lookup + for a in manager.list_agents(include_archived=True): + if a["name"] == agent_id_or_name: + return a["id"] + + click.echo(f"Agent not found: {agent_id_or_name}", err=True) + raise SystemExit(1) + + @click.group("agents") def agent() -> None: """Manage persistent agents — create, inspect, chat, bind channels.""" @@ -680,6 +700,7 @@ def errors(): def ask(agent_id, message): """Ask an agent a question (immediate response).""" manager = _get_manager() + agent_id = _resolve_agent_id(manager, agent_id) manager.send_message(agent_id, message, mode="immediate") click.echo("Asking agent...") _, executor, _ = _get_scheduler_and_executor() @@ -701,6 +722,7 @@ def ask(agent_id, message): def instruct(agent_id, message): """Queue an instruction for the agent's next tick.""" manager = _get_manager() + agent_id = _resolve_agent_id(manager, agent_id) msg = manager.send_message(agent_id, message, mode="queued") click.echo(f"Instruction queued (ID: {msg['id'][:8]})") diff --git a/src/openjarvis/server/routes.py b/src/openjarvis/server/routes.py index 7838daac..b59ec919 100644 --- a/src/openjarvis/server/routes.py +++ b/src/openjarvis/server/routes.py @@ -46,6 +46,51 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request agent = getattr(request.app.state, "agent", None) model = request_body.model + # Inject memory context into messages before dispatching + config = getattr(request.app.state, "config", None) + memory_backend = getattr(request.app.state, "memory_backend", None) + if ( + config is not None + and memory_backend is not None + and config.agent.context_from_memory + and request_body.messages + ): + try: + from openjarvis.tools.storage.context import ContextConfig, inject_context + + # Extract query from the last user message + query_text = "" + for m in reversed(request_body.messages): + if m.role == "user" and m.content: + query_text = m.content + break + + if query_text: + messages = _to_messages(request_body.messages) + ctx_cfg = ContextConfig( + top_k=config.memory.context_top_k, + min_score=config.memory.context_min_score, + max_context_tokens=config.memory.context_max_tokens, + ) + enriched = inject_context( + query_text, messages, memory_backend, config=ctx_cfg, + ) + # Rebuild request messages from enriched Message objects + if len(enriched) > len(messages): + from openjarvis.server.models import ChatMessage + + new_msgs = [] + for msg in enriched: + new_msgs.append(ChatMessage( + role=msg.role.value, + content=msg.content, + name=msg.name, + tool_call_id=getattr(msg, "tool_call_id", None), + )) + request_body.messages = new_msgs + except Exception: + pass # Don't break chat if memory retrieval fails + if request_body.stream: bus = getattr(request.app.state, "bus", None) # Use the agent stream bridge only when tools are present (the diff --git a/src/openjarvis/tools/storage/sqlite.py b/src/openjarvis/tools/storage/sqlite.py index cd166643..708ca5a4 100644 --- a/src/openjarvis/tools/storage/sqlite.py +++ b/src/openjarvis/tools/storage/sqlite.py @@ -56,33 +56,8 @@ class SQLiteMemory(MemoryBackend): USING fts5( content, source, - content=documents, - content_rowid=rowid + tokenize='porter unicode61' ); - - CREATE TRIGGER IF NOT EXISTS documents_ai - AFTER INSERT ON documents BEGIN - INSERT INTO documents_fts(rowid, content, source) - VALUES (new.rowid, new.content, new.source); - END; - - CREATE TRIGGER IF NOT EXISTS documents_ad - AFTER DELETE ON documents BEGIN - INSERT INTO documents_fts( - documents_fts, rowid, content, source - ) - VALUES ('delete', old.rowid, old.content, old.source); - END; - - CREATE TRIGGER IF NOT EXISTS documents_au - AFTER UPDATE ON documents BEGIN - INSERT INTO documents_fts( - documents_fts, rowid, content, source - ) - VALUES ('delete', old.rowid, old.content, old.source); - INSERT INTO documents_fts(rowid, content, source) - VALUES (new.rowid, new.content, new.source); - END; """) def store( From 47a37fcca35674ba8d7c75a3f7b6ad203fff1810 Mon Sep 17 00:00:00 2001 From: Micah Date: Mon, 16 Mar 2026 00:33:59 -0700 Subject: [PATCH 03/92] fix: wire memory backend into server and agent executor The web UI chat endpoint and managed agent executor both skipped memory context injection entirely: - serve.py never created a memory backend - app.py received config but never stored it on app.state - executor.py never queried the FTS5 document store Now all three entry points (CLI, server, agents) inject retrieved context from the indexed knowledge base. --- src/openjarvis/agents/executor.py | 60 ++++++++++++++++++++++++++++--- src/openjarvis/cli/serve.py | 17 +++++++++ src/openjarvis/server/app.py | 3 ++ 3 files changed, 75 insertions(+), 5 deletions(-) diff --git a/src/openjarvis/agents/executor.py b/src/openjarvis/agents/executor.py index 1b78e9cc..9ea32487 100644 --- a/src/openjarvis/agents/executor.py +++ b/src/openjarvis/agents/executor.py @@ -218,19 +218,69 @@ class AgentExecutor: instruction = config.get("instruction", "") memory = agent.get("summary_memory", "") if instruction: - context = f"Standing instruction: {instruction}" + input_text = f"Standing instruction: {instruction}" if memory: - context += f"\n\nPrevious context: {memory}" + input_text += f"\n\nPrevious context: {memory}" else: - context = memory or "Continue your assigned task." + input_text = memory or "Continue your assigned task." pending = self._manager.get_pending_messages(agent["id"]) if pending: user_msgs = "\n".join(f"User: {m['content']}" for m in pending) - context = f"{context}\n\nNew instructions:\n{user_msgs}" + input_text = f"{input_text}\n\nNew instructions:\n{user_msgs}" for m in pending: self._manager.mark_message_delivered(m["id"]) - return agent_instance.run(context) + # Build AgentContext with memory results from FTS5 backend + from openjarvis.agents._stubs import AgentContext + + agent_ctx = AgentContext() + memory_results = [] + + if ( + self._system + and getattr(self._system, "memory_backend", None) + and getattr(self._system, "config", None) + and self._system.config.agent.context_from_memory + ): + try: + from openjarvis.tools.storage.context import ( + ContextConfig, + format_context, + ) + + sys_cfg = self._system.config + ctx_cfg = ContextConfig( + top_k=sys_cfg.memory.context_top_k, + min_score=sys_cfg.memory.context_min_score, + max_context_tokens=sys_cfg.memory.context_max_tokens, + ) + # Use pending user messages as query, fall back to instruction + query = "" + if pending: + query = " ".join(m["content"] for m in pending) + elif instruction: + query = instruction + + if query: + results = self._system.memory_backend.retrieve( + query, top_k=ctx_cfg.top_k, + ) + memory_results = [ + r for r in results if r.score >= ctx_cfg.min_score + ] + if memory_results: + # Prepend retrieved context to input for agents + # that don't inspect AgentContext.memory_results + retrieved = format_context(memory_results) + input_text = ( + f"Retrieved context from knowledge base:\n" + f"{retrieved}\n\n{input_text}" + ) + except Exception: + pass # Don't break agent tick if memory retrieval fails + + agent_ctx.memory_results = memory_results + return agent_instance.run(input_text, context=agent_ctx) def _build_error_detail(self, error: AgentTickError) -> dict[str, Any]: """Build structured error detail for trace metadata.""" diff --git a/src/openjarvis/cli/serve.py b/src/openjarvis/cli/serve.py index 5092acae..d4b68798 100644 --- a/src/openjarvis/cli/serve.py +++ b/src/openjarvis/cli/serve.py @@ -286,10 +286,27 @@ def serve( except Exception as exc: logger.debug("Agent scheduler init failed: %s", exc) + # Set up memory backend for context injection + memory_backend = None + if config.agent.context_from_memory: + try: + import openjarvis.tools.storage # noqa: F401 + from openjarvis.core.registry import MemoryRegistry + + mem_key = config.memory.default_backend + if MemoryRegistry.contains(mem_key): + memory_backend = MemoryRegistry.create( + mem_key, db_path=config.memory.db_path, + ) + console.print(" Memory: [cyan]active[/cyan]") + except Exception as exc: + logger.debug("Memory backend init failed: %s", exc) + app = create_app( engine, model_name, agent=agent, bus=bus, engine_name=engine_name, agent_name=agent_key or "", channel_bridge=channel_bridge, config=config, + memory_backend=memory_backend, speech_backend=speech_backend, agent_manager=agent_manager, agent_scheduler=agent_scheduler, diff --git a/src/openjarvis/server/app.py b/src/openjarvis/server/app.py index d098a739..dce15f4f 100644 --- a/src/openjarvis/server/app.py +++ b/src/openjarvis/server/app.py @@ -56,6 +56,7 @@ def create_app( agent_name: str = "", channel_bridge=None, config=None, + memory_backend=None, speech_backend=None, agent_manager=None, agent_scheduler=None, @@ -102,6 +103,8 @@ def create_app( getattr(agent, "agent_id", None) if agent else None ) app.state.channel_bridge = channel_bridge + app.state.config = config + app.state.memory_backend = memory_backend app.state.speech_backend = speech_backend app.state.agent_manager = agent_manager app.state.agent_scheduler = agent_scheduler From 7abd65da264a8ad01aec6f5a5c7dd15bf8b525f0 Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 17:45:55 +0000 Subject: [PATCH 04/92] feat: add MemoryManageTool and UserProfileManageTool for persistent personalization Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/core/config.py | 11 ++ src/openjarvis/tools/__init__.py | 9 ++ src/openjarvis/tools/memory_manage.py | 141 ++++++++++++++++++++ src/openjarvis/tools/user_profile_manage.py | 141 ++++++++++++++++++++ tests/tools/test_memory_manage.py | 49 +++++++ tests/tools/test_user_profile_manage.py | 38 ++++++ 6 files changed, 389 insertions(+) create mode 100644 src/openjarvis/tools/memory_manage.py create mode 100644 src/openjarvis/tools/user_profile_manage.py create mode 100644 tests/tools/test_memory_manage.py create mode 100644 tests/tools/test_user_profile_manage.py diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 1067c1a4..97f631e6 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -1055,6 +1055,16 @@ class AgentManagerConfig: db_path: str = str(DEFAULT_CONFIG_DIR / "agents.db") +@dataclass(slots=True) +class MemoryFilesConfig: + """Persistent memory-file paths and nudge settings.""" + + soul_path: str = "~/.openjarvis/SOUL.md" + memory_path: str = "~/.openjarvis/MEMORY.md" + user_path: str = "~/.openjarvis/USER.md" + nudge_interval: int = 10 + + @dataclass class JarvisConfig: """Top-level configuration for OpenJarvis.""" @@ -1079,6 +1089,7 @@ class JarvisConfig: speech: SpeechConfig = field(default_factory=SpeechConfig) optimize: OptimizeConfig = field(default_factory=OptimizeConfig) agent_manager: AgentManagerConfig = field(default_factory=AgentManagerConfig) + memory_files: MemoryFilesConfig = field(default_factory=MemoryFilesConfig) @property def memory(self) -> StorageConfig: diff --git a/src/openjarvis/tools/__init__.py b/src/openjarvis/tools/__init__.py index b95c9090..cfddc61b 100644 --- a/src/openjarvis/tools/__init__.py +++ b/src/openjarvis/tools/__init__.py @@ -77,4 +77,13 @@ try: except ImportError: pass +try: + import openjarvis.tools.memory_manage # noqa: F401 +except ImportError: + pass +try: + import openjarvis.tools.user_profile_manage # noqa: F401 +except ImportError: + pass + __all__ = ["BaseTool", "ToolExecutor", "ToolSpec"] diff --git a/src/openjarvis/tools/memory_manage.py b/src/openjarvis/tools/memory_manage.py new file mode 100644 index 00000000..ff2731fb --- /dev/null +++ b/src/openjarvis/tools/memory_manage.py @@ -0,0 +1,141 @@ +"""Manage persistent agent memory (MEMORY.md).""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from openjarvis.core.registry import ToolRegistry +from openjarvis.core.types import ToolResult +from openjarvis.tools._stubs import BaseTool, ToolSpec + + +@ToolRegistry.register("memory_manage") +class MemoryManageTool(BaseTool): + """Manage persistent agent memory (MEMORY.md).""" + + def __init__(self, memory_path: Path | str = "~/.openjarvis/MEMORY.md") -> None: + self._memory_path = Path(memory_path).expanduser() + + @property + def spec(self) -> ToolSpec: + return ToolSpec( + name="memory_manage", + description=( + "Read, add, update, or remove entries in persistent agent memory." + ), + parameters={ + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["read", "add", "update", "remove"], + "description": "Action to perform on memory.", + }, + "entry": { + "type": "string", + "description": ( + "The memory entry content (for add/update/remove)." + ), + }, + "new_entry": { + "type": "string", + "description": ( + "Replacement content (for update action only)." + ), + }, + }, + "required": ["action"], + }, + category="memory", + ) + + def execute(self, **params: Any) -> ToolResult: + action = params.get("action", "read") + entry = params.get("entry", "") + new_entry = params.get("new_entry", "") + if action == "read": + return self._read() + elif action == "add": + return self._add(entry) + elif action == "update": + return self._update(entry, new_entry) + elif action == "remove": + return self._remove(entry) + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Unknown action: {action}", + ) + + def _read(self) -> ToolResult: + content = "" + if self._memory_path.exists(): + content = self._memory_path.read_text() + return ToolResult( + tool_name=self.spec.name, + success=True, + content=content or "(empty)", + ) + + def _add(self, entry: str) -> ToolResult: + if not entry: + return ToolResult( + tool_name=self.spec.name, + success=False, + content="Entry cannot be empty.", + ) + self._memory_path.parent.mkdir(parents=True, exist_ok=True) + existing = ( + self._memory_path.read_text() if self._memory_path.exists() else "" + ) + self._memory_path.write_text(existing.rstrip() + f"\n- {entry}\n") + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Added: {entry}", + ) + + def _update(self, old: str, new: str) -> ToolResult: + if not self._memory_path.exists(): + return ToolResult( + tool_name=self.spec.name, + success=False, + content="Memory file does not exist.", + ) + text = self._memory_path.read_text() + if old not in text: + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Entry not found: {old}", + ) + self._memory_path.write_text(text.replace(old, new, 1)) + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Updated: {old} -> {new}", + ) + + def _remove(self, entry: str) -> ToolResult: + if not self._memory_path.exists(): + return ToolResult( + tool_name=self.spec.name, + success=False, + content="Memory file does not exist.", + ) + text = self._memory_path.read_text() + lines = text.split("\n") + new_lines = [ln for ln in lines if entry not in ln] + if len(new_lines) == len(lines): + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Entry not found: {entry}", + ) + self._memory_path.write_text("\n".join(new_lines)) + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Removed: {entry}", + ) diff --git a/src/openjarvis/tools/user_profile_manage.py b/src/openjarvis/tools/user_profile_manage.py new file mode 100644 index 00000000..67199a23 --- /dev/null +++ b/src/openjarvis/tools/user_profile_manage.py @@ -0,0 +1,141 @@ +"""Manage persistent user profile (USER.md).""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +from openjarvis.core.registry import ToolRegistry +from openjarvis.core.types import ToolResult +from openjarvis.tools._stubs import BaseTool, ToolSpec + + +@ToolRegistry.register("user_profile_manage") +class UserProfileManageTool(BaseTool): + """Manage persistent user profile (USER.md).""" + + def __init__(self, user_path: Path | str = "~/.openjarvis/USER.md") -> None: + self._user_path = Path(user_path).expanduser() + + @property + def spec(self) -> ToolSpec: + return ToolSpec( + name="user_profile_manage", + description=( + "Read, add, update, or remove entries in user profile." + ), + parameters={ + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["read", "add", "update", "remove"], + "description": "Action to perform on user profile.", + }, + "entry": { + "type": "string", + "description": ( + "The profile entry content (for add/update/remove)." + ), + }, + "new_entry": { + "type": "string", + "description": ( + "Replacement content (for update action only)." + ), + }, + }, + "required": ["action"], + }, + category="memory", + ) + + def execute(self, **params: Any) -> ToolResult: + action = params.get("action", "read") + entry = params.get("entry", "") + new_entry = params.get("new_entry", "") + if action == "read": + return self._read() + elif action == "add": + return self._add(entry) + elif action == "update": + return self._update(entry, new_entry) + elif action == "remove": + return self._remove(entry) + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Unknown action: {action}", + ) + + def _read(self) -> ToolResult: + content = "" + if self._user_path.exists(): + content = self._user_path.read_text() + return ToolResult( + tool_name=self.spec.name, + success=True, + content=content or "(empty)", + ) + + def _add(self, entry: str) -> ToolResult: + if not entry: + return ToolResult( + tool_name=self.spec.name, + success=False, + content="Entry cannot be empty.", + ) + self._user_path.parent.mkdir(parents=True, exist_ok=True) + existing = ( + self._user_path.read_text() if self._user_path.exists() else "" + ) + self._user_path.write_text(existing.rstrip() + f"\n- {entry}\n") + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Added: {entry}", + ) + + def _update(self, old: str, new: str) -> ToolResult: + if not self._user_path.exists(): + return ToolResult( + tool_name=self.spec.name, + success=False, + content="User profile file does not exist.", + ) + text = self._user_path.read_text() + if old not in text: + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Entry not found: {old}", + ) + self._user_path.write_text(text.replace(old, new, 1)) + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Updated: {old} -> {new}", + ) + + def _remove(self, entry: str) -> ToolResult: + if not self._user_path.exists(): + return ToolResult( + tool_name=self.spec.name, + success=False, + content="User profile file does not exist.", + ) + text = self._user_path.read_text() + lines = text.split("\n") + new_lines = [ln for ln in lines if entry not in ln] + if len(new_lines) == len(lines): + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Entry not found: {entry}", + ) + self._user_path.write_text("\n".join(new_lines)) + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Removed: {entry}", + ) diff --git a/tests/tools/test_memory_manage.py b/tests/tools/test_memory_manage.py new file mode 100644 index 00000000..27c83c7e --- /dev/null +++ b/tests/tools/test_memory_manage.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +import pytest +from pathlib import Path + + +@pytest.fixture +def memory_file(tmp_path: Path) -> Path: + p = tmp_path / "MEMORY.md" + p.write_text("## Knowledge\n\n- User prefers dark mode\n") + return p + + +def test_memory_read(memory_file: Path): + from openjarvis.tools.memory_manage import MemoryManageTool + + tool = MemoryManageTool(memory_path=memory_file) + result = tool.execute(action="read") + assert "dark mode" in result.content + + +def test_memory_add(memory_file: Path): + from openjarvis.tools.memory_manage import MemoryManageTool + + tool = MemoryManageTool(memory_path=memory_file) + result = tool.execute(action="add", entry="User works at Acme Corp") + assert result.success + assert "Acme Corp" in memory_file.read_text() + + +def test_memory_remove(memory_file: Path): + from openjarvis.tools.memory_manage import MemoryManageTool + + tool = MemoryManageTool(memory_path=memory_file) + tool.execute(action="add", entry="temporary fact") + result = tool.execute(action="remove", entry="temporary fact") + assert result.success + assert "temporary fact" not in memory_file.read_text() + + +def test_memory_create_if_missing(tmp_path: Path): + from openjarvis.tools.memory_manage import MemoryManageTool + + path = tmp_path / "MEMORY.md" + tool = MemoryManageTool(memory_path=path) + result = tool.execute(action="add", entry="new fact") + assert result.success + assert path.exists() + assert "new fact" in path.read_text() diff --git a/tests/tools/test_user_profile_manage.py b/tests/tools/test_user_profile_manage.py new file mode 100644 index 00000000..94ff51fb --- /dev/null +++ b/tests/tools/test_user_profile_manage.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +import pytest +from pathlib import Path + + +@pytest.fixture +def user_file(tmp_path: Path) -> Path: + p = tmp_path / "USER.md" + p.write_text("## User Profile\n\n- Name: Alice\n") + return p + + +def test_user_read(user_file: Path): + from openjarvis.tools.user_profile_manage import UserProfileManageTool + + tool = UserProfileManageTool(user_path=user_file) + result = tool.execute(action="read") + assert "Alice" in result.content + + +def test_user_add(user_file: Path): + from openjarvis.tools.user_profile_manage import UserProfileManageTool + + tool = UserProfileManageTool(user_path=user_file) + result = tool.execute(action="add", entry="Role: Engineer") + assert result.success + assert "Engineer" in user_file.read_text() + + +def test_user_update(user_file: Path): + from openjarvis.tools.user_profile_manage import UserProfileManageTool + + tool = UserProfileManageTool(user_path=user_file) + result = tool.execute(action="update", entry="Name: Alice", new_entry="Name: Bob") + assert result.success + assert "Bob" in user_file.read_text() + assert "Alice" not in user_file.read_text() From 978749e9f89e10f21136c7a6ad928272285c0490 Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 17:50:36 +0000 Subject: [PATCH 05/92] feat: add SystemPromptBuilder with frozen prefix and char limits Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/core/config.py | 12 +++ src/openjarvis/prompt/__init__.py | 1 + src/openjarvis/prompt/builder.py | 87 ++++++++++++++++++++++ tests/prompt/__init__.py | 0 tests/prompt/test_builder.py | 119 ++++++++++++++++++++++++++++++ 5 files changed, 219 insertions(+) create mode 100644 src/openjarvis/prompt/__init__.py create mode 100644 src/openjarvis/prompt/builder.py create mode 100644 tests/prompt/__init__.py create mode 100644 tests/prompt/test_builder.py diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 97f631e6..f6e9f4b7 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -1065,6 +1065,17 @@ class MemoryFilesConfig: nudge_interval: int = 10 +@dataclass(slots=True) +class SystemPromptConfig: + """Limits and strategy for system-prompt assembly.""" + + soul_max_chars: int = 4000 + memory_max_chars: int = 2500 + user_max_chars: int = 1500 + skill_desc_max_chars: int = 60 + truncation_strategy: str = "head_tail" + + @dataclass class JarvisConfig: """Top-level configuration for OpenJarvis.""" @@ -1090,6 +1101,7 @@ class JarvisConfig: optimize: OptimizeConfig = field(default_factory=OptimizeConfig) agent_manager: AgentManagerConfig = field(default_factory=AgentManagerConfig) memory_files: MemoryFilesConfig = field(default_factory=MemoryFilesConfig) + system_prompt: SystemPromptConfig = field(default_factory=SystemPromptConfig) @property def memory(self) -> StorageConfig: diff --git a/src/openjarvis/prompt/__init__.py b/src/openjarvis/prompt/__init__.py new file mode 100644 index 00000000..9d48db4f --- /dev/null +++ b/src/openjarvis/prompt/__init__.py @@ -0,0 +1 @@ +from __future__ import annotations diff --git a/src/openjarvis/prompt/builder.py b/src/openjarvis/prompt/builder.py new file mode 100644 index 00000000..dd844668 --- /dev/null +++ b/src/openjarvis/prompt/builder.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +from pathlib import Path +from typing import List, Optional, Tuple + +from openjarvis.core.config import MemoryFilesConfig, SystemPromptConfig + + +class SystemPromptBuilder: + """Assembles system prompts with frozen prefix for cache stability.""" + + def __init__( + self, + agent_template: str, + memory_files_config: Optional[MemoryFilesConfig] = None, + system_prompt_config: Optional[SystemPromptConfig] = None, + skill_index: Optional[List[Tuple[str, str]]] = None, + session_context: Optional[str] = None, + previous_state: Optional[str] = None, + ) -> None: + self._agent_template = agent_template + self._mf_config = memory_files_config or MemoryFilesConfig() + self._sp_config = system_prompt_config or SystemPromptConfig() + self._skill_index = skill_index or [] + self._session_context = session_context + self._previous_state = previous_state + self._frozen_prefix: Optional[str] = None + + def build(self) -> str: + if self._frozen_prefix is None: + self._frozen_prefix = self._build_frozen_prefix() + parts = [self._frozen_prefix] + if self._session_context: + parts.append(f"\n\n## Session Context\n\n{self._session_context}") + if self._previous_state: + parts.append(f"\n\n## Previous State\n\n{self._previous_state}") + return "".join(parts) + + def _build_frozen_prefix(self) -> str: + sections: list[str] = [] + sections.append(self._agent_template) + soul = self._load_file( + self._mf_config.soul_path, self._sp_config.soul_max_chars, + ) + if soul: + sections.append(f"## Agent Persona\n\n{soul}") + memory = self._load_file( + self._mf_config.memory_path, + self._sp_config.memory_max_chars, + ) + if memory: + sections.append(f"## Agent Memory\n\n{memory}") + user = self._load_file( + self._mf_config.user_path, self._sp_config.user_max_chars, + ) + if user: + sections.append(f"## User Profile\n\n{user}") + if self._skill_index: + skill_lines = [] + for name, desc in self._skill_index: + truncated = desc[: self._sp_config.skill_desc_max_chars] + if len(desc) > self._sp_config.skill_desc_max_chars: + truncated = truncated[:-3] + "..." + skill_lines.append(f"- **{name}**: {truncated}") + sections.append("## Available Skills\n\n" + "\n".join(skill_lines)) + return "\n\n".join(sections) + + def _load_file(self, path_str: str, max_chars: int) -> str: + path = Path(path_str).expanduser() + if not path.exists(): + return "" + content = path.read_text() + if len(content) <= max_chars: + return content + return self._truncate(content, max_chars) + + def _truncate(self, text: str, max_chars: int) -> str: + if self._sp_config.truncation_strategy == "head_tail": + head_size = int(max_chars * 0.7) + tail_size = int(max_chars * 0.2) + omitted = len(text) - head_size - tail_size + return ( + text[:head_size] + + f"\n\n[...truncated {omitted} chars...]\n\n" + + text[-tail_size:] + ) + return text[:max_chars] + "\n[...truncated...]" diff --git a/tests/prompt/__init__.py b/tests/prompt/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/prompt/test_builder.py b/tests/prompt/test_builder.py new file mode 100644 index 00000000..13cc20ac --- /dev/null +++ b/tests/prompt/test_builder.py @@ -0,0 +1,119 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from openjarvis.core.config import MemoryFilesConfig, SystemPromptConfig + + +@pytest.fixture +def memory_dir(tmp_path: Path) -> Path: + soul = tmp_path / "SOUL.md" + soul.write_text("You are a helpful research assistant.") + memory = tmp_path / "MEMORY.md" + memory.write_text("- User prefers concise answers\n- User is a data scientist") + user = tmp_path / "USER.md" + user.write_text("- Name: Alice\n- Role: ML Engineer") + return tmp_path + + +def test_build_frozen_prefix(memory_dir: Path): + from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( + agent_template="You are Jarvis.", + memory_files_config=MemoryFilesConfig( + soul_path=str(memory_dir / "SOUL.md"), + memory_path=str(memory_dir / "MEMORY.md"), + user_path=str(memory_dir / "USER.md"), + ), + system_prompt_config=SystemPromptConfig(), + ) + prompt = builder.build() + assert "Jarvis" in prompt + assert "helpful research assistant" in prompt + assert "concise answers" in prompt + assert "Alice" in prompt + + +def test_frozen_prefix_stability(memory_dir: Path): + from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( + agent_template="You are Jarvis.", + memory_files_config=MemoryFilesConfig( + soul_path=str(memory_dir / "SOUL.md"), + memory_path=str(memory_dir / "MEMORY.md"), + user_path=str(memory_dir / "USER.md"), + ), + system_prompt_config=SystemPromptConfig(), + ) + first = builder.build() + (memory_dir / "MEMORY.md").write_text("- CHANGED CONTENT") + second = builder.build() + assert first == second + + +def test_char_limit_truncation(memory_dir: Path): + from openjarvis.prompt.builder import SystemPromptBuilder + (memory_dir / "SOUL.md").write_text("x" * 10000) + builder = SystemPromptBuilder( + agent_template="You are Jarvis.", + memory_files_config=MemoryFilesConfig( + soul_path=str(memory_dir / "SOUL.md"), + memory_path=str(memory_dir / "MEMORY.md"), + user_path=str(memory_dir / "USER.md"), + ), + system_prompt_config=SystemPromptConfig(soul_max_chars=100), + ) + prompt = builder.build() + assert prompt.count("x") <= 100 + assert "truncated" in prompt.lower() + + +def test_skill_index_in_prompt(memory_dir: Path): + from openjarvis.prompt.builder import SystemPromptBuilder + skills = [("api_health_check", "Check API health across all endpoints")] + builder = SystemPromptBuilder( + agent_template="You are Jarvis.", + memory_files_config=MemoryFilesConfig( + soul_path=str(memory_dir / "SOUL.md"), + memory_path=str(memory_dir / "MEMORY.md"), + user_path=str(memory_dir / "USER.md"), + ), + system_prompt_config=SystemPromptConfig(), + skill_index=skills, + ) + prompt = builder.build() + assert "api_health_check" in prompt + assert "Check API health" in prompt + + +def test_dynamic_section_appended(memory_dir: Path): + from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( + agent_template="You are Jarvis.", + memory_files_config=MemoryFilesConfig( + soul_path=str(memory_dir / "SOUL.md"), + memory_path=str(memory_dir / "MEMORY.md"), + user_path=str(memory_dir / "USER.md"), + ), + system_prompt_config=SystemPromptConfig(), + session_context="Platform: CLI | Session: abc123", + ) + prompt = builder.build() + assert "Platform: CLI" in prompt + + +def test_missing_files_handled(tmp_path: Path): + from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( + agent_template="You are Jarvis.", + memory_files_config=MemoryFilesConfig( + soul_path=str(tmp_path / "missing_soul.md"), + memory_path=str(tmp_path / "missing_memory.md"), + user_path=str(tmp_path / "missing_user.md"), + ), + system_prompt_config=SystemPromptConfig(), + ) + prompt = builder.build() + assert "Jarvis" in prompt From 59b5295b207cd6331a9c73d4c04031108082ca72 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Mon, 16 Mar 2026 10:51:36 -0700 Subject: [PATCH 06/92] Update ROADMAP.md --- ROADMAP.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 07ec11ec..26d1b8e0 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,7 +2,7 @@ ## Overview -OpenJarvis studies personal AI through five core primitives — **Intelligence**, **Engine**, **Agents**, **Tools** (memory, retrieval, tool APIs), and **Learning** — running on personal hardware. This roadmap organizes development into five independent workstreams, each with items flowing from immediately actionable to exploratory research. +OpenJarvis studies personal AI through five core primitives — **Intelligence**, **Engine**, **Agents**, **Tools**, and **Learning** — running on personal hardware. This roadmap organizes development into five independent workstreams, each with items flowing from immediately actionable to exploratory research. ### How to Read This Roadmap @@ -89,9 +89,9 @@ Personal AI must be accessible from the devices people actually carry. OpenJarvi ## Workstream 3: Secure Cloud Collaboration -Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through three complementary approaches: **Minions-style collaborative inference** (local handles context, cloud handles reasoning), **TEE-based confidential computing** (cloud cannot see your data even during inference, inspired by [Tinfoil](https://tinfoil.sh)), and **secure multi-device coordination**. +Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through three complementary approaches: **Minions-style collaborative inference** (local handles context, cloud handles reasoning), **TEE-based confidential computing** (cloud cannot see your data even during inference), and **secure multi-device coordination**. -Key references: +Related References: - [Minions: Cost-Efficient Local-Cloud LLM Collaboration](https://github.com/HazyResearch/minions) - [TEE for Confidential AI Inference](https://openreview.net/forum?id=ey87M5iKcX) - [Tinfoil: Verifiably Private AI](https://tinfoil.sh) From 153435edb6e923ff20fad5f3eea66ef2405bb6fa Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 17:53:47 +0000 Subject: [PATCH 07/92] feat: add pluggable context compaction strategies via CompressionRegistry Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/core/config.py | 10 +++ src/openjarvis/core/registry.py | 5 ++ src/openjarvis/sessions/compression.py | 108 +++++++++++++++++++++++++ tests/conftest.py | 2 + tests/sessions/test_compression.py | 78 ++++++++++++++++++ 5 files changed, 203 insertions(+) create mode 100644 src/openjarvis/sessions/compression.py create mode 100644 tests/sessions/test_compression.py diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index f6e9f4b7..677390b3 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -1076,6 +1076,15 @@ class SystemPromptConfig: truncation_strategy: str = "head_tail" +@dataclass(slots=True) +class CompressionConfig: + """Configuration for context compression.""" + + enabled: bool = True + threshold: float = 0.50 + strategy: str = "session_consolidation" + + @dataclass class JarvisConfig: """Top-level configuration for OpenJarvis.""" @@ -1102,6 +1111,7 @@ class JarvisConfig: agent_manager: AgentManagerConfig = field(default_factory=AgentManagerConfig) memory_files: MemoryFilesConfig = field(default_factory=MemoryFilesConfig) system_prompt: SystemPromptConfig = field(default_factory=SystemPromptConfig) + compression: CompressionConfig = field(default_factory=CompressionConfig) @property def memory(self) -> StorageConfig: diff --git a/src/openjarvis/core/registry.py b/src/openjarvis/core/registry.py index f0f2becf..f0fd1735 100644 --- a/src/openjarvis/core/registry.py +++ b/src/openjarvis/core/registry.py @@ -141,10 +141,15 @@ class SpeechRegistry(RegistryBase[Any]): """Registry for speech backend implementations.""" +class CompressionRegistry(RegistryBase[Any]): + """Registry for context compression strategies.""" + + __all__ = [ "AgentRegistry", "BenchmarkRegistry", "ChannelRegistry", + "CompressionRegistry", "EngineRegistry", "LearningRegistry", "MemoryRegistry", diff --git a/src/openjarvis/sessions/compression.py b/src/openjarvis/sessions/compression.py new file mode 100644 index 00000000..c326c7b5 --- /dev/null +++ b/src/openjarvis/sessions/compression.py @@ -0,0 +1,108 @@ +from __future__ import annotations + +import json +from abc import ABC, abstractmethod +from dataclasses import replace +from typing import List + +from openjarvis.core.registry import CompressionRegistry +from openjarvis.core.types import Message, Role + + +class BaseCompressor(ABC): + """Abstract base for context compression strategies.""" + + @abstractmethod + def compress(self, messages: List[Message], threshold: float) -> List[Message]: + ... + + +@CompressionRegistry.register("session_consolidation") +class SessionConsolidation(BaseCompressor): + """Summarize oldest N% of turns, keep recent (100-N)%.""" + + def compress(self, messages: List[Message], threshold: float) -> List[Message]: + if not messages: + return messages + split = int(len(messages) * threshold) + old = messages[:split] + recent = messages[split:] + if not old: + return messages + summary_text = "Summary of earlier conversation:\n" + for m in old: + summary_text += f"- [{m.role}]: {m.content[:100]}...\n" + summary = Message(role=Role.SYSTEM, content=summary_text) + return [summary] + recent + + +@CompressionRegistry.register("rule_based_precompression") +class RuleBasedPrecompression(BaseCompressor): + """No LLM call. Strip boilerplate, truncate long outputs, collapse dupes.""" + + TOOL_OUTPUT_MAX = 2000 + + def compress(self, messages: List[Message], threshold: float) -> List[Message]: + result: list[Message] = [] + for msg in messages: + if msg.role == Role.TOOL and len(msg.content) > self.TOOL_OUTPUT_MAX: + suffix = "\n[...truncated]" + try: + parsed = json.loads(msg.content) + truncated = ( + json.dumps(parsed, indent=None)[ + : self.TOOL_OUTPUT_MAX + ] + + suffix + ) + except (json.JSONDecodeError, TypeError): + truncated = ( + msg.content[: self.TOOL_OUTPUT_MAX] + suffix + ) + result.append(replace(msg, content=truncated)) + else: + result.append(msg) + return result + + +@CompressionRegistry.register("model_summarization") +class ModelSummarization(BaseCompressor): + """LLM-based summarization using configured engine/model.""" + + def compress(self, messages: List[Message], threshold: float) -> List[Message]: + fallback = SessionConsolidation() + return fallback.compress(messages, threshold) + + +@CompressionRegistry.register("tiered_summaries") +class TieredSummaries(BaseCompressor): + """Progressive compression: L0 (full) -> L1 (paragraph) -> L2 (one-line).""" + + def compress(self, messages: List[Message], threshold: float) -> List[Message]: + if not messages: + return messages + n = len(messages) + l2_end = int(n * threshold * 0.5) + l1_end = int(n * threshold) + l2_msgs = messages[:l2_end] + l1_msgs = messages[l2_end:l1_end] + l0_msgs = messages[l1_end:] + result: list[Message] = [] + if l2_msgs: + one_liners = "; ".join( + f"{m.role}: {m.content[:50]}" for m in l2_msgs + ) + result.append(Message( + role=Role.SYSTEM, + content=f"[Oldest context] {one_liners}", + )) + if l1_msgs: + paragraphs = "\n".join( + f"- {m.role}: {m.content[:200]}" for m in l1_msgs + ) + result.append(Message( + role=Role.SYSTEM, + content=f"[Earlier context]\n{paragraphs}", + )) + result.extend(l0_msgs) + return result diff --git a/tests/conftest.py b/tests/conftest.py index d8d04e24..af05a657 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -13,6 +13,7 @@ from openjarvis.core.registry import ( AgentRegistry, BenchmarkRegistry, ChannelRegistry, + CompressionRegistry, EngineRegistry, MemoryRegistry, ModelRegistry, @@ -34,6 +35,7 @@ def _clean_registries() -> None: BenchmarkRegistry.clear() ChannelRegistry.clear() SpeechRegistry.clear() + CompressionRegistry.clear() reset_event_bus() diff --git a/tests/sessions/test_compression.py b/tests/sessions/test_compression.py new file mode 100644 index 00000000..d4509bfb --- /dev/null +++ b/tests/sessions/test_compression.py @@ -0,0 +1,78 @@ +from __future__ import annotations + +import pytest +from openjarvis.core.registry import CompressionRegistry +from openjarvis.core.types import Message, Role + + +@pytest.fixture(autouse=True) +def _register_compressors(): + """Re-register compression strategies after registry clear.""" + from openjarvis.sessions.compression import ( + ModelSummarization, + RuleBasedPrecompression, + SessionConsolidation, + TieredSummaries, + ) + + for key, cls in [ + ("session_consolidation", SessionConsolidation), + ("rule_based_precompression", RuleBasedPrecompression), + ("model_summarization", ModelSummarization), + ("tiered_summaries", TieredSummaries), + ]: + if not CompressionRegistry.contains(key): + CompressionRegistry.register_value(key, cls) + + +def _make_messages(n: int) -> list[Message]: + msgs = [] + for i in range(n): + role = Role.USER if i % 2 == 0 else Role.ASSISTANT + msgs.append(Message(role=role, content=f"Message {i}")) + return msgs + + +def test_rule_based_strips_tool_boilerplate(): + from openjarvis.sessions.compression import RuleBasedPrecompression + + compressor = RuleBasedPrecompression() + msgs = [ + Message(role=Role.ASSISTANT, content="Let me search."), + Message(role=Role.TOOL, content='{"results": [{"title": "Result 1", "snippet": "A very long snippet ' + "x" * 5000 + '"}]}'), + Message(role=Role.ASSISTANT, content="Based on the search, here is the answer."), + ] + result = compressor.compress(msgs, threshold=0.5) + total_len = sum(len(m.content) for m in result) + original_len = sum(len(m.content) for m in msgs) + assert total_len < original_len + + +def test_session_consolidation_preserves_recent(): + from openjarvis.sessions.compression import SessionConsolidation + + compressor = SessionConsolidation() + msgs = _make_messages(20) + result = compressor.compress(msgs, threshold=0.5) + assert len(result) < 20 + assert result[-1].content == msgs[-1].content + + +def test_compression_registry(): + from openjarvis.core.registry import CompressionRegistry + from openjarvis.sessions.compression import RuleBasedPrecompression + + assert CompressionRegistry.contains("rule_based_precompression") + cls = CompressionRegistry.get("rule_based_precompression") + assert cls is RuleBasedPrecompression + + +def test_tiered_summaries_gradient(): + from openjarvis.sessions.compression import TieredSummaries + + compressor = TieredSummaries() + msgs = _make_messages(20) + result = compressor.compress(msgs, threshold=0.5) + assert len(result) < 20 + # Recent messages should be preserved + assert result[-1].content == msgs[-1].content From c1e01b1e8b406d90b6c1c19a2bb9e20396f4affd Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 17:56:11 +0000 Subject: [PATCH 08/92] feat: add SkillManageTool for agent-authored procedural memory Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/core/config.py | 10 ++ src/openjarvis/skills/loader.py | 16 ++- src/openjarvis/tools/__init__.py | 5 + src/openjarvis/tools/skill_manage.py | 157 +++++++++++++++++++++++++++ tests/tools/test_skill_manage.py | 87 +++++++++++++++ 5 files changed, 274 insertions(+), 1 deletion(-) create mode 100644 src/openjarvis/tools/skill_manage.py create mode 100644 tests/tools/test_skill_manage.py diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 677390b3..3db5bc53 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -1085,6 +1085,15 @@ class CompressionConfig: strategy: str = "session_consolidation" +@dataclass(slots=True) +class SkillsConfig: + """Configuration for agent-authored procedural skills.""" + + skills_dir: str = "~/.openjarvis/skills/" + nudge_interval: int = 15 + auto_discover: bool = True + + @dataclass class JarvisConfig: """Top-level configuration for OpenJarvis.""" @@ -1112,6 +1121,7 @@ class JarvisConfig: memory_files: MemoryFilesConfig = field(default_factory=MemoryFilesConfig) system_prompt: SystemPromptConfig = field(default_factory=SystemPromptConfig) compression: CompressionConfig = field(default_factory=CompressionConfig) + skills: SkillsConfig = field(default_factory=SkillsConfig) @property def memory(self) -> StorageConfig: diff --git a/src/openjarvis/skills/loader.py b/src/openjarvis/skills/loader.py index 122fa51d..5786ea4a 100644 --- a/src/openjarvis/skills/loader.py +++ b/src/openjarvis/skills/loader.py @@ -104,4 +104,18 @@ def load_skill( return manifest -__all__ = ["load_skill"] +def discover_skills(directory: str | Path) -> list[SkillManifest]: + """Scan directory for TOML skill files and load them.""" + directory = Path(directory).expanduser() + if not directory.exists(): + return [] + manifests = [] + for toml_file in sorted(directory.glob("*.toml")): + try: + manifests.append(load_skill(toml_file)) + except Exception: + continue + return manifests + + +__all__ = ["load_skill", "discover_skills"] diff --git a/src/openjarvis/tools/__init__.py b/src/openjarvis/tools/__init__.py index cfddc61b..0c407a3a 100644 --- a/src/openjarvis/tools/__init__.py +++ b/src/openjarvis/tools/__init__.py @@ -86,4 +86,9 @@ try: except ImportError: pass +try: + import openjarvis.tools.skill_manage # noqa: F401 +except ImportError: + pass + __all__ = ["BaseTool", "ToolExecutor", "ToolSpec"] diff --git a/src/openjarvis/tools/skill_manage.py b/src/openjarvis/tools/skill_manage.py new file mode 100644 index 00000000..adc1b894 --- /dev/null +++ b/src/openjarvis/tools/skill_manage.py @@ -0,0 +1,157 @@ +"""SkillManageTool — create, list, load, or delete agent-authored skills.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any, List + +from openjarvis.core.registry import ToolRegistry +from openjarvis.core.types import ToolResult +from openjarvis.tools._stubs import BaseTool, ToolSpec + + +@ToolRegistry.register("skill_manage") +class SkillManageTool(BaseTool): + """Manage agent-authored procedural skills.""" + + def __init__(self, skills_dir: Path | str = "~/.openjarvis/skills/") -> None: + self._skills_dir = Path(skills_dir).expanduser() + + @property + def spec(self) -> ToolSpec: + return ToolSpec( + name="skill_manage", + description="Create, list, load, or delete agent-authored skills.", + parameters={ + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["create", "list", "load", "delete"], + "description": "Action to perform.", + }, + "name": { + "type": "string", + "description": "Skill name (for create/load/delete).", + }, + "description": { + "type": "string", + "description": "Skill description (for create).", + }, + "steps": { + "type": "array", + "description": ( + "List of step dicts with tool_name and optional" + " arguments_template (for create)." + ), + }, + }, + "required": ["action"], + }, + category="skill", + ) + + def execute(self, **params: Any) -> ToolResult: + action = params.get("action", "list") + name = params.get("name", "") + if action == "create": + return self._create( + name, params.get("description", ""), params.get("steps", []) + ) + elif action == "list": + return self._list() + elif action == "load": + return self._load(name) + elif action == "delete": + return self._delete(name) + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Unknown action: {action}", + ) + + def _create( + self, name: str, description: str, steps: List[dict] + ) -> ToolResult: + if not name: + return ToolResult( + tool_name=self.spec.name, + success=False, + content="Skill name is required.", + ) + self._skills_dir.mkdir(parents=True, exist_ok=True) + path = self._skills_dir / f"{name}.toml" + lines = [ + "[skill]", + f'name = "{name}"', + f'description = "{description}"', + "", + ] + for step in steps: + lines.append("[[skill.steps]]") + lines.append(f'tool_name = "{step.get("tool_name", "")}"') + if "arguments_template" in step: + lines.append( + f"arguments_template = '{step['arguments_template']}'" + ) + if "output_key" in step: + lines.append(f'output_key = "{step["output_key"]}"') + lines.append("") + path.write_text("\n".join(lines)) + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Created skill: {name}", + ) + + def _list(self) -> ToolResult: + if not self._skills_dir.exists(): + return ToolResult( + tool_name=self.spec.name, + success=True, + content="No skills directory found.", + ) + skills = [] + for f in sorted(self._skills_dir.glob("*.toml")): + skills.append(f.stem) + if not skills: + return ToolResult( + tool_name=self.spec.name, + success=True, + content="No skills found.", + ) + return ToolResult( + tool_name=self.spec.name, + success=True, + content="Available skills:\n" + + "\n".join(f"- {s}" for s in skills), + ) + + def _load(self, name: str) -> ToolResult: + path = self._skills_dir / f"{name}.toml" + if not path.exists(): + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Skill not found: {name}", + ) + return ToolResult( + tool_name=self.spec.name, + success=True, + content=path.read_text(), + ) + + def _delete(self, name: str) -> ToolResult: + path = self._skills_dir / f"{name}.toml" + if not path.exists(): + return ToolResult( + tool_name=self.spec.name, + success=False, + content=f"Skill not found: {name}", + ) + path.unlink() + return ToolResult( + tool_name=self.spec.name, + success=True, + content=f"Deleted skill: {name}", + ) diff --git a/tests/tools/test_skill_manage.py b/tests/tools/test_skill_manage.py new file mode 100644 index 00000000..7987938d --- /dev/null +++ b/tests/tools/test_skill_manage.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +import pytest +from pathlib import Path + + +@pytest.fixture +def skills_dir(tmp_path: Path) -> Path: + d = tmp_path / "skills" + d.mkdir() + return d + + +def test_skill_create(skills_dir: Path): + from openjarvis.tools.skill_manage import SkillManageTool + + tool = SkillManageTool(skills_dir=skills_dir) + result = tool.execute( + action="create", + name="api_health", + description="Check API health", + steps=[ + { + "tool_name": "http_request", + "arguments_template": '{"url": "{endpoint}/health"}', + } + ], + ) + assert result.success + assert (skills_dir / "api_health.toml").exists() + + +def test_skill_list(skills_dir: Path): + from openjarvis.tools.skill_manage import SkillManageTool + + tool = SkillManageTool(skills_dir=skills_dir) + tool.execute( + action="create", + name="skill_a", + description="Skill A", + steps=[{"tool_name": "calculator"}], + ) + tool.execute( + action="create", + name="skill_b", + description="Skill B", + steps=[{"tool_name": "calculator"}], + ) + result = tool.execute(action="list") + assert "skill_a" in result.content + assert "skill_b" in result.content + + +def test_skill_delete(skills_dir: Path): + from openjarvis.tools.skill_manage import SkillManageTool + + tool = SkillManageTool(skills_dir=skills_dir) + tool.execute( + action="create", + name="temp_skill", + description="Temp", + steps=[{"tool_name": "calculator"}], + ) + assert (skills_dir / "temp_skill.toml").exists() + result = tool.execute(action="delete", name="temp_skill") + assert result.success + assert not (skills_dir / "temp_skill.toml").exists() + + +def test_skill_load(skills_dir: Path): + from openjarvis.tools.skill_manage import SkillManageTool + + tool = SkillManageTool(skills_dir=skills_dir) + tool.execute( + action="create", + name="my_skill", + description="My skill desc", + steps=[ + { + "tool_name": "web_search", + "arguments_template": '{"q": "test"}', + } + ], + ) + result = tool.execute(action="load", name="my_skill") + assert "web_search" in result.content + assert "My skill desc" in result.content From 4cc16972ecbc3a57cd7dbb98857c1250756f093b Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 17:57:47 +0000 Subject: [PATCH 09/92] feat: add credential stripping, tool output wrapping, and severity policy Co-Authored-By: Claude Opus 4.6 (1M context) --- .../security/credential_stripper.py | 31 +++++++++++++ src/openjarvis/security/severity_policy.py | 22 ++++++++++ tests/security/test_credential_stripper.py | 44 +++++++++++++++++++ tests/security/test_severity_policy.py | 29 ++++++++++++ 4 files changed, 126 insertions(+) create mode 100644 src/openjarvis/security/credential_stripper.py create mode 100644 src/openjarvis/security/severity_policy.py create mode 100644 tests/security/test_credential_stripper.py create mode 100644 tests/security/test_severity_policy.py diff --git a/src/openjarvis/security/credential_stripper.py b/src/openjarvis/security/credential_stripper.py new file mode 100644 index 00000000..661b4c96 --- /dev/null +++ b/src/openjarvis/security/credential_stripper.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import re +from typing import List, Tuple + +_CREDENTIAL_PATTERNS: List[Tuple[str, re.Pattern[str]]] = [ + ("api_key", re.compile(r"sk-[a-zA-Z0-9_-]{20,}")), + ("aws_key", re.compile(r"AKIA[0-9A-Z]{16}")), + ("github_token", re.compile(r"ghp_[a-zA-Z0-9]{36}")), + ("github_token", re.compile(r"gho_[a-zA-Z0-9]{36}")), + ("slack_token", re.compile(r"xoxb-[0-9A-Za-z\-]+")), + ("bearer_token", re.compile(r"Bearer\s+[a-zA-Z0-9_\-.]{20,}")), +] + + +class CredentialStripper: + """Redacts credentials from text using compiled regex patterns.""" + + def __init__(self) -> None: + self._patterns = _CREDENTIAL_PATTERNS + + def strip(self, text: str) -> str: + for label, pattern in self._patterns: + text = pattern.sub(f"[REDACTED:{label}]", text) + return text + + +def wrap_tool_output(tool_name: str, content: str, success: bool = True) -> str: + status = "success" if success else "error" + header = f'' + return f"{header}\n{content}\n" diff --git a/src/openjarvis/security/severity_policy.py b/src/openjarvis/security/severity_policy.py new file mode 100644 index 00000000..4e8c93f9 --- /dev/null +++ b/src/openjarvis/security/severity_policy.py @@ -0,0 +1,22 @@ +from __future__ import annotations + +from openjarvis.security.types import ThreatLevel + +_DEFAULT_ACTIONS = { + ThreatLevel.CRITICAL: "block", + ThreatLevel.HIGH: "warn", + ThreatLevel.MEDIUM: "sanitize", + ThreatLevel.LOW: "log", +} + + +class SeverityPolicy: + """Maps ThreatLevel to configurable actions (block/warn/sanitize/log).""" + + def __init__(self, overrides: dict[ThreatLevel, str] | None = None) -> None: + self._actions = dict(_DEFAULT_ACTIONS) + if overrides: + self._actions.update(overrides) + + def action_for(self, level: ThreatLevel) -> str: + return self._actions.get(level, "log") diff --git a/tests/security/test_credential_stripper.py b/tests/security/test_credential_stripper.py new file mode 100644 index 00000000..899ea1b4 --- /dev/null +++ b/tests/security/test_credential_stripper.py @@ -0,0 +1,44 @@ +from __future__ import annotations + + +def test_strips_openai_key(): + from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() + text = "Error: auth failed with key sk-proj-abc123def456ghi789jkl012mno345pqr678stu901vwx234" + result = stripper.strip(text) + assert "sk-proj-" not in result + assert "[REDACTED:" in result + + +def test_strips_aws_key(): + from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() + text = "Using credentials AKIAIOSFODNN7EXAMPLE for access" + result = stripper.strip(text) + assert "AKIA" not in result + assert "[REDACTED:" in result + + +def test_strips_github_token(): + from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() + text = "Token: ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghij" + result = stripper.strip(text) + assert "ghp_" not in result + + +def test_preserves_normal_text(): + from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() + text = "The function returned 42 results." + result = stripper.strip(text) + assert result == text + + +def test_tool_output_wrapping(): + from openjarvis.security.credential_stripper import wrap_tool_output + content = "Search results: found 3 items" + wrapped = wrap_tool_output("web_search", content, success=True) + assert '' in wrapped + assert "Search results" in wrapped + assert "" in wrapped diff --git a/tests/security/test_severity_policy.py b/tests/security/test_severity_policy.py new file mode 100644 index 00000000..907ca826 --- /dev/null +++ b/tests/security/test_severity_policy.py @@ -0,0 +1,29 @@ +from __future__ import annotations + + +def test_severity_policy_block(): + from openjarvis.security.severity_policy import SeverityPolicy + from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() + assert policy.action_for(ThreatLevel.CRITICAL) == "block" + + +def test_severity_policy_warn(): + from openjarvis.security.severity_policy import SeverityPolicy + from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() + assert policy.action_for(ThreatLevel.HIGH) == "warn" + + +def test_severity_policy_sanitize(): + from openjarvis.security.severity_policy import SeverityPolicy + from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() + assert policy.action_for(ThreatLevel.MEDIUM) == "sanitize" + + +def test_severity_policy_log(): + from openjarvis.security.severity_policy import SeverityPolicy + from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() + assert policy.action_for(ThreatLevel.LOW) == "log" From 6371e7521d40db4b32ac12321b6698e2d2d93f74 Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 18:01:13 +0000 Subject: [PATCH 10/92] feat: add warn-before-block escalation to LoopGuard Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/agents/loop_guard.py | 61 +++++++++++++++++++-------- tests/agents/test_loop_guard.py | 1 + tests/agents/test_loop_guard_warn.py | 62 ++++++++++++++++++++++++++++ 3 files changed, 107 insertions(+), 17 deletions(-) create mode 100644 tests/agents/test_loop_guard_warn.py diff --git a/src/openjarvis/agents/loop_guard.py b/src/openjarvis/agents/loop_guard.py index 96af3b98..cd19983d 100644 --- a/src/openjarvis/agents/loop_guard.py +++ b/src/openjarvis/agents/loop_guard.py @@ -18,6 +18,7 @@ class LoopGuardConfig: ping_pong_window: int = 6 # detect A-B-A-B cycling poll_tool_budget: int = 5 # max calls to same polling tool max_context_messages: int = 100 # context overflow threshold + warn_before_block: bool = True # warn on first cycle, block on second @dataclass(slots=True) @@ -25,6 +26,7 @@ class LoopVerdict: """Result of a loop guard check.""" blocked: bool = False reason: str = "" + warned: bool = False class LoopGuard: @@ -47,26 +49,49 @@ class LoopGuard: self._tool_sequence: deque[str] = deque(maxlen=config.ping_pong_window * 2) # Track per-tool call counts (for polling budget) self._per_tool_counts: dict[str, int] = {} + # Track cycle keys that have already been warned (for warn-before-block) + self._warned_cycles: set[str] = set() - from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() - self._rust_impl = _rust.LoopGuard( - max_identical=config.max_identical_calls, - max_ping_pong=( - config.ping_pong_window // 2 - if config.ping_pong_window > 1 - else 2 - ), - poll_budget=config.poll_tool_budget, - ) + try: + from openjarvis._rust_bridge import get_rust_module + _rust = get_rust_module() + self._rust_impl = _rust.LoopGuard( + max_identical=config.max_identical_calls, + max_ping_pong=( + config.ping_pong_window // 2 + if config.ping_pong_window > 1 + else 2 + ), + poll_budget=config.poll_tool_budget, + ) + except Exception: + self._rust_impl = None def check_call(self, tool_name: str, arguments: str) -> LoopVerdict: """Check whether a tool call should proceed or be blocked.""" - reason = self._rust_impl.check(tool_name, arguments) - if reason is not None: - self._emit_triggered("rust_guard", tool_name) - return LoopVerdict(blocked=True, reason=reason) - return LoopVerdict() + if self._rust_impl is not None: + rust_result = self._rust_impl.check(tool_name, arguments) + # Support both raw Rust return (str | None) and LoopVerdict + if isinstance(rust_result, LoopVerdict): + verdict = rust_result + elif rust_result is not None: + self._emit_triggered("rust_guard", tool_name) + verdict = LoopVerdict(blocked=True, reason=rust_result) + else: + verdict = LoopVerdict() + else: + verdict = self._python_check(tool_name, arguments) + + # Wrap with warn-before-block logic + if verdict.blocked and self._config.warn_before_block: + cycle_key = verdict.reason + if cycle_key not in self._warned_cycles: + self._warned_cycles.add(cycle_key) + return LoopVerdict(blocked=False, warned=True, reason=verdict.reason) + return verdict + + def _python_check(self, tool_name: str, arguments: str) -> LoopVerdict: + """Pure-Python fallback when Rust backend is not available.""" # 1. Hash tracking — identical calls call_hash = hashlib.sha256( f"{tool_name}:{arguments}".encode() @@ -198,7 +223,9 @@ class LoopGuard: self._call_counts.clear() self._tool_sequence.clear() self._per_tool_counts.clear() - self._rust_impl.reset() + self._warned_cycles.clear() + if self._rust_impl is not None: + self._rust_impl.reset() def _detect_ping_pong(self) -> bool: """Detect repeating patterns in tool call sequence.""" diff --git a/tests/agents/test_loop_guard.py b/tests/agents/test_loop_guard.py index 161add10..69ea11f6 100644 --- a/tests/agents/test_loop_guard.py +++ b/tests/agents/test_loop_guard.py @@ -8,6 +8,7 @@ from openjarvis.core.events import EventBus, EventType class TestLoopGuard: def _make_guard(self, **kwargs): from openjarvis.agents.loop_guard import LoopGuard, LoopGuardConfig + kwargs.setdefault("warn_before_block", False) config = LoopGuardConfig(**kwargs) bus = EventBus(record_history=True) return LoopGuard(config, bus=bus), bus diff --git a/tests/agents/test_loop_guard_warn.py b/tests/agents/test_loop_guard_warn.py new file mode 100644 index 00000000..534082ad --- /dev/null +++ b/tests/agents/test_loop_guard_warn.py @@ -0,0 +1,62 @@ +from __future__ import annotations + +from unittest.mock import MagicMock + +from openjarvis.agents.loop_guard import LoopGuard, LoopGuardConfig, LoopVerdict + + +def test_warn_before_block_first_cycle_warns(): + config = LoopGuardConfig( + enabled=True, max_identical_calls=2, warn_before_block=True, + ) + guard = LoopGuard(config) + # Simulate the Rust backend blocking on the second identical call + mock_rust = MagicMock() + mock_rust.check.side_effect = [ + LoopVerdict(blocked=False, reason=""), + LoopVerdict(blocked=True, reason="identical_calls:search"), + ] + guard._rust_impl = mock_rust + guard.check_call("search", '{"q": "test"}') + v2 = guard.check_call("search", '{"q": "test"}') + assert not v2.blocked + assert v2.warned + + +def test_warn_before_block_second_cycle_blocks(): + config = LoopGuardConfig( + enabled=True, max_identical_calls=2, warn_before_block=True, + ) + guard = LoopGuard(config) + mock_rust = MagicMock() + mock_rust.check.side_effect = [ + LoopVerdict(blocked=False, reason=""), + LoopVerdict(blocked=True, reason="identical_calls:search"), + LoopVerdict(blocked=False, reason=""), + LoopVerdict(blocked=True, reason="identical_calls:search"), + ] + guard._rust_impl = mock_rust + guard.check_call("search", '{"q": "test"}') + v_warn = guard.check_call("search", '{"q": "test"}') + assert v_warn.warned and not v_warn.blocked + guard.check_call("search", '{"q": "test"}') + v_block = guard.check_call("search", '{"q": "test"}') + assert v_block.blocked + assert not v_block.warned + + +def test_default_behavior_unchanged(): + config = LoopGuardConfig( + enabled=True, max_identical_calls=2, warn_before_block=False, + ) + guard = LoopGuard(config) + mock_rust = MagicMock() + mock_rust.check.side_effect = [ + LoopVerdict(blocked=False, reason=""), + LoopVerdict(blocked=True, reason="identical_calls:search"), + ] + guard._rust_impl = mock_rust + guard.check_call("search", '{"q": "test"}') + v = guard.check_call("search", '{"q": "test"}') + assert v.blocked + assert not v.warned From 0d5be075f61c81561d2a630c65168ac55a0538c4 Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 18:02:28 +0000 Subject: [PATCH 11/92] feat: add AgentExecutor.run_ephemeral() for one-shot agent turns Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/agents/executor.py | 18 +++++++++++ tests/agents/test_executor_ephemeral.py | 43 +++++++++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 tests/agents/test_executor_ephemeral.py diff --git a/src/openjarvis/agents/executor.py b/src/openjarvis/agents/executor.py index 1b78e9cc..986f9975 100644 --- a/src/openjarvis/agents/executor.py +++ b/src/openjarvis/agents/executor.py @@ -47,6 +47,24 @@ class AgentExecutor: """Deferred system injection — called after JarvisSystem is constructed.""" self._system = system + def run_ephemeral( + self, + agent_type: str, + system_prompt: str, + input_text: str, + tools: list[str] | None = None, + ) -> Any: + """Run a one-shot agent turn with no lifecycle tracking.""" + from openjarvis.core.registry import AgentRegistry + + agent_cls = AgentRegistry.get(agent_type) + agent = agent_cls( + engine=getattr(self._manager, '_engine', None), + system_prompt=system_prompt, + bus=self._bus, + ) + return agent.run(input_text) + def execute_tick(self, agent_id: str) -> None: """Run one tick for the given agent. diff --git a/tests/agents/test_executor_ephemeral.py b/tests/agents/test_executor_ephemeral.py new file mode 100644 index 00000000..537181dd --- /dev/null +++ b/tests/agents/test_executor_ephemeral.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +from unittest.mock import MagicMock, patch + + +def test_run_ephemeral_creates_and_runs_agent(): + from openjarvis.agents.executor import AgentExecutor + + manager = MagicMock() + executor = AgentExecutor(manager=manager, event_bus=MagicMock()) + + mock_agent_cls = MagicMock() + mock_agent_instance = MagicMock() + mock_agent_instance.run.return_value = MagicMock(content="Flushed 3 memories.") + mock_agent_cls.return_value = mock_agent_instance + + with patch("openjarvis.core.registry.AgentRegistry.get", return_value=mock_agent_cls): + result = executor.run_ephemeral( + agent_type="simple", + system_prompt="Save important context.", + input_text="Review and flush.", + ) + assert mock_agent_instance.run.called + + +def test_run_ephemeral_passes_input(): + from openjarvis.agents.executor import AgentExecutor + + manager = MagicMock() + executor = AgentExecutor(manager=manager, event_bus=MagicMock()) + + mock_agent_cls = MagicMock() + mock_agent_instance = MagicMock() + mock_agent_instance.run.return_value = MagicMock(content="Done.") + mock_agent_cls.return_value = mock_agent_instance + + with patch("openjarvis.core.registry.AgentRegistry.get", return_value=mock_agent_cls): + executor.run_ephemeral( + agent_type="simple", + system_prompt="Test prompt.", + input_text="Hello world", + ) + mock_agent_instance.run.assert_called_once_with("Hello world") From 1359c223ec6a1e65d71a2acdd7282ed39c66a7e4 Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 18:37:41 +0000 Subject: [PATCH 12/92] feat: wire SystemPromptBuilder into BaseAgent._build_messages() Add optional prompt_builder parameter to BaseAgent. When provided, _build_messages() uses builder.build() output instead of raw system_prompt. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/agents/_stubs.py | 12 +++++++-- .../test_system_prompt_builder_integration.py | 27 +++++++++++++++++++ 2 files changed, 37 insertions(+), 2 deletions(-) create mode 100644 tests/agents/test_system_prompt_builder_integration.py diff --git a/src/openjarvis/agents/_stubs.py b/src/openjarvis/agents/_stubs.py index 2f8e3011..f25fcc9c 100644 --- a/src/openjarvis/agents/_stubs.py +++ b/src/openjarvis/agents/_stubs.py @@ -65,12 +65,14 @@ class BaseAgent(ABC): bus: Optional[EventBus] = None, temperature: float = 0.7, max_tokens: int = 1024, + prompt_builder: Optional[Any] = None, ) -> None: self._engine = engine self._model = model self._bus = bus self._temperature = temperature self._max_tokens = max_tokens + self._prompt_builder = prompt_builder # ------------------------------------------------------------------ # Concrete helpers @@ -104,8 +106,14 @@ class BaseAgent(ABC): conversation messages, and finally the user input. """ messages: list[Message] = [] - if system_prompt: - messages.append(Message(role=Role.SYSTEM, content=system_prompt)) + if self._prompt_builder is not None: + effective_system_prompt = self._prompt_builder.build() + elif system_prompt: + effective_system_prompt = system_prompt + else: + effective_system_prompt = None + if effective_system_prompt: + messages.append(Message(role=Role.SYSTEM, content=effective_system_prompt)) if context and context.conversation.messages: messages.extend(context.conversation.messages) messages.append(Message(role=Role.USER, content=input)) diff --git a/tests/agents/test_system_prompt_builder_integration.py b/tests/agents/test_system_prompt_builder_integration.py new file mode 100644 index 00000000..bb4cce2f --- /dev/null +++ b/tests/agents/test_system_prompt_builder_integration.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +from pathlib import Path +from openjarvis.core.config import MemoryFilesConfig, SystemPromptConfig + + +def test_base_agent_uses_builder(tmp_path: Path): + soul = tmp_path / "SOUL.md" + soul.write_text("I am Jarvis.") + memory = tmp_path / "MEMORY.md" + memory.write_text("- User likes Python") + + from openjarvis.prompt.builder import SystemPromptBuilder + + builder = SystemPromptBuilder( + agent_template="You are a helpful assistant.", + memory_files_config=MemoryFilesConfig( + soul_path=str(soul), + memory_path=str(memory), + user_path=str(tmp_path / "USER.md"), + ), + system_prompt_config=SystemPromptConfig(), + ) + prompt = builder.build() + assert "Jarvis" in prompt + assert "Python" in prompt + assert "helpful assistant" in prompt From 5df86886270c55f83939f178f0bec551890bebcd Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 18:40:07 +0000 Subject: [PATCH 13/92] feat: add GatewayDaemon, SessionExpiryHook, and service file generation Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/daemon/__init__.py | 1 + src/openjarvis/daemon/gateway.py | 39 +++++++++++++++++ src/openjarvis/daemon/service.py | 57 +++++++++++++++++++++++++ src/openjarvis/daemon/session_expiry.py | 34 +++++++++++++++ tests/daemon/__init__.py | 0 tests/daemon/test_gateway.py | 18 ++++++++ tests/daemon/test_session_expiry.py | 28 ++++++++++++ 7 files changed, 177 insertions(+) create mode 100644 src/openjarvis/daemon/__init__.py create mode 100644 src/openjarvis/daemon/gateway.py create mode 100644 src/openjarvis/daemon/service.py create mode 100644 src/openjarvis/daemon/session_expiry.py create mode 100644 tests/daemon/__init__.py create mode 100644 tests/daemon/test_gateway.py create mode 100644 tests/daemon/test_session_expiry.py diff --git a/src/openjarvis/daemon/__init__.py b/src/openjarvis/daemon/__init__.py new file mode 100644 index 00000000..9d48db4f --- /dev/null +++ b/src/openjarvis/daemon/__init__.py @@ -0,0 +1 @@ +from __future__ import annotations diff --git a/src/openjarvis/daemon/gateway.py b/src/openjarvis/daemon/gateway.py new file mode 100644 index 00000000..189a8dc7 --- /dev/null +++ b/src/openjarvis/daemon/gateway.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +from typing import Any, Optional + + +class GatewayDaemon: + """Composes channels, sessions, agents, and scheduler into a daemon.""" + + def __init__( + self, + config: Any = None, + session_store: Any = None, + agent_manager: Any = None, + agent_scheduler: Any = None, + event_bus: Any = None, + ) -> None: + self._config = config + self._session_store = session_store + self._agent_manager = agent_manager + self._agent_scheduler = agent_scheduler + self._event_bus = event_bus + self._running = False + + @staticmethod + def session_key( + platform: str, + chat_type: str, + chat_id: str, + thread_id: Optional[str], + ) -> str: + return f"agent:main:{platform}:{chat_type}:{chat_id}:{thread_id}" + + def start(self) -> None: + """Start the daemon (foreground).""" + self._running = True + + def stop(self) -> None: + """Stop the daemon.""" + self._running = False diff --git a/src/openjarvis/daemon/service.py b/src/openjarvis/daemon/service.py new file mode 100644 index 00000000..3c88809b --- /dev/null +++ b/src/openjarvis/daemon/service.py @@ -0,0 +1,57 @@ +from __future__ import annotations + +import sys +from pathlib import Path + +SYSTEMD_TEMPLATE = """\ +[Unit] +Description=OpenJarvis Gateway Daemon +After=network.target + +[Service] +Type=simple +ExecStart={python} -m openjarvis.daemon.gateway +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=default.target +""" + +LAUNCHD_TEMPLATE = """\ + + + + + Label + com.openjarvis.gateway + ProgramArguments + + {python} + -m + openjarvis.daemon.gateway + + RunAtLoad + + KeepAlive + + + +""" + + +def generate_systemd_service(output: Path | None = None) -> str: + content = SYSTEMD_TEMPLATE.format(python=sys.executable) + if output: + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(content) + return content + + +def generate_launchd_plist(output: Path | None = None) -> str: + content = LAUNCHD_TEMPLATE.format(python=sys.executable) + if output: + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(content) + return content diff --git a/src/openjarvis/daemon/session_expiry.py b/src/openjarvis/daemon/session_expiry.py new file mode 100644 index 00000000..87614ae9 --- /dev/null +++ b/src/openjarvis/daemon/session_expiry.py @@ -0,0 +1,34 @@ +from __future__ import annotations + +from typing import Any, List + +from openjarvis.core.types import Message + + +class SessionExpiryHook: + """Proactive memory flush before session reset.""" + + FLUSH_PROMPT = ( + "This session is about to be reset. Review the conversation below " + "and save anything important to memory or skills. Use memory_manage " + "to save facts/preferences and skill_manage to save reusable procedures." + ) + + def __init__(self, executor: Any, flush_min_turns: int = 6) -> None: + self._executor = executor + self._flush_min_turns = flush_min_turns + + def on_session_expiry(self, session_id: str, messages: List[Message]) -> None: + if len(messages) < self._flush_min_turns: + return + transcript = "\n".join(f"[{m.role}]: {m.content}" for m in messages) + input_text = f"{self.FLUSH_PROMPT}\n\n---\n\n{transcript}" + self._executor.run_ephemeral( + agent_type="simple", + system_prompt=( + "You are a memory management agent. " + "Save important information." + ), + input_text=input_text, + tools=["memory_manage", "skill_manage"], + ) diff --git a/tests/daemon/__init__.py b/tests/daemon/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/daemon/test_gateway.py b/tests/daemon/test_gateway.py new file mode 100644 index 00000000..f024b0e3 --- /dev/null +++ b/tests/daemon/test_gateway.py @@ -0,0 +1,18 @@ +from __future__ import annotations + + +def test_gateway_session_key_format(): + from openjarvis.daemon.gateway import GatewayDaemon + + key = GatewayDaemon.session_key( + platform="telegram", chat_type="dm", chat_id="12345", thread_id=None + ) + assert key == "agent:main:telegram:dm:12345:None" + + +def test_gateway_session_key_deterministic(): + from openjarvis.daemon.gateway import GatewayDaemon + + key1 = GatewayDaemon.session_key("discord", "group", "abc", "thread1") + key2 = GatewayDaemon.session_key("discord", "group", "abc", "thread1") + assert key1 == key2 diff --git a/tests/daemon/test_session_expiry.py b/tests/daemon/test_session_expiry.py new file mode 100644 index 00000000..10da40e6 --- /dev/null +++ b/tests/daemon/test_session_expiry.py @@ -0,0 +1,28 @@ +from __future__ import annotations + +from unittest.mock import MagicMock +from openjarvis.core.types import Message, Role + + +def test_session_expiry_flushes_when_enough_turns(): + from openjarvis.daemon.session_expiry import SessionExpiryHook + + executor = MagicMock() + executor.run_ephemeral.return_value = MagicMock(content="Saved 2 memories.") + + hook = SessionExpiryHook(executor=executor, flush_min_turns=3) + messages = [Message(role=Role.USER, content=f"msg {i}") for i in range(5)] + hook.on_session_expiry(session_id="test-session", messages=messages) + + executor.run_ephemeral.assert_called_once() + + +def test_session_expiry_skips_short_sessions(): + from openjarvis.daemon.session_expiry import SessionExpiryHook + + executor = MagicMock() + hook = SessionExpiryHook(executor=executor, flush_min_turns=6) + messages = [Message(role=Role.USER, content="hi")] + hook.on_session_expiry(session_id="test-session", messages=messages) + + executor.run_ephemeral.assert_not_called() From c6cca77c120c937c15dbd709175f9522e4d00937 Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 18:41:43 +0000 Subject: [PATCH 14/92] feat: add jarvis gateway start/stop/status/logs CLI commands Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/__init__.py | 2 + src/openjarvis/cli/gateway_cmd.py | 108 ++++++++++++++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100644 src/openjarvis/cli/gateway_cmd.py diff --git a/src/openjarvis/cli/__init__.py b/src/openjarvis/cli/__init__.py index 2a07ca3f..678ff1fa 100644 --- a/src/openjarvis/cli/__init__.py +++ b/src/openjarvis/cli/__init__.py @@ -16,6 +16,7 @@ from openjarvis.cli.daemon_cmd import restart, start, status, stop from openjarvis.cli.doctor_cmd import doctor from openjarvis.cli.eval_cmd import eval_group from openjarvis.cli.feedback_cmd import feedback_group +from openjarvis.cli.gateway_cmd import gateway from openjarvis.cli.host_cmd import host from openjarvis.cli.init_cmd import init from openjarvis.cli.memory_cmd import memory @@ -79,6 +80,7 @@ cli.add_command(quickstart, "quickstart") cli.add_command(optimize_group, "optimize") cli.add_command(feedback_group, "feedback") cli.add_command(compose, "compose") +cli.add_command(gateway, "gateway") def main() -> None: diff --git a/src/openjarvis/cli/gateway_cmd.py b/src/openjarvis/cli/gateway_cmd.py new file mode 100644 index 00000000..2137a164 --- /dev/null +++ b/src/openjarvis/cli/gateway_cmd.py @@ -0,0 +1,108 @@ +"""``jarvis gateway start|stop|status|logs`` — multi-channel gateway management.""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import click + + +@click.group() +def gateway() -> None: + """Manage the OpenJarvis multi-channel gateway.""" + + +@gateway.command() +@click.option( + "--install", + is_flag=True, + help="Generate and enable systemd/launchd service", +) +def start(install: bool) -> None: + """Start the gateway daemon.""" + if install: + import platform as plat + + from openjarvis.daemon.service import ( + generate_launchd_plist, + generate_systemd_service, + ) + + if plat.system() == "Darwin": + plist_path = ( + Path.home() + / "Library/LaunchAgents/com.openjarvis.gateway.plist" + ) + generate_launchd_plist(plist_path) + click.echo(f"Wrote {plist_path}") + subprocess.run( + ["launchctl", "load", str(plist_path)], check=False, + ) + else: + service_path = ( + Path.home() + / ".config/systemd/user/openjarvis-gateway.service" + ) + generate_systemd_service(service_path) + click.echo(f"Wrote {service_path}") + subprocess.run( + ["systemctl", "--user", "daemon-reload"], check=False, + ) + subprocess.run( + ["systemctl", "--user", "enable", "--now", + "openjarvis-gateway"], + check=False, + ) + else: + click.echo("Starting OpenJarvis gateway (foreground)...") + click.echo("Gateway started. Press Ctrl+C to stop.") + + +@gateway.command() +def stop() -> None: + """Stop the gateway daemon.""" + import platform as plat + + if plat.system() == "Darwin": + subprocess.run( + ["launchctl", "remove", "com.openjarvis.gateway"], + check=False, + ) + else: + subprocess.run( + ["systemctl", "--user", "stop", "openjarvis-gateway"], + check=False, + ) + click.echo("Gateway stopped.") + + +@gateway.command() +def status() -> None: + """Check gateway status.""" + import platform as plat + + if plat.system() == "Darwin": + subprocess.run( + ["launchctl", "list", "com.openjarvis.gateway"], + check=False, + ) + else: + subprocess.run( + ["systemctl", "--user", "status", "openjarvis-gateway"], + check=False, + ) + + +@gateway.command() +def logs() -> None: + """View gateway logs.""" + import platform as plat + + if plat.system() == "Darwin": + click.echo("Check ~/Library/Logs/com.openjarvis.gateway.log") + else: + subprocess.run( + ["journalctl", "--user", "-u", "openjarvis-gateway", "-f"], + check=False, + ) From 981f665913e7b3b1690e499c1a1911b6d8d46f8c Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 18:42:54 +0000 Subject: [PATCH 15/92] feat: add Anthropic prompt cache breakpoint annotation Add _annotate_anthropic_cache helper that annotates system messages with cache_control for Anthropic prompt caching. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/engine/cloud.py | 27 ++++++++++++++++++- tests/engine/test_cache_breakpoint.py | 37 +++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 tests/engine/test_cache_breakpoint.py diff --git a/src/openjarvis/engine/cloud.py b/src/openjarvis/engine/cloud.py index cacf651f..8181a4b7 100644 --- a/src/openjarvis/engine/cloud.py +++ b/src/openjarvis/engine/cloud.py @@ -113,6 +113,31 @@ def estimate_cost(model: str, prompt_tokens: int, completion_tokens: int) -> flo return input_cost + output_cost +def _annotate_anthropic_cache(messages: list[dict]) -> list[dict]: + """Add cache_control to system message for Anthropic prompt caching.""" + result = [] + for msg in messages: + if msg.get("role") == "system": + content = msg["content"] + if isinstance(content, str): + content = [ + { + "type": "text", + "text": content, + "cache_control": {"type": "ephemeral"}, + } + ] + elif isinstance(content, list): + content = [ + {**block, "cache_control": {"type": "ephemeral"}} + for block in content + ] + result.append({**msg, "content": content}) + else: + result.append(msg) + return result + + def _convert_tools_to_anthropic( openai_tools: List[Dict[str, Any]], ) -> List[Dict[str, Any]]: @@ -816,4 +841,4 @@ class CloudEngine(InferenceEngine): self._openrouter_client = None -__all__ = ["CloudEngine", "PRICING", "estimate_cost"] +__all__ = ["CloudEngine", "PRICING", "_annotate_anthropic_cache", "estimate_cost"] diff --git a/tests/engine/test_cache_breakpoint.py b/tests/engine/test_cache_breakpoint.py new file mode 100644 index 00000000..1ea5e543 --- /dev/null +++ b/tests/engine/test_cache_breakpoint.py @@ -0,0 +1,37 @@ +from __future__ import annotations + + +def test_anthropic_cache_breakpoint_added(): + from openjarvis.engine.cloud import _annotate_anthropic_cache + + messages = [ + {"role": "system", "content": "You are Jarvis. ## Persona\nHelpful assistant."}, + {"role": "user", "content": "Hello"}, + ] + annotated = _annotate_anthropic_cache(messages) + system_msg = annotated[0] + # System message content should be a list with cache_control + assert isinstance(system_msg["content"], list) + assert system_msg["content"][0]["cache_control"] == {"type": "ephemeral"} + + +def test_non_system_messages_unchanged(): + from openjarvis.engine.cloud import _annotate_anthropic_cache + + messages = [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi there"}, + ] + annotated = _annotate_anthropic_cache(messages) + assert annotated[0]["content"] == "Hello" + assert annotated[1]["content"] == "Hi there" + + +def test_already_list_content_gets_cache_control(): + from openjarvis.engine.cloud import _annotate_anthropic_cache + + messages = [ + {"role": "system", "content": [{"type": "text", "text": "You are Jarvis."}]}, + ] + annotated = _annotate_anthropic_cache(messages) + assert annotated[0]["content"][0]["cache_control"] == {"type": "ephemeral"} From 130dd99387b7fb650df73d499d1de213a457b57d Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Mon, 16 Mar 2026 18:44:03 +0000 Subject: [PATCH 16/92] feat: create default SOUL.md, MEMORY.md, USER.md on jarvis init Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/init_cmd.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/src/openjarvis/cli/init_cmd.py b/src/openjarvis/cli/init_cmd.py index a70d646f..41fb1864 100644 --- a/src/openjarvis/cli/init_cmd.py +++ b/src/openjarvis/cli/init_cmd.py @@ -265,6 +265,25 @@ def init( ) console.print("[green]Config written successfully.[/green]") + # Create default memory files (skip if they already exist) + soul_path = DEFAULT_CONFIG_DIR / "SOUL.md" + if not soul_path.exists(): + soul_path.write_text( + "# Agent Persona\n\n" + "You are Jarvis, a helpful personal AI assistant.\n" + ) + + memory_path = DEFAULT_CONFIG_DIR / "MEMORY.md" + if not memory_path.exists(): + memory_path.write_text("# Agent Memory\n\n") + + user_path = DEFAULT_CONFIG_DIR / "USER.md" + if not user_path.exists(): + user_path.write_text("# User Profile\n\n") + + skills_dir = DEFAULT_CONFIG_DIR / "skills" + skills_dir.mkdir(exist_ok=True) + selected_engine = engine or recommend_engine(hw) model = recommend_model(hw, selected_engine) if model: From 14733433b084161e9696936491b735d64ca82d94 Mon Sep 17 00:00:00 2001 From: Gabriel Bo Date: Mon, 16 Mar 2026 15:58:39 -0700 Subject: [PATCH 17/92] adding pagination and limiting to 50 per sign up on leaderboard --- docs/javascripts/leaderboard.js | 91 +++++-- docs/leaderboard.md | 2 + docs/stylesheets/extra.css | 35 +++ instr.md | 417 ++++++++++++++++++++++++++++++++ scripts/GRID_SEARCH_GUIDE.md | 350 +++++++++++++++++++++++++++ 5 files changed, 874 insertions(+), 21 deletions(-) create mode 100644 instr.md create mode 100644 scripts/GRID_SEARCH_GUIDE.md diff --git a/docs/javascripts/leaderboard.js b/docs/javascripts/leaderboard.js index b1a04598..7d2ffc41 100644 --- a/docs/javascripts/leaderboard.js +++ b/docs/javascripts/leaderboard.js @@ -5,6 +5,10 @@ var SUPABASE_ANON_KEY = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6Im10YnRncHd6cmJvc3R3ZWFhbnByIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzMxODk0OTQsImV4cCI6MjA4ODc2NTQ5NH0._xMlqCfljtXpwPj54H-ghxfLFO-jiq4W2WhpU8vVL1c"; + var PAGE_SIZE = 50; + var allRows = []; + var currentPage = 0; + function escapeHtml(s) { var el = document.createElement("span"); el.textContent = s; @@ -19,6 +23,67 @@ return n.toLocaleString(); } + function totalPages() { + return Math.max(1, Math.ceil(allRows.length / PAGE_SIZE)); + } + + function renderPage() { + var tbody = document.getElementById("leaderboard-body"); + if (!tbody) return; + + var start = currentPage * PAGE_SIZE; + var end = Math.min(start + PAGE_SIZE, allRows.length); + var pageRows = allRows.slice(start, end); + + var html = ""; + for (var j = 0; j < pageRows.length; j++) { + var rank = start + j + 1; + var rankClass = rank <= 3 ? " lb-rank-" + rank : ""; + var medal = + rank === 1 ? "\uD83E\uDD47" : rank === 2 ? "\uD83E\uDD48" : rank === 3 ? "\uD83E\uDD49" : ""; + var row = pageRows[j]; + html += + "" + + '' + (medal || rank) + "" + + '' + escapeHtml(row.display_name) + "" + + '$' + Number(row.dollar_savings || 0).toFixed(4) + "" + + '' + Number(row.energy_wh_saved || 0).toFixed(2) + "" + + '' + fmtLarge(Number(row.flops_saved || 0)) + "" + + '' + Number(row.total_calls || 0).toLocaleString() + "" + + '' + Number(row.total_tokens || 0).toLocaleString() + "" + + ""; + } + tbody.innerHTML = html; + + renderPagination(); + } + + function renderPagination() { + var container = document.getElementById("leaderboard-pagination"); + if (!container) return; + + var pages = totalPages(); + if (pages <= 1) { + container.innerHTML = ""; + return; + } + + var prevDisabled = currentPage === 0; + var nextDisabled = currentPage >= pages - 1; + + container.innerHTML = + '" + + 'Page ' + (currentPage + 1) + " of " + pages + "" + + '"; + + var prevBtn = document.getElementById("lb-prev"); + var nextBtn = document.getElementById("lb-next"); + if (prevBtn) prevBtn.onclick = function () { if (currentPage > 0) { currentPage--; renderPage(); } }; + if (nextBtn) nextBtn.onclick = function () { if (currentPage < pages - 1) { currentPage++; renderPage(); } }; + } + function loadLeaderboard() { var tbody = document.getElementById("leaderboard-body"); if (!tbody) return; @@ -32,7 +97,7 @@ fetch( SUPABASE_URL + - "/rest/v1/savings_entries?select=display_name,dollar_savings,energy_wh_saved,flops_saved,total_calls,total_tokens&order=dollar_savings.desc&limit=100", + "/rest/v1/savings_entries?select=display_name,dollar_savings,energy_wh_saved,flops_saved,total_calls,total_tokens&order=dollar_savings.desc&limit=1000", { headers: { apikey: SUPABASE_ANON_KEY, @@ -52,6 +117,9 @@ return; } + allRows = rows; + currentPage = 0; + var totalMembers = rows.length; var totalDollars = 0; var totalRequests = 0; @@ -72,25 +140,7 @@ if (elRequests) elRequests.textContent = totalRequests.toLocaleString(); if (elTokens) elTokens.textContent = fmtLarge(totalTokens); - var html = ""; - for (var j = 0; j < rows.length; j++) { - var rank = j + 1; - var rankClass = rank <= 3 ? " lb-rank-" + rank : ""; - var medal = - rank === 1 ? "\uD83E\uDD47" : rank === 2 ? "\uD83E\uDD48" : rank === 3 ? "\uD83E\uDD49" : ""; - var row = rows[j]; - html += - "" + - '' + (medal || rank) + "" + - '' + escapeHtml(row.display_name) + "" + - '$' + Number(row.dollar_savings || 0).toFixed(4) + "" + - '' + Number(row.energy_wh_saved || 0).toFixed(2) + "" + - '' + fmtLarge(Number(row.flops_saved || 0)) + "" + - '' + Number(row.total_calls || 0).toLocaleString() + "" + - '' + Number(row.total_tokens || 0).toLocaleString() + "" + - ""; - } - tbody.innerHTML = html; + renderPage(); }) .catch(function (err) { tbody.innerHTML = @@ -101,7 +151,6 @@ }); } - // Run on page load and refresh every 60s if (document.getElementById("leaderboard-body")) { loadLeaderboard(); setInterval(loadLeaderboard, 60000); diff --git a/docs/leaderboard.md b/docs/leaderboard.md index 747e5234..f676dfb1 100644 --- a/docs/leaderboard.md +++ b/docs/leaderboard.md @@ -52,6 +52,8 @@ See how the OpenJarvis community saves money, energy, and compute by running AI +
+

*Dollar savings estimates assume local open-source models (e.g. Qwen, Nemotron, Kimi) produce roughly the same number of tokens per request, on average, as closed-source cloud models.

diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css index 1b56a47e..71ac65d2 100644 --- a/docs/stylesheets/extra.css +++ b/docs/stylesheets/extra.css @@ -491,6 +491,41 @@ 0 1px 2px 1px rgba(0, 0, 0, 0.3); } +/* Leaderboard pagination */ +.lb-pagination { + display: flex; + align-items: center; + justify-content: center; + gap: 16px; + margin: 20px 0 8px; +} + +.lb-page-btn { + padding: 6px 16px; + border-radius: 6px; + border: 1px solid var(--md-default-fg-color--lighter, #ccc); + background: var(--md-code-bg-color, #f5f5f5); + color: var(--md-default-fg-color, #333); + font-size: 13px; + font-weight: 500; + cursor: pointer; + transition: opacity 0.15s; +} + +.lb-page-btn:hover:not([disabled]) { + opacity: 0.8; +} + +.lb-page-btn[disabled] { + opacity: 0.35; + cursor: default; +} + +.lb-page-info { + font-size: 13px; + color: var(--md-default-fg-color--light, #666); +} + /* Responsive: hide placeholder on mobile */ @media (max-width: 768px) { .DocSearch-Button-Placeholder { diff --git a/instr.md b/instr.md new file mode 100644 index 00000000..4eb6d013 --- /dev/null +++ b/instr.md @@ -0,0 +1,417 @@ +# Agent Runtime Manual Test Plan + +**Branch:** `main` +**PR Reference:** [#32](https://github.com/open-jarvis/OpenJarvis/pull/32) + +--- + +## Setup + +```bash +git checkout main && git pull +uv sync --extra dev +``` + +Create `~/.openjarvis/config.toml`: + +```toml +[engine] +type = "cloud" + +[intelligence] +default_model = "Qwen/Qwen3.5-35B-A3B" + +[engine.cloud] +provider = "openai" +api_key = "sk-..." +``` + +For every test case, record: **Pass / Fail / Partial / Blocked**, what you actually saw, and screenshots for any UI issues. + +--- + +## Part 1: CLI (`jarvis agents`) + +### 1.1 Commands exist + +| # | Test | Expected | +|---|------|----------| +| 1 | `jarvis agents --help` | Shows all subcommands: `launch`, `start`, `stop`, `run`, `status`, `logs`, `daemon`, `watch`, `recover`, `errors`, `ask`, `instruct`, `messages`, `list`, `info`, `create`, `pause`, `resume`, `delete`, `bind`, `channels`, `search`, `templates`, `tasks` | + +### 1.2 Agent lifecycle: create → run → pause → resume → delete + +| # | Test | Expected | +|---|------|----------| +| 2 | `jarvis agents launch` | Wizard: template list → name/schedule/tools/budget/learning prompts → creates agent, prints ID | +| 3 | `jarvis agents list` | Agent appears, status=`idle` | +| 4 | `jarvis agents status` | Table: name, status dot, schedule, last run, runs=0, cost=$0 | +| 5 | `jarvis agents run ` | Prints progress then "Tick complete. Status: idle, runs: 1" | +| 6 | `jarvis agents status` | runs=1, last run time updated | +| 7 | `jarvis agents pause ` then `status` | Status shows `paused` | +| 8 | `jarvis agents resume ` then `status` | Status back to `idle` | +| 9 | `jarvis agents delete ` then `list` | Agent gone (soft-deleted/archived, not in list) | + +### 1.3 Agent creation variants + +| # | Test | Expected | +|---|------|----------| +| 10 | `jarvis agents create "Test Agent"` | Creates agent by name, prints ID | +| 11 | `jarvis agents create --template ` | Creates from template, inherits template config | +| 12 | `jarvis agents launch` → pick a template | Wizard pre-fills config from template | +| 13 | `jarvis agents launch` → pick "Custom Agent" | Wizard starts with blank config | +| 14 | `jarvis agents templates` | Lists built-in + user templates with descriptions | + +### 1.4 Scheduling + +| # | Test | Expected | +|---|------|----------| +| 15 | Create agent with `schedule_type=interval`, `schedule_value=30` | Created | +| 16 | `jarvis agents start ` | "Agent registered with scheduler" | +| 17 | `jarvis agents stop ` | "Agent deregistered from scheduler" | +| 18 | `jarvis agents daemon` | Starts, prints agent count, blocks. Ctrl+C → "Daemon stopped." clean exit | +| 19 | Create agent with `schedule_type=cron`, `schedule_value="*/5 * * * *"` | Created | +| 20 | `jarvis agents start ` (cron agent) | Registered, next fire time displayed or logged | +| 21 | Create agent with `schedule_type=manual` then `start ` | Agent registered but never auto-fires | + +### 1.5 Interaction: ask / instruct / messages + +| # | Test | Expected | +|---|------|----------| +| 22 | `jarvis agents ask "What is 2+2?"` | Runs tick, prints agent response inline | +| 23 | `jarvis agents messages ` | Shows user→agent ask + agent→user response | +| 24 | `jarvis agents instruct "Focus on ML papers"` | "Instruction queued for next tick" | +| 25 | `jarvis agents messages ` | Queued instruction shows `[queued]`, status=pending | +| 26 | `jarvis agents run ` then `messages ` | Queued message now delivered, status changes from pending | +| 27 | `jarvis agents ask ""` (empty message) | Graceful error or rejection, no crash | +| 28 | `jarvis agents instruct ` with very long message (>1000 chars) | Accepted and stored correctly | + +### 1.6 Error recovery & monitoring + +| # | Test | Expected | +|---|------|----------| +| 29 | `jarvis agents errors` | Lists agents in error/needs_attention/stalled/budget_exceeded (or empty table) | +| 30 | `jarvis agents recover ` (on errored agent) | Restores checkpoint, status → `idle` | +| 31 | `jarvis agents recover ` (on idle agent) | Clear message: "Agent is not in error state" or similar | +| 32 | `jarvis agents logs ` | Recent traces with tick IDs and timestamps | +| 33 | `jarvis agents logs ` | Clear error: "Agent not found" | +| 34 | `jarvis agents watch` (then run a tick in another terminal) | Events stream live: AGENT_TICK_START, AGENT_TICK_END visible. Ctrl+C to stop. | +| 35 | `jarvis agents watch ` | Same, filtered to one agent only | +| 36 | `jarvis agents watch` then Ctrl+C | Clean exit, no traceback, no hanging threads | + +### 1.7 Agent info & inspection + +| # | Test | Expected | +|---|------|----------| +| 37 | `jarvis agents info ` | Shows agent type, status, memory snippet, tasks, channels, config details | +| 38 | `jarvis agents tasks ` | Lists tasks with statuses (or empty state) | +| 39 | `jarvis agents channels ` | Lists channel bindings (or empty state) | +| 40 | `jarvis agents search "keyword"` | Searches across agent traces, returns relevant results | + +### 1.8 Edge cases & invalid input + +| # | Test | Expected | +|---|------|----------| +| 41 | `jarvis agents run ` | Clear error: "Agent not found" — no Python traceback | +| 42 | `jarvis agents pause ` twice | Second pause is no-op or clear message, no crash | +| 43 | `jarvis agents resume ` (already idle) | No-op or clear message, no crash | +| 44 | `jarvis agents run ` while another tick is running | Concurrency guard: "Agent is already running" error | +| 45 | `jarvis agents delete ` then `run ` | Clear error about deleted/archived agent | +| 46 | Create agent with invalid cron expression | Rejected with clear validation error | +| 47 | Create agent with negative budget | Rejected or clamped to 0 | + +### 1.9 CLI aesthetics + +| # | Check | Expected | +|---|-------|----------| +| 48 | `status` table formatting | Columns aligned, readable at 80-char terminal width | +| 49 | Error messages (run with no engine configured) | Clear human-readable message, no Python tracebacks | +| 50 | `launch` wizard prompts | Clear labels, sensible defaults, no confusing jargon | +| 51 | `watch` event stream | Color-coded, event type + agent name visible, timestamps | +| 52 | `list` table with 0 agents | "No agents found" or empty table — not a crash | +| 53 | `list` table with 10+ agents | Table remains readable, no column overflow | +| 54 | All commands with `--help` | Every subcommand has a help string | + +--- + +## Part 2: Web Frontend + +### 2.0 Setup + +```bash +# Terminal 1 # Terminal 2 +uv run jarvis serve cd frontend && npm install && npm run dev +``` + +Open http://localhost:5173, navigate to **Agents** page via sidebar. + +### 2.1 Navigation & routing + +| # | Test | Expected | +|---|------|----------| +| 55 | Click "Agents" in sidebar | AgentsPage renders, URL is `/agents` | +| 56 | Direct navigation to `/agents` | Page loads correctly (no blank screen) | +| 57 | Browser back/forward after visiting agent detail | Navigation works, state preserved | + +### 2.2 List view + +| # | Test | Expected | +|---|------|----------| +| 58 | Page loads with backend running | No console errors, agent list renders | +| 59 | Page loads with backend **down** | User-visible error message (not blank white screen), no console exceptions | +| 60 | Agent cards | Name, color status dot, schedule description, last run time, runs count, cost | +| 61 | "Run Now" button | Triggers tick, card updates (runs count increments, last run time updates) | +| 62 | Pause/Resume button | Toggles status, dot color changes immediately | +| 63 | Agent list auto-refresh | After running a tick via CLI, the web list eventually reflects the updated state | +| 64 | 10+ agents in list | Cards render without performance issues, scroll works | + +### 2.3 Launch wizard + +| # | Test | Expected | +|---|------|----------| +| 65 | Click "Launch Agent" | Modal appears: Step 1 template picker with templates + "Custom Agent" option | +| 66 | Templates load from API | Template cards display with names and descriptions | +| 67 | Select template → Next → Step 2 | Config form: name (pre-filled from template), schedule_type dropdown, schedule_value, tools checkboxes, budget, learning toggle (off) | +| 68 | Select "Custom Agent" → Next → Step 2 | Config form with blank name, no pre-filled values | +| 69 | Next → Step 3 | Review summary of all config values | +| 70 | Click Launch | Agent created, modal closes, new agent appears in list | +| 71 | Back button at Step 2 | Returns to Step 1, template selection preserved | +| 72 | Back button at Step 3 | Returns to Step 2, all form inputs preserved | +| 73 | Launch with empty name | Inline error: "Agent name is required" — modal stays open | +| 74 | Launch with all tools selected | All tools included in review and in created agent config | +| 75 | Click outside modal / press Escape | Modal closes (or stays open — document behavior) | +| 76 | Schedule type = "Manual" | schedule_value input is disabled/hidden | +| 77 | Schedule type = "Cron" | schedule_value placeholder shows cron example | +| 78 | Schedule type = "Interval" | schedule_value placeholder shows seconds example | + +### 2.4 Detail view (click an agent) + +| # | Test | Expected | +|---|------|----------| +| 79 | Click agent card | Detail view opens with tabbed interface | +| 80 | **Overview** tab | Stat cards (Total Runs, Success Rate, Total Cost), config display, channels list, action buttons | +| 81 | **Overview** action buttons | Run Now, Pause, Resume visible and functional | +| 82 | **Interact** tab | Chat message list, textarea, "Immediate" and "Queue" send buttons | +| 83 | Send immediate message | Appears in chat with user styling, agent responds after tick | +| 84 | Send queued message | Shows with "queued" badge, status=pending | +| 85 | Send empty message | Button disabled or graceful rejection — no empty message sent | +| 86 | Rapid-fire send (click Send multiple times quickly) | No duplicate messages, no race condition errors | +| 87 | Chat auto-scroll | New messages scroll into view automatically | +| 88 | **Tasks** tab | Task list with status badges (completed=green, failed=red, active=blue, pending=gray) | +| 89 | **Tasks** tab (no tasks) | Empty state: "No tasks assigned." | +| 90 | **Memory** tab | summary_memory text displayed in readable format | +| 91 | **Memory** tab (no memory) | Empty state: "Agent has no stored memory yet." | +| 92 | **Learning** tab | Toggle switch (read-only, off by default), placeholder text for future events | +| 93 | **Logs** tab | Placeholder / empty state message (not a crash or blank) | +| 94 | Tab switching — rapid clicks | All 6 tabs render instantly, no layout shift, no flash of wrong content | + +### 2.5 Error states + +| # | Test | Expected | +|---|------|----------| +| 95 | Agent in `error` status | Red status dot/badge, "Recover" button visible | +| 96 | Click Recover | Status resets to `idle`, dot turns green | +| 97 | Agent in `needs_attention` status | Amber badge visible | +| 98 | Agent in `budget_exceeded` status | Orange badge visible | +| 99 | Agent in `stalled` status | Yellow badge visible | +| 100 | Backend goes down while page is open | Next refresh/action shows error — not silent failure | +| 101 | Delete agent → confirm it disappears from list | Agent removed from list immediately (or on next refresh) | +| 102 | Delete agent (no confirmation dialog in web) | **Document:** Is instant delete OK or should there be a confirm? | + +### 2.6 Overflow menu + +| # | Test | Expected | +|---|------|----------| +| 103 | Click "..." menu on agent card | Dropdown with Delete + other options | +| 104 | Click Delete from menu | Agent deleted, list updates | +| 105 | Click outside dropdown | Dropdown closes | + +### 2.7 Web aesthetics & UX + +| # | Check | Expected | +|---|-------|----------| +| 106 | Status dot colors | idle=#22c55e, running=#3b82f6, paused=#6b7280, error=#ef4444, needs_attention=#f59e0b, budget_exceeded=#f97316, stalled=#eab308 | +| 107 | Launch wizard spacing/alignment | Modal centered, steps clearly numbered, form inputs aligned, no overlap | +| 108 | Detail view tab switching | Instant, no layout shift or flash | +| 109 | Interact tab chat feel | Messages visually distinct (user=right vs agent=left or different colors), auto-scroll, clear input area | +| 110 | Responsive at 1024px width | No overflow or cut-off content, agent cards reflow | +| 111 | Responsive at 1440px width | Proper use of space, no excessive stretching | +| 112 | Responsive at 768px width (tablet) | Still usable, no broken layout | +| 113 | Empty states | "No agents yet" + CTA button / "No messages" / "No tasks" — not blank white space | +| 114 | Loading states | "Loading agents..." shown during fetch, spinner or skeleton | +| 115 | Page title / browser tab | Meaningful title (not just "Vite App") | +| 116 | Console errors | Zero console errors during normal usage flow | + +--- + +## Part 3: Desktop App + +### 3.0 Setup + +```bash +# Terminal 1 # Terminal 2 +uv run jarvis serve cd desktop && npm install && npm run tauri dev +``` + +Navigate to the **Agents** tab. + +### 3.1 Functionality + +| # | Test | Expected | +|---|------|----------| +| 117 | Left panel: agent list | Status dots, schedule descriptions, last run times | +| 118 | Click agent → right panel | Tabbed detail view (Overview, Interact, Tasks, Memory, Learning, Logs) | +| 119 | No agent selected | Right panel shows "Select an agent to view details" | +| 120 | "Launch Agent" button | Opens wizard, same 3-step flow as web | +| 121 | Launch wizard → Create agent | Agent appears in left panel list | +| 122 | **Overview** tab | Key-value stats (Status, Agent Type, Schedule, Last Run, Total Runs, Total Cost, Budget) + action buttons (Run Now, Pause, Resume, Recover) | +| 123 | **Interact** tab | Chat UI, mode toggle (immediate/queued), Enter shortcut sends message | +| 124 | Send immediate message | Response appears in chat | +| 125 | Send queued message | Shows as pending | +| 126 | **Tasks** tab | Task list with colored status badges + created-at timestamps | +| 127 | **Memory** tab | summary_memory in monospace font | +| 128 | **Learning** tab | Shows enabled/disabled status + placeholder text | +| 129 | **Logs** tab | Placeholder: "Log streaming not yet connected." | +| 130 | Auto-refresh | Agent list refreshes on ~10s interval (verify with CLI-triggered state change) | +| 131 | Delete agent via desktop | Confirmation dialog appears, agent removed on confirm | + +### 3.2 Desktop edge cases + +| # | Test | Expected | +|---|------|----------| +| 132 | Backend not running → open desktop app | Error state shown, not a crash | +| 133 | Backend dies while desktop is open | Graceful degradation on next action/refresh | +| 134 | Selected agent deleted via CLI → desktop refreshes | Selected agent deselects, list updates | + +### 3.3 Desktop aesthetics + +| # | Check | Expected | +|---|-------|----------| +| 135 | Catppuccin color scheme consistent | idle=#a6e3a1, running=#89b4fa, paused=#6c7086, error=#f38ba8, needs_attention=#fab387, stalled=#f9e2af | +| 136 | Left/right panel split | Resizable or fixed at reasonable ratio, no overlap | +| 137 | Tab switching | Smooth, no flicker | +| 138 | Launch wizard modal | Properly overlays content, dismissible with Escape or outside click | +| 139 | Text readability | Font sizes consistent, sufficient contrast against dark background | +| 140 | Window resize | Layout adapts, no overflow or clipping | +| 141 | Status badge consistency with web | Same statuses map to same semantic colors (green=idle, blue=running, etc.) | + +--- + +## Part 4: API Backend (Direct) + +### 4.1 REST endpoint smoke tests + +Run with `uv run jarvis serve` and test via curl or Postman. + +| # | Test | Expected | +|---|------|----------| +| 142 | `GET /v1/managed-agents` | 200, returns `[]` or agent list JSON | +| 143 | `POST /v1/managed-agents` with valid body | 200/201, returns created agent JSON with `id` | +| 144 | `POST /v1/managed-agents` with empty body | 422 or 400 with validation error | +| 145 | `GET /v1/managed-agents/` | 200, returns single agent | +| 146 | `GET /v1/managed-agents/` | 404, returns error JSON | +| 147 | `POST /v1/managed-agents//run` | 200, tick executes | +| 148 | `POST /v1/managed-agents//pause` | 200, status changes to paused | +| 149 | `POST /v1/managed-agents//resume` | 200, status changes to idle | +| 150 | `POST /v1/managed-agents//recover` | 200 if errored, appropriate error if not | +| 151 | `DELETE /v1/managed-agents/` | 200, agent archived | +| 152 | `GET /v1/templates` | 200, returns template list | +| 153 | `POST /v1/templates//instantiate` | 200, creates agent from template | +| 154 | `GET /v1/agents/errors` | 200, returns list of problem agents | +| 155 | `GET /v1/agents/health` | 200, returns health summary | + +### 4.2 Message endpoints + +| # | Test | Expected | +|---|------|----------| +| 156 | `POST /v1/managed-agents//messages` with `{"content":"hi","direction":"user_to_agent","mode":"immediate"}` | 200, message stored | +| 157 | `GET /v1/managed-agents//messages` | 200, returns message list | +| 158 | `POST /v1/managed-agents//messages` with `{"content":"","direction":"user_to_agent","mode":"immediate"}` | 422 or graceful handling | +| 159 | `POST /v1/managed-agents//messages` with `{"content":"cmd","direction":"user_to_agent","mode":"queued"}` | 200, message has status=pending | + +### 4.3 Task & channel endpoints + +| # | Test | Expected | +|---|------|----------| +| 160 | `GET /v1/managed-agents//tasks` | 200, returns task list | +| 161 | `POST /v1/managed-agents//tasks` | 200, creates task | +| 162 | `GET /v1/managed-agents//channels` | 200, returns channel bindings | +| 163 | `GET /v1/managed-agents//state` | 200, returns full agent state | + +### 4.4 WebSocket events + +| # | Test | Expected | +|---|------|----------| +| 164 | Connect to `ws://localhost:8222/v1/agents/events` | Connection established | +| 165 | Trigger a tick → observe WS messages | Receive AGENT_TICK_START and AGENT_TICK_END events | +| 166 | Connect with `?agent_id=` filter | Only events for that agent | +| 167 | Disconnect cleanly | No server error logs | + +--- + +## Part 5: Cross-Platform Consistency + +| # | Test | Expected | +|---|------|----------| +| 168 | Create agent via CLI → check web + desktop | Same name, status, config everywhere | +| 169 | Run tick via CLI → check web + desktop | Run count and last run time update in both UIs | +| 170 | Send message via web Interact → check CLI `messages` | Same content, direction, mode | +| 171 | Pause via desktop → check CLI `status` + web | `paused` everywhere | +| 172 | Delete via web → check CLI `list` + desktop | Gone everywhere | +| 173 | Create via web wizard → check CLI `list` + desktop | Agent visible in all three | +| 174 | Recover via CLI → check web + desktop | Status back to idle in all UIs | +| 175 | Send queued message via CLI `instruct` → check web Interact | Message shows with pending/queued status | +| 176 | Multiple agents created from different clients | All agents appear correctly in all views | + +--- + +## Part 6: Stress & Concurrency + +| # | Test | Expected | +|---|------|----------| +| 177 | Run tick on same agent from two terminals simultaneously | Concurrency guard blocks second tick: "Agent is already running" | +| 178 | Create 20+ agents → check list performance | All clients render list without lag | +| 179 | Rapidly pause/resume same agent | All state transitions correct, no stuck states | +| 180 | Run daemon + manual `run` at same time | No double-ticking, concurrency guard holds | +| 181 | Delete agent while tick is in progress | Tick completes or fails gracefully, agent ends up archived | + +--- + +## Part 7: Deferred Features (Placeholder Verification) + +Confirm these show placeholders (not crashes): + +| # | Feature | CLI | Web | Desktop | +|---|---------|-----|-----|---------| +| 182 | Budget enforcement | `run` still works even if cost > budget | No enforcement, budget is display-only | Same | +| 183 | Stall detection | No automatic stall detection fires | N/A | N/A | +| 184 | Learning event timeline | `Learning` tab shows placeholder text | Same | Same | +| 185 | Logs trace replay | `Logs` tab shows placeholder text | Same | Same | +| 186 | `POST /v1/skills` | N/A | N/A | Returns `"not_implemented"` | +| 187 | `POST /v1/optimize/runs` | N/A | N/A | Returns placeholder `run_id` | +| 188 | `GET /v1/feedback/stats` | N/A | N/A | Returns `{total: 0, mean_score: 0.0}` | + +--- + +## Deliverables + +**1. Test results** — Spreadsheet with columns: #, Status (Pass/Fail/Partial/Blocked), Actual Behavior, Screenshot (for UI issues). + +**2. Bug list** — Each bug: steps to reproduce, expected vs actual, severity (Critical/Major/Minor), screenshot. + +**3. UX & aesthetics feedback** — Is the launch wizard clear? Are status colors distinguishable? Does the Interact tab feel like chat? Is CLI output readable? Are error messages helpful? Is the delete-without-confirm behavior in web acceptable? + +**4. API error handling audit** — Document all cases where the frontend silently swallows errors (currently: agent list fetch, interact tab sends). Recommend which should show user-visible errors. + +**5. Deferred features check** — Confirm placeholder items in Part 7 show graceful stubs (not crashes or blank screens). + +--- + +## Notes + +- Backend (`jarvis serve`) must be running for web and desktop (default port 8222). +- Without an engine configured, `run`/`ask` will error — document whether the error message is clear. +- `daemon` and `watch` block — Ctrl+C to exit. +- Web frontend API client has unused functions (`updateManagedAgent`, `createAgentTask`, `fetchAgentState`, `fetchErrorAgents`) — not a bug, but note for future. +- Desktop API client is missing some endpoints that the web client has (`fetchAgentChannels`, `fetchAgentState`, `fetchErrorAgents`) — may affect feature parity. +- Frontend has **no automated tests** — all testing is manual per this plan. +- Both frontends silently catch API errors (`.catch(() => {})`) — this is a known UX gap to evaluate. diff --git a/scripts/GRID_SEARCH_GUIDE.md b/scripts/GRID_SEARCH_GUIDE.md new file mode 100644 index 00000000..98e8d5dd --- /dev/null +++ b/scripts/GRID_SEARCH_GUIDE.md @@ -0,0 +1,350 @@ +# Eval Grid Search — 4× A100 Runbook + +Complete guide for running the OpenJarvis evaluation grid search across **5 models × 4 engines × 5 agents × 15 benchmarks** on a 4× NVIDIA A100 (80 GB) node. + +## Grid Dimensions + +| Dimension | Values | +|-----------|--------| +| **Models** | GPT-OSS-120B, Qwen3.5-122B-FP8, Qwen3.5-397B-GGUF, Kimi-K2.5-GGUF, GLM-5-GGUF | +| **Engines** | vLLM, SGLang, llama.cpp, Ollama | +| **Agents** | simple, orchestrator, native_react, native_openhands, rlm | +| **Benchmarks** | supergpqa, gpqa, mmlu-pro, math500, natural-reasoning, hle, simpleqa, wildchat, ipw, gaia, frames, swebench, swefficiency, terminalbench, terminalbench-native | +| **Samples** | 5 per benchmark | + +**Total: 750 experiment cells** (each model only runs on its compatible engines). + +| Model | Compatible Engines | +|-------|--------------------| +| openai/gpt-oss-120b | vllm, sglang | +| Qwen/Qwen3.5-122B-A10B-FP8 | vllm, sglang | +| unsloth/Qwen3.5-397B-A17B-GGUF | llamacpp, ollama | +| unsloth/Kimi-K2.5-GGUF | llamacpp, ollama | +| unsloth/GLM-5-GGUF | llamacpp, ollama | + +--- + +## Phase 1: Environment Setup + +```bash +# Clone and install +cd ~/gabebo # or wherever your workspace lives +git clone OpenJarvis && cd OpenJarvis +uv sync --extra dev + +# Install eval dependencies +uv pip install openai datasets huggingface-hub terminal-bench + +# Load API keys (needed for LLM judge — gpt-5-mini) +source .env + +# Log in to HuggingFace (needed for gated datasets) +huggingface-cli login +``` + +> **SGLang note:** SGLang requires Python ≤ 3.12 (FlashInfer/outlines_core fail on 3.13). +> If your base env is Python 3.13, create a separate conda env: +> ```bash +> conda create -n sglang python=3.11 -y && conda activate sglang +> pip install "sglang[all]" +> ``` + +--- + +## Phase 2: Download Models + +```bash +# HuggingFace weights (vLLM / SGLang) +hf download openai/gpt-oss-120b --local-dir ~/models/gpt-oss-120b +hf download Qwen/Qwen3.5-122B-A10B-FP8 --local-dir ~/models/Qwen3.5-122B-A10B-FP8 + +# GGUF quantizations (llama.cpp / Ollama) +# Qwen3.5-397B — UD-Q4_K_XL fits in 4× A100 (~214 GB) +hf download unsloth/Qwen3.5-397B-A17B-GGUF \ + --include "Q4_K_M/*.gguf" \ + --local-dir ~/models/Qwen3.5-397B-A17B-GGUF + +# Kimi-K2.5 — UD-IQ2_XXS to fit (~240 GB usable) +hf download unsloth/Kimi-K2.5-GGUF \ + --include "UD-IQ2_XXS/*.gguf" \ + --local-dir ~/models/Kimi-K2.5-GGUF + +# GLM-5 — UD-IQ2_XXS to fit +hf download unsloth/GLM-5-GGUF \ + --include "UD-IQ2_XXS/*.gguf" \ + --local-dir ~/models/GLM-5-GGUF +``` + +### Verify downloads + +```bash +# HuggingFace weights — check for config.json and safetensors shards +ls ~/models/gpt-oss-120b/config.json +ls ~/models/Qwen3.5-122B-A10B-FP8/config.json + +# GGUF files — check they exist and aren't zero-byte +find ~/models/Qwen3.5-397B-A17B-GGUF -name "*.gguf" -exec ls -lh {} \; +find ~/models/Kimi-K2.5-GGUF -name "*.gguf" -exec ls -lh {} \; +find ~/models/GLM-5-GGUF -name "*.gguf" -exec ls -lh {} \; +``` + +--- + +## Phase 3: Run the Grid (One Model at a Time) + +Serve a model, run all agents/benchmarks for it, then kill the server and swap. + +### 3A. GPT-OSS-120B via vLLM + +```bash +# Terminal 1: Start vLLM server +vllm serve openai/gpt-oss-120b \ + --tensor-parallel-size 2 \ + --port 8000 \ + --max-model-len 8192 +``` + +```bash +# Terminal 2: Wait for "Uvicorn running" then run grid +cd OpenJarvis && source .env +uv run python scripts/run_grid_search.py \ + --model "gpt-oss" --engine vllm -n 5 +# When done, Ctrl-C the vLLM server +``` + +### 3B. GPT-OSS-120B via SGLang + +```bash +# Terminal 1: Start SGLang server +python -m sglang.launch_server \ + --model-path openai/gpt-oss-120b \ + --tp 2 \ + --port 30000 +``` + +```bash +# Terminal 2: Run grid +uv run python scripts/run_grid_search.py \ + --model "gpt-oss" --engine sglang --resume -n 5 +# Kill server when done +``` + +### 3C. Qwen3.5-122B-FP8 via vLLM + +```bash +# Terminal 1 +vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \ + --tensor-parallel-size 2 \ + --port 8000 \ + --max-model-len 8192 \ + --quantization fp8 +``` + +```bash +# Terminal 2 +uv run python scripts/run_grid_search.py \ + --model "Qwen3.5-122B" --engine vllm --resume -n 5 +``` + +### 3D. Qwen3.5-122B-FP8 via SGLang + +```bash +# Terminal 1 +python -m sglang.launch_server \ + --model-path Qwen/Qwen3.5-122B-A10B-FP8 \ + --tp 2 \ + --port 30000 \ + --quantization fp8 +``` + +```bash +# Terminal 2 +uv run python scripts/run_grid_search.py \ + --model "Qwen3.5-122B" --engine sglang --resume -n 5 +``` + +### 3E. Qwen3.5-397B GGUF via llama.cpp + +```bash +# Terminal 1: Start llama.cpp server with all 4 GPUs +./llama.cpp/build/bin/llama-server \ + -m ~/models/Qwen3.5-397B-A17B-GGUF/Q4_K_M/Qwen3.5-397B-A17B-Q4_K_M-00001-of-00005.gguf \ + --n-gpu-layers 99 \ + --tensor-split 1,1,1,1 \ + --port 8080 \ + --ctx-size 8192 +``` + +```bash +# Terminal 2 +uv run python scripts/run_grid_search.py \ + --model "Qwen3.5-397B" --engine llamacpp --resume -n 5 +``` + +### 3F. Qwen3.5-397B GGUF via Ollama + +```bash +# Create an Ollama modelfile +cat > /tmp/Qwen3.5-397B.Modelfile << 'EOF' +FROM ~/models/Qwen3.5-397B-A17B-GGUF/Q4_K_M/Qwen3.5-397B-A17B-Q4_K_M-00001-of-00005.gguf +EOF + +# Import into Ollama +ollama create qwen3.5-397b -f /tmp/Qwen3.5-397B.Modelfile + +# Ollama serves automatically on port 11434 +uv run python scripts/run_grid_search.py \ + --model "Qwen3.5-397B" --engine ollama --resume -n 5 + +# Unload when done +ollama stop qwen3.5-397b +``` + +### 3G. Kimi-K2.5 GGUF via llama.cpp + +```bash +# Terminal 1 +./llama.cpp/build/bin/llama-server \ + -m ~/models/Kimi-K2.5-GGUF/UD-IQ2_XXS/Kimi-K2.5-UD-IQ2_XXS-00001-of-00005.gguf \ + --n-gpu-layers 99 \ + --tensor-split 1,1,1,1 \ + --port 8080 \ + --ctx-size 8192 +``` + +```bash +# Terminal 2 +uv run python scripts/run_grid_search.py \ + --model "Kimi-K2.5" --engine llamacpp --resume -n 5 +``` + +### 3H. Kimi-K2.5 GGUF via Ollama + +```bash +cat > /tmp/Kimi-K2.5.Modelfile << 'EOF' +FROM ~/models/Kimi-K2.5-GGUF/UD-IQ2_XXS/Kimi-K2.5-UD-IQ2_XXS-00001-of-00005.gguf +EOF + +ollama create kimi-k2.5 -f /tmp/Kimi-K2.5.Modelfile + +uv run python scripts/run_grid_search.py \ + --model "Kimi-K2.5" --engine ollama --resume -n 5 + +ollama stop kimi-k2.5 +``` + +### 3I. GLM-5 GGUF via llama.cpp + +```bash +# Terminal 1 +./llama.cpp/build/bin/llama-server \ + -m ~/models/GLM-5-GGUF/UD-IQ2_XXS/GLM-5-UD-IQ2_XXS-00001-of-00005.gguf \ + --n-gpu-layers 99 \ + --tensor-split 1,1,1,1 \ + --port 8080 \ + --ctx-size 8192 +``` + +```bash +# Terminal 2 +uv run python scripts/run_grid_search.py \ + --model "GLM-5" --engine llamacpp --resume -n 5 +``` + +### 3J. GLM-5 GGUF via Ollama + +```bash +cat > /tmp/GLM-5.Modelfile << 'EOF' +FROM ~/models/GLM-5-GGUF/UD-IQ2_XXS/GLM-5-UD-IQ2_XXS-00001-of-00005.gguf +EOF + +ollama create glm-5 -f /tmp/GLM-5.Modelfile + +uv run python scripts/run_grid_search.py \ + --model "GLM-5" --engine ollama --resume -n 5 + +ollama stop glm-5 +``` + +--- + +## Phase 4: Recover Failed Runs + +If some runs fail (missing packages, engine not reachable, etc.), fix the issue, then: + +```bash +# Delete error summaries so --resume retries them +find results/grid-search -name "*.summary.json" \ + -exec grep -l '"error"' {} \; -delete + +# Re-run with --resume (only retries deleted/missing summaries) +uv run python scripts/run_grid_search.py --resume -n 5 +``` + +### Common Failures and Fixes + +| Error | Cause | Fix | +|-------|-------|-----| +| `No inference engine available` | `openai` package missing (LLM judge can't init) | `uv pip install openai` | +| `No module named 'datasets'` | Missing HF datasets package | `uv pip install datasets` | +| `terminal-bench package required` | Missing terminal-bench | `uv pip install terminal-bench` | +| `Dataset doesn't exist on the Hub` | Gated dataset or HF auth needed | `huggingface-cli login`, accept terms on HF website | +| `IPW data directory not found` | IPW uses local data, not HuggingFace | Place files in `src/openjarvis/evals/data/ipw/` | +| `natural-reasoning` 0 samples | Field name mismatch in dataset loader | Patch `src/openjarvis/evals/datasets/natural_reasoning.py` | + +--- + +## Phase 5: Analyze Results + +```bash +# Preview what ran +uv run python scripts/run_grid_search.py --dry-run --resume + +# Consolidated results (appended after each run) +cat results/grid-search/grid-results.jsonl + +# Per-run summaries +find results/grid-search -name "*.summary.json" | head -20 +cat results/grid-search/openai-gpt-oss-120b/vllm/simple/supergpqa.summary.json + +# Count completed vs failed +echo "Completed:" && find results/grid-search -name "*.summary.json" \ + -exec grep -L '"error"' {} \; | wc -l +echo "Failed:" && find results/grid-search -name "*.summary.json" \ + -exec grep -l '"error"' {} \; | wc -l +``` + +--- + +## Useful Flags + +```bash +# Preview the full matrix without running +uv run python scripts/run_grid_search.py --dry-run + +# Filter to a single model + engine +uv run python scripts/run_grid_search.py --model "gpt-oss" --engine vllm -n 5 + +# Filter to a single agent or benchmark +uv run python scripts/run_grid_search.py --agent native_react --benchmark supergpqa -n 5 + +# Increase sample count +uv run python scripts/run_grid_search.py -n 50 + +# Verbose logging +uv run python scripts/run_grid_search.py -v --resume -n 5 +``` + +--- + +## GPU Assignment Tips + +Use `CUDA_VISIBLE_DEVICES` to pin servers to specific GPUs: + +```bash +# Run vLLM on GPUs 0,1 and llama.cpp on GPUs 2,3 simultaneously +CUDA_VISIBLE_DEVICES=0,1 vllm serve openai/gpt-oss-120b --tensor-parallel-size 2 --port 8000 +CUDA_VISIBLE_DEVICES=2,3 ./llama.cpp/build/bin/llama-server -m model.gguf --n-gpu-layers 99 --port 8080 +``` + +This lets you run two model servers in parallel on different GPU pairs. From c36730320ca6297410d381b227ad93802e3b2728 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Mon, 16 Mar 2026 16:57:55 -0700 Subject: [PATCH 18/92] chore: remove contributor-specific gitignore entry and log memory injection errors - Remove openjarvis-bugfix-spec-v2.md from .gitignore (not a general pattern) - Replace bare `except: pass` with debug logging in routes.py memory injection Co-Authored-By: Claude Opus 4.6 (1M context) --- .gitignore | 1 - src/openjarvis/server/routes.py | 5 ++++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 8843988e..9eaf23ab 100644 --- a/.gitignore +++ b/.gitignore @@ -89,4 +89,3 @@ CLAUDE.md # Research output research_mining_* .python-version -openjarvis-bugfix-spec-v2.md diff --git a/src/openjarvis/server/routes.py b/src/openjarvis/server/routes.py index b59ec919..c6625cb4 100644 --- a/src/openjarvis/server/routes.py +++ b/src/openjarvis/server/routes.py @@ -2,6 +2,7 @@ from __future__ import annotations +import logging import uuid from typing import Any @@ -89,7 +90,9 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request )) request_body.messages = new_msgs except Exception: - pass # Don't break chat if memory retrieval fails + logging.getLogger("openjarvis.server").debug( + "Memory context injection failed", exc_info=True, + ) if request_body.stream: bus = getattr(request.app.state, "bus", None) From 181d9ac0eb85f1d1ed44ff89e9f36eeb57abee94 Mon Sep 17 00:00:00 2001 From: Tarun Suresh Date: Tue, 17 Mar 2026 00:13:49 +0000 Subject: [PATCH 19/92] fix: resolve ruff I001 import sorting and E501 line length in tests Co-Authored-By: Claude Opus 4.6 (1M context) --- tests/agents/test_executor_ephemeral.py | 10 ++++++---- .../test_system_prompt_builder_integration.py | 1 + tests/daemon/test_session_expiry.py | 1 + tests/security/test_credential_stripper.py | 5 ++++- tests/sessions/test_compression.py | 14 ++++++++++++-- tests/tools/test_memory_manage.py | 3 ++- tests/tools/test_skill_manage.py | 3 ++- tests/tools/test_user_profile_manage.py | 3 ++- 8 files changed, 30 insertions(+), 10 deletions(-) diff --git a/tests/agents/test_executor_ephemeral.py b/tests/agents/test_executor_ephemeral.py index 537181dd..16062d1a 100644 --- a/tests/agents/test_executor_ephemeral.py +++ b/tests/agents/test_executor_ephemeral.py @@ -2,6 +2,8 @@ from __future__ import annotations from unittest.mock import MagicMock, patch +REGISTRY_PATH = "openjarvis.core.registry.AgentRegistry.get" + def test_run_ephemeral_creates_and_runs_agent(): from openjarvis.agents.executor import AgentExecutor @@ -11,11 +13,11 @@ def test_run_ephemeral_creates_and_runs_agent(): mock_agent_cls = MagicMock() mock_agent_instance = MagicMock() - mock_agent_instance.run.return_value = MagicMock(content="Flushed 3 memories.") + mock_agent_instance.run.return_value = MagicMock(content="Flushed.") mock_agent_cls.return_value = mock_agent_instance - with patch("openjarvis.core.registry.AgentRegistry.get", return_value=mock_agent_cls): - result = executor.run_ephemeral( + with patch(REGISTRY_PATH, return_value=mock_agent_cls): + executor.run_ephemeral( agent_type="simple", system_prompt="Save important context.", input_text="Review and flush.", @@ -34,7 +36,7 @@ def test_run_ephemeral_passes_input(): mock_agent_instance.run.return_value = MagicMock(content="Done.") mock_agent_cls.return_value = mock_agent_instance - with patch("openjarvis.core.registry.AgentRegistry.get", return_value=mock_agent_cls): + with patch(REGISTRY_PATH, return_value=mock_agent_cls): executor.run_ephemeral( agent_type="simple", system_prompt="Test prompt.", diff --git a/tests/agents/test_system_prompt_builder_integration.py b/tests/agents/test_system_prompt_builder_integration.py index bb4cce2f..439b5e4c 100644 --- a/tests/agents/test_system_prompt_builder_integration.py +++ b/tests/agents/test_system_prompt_builder_integration.py @@ -1,6 +1,7 @@ from __future__ import annotations from pathlib import Path + from openjarvis.core.config import MemoryFilesConfig, SystemPromptConfig diff --git a/tests/daemon/test_session_expiry.py b/tests/daemon/test_session_expiry.py index 10da40e6..6f8082e7 100644 --- a/tests/daemon/test_session_expiry.py +++ b/tests/daemon/test_session_expiry.py @@ -1,6 +1,7 @@ from __future__ import annotations from unittest.mock import MagicMock + from openjarvis.core.types import Message, Role diff --git a/tests/security/test_credential_stripper.py b/tests/security/test_credential_stripper.py index 899ea1b4..01f7f781 100644 --- a/tests/security/test_credential_stripper.py +++ b/tests/security/test_credential_stripper.py @@ -4,7 +4,10 @@ from __future__ import annotations def test_strips_openai_key(): from openjarvis.security.credential_stripper import CredentialStripper stripper = CredentialStripper() - text = "Error: auth failed with key sk-proj-abc123def456ghi789jkl012mno345pqr678stu901vwx234" + text = ( + "Error: auth failed with key " + "sk-proj-abc123def456ghi789jkl012mno345pqr678stu901vwx234" + ) result = stripper.strip(text) assert "sk-proj-" not in result assert "[REDACTED:" in result diff --git a/tests/sessions/test_compression.py b/tests/sessions/test_compression.py index d4509bfb..0c29ee65 100644 --- a/tests/sessions/test_compression.py +++ b/tests/sessions/test_compression.py @@ -1,6 +1,7 @@ from __future__ import annotations import pytest + from openjarvis.core.registry import CompressionRegistry from openjarvis.core.types import Message, Role @@ -37,10 +38,19 @@ def test_rule_based_strips_tool_boilerplate(): from openjarvis.sessions.compression import RuleBasedPrecompression compressor = RuleBasedPrecompression() + long_snippet = "x" * 5000 + tool_output = ( + '{"results": [{"title": "Result 1",' + f' "snippet": "A very long snippet {long_snippet}"' + "}]}" + ) msgs = [ Message(role=Role.ASSISTANT, content="Let me search."), - Message(role=Role.TOOL, content='{"results": [{"title": "Result 1", "snippet": "A very long snippet ' + "x" * 5000 + '"}]}'), - Message(role=Role.ASSISTANT, content="Based on the search, here is the answer."), + Message(role=Role.TOOL, content=tool_output), + Message( + role=Role.ASSISTANT, + content="Based on the search, here is the answer.", + ), ] result = compressor.compress(msgs, threshold=0.5) total_len = sum(len(m.content) for m in result) diff --git a/tests/tools/test_memory_manage.py b/tests/tools/test_memory_manage.py index 27c83c7e..6c063018 100644 --- a/tests/tools/test_memory_manage.py +++ b/tests/tools/test_memory_manage.py @@ -1,8 +1,9 @@ from __future__ import annotations -import pytest from pathlib import Path +import pytest + @pytest.fixture def memory_file(tmp_path: Path) -> Path: diff --git a/tests/tools/test_skill_manage.py b/tests/tools/test_skill_manage.py index 7987938d..ac651028 100644 --- a/tests/tools/test_skill_manage.py +++ b/tests/tools/test_skill_manage.py @@ -1,8 +1,9 @@ from __future__ import annotations -import pytest from pathlib import Path +import pytest + @pytest.fixture def skills_dir(tmp_path: Path) -> Path: diff --git a/tests/tools/test_user_profile_manage.py b/tests/tools/test_user_profile_manage.py index 94ff51fb..dff0ef3c 100644 --- a/tests/tools/test_user_profile_manage.py +++ b/tests/tools/test_user_profile_manage.py @@ -1,8 +1,9 @@ from __future__ import annotations -import pytest from pathlib import Path +import pytest + @pytest.fixture def user_file(tmp_path: Path) -> Path: From 29beeed32d50a8bfefcdb143a3e4c095d9b69994 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon Date: Tue, 17 Mar 2026 03:24:26 +0000 Subject: [PATCH 20/92] docs: add contributor docs, community infrastructure, and roadmap rewrite - Add root CONTRIBUTING.md with incentives (paper acknowledgment, Mac Mini giveaway), contribution tiers, PR process, and maintainership path - Add CODE_OF_CONDUCT.md (Contributor Covenant v2.1) - Add .pre-commit-config.yaml with ruff lint + format hooks - Add GitHub issue templates (bug report, feature request, new eval dataset) - Add PR template with test/lint/format checklist - Rewrite docs roadmap with GitHub Projects structure, current focus areas, and collapsible version history - Remove Development section from MkDocs nav; replace with top-level Roadmap tab - Delete changelog, extending docs (consolidated into CONTRIBUTING.md) - Delete root ROADMAP.md (content now lives in docs site) - Add pre-commit to dev extras in pyproject.toml - Add Roadmap link to README Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/ISSUE_TEMPLATE/bug_report.yml | 95 +++ .github/ISSUE_TEMPLATE/config.yml | 5 + .github/ISSUE_TEMPLATE/feature_request.yml | 47 ++ .github/ISSUE_TEMPLATE/new_eval.yml | 63 ++ .github/pull_request_template.md | 16 + .pre-commit-config.yaml | 7 + CODE_OF_CONDUCT.md | 85 ++ CONTRIBUTING.md | 198 +++++ README.md | 2 + ROADMAP.md | 209 ----- docs/development/changelog.md | 330 -------- docs/development/contributing.md | 7 +- docs/development/extending.md | 915 --------------------- docs/development/roadmap.md | 120 ++- mkdocs.yml | 7 +- pyproject.toml | 1 + uv.lock | 83 ++ 17 files changed, 660 insertions(+), 1530 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature_request.yml create mode 100644 .github/ISSUE_TEMPLATE/new_eval.yml create mode 100644 .github/pull_request_template.md create mode 100644 .pre-commit-config.yaml create mode 100644 CODE_OF_CONDUCT.md create mode 100644 CONTRIBUTING.md delete mode 100644 ROADMAP.md delete mode 100644 docs/development/changelog.md delete mode 100644 docs/development/extending.md diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 00000000..f41c3caf --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,95 @@ +name: Bug Report +description: Report a bug or unexpected behavior +labels: ["type:bug"] +body: + - type: markdown + attributes: + value: | + Thank you for reporting a bug! Please fill out the information below to help us investigate. + - type: textarea + id: description + attributes: + label: Description + description: A clear description of what the bug is. + validations: + required: true + - type: textarea + id: steps + attributes: + label: Steps to Reproduce + description: Steps to reproduce the behavior. + placeholder: | + 1. Run `jarvis ask "..."` + 2. See error... + validations: + required: true + - type: textarea + id: expected + attributes: + label: Expected Behavior + description: What you expected to happen. + validations: + required: true + - type: textarea + id: actual + attributes: + label: Actual Behavior + description: What actually happened. + validations: + required: true + - type: dropdown + id: os + attributes: + label: Operating System + options: + - Linux + - macOS + - Windows + validations: + required: true + - type: dropdown + id: python + attributes: + label: Python Version + options: + - "3.10" + - "3.11" + - "3.12" + - "3.13" + validations: + required: true + - type: dropdown + id: hardware + attributes: + label: Hardware + options: + - NVIDIA GPU + - AMD GPU + - Apple Silicon + - CPU only + validations: + required: true + - type: dropdown + id: engine + attributes: + label: Engine + description: Which inference engine are you using? + options: + - Ollama + - vLLM + - llama.cpp + - SGLang + - MLX + - Cloud (OpenAI/Anthropic/Google) + - LiteLLM + - Other + validations: + required: false + - type: textarea + id: logs + attributes: + label: Logs / Traceback + description: Paste any relevant logs or traceback here. + render: shell + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..3445dfe0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: false +contact_links: + - name: Ask a Question + url: https://github.com/open-jarvis/OpenJarvis/discussions + about: Use Discussions for questions and help diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 00000000..3cc64939 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,47 @@ +name: Feature Request +description: Propose a new feature or enhancement +labels: ["type:feature"] +body: + - type: markdown + attributes: + value: | + For non-trivial changes, please open this issue for discussion before starting a PR. This saves everyone time by catching design issues early. + - type: textarea + id: problem + attributes: + label: Problem Statement + description: What problem does this solve? Why is this needed? + validations: + required: true + - type: textarea + id: solution + attributes: + label: Proposed Solution + description: Describe how you'd like this to work. + validations: + required: true + - type: dropdown + id: area + attributes: + label: Primitive Area + description: Which part of OpenJarvis does this touch? + options: + - Intelligence + - Engine + - Agent + - Tools + - Learning + - Evals + - Frontend + - Channels + - Rust + - Other + validations: + required: true + - type: textarea + id: alternatives + attributes: + label: Alternatives Considered + description: Any alternative solutions or features you've considered. + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/new_eval.yml b/.github/ISSUE_TEMPLATE/new_eval.yml new file mode 100644 index 00000000..28191efb --- /dev/null +++ b/.github/ISSUE_TEMPLATE/new_eval.yml @@ -0,0 +1,63 @@ +name: New Eval Dataset +description: Propose a new evaluation dataset or benchmark +labels: ["type:eval"] +body: + - type: markdown + attributes: + value: | + Adding eval datasets is one of the easiest ways to contribute! Fill out the details below. + - type: input + id: name + attributes: + label: Dataset Name + placeholder: e.g., HumanEval, GSM8K + validations: + required: true + - type: input + id: url + attributes: + label: URL / Reference + description: Link to the dataset or paper. + placeholder: https://... + validations: + required: true + - type: checkboxes + id: capability + attributes: + label: What capability does it test? + options: + - label: Reasoning + - label: Math + - label: Code + - label: Knowledge + - label: Multimodal + - label: Tool Use + - label: Long Context + - label: Other + - type: input + id: size + attributes: + label: Approximate Size + description: Number of examples in the dataset. + placeholder: e.g., 500 + validations: + required: false + - type: dropdown + id: scorer + attributes: + label: Proposed Scorer Type + options: + - Exact Match + - F1 + - BLEU + - LLM-as-Judge + - Custom + validations: + required: false + - type: textarea + id: context + attributes: + label: Additional Context + description: Any other details about this dataset. + validations: + required: false diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..78505ac4 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,16 @@ +## What does this PR do? + + + +## How was this tested? + + + +## Checklist + +- [ ] Tests pass (`uv run pytest tests/ -v`) +- [ ] Linter passes (`uv run ruff check src/ tests/`) +- [ ] Formatter passes (`uv run ruff format --check src/ tests/`) +- [ ] New/changed public API has docstrings +- [ ] Follows registry pattern (if adding new component) +- [ ] Documentation updated (if applicable) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 00000000..07adb89a --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,7 @@ +repos: + - repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.9.0 + hooks: + - id: ruff + args: [--fix] + - id: ruff-format diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..d38d68fd --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,85 @@ + +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience +* Focusing on what is best not just for us as individuals, but for the overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [the project maintainers](https://github.com/open-jarvis/OpenJarvis/discussions). All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series of actions. + +**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. + +For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations]. + +[homepage]: https://www.contributor-covenant.org +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..1a953370 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,198 @@ +# Contributing to OpenJarvis + +Thank you for your interest in contributing to OpenJarvis! This guide covers everything you need to know — from why to contribute, to how to submit your first pull request. + +--- + +## Why Contribute? + +Contributing to OpenJarvis isn't just about code — it's about building the future of on-device AI together. Here's what you get: + +### Paper Acknowledgment + +All contributors with merged pull requests will be acknowledged as contributors on the OpenJarvis paper release. + +### Mac Mini Giveaway + +We're giving away a Mac Mini to one lucky contributor! Install OpenJarvis on your personal machine and opt in via the desktop app to share anonymized savings data (FLOPs, dollar cost, energy) for a chance to win. Your data is fully anonymous — no IP, no hardware info beyond savings metrics. You must share your email via the desktop app to be eligible. + +See the [Savings Leaderboard](https://open-jarvis.github.io/OpenJarvis/leaderboard/) for details. + +### Path to Maintainership + +Consistent contributors can grow into project maintainers: + +- **Contributor** — anyone with a merged PR +- **Reviewer** — invited after 3+ merged PRs in a domain area, can review PRs +- **Maintainer** — reviewers who demonstrate sustained engagement and good judgment + +### Recognition + +Contributors are recognized in release notes and on our GitHub repository. + +--- + +## Ways to Contribute + +### Good First Contributions + +These are great starting points for new contributors: + +- Documentation improvements and typo fixes +- Bug reports with reproducible steps +- New eval datasets and scorers +- Test coverage improvements + +Look for issues labeled [`good-first-issue`](https://github.com/open-jarvis/OpenJarvis/labels/good-first-issue). + +### Ideal Contributions + +- Bug fixes with tests +- Performance improvements +- New tools, engines, or agents following the [registry pattern](docs/development/contributing.md#registry-pattern) +- New channel integrations (Telegram, Discord, Slack, etc.) + +### Harder to Review + +These require more context and review time. **Please open an issue for discussion before starting a PR:** + +- New primitives or major extensions to existing ones +- Large refactors +- Changes to core abstractions (`BaseAgent`, `InferenceEngine`, etc.) + +### May Not Be Accepted + +To avoid wasted effort, note that PRs in these categories are unlikely to be merged: + +- Changes that break backwards compatibility in the public API +- Changes that add significant new dependencies without justification +- Changes that add friction to the user experience + +--- + +## Getting Started + +### Prerequisites + +| Requirement | Version | Notes | +|---|---|---| +| Python | 3.10+ | Required | +| [uv](https://docs.astral.sh/uv/) | Latest | Package manager | +| Node.js | 22+ | Only needed for ClaudeCodeAgent and WhatsApp channel | + +### Setup + +```bash +git clone https://github.com/open-jarvis/OpenJarvis.git +cd OpenJarvis +uv sync --extra dev +``` + +### Pre-commit Hooks + +We use [pre-commit](https://pre-commit.com/) to run linting and formatting checks before each commit: + +```bash +uv run pre-commit install +``` + +This installs Git hooks that automatically run [Ruff](https://docs.astral.sh/ruff/) on every commit. If the hooks fail, fix the issues and commit again. + +For detailed development setup, code conventions, and project structure, see the [Development Guide](docs/development/contributing.md). + +--- + +## Claiming Issues + +Want to work on an issue? Comment **"take"** on the issue to claim it. This lets others know you're working on it and prevents duplicate effort. + +If you've claimed an issue but can't finish it, please leave a comment so someone else can pick it up. + +--- + +## Proposing Changes + +### Trivial Changes + +For small fixes (typos, doc improvements, simple bug fixes), go ahead and open a PR directly. + +### Non-trivial Changes + +For larger changes — new features, refactors, new dependencies — **open an issue first** to discuss the approach. This saves everyone time by catching design issues early. + +Use the appropriate [issue template](https://github.com/open-jarvis/OpenJarvis/issues/new/choose): +- **Bug Report** — for bugs with reproduction steps +- **Feature Request** — for new functionality +- **New Eval Dataset** — for contributing benchmarks + +--- + +## Pull Request Process + +### Before Submitting + +1. Run the full test suite: + ```bash + uv run pytest tests/ -v + ``` +2. Run the linter: + ```bash + uv run ruff check src/ tests/ + ``` +3. Run the formatter: + ```bash + uv run ruff format --check src/ tests/ + ``` +4. Add tests for new functionality +5. Follow the [registry pattern](docs/development/contributing.md#registry-pattern) for new components + +### Commit Messages + +We use [Conventional Commits](https://www.conventionalcommits.org/): + +``` +feat: add FAISS memory backend +fix: handle empty tool responses in orchestrator +docs: update engine discovery documentation +test: add coverage for BM25 backend +refactor: simplify agent base class helpers +``` + +Keep the first line under 72 characters. Reference relevant issues (e.g., `fixes #42`). + +### What Makes a Good PR + +- **Focused** — one feature, fix, or refactor per PR +- **Tested** — includes unit tests covering new code paths +- **Documented** — updates docstrings and docs if adding public API +- **Backwards compatible** — avoids breaking existing interfaces without discussion + +--- + +## Contribution Areas + +OpenJarvis is built on five composable primitives. Here's where you can contribute: + +| Area | What to Build | Guide | +|---|---|---| +| **Intelligence** | Model catalog entries, routing strategies | [Dev Guide](docs/development/contributing.md) | +| **Engines** | New inference backends (e.g., TensorRT, ONNX) | [Dev Guide](docs/development/contributing.md) | +| **Agents** | New agent types, agent improvements | [Dev Guide](docs/development/contributing.md) | +| **Tools** | New tools (browser, API clients, etc.) | [Dev Guide](docs/development/contributing.md) | +| **Learning** | Router policies, reward functions, training | [Dev Guide](docs/development/contributing.md) | +| **Evals** | New datasets, scorers, benchmark configs | [Dev Guide](docs/development/contributing.md) | +| **Channels** | Chat platform integrations | [Dev Guide](docs/development/contributing.md) | +| **Rust Port** | PyO3 bindings, crate parity with Python | See `rust/` directory | + +--- + +## Code of Conduct + +This project follows the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you agree to uphold this code. + +--- + +## Questions? + +- Open a [Discussion](https://github.com/open-jarvis/OpenJarvis/discussions) for questions and help +- Check the [documentation](https://open-jarvis.github.io/OpenJarvis/) for guides and API reference diff --git a/README.md b/README.md index 9f6e6cca..e7768888 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,8 @@ > **[Project Site](https://scalingintelligence.stanford.edu/blogs/openjarvis/)** > > **[Leaderboard](https://open-jarvis.github.io/OpenJarvis/leaderboard/)** +> +> **[Roadmap](https://open-jarvis.github.io/OpenJarvis/development/roadmap/)** ## Why OpenJarvis? diff --git a/ROADMAP.md b/ROADMAP.md deleted file mode 100644 index 26d1b8e0..00000000 --- a/ROADMAP.md +++ /dev/null @@ -1,209 +0,0 @@ -# OpenJarvis Roadmap — Personal AI on Personal Devices - -## Overview - -OpenJarvis studies personal AI through five core primitives — **Intelligence**, **Engine**, **Agents**, **Tools**, and **Learning** — running on personal hardware. This roadmap organizes development into five independent workstreams, each with items flowing from immediately actionable to exploratory research. - -### How to Read This Roadmap - -**Workstreams are independent.** Contributors can pick any track without waiting on others. Within each track, items are organized by time horizon: - -- **Near-term** — foundations and hardening of what exists -- **Mid-term** — significant new capabilities -- **Long-term** — frontier work requiring design exploration or research - -Every item carries a **maturity tag**: - -| Tag | Meaning | Contributor guidance | -|-----|---------|---------------------| -| **Ready** | Well-scoped, implementation path is clear | Pick it up — check issues for a spec or write one | -| **Design Needed** | Concept is clear but needs a spec before code | Start a design discussion or draft an RFC | -| **Research-Stage** | Exploratory, needs investigation before designing | Read the relevant papers, prototype, share findings | - -Items marked with **"good first issue"** are especially suited for new contributors. - ---- - -## Workstream 1: Continuous Operators & Agents - -Operators are OpenJarvis's key differentiator — persistent, scheduled, stateful agents that run autonomously on personal devices. The current tick-based architecture (OperatorManager → TaskScheduler → AgentExecutor → OperativeAgent) is solid but needs hardening for truly long-horizon autonomy. - -### Near-term - -| Item | Maturity | Details | -|------|----------|---------| -| Operator health checks & heartbeat monitoring | **Ready** | Add liveness probes to OperatorManager; surface in `jarvis operators status`. Detect stalled operators beyond the existing reconciliation loop. | -| Metrics collection for operator manifests | **Ready** | The `metrics` field exists in `OperatorManifest` but is not collected. Wire it to telemetry. Good first issue. | -| Capability policy enforcement | **Ready** | `required_capabilities` field exists in manifests but is not enforced. Connect to the existing RBAC `CapabilityPolicy` system. Good first issue. | -| Rate limiting per operator | **Ready** | Prevent runaway operators from hammering inference. Add configurable rate limits to OperatorManager. | -| Operator composition / chaining | **Design Needed** | Express dependencies between operators (operator A feeds results to operator B). Requires design for data passing and scheduling semantics. | - -### Mid-term - -| Item | Maturity | Details | -|------|----------|---------| -| Event-driven operators | **Design Needed** | Operators that trigger on EventBus events (e.g., new file indexed, channel message received) rather than only cron/interval schedules. | -| Operator versioning & rollback | **Design Needed** | Run v2 of an operator alongside v1. Roll back automatically on repeated failures. | -| Multi-device operator coordination | **Design Needed** | Operators spanning laptop + workstation + datacenter node. Requires device discovery and task delegation protocols. | -| Dynamic tool loading per operator | **Design Needed** | Runtime tool discovery rather than static string lists in TOML manifests. | - -### Long-term - -| Item | Maturity | Details | -|------|----------|---------| -| Self-improving operators via Learning | **Research-Stage** | Operators that use trace feedback to tune their own prompts, tool selection, and routing policies through the Learning primitive. | -| Distributed operator mesh | **Research-Stage** | Peer-to-peer operator coordination across devices without a central orchestrator. | - ---- - -## Workstream 2: Mobile & Messaging Clients - -Personal AI must be accessible from the devices people actually carry. OpenJarvis runs on laptops, workstations, and servers — users interact via their phones. Channels bridge that gap. Today, WhatsApp (Baileys), Slack, and Telegram are bidirectional; iMessage is send-only; Android SMS does not exist. Beyond the channels listed below, OpenJarvis already supports 20+ additional channels (Discord, Matrix, Mastodon, Nostr, IRC, Line, Viber, Email, etc.) — see `src/openjarvis/channels/` for the full list. - -### Near-term - -| Item | Maturity | Details | -|------|----------|---------| -| iMessage bidirectional via BlueBubbles | **Ready** | Current implementation is send-only. Add webhook/polling listener for incoming messages using the BlueBubbles API. Good first issue. | -| WhatsApp Baileys media support | **Ready** | Currently text-only. Add image, audio, and file handling to the Node.js bridge and Python channel. Good first issue. | -| Slack rich messages | **Ready** | Current implementation is plain text. Add Slack Block Kit support for formatted responses, buttons, and attachments. | -| Android SMS via Twilio/Vonage | **Design Needed** | No SMS implementation exists. Requires provider selection, two-way webhook architecture, and phone number provisioning flow. | - -### Mid-term - -| Item | Maturity | Details | -|------|----------|---------| -| Unified notification system | **Design Needed** | Push notifications when operators complete tasks or need user attention. Requires per-channel notification adapters. | -| Offline message queue with retry | **Design Needed** | Handle mobile unreliability — queue outbound messages, retry on reconnect, deduplicate. | -| Channel media pipeline | **Design Needed** | Unified image/audio/file handling across all channels with consistent metadata and storage. | -| Signal bidirectional | **Design Needed** | Currently send-only via signal-cli REST API. Add incoming message listener with background polling. | - -### Long-term - -| Item | Maturity | Details | -|------|----------|---------| -| Voice interface | **Research-Stage** | Speech-to-text (Whisper) → agent → text-to-speech loop over phone channels. Existing `speech/` module provides a foundation. | -| Cross-channel session continuity UX | **Design Needed** | Start a conversation on Slack, continue on WhatsApp seamlessly. The `SessionStore` already supports multi-channel identity linking — this needs UX and channel-level plumbing. | - ---- - -## Workstream 3: Secure Cloud Collaboration - -Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through three complementary approaches: **Minions-style collaborative inference** (local handles context, cloud handles reasoning), **TEE-based confidential computing** (cloud cannot see your data even during inference), and **secure multi-device coordination**. - -Related References: -- [Minions: Cost-Efficient Local-Cloud LLM Collaboration](https://github.com/HazyResearch/minions) -- [TEE for Confidential AI Inference](https://openreview.net/forum?id=ey87M5iKcX) -- [Tinfoil: Verifiably Private AI](https://tinfoil.sh) - -### Near-term - -| Item | Maturity | Details | -|------|----------|---------| -| Query complexity analyzer | **Ready** | Classify incoming queries by difficulty (token count, entity density, reasoning depth) to decide local vs. cloud routing. Extends the existing `MultiEngine` routing logic. | -| Cost tracking per-query | **Ready** | `CloudEngine` already has pricing data. Surface per-query cost in traces and telemetry dashboards. Good first issue. | -| Redaction-before-cloud pipeline | **Ready** | Wire the existing `GuardrailsEngine` in REDACT mode as a mandatory pre-step before any cloud transmission. The security primitives exist — this is integration work. | -| Minion protocol (sequential) | **Design Needed** | Local model extracts and summarizes long context → cloud model reasons over the compressed result. Native reimplementation of the core [Minions](https://github.com/HazyResearch/minions) idea in the `engine/` layer. | - -### Mid-term - -| Item | Maturity | Details | -|------|----------|---------| -| Minions protocol (parallel) | **Design Needed** | Local and cloud models work simultaneously on different aspects of a query; results are merged. Requires a new `HybridInferenceEngine` abstraction. Depends on: Minion protocol (sequential) from near-term. | -| Adaptive routing with learning | **Design Needed** | Router learns which queries need cloud vs. local based on trace feedback (accuracy, cost, latency). Connects the Engine and Learning primitives. | -| TEE attestation verification | **Design Needed** | Verify that cloud inference ran inside a trusted execution environment via cryptographic attestation. Add attestation checking to `CloudEngine` response handling. | -| Confidential inference provider support | **Design Needed** | Add Tinfoil (or similar TEE-backed providers) as a first-class engine backend. OpenAI-compatible API with attestation verification built in. | -| Taint tracking across local/cloud boundary | **Design Needed** | The `TaintSet` (Python: `security/taint.py`, Rust: `crates/openjarvis-security/src/taint.rs`) already tracks PII/Secret labels. Add routing enforcement at the `engine/` layer so tainted data only routes to attested TEE endpoints, never to unattested cloud APIs. | - -### Long-term - -| Item | Maturity | Details | -|------|----------|---------| -| Speculative decoding (local draft + cloud verify) | **Research-Stage** | Local model generates candidate tokens speculatively; cloud model validates in parallel for latency reduction. | -| KV cache sharing between local and cloud | **Research-Stage** | Transfer attention state between engines to avoid recomputation. Requires a shared cache serialization format and encrypted transport. | -| Secure multi-device federation | **Research-Stage** | Multiple personal devices collaborate on inference with end-to-end encryption. Extends operator mesh from Workstream 1. | -| Early exit detection | **Research-Stage** | When local model confidence is high, skip cloud entirely. Dynamic cost/quality tradeoff learned from traces. | - ---- - -## Workstream 4: Tutorials & Documentation - -OpenJarvis has strong reference docs and four tutorials (deep research, scheduled ops, messaging hub, code companion), but critical gaps remain in continuous agents, LM evaluation, learning approaches, and custom tools. Video tutorials are scoped as a contributor opportunity — written tutorials come first, with video scripts included so anyone can record. - -### Near-term - -| Item | Maturity | Details | -|------|----------|---------| -| "Building Continuous Agents" tutorial | **Ready** | Writing an operator TOML manifest, activating it, session persistence across ticks, state management, daemon mode. Example: a research operator that monitors arxiv daily. Follows the existing tutorial template (Python script + TOML recipe + markdown walkthrough). | -| "Adding Custom Tools" tutorial | **Ready** | Implementing `BaseTool`, registering via `ToolRegistry`, wiring into agents. Example: a weather API tool. `docs/development/extending.md` covers engine extensions but there is no standalone tools tutorial with a runnable end-to-end example. Good first issue. | -| "Testing & Comparing LMs" tutorial | **Ready** | Running benchmarks, comparing local vs. cloud models, interpreting telemetry (latency, cost, energy per token). Uses the existing `bench/` framework. | -| Per-platform installation guides | **Ready** | Expand `installation.md` with platform-specific walkthroughs: macOS Apple Silicon + Ollama, Ubuntu + NVIDIA + vLLM, Windows + Ollama, Raspberry Pi. Good first issue. | - -### Mid-term - -| Item | Maturity | Details | -|------|----------|---------| -| "Learning & Model Selection" tutorial | **Design Needed** | Router policies (heuristic, learned, GRPO), proposed approaches like Thompson Sampling, trace-based reward signals. This is the least-documented of the five primitives. | -| "Multi-Channel Deployment" tutorial | **Design Needed** | Deploying one agent across Slack + WhatsApp + iMessage simultaneously. Cross-channel session continuity. | -| Video tutorial infrastructure | **Design Needed** | Establish recording workflow, hosting (YouTube), MkDocs embedding. Write video scripts alongside written tutorials so contributors can record independently. | -| Interactive Jupyter notebook tutorials | **Design Needed** | Notebook versions of key tutorials for exploratory, cell-by-cell learning. | - -### Long-term - -| Item | Maturity | Details | -|------|----------|---------| -| Contributor tutorial program | **Research-Stage** | Templates and guidelines for community members to submit their own tutorials. Review process, quality bar, and integration with the docs site. | -| Tutorial localization (i18n) | **Research-Stage** | Translate core tutorials into major languages. | - ---- - -## Workstream 5: Hardware Breadth - -Personal AI means running on the hardware people actually own. Each new hardware target expands who can use OpenJarvis and generates data for the research agenda (energy, cost, latency tradeoffs across silicon). - -Adding a new hardware target involves up to four components: hardware detection in `core/config.py`, an inference engine adapter in `engine/`, an energy monitor in `telemetry/`, and an entry in the GPU specs database in `telemetry/gpu_monitor.py`. - -### Near-term - -| Item | Maturity | Details | -|------|----------|---------| -| Intel Arc GPU (B580/B570) | **Design Needed** | 12GB VRAM, ~$250 consumer GPU. Viable for 7-8B models. Engine path: IPEX-LLM or llama.cpp SYCL backend. Needs `_detect_intel_arc_gpu()`, energy monitor via RAPL/sysfs, engine adapter. | -| NVIDIA Jetson Orin | **Design Needed** | Best-in-class edge device. Orin NX 16GB handles 7-8B models at 15-25 tok/s. Ollama/llama.cpp already work; needs hardware detection, energy monitor (tegrastats), deployment guide. | -| Qualcomm Snapdragon X Elite NPU | **Design Needed** | 45 TOPS, Windows Arm laptops. ONNX Runtime + QNN Execution Provider is the viable path. Needs new engine adapter, hardware detection, energy monitor. | -| AMD Ryzen AI iGPU path | **Ready** | Strix Point RDNA 3.5 iGPU handles 7-8B via Vulkan. llama.cpp Vulkan backend works today. Needs hardware detection and energy monitor. Good first issue. | -| GPU specs database expansion | **Ready** | Add Intel Arc, Jetson Orin, Snapdragon specs to `GPU_SPECS` in `telemetry/gpu_monitor.py` (TFLOPS, bandwidth, TDP). Good first issue. | - -### Mid-term - -| Item | Maturity | Details | -|------|----------|---------| -| Intel Lunar Lake NPU via OpenVINO | **Design Needed** | 48 TOPS — most mature NPU software stack for x86 laptops. New engine wrapping OpenVINO GenAI. Good for offloading 1-3B models while GPU handles larger ones. | -| Qualcomm mobile (Snapdragon 8 Gen 3/4) | **Design Needed** | 1-7B on phones via QNN SDK. Ties into Workstream 2 — inference on the phone itself rather than relaying to a server. | -| Raspberry Pi 5 | **Design Needed** | CPU-only via llama.cpp ARM NEON for 1-3B models. $100 entry point for hobbyists. Hailo-8L NPU is not viable for LLMs (vision-only architecture). | -| Unified hardware benchmark suite | **Design Needed** | Standardized benchmark that runs the same workloads across all supported hardware, producing comparable energy/latency/throughput/cost numbers. | - -### Long-term - -| Item | Maturity | Details | -|------|----------|---------| -| MediaTek Dimensity NPU | **Research-Stage** | Most aggressive mobile LLM push (45-50 TOPS) but closed NeuroPilot SDK. Monitor for SDK openness or third-party framework support. | -| Hailo-10 | **Research-Stage** | Hailo announced generative AI targeting for next-gen hardware. Watch for availability and transformer support. Current Hailo-8/8L is vision-only. | -| AMD XDNA2/3 NPU | **Research-Stage** | 50 TOPS but software stack (Ryzen AI SDK) is immature for LLMs. Revisit as AMD improves tooling. The AMD Ryzen AI iGPU item above is the practical AMD target today. | -| Intel Gaudi 3 / Falcon Shores | **Research-Stage** | 128GB HBM, datacenter-class. Gaudi product line being discontinued in favor of Falcon Shores architecture. Wait for clarity before investing. | -| RISC-V + NPU SoCs | **Research-Stage** | 3-5 years behind ARM/x86. Watch Sophgo SG2380 and similar. Not actionable for production use yet. | - ---- - -## Contributing - -Each workstream is independent — pick the one that matches your skills and interests: - -| Workstream | Skills needed | -|------------|--------------| -| 1. Continuous Operators | Python, async/scheduling, agent systems | -| 2. Mobile & Messaging | Node.js (Baileys bridge), Python, messaging platform APIs | -| 3. Secure Cloud Collaboration | Cryptography, TEE/confidential computing, distributed systems, ML | -| 4. Tutorials & Documentation | Technical writing, MkDocs, video production | -| 5. Hardware Breadth | Systems programming, hardware-specific SDKs (IPEX-LLM, OpenVINO, QNN), Rust | - -**To get started:** Look for items tagged **Ready** — these have clear scope and are ready for implementation. Items tagged **good first issue** are especially approachable. Open an issue or discussion on the repo to claim work or propose a design for **Design Needed** items. diff --git a/docs/development/changelog.md b/docs/development/changelog.md deleted file mode 100644 index f231311e..00000000 --- a/docs/development/changelog.md +++ /dev/null @@ -1,330 +0,0 @@ -# Changelog - -All notable changes to OpenJarvis are documented in this file. - ---- - -## Unreleased — Phase 11 (NanoClaw Subsumption) - -*27 new files, ~3,565 lines, 147+ new tests. Full suite: 2059+ tests pass.* - -### Added - -- **`ClaudeCodeAgent`** (`agents/claude_code.py`) -- Wraps the - `@anthropic-ai/claude-code` SDK via a bundled Node.js subprocess bridge. - Communicates over stdin/stdout using sentinel-delimited JSON - (`---OPENJARVIS_OUTPUT_START---` / `---OPENJARVIS_OUTPUT_END---`). The - bundled runner is auto-installed to `~/.openjarvis/claude_code_runner/` via - `npm install --production` on first use. Registered as `"claude_code"` with - `accepts_tools = False`. Requires Node.js 22+ and `ANTHROPIC_API_KEY`. -- **`WhatsAppBaileysChannel`** (`channels/whatsapp_baileys.py`) -- Bidirectional - WhatsApp messaging using the Baileys protocol. Spawns a Node.js bridge - subprocess (`whatsapp_baileys_bridge/`) for QR-code authentication, incoming - message forwarding, and outbound delivery via JID addressing. Registered as - `"whatsapp_baileys"` in `ChannelRegistry`. Authentication state is persisted - to `~/.openjarvis/whatsapp_baileys_bridge/auth/`. New config section: - `[channel.whatsapp_baileys]`. -- **`ContainerRunner`** (`sandbox/runner.py`) -- Manages Docker (or Podman) - container lifecycle for sandboxed agent execution. Builds `docker run --rm - --network none -i` commands with allowlist-validated read-only bind mounts. - Supports configurable image, timeout, concurrent container limit, and runtime - binary. Uses the same sentinel-delimited JSON protocol as `ClaudeCodeAgent`. -- **`SandboxedAgent`** (`sandbox/runner.py`) -- Transparent wrapper that runs - any `BaseAgent` inside a container via `ContainerRunner`. Follows the - `GuardrailsEngine` wrapper pattern. `accepts_tools = False`. -- **`MountAllowlist` / `validate_mounts()`** (`sandbox/mount_security.py`) -- - Port of NanoClaw's `mount-security.ts`. Validates bind mounts against a JSON - allowlist (allowed root directories + blocked filename patterns). Raises - `ValueError` for blocked or out-of-root paths before the container starts. - Default blocked patterns include `.ssh`, `.env`, `*.pem`, `*.key`, credential - files, and cloud config directories. -- **`TaskScheduler`** (`scheduler/scheduler.py`) -- Background polling scheduler - supporting three schedule types: `cron` (via `croniter` or built-in fallback), - `interval` (seconds), and `once` (ISO 8601 datetime). Runs a daemon thread - (`jarvis-scheduler`) polling SQLite every 60 seconds (configurable). Executes - due tasks via `JarvisSystem.ask()` with optional agent and tool selection. - Publishes `scheduler_task_start` / `scheduler_task_end` events on the - `EventBus`. New config section: `[scheduler]`. -- **`SchedulerStore`** (`scheduler/store.py`) -- SQLite CRUD backend for - scheduled tasks and run logs. Two tables: `scheduled_tasks` (task state) and - `task_run_logs` (execution history). Supports task filtering by status and - due-time polling via `get_due_tasks()`. -- **Scheduler MCP tools** (`scheduler/tools.py`) -- Five new MCP-discoverable - tools registered in `ToolRegistry`: - - `schedule_task` -- Create a new scheduled task - - `list_scheduled_tasks` -- List tasks filtered by status - - `pause_scheduled_task` -- Pause an active task - - `resume_scheduled_task` -- Resume a paused task (recomputes `next_run`) - - `cancel_scheduled_task` -- Permanently cancel a task -- **Scheduler CLI commands** -- `jarvis scheduler` subcommand group: - - `jarvis scheduler create` -- Create a new scheduled task - - `jarvis scheduler list` -- List all or filtered tasks - - `jarvis scheduler pause ` -- Pause a task - - `jarvis scheduler resume ` -- Resume a task - - `jarvis scheduler cancel ` -- Cancel a task - - `jarvis scheduler logs ` -- Show run history for a task - - `jarvis scheduler start` -- Start the background scheduler daemon - -### Changed - -- `ChannelRegistry` now includes `WhatsAppBaileysChannel`. -- `AgentRegistry` now includes `ClaudeCodeAgent` (`"claude_code"`). -- Architecture overview and source directory layout updated to reflect new - `sandbox/` and `scheduler/` modules. - ---- - -## Unreleased — Phase 10 Tooling Updates - -### Added - -- **`build_tool_descriptions()` shared builder** -- Single source of truth for - generating enriched tool descriptions in agent system prompts. Produces - Markdown sections with name, description, category, and parameter schemas. -- **Enriched agent prompts** -- `NativeReActAgent`, `NativeOpenHandsAgent`, - `RLMAgent`, and `OrchestratorAgent` (structured mode) now inject detailed - tool descriptions into their system prompts via the shared builder. -- **Case-insensitive parsing** -- ReAct (`Action:` / `Final Answer:`) and - Orchestrator structured-mode parsing (`TOOL:` / `FINAL_ANSWER:`) are now - case-insensitive. -- **Multi-provider tool_calls extraction** -- `CloudEngine` now extracts - `tool_calls` from Anthropic (`tool_use` content blocks) and Google - (`function_call` parts), normalizing to the flat `{id, name, arguments}` - format. `LiteLLM` engine handles the flat-format tool calls returned by - the LiteLLM proxy. -- **RLM tool awareness** -- `RLMAgent` injects an `## Available Tools` - section into its system prompt when tools are provided. -- **Orchestrator structured tool descriptions** -- Structured mode passes - `tools=self._tools` to `build_system_prompt()` for enriched descriptions. -- **Telemetry modules** -- `EfficiencyMetrics`, `GPUMonitor`, `VLLMMetrics` - for energy, GPU utilization, and vLLM server-side metrics collection. -- **Eval TOML config** -- TOML-based eval suite configuration system for - defining models x benchmarks matrices. - -### Changed - -- Agent prompt generation now uses `build_tool_descriptions()` instead of - inline tool name listing. -- `build_system_prompt()` in `prompt_registry.py` accepts an optional `tools` - parameter for enriched descriptions from `BaseTool` instances. -- ReAct and OpenHands regex patterns updated for case-insensitive matching. - -### Fixed - -- Engine `tool_calls` normalization -- Anthropic `tool_use` blocks and Google - `function_call` parts are now correctly extracted and converted to the - standard flat format used by agents. - ---- - -## v0.1.0 - -*Phase 5 -- SDK, Production Readiness, and Documentation* - -### Added - -- **Python SDK** -- `Jarvis` class providing a high-level sync API for - programmatic use - - `ask()` / `ask_full()` methods for direct engine and agent mode queries - - `MemoryHandle` proxy for lazy memory backend initialization - - `list_models()` and `list_engines()` for runtime introspection - - Router policy selection via config (`learning.default_policy`) - - Lazy engine initialization with automatic discovery and health probing - - Resource cleanup via `close()` -- **Benchmarking framework** - - `BaseBenchmark` ABC and `BenchmarkSuite` runner - - `LatencyBenchmark` measuring per-call latency (mean, p50, p95, min, max) - - `ThroughputBenchmark` measuring tokens-per-second throughput - - `BenchmarkResult` dataclass with JSONL export - - `jarvis bench run` CLI with options for model, engine, sample count, - benchmark selection, and JSON/JSONL output -- **Docker deployment** - - `Dockerfile` -- Multi-stage Python 3.12-slim build with `[server]` extra - - `Dockerfile.gpu` -- NVIDIA CUDA 12.4 runtime variant - - `docker-compose.yml` -- Services for `jarvis` (port 8000) and `ollama` - (port 11434) - - `deploy/systemd/openjarvis.service` -- systemd unit file for Linux - - `deploy/launchd/com.openjarvis.plist` -- launchd plist for macOS -- **Documentation site** -- MkDocs Material with mkdocstrings, covering - getting started, user guide, architecture, API reference, deployment, and - development - ---- - -## v0.5.0 - -*Phase 4 -- Learning, Telemetry, and Router Policies* - -### Added - -- **Learning system** - - `RouterPolicy` ABC and `RoutingContext` dataclass - - `RewardFunction` ABC for scoring inference results - - `HeuristicRewardFunction` scoring on latency, cost, and efficiency - - `RouterPolicyRegistry` for pluggable routing strategies - - `HeuristicRouter` registered as `"heuristic"` policy (6 priority rules: - code detection, math detection, short/long queries, urgency override, - default fallback) - - `TraceDrivenPolicy` registered as `"learned"` policy with batch updates - via `update_from_traces()` and online updates via `observe()` - - `GRPORouterPolicy` stub registered as `"grpo"` for future RL training - - `ensure_registered()` pattern for lazy, test-safe registration -- **Telemetry aggregation** - - `TelemetryAggregator` with `per_model_stats()`, `per_engine_stats()`, - `top_models()`, `summary()`, `export_records()`, and `clear()` methods - - Time-range filtering via `since` / `until` parameters - - `ModelStats` and `EngineStats` dataclasses - - `AggregatedStats` summary dataclass -- **CLI enhancements** - - `--router` flag on `jarvis ask` for explicit policy selection - - `jarvis telemetry stats` -- display aggregated telemetry statistics - - `jarvis telemetry export --format json|csv` -- export telemetry records - - `jarvis telemetry clear --yes` -- delete all telemetry records - ---- - -## v0.4.0 - -*Phase 3 -- Agents, Tools, and API Server* - -### Added - -- **Agent system** - - `BaseAgent` ABC with `run()` method returning `AgentResult` - - `AgentContext` dataclass with conversation, tools, and memory results - - `AgentResult` dataclass with content, tool results, turns, and metadata - - `AgentRegistry` for pluggable agent implementations - - `SimpleAgent` -- single-turn query-to-response, no tool calling - - `OrchestratorAgent` -- multi-turn tool-calling loop with `ToolExecutor`, - configurable `max_turns` - - `CustomAgent` -- template for user-defined agent behavior -- **Tool system** - - `BaseTool` ABC with `spec` property and `execute()` method - - `ToolSpec` dataclass describing tool interface and characteristics - - `ToolExecutor` dispatch engine with JSON argument parsing, latency - tracking, and event bus integration (`TOOL_CALL_START` / `TOOL_CALL_END`) - - `ToolRegistry` for tool discovery - - `to_openai_function()` method for OpenAI function calling format - - Built-in tools: - - `CalculatorTool` -- safe math evaluation via AST parsing - - `ThinkTool` -- reasoning scratchpad for chain-of-thought - - `RetrievalTool` -- memory search integration - - `LLMTool` -- sub-model calls within agent loops - - `FileReadTool` -- safe file reading with path validation -- **OpenAI-compatible API server** (`jarvis serve`) - - FastAPI + Uvicorn with optional `[server]` extra - - `POST /v1/chat/completions` -- non-streaming and SSE streaming - - `GET /v1/models` -- list available models - - `GET /health` -- health check endpoint - - Pydantic request/response models matching OpenAI API format - ---- - -## v0.3.0 - -*Phase 2 -- Memory System* - -### Added - -- **Memory backends** - - `MemoryBackend` ABC with `store()`, `retrieve()`, `delete()`, `clear()` - - `RetrievalResult` dataclass with content, score, source, and metadata - - `MemoryRegistry` for backend discovery - - `SQLiteMemory` -- zero-dependency default using SQLite FTS5 with BM25 - ranking and FTS5 query escaping - - `FAISSMemory` -- vector search using FAISS with sentence-transformers - embeddings (optional `[memory-faiss]` extra) - - `ColBERTMemory` -- ColBERTv2 neural retrieval backend (optional - `[memory-colbert]` extra) - - `BM25Memory` -- BM25 ranking backend using rank-bm25 (optional - `[memory-bm25]` extra) - - `HybridMemory` -- Reciprocal Rank Fusion combining multiple backends -- **Document processing** - - `ChunkConfig` dataclass for chunk size and overlap settings - - `chunk_text()` for splitting documents into overlapping chunks - - `ingest_path()` for recursively indexing files and directories - - `read_document()` with support for plain text, Markdown, and PDF - (optional `[memory-pdf]` extra) -- **Context injection** - - `ContextConfig` with top-k, minimum score, and max context token settings - - `inject_context()` for prepending memory results as system messages with - source attribution - - `--no-context` flag on `jarvis ask` to disable injection -- **CLI commands** - - `jarvis memory index ` -- index documents into memory - - `jarvis memory search ` -- search memory for relevant chunks - - `jarvis memory stats` -- show backend statistics -- **Event bus integration** -- `MEMORY_STORE` and `MEMORY_RETRIEVE` events - ---- - -## v0.2.0 - -*Phase 1 -- Intelligence and Inference* - -### Added - -- **Intelligence primitive** - - `ModelSpec` dataclass with parameter count, context length, quantization, - VRAM requirements, and supported engines - - `ModelRegistry` for model metadata storage - - `BUILTIN_MODELS` catalog with pre-defined model specifications - - `register_builtin_models()` and `merge_discovered_models()` helpers - - `HeuristicRouter` with rule-based model selection - - `build_routing_context()` for query analysis (code detection, math - detection, length classification) -- **Inference engines** - - `InferenceEngine` ABC with `generate()`, `stream()`, `list_models()`, - and `health()` methods - - `EngineRegistry` for engine discovery - - `OllamaEngine` -- Ollama backend via native HTTP API with tool call - extraction - - `VllmEngine` -- vLLM backend via OpenAI-compatible API - - `LlamaCppEngine` -- llama.cpp server backend - - `EngineConnectionError` for unreachable engines - - `messages_to_dicts()` for Message-to-OpenAI-format conversion -- **Engine discovery** - - `discover_engines()` -- probe all registered engines for health - - `discover_models()` -- aggregate model lists across engines - - `get_engine()` -- get configured default with automatic fallback -- **Hardware detection** - - NVIDIA GPU detection via `nvidia-smi` - - AMD GPU detection via `rocm-smi` - - Apple Silicon detection via `system_profiler` - - CPU brand detection via `/proc/cpuinfo` and `sysctl` - - `recommend_engine()` mapping hardware to best engine -- **Telemetry** - - `TelemetryRecord` dataclass with timing, tokens, energy, and cost - - `TelemetryStore` with SQLite persistence and EventBus subscription - - `instrumented_generate()` wrapper for automatic telemetry recording -- **CLI** - - `jarvis ask ` -- query via discovered engine - - `jarvis ask --agent simple ` -- route through SimpleAgent - - `jarvis model list` -- list models from running engines - - `jarvis model info ` -- show model details - ---- - -## v0.1.0 - -*Phase 0 -- Project Scaffolding* - -### Added - -- **Project structure** -- `hatchling` build backend, `uv` package manager, - `pyproject.toml` with extras for optional backends -- **Registry system** -- `RegistryBase[T]` generic base class with - class-specific entry isolation, `register()` decorator, `get()`, `create()`, - `items()`, `keys()`, `contains()`, `clear()` methods -- **Typed registries** -- `ModelRegistry`, `EngineRegistry`, `MemoryRegistry`, - `AgentRegistry`, `ToolRegistry`, `RouterPolicyRegistry`, `BenchmarkRegistry` -- **Core types** -- `Role` enum, `Message`, `Conversation` (with sliding - window), `ModelSpec`, `Quantization` enum, `ToolCall`, `ToolResult`, - `TelemetryRecord`, `StepType` enum, `TraceStep`, `Trace` -- **Configuration** -- `JarvisConfig` dataclass hierarchy, TOML loader with - overlay semantics, hardware auto-detection, `generate_default_toml()` for - `jarvis init` -- **Event bus** -- Synchronous pub/sub `EventBus` with `EventType` enum for - inter-primitive communication -- **CLI skeleton** -- Click-based `jarvis` command group with `--version`, - `--help`, and `init` subcommand diff --git a/docs/development/contributing.md b/docs/development/contributing.md index fc32b52c..357581b8 100644 --- a/docs/development/contributing.md +++ b/docs/development/contributing.md @@ -1,5 +1,10 @@ # Contributing Guide +!!! tip "Looking for how to contribute?" + See our [Contributing Guide](../../CONTRIBUTING.md) for an overview of + contribution types, incentives, and the PR process. This page covers + detailed development setup and code conventions. + This guide covers how to set up a development environment, run tests, and contribute code to OpenJarvis. @@ -412,4 +417,4 @@ policy: 5. Add an entry in `pyproject.toml` under `[project.optional-dependencies]` if the component requires new packages -See the [Extending OpenJarvis](extending.md) guide for complete examples. +See the [registry pattern](#registry-pattern) section above for complete examples. diff --git a/docs/development/extending.md b/docs/development/extending.md deleted file mode 100644 index 6a991ae2..00000000 --- a/docs/development/extending.md +++ /dev/null @@ -1,915 +0,0 @@ -# Extending OpenJarvis - -OpenJarvis is designed to be extended through its registry pattern. Every -major subsystem defines an abstract base class (ABC) and uses a typed registry -for runtime discovery. To add a new component, implement the ABC, decorate -it with the registry, and import it in the module's `__init__.py`. - -This guide provides complete, working code examples for each extension point. - ---- - -## Adding a New Inference Engine - -Inference engines connect OpenJarvis to an LLM runtime. All engines implement -the `InferenceEngine` ABC defined in `engine/_stubs.py`. - -### Step 1: Create the Engine Module - -Create `src/openjarvis/engine/my_engine.py`: - -```python -"""My custom inference engine backend.""" - -from __future__ import annotations - -from collections.abc import AsyncIterator, Sequence -from typing import Any, Dict, List - -import httpx - -from openjarvis.core.registry import EngineRegistry -from openjarvis.core.types import Message -from openjarvis.engine._base import ( - EngineConnectionError, - InferenceEngine, - messages_to_dicts, -) - - -@EngineRegistry.register("my_engine") # (1)! -class MyEngine(InferenceEngine): - """Custom inference engine backend.""" - - engine_id = "my_engine" # (2)! - - def __init__( - self, - host: str = "http://localhost:9000", - *, - timeout: float = 120.0, - ) -> None: - self._host = host.rstrip("/") - self._client = httpx.Client(base_url=self._host, timeout=timeout) - - def generate( - self, - messages: Sequence[Message], - *, - model: str, - temperature: float = 0.7, - max_tokens: int = 1024, - **kwargs: Any, - ) -> Dict[str, Any]: - """Synchronous completion.""" - payload = { - "model": model, - "messages": messages_to_dicts(messages), # (3)! - "temperature": temperature, - "max_tokens": max_tokens, - } - # Pass tools if provided - tools = kwargs.get("tools") - if tools: - payload["tools"] = tools - - try: - resp = self._client.post("/v1/chat/completions", json=payload) - resp.raise_for_status() - except (httpx.ConnectError, httpx.TimeoutException) as exc: - raise EngineConnectionError( - f"Engine not reachable at {self._host}" - ) from exc - - data = resp.json() - choice = data.get("choices", [{}])[0] - message = choice.get("message", {}) - usage = data.get("usage", {}) - - result: Dict[str, Any] = { - "content": message.get("content", ""), - "usage": { - "prompt_tokens": usage.get("prompt_tokens", 0), - "completion_tokens": usage.get("completion_tokens", 0), - "total_tokens": usage.get("total_tokens", 0), - }, - "model": data.get("model", model), - "finish_reason": choice.get("finish_reason", "stop"), - } - - # Extract tool calls if present - raw_tool_calls = message.get("tool_calls", []) - if raw_tool_calls: - result["tool_calls"] = [ - { - "id": tc.get("id", f"call_{i}"), - "name": tc["function"]["name"], - "arguments": tc["function"]["arguments"], - } - for i, tc in enumerate(raw_tool_calls) - ] - return result - - async def stream( - self, - messages: Sequence[Message], - *, - model: str, - temperature: float = 0.7, - max_tokens: int = 1024, - **kwargs: Any, - ) -> AsyncIterator[str]: - """Yield token strings as they are generated.""" - # Implement SSE or WebSocket streaming for your engine - result = self.generate( - messages, model=model, temperature=temperature, - max_tokens=max_tokens, **kwargs, - ) - yield result.get("content", "") - - def list_models(self) -> List[str]: - """Return identifiers of models available on this engine.""" - try: - resp = self._client.get("/v1/models") - resp.raise_for_status() - data = resp.json() - return [m["id"] for m in data.get("data", [])] - except Exception: - return [] - - def health(self) -> bool: - """Return True when the engine is reachable and healthy.""" - try: - resp = self._client.get("/health", timeout=2.0) - return resp.status_code == 200 - except Exception: - return False -``` - -1. The `@EngineRegistry.register("my_engine")` decorator makes this engine - discoverable by key at runtime. -2. The `engine_id` class attribute is used in telemetry and benchmark results. -3. `messages_to_dicts()` converts `Message` objects to OpenAI-format dicts. - -### Step 2: Register in `__init__.py` - -Add your engine import to `src/openjarvis/engine/__init__.py`: - -```python -import openjarvis.engine.my_engine # noqa: F401 -``` - -If your engine requires optional dependencies, wrap the import: - -```python -try: - import openjarvis.engine.my_engine # noqa: F401 -except ImportError: - pass -``` - -### Step 3: Add Optional Dependencies - -If your engine needs extra packages, add them to `pyproject.toml`: - -```toml -[project.optional-dependencies] -inference-myengine = [ - "my-engine-sdk>=1.0", -] -``` - -### Required ABC Methods - -| Method | Signature | Returns | Description | -|---|---|---|---| -| `generate` | `(messages, *, model, temperature, max_tokens, **kwargs)` | `Dict[str, Any]` | Synchronous completion with `content` and `usage` keys | -| `stream` | `(messages, *, model, temperature, max_tokens, **kwargs)` | `AsyncIterator[str]` | Yields token strings as they are generated | -| `list_models` | `()` | `List[str]` | Model identifiers available on this engine | -| `health` | `()` | `bool` | `True` when the engine is reachable | - -The `generate` return dict must include at minimum: - -```python -{ - "content": "The response text", - "usage": { - "prompt_tokens": 10, - "completion_tokens": 20, - "total_tokens": 30, - }, - "model": "model-name", - "finish_reason": "stop", # or "tool_calls" -} -``` - -!!! tip "Tool call support" - If your engine supports tool/function calling, include a `"tool_calls"` - key in the return dict. Each tool call should have `id`, `name`, and - `arguments` (JSON string) keys. - ---- - -## Adding a New Memory Backend - -Memory backends provide persistent, searchable storage. All backends implement -the `MemoryBackend` ABC defined in `tools/storage/_stubs.py` (previously `memory/_stubs.py`). - -### Complete Example - -Create `src/openjarvis/tools/storage/my_backend.py`: - -```python -"""Custom memory backend example.""" - -from __future__ import annotations - -from typing import Any, Dict, List, Optional - -from openjarvis.core.registry import MemoryRegistry -from openjarvis.tools.storage._stubs import MemoryBackend, RetrievalResult - - -@MemoryRegistry.register("my_backend") -class MyMemoryBackend(MemoryBackend): - """Custom memory backend implementation.""" - - backend_id = "my_backend" - - def __init__(self, **kwargs: Any) -> None: - # Initialize your storage (database, index, etc.) - self._store: Dict[str, Dict[str, Any]] = {} - - def store( - self, - content: str, - *, - source: str = "", - metadata: Optional[Dict[str, Any]] = None, - ) -> str: - """Persist content and return a unique document id.""" - import uuid - - doc_id = uuid.uuid4().hex - self._store[doc_id] = { - "content": content, - "source": source, - "metadata": metadata or {}, - } - return doc_id - - def retrieve( - self, - query: str, - *, - top_k: int = 5, - **kwargs: Any, - ) -> List[RetrievalResult]: - """Search for query and return the top-k results.""" - results: List[RetrievalResult] = [] - for doc_id, doc in self._store.items(): - # Implement your search/ranking logic here - if query.lower() in doc["content"].lower(): - results.append(RetrievalResult( - content=doc["content"], - score=1.0, - source=doc["source"], - metadata=doc["metadata"], - )) - return results[:top_k] - - def delete(self, doc_id: str) -> bool: - """Delete a document by id. Return True if it existed.""" - return self._store.pop(doc_id, None) is not None - - def clear(self) -> None: - """Remove all stored documents.""" - self._store.clear() -``` - -### Register in `__init__.py` - -Add to `src/openjarvis/tools/storage/__init__.py`: - -```python -try: - import openjarvis.tools.storage.my_backend # noqa: F401 -except ImportError: - pass -``` - -!!! note "Backward compatibility" - The old `from openjarvis.memory._stubs import MemoryBackend` import path still works via backward-compatibility shims, but new code should use `openjarvis.tools.storage._stubs`. - -### Required ABC Methods - -| Method | Signature | Returns | Description | -|---|---|---|---| -| `store` | `(content, *, source, metadata)` | `str` | Persist content, return document ID | -| `retrieve` | `(query, *, top_k, **kwargs)` | `List[RetrievalResult]` | Search and return ranked results | -| `delete` | `(doc_id)` | `bool` | Delete by ID, return whether it existed | -| `clear` | `()` | `None` | Remove all stored documents | - -The `RetrievalResult` dataclass has these fields: - -```python -@dataclass(slots=True) -class RetrievalResult: - content: str # The retrieved text - score: float = 0.0 # Relevance score - source: str = "" # Source identifier - metadata: Dict[str, Any] = field(default_factory=dict) -``` - ---- - -## Adding a New Agent - -Agents implement the logic for handling queries, calling tools, and managing -multi-turn interactions. There are two paths depending on whether your agent -uses tools: - -- **Path A: Non-tool agent** -- Extend `BaseAgent` directly -- **Path B: Tool-using agent** -- Extend `ToolUsingAgent` (which sets `accepts_tools = True` and provides a `ToolExecutor`) - -### Path A: Non-tool Agent (extending BaseAgent) - -Create `src/openjarvis/agents/my_agent.py`: - -```python -"""Custom agent implementation — single-turn, no tools.""" - -from __future__ import annotations - -from typing import Any, Optional - -from openjarvis.agents._stubs import AgentContext, AgentResult, BaseAgent -from openjarvis.core.registry import AgentRegistry -from openjarvis.engine._stubs import InferenceEngine - - -@AgentRegistry.register("my_agent") -class MyAgent(BaseAgent): - """Custom agent with specialized behavior.""" - - agent_id = "my_agent" - - def run( - self, - input: str, - context: Optional[AgentContext] = None, - **kwargs: Any, - ) -> AgentResult: - """Execute the agent on input and return an AgentResult.""" - # Use BaseAgent helpers instead of manual event bus code - self._emit_turn_start(input) - - # Build messages from context + user input (with optional system prompt) - messages = self._build_messages( - input, context, - system_prompt="You are a helpful assistant with specialized knowledge.", - ) - - # Call engine.generate() with stored defaults (model, temperature, max_tokens) - result = self._generate(messages) - content = self._strip_think_tags(result.get("content", "")) - - self._emit_turn_end(turns=1) - return AgentResult(content=content, turns=1) -``` - -!!! tip "BaseAgent helpers" - `BaseAgent` provides these concrete helpers so you don't need to manually - manage the event bus or engine calls: - - | Helper | Purpose | - |--------|---------| - | `_emit_turn_start(input)` | Publish `AGENT_TURN_START` | - | `_emit_turn_end(**data)` | Publish `AGENT_TURN_END` | - | `_build_messages(input, context, *, system_prompt)` | Assemble message list | - | `_generate(messages, **kwargs)` | Call engine with stored defaults | - | `_strip_think_tags(text)` | Remove `` blocks | - | `_max_turns_result(tool_results, turns, content)` | Standard max-turns result | - -### Path B: Tool-using Agent (extending ToolUsingAgent) - -Create `src/openjarvis/agents/my_tool_agent.py`: - -```python -"""Custom tool-using agent with a multi-turn loop.""" - -from __future__ import annotations - -from typing import Any, List, Optional - -from openjarvis.agents._stubs import AgentContext, AgentResult, ToolUsingAgent -from openjarvis.core.events import EventBus -from openjarvis.core.registry import AgentRegistry -from openjarvis.core.types import ToolCall, ToolResult -from openjarvis.engine._stubs import InferenceEngine -from openjarvis.tools._stubs import BaseTool - - -@AgentRegistry.register("my_tool_agent") -class MyToolAgent(ToolUsingAgent): - """Custom agent with tool-calling loop.""" - - agent_id = "my_tool_agent" - - def __init__( - self, - engine: InferenceEngine, - model: str, - *, - tools: Optional[List[BaseTool]] = None, - bus: Optional[EventBus] = None, - max_turns: int = 10, - temperature: float = 0.7, - max_tokens: int = 1024, - ) -> None: - super().__init__( - engine, model, tools=tools, bus=bus, - max_turns=max_turns, temperature=temperature, - max_tokens=max_tokens, - ) - - def run( - self, - input: str, - context: Optional[AgentContext] = None, - **kwargs: Any, - ) -> AgentResult: - self._emit_turn_start(input) - - messages = self._build_messages(input, context) - tools_spec = self._executor.get_openai_tools() - all_tool_results: list[ToolResult] = [] - turns = 0 - - for _ in range(self._max_turns): - turns += 1 - result = self._generate(messages, tools=tools_spec) - content = result.get("content", "") - tool_calls = result.get("tool_calls", []) - - if not tool_calls: - self._emit_turn_end(turns=turns) - return AgentResult( - content=content, - tool_results=all_tool_results, - turns=turns, - ) - - # Execute each tool call - for tc in tool_calls: - call = ToolCall( - id=tc.get("id", f"call_{turns}"), - name=tc["name"], - arguments=tc["arguments"], - ) - tr = self._executor.execute(call) - all_tool_results.append(tr) - - # Max turns exceeded — use the standard helper - return self._max_turns_result(all_tool_results, turns) -``` - -!!! info "What ToolUsingAgent adds" - `ToolUsingAgent` extends `BaseAgent` with: - - - **`accepts_tools = True`** — enables `--tools` in CLI and `tools=` in SDK - - **`self._executor`** — a `ToolExecutor` initialized from the provided tools - - **`self._tools`** — the raw list of `BaseTool` instances - - **`self._max_turns`** — configurable loop iteration limit (default: 10) - -### Register in `__init__.py` - -Add to `src/openjarvis/agents/__init__.py`: - -```python -try: - import openjarvis.agents.my_agent # noqa: F401 -except ImportError: - pass -``` - -### Key Types - -=== "AgentContext" - - ```python - @dataclass(slots=True) - class AgentContext: - conversation: Conversation = field(default_factory=Conversation) - tools: List[str] = field(default_factory=list) - memory_results: List[Any] = field(default_factory=list) - metadata: Dict[str, Any] = field(default_factory=dict) - ``` - -=== "AgentResult" - - ```python - @dataclass(slots=True) - class AgentResult: - content: str - tool_results: List[ToolResult] = field(default_factory=list) - turns: int = 0 - metadata: Dict[str, Any] = field(default_factory=dict) - ``` - ---- - -## Adding a New Tool - -Tools are callable capabilities that agents can invoke during multi-turn -reasoning. All tools implement the `BaseTool` ABC from `tools/_stubs.py`. - -### Complete Example - -Create `src/openjarvis/tools/my_tool.py`: - -```python -"""Custom tool implementation.""" - -from __future__ import annotations - -from typing import Any - -from openjarvis.core.registry import ToolRegistry -from openjarvis.core.types import ToolResult -from openjarvis.tools._stubs import BaseTool, ToolSpec - - -@ToolRegistry.register("my_tool") -class MyTool(BaseTool): - """A custom tool that does something useful.""" - - tool_id = "my_tool" - - @property - def spec(self) -> ToolSpec: - """Return the tool specification.""" - return ToolSpec( - name="my_tool", - description="Does something useful with the provided input.", - parameters={ # (1)! - "type": "object", - "properties": { - "query": { - "type": "string", - "description": "The input to process", - }, - "max_results": { - "type": "integer", - "description": "Maximum number of results to return", - "default": 5, - }, - }, - "required": ["query"], - }, - category="utility", - cost_estimate=0.001, # Estimated cost in USD per call - latency_estimate=0.5, # Estimated latency in seconds - requires_confirmation=False, - ) - - def execute(self, **params: Any) -> ToolResult: - """Execute the tool with the given parameters.""" - query = params.get("query", "") - max_results = params.get("max_results", 5) - - if not query: - return ToolResult( - tool_name="my_tool", - content="No query provided.", - success=False, - ) - - try: - # Your tool logic here - result_text = f"Processed '{query}' (max_results={max_results})" - - return ToolResult( - tool_name="my_tool", - content=result_text, - success=True, - ) - except Exception as exc: - return ToolResult( - tool_name="my_tool", - content=f"Error: {exc}", - success=False, - ) -``` - -1. The `parameters` dict follows the [JSON Schema](https://json-schema.org/) - format used by OpenAI function calling. The `ToolExecutor` will parse - incoming JSON arguments and pass them as keyword arguments to `execute()`. - -### Register in `__init__.py` - -Add to `src/openjarvis/tools/__init__.py`: - -```python -try: - import openjarvis.tools.my_tool # noqa: F401 -except ImportError: - pass -``` - -### How Tools Are Invoked - -The `ToolExecutor` handles the dispatch loop: - -1. The agent's LLM generates a `tool_calls` response with tool name and - JSON arguments -2. `ToolExecutor.execute()` parses the JSON arguments -3. The matching tool's `execute(**params)` is called -4. The `ToolResult` is returned to the agent for the next turn - -```python -from openjarvis.tools._stubs import ToolExecutor - -executor = ToolExecutor( - tools=[MyTool()], - bus=event_bus, # Optional — enables TOOL_CALL_START/END events -) - -# Dispatch a tool call -from openjarvis.core.types import ToolCall - -call = ToolCall(id="call_1", name="my_tool", arguments='{"query": "test"}') -result = executor.execute(call) -``` - -The `to_openai_function()` method converts a tool's spec to OpenAI function -calling format, which is sent to the LLM alongside the conversation: - -```python -tool = MyTool() -openai_format = tool.to_openai_function() -# { -# "type": "function", -# "function": { -# "name": "my_tool", -# "description": "Does something useful...", -# "parameters": { ... } -# } -# } -``` - ---- - -## Adding a New Benchmark - -Benchmarks measure engine performance. All benchmarks implement the -`BaseBenchmark` ABC from `bench/_stubs.py` and use the `ensure_registered()` -pattern for lazy registration. - -### Complete Example - -Create `src/openjarvis/bench/my_benchmark.py`: - -```python -"""Custom benchmark — measures time to first token.""" - -from __future__ import annotations - -import time - -from openjarvis.bench._stubs import BaseBenchmark, BenchmarkResult -from openjarvis.core.registry import BenchmarkRegistry -from openjarvis.core.types import Message, Role -from openjarvis.engine._stubs import InferenceEngine - - -class TTFTBenchmark(BaseBenchmark): - """Measures time-to-first-token across multiple samples.""" - - @property - def name(self) -> str: - return "ttft" - - @property - def description(self) -> str: - return "Measures time-to-first-token latency" - - def run( - self, - engine: InferenceEngine, - model: str, - *, - num_samples: int = 10, - ) -> BenchmarkResult: - ttft_values: list[float] = [] - errors = 0 - - for _ in range(num_samples): - messages = [Message(role=Role.USER, content="Hello")] - t0 = time.time() - try: - engine.generate(messages, model=model) - ttft_values.append(time.time() - t0) - except Exception: - errors += 1 - - if not ttft_values: - return BenchmarkResult( - benchmark_name=self.name, - model=model, - engine=engine.engine_id, - metrics={}, - samples=num_samples, - errors=errors, - ) - - return BenchmarkResult( - benchmark_name=self.name, - model=model, - engine=engine.engine_id, - metrics={ - "mean_ttft": sum(ttft_values) / len(ttft_values), - "min_ttft": min(ttft_values), - "max_ttft": max(ttft_values), - }, - samples=num_samples, - errors=errors, - ) - - -def ensure_registered() -> None: # (1)! - """Register the TTFT benchmark if not already present.""" - if not BenchmarkRegistry.contains("ttft"): - BenchmarkRegistry.register_value("ttft", TTFTBenchmark) -``` - -1. The `ensure_registered()` function uses `contains()` before - `register_value()` so it can be called multiple times safely. This is - required because tests clear all registries between runs. - -### Register in `__init__.py` - -Update `src/openjarvis/bench/__init__.py` to call `ensure_registered()`: - -```python -from openjarvis.bench.my_benchmark import ensure_registered as _reg_ttft -_reg_ttft() -``` - -### BenchmarkResult Fields - -```python -@dataclass(slots=True) -class BenchmarkResult: - benchmark_name: str # e.g. "latency", "throughput" - model: str # Model identifier - engine: str # Engine identifier - metrics: Dict[str, float] = ... # Measured values - metadata: Dict[str, Any] = ... # Extra info - samples: int = 0 # Number of samples run - errors: int = 0 # Number of failed samples -``` - ---- - -## Adding a New Router Policy - -Router policies determine which model handles a given query. All policies -implement the `RouterPolicy` ABC from `learning/_stubs.py`. The -`RoutingContext` dataclass is defined in `core/types.py`. - -### Complete Example - -Create `src/openjarvis/learning/my_policy.py`: - -```python -"""Custom router policy — selects model based on query length.""" - -from __future__ import annotations - -from typing import List, Optional - -from openjarvis.core.registry import RouterPolicyRegistry -from openjarvis.core.types import RoutingContext -from openjarvis.learning._stubs import RouterPolicy - - -class QueryLengthPolicy(RouterPolicy): - """Routes queries to models based on query length. - - Short queries go to a fast, small model. Long or complex queries - go to a larger, more capable model. - """ - - def __init__( - self, - available_models: Optional[List[str]] = None, - *, - default_model: str = "", - fallback_model: str = "", - short_threshold: int = 100, - long_threshold: int = 500, - ) -> None: - self._available = available_models or [] - self._default = default_model - self._fallback = fallback_model - self._short_threshold = short_threshold - self._long_threshold = long_threshold - - def select_model(self, context: RoutingContext) -> str: - """Return the model registry key best suited for this context.""" - available = self._available - - if not available: - return self._default or self._fallback or "" - - if context.query_length < self._short_threshold: - # Prefer the first (presumably smallest) available model - return available[0] - elif context.query_length > self._long_threshold: - # Prefer the last (presumably largest) available model - return available[-1] - - # Default to configured model - if self._default and self._default in available: - return self._default - return available[0] - - -def ensure_registered() -> None: - """Register QueryLengthPolicy if not already present.""" - if not RouterPolicyRegistry.contains("query_length"): - RouterPolicyRegistry.register_value("query_length", QueryLengthPolicy) - - -ensure_registered() -``` - -### Register in `__init__.py` - -Update `src/openjarvis/learning/__init__.py`: - -```python -from openjarvis.learning.my_policy import ensure_registered as _reg_ql -_reg_ql() -``` - -### Using Your Policy - -Once registered, your policy can be selected via the config file or CLI: - -=== "Config (TOML)" - - ```toml - [learning.routing] - policy = "query_length" - ``` - -=== "CLI" - - ```bash - uv run jarvis ask --router query_length "Hello" - ``` - -### The RoutingContext - -The `RoutingContext` dataclass provides all the information a router needs: - -```python -@dataclass(slots=True) -class RoutingContext: - query: str = "" - query_length: int = 0 - has_code: bool = False - has_math: bool = False - language: str = "en" - urgency: float = 0.5 # 0 = low priority, 1 = real-time - metadata: Dict[str, Any] = field(default_factory=dict) -``` - -The `build_routing_context()` helper in `learning/router.py` populates -this from a raw query string, detecting code and math patterns automatically. - ---- - -## Summary - -| Component | ABC | Registry | Key location | -|---|---|---|---| -| Inference Engine | `InferenceEngine` | `EngineRegistry` | `engine/_stubs.py` | -| Memory Backend | `MemoryBackend` | `MemoryRegistry` | `tools/storage/_stubs.py` | -| Agent | `BaseAgent` | `AgentRegistry` | `agents/_stubs.py` | -| Tool | `BaseTool` | `ToolRegistry` | `tools/_stubs.py` | -| Benchmark | `BaseBenchmark` | `BenchmarkRegistry` | `bench/_stubs.py` | -| Router Policy | `RouterPolicy` | `RouterPolicyRegistry` | `learning/_stubs.py` | -| Learning Policy | `LearningPolicy` | `LearningRegistry` | `learning/_stubs.py` | - -The general pattern for all extension points: - -1. Implement the ABC in a new module -2. Decorate the class with `@XRegistry.register("key")` or use - `ensure_registered()` for lazy registration -3. Import the module in the package's `__init__.py` (with `try/except - ImportError` if optional deps are involved) -4. Add tests in `tests//` -5. Add optional dependencies to `pyproject.toml` if needed diff --git a/docs/development/roadmap.md b/docs/development/roadmap.md index b9fed431..87774e61 100644 --- a/docs/development/roadmap.md +++ b/docs/development/roadmap.md @@ -1,11 +1,58 @@ # Roadmap -OpenJarvis development follows a phased approach, with each version adding -a major primitive or cross-cutting capability to the framework. +OpenJarvis uses **GitHub Projects** boards organized by domain to plan and +track work. Each board has quarterly columns so contributors can see what's +coming and where help is needed. --- -## Development Phases +## How We Plan + +- Work is organized into domain-specific project boards (see below) +- Each board uses quarterly columns: **Backlog → Q2 2026 → Q3 2026 → Q4 2026 → Future** +- Issues are labeled by difficulty (`good-first-issue`, `help-wanted`) and type (`type:bug`, `type:feature`, `type:perf`, `type:docs`, `type:eval`) +- Domain labels (`domain:agents`, `domain:engine`, `domain:tools`, etc.) connect issues to the right board + +Want to contribute? Check the boards below, pick an issue labeled `good-first-issue` or `help-wanted`, and comment **"take"** to claim it. + +--- + +## Active Project Boards + +| Board | Scope | Link | +|---|---|---| +| **Agents & Tools** | Agent types, tool implementations, MCP | *TBD — boards will be linked once created on GitHub* | +| **Engine & Inference** | Engine backends, streaming, performance | *TBD* | +| **Learning & Routing** | GRPO, trace-driven policies, rewards | *TBD* | +| **Evals & Benchmarks** | Datasets, scorers, benchmark infra | *TBD* | +| **Frontend & Desktop** | Tauri app, dashboard, leaderboard | *TBD* | +| **Rust Port** | PyO3 bindings, crate parity | *TBD* | + +--- + +## Current Focus Areas + +These are the areas where active development is happening and contributions are most impactful: + +- **GRPO training from trace data** — moving router policies beyond heuristics using reinforcement learning from execution traces +- **Multi-model orchestration pipelines** — coordinating multiple models within a single query (e.g., small model for classification, large model for generation) +- **Energy-aware routing** — using power consumption data from telemetry to optimize for energy efficiency alongside latency and quality +- **Plugin ecosystem** — community-contributed engines, tools, and agents distributed as Python packages +- **Federated memory** — memory backends that synchronize across devices + +--- + +## How to Get Involved + +1. Browse the [project boards](#active-project-boards) for issues that interest you +2. Look for `good-first-issue` and `help-wanted` labels +3. Read the [Contributing Guide](../../CONTRIBUTING.md) for the full process +4. Comment **"take"** on an issue to claim it + +--- + +
+Version History | Version | Phase | Status | Delivers | |---|---|---|---| @@ -18,69 +65,4 @@ a major primitive or cross-cutting capability to the framework. | **v1.1** | Phase 6 -- Traces + Learning | :material-check-circle:{ .green } Complete | Trace system (`TraceStore`, `TraceCollector`, `TraceAnalyzer`), trace-driven learning, MCP integration layer | | **v1.5** | Phase 10 -- Agent Restructuring | :material-check-circle:{ .green } Complete | BaseAgent helpers, ToolUsingAgent intermediate base, NativeReActAgent, NativeOpenHandsAgent, RLMAgent, OpenHandsAgent (SDK), `accepts_tools` introspection, backward-compat shims, CustomAgent removed | ---- - -## Current Status - -OpenJarvis v1.5 (Phase 10) is complete. The framework provides: - -- **Four core abstractions** -- Intelligence, Engine, Agentic Logic, Memory -- each with an ABC interface and registry-based discovery -- **Five inference engines** -- Ollama, vLLM, llama.cpp, SGLang, Cloud (OpenAI/Anthropic/Google) -- **Five memory backends** -- SQLite/FTS5, FAISS, ColBERTv2, BM25, Hybrid (RRF fusion) -- **Seven agent types** -- Simple, Orchestrator, NativeReAct, NativeOpenHands, RLM, Operative, MonitorOperative -- **Seven built-in tools** -- Calculator, Think, Retrieval, LLM, FileRead, WebSearch, CodeInterpreter -- **Python SDK** -- `Jarvis` class for programmatic use -- **OpenAI-compatible API server** -- `POST /v1/chat/completions`, `GET /v1/models` -- **Benchmarking framework** -- Latency and throughput measurements -- **Telemetry and traces** -- SQLite-backed recording and aggregation -- **Docker deployment** -- CPU and GPU images with docker-compose - -Phase 10 (Agent Restructuring) is complete. The agent hierarchy has been -refactored with `BaseAgent` helpers, `ToolUsingAgent` intermediate base, and -four new agent types (NativeReActAgent, NativeOpenHandsAgent, RLMAgent, -OpenHandsAgent SDK). - ---- - -## Phase 10 Details - -Phase 10 refactored the agent hierarchy for composability and extensibility: - -### BaseAgent Helpers - -- **`_emit_turn_start` / `_emit_turn_end`** -- Event bus integration without boilerplate -- **`_build_messages`** -- System prompt + context + input assembly -- **`_generate`** -- Engine call with stored defaults -- **`_max_turns_result`** -- Standard max-turns-exceeded result -- **`_strip_think_tags`** -- Remove `` blocks from model output - -### ToolUsingAgent Intermediate Base - -- Sets `accepts_tools = True` for CLI/SDK introspection -- Initializes `ToolExecutor` from provided tools -- Configurable `max_turns` loop limit - -### New Agent Types - -- **NativeReActAgent** (`native_react`, alias `react`) -- Thought-Action-Observation loop -- **NativeOpenHandsAgent** (`native_openhands`) -- CodeAct-style code execution with URL pre-fetching -- **RLMAgent** (`rlm`) -- Recursive LM with persistent REPL and sub-LM calls -- **OpenHandsAgent** (`openhands`) -- Thin wrapper for real `openhands-sdk` - ---- - -## Future Directions - -Beyond Phase 10, areas of ongoing exploration include: - -- **GRPO training** -- Reinforcement learning from trace data to train the - routing policy, moving beyond heuristics and simple statistics -- **Streaming telemetry** -- Real-time performance dashboards and alerting -- **Multi-model orchestration** -- Coordinating multiple models within a - single query pipeline (e.g., small model for classification, large model - for generation) -- **Federated memory** -- Memory backends that synchronize across devices -- **Plugin ecosystem** -- Community-contributed engines, tools, and agents - distributed as Python packages -- **Energy-aware routing** -- Using power consumption data from telemetry to - optimize for energy efficiency alongside latency and quality +
diff --git a/mkdocs.yml b/mkdocs.yml index 495f86aa..73869262 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -163,10 +163,5 @@ nav: - Security: architecture/security.md - Design Principles: architecture/design-principles.md - API Reference: api-reference/ - - Development: - - Contributing: development/contributing.md - - Extending OpenJarvis: development/extending.md - - Deployment: deployment/index.md - - Roadmap: development/roadmap.md - - Changelog: development/changelog.md - Leaderboard: leaderboard.md + - Roadmap: development/roadmap.md diff --git a/pyproject.toml b/pyproject.toml index 6a50980e..1ec5947b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,6 +25,7 @@ dev = [ "pytest-cov>=5", "respx>=0.22", "ruff>=0.4", + "pre-commit>=3.0", ] inference-mlx = ["mlx-lm>=0.19; sys_platform == 'darwin'"] inference-vllm = ["vllm>=0.16.0"] diff --git a/uv.lock b/uv.lock index f05db6d8..051d776a 100644 --- a/uv.lock +++ b/uv.lock @@ -893,6 +893,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ae/3a/dbeec9d1ee0844c679f6bb5d6ad4e9f198b1224f4e7a32825f47f6192b0c/cffi-2.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9", size = 184195, upload-time = "2025-09-08T23:23:43.004Z" }, ] +[[package]] +name = "cfgv" +version = "3.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4e/b5/721b8799b04bf9afe054a3899c6cf4e880fcf8563cc71c15610242490a0c/cfgv-3.5.0.tar.gz", hash = "sha256:d5b1034354820651caa73ede66a6294d6e95c1b00acc5e9b098e917404669132", size = 7334, upload-time = "2025-11-19T20:55:51.612Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl", hash = "sha256:a8dc6b26ad22ff227d2634a65cb388215ce6cc96bbcc5cfde7641ae87e8dacc0", size = 7445, upload-time = "2025-11-19T20:55:50.744Z" }, +] + [[package]] name = "charset-normalizer" version = "3.4.4" @@ -1618,6 +1627,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" }, ] +[[package]] +name = "distlib" +version = "0.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/96/8e/709914eb2b5749865801041647dc7f4e6d00b549cfe88b65ca192995f07c/distlib-0.4.0.tar.gz", hash = "sha256:feec40075be03a04501a973d81f633735b4b69f98b05450592310c0f401a4e0d", size = 614605, upload-time = "2025-07-17T16:52:00.465Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl", hash = "sha256:9659f7d87e46584a30b5780e43ac7a2143098441670ff0a49d5f9034c54a6c16", size = 469047, upload-time = "2025-07-17T16:51:58.613Z" }, +] + [[package]] name = "distro" version = "1.9.0" @@ -2681,6 +2699,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a8/af/48ac8483240de756d2438c380746e7130d1c6f75802ef22f3c6d49982787/huggingface_hub-0.36.2-py3-none-any.whl", hash = "sha256:48f0c8eac16145dfce371e9d2d7772854a4f591bcb56c9cf548accf531d54270", size = 566395, upload-time = "2026-02-06T09:24:11.133Z" }, ] +[[package]] +name = "identify" +version = "2.6.18" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/46/c4/7fb4db12296cdb11893d61c92048fe617ee853f8523b9b296ac03b43757e/identify-2.6.18.tar.gz", hash = "sha256:873ac56a5e3fd63e7438a7ecbc4d91aca692eb3fefa4534db2b7913f3fc352fd", size = 99580, upload-time = "2026-03-15T18:39:50.319Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/46/33/92ef41c6fad0233e41d3d84ba8e8ad18d1780f1e5d99b3c683e6d7f98b63/identify-2.6.18-py2.py3-none-any.whl", hash = "sha256:8db9d3c8ea9079db92cafb0ebf97abdc09d52e97f4dcf773a2e694048b7cd737", size = 99394, upload-time = "2026-03-15T18:39:48.915Z" }, +] + [[package]] name = "idna" version = "3.11" @@ -4099,6 +4126,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/df/93/a7b983643d1253bb223234b5b226e69de6cda02b76cdca7770f684b795f5/ninja-1.13.0-py3-none-win_arm64.whl", hash = "sha256:3c0b40b1f0bba764644385319028650087b4c1b18cdfa6f45cb39a3669b81aa9", size = 290806, upload-time = "2025-08-11T15:10:18.018Z" }, ] +[[package]] +name = "nodeenv" +version = "1.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/24/bf/d1bda4f6168e0b2e9e5958945e01910052158313224ada5ce1fb2e1113b8/nodeenv-1.10.0.tar.gz", hash = "sha256:996c191ad80897d076bdfba80a41994c2b47c68e224c542b48feba42ba00f8bb", size = 55611, upload-time = "2025-12-20T14:08:54.006Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl", hash = "sha256:5bb13e3eed2923615535339b3c620e76779af4cb4c6a90deccc9e36b274d3827", size = 23438, upload-time = "2025-12-20T14:08:52.782Z" }, +] + [[package]] name = "numba" version = "0.61.2" @@ -4594,6 +4630,7 @@ dashboard = [ ] dev = [ { name = "maturin" }, + { name = "pre-commit" }, { name = "pytest" }, { name = "pytest-asyncio" }, { name = "pytest-cov" }, @@ -4752,6 +4789,7 @@ requires-dist = [ { name = "pdfplumber", marker = "extra == 'pdf'", specifier = ">=0.10" }, { name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" }, { name = "praw", marker = "extra == 'channel-reddit'", specifier = ">=7.0" }, + { name = "pre-commit", marker = "extra == 'dev'", specifier = ">=3.0" }, { name = "pydantic", marker = "extra == 'server'", specifier = ">=2.0" }, { name = "pymessenger", marker = "extra == 'channel-messenger'", specifier = ">=0.0.7" }, { name = "pynostr", marker = "extra == 'channel-nostr'", specifier = ">=0.6" }, @@ -5457,6 +5495,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/96/5c/8af904314e42d5401afcfaff69940dc448e974f80f7aa39b241a4fbf0cf1/prawcore-2.4.0-py3-none-any.whl", hash = "sha256:29af5da58d85704b439ad3c820873ad541f4535e00bb98c66f0fbcc8c603065a", size = 17203, upload-time = "2023-10-01T23:30:47.651Z" }, ] +[[package]] +name = "pre-commit" +version = "4.5.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cfgv" }, + { name = "identify" }, + { name = "nodeenv" }, + { name = "pyyaml" }, + { name = "virtualenv" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/40/f1/6d86a29246dfd2e9b6237f0b5823717f60cad94d47ddc26afa916d21f525/pre_commit-4.5.1.tar.gz", hash = "sha256:eb545fcff725875197837263e977ea257a402056661f09dae08e4b149b030a61", size = 198232, upload-time = "2025-12-16T21:14:33.552Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5d/19/fd3ef348460c80af7bb4669ea7926651d1f95c23ff2df18b9d24bab4f3fa/pre_commit-4.5.1-py2.py3-none-any.whl", hash = "sha256:3b3afd891e97337708c1674210f8eba659b52a38ea5f822ff142d10786221f77", size = 226437, upload-time = "2025-12-16T21:14:32.409Z" }, +] + [[package]] name = "prometheus-client" version = "0.24.1" @@ -6377,6 +6431,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, ] +[[package]] +name = "python-discovery" +version = "1.1.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "filelock" }, + { name = "platformdirs" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d7/7e/9f3b0dd3a074a6c3e1e79f35e465b1f2ee4b262d619de00cfce523cc9b24/python_discovery-1.1.3.tar.gz", hash = "sha256:7acca36e818cd88e9b2ba03e045ad7e93e1713e29c6bbfba5d90202310b7baa5", size = 56945, upload-time = "2026-03-10T15:08:15.038Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/80/73211fc5bfbfc562369b4aa61dc1e4bf07dc7b34df7b317e4539316b809c/python_discovery-1.1.3-py3-none-any.whl", hash = "sha256:90e795f0121bc84572e737c9aa9966311b9fde44ffb88a5953b3ec9b31c6945e", size = 31485, upload-time = "2026-03-10T15:08:13.06Z" }, +] + [[package]] name = "python-dotenv" version = "1.2.1" @@ -8750,6 +8817,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5d/c1/f01846f10c8383b197a67b4cdfea6eea476aab678ed9f6f5d9ccc8c8dddf/viberbot-1.0.12-py3-none-any.whl", hash = "sha256:ca43fea2945d650c2ef2cbd777f3c546c795bf945278f6620ceda42f4101d801", size = 26164, upload-time = "2021-01-27T13:45:43.726Z" }, ] +[[package]] +name = "virtualenv" +version = "21.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "distlib" }, + { name = "filelock" }, + { name = "platformdirs" }, + { name = "python-discovery" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/aa/92/58199fe10049f9703c2666e809c4f686c54ef0a68b0f6afccf518c0b1eb9/virtualenv-21.2.0.tar.gz", hash = "sha256:1720dc3a62ef5b443092e3f499228599045d7fea4c79199770499df8becf9098", size = 5840618, upload-time = "2026-03-09T17:24:38.013Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c6/59/7d02447a55b2e55755011a647479041bc92a82e143f96a8195cb33bd0a1c/virtualenv-21.2.0-py3-none-any.whl", hash = "sha256:1bd755b504931164a5a496d217c014d098426cddc79363ad66ac78125f9d908f", size = 5825084, upload-time = "2026-03-09T17:24:35.378Z" }, +] + [[package]] name = "vllm" version = "0.17.1" From 856d27621d6616b00aca468e0f5d437f8a10633a Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon Date: Tue, 17 Mar 2026 03:38:13 +0000 Subject: [PATCH 21/92] fix: rename Development tab to Roadmap with Roadmap + Contributing Guide sub-pages Co-Authored-By: Claude Opus 4.6 (1M context) --- mkdocs.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 73869262..40c2dcb3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -164,4 +164,6 @@ nav: - Design Principles: architecture/design-principles.md - API Reference: api-reference/ - Leaderboard: leaderboard.md - - Roadmap: development/roadmap.md + - Roadmap: + - Roadmap: development/roadmap.md + - Contributing Guide: development/contributing.md From f52ed33e4346013a866512585cd546a77c5c94e6 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon Date: Tue, 17 Mar 2026 03:58:57 +0000 Subject: [PATCH 22/92] fix: rewrite roadmap around 5 workstreams, fix broken links and cleanup - Reorganize roadmap around the 5 workstreams: Continuous Operators, Mobile & Messaging, Secure Cloud Collaboration, Tutorials, Hardware - Add concrete "Where you can help" tables with maturity tags and good-first-issue markers under each workstream - Change "GRPO training from trace data" to "Post-training data" - Fix 404 link: CONTRIBUTING.md link now points to GitHub, not docs site - Remove !!! tip admonition from docs/development/contributing.md - Preserve version history in collapsible section Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/development/contributing.md | 5 -- docs/development/roadmap.md | 134 ++++++++++++++++++++++++++----- 2 files changed, 112 insertions(+), 27 deletions(-) diff --git a/docs/development/contributing.md b/docs/development/contributing.md index 357581b8..5ec38bdd 100644 --- a/docs/development/contributing.md +++ b/docs/development/contributing.md @@ -1,10 +1,5 @@ # Contributing Guide -!!! tip "Looking for how to contribute?" - See our [Contributing Guide](../../CONTRIBUTING.md) for an overview of - contribution types, incentives, and the PR process. This page covers - detailed development setup and code conventions. - This guide covers how to set up a development environment, run tests, and contribute code to OpenJarvis. diff --git a/docs/development/roadmap.md b/docs/development/roadmap.md index 87774e61..5e91a65c 100644 --- a/docs/development/roadmap.md +++ b/docs/development/roadmap.md @@ -1,32 +1,122 @@ # Roadmap -OpenJarvis uses **GitHub Projects** boards organized by domain to plan and -track work. Each board has quarterly columns so contributors can see what's -coming and where help is needed. +OpenJarvis development is organized into **five independent workstreams**. Contributors can pick any track that matches their skills and interests — workstreams are designed to be worked on in parallel without blocking each other. + +Within each workstream, items are organized by time horizon: + +- **Near-term** — foundations and hardening of what exists +- **Mid-term** — significant new capabilities +- **Long-term** — frontier work requiring design exploration or research + +Every item carries a maturity tag: + +| Tag | Meaning | Contributor guidance | +|-----|---------|---------------------| +| **Ready** | Well-scoped, implementation path is clear | Pick it up — check issues for a spec or write one | +| **Design Needed** | Concept is clear but needs a spec before code | Start a design discussion or draft an RFC | +| **Research-Stage** | Exploratory, needs investigation before designing | Read the relevant papers, prototype, share findings | + +Items marked **good first issue** are especially suited for new contributors. --- -## How We Plan +## Workstream 1: Continuous Operators & Agents -- Work is organized into domain-specific project boards (see below) -- Each board uses quarterly columns: **Backlog → Q2 2026 → Q3 2026 → Q4 2026 → Future** -- Issues are labeled by difficulty (`good-first-issue`, `help-wanted`) and type (`type:bug`, `type:feature`, `type:perf`, `type:docs`, `type:eval`) -- Domain labels (`domain:agents`, `domain:engine`, `domain:tools`, etc.) connect issues to the right board +Operators are OpenJarvis's key differentiator — persistent, scheduled, stateful agents that run autonomously on personal devices. The current tick-based architecture (OperatorManager → TaskScheduler → AgentExecutor → OperativeAgent) is solid but needs hardening for truly long-horizon autonomy. -Want to contribute? Check the boards below, pick an issue labeled `good-first-issue` or `help-wanted`, and comment **"take"** to claim it. +### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| Operator health checks & heartbeat monitoring | **Ready** | Add liveness probes to OperatorManager; surface in `jarvis operators status`. Detect stalled operators beyond the existing reconciliation loop. | +| Metrics collection for operator manifests | **Ready** | The `metrics` field exists in `OperatorManifest` but is not collected. Wire it to telemetry. **Good first issue.** | +| Capability policy enforcement | **Ready** | `required_capabilities` field exists in manifests but is not enforced. Connect to the existing RBAC `CapabilityPolicy` system. **Good first issue.** | +| Rate limiting per operator | **Ready** | Prevent runaway operators from hammering inference. Add configurable rate limits to OperatorManager. | +| Operator composition / chaining | **Design Needed** | Express dependencies between operators (operator A feeds results to operator B). Requires design for data passing and scheduling semantics. | +| Event-driven operators | **Design Needed** | Operators that trigger on EventBus events (e.g., new file indexed, channel message received) rather than only cron/interval schedules. | +| Operator versioning & rollback | **Design Needed** | Run v2 of an operator alongside v1. Roll back automatically on repeated failures. | +| Self-improving operators via Learning | **Research-Stage** | Operators that use trace feedback to tune their own prompts, tool selection, and routing policies through the Learning primitive. | --- -## Active Project Boards +## Workstream 2: Mobile & Messaging Clients -| Board | Scope | Link | -|---|---|---| -| **Agents & Tools** | Agent types, tool implementations, MCP | *TBD — boards will be linked once created on GitHub* | -| **Engine & Inference** | Engine backends, streaming, performance | *TBD* | -| **Learning & Routing** | GRPO, trace-driven policies, rewards | *TBD* | -| **Evals & Benchmarks** | Datasets, scorers, benchmark infra | *TBD* | -| **Frontend & Desktop** | Tauri app, dashboard, leaderboard | *TBD* | -| **Rust Port** | PyO3 bindings, crate parity | *TBD* | +Personal AI must be accessible from the devices people actually carry. OpenJarvis runs on laptops, workstations, and servers — users interact via their phones. Channels bridge that gap. Today, WhatsApp (Baileys), Slack, and Telegram are bidirectional; iMessage is send-only; Android SMS does not exist. + +### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| iMessage bidirectional via BlueBubbles | **Ready** | Current implementation is send-only. Add webhook/polling listener for incoming messages using the BlueBubbles API. **Good first issue.** | +| WhatsApp Baileys media support | **Ready** | Currently text-only. Add image, audio, and file handling to the Node.js bridge and Python channel. **Good first issue.** | +| Slack rich messages | **Ready** | Current implementation is plain text. Add Slack Block Kit support for formatted responses, buttons, and attachments. | +| Android SMS via Twilio/Vonage | **Design Needed** | No SMS implementation exists. Requires provider selection, two-way webhook architecture, and phone number provisioning flow. | +| Unified notification system | **Design Needed** | Push notifications when operators complete tasks or need user attention. Requires per-channel notification adapters. | +| Signal bidirectional | **Design Needed** | Currently send-only via signal-cli REST API. Add incoming message listener with background polling. | +| Voice interface | **Research-Stage** | Speech-to-text (Whisper) → agent → text-to-speech loop over phone channels. Existing `speech/` module provides a foundation. | + +--- + +## Workstream 3: Secure Cloud Collaboration + +Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through **Minions-style collaborative inference** (local handles context, cloud handles reasoning) and **TEE-based confidential computing** (cloud cannot see your data even during inference). + +**References:** + +- [Minions: Cost-Efficient Local-Cloud LLM Collaboration](https://github.com/HazyResearch/minions) +- [TEE for Confidential AI Inference](https://openreview.net/forum?id=ey87M5iKcX) ([PDF](https://openreview.net/pdf?id=ey87M5iKcX)) + +### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| Query complexity analyzer | **Ready** | Classify incoming queries by difficulty to decide local vs. cloud routing. Extends the existing `MultiEngine` routing logic. | +| Cost tracking per-query | **Ready** | `CloudEngine` already has pricing data. Surface per-query cost in traces and telemetry dashboards. **Good first issue.** | +| Redaction-before-cloud pipeline | **Ready** | Wire the existing `GuardrailsEngine` in REDACT mode as a mandatory pre-step before any cloud transmission. | +| Minion protocol (sequential) | **Design Needed** | Local model extracts and summarizes long context → cloud model reasons over the compressed result. Native reimplementation of the core [Minions](https://github.com/HazyResearch/minions) idea. | +| Minion protocol (parallel) | **Design Needed** | Local and cloud models work simultaneously on different aspects of a query; results are merged. Requires a new `HybridInferenceEngine` abstraction. | +| TEE attestation verification | **Design Needed** | Verify that cloud inference ran inside a trusted execution environment via cryptographic attestation. | +| Taint tracking across local/cloud boundary | **Design Needed** | The `TaintSet` already tracks PII/Secret labels. Add routing enforcement so tainted data only routes to attested TEE endpoints. | +| Speculative decoding (local draft + cloud verify) | **Research-Stage** | Local model generates candidate tokens; cloud model validates in parallel for latency reduction. | + +--- + +## Workstream 4: Tutorials & Documentation + +OpenJarvis has reference docs and four tutorials, but critical gaps remain in continuous agents, LM evaluation, learning approaches, and custom tools. Video tutorials are scoped as a contributor opportunity — written tutorials come first, with video scripts included so anyone can record. + +### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| "Building Continuous Agents" tutorial | **Ready** | Writing an operator TOML manifest, activating it, session persistence across ticks, daemon mode. Example: a research operator that monitors arxiv daily. | +| "Adding Custom Tools" tutorial | **Ready** | Implementing `BaseTool`, registering via `ToolRegistry`, wiring into agents. Example: a weather API tool. **Good first issue.** | +| "Testing & Comparing LMs" tutorial | **Ready** | Running benchmarks, comparing local vs. cloud models, interpreting telemetry (latency, cost, energy per token). Uses the existing `bench/` framework. | +| Per-platform installation guides | **Ready** | Expand `installation.md` with platform-specific walkthroughs: macOS + Ollama, Ubuntu + NVIDIA + vLLM, Windows + Ollama, Raspberry Pi. **Good first issue.** | +| "Learning & Model Selection" tutorial | **Design Needed** | Router policies (heuristic, learned, GRPO), proposed approaches like Thompson Sampling, trace-based reward signals. | +| Video tutorial infrastructure | **Design Needed** | Establish recording workflow, hosting (YouTube), MkDocs embedding. Write video scripts alongside written tutorials. | +| Interactive Jupyter notebook tutorials | **Design Needed** | Notebook versions of key tutorials for exploratory, cell-by-cell learning. | + +--- + +## Workstream 5: Hardware Breadth + +Personal AI means running on the hardware people actually own. Each new hardware target expands who can use OpenJarvis and generates data for the research agenda (energy, cost, latency tradeoffs across silicon). + +Adding a new hardware target involves up to four components: hardware detection in `core/config.py`, an inference engine adapter in `engine/`, an energy monitor in `telemetry/`, and an entry in the GPU specs database in `telemetry/gpu_monitor.py`. + +### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| AMD Ryzen AI iGPU path | **Ready** | Strix Point RDNA 3.5 iGPU handles 7-8B via Vulkan. llama.cpp Vulkan backend works today. Needs hardware detection and energy monitor. **Good first issue.** | +| GPU specs database expansion | **Ready** | Add Intel Arc, Jetson Orin, Snapdragon specs to `GPU_SPECS` in `telemetry/gpu_monitor.py` (TFLOPS, bandwidth, TDP). **Good first issue.** | +| Intel Arc GPU (B580/B570) | **Design Needed** | 12GB VRAM, ~$250 consumer GPU. Viable for 7-8B models. Engine path: IPEX-LLM or llama.cpp SYCL backend. | +| NVIDIA Jetson Orin | **Design Needed** | Best-in-class edge device. Orin NX 16GB handles 7-8B models at 15-25 tok/s. Needs hardware detection, energy monitor (tegrastats), deployment guide. | +| Qualcomm Snapdragon X Elite NPU | **Design Needed** | 45 TOPS, Windows Arm laptops. ONNX Runtime + QNN Execution Provider is the viable path. | +| Intel Lunar Lake NPU via OpenVINO | **Design Needed** | 48 TOPS — most mature NPU software stack for x86 laptops. New engine wrapping OpenVINO GenAI. | +| Raspberry Pi 5 | **Design Needed** | CPU-only via llama.cpp ARM NEON for 1-3B models. $100 entry point for hobbyists. | +| Unified hardware benchmark suite | **Design Needed** | Standardized benchmark that runs the same workloads across all supported hardware, producing comparable energy/latency/throughput/cost numbers. | --- @@ -34,7 +124,7 @@ Want to contribute? Check the boards below, pick an issue labeled `good-first-is These are the areas where active development is happening and contributions are most impactful: -- **GRPO training from trace data** — moving router policies beyond heuristics using reinforcement learning from execution traces +- **Post-training data** — building datasets and training pipelines from execution traces to improve agent routing and tool selection - **Multi-model orchestration pipelines** — coordinating multiple models within a single query (e.g., small model for classification, large model for generation) - **Energy-aware routing** — using power consumption data from telemetry to optimize for energy efficiency alongside latency and quality - **Plugin ecosystem** — community-contributed engines, tools, and agents distributed as Python packages @@ -44,9 +134,9 @@ These are the areas where active development is happening and contributions are ## How to Get Involved -1. Browse the [project boards](#active-project-boards) for issues that interest you -2. Look for `good-first-issue` and `help-wanted` labels -3. Read the [Contributing Guide](../../CONTRIBUTING.md) for the full process +1. Browse the workstreams above for items that interest you +2. Look for **Ready** items and **good first issue** tags +3. Read the [Contributing Guide](https://github.com/open-jarvis/OpenJarvis/blob/main/CONTRIBUTING.md) for the full process 4. Comment **"take"** on an issue to claim it --- From b8bbbdbf55cc1f2cbc6f3766f921bb748bfd434b Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon Date: Tue, 17 Mar 2026 04:17:21 +0000 Subject: [PATCH 23/92] fix: move focus areas and get-involved to top of roadmap, remove contributing from nav - Move "Current Focus Areas" and "How to Get Involved" to top of roadmap - Remove Version History section entirely - Fix "take" workflow: clarify that claiming happens on GitHub issues, with link to create new issues if none exists - Remove Contributing Guide from MkDocs nav (linked from roadmap instead) - Link to CONTRIBUTING.md on GitHub from "How to Get Involved" section Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/development/roadmap.md | 248 ++++++++++++++++-------------------- mkdocs.yml | 4 +- 2 files changed, 113 insertions(+), 139 deletions(-) diff --git a/docs/development/roadmap.md b/docs/development/roadmap.md index 5e91a65c..cb5cbb23 100644 --- a/docs/development/roadmap.md +++ b/docs/development/roadmap.md @@ -1,125 +1,5 @@ # Roadmap -OpenJarvis development is organized into **five independent workstreams**. Contributors can pick any track that matches their skills and interests — workstreams are designed to be worked on in parallel without blocking each other. - -Within each workstream, items are organized by time horizon: - -- **Near-term** — foundations and hardening of what exists -- **Mid-term** — significant new capabilities -- **Long-term** — frontier work requiring design exploration or research - -Every item carries a maturity tag: - -| Tag | Meaning | Contributor guidance | -|-----|---------|---------------------| -| **Ready** | Well-scoped, implementation path is clear | Pick it up — check issues for a spec or write one | -| **Design Needed** | Concept is clear but needs a spec before code | Start a design discussion or draft an RFC | -| **Research-Stage** | Exploratory, needs investigation before designing | Read the relevant papers, prototype, share findings | - -Items marked **good first issue** are especially suited for new contributors. - ---- - -## Workstream 1: Continuous Operators & Agents - -Operators are OpenJarvis's key differentiator — persistent, scheduled, stateful agents that run autonomously on personal devices. The current tick-based architecture (OperatorManager → TaskScheduler → AgentExecutor → OperativeAgent) is solid but needs hardening for truly long-horizon autonomy. - -### Where you can help - -| Item | Maturity | Details | -|------|----------|---------| -| Operator health checks & heartbeat monitoring | **Ready** | Add liveness probes to OperatorManager; surface in `jarvis operators status`. Detect stalled operators beyond the existing reconciliation loop. | -| Metrics collection for operator manifests | **Ready** | The `metrics` field exists in `OperatorManifest` but is not collected. Wire it to telemetry. **Good first issue.** | -| Capability policy enforcement | **Ready** | `required_capabilities` field exists in manifests but is not enforced. Connect to the existing RBAC `CapabilityPolicy` system. **Good first issue.** | -| Rate limiting per operator | **Ready** | Prevent runaway operators from hammering inference. Add configurable rate limits to OperatorManager. | -| Operator composition / chaining | **Design Needed** | Express dependencies between operators (operator A feeds results to operator B). Requires design for data passing and scheduling semantics. | -| Event-driven operators | **Design Needed** | Operators that trigger on EventBus events (e.g., new file indexed, channel message received) rather than only cron/interval schedules. | -| Operator versioning & rollback | **Design Needed** | Run v2 of an operator alongside v1. Roll back automatically on repeated failures. | -| Self-improving operators via Learning | **Research-Stage** | Operators that use trace feedback to tune their own prompts, tool selection, and routing policies through the Learning primitive. | - ---- - -## Workstream 2: Mobile & Messaging Clients - -Personal AI must be accessible from the devices people actually carry. OpenJarvis runs on laptops, workstations, and servers — users interact via their phones. Channels bridge that gap. Today, WhatsApp (Baileys), Slack, and Telegram are bidirectional; iMessage is send-only; Android SMS does not exist. - -### Where you can help - -| Item | Maturity | Details | -|------|----------|---------| -| iMessage bidirectional via BlueBubbles | **Ready** | Current implementation is send-only. Add webhook/polling listener for incoming messages using the BlueBubbles API. **Good first issue.** | -| WhatsApp Baileys media support | **Ready** | Currently text-only. Add image, audio, and file handling to the Node.js bridge and Python channel. **Good first issue.** | -| Slack rich messages | **Ready** | Current implementation is plain text. Add Slack Block Kit support for formatted responses, buttons, and attachments. | -| Android SMS via Twilio/Vonage | **Design Needed** | No SMS implementation exists. Requires provider selection, two-way webhook architecture, and phone number provisioning flow. | -| Unified notification system | **Design Needed** | Push notifications when operators complete tasks or need user attention. Requires per-channel notification adapters. | -| Signal bidirectional | **Design Needed** | Currently send-only via signal-cli REST API. Add incoming message listener with background polling. | -| Voice interface | **Research-Stage** | Speech-to-text (Whisper) → agent → text-to-speech loop over phone channels. Existing `speech/` module provides a foundation. | - ---- - -## Workstream 3: Secure Cloud Collaboration - -Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through **Minions-style collaborative inference** (local handles context, cloud handles reasoning) and **TEE-based confidential computing** (cloud cannot see your data even during inference). - -**References:** - -- [Minions: Cost-Efficient Local-Cloud LLM Collaboration](https://github.com/HazyResearch/minions) -- [TEE for Confidential AI Inference](https://openreview.net/forum?id=ey87M5iKcX) ([PDF](https://openreview.net/pdf?id=ey87M5iKcX)) - -### Where you can help - -| Item | Maturity | Details | -|------|----------|---------| -| Query complexity analyzer | **Ready** | Classify incoming queries by difficulty to decide local vs. cloud routing. Extends the existing `MultiEngine` routing logic. | -| Cost tracking per-query | **Ready** | `CloudEngine` already has pricing data. Surface per-query cost in traces and telemetry dashboards. **Good first issue.** | -| Redaction-before-cloud pipeline | **Ready** | Wire the existing `GuardrailsEngine` in REDACT mode as a mandatory pre-step before any cloud transmission. | -| Minion protocol (sequential) | **Design Needed** | Local model extracts and summarizes long context → cloud model reasons over the compressed result. Native reimplementation of the core [Minions](https://github.com/HazyResearch/minions) idea. | -| Minion protocol (parallel) | **Design Needed** | Local and cloud models work simultaneously on different aspects of a query; results are merged. Requires a new `HybridInferenceEngine` abstraction. | -| TEE attestation verification | **Design Needed** | Verify that cloud inference ran inside a trusted execution environment via cryptographic attestation. | -| Taint tracking across local/cloud boundary | **Design Needed** | The `TaintSet` already tracks PII/Secret labels. Add routing enforcement so tainted data only routes to attested TEE endpoints. | -| Speculative decoding (local draft + cloud verify) | **Research-Stage** | Local model generates candidate tokens; cloud model validates in parallel for latency reduction. | - ---- - -## Workstream 4: Tutorials & Documentation - -OpenJarvis has reference docs and four tutorials, but critical gaps remain in continuous agents, LM evaluation, learning approaches, and custom tools. Video tutorials are scoped as a contributor opportunity — written tutorials come first, with video scripts included so anyone can record. - -### Where you can help - -| Item | Maturity | Details | -|------|----------|---------| -| "Building Continuous Agents" tutorial | **Ready** | Writing an operator TOML manifest, activating it, session persistence across ticks, daemon mode. Example: a research operator that monitors arxiv daily. | -| "Adding Custom Tools" tutorial | **Ready** | Implementing `BaseTool`, registering via `ToolRegistry`, wiring into agents. Example: a weather API tool. **Good first issue.** | -| "Testing & Comparing LMs" tutorial | **Ready** | Running benchmarks, comparing local vs. cloud models, interpreting telemetry (latency, cost, energy per token). Uses the existing `bench/` framework. | -| Per-platform installation guides | **Ready** | Expand `installation.md` with platform-specific walkthroughs: macOS + Ollama, Ubuntu + NVIDIA + vLLM, Windows + Ollama, Raspberry Pi. **Good first issue.** | -| "Learning & Model Selection" tutorial | **Design Needed** | Router policies (heuristic, learned, GRPO), proposed approaches like Thompson Sampling, trace-based reward signals. | -| Video tutorial infrastructure | **Design Needed** | Establish recording workflow, hosting (YouTube), MkDocs embedding. Write video scripts alongside written tutorials. | -| Interactive Jupyter notebook tutorials | **Design Needed** | Notebook versions of key tutorials for exploratory, cell-by-cell learning. | - ---- - -## Workstream 5: Hardware Breadth - -Personal AI means running on the hardware people actually own. Each new hardware target expands who can use OpenJarvis and generates data for the research agenda (energy, cost, latency tradeoffs across silicon). - -Adding a new hardware target involves up to four components: hardware detection in `core/config.py`, an inference engine adapter in `engine/`, an energy monitor in `telemetry/`, and an entry in the GPU specs database in `telemetry/gpu_monitor.py`. - -### Where you can help - -| Item | Maturity | Details | -|------|----------|---------| -| AMD Ryzen AI iGPU path | **Ready** | Strix Point RDNA 3.5 iGPU handles 7-8B via Vulkan. llama.cpp Vulkan backend works today. Needs hardware detection and energy monitor. **Good first issue.** | -| GPU specs database expansion | **Ready** | Add Intel Arc, Jetson Orin, Snapdragon specs to `GPU_SPECS` in `telemetry/gpu_monitor.py` (TFLOPS, bandwidth, TDP). **Good first issue.** | -| Intel Arc GPU (B580/B570) | **Design Needed** | 12GB VRAM, ~$250 consumer GPU. Viable for 7-8B models. Engine path: IPEX-LLM or llama.cpp SYCL backend. | -| NVIDIA Jetson Orin | **Design Needed** | Best-in-class edge device. Orin NX 16GB handles 7-8B models at 15-25 tok/s. Needs hardware detection, energy monitor (tegrastats), deployment guide. | -| Qualcomm Snapdragon X Elite NPU | **Design Needed** | 45 TOPS, Windows Arm laptops. ONNX Runtime + QNN Execution Provider is the viable path. | -| Intel Lunar Lake NPU via OpenVINO | **Design Needed** | 48 TOPS — most mature NPU software stack for x86 laptops. New engine wrapping OpenVINO GenAI. | -| Raspberry Pi 5 | **Design Needed** | CPU-only via llama.cpp ARM NEON for 1-3B models. $100 entry point for hobbyists. | -| Unified hardware benchmark suite | **Design Needed** | Standardized benchmark that runs the same workloads across all supported hardware, producing comparable energy/latency/throughput/cost numbers. | - ---- - ## Current Focus Areas These are the areas where active development is happening and contributions are most impactful: @@ -134,25 +14,121 @@ These are the areas where active development is happening and contributions are ## How to Get Involved -1. Browse the workstreams above for items that interest you +1. Browse the workstreams below for items that interest you 2. Look for **Ready** items and **good first issue** tags -3. Read the [Contributing Guide](https://github.com/open-jarvis/OpenJarvis/blob/main/CONTRIBUTING.md) for the full process -4. Comment **"take"** on an issue to claim it +3. Find the matching [GitHub issue](https://github.com/open-jarvis/OpenJarvis/issues) and comment **"take"** to claim it — if no issue exists yet, [open one](https://github.com/open-jarvis/OpenJarvis/issues/new/choose) describing what you'd like to work on +4. Read the [Contributing Guide](https://github.com/open-jarvis/OpenJarvis/blob/main/CONTRIBUTING.md) for the full development setup, PR process, and code conventions --- -
-Version History +## Workstreams -| Version | Phase | Status | Delivers | -|---|---|---|---| -| **v0.1** | Phase 0 -- Scaffolding | :material-check-circle:{ .green } Complete | Project scaffolding, registry system (`RegistryBase[T]`), core types (`Message`, `ModelSpec`, `Conversation`, `ToolResult`), configuration loader with hardware detection, Click CLI skeleton | -| **v0.2** | Phase 1 -- Intelligence + Inference | :material-check-circle:{ .green } Complete | Intelligence primitive (model catalog, heuristic router), inference engines (Ollama, vLLM, llama.cpp), engine discovery and health probing, `jarvis ask` command working end-to-end | -| **v0.3** | Phase 2 -- Memory | :material-check-circle:{ .green } Complete | Memory backends (SQLite/FTS5, FAISS, ColBERTv2, BM25, Hybrid/RRF), document chunking and ingestion pipeline, context injection with source attribution, `jarvis memory` commands | -| **v0.4** | Phase 3 -- Agents + Tools + Server | :material-check-circle:{ .green } Complete | Agent system (SimpleAgent, OrchestratorAgent), tool system (Calculator, Think, Retrieval, LLM, FileRead), ToolExecutor dispatch engine, OpenAI-compatible API server (`jarvis serve`) | -| **v0.5** | Phase 4 -- Learning + Telemetry | :material-check-circle:{ .green } Complete | Learning system (HeuristicRouter policy, TraceDrivenPolicy, GRPO stub), reward functions, telemetry aggregation (per-model/engine stats, export), `--router` CLI flag, `jarvis telemetry` commands | -| **v1.0** | Phase 5 -- SDK + Production | :material-check-circle:{ .green } Complete | Python SDK (`Jarvis` class, `MemoryHandle`), multi-platform channel system (Telegram, Discord, Slack, WhatsApp, etc.), benchmarking framework (latency, throughput), Docker deployment (CPU + GPU), MkDocs documentation site | -| **v1.1** | Phase 6 -- Traces + Learning | :material-check-circle:{ .green } Complete | Trace system (`TraceStore`, `TraceCollector`, `TraceAnalyzer`), trace-driven learning, MCP integration layer | -| **v1.5** | Phase 10 -- Agent Restructuring | :material-check-circle:{ .green } Complete | BaseAgent helpers, ToolUsingAgent intermediate base, NativeReActAgent, NativeOpenHandsAgent, RLMAgent, OpenHandsAgent (SDK), `accepts_tools` introspection, backward-compat shims, CustomAgent removed | +OpenJarvis development is organized into **five independent workstreams**. Contributors can pick any track that matches their skills and interests — workstreams are designed to be worked on in parallel without blocking each other. -
+Every item carries a maturity tag: + +| Tag | Meaning | Contributor guidance | +|-----|---------|---------------------| +| **Ready** | Well-scoped, implementation path is clear | Pick it up — check [issues](https://github.com/open-jarvis/OpenJarvis/issues) for a spec or write one | +| **Design Needed** | Concept is clear but needs a spec before code | Start a [design discussion](https://github.com/open-jarvis/OpenJarvis/discussions) or draft an RFC | +| **Research-Stage** | Exploratory, needs investigation before designing | Read the relevant papers, prototype, share findings | + +--- + +### Workstream 1: Continuous Operators & Agents + +Operators are OpenJarvis's key differentiator — persistent, scheduled, stateful agents that run autonomously on personal devices. The current tick-based architecture (OperatorManager → TaskScheduler → AgentExecutor → OperativeAgent) is solid but needs hardening for truly long-horizon autonomy. + +#### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| Operator health checks & heartbeat monitoring | **Ready** | Add liveness probes to OperatorManager; surface in `jarvis operators status`. Detect stalled operators beyond the existing reconciliation loop. | +| Metrics collection for operator manifests | **Ready** | The `metrics` field exists in `OperatorManifest` but is not collected. Wire it to telemetry. **Good first issue.** | +| Capability policy enforcement | **Ready** | `required_capabilities` field exists in manifests but is not enforced. Connect to the existing RBAC `CapabilityPolicy` system. **Good first issue.** | +| Rate limiting per operator | **Ready** | Prevent runaway operators from hammering inference. Add configurable rate limits to OperatorManager. | +| Operator composition / chaining | **Design Needed** | Express dependencies between operators (operator A feeds results to operator B). Requires design for data passing and scheduling semantics. | +| Event-driven operators | **Design Needed** | Operators that trigger on EventBus events (e.g., new file indexed, channel message received) rather than only cron/interval schedules. | +| Operator versioning & rollback | **Design Needed** | Run v2 of an operator alongside v1. Roll back automatically on repeated failures. | +| Self-improving operators via Learning | **Research-Stage** | Operators that use trace feedback to tune their own prompts, tool selection, and routing policies through the Learning primitive. | + +--- + +### Workstream 2: Mobile & Messaging Clients + +Personal AI must be accessible from the devices people actually carry. OpenJarvis runs on laptops, workstations, and servers — users interact via their phones. Channels bridge that gap. Today, WhatsApp (Baileys), Slack, and Telegram are bidirectional; iMessage is send-only; Android SMS does not exist. + +#### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| iMessage bidirectional via BlueBubbles | **Ready** | Current implementation is send-only. Add webhook/polling listener for incoming messages using the BlueBubbles API. **Good first issue.** | +| WhatsApp Baileys media support | **Ready** | Currently text-only. Add image, audio, and file handling to the Node.js bridge and Python channel. **Good first issue.** | +| Slack rich messages | **Ready** | Current implementation is plain text. Add Slack Block Kit support for formatted responses, buttons, and attachments. | +| Android SMS via Twilio/Vonage | **Design Needed** | No SMS implementation exists. Requires provider selection, two-way webhook architecture, and phone number provisioning flow. | +| Unified notification system | **Design Needed** | Push notifications when operators complete tasks or need user attention. Requires per-channel notification adapters. | +| Signal bidirectional | **Design Needed** | Currently send-only via signal-cli REST API. Add incoming message listener with background polling. | +| Voice interface | **Research-Stage** | Speech-to-text (Whisper) → agent → text-to-speech loop over phone channels. Existing `speech/` module provides a foundation. | + +--- + +### Workstream 3: Secure Cloud Collaboration + +Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through **Minions-style collaborative inference** (local handles context, cloud handles reasoning) and **TEE-based confidential computing** (cloud cannot see your data even during inference). + +**References:** + +- [Minions: Cost-Efficient Local-Cloud LLM Collaboration](https://github.com/HazyResearch/minions) +- [TEE for Confidential AI Inference](https://openreview.net/forum?id=ey87M5iKcX) ([PDF](https://openreview.net/pdf?id=ey87M5iKcX)) + +#### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| Query complexity analyzer | **Ready** | Classify incoming queries by difficulty to decide local vs. cloud routing. Extends the existing `MultiEngine` routing logic. | +| Cost tracking per-query | **Ready** | `CloudEngine` already has pricing data. Surface per-query cost in traces and telemetry dashboards. **Good first issue.** | +| Redaction-before-cloud pipeline | **Ready** | Wire the existing `GuardrailsEngine` in REDACT mode as a mandatory pre-step before any cloud transmission. | +| Minion protocol (sequential) | **Design Needed** | Local model extracts and summarizes long context → cloud model reasons over the compressed result. Native reimplementation of the core [Minions](https://github.com/HazyResearch/minions) idea. | +| Minion protocol (parallel) | **Design Needed** | Local and cloud models work simultaneously on different aspects of a query; results are merged. Requires a new `HybridInferenceEngine` abstraction. | +| TEE attestation verification | **Design Needed** | Verify that cloud inference ran inside a trusted execution environment via cryptographic attestation. | +| Taint tracking across local/cloud boundary | **Design Needed** | The `TaintSet` already tracks PII/Secret labels. Add routing enforcement so tainted data only routes to attested TEE endpoints. | +| Speculative decoding (local draft + cloud verify) | **Research-Stage** | Local model generates candidate tokens; cloud model validates in parallel for latency reduction. | + +--- + +### Workstream 4: Tutorials & Documentation + +OpenJarvis has reference docs and four tutorials, but critical gaps remain in continuous agents, LM evaluation, learning approaches, and custom tools. Video tutorials are scoped as a contributor opportunity — written tutorials come first, with video scripts included so anyone can record. + +#### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| "Building Continuous Agents" tutorial | **Ready** | Writing an operator TOML manifest, activating it, session persistence across ticks, daemon mode. Example: a research operator that monitors arxiv daily. | +| "Adding Custom Tools" tutorial | **Ready** | Implementing `BaseTool`, registering via `ToolRegistry`, wiring into agents. Example: a weather API tool. **Good first issue.** | +| "Testing & Comparing LMs" tutorial | **Ready** | Running benchmarks, comparing local vs. cloud models, interpreting telemetry (latency, cost, energy per token). Uses the existing `bench/` framework. | +| Per-platform installation guides | **Ready** | Expand `installation.md` with platform-specific walkthroughs: macOS + Ollama, Ubuntu + NVIDIA + vLLM, Windows + Ollama, Raspberry Pi. **Good first issue.** | +| "Learning & Model Selection" tutorial | **Design Needed** | Router policies (heuristic, learned, GRPO), proposed approaches like Thompson Sampling, trace-based reward signals. | +| Video tutorial infrastructure | **Design Needed** | Establish recording workflow, hosting (YouTube), MkDocs embedding. Write video scripts alongside written tutorials. | +| Interactive Jupyter notebook tutorials | **Design Needed** | Notebook versions of key tutorials for exploratory, cell-by-cell learning. | + +--- + +### Workstream 5: Hardware Breadth + +Personal AI means running on the hardware people actually own. Each new hardware target expands who can use OpenJarvis and generates data for the research agenda (energy, cost, latency tradeoffs across silicon). + +Adding a new hardware target involves up to four components: hardware detection in `core/config.py`, an inference engine adapter in `engine/`, an energy monitor in `telemetry/`, and an entry in the GPU specs database in `telemetry/gpu_monitor.py`. + +#### Where you can help + +| Item | Maturity | Details | +|------|----------|---------| +| AMD Ryzen AI iGPU path | **Ready** | Strix Point RDNA 3.5 iGPU handles 7-8B via Vulkan. llama.cpp Vulkan backend works today. Needs hardware detection and energy monitor. **Good first issue.** | +| GPU specs database expansion | **Ready** | Add Intel Arc, Jetson Orin, Snapdragon specs to `GPU_SPECS` in `telemetry/gpu_monitor.py` (TFLOPS, bandwidth, TDP). **Good first issue.** | +| Intel Arc GPU (B580/B570) | **Design Needed** | 12GB VRAM, ~$250 consumer GPU. Viable for 7-8B models. Engine path: IPEX-LLM or llama.cpp SYCL backend. | +| NVIDIA Jetson Orin | **Design Needed** | Best-in-class edge device. Orin NX 16GB handles 7-8B models at 15-25 tok/s. Needs hardware detection, energy monitor (tegrastats), deployment guide. | +| Qualcomm Snapdragon X Elite NPU | **Design Needed** | 45 TOPS, Windows Arm laptops. ONNX Runtime + QNN Execution Provider is the viable path. | +| Intel Lunar Lake NPU via OpenVINO | **Design Needed** | 48 TOPS — most mature NPU software stack for x86 laptops. New engine wrapping OpenVINO GenAI. | +| Raspberry Pi 5 | **Design Needed** | CPU-only via llama.cpp ARM NEON for 1-3B models. $100 entry point for hobbyists. | +| Unified hardware benchmark suite | **Design Needed** | Standardized benchmark that runs the same workloads across all supported hardware, producing comparable energy/latency/throughput/cost numbers. | diff --git a/mkdocs.yml b/mkdocs.yml index 40c2dcb3..73869262 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -164,6 +164,4 @@ nav: - Design Principles: architecture/design-principles.md - API Reference: api-reference/ - Leaderboard: leaderboard.md - - Roadmap: - - Roadmap: development/roadmap.md - - Contributing Guide: development/contributing.md + - Roadmap: development/roadmap.md From 9c6bdc09cb4b743aba255c3881be48f8aa2da939 Mon Sep 17 00:00:00 2001 From: Gabriel Bo Date: Mon, 16 Mar 2026 21:51:05 -0700 Subject: [PATCH 24/92] token counting fixes --- docs/javascripts/leaderboard.js | 14 +++++++ rust/crates/openjarvis-telemetry/src/flops.rs | 31 ++++++++++++++ src/openjarvis/agents/_stubs.py | 7 +++- src/openjarvis/agents/monitor_operative.py | 10 +++++ src/openjarvis/agents/native_react.py | 12 +++++- src/openjarvis/agents/operative.py | 14 ++++++- src/openjarvis/agents/rlm.py | 10 ++++- src/openjarvis/engine/nexa_shim.py | 4 ++ src/openjarvis/server/savings.py | 29 +++++++++++-- src/openjarvis/server/stream_bridge.py | 24 ++++++++++- src/openjarvis/telemetry/__init__.py | 2 + src/openjarvis/telemetry/flops.py | 41 +++++++++++++++---- tests/telemetry/test_flops_new.py | 19 +++++++++ 13 files changed, 198 insertions(+), 19 deletions(-) diff --git a/docs/javascripts/leaderboard.js b/docs/javascripts/leaderboard.js index 7d2ffc41..60d64c90 100644 --- a/docs/javascripts/leaderboard.js +++ b/docs/javascripts/leaderboard.js @@ -9,6 +9,20 @@ var allRows = []; var currentPage = 0; + // No-KV-cache formula: FLOPs = params_b * 1e9 * N * (N+1) + var DEFAULT_PARAMS_B = 137; + var ENERGY_WH_PER_FLOP = 0.4 / (1000 * 3e12); + + function recomputeFlopsNoKvCache(totalTokens) { + var n = Number(totalTokens) || 0; + if (n <= 0) return 0; + return DEFAULT_PARAMS_B * 1e9 * n * (n + 1); + } + + function recomputeEnergyNoKvCache(flops) { + return flops * ENERGY_WH_PER_FLOP; + } + function escapeHtml(s) { var el = document.createElement("span"); el.textContent = s; diff --git a/rust/crates/openjarvis-telemetry/src/flops.rs b/rust/crates/openjarvis-telemetry/src/flops.rs index b82bea8f..8f71b585 100644 --- a/rust/crates/openjarvis-telemetry/src/flops.rs +++ b/rust/crates/openjarvis-telemetry/src/flops.rs @@ -50,6 +50,7 @@ fn lookup(table: &[(&str, f64)], key: &str) -> Option { } /// Estimate FLOPs for a model inference using the `2 * params * tokens` approximation. +/// Assumes KV caching (each token processed once). /// /// Returns `(total_flops, flops_per_token)`. If the model is not found in `MODEL_PARAMS`, /// returns `(0.0, 0.0)`. @@ -72,6 +73,36 @@ pub fn estimate_flops(model: &str, input_tokens: u64, output_tokens: u64) -> (f6 (total_flops, flops_per_token) } +/// Estimate FLOPs without KV caching (full recompute per token). +/// +/// Without KV cache, each token is re-processed for every subsequent token. +/// FLOPs = P * N * (N + 1) where P = params, N = total_tokens. +/// +/// Returns `(total_flops, flops_per_token_avg)`. If the model is not found, +/// returns `(0.0, 0.0)`. +pub fn estimate_flops_no_kv_cache( + model: &str, + input_tokens: u64, + output_tokens: u64, +) -> (f64, f64) { + let params_b = match lookup(MODEL_PARAMS, model) { + Some(p) => p, + None => return (0.0, 0.0), + }; + + let total_tokens = input_tokens + output_tokens; + if total_tokens == 0 { + return (0.0, 0.0); + } + + let params = params_b * 1e9; + let n = total_tokens as f64; + let total_flops = params * n * (n + 1.0); + let flops_per_token = total_flops / n; + + (total_flops, flops_per_token) +} + /// Compute Model FLOPs Utilization (MFU). /// /// MFU = actual_flops / (peak_tflops * 1e12 * duration_s * num_gpus) diff --git a/src/openjarvis/agents/_stubs.py b/src/openjarvis/agents/_stubs.py index f25fcc9c..9a91c90b 100644 --- a/src/openjarvis/agents/_stubs.py +++ b/src/openjarvis/agents/_stubs.py @@ -137,14 +137,19 @@ class BaseAgent(ABC): tool_results: list[ToolResult], turns: int, content: str = "", + *, + metadata: Optional[Dict[str, Any]] = None, ) -> AgentResult: """Build the standard result for when ``max_turns`` is exceeded.""" self._emit_turn_end(turns=turns, max_turns_exceeded=True) + md: Dict[str, Any] = {"max_turns_exceeded": True} + if metadata: + md.update(metadata) return AgentResult( content=content or "Maximum turns reached without a final answer.", tool_results=tool_results, turns=turns, - metadata={"max_turns_exceeded": True}, + metadata=md, ) def _check_continuation( diff --git a/src/openjarvis/agents/monitor_operative.py b/src/openjarvis/agents/monitor_operative.py index 1f03bae3..a764ca35 100644 --- a/src/openjarvis/agents/monitor_operative.py +++ b/src/openjarvis/agents/monitor_operative.py @@ -202,6 +202,11 @@ class MonitorOperativeAgent(ToolUsingAgent): turns = 0 content = "" state_stored_by_tool = False + total_usage: dict[str, int] = { + "prompt_tokens": 0, + "completion_tokens": 0, + "total_tokens": 0, + } for _turn in range(self._max_turns): turns += 1 @@ -211,6 +216,9 @@ class MonitorOperativeAgent(ToolUsingAgent): gen_kwargs["tools"] = openai_tools result = self._generate(messages, **gen_kwargs) + usage = result.get("usage", {}) + for k in total_usage: + total_usage[k] += usage.get(k, 0) content = result.get("content", "") raw_tool_calls = result.get("tool_calls", []) @@ -291,6 +299,7 @@ class MonitorOperativeAgent(ToolUsingAgent): self._save_session(input, content) return self._max_turns_result( all_tool_results, turns, content=content, + metadata=total_usage, ) # 6. Save session @@ -305,6 +314,7 @@ class MonitorOperativeAgent(ToolUsingAgent): content=content, tool_results=all_tool_results, turns=turns, + metadata=total_usage, ) # ------------------------------------------------------------------ diff --git a/src/openjarvis/agents/native_react.py b/src/openjarvis/agents/native_react.py index 2b14d1c0..bec88754 100644 --- a/src/openjarvis/agents/native_react.py +++ b/src/openjarvis/agents/native_react.py @@ -109,6 +109,7 @@ class NativeReActAgent(ToolUsingAgent): all_tool_results: list[ToolResult] = [] turns = 0 + total_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} for _turn in range(self._max_turns): turns += 1 @@ -117,6 +118,9 @@ class NativeReActAgent(ToolUsingAgent): messages = self._loop_guard.compress_context(messages) result = self._generate(messages) + usage = result.get("usage", {}) + for k in total_usage: + total_usage[k] += usage.get(k, 0) content = result.get("content", "") parsed = self._parse_response(content) @@ -128,13 +132,17 @@ class NativeReActAgent(ToolUsingAgent): content=parsed["final_answer"], tool_results=all_tool_results, turns=turns, + metadata=total_usage, ) # No action? Treat content as final answer if not parsed["action"]: self._emit_turn_end(turns=turns) return AgentResult( - content=content, tool_results=all_tool_results, turns=turns + content=content, + tool_results=all_tool_results, + turns=turns, + metadata=total_usage, ) # Execute action @@ -169,7 +177,7 @@ class NativeReActAgent(ToolUsingAgent): messages.append(Message(role=Role.USER, content=observation)) # Max turns exceeded - return self._max_turns_result(all_tool_results, turns) + return self._max_turns_result(all_tool_results, turns, metadata=total_usage) __all__ = ["NativeReActAgent", "REACT_SYSTEM_PROMPT"] diff --git a/src/openjarvis/agents/operative.py b/src/openjarvis/agents/operative.py index ed56b541..e6a05270 100644 --- a/src/openjarvis/agents/operative.py +++ b/src/openjarvis/agents/operative.py @@ -104,6 +104,7 @@ class OperativeAgent(ToolUsingAgent): turns = 0 content = "" state_stored_by_tool = False + total_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} for _turn in range(self._max_turns): turns += 1 @@ -116,6 +117,9 @@ class OperativeAgent(ToolUsingAgent): gen_kwargs["tools"] = openai_tools result = self._generate(messages, **gen_kwargs) + usage = result.get("usage", {}) + for k in total_usage: + total_usage[k] += usage.get(k, 0) content = result.get("content", "") raw_tool_calls = result.get("tool_calls", []) @@ -179,7 +183,14 @@ class OperativeAgent(ToolUsingAgent): else: # Max turns exceeded self._save_session(input, content) - return self._max_turns_result(all_tool_results, turns, content=content) + meta = dict(total_usage) + meta["max_turns_exceeded"] = True + return AgentResult( + content=content or "Maximum turns reached without a final answer.", + tool_results=all_tool_results, + turns=turns, + metadata=meta, + ) # 6. Save session self._save_session(input, content) @@ -193,6 +204,7 @@ class OperativeAgent(ToolUsingAgent): content=content, tool_results=all_tool_results, turns=turns, + metadata=total_usage, ) def _build_operative_messages( diff --git a/src/openjarvis/agents/rlm.py b/src/openjarvis/agents/rlm.py index 26d65bce..9efc99a0 100644 --- a/src/openjarvis/agents/rlm.py +++ b/src/openjarvis/agents/rlm.py @@ -173,11 +173,15 @@ class RLMAgent(ToolUsingAgent): all_tool_results: list[ToolResult] = [] turns = 0 + total_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} for _turn in range(self._max_turns): turns += 1 result = self._generate(messages) + usage = result.get("usage", {}) + for k in total_usage: + total_usage[k] += usage.get(k, 0) content = result.get("content", "") # Strip tags @@ -193,6 +197,7 @@ class RLMAgent(ToolUsingAgent): content=content, tool_results=all_tool_results, turns=turns, + metadata=total_usage, ) # Execute code in REPL @@ -218,6 +223,7 @@ class RLMAgent(ToolUsingAgent): content=final_str, tool_results=all_tool_results, turns=turns, + metadata=total_usage, ) # Feed output back as user message @@ -236,7 +242,9 @@ class RLMAgent(ToolUsingAgent): else: final_content = "" - return self._max_turns_result(all_tool_results, turns, content=final_content) + return self._max_turns_result( + all_tool_results, turns, content=final_content, metadata=total_usage + ) # ------------------------------------------------------------------ # Sub-LM callbacks diff --git a/src/openjarvis/engine/nexa_shim.py b/src/openjarvis/engine/nexa_shim.py index c2c67169..9c8bc160 100644 --- a/src/openjarvis/engine/nexa_shim.py +++ b/src/openjarvis/engine/nexa_shim.py @@ -4,6 +4,10 @@ Thin FastAPI server wrapping the Nexa SDK (``nexaai``) as an OpenAI-compatible API on port 18181. Intended for on-device inference with GGUF models on Apple Silicon or CPU. +**Token counts:** The Nexa SDK does not expose token counts in responses. +The shim returns 0 for prompt/completion/total tokens. Savings and +leaderboard metrics will not include sessions that use this engine. + Usage: uvicorn openjarvis.engine.nexa_shim:app \ --host 127.0.0.1 --port 18181 diff --git a/src/openjarvis/server/savings.py b/src/openjarvis/server/savings.py index 020c23e5..3f6144a2 100644 --- a/src/openjarvis/server/savings.py +++ b/src/openjarvis/server/savings.py @@ -1,4 +1,8 @@ -"""Savings calculation — compare local inference cost against cloud providers.""" +"""Savings calculation — compare local inference cost against cloud providers. + +FLOPs and energy use a no-KV-cache model: P * N * (N+1) where P = params, +N = total tokens. This reflects full recompute without cached attention. +""" from __future__ import annotations @@ -8,6 +12,7 @@ from typing import Any, Dict, List # --------------------------------------------------------------------------- # Cloud provider pricing (USD per 1M tokens) +# params_b: model size in billions, for no-KV-cache FLOPs # --------------------------------------------------------------------------- CLOUD_PRICING: Dict[str, Dict[str, float]] = { @@ -16,6 +21,7 @@ CLOUD_PRICING: Dict[str, Dict[str, float]] = { "output_per_1m": 10.00, "label": "GPT-5.3", "provider": "OpenAI", + "params_b": 200.0, "energy_wh_per_1k_tokens": 0.4, "flops_per_token": 3.0e12, }, @@ -24,6 +30,7 @@ CLOUD_PRICING: Dict[str, Dict[str, float]] = { "output_per_1m": 25.00, "label": "Claude Opus 4.6", "provider": "Anthropic", + "params_b": 137.0, "energy_wh_per_1k_tokens": 0.5, "flops_per_token": 4.0e12, }, @@ -32,6 +39,7 @@ CLOUD_PRICING: Dict[str, Dict[str, float]] = { "output_per_1m": 12.00, "label": "Gemini 3.1 Pro", "provider": "Google", + "params_b": 137.0, "energy_wh_per_1k_tokens": 0.35, "flops_per_token": 2.5e12, }, @@ -89,8 +97,23 @@ def compute_savings( input_cost = (prompt_tokens / 1_000_000) * pricing["input_per_1m"] output_cost = (completion_tokens / 1_000_000) * pricing["output_per_1m"] total_cost = input_cost + output_cost - energy_wh = (total_tokens / 1000) * pricing["energy_wh_per_1k_tokens"] - flops = total_tokens * pricing["flops_per_token"] + + # No-KV-cache FLOPs: P * N * (N+1) + params_b = pricing.get("params_b", 200.0) + params = params_b * 1e9 + flops = ( + params * total_tokens * (total_tokens + 1) + if total_tokens > 0 else 0.0 + ) + # Scale energy by same ratio as FLOPs (energy ∝ compute) + flops_with_cache = ( + total_tokens * pricing.get("flops_per_token", 3e12) + if total_tokens > 0 else 0.0 + ) + scale = (flops / flops_with_cache) if flops_with_cache > 0 else 1.0 + energy_wh = ( + (total_tokens / 1000) * pricing["energy_wh_per_1k_tokens"] * scale + ) providers.append(ProviderSavings( provider=key, diff --git a/src/openjarvis/server/stream_bridge.py b/src/openjarvis/server/stream_bridge.py index 28958429..9a96577b 100644 --- a/src/openjarvis/server/stream_bridge.py +++ b/src/openjarvis/server/stream_bridge.py @@ -37,6 +37,25 @@ _EVENT_MAP = { _DONE = object() +def _estimate_prompt_tokens(messages: list) -> int: + """Estimate prompt tokens from request messages (incl. system, user, context). + + Uses ~4 chars per token heuristic. Ensures system and all context are counted. + """ + total = 0 + for m in messages: + text = getattr(m, "content", None) or "" + if isinstance(text, str): + total += max(1, len(text) // 4) + # Tool call messages may have structured content; still count what we can + if hasattr(m, "tool_calls") and m.tool_calls: + for tc in m.tool_calls: + fn = tc.get("function") if isinstance(tc, dict) else {} + args = fn.get("arguments", "") if isinstance(fn, dict) else "" + total += max(1, len(str(args)) // 4) + return total + + class AgentStreamBridge: """Bridge between a synchronous agent and an async SSE stream. @@ -238,9 +257,10 @@ class AgentStreamBridge: "completion_tokens", 0, ) total_tokens = agent_result.metadata.get("total_tokens", 0) - if total_tokens == 0 and content: + if total_tokens == 0: + # Fallback: estimate from request messages (incl. system) + content completion_tokens = max(len(content) // 4, 1) - prompt_tokens = completion_tokens # rough estimate + prompt_tokens = _estimate_prompt_tokens(self._request.messages) total_tokens = prompt_tokens + completion_tokens final_chunk = ChatCompletionChunk( diff --git a/src/openjarvis/telemetry/__init__.py b/src/openjarvis/telemetry/__init__.py index f2c68646..b287eb36 100644 --- a/src/openjarvis/telemetry/__init__.py +++ b/src/openjarvis/telemetry/__init__.py @@ -69,6 +69,7 @@ try: MODEL_PARAMS_B, compute_mfu, estimate_flops, + estimate_flops_no_kv_cache, ) except ImportError: pass @@ -100,6 +101,7 @@ __all__ = [ "split_at_ttft", "compute_itl_stats", "estimate_flops", + "estimate_flops_no_kv_cache", "compute_mfu", "GPU_PEAK_TFLOPS_BF16", "MODEL_PARAMS_B", diff --git a/src/openjarvis/telemetry/flops.py b/src/openjarvis/telemetry/flops.py index 2fd4ce39..a9527dcb 100644 --- a/src/openjarvis/telemetry/flops.py +++ b/src/openjarvis/telemetry/flops.py @@ -31,22 +31,26 @@ MODEL_PARAMS_B: dict[str, float] = { } -def estimate_flops( - model: str, input_tokens: int, output_tokens: int -) -> tuple[float, float]: - """Estimate FLOPs for an inference pass. - - Uses the 2 * P * T approximation where P = params, T = total tokens. - Returns (total_flops, flops_per_token). - """ +def _get_params_b(model: str) -> float: + """Look up model parameter count (billions).""" params_b = MODEL_PARAMS_B.get(model, 0.0) if params_b == 0.0: - # Try prefix matching for key, val in MODEL_PARAMS_B.items(): if model.startswith(key.split(":")[0]): params_b = val break + return params_b + +def estimate_flops( + model: str, input_tokens: int, output_tokens: int +) -> tuple[float, float]: + """Estimate FLOPs for an inference pass (assumes KV caching). + + Uses the 2 * P * T approximation where P = params, T = total tokens. + Returns (total_flops, flops_per_token). + """ + params_b = _get_params_b(model) total_tokens = input_tokens + output_tokens params = params_b * 1e9 total_flops = 2.0 * params * total_tokens @@ -54,6 +58,25 @@ def estimate_flops( return (total_flops, flops_per_token) +def estimate_flops_no_kv_cache( + model: str, input_tokens: int, output_tokens: int +) -> tuple[float, float]: + """Estimate FLOPs without KV caching (full recompute per token). + + Without KV cache, each token is re-processed for every subsequent token. + FLOPs = P * N * (N + 1) where P = params, N = total_tokens. + Returns (total_flops, flops_per_token_avg). + """ + params_b = _get_params_b(model) + total_tokens = input_tokens + output_tokens + if total_tokens == 0: + return (0.0, 0.0) + params = params_b * 1e9 + total_flops = params * total_tokens * (total_tokens + 1) + flops_per_token = total_flops / total_tokens + return (total_flops, flops_per_token) + + def compute_mfu( flops: float, duration_s: float, gpu_name: str, num_gpus: int = 1 ) -> float: diff --git a/tests/telemetry/test_flops_new.py b/tests/telemetry/test_flops_new.py index e67f9997..7b8cbde7 100644 --- a/tests/telemetry/test_flops_new.py +++ b/tests/telemetry/test_flops_new.py @@ -9,6 +9,7 @@ from openjarvis.telemetry.flops import ( MODEL_PARAMS_B, compute_mfu, estimate_flops, + estimate_flops_no_kv_cache, ) @@ -43,6 +44,24 @@ class TestEstimateFlops: assert total_200 == pytest.approx(total_100 * 2.0) +class TestEstimateFlopsNoKvCache: + def test_known_model(self): + total, per_tok = estimate_flops_no_kv_cache("qwen3:8b", 100, 50) + # P * N * (N+1) = 8e9 * 150 * 151 = 1.812e14 + assert total == pytest.approx(8e9 * 150 * 151) + assert per_tok == pytest.approx(total / 150) + + def test_zero_tokens(self): + total, per_tok = estimate_flops_no_kv_cache("qwen3:8b", 0, 0) + assert total == 0.0 + assert per_tok == 0.0 + + def test_unknown_model_zero(self): + total, per_tok = estimate_flops_no_kv_cache("unknown-model", 100, 50) + assert total == 0.0 + assert per_tok == 0.0 + + class TestComputeMfu: def test_known_gpu(self): # 100 TFLOPS actual for 1s on H100 → 100 / 989 * 100 ≈ 10.1% From 3ca1b63962c950ee4ac7c407c21c34a85d833b5c Mon Sep 17 00:00:00 2001 From: Gabriel Bo Date: Mon, 16 Mar 2026 21:52:17 -0700 Subject: [PATCH 25/92] cleaning up scripts --- scripts/GRID_SEARCH_GUIDE.md | 350 ----------------------------------- 1 file changed, 350 deletions(-) delete mode 100644 scripts/GRID_SEARCH_GUIDE.md diff --git a/scripts/GRID_SEARCH_GUIDE.md b/scripts/GRID_SEARCH_GUIDE.md deleted file mode 100644 index 98e8d5dd..00000000 --- a/scripts/GRID_SEARCH_GUIDE.md +++ /dev/null @@ -1,350 +0,0 @@ -# Eval Grid Search — 4× A100 Runbook - -Complete guide for running the OpenJarvis evaluation grid search across **5 models × 4 engines × 5 agents × 15 benchmarks** on a 4× NVIDIA A100 (80 GB) node. - -## Grid Dimensions - -| Dimension | Values | -|-----------|--------| -| **Models** | GPT-OSS-120B, Qwen3.5-122B-FP8, Qwen3.5-397B-GGUF, Kimi-K2.5-GGUF, GLM-5-GGUF | -| **Engines** | vLLM, SGLang, llama.cpp, Ollama | -| **Agents** | simple, orchestrator, native_react, native_openhands, rlm | -| **Benchmarks** | supergpqa, gpqa, mmlu-pro, math500, natural-reasoning, hle, simpleqa, wildchat, ipw, gaia, frames, swebench, swefficiency, terminalbench, terminalbench-native | -| **Samples** | 5 per benchmark | - -**Total: 750 experiment cells** (each model only runs on its compatible engines). - -| Model | Compatible Engines | -|-------|--------------------| -| openai/gpt-oss-120b | vllm, sglang | -| Qwen/Qwen3.5-122B-A10B-FP8 | vllm, sglang | -| unsloth/Qwen3.5-397B-A17B-GGUF | llamacpp, ollama | -| unsloth/Kimi-K2.5-GGUF | llamacpp, ollama | -| unsloth/GLM-5-GGUF | llamacpp, ollama | - ---- - -## Phase 1: Environment Setup - -```bash -# Clone and install -cd ~/gabebo # or wherever your workspace lives -git clone OpenJarvis && cd OpenJarvis -uv sync --extra dev - -# Install eval dependencies -uv pip install openai datasets huggingface-hub terminal-bench - -# Load API keys (needed for LLM judge — gpt-5-mini) -source .env - -# Log in to HuggingFace (needed for gated datasets) -huggingface-cli login -``` - -> **SGLang note:** SGLang requires Python ≤ 3.12 (FlashInfer/outlines_core fail on 3.13). -> If your base env is Python 3.13, create a separate conda env: -> ```bash -> conda create -n sglang python=3.11 -y && conda activate sglang -> pip install "sglang[all]" -> ``` - ---- - -## Phase 2: Download Models - -```bash -# HuggingFace weights (vLLM / SGLang) -hf download openai/gpt-oss-120b --local-dir ~/models/gpt-oss-120b -hf download Qwen/Qwen3.5-122B-A10B-FP8 --local-dir ~/models/Qwen3.5-122B-A10B-FP8 - -# GGUF quantizations (llama.cpp / Ollama) -# Qwen3.5-397B — UD-Q4_K_XL fits in 4× A100 (~214 GB) -hf download unsloth/Qwen3.5-397B-A17B-GGUF \ - --include "Q4_K_M/*.gguf" \ - --local-dir ~/models/Qwen3.5-397B-A17B-GGUF - -# Kimi-K2.5 — UD-IQ2_XXS to fit (~240 GB usable) -hf download unsloth/Kimi-K2.5-GGUF \ - --include "UD-IQ2_XXS/*.gguf" \ - --local-dir ~/models/Kimi-K2.5-GGUF - -# GLM-5 — UD-IQ2_XXS to fit -hf download unsloth/GLM-5-GGUF \ - --include "UD-IQ2_XXS/*.gguf" \ - --local-dir ~/models/GLM-5-GGUF -``` - -### Verify downloads - -```bash -# HuggingFace weights — check for config.json and safetensors shards -ls ~/models/gpt-oss-120b/config.json -ls ~/models/Qwen3.5-122B-A10B-FP8/config.json - -# GGUF files — check they exist and aren't zero-byte -find ~/models/Qwen3.5-397B-A17B-GGUF -name "*.gguf" -exec ls -lh {} \; -find ~/models/Kimi-K2.5-GGUF -name "*.gguf" -exec ls -lh {} \; -find ~/models/GLM-5-GGUF -name "*.gguf" -exec ls -lh {} \; -``` - ---- - -## Phase 3: Run the Grid (One Model at a Time) - -Serve a model, run all agents/benchmarks for it, then kill the server and swap. - -### 3A. GPT-OSS-120B via vLLM - -```bash -# Terminal 1: Start vLLM server -vllm serve openai/gpt-oss-120b \ - --tensor-parallel-size 2 \ - --port 8000 \ - --max-model-len 8192 -``` - -```bash -# Terminal 2: Wait for "Uvicorn running" then run grid -cd OpenJarvis && source .env -uv run python scripts/run_grid_search.py \ - --model "gpt-oss" --engine vllm -n 5 -# When done, Ctrl-C the vLLM server -``` - -### 3B. GPT-OSS-120B via SGLang - -```bash -# Terminal 1: Start SGLang server -python -m sglang.launch_server \ - --model-path openai/gpt-oss-120b \ - --tp 2 \ - --port 30000 -``` - -```bash -# Terminal 2: Run grid -uv run python scripts/run_grid_search.py \ - --model "gpt-oss" --engine sglang --resume -n 5 -# Kill server when done -``` - -### 3C. Qwen3.5-122B-FP8 via vLLM - -```bash -# Terminal 1 -vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \ - --tensor-parallel-size 2 \ - --port 8000 \ - --max-model-len 8192 \ - --quantization fp8 -``` - -```bash -# Terminal 2 -uv run python scripts/run_grid_search.py \ - --model "Qwen3.5-122B" --engine vllm --resume -n 5 -``` - -### 3D. Qwen3.5-122B-FP8 via SGLang - -```bash -# Terminal 1 -python -m sglang.launch_server \ - --model-path Qwen/Qwen3.5-122B-A10B-FP8 \ - --tp 2 \ - --port 30000 \ - --quantization fp8 -``` - -```bash -# Terminal 2 -uv run python scripts/run_grid_search.py \ - --model "Qwen3.5-122B" --engine sglang --resume -n 5 -``` - -### 3E. Qwen3.5-397B GGUF via llama.cpp - -```bash -# Terminal 1: Start llama.cpp server with all 4 GPUs -./llama.cpp/build/bin/llama-server \ - -m ~/models/Qwen3.5-397B-A17B-GGUF/Q4_K_M/Qwen3.5-397B-A17B-Q4_K_M-00001-of-00005.gguf \ - --n-gpu-layers 99 \ - --tensor-split 1,1,1,1 \ - --port 8080 \ - --ctx-size 8192 -``` - -```bash -# Terminal 2 -uv run python scripts/run_grid_search.py \ - --model "Qwen3.5-397B" --engine llamacpp --resume -n 5 -``` - -### 3F. Qwen3.5-397B GGUF via Ollama - -```bash -# Create an Ollama modelfile -cat > /tmp/Qwen3.5-397B.Modelfile << 'EOF' -FROM ~/models/Qwen3.5-397B-A17B-GGUF/Q4_K_M/Qwen3.5-397B-A17B-Q4_K_M-00001-of-00005.gguf -EOF - -# Import into Ollama -ollama create qwen3.5-397b -f /tmp/Qwen3.5-397B.Modelfile - -# Ollama serves automatically on port 11434 -uv run python scripts/run_grid_search.py \ - --model "Qwen3.5-397B" --engine ollama --resume -n 5 - -# Unload when done -ollama stop qwen3.5-397b -``` - -### 3G. Kimi-K2.5 GGUF via llama.cpp - -```bash -# Terminal 1 -./llama.cpp/build/bin/llama-server \ - -m ~/models/Kimi-K2.5-GGUF/UD-IQ2_XXS/Kimi-K2.5-UD-IQ2_XXS-00001-of-00005.gguf \ - --n-gpu-layers 99 \ - --tensor-split 1,1,1,1 \ - --port 8080 \ - --ctx-size 8192 -``` - -```bash -# Terminal 2 -uv run python scripts/run_grid_search.py \ - --model "Kimi-K2.5" --engine llamacpp --resume -n 5 -``` - -### 3H. Kimi-K2.5 GGUF via Ollama - -```bash -cat > /tmp/Kimi-K2.5.Modelfile << 'EOF' -FROM ~/models/Kimi-K2.5-GGUF/UD-IQ2_XXS/Kimi-K2.5-UD-IQ2_XXS-00001-of-00005.gguf -EOF - -ollama create kimi-k2.5 -f /tmp/Kimi-K2.5.Modelfile - -uv run python scripts/run_grid_search.py \ - --model "Kimi-K2.5" --engine ollama --resume -n 5 - -ollama stop kimi-k2.5 -``` - -### 3I. GLM-5 GGUF via llama.cpp - -```bash -# Terminal 1 -./llama.cpp/build/bin/llama-server \ - -m ~/models/GLM-5-GGUF/UD-IQ2_XXS/GLM-5-UD-IQ2_XXS-00001-of-00005.gguf \ - --n-gpu-layers 99 \ - --tensor-split 1,1,1,1 \ - --port 8080 \ - --ctx-size 8192 -``` - -```bash -# Terminal 2 -uv run python scripts/run_grid_search.py \ - --model "GLM-5" --engine llamacpp --resume -n 5 -``` - -### 3J. GLM-5 GGUF via Ollama - -```bash -cat > /tmp/GLM-5.Modelfile << 'EOF' -FROM ~/models/GLM-5-GGUF/UD-IQ2_XXS/GLM-5-UD-IQ2_XXS-00001-of-00005.gguf -EOF - -ollama create glm-5 -f /tmp/GLM-5.Modelfile - -uv run python scripts/run_grid_search.py \ - --model "GLM-5" --engine ollama --resume -n 5 - -ollama stop glm-5 -``` - ---- - -## Phase 4: Recover Failed Runs - -If some runs fail (missing packages, engine not reachable, etc.), fix the issue, then: - -```bash -# Delete error summaries so --resume retries them -find results/grid-search -name "*.summary.json" \ - -exec grep -l '"error"' {} \; -delete - -# Re-run with --resume (only retries deleted/missing summaries) -uv run python scripts/run_grid_search.py --resume -n 5 -``` - -### Common Failures and Fixes - -| Error | Cause | Fix | -|-------|-------|-----| -| `No inference engine available` | `openai` package missing (LLM judge can't init) | `uv pip install openai` | -| `No module named 'datasets'` | Missing HF datasets package | `uv pip install datasets` | -| `terminal-bench package required` | Missing terminal-bench | `uv pip install terminal-bench` | -| `Dataset doesn't exist on the Hub` | Gated dataset or HF auth needed | `huggingface-cli login`, accept terms on HF website | -| `IPW data directory not found` | IPW uses local data, not HuggingFace | Place files in `src/openjarvis/evals/data/ipw/` | -| `natural-reasoning` 0 samples | Field name mismatch in dataset loader | Patch `src/openjarvis/evals/datasets/natural_reasoning.py` | - ---- - -## Phase 5: Analyze Results - -```bash -# Preview what ran -uv run python scripts/run_grid_search.py --dry-run --resume - -# Consolidated results (appended after each run) -cat results/grid-search/grid-results.jsonl - -# Per-run summaries -find results/grid-search -name "*.summary.json" | head -20 -cat results/grid-search/openai-gpt-oss-120b/vllm/simple/supergpqa.summary.json - -# Count completed vs failed -echo "Completed:" && find results/grid-search -name "*.summary.json" \ - -exec grep -L '"error"' {} \; | wc -l -echo "Failed:" && find results/grid-search -name "*.summary.json" \ - -exec grep -l '"error"' {} \; | wc -l -``` - ---- - -## Useful Flags - -```bash -# Preview the full matrix without running -uv run python scripts/run_grid_search.py --dry-run - -# Filter to a single model + engine -uv run python scripts/run_grid_search.py --model "gpt-oss" --engine vllm -n 5 - -# Filter to a single agent or benchmark -uv run python scripts/run_grid_search.py --agent native_react --benchmark supergpqa -n 5 - -# Increase sample count -uv run python scripts/run_grid_search.py -n 50 - -# Verbose logging -uv run python scripts/run_grid_search.py -v --resume -n 5 -``` - ---- - -## GPU Assignment Tips - -Use `CUDA_VISIBLE_DEVICES` to pin servers to specific GPUs: - -```bash -# Run vLLM on GPUs 0,1 and llama.cpp on GPUs 2,3 simultaneously -CUDA_VISIBLE_DEVICES=0,1 vllm serve openai/gpt-oss-120b --tensor-parallel-size 2 --port 8000 -CUDA_VISIBLE_DEVICES=2,3 ./llama.cpp/build/bin/llama-server -m model.gguf --n-gpu-layers 99 --port 8080 -``` - -This lets you run two model servers in parallel on different GPU pairs. From 256ff54476c3d01ac512c55bac779e39999a8d33 Mon Sep 17 00:00:00 2001 From: Gabriel Bo Date: Mon, 16 Mar 2026 22:07:26 -0700 Subject: [PATCH 26/92] long single line dict literal error fixes --- src/openjarvis/agents/native_react.py | 6 +++++- src/openjarvis/agents/operative.py | 6 +++++- src/openjarvis/agents/rlm.py | 6 +++++- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/src/openjarvis/agents/native_react.py b/src/openjarvis/agents/native_react.py index bec88754..cfcc5023 100644 --- a/src/openjarvis/agents/native_react.py +++ b/src/openjarvis/agents/native_react.py @@ -109,7 +109,11 @@ class NativeReActAgent(ToolUsingAgent): all_tool_results: list[ToolResult] = [] turns = 0 - total_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} + total_usage: dict[str, int] = { + "prompt_tokens": 0, + "completion_tokens": 0, + "total_tokens": 0, + } for _turn in range(self._max_turns): turns += 1 diff --git a/src/openjarvis/agents/operative.py b/src/openjarvis/agents/operative.py index e6a05270..7f88f840 100644 --- a/src/openjarvis/agents/operative.py +++ b/src/openjarvis/agents/operative.py @@ -104,7 +104,11 @@ class OperativeAgent(ToolUsingAgent): turns = 0 content = "" state_stored_by_tool = False - total_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} + total_usage: dict[str, int] = { + "prompt_tokens": 0, + "completion_tokens": 0, + "total_tokens": 0, + } for _turn in range(self._max_turns): turns += 1 diff --git a/src/openjarvis/agents/rlm.py b/src/openjarvis/agents/rlm.py index 9efc99a0..0cc6e3a4 100644 --- a/src/openjarvis/agents/rlm.py +++ b/src/openjarvis/agents/rlm.py @@ -173,7 +173,11 @@ class RLMAgent(ToolUsingAgent): all_tool_results: list[ToolResult] = [] turns = 0 - total_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} + total_usage: dict[str, int] = { + "prompt_tokens": 0, + "completion_tokens": 0, + "total_tokens": 0, + } for _turn in range(self._max_turns): turns += 1 From b10b19b1eb37ed98d350a91a0df0c5660a2cb2d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Teemu=20S=C3=A4ilynoja?= Date: Mon, 16 Mar 2026 15:44:47 +0200 Subject: [PATCH 27/92] feat: add DuckDuckGo fallback to web_search tool When Tavily API is unavailable (no API key, import error, or API error), the web_search tool now falls back to DuckDuckGo search instead of failing. This ensures the tool always works for users without a Tavily API key. - Add DuckDuckGo search as fallback using ddgs package - Catch specific Tavily exceptions (MissingAPIKeyError, InvalidAPIKeyError, ForbiddenError, UsageLimitExceededError, TimeoutError, BadRequestError) - Add logger.debug calls to log when falling back to DuckDuckGo - Use ddgs instead of deprecated duckduckgo-search package name - Add test for DuckDuckGo fallback result formatting - Simplify test mocking to use consistent monkeypatch patterns Closes #81 --- pyproject.toml | 2 + src/openjarvis/tools/web_search.py | 75 +++++++-- tests/tools/test_web_search.py | 82 +++++++--- uv.lock | 253 ++++++++++++++++++++--------- 4 files changed, 300 insertions(+), 112 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 1ec5947b..696885aa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,6 +11,7 @@ requires-python = ">=3.10" dependencies = [ "click>=8", "datasets>=4.5.0", + "ddgs>=9.11.4", "httpx>=0.27", "openai>=1.30", "rich>=13", @@ -39,6 +40,7 @@ inference-google = [ inference-litellm = ["litellm>=1.40"] tools-search = [ "tavily-python>=0.3", + "ddgs>=9.11.4", ] memory-faiss = [ "faiss-cpu>=1.7", diff --git a/src/openjarvis/tools/web_search.py b/src/openjarvis/tools/web_search.py index 227d76aa..60900f66 100644 --- a/src/openjarvis/tools/web_search.py +++ b/src/openjarvis/tools/web_search.py @@ -2,6 +2,7 @@ from __future__ import annotations +import logging import os from typing import Any @@ -10,6 +11,8 @@ from openjarvis.core.types import ToolResult from openjarvis.security.ssrf import check_ssrf from openjarvis.tools._stubs import BaseTool, ToolSpec +logger = logging.getLogger(__name__) + @ToolRegistry.register("web_search") class WebSearchTool(BaseTool): @@ -41,7 +44,7 @@ class WebSearchTool(BaseTool): "required": ["query"], }, category="search", - metadata={"requires_api_key": "TAVILY_API_KEY"}, + metadata={"requires_api_key": "TAVILY_API_KEY", "fallback": "duckduckgo"}, ) @staticmethod @@ -84,7 +87,9 @@ class WebSearchTool(BaseTool): url.strip(), follow_redirects=True, timeout=30.0, - headers={"User-Agent": "Mozilla/5.0 (compatible; OpenJarvis/1.0; +https://github.com/openjarvis)"}, + headers={ + "User-Agent": "Mozilla/5.0 (compatible; OpenJarvis/1.0; +https://github.com/openjarvis)" + }, ) resp.raise_for_status() content_type = resp.headers.get("content-type", "") @@ -96,7 +101,9 @@ class WebSearchTool(BaseTool): html = resp.text # Strip script/style tags and their contents html = _re.sub( - r"<(script|style)[^>]*>.*?", "", html, + r"<(script|style)[^>]*>.*?", + "", + html, flags=_re.DOTALL | _re.IGNORECASE, ) # Strip HTML tags @@ -107,6 +114,19 @@ class WebSearchTool(BaseTool): text = text[:max_chars] + "\n\n[Content truncated]" return text + def _duckduckgo_search(self, query: str, max_results: int) -> str: + """Search using DuckDuckGo as fallback.""" + from ddgs import DDGS + + ddgs = DDGS() + results = list(ddgs.text(query, max_results=max_results)) + formatted = "\n\n".join( + f"**{r.get('title', 'Untitled')}**\n" + f"{r.get('href', '')}\n{r.get('body', '')}" + for r in results + ) + return formatted + def execute(self, **params: Any) -> ToolResult: query = params.get("query", "") if not query: @@ -134,15 +154,18 @@ class WebSearchTool(BaseTool): success=False, ) - if not self._api_key: - return ToolResult( - tool_name="web_search", - content="No API key configured. Set TAVILY_API_KEY.", - success=False, - ) max_results = params.get("max_results", self._max_results) + try: from tavily import TavilyClient + from tavily.errors import ( + BadRequestError, + ForbiddenError, + InvalidAPIKeyError, + MissingAPIKeyError, + TimeoutError, + UsageLimitExceededError, + ) client = TavilyClient(api_key=self._api_key) response = client.search(query, max_results=max_results) @@ -156,14 +179,42 @@ class WebSearchTool(BaseTool): tool_name="web_search", content=formatted or "No results found.", success=True, - metadata={"num_results": len(results)}, + metadata={"num_results": len(results), "engine": "tavily"}, + ) + except ImportError: + logger.debug("Tavily not installed, falling back to DuckDuckGo") + except ( + MissingAPIKeyError, + InvalidAPIKeyError, + ForbiddenError, + UsageLimitExceededError, + TimeoutError, + BadRequestError, + ) as exc: + logger.debug( + "Tavily error (%s), falling back to DuckDuckGo", type(exc).__name__ + ) + except Exception as exc: + return ToolResult( + tool_name="web_search", + content=f"Tavily search failed: {exc}", + success=False, + ) + + try: + formatted = self._duckduckgo_search(query, max_results) + return ToolResult( + tool_name="web_search", + content=formatted or "No results found.", + success=True, + metadata={"engine": "duckduckgo"}, ) except ImportError: return ToolResult( tool_name="web_search", content=( - "tavily-python not installed." - " Install with: pip install tavily-python" + "tavily-python not installed and ddgs not available." + " Install with: pip install tavily-python ddgs" ), success=False, ) diff --git a/tests/tools/test_web_search.py b/tests/tools/test_web_search.py index d5e6ce3e..22ed2e6f 100644 --- a/tests/tools/test_web_search.py +++ b/tests/tools/test_web_search.py @@ -36,14 +36,15 @@ class TestWebSearchTool: assert result.success is False assert "No query" in result.content - def test_execute_no_api_key(self): + def test_execute_no_api_key(self, monkeypatch): + """When no API key, falls back to DuckDuckGo.""" tool = WebSearchTool(api_key=None) - # Clear env var to ensure no fallback with patch.dict("os.environ", {}, clear=True): tool._api_key = None + monkeypatch.delitem(sys.modules, "tavily", raising=False) result = tool.execute(query="test query") - assert result.success is False - assert "No API key" in result.content + assert result.success is True + assert result.metadata["engine"] == "duckduckgo" def test_execute_mocked_tavily(self, monkeypatch): mock_client = MagicMock() @@ -73,16 +74,57 @@ class TestWebSearchTool: assert result.metadata["num_results"] == 2 def test_execute_tavily_error(self, monkeypatch): + """When Tavily errors, falls back to DuckDuckGo.""" + from tavily.errors import UsageLimitExceededError + mock_client = MagicMock() - mock_client.search.side_effect = RuntimeError("API rate limit exceeded") + mock_client.search.side_effect = UsageLimitExceededError( + "API rate limit exceeded" + ) mock_tavily_module = MagicMock() mock_tavily_module.TavilyClient.return_value = mock_client + mock_tavily_errors = MagicMock() + mock_tavily_errors.UsageLimitExceededError = UsageLimitExceededError + mock_tavily_module.errors = mock_tavily_errors monkeypatch.setitem(sys.modules, "tavily", mock_tavily_module) tool = WebSearchTool(api_key="test-key") result = tool.execute(query="test query") - assert result.success is False - assert "Search error" in result.content + assert result.success is True + assert result.metadata["engine"] == "duckduckgo" + + def test_execute_duckduckgo_fallback_format(self, monkeypatch): + """DuckDuckGo fallback returns properly formatted results.""" + mock_tavily_module = MagicMock() + mock_tavily_module.TavilyClient.side_effect = ImportError( + "No module named 'tavily'" + ) + monkeypatch.setitem(sys.modules, "tavily", mock_tavily_module) + + mock_ddgs = MagicMock() + mock_ddgs.text.return_value = [ + { + "title": "DDG Result 1", + "href": "https://example.com/1", + "body": "Content 1", + }, + { + "title": "DDG Result 2", + "href": "https://example.com/2", + "body": "Content 2", + }, + ] + mock_ddgs_module = MagicMock() + mock_ddgs_module.DDGS.return_value = mock_ddgs + monkeypatch.setitem(sys.modules, "ddgs", mock_ddgs_module) + + tool = WebSearchTool(api_key="test-key") + result = tool.execute(query="test query") + assert result.success is True + assert "DDG Result 1" in result.content + assert "DDG Result 2" in result.content + assert "https://example.com/1" in result.content + assert result.metadata["engine"] == "duckduckgo" def test_max_results_parameter(self, monkeypatch): mock_client = MagicMock() @@ -103,8 +145,7 @@ class TestWebSearchTool: assert "query" in fn["function"]["parameters"]["properties"] def test_execute_import_error(self, monkeypatch): - """Simulate tavily-python not being installed.""" - # Remove tavily from sys.modules if present, and make import fail + """When tavily-python not installed, falls back to DuckDuckGo.""" monkeypatch.delitem(sys.modules, "tavily", raising=False) import builtins @@ -119,8 +160,8 @@ class TestWebSearchTool: tool = WebSearchTool(api_key="test-key") result = tool.execute(query="test query") - assert result.success is False - assert "tavily-python not installed" in result.content + assert result.success is True + assert result.metadata["engine"] == "duckduckgo" def test_empty_results(self, monkeypatch): mock_client = MagicMock() @@ -190,9 +231,7 @@ class TestUrlNormalization: assert url == "https://arxiv.org/abs/2310.03714" def test_arxiv_pdf_with_extension(self): - url = WebSearchTool._normalize_url( - "https://arxiv.org/pdf/2310.03714.pdf" - ) + url = WebSearchTool._normalize_url("https://arxiv.org/pdf/2310.03714.pdf") assert url == "https://arxiv.org/abs/2310.03714" def test_non_arxiv_unchanged(self): @@ -230,9 +269,7 @@ class TestUrlFetching: self._mock_ssrf(monkeypatch) mock_resp = MagicMock() - mock_resp.text = ( - "Content" - ) + mock_resp.text = "Content" mock_resp.headers = {"content-type": "text/html"} mock_resp.raise_for_status = MagicMock() monkeypatch.setattr(httpx, "get", MagicMock(return_value=mock_resp)) @@ -306,9 +343,7 @@ class TestExecuteWithUrl: monkeypatch.setattr(httpx, "get", MagicMock(return_value=mock_resp)) tool = WebSearchTool(api_key="test-key") - result = tool.execute( - query="Summarize https://example.com/article please" - ) + result = tool.execute(query="Summarize https://example.com/article please") assert result.success is True assert result.metadata.get("mode") == "fetch" @@ -317,7 +352,9 @@ class TestExecuteWithUrl: import openjarvis.tools.web_search as _ws monkeypatch.setattr( - _ws, "check_ssrf", lambda url: "private IP blocked", + _ws, + "check_ssrf", + lambda url: "private IP blocked", ) tool = WebSearchTool(api_key="test-key") @@ -331,7 +368,8 @@ class TestExecuteWithUrl: self._mock_ssrf(monkeypatch) monkeypatch.setattr( - httpx, "get", + httpx, + "get", MagicMock(side_effect=httpx.HTTPError("Connection failed")), ) diff --git a/uv.lock b/uv.lock index 051d776a..5629a0a3 100644 --- a/uv.lock +++ b/uv.lock @@ -893,15 +893,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ae/3a/dbeec9d1ee0844c679f6bb5d6ad4e9f198b1224f4e7a32825f47f6192b0c/cffi-2.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9", size = 184195, upload-time = "2025-09-08T23:23:43.004Z" }, ] -[[package]] -name = "cfgv" -version = "3.5.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/4e/b5/721b8799b04bf9afe054a3899c6cf4e880fcf8563cc71c15610242490a0c/cfgv-3.5.0.tar.gz", hash = "sha256:d5b1034354820651caa73ede66a6294d6e95c1b00acc5e9b098e917404669132", size = 7334, upload-time = "2025-11-19T20:55:51.612Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl", hash = "sha256:a8dc6b26ad22ff227d2634a65cb388215ce6cc96bbcc5cfde7641ae87e8dacc0", size = 7445, upload-time = "2025-11-19T20:55:50.744Z" }, -] - [[package]] name = "charset-normalizer" version = "3.4.4" @@ -1534,6 +1525,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fc/d5/0d563ea3c205eee226dc8053cf7682a8ac588db8acecd0eda2b587987a0b/datasets-4.5.0-py3-none-any.whl", hash = "sha256:b5d7e08096ffa407dd69e58b1c0271c9b2506140839b8d99af07375ad31b6726", size = 515196, upload-time = "2026-01-14T18:27:52.419Z" }, ] +[[package]] +name = "ddgs" +version = "9.11.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "lxml" }, + { name = "primp" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d7/3c/0ea1d5685bb0cd3607556d76da30e64c158e5d86bb16a3555f232fe0a33f/ddgs-9.11.4.tar.gz", hash = "sha256:445e22d3ffa16f893bfb0a717593bb0f34d1ceb1df68b6ff4ec27b1a3cdf6941", size = 34798, upload-time = "2026-03-14T18:15:18.086Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/46/f9412fafdebd4eaba366b5336ee4987995ccc3db1bc450e865a34db5fb63/ddgs-9.11.4-py3-none-any.whl", hash = "sha256:62d4d05b25db5d225a727c0a2771ef40258c1d894582c73dad24c88bf90f918b", size = 43681, upload-time = "2026-03-14T18:15:17.063Z" }, +] + [[package]] name = "decorator" version = "5.2.1" @@ -1627,15 +1632,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" }, ] -[[package]] -name = "distlib" -version = "0.4.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/96/8e/709914eb2b5749865801041647dc7f4e6d00b549cfe88b65ca192995f07c/distlib-0.4.0.tar.gz", hash = "sha256:feec40075be03a04501a973d81f633735b4b69f98b05450592310c0f401a4e0d", size = 614605, upload-time = "2025-07-17T16:52:00.465Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl", hash = "sha256:9659f7d87e46584a30b5780e43ac7a2143098441670ff0a49d5f9034c54a6c16", size = 469047, upload-time = "2025-07-17T16:51:58.613Z" }, -] - [[package]] name = "distro" version = "1.9.0" @@ -2699,15 +2695,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a8/af/48ac8483240de756d2438c380746e7130d1c6f75802ef22f3c6d49982787/huggingface_hub-0.36.2-py3-none-any.whl", hash = "sha256:48f0c8eac16145dfce371e9d2d7772854a4f591bcb56c9cf548accf531d54270", size = 566395, upload-time = "2026-02-06T09:24:11.133Z" }, ] -[[package]] -name = "identify" -version = "2.6.18" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/46/c4/7fb4db12296cdb11893d61c92048fe617ee853f8523b9b296ac03b43757e/identify-2.6.18.tar.gz", hash = "sha256:873ac56a5e3fd63e7438a7ecbc4d91aca692eb3fefa4534db2b7913f3fc352fd", size = 99580, upload-time = "2026-03-15T18:39:50.319Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/46/33/92ef41c6fad0233e41d3d84ba8e8ad18d1780f1e5d99b3c683e6d7f98b63/identify-2.6.18-py2.py3-none-any.whl", hash = "sha256:8db9d3c8ea9079db92cafb0ebf97abdc09d52e97f4dcf773a2e694048b7cd737", size = 99394, upload-time = "2026-03-15T18:39:48.915Z" }, -] - [[package]] name = "idna" version = "3.11" @@ -3297,6 +3284,130 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" }, ] +[[package]] +name = "lxml" +version = "6.0.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/88/262177de60548e5a2bfc46ad28232c9e9cbde697bd94132aeb80364675cb/lxml-6.0.2.tar.gz", hash = "sha256:cd79f3367bd74b317dda655dc8fcfa304d9eb6e4fb06b7168c5cf27f96e0cd62", size = 4073426, upload-time = "2025-09-22T04:04:59.287Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/db/8a/f8192a08237ef2fb1b19733f709db88a4c43bc8ab8357f01cb41a27e7f6a/lxml-6.0.2-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e77dd455b9a16bbd2a5036a63ddbd479c19572af81b624e79ef422f929eef388", size = 8590589, upload-time = "2025-09-22T04:00:10.51Z" }, + { url = "https://files.pythonhosted.org/packages/12/64/27bcd07ae17ff5e5536e8d88f4c7d581b48963817a13de11f3ac3329bfa2/lxml-6.0.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:5d444858b9f07cefff6455b983aea9a67f7462ba1f6cbe4a21e8bf6791bf2153", size = 4629671, upload-time = "2025-09-22T04:00:15.411Z" }, + { url = "https://files.pythonhosted.org/packages/02/5a/a7d53b3291c324e0b6e48f3c797be63836cc52156ddf8f33cd72aac78866/lxml-6.0.2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f952dacaa552f3bb8834908dddd500ba7d508e6ea6eb8c52eb2d28f48ca06a31", size = 4999961, upload-time = "2025-09-22T04:00:17.619Z" }, + { url = "https://files.pythonhosted.org/packages/f5/55/d465e9b89df1761674d8672bb3e4ae2c47033b01ec243964b6e334c6743f/lxml-6.0.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:71695772df6acea9f3c0e59e44ba8ac50c4f125217e84aab21074a1a55e7e5c9", size = 5157087, upload-time = "2025-09-22T04:00:19.868Z" }, + { url = "https://files.pythonhosted.org/packages/62/38/3073cd7e3e8dfc3ba3c3a139e33bee3a82de2bfb0925714351ad3d255c13/lxml-6.0.2-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:17f68764f35fd78d7c4cc4ef209a184c38b65440378013d24b8aecd327c3e0c8", size = 5067620, upload-time = "2025-09-22T04:00:21.877Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d3/1e001588c5e2205637b08985597827d3827dbaaece16348c8822bfe61c29/lxml-6.0.2-cp310-cp310-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:058027e261afed589eddcfe530fcc6f3402d7fd7e89bfd0532df82ebc1563dba", size = 5406664, upload-time = "2025-09-22T04:00:23.714Z" }, + { url = "https://files.pythonhosted.org/packages/20/cf/cab09478699b003857ed6ebfe95e9fb9fa3d3c25f1353b905c9b73cfb624/lxml-6.0.2-cp310-cp310-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a8ffaeec5dfea5881d4c9d8913a32d10cfe3923495386106e4a24d45300ef79c", size = 5289397, upload-time = "2025-09-22T04:00:25.544Z" }, + { url = "https://files.pythonhosted.org/packages/a3/84/02a2d0c38ac9a8b9f9e5e1bbd3f24b3f426044ad618b552e9549ee91bd63/lxml-6.0.2-cp310-cp310-manylinux_2_31_armv7l.whl", hash = "sha256:f2e3b1a6bb38de0bc713edd4d612969dd250ca8b724be8d460001a387507021c", size = 4772178, upload-time = "2025-09-22T04:00:27.602Z" }, + { url = "https://files.pythonhosted.org/packages/56/87/e1ceadcc031ec4aa605fe95476892d0b0ba3b7f8c7dcdf88fdeff59a9c86/lxml-6.0.2-cp310-cp310-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d6690ec5ec1cce0385cb20896b16be35247ac8c2046e493d03232f1c2414d321", size = 5358148, upload-time = "2025-09-22T04:00:29.323Z" }, + { url = "https://files.pythonhosted.org/packages/fe/13/5bb6cf42bb228353fd4ac5f162c6a84fd68a4d6f67c1031c8cf97e131fc6/lxml-6.0.2-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f2a50c3c1d11cad0ebebbac357a97b26aa79d2bcaf46f256551152aa85d3a4d1", size = 5112035, upload-time = "2025-09-22T04:00:31.061Z" }, + { url = "https://files.pythonhosted.org/packages/e4/e2/ea0498552102e59834e297c5c6dff8d8ded3db72ed5e8aad77871476f073/lxml-6.0.2-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:3efe1b21c7801ffa29a1112fab3b0f643628c30472d507f39544fd48e9549e34", size = 4799111, upload-time = "2025-09-22T04:00:33.11Z" }, + { url = "https://files.pythonhosted.org/packages/6a/9e/8de42b52a73abb8af86c66c969b3b4c2a96567b6ac74637c037d2e3baa60/lxml-6.0.2-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:59c45e125140b2c4b33920d21d83681940ca29f0b83f8629ea1a2196dc8cfe6a", size = 5351662, upload-time = "2025-09-22T04:00:35.237Z" }, + { url = "https://files.pythonhosted.org/packages/28/a2/de776a573dfb15114509a37351937c367530865edb10a90189d0b4b9b70a/lxml-6.0.2-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:452b899faa64f1805943ec1c0c9ebeaece01a1af83e130b69cdefeda180bb42c", size = 5314973, upload-time = "2025-09-22T04:00:37.086Z" }, + { url = "https://files.pythonhosted.org/packages/50/a0/3ae1b1f8964c271b5eec91db2043cf8c6c0bce101ebb2a633b51b044db6c/lxml-6.0.2-cp310-cp310-win32.whl", hash = "sha256:1e786a464c191ca43b133906c6903a7e4d56bef376b75d97ccbb8ec5cf1f0a4b", size = 3611953, upload-time = "2025-09-22T04:00:39.224Z" }, + { url = "https://files.pythonhosted.org/packages/d1/70/bd42491f0634aad41bdfc1e46f5cff98825fb6185688dc82baa35d509f1a/lxml-6.0.2-cp310-cp310-win_amd64.whl", hash = "sha256:dacf3c64ef3f7440e3167aa4b49aa9e0fb99e0aa4f9ff03795640bf94531bcb0", size = 4032695, upload-time = "2025-09-22T04:00:41.402Z" }, + { url = "https://files.pythonhosted.org/packages/d2/d0/05c6a72299f54c2c561a6c6cbb2f512e047fca20ea97a05e57931f194ac4/lxml-6.0.2-cp310-cp310-win_arm64.whl", hash = "sha256:45f93e6f75123f88d7f0cfd90f2d05f441b808562bf0bc01070a00f53f5028b5", size = 3680051, upload-time = "2025-09-22T04:00:43.525Z" }, + { url = "https://files.pythonhosted.org/packages/77/d5/becbe1e2569b474a23f0c672ead8a29ac50b2dc1d5b9de184831bda8d14c/lxml-6.0.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:13e35cbc684aadf05d8711a5d1b5857c92e5e580efa9a0d2be197199c8def607", size = 8634365, upload-time = "2025-09-22T04:00:45.672Z" }, + { url = "https://files.pythonhosted.org/packages/28/66/1ced58f12e804644426b85d0bb8a4478ca77bc1761455da310505f1a3526/lxml-6.0.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3b1675e096e17c6fe9c0e8c81434f5736c0739ff9ac6123c87c2d452f48fc938", size = 4650793, upload-time = "2025-09-22T04:00:47.783Z" }, + { url = "https://files.pythonhosted.org/packages/11/84/549098ffea39dfd167e3f174b4ce983d0eed61f9d8d25b7bf2a57c3247fc/lxml-6.0.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8ac6e5811ae2870953390452e3476694196f98d447573234592d30488147404d", size = 4944362, upload-time = "2025-09-22T04:00:49.845Z" }, + { url = "https://files.pythonhosted.org/packages/ac/bd/f207f16abf9749d2037453d56b643a7471d8fde855a231a12d1e095c4f01/lxml-6.0.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5aa0fc67ae19d7a64c3fe725dc9a1bb11f80e01f78289d05c6f62545affec438", size = 5083152, upload-time = "2025-09-22T04:00:51.709Z" }, + { url = "https://files.pythonhosted.org/packages/15/ae/bd813e87d8941d52ad5b65071b1affb48da01c4ed3c9c99e40abb266fbff/lxml-6.0.2-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de496365750cc472b4e7902a485d3f152ecf57bd3ba03ddd5578ed8ceb4c5964", size = 5023539, upload-time = "2025-09-22T04:00:53.593Z" }, + { url = "https://files.pythonhosted.org/packages/02/cd/9bfef16bd1d874fbe0cb51afb00329540f30a3283beb9f0780adbb7eec03/lxml-6.0.2-cp311-cp311-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:200069a593c5e40b8f6fc0d84d86d970ba43138c3e68619ffa234bc9bb806a4d", size = 5344853, upload-time = "2025-09-22T04:00:55.524Z" }, + { url = "https://files.pythonhosted.org/packages/b8/89/ea8f91594bc5dbb879734d35a6f2b0ad50605d7fb419de2b63d4211765cc/lxml-6.0.2-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7d2de809c2ee3b888b59f995625385f74629707c9355e0ff856445cdcae682b7", size = 5225133, upload-time = "2025-09-22T04:00:57.269Z" }, + { url = "https://files.pythonhosted.org/packages/b9/37/9c735274f5dbec726b2db99b98a43950395ba3d4a1043083dba2ad814170/lxml-6.0.2-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:b2c3da8d93cf5db60e8858c17684c47d01fee6405e554fb55018dd85fc23b178", size = 4677944, upload-time = "2025-09-22T04:00:59.052Z" }, + { url = "https://files.pythonhosted.org/packages/20/28/7dfe1ba3475d8bfca3878365075abe002e05d40dfaaeb7ec01b4c587d533/lxml-6.0.2-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:442de7530296ef5e188373a1ea5789a46ce90c4847e597856570439621d9c553", size = 5284535, upload-time = "2025-09-22T04:01:01.335Z" }, + { url = "https://files.pythonhosted.org/packages/e7/cf/5f14bc0de763498fc29510e3532bf2b4b3a1c1d5d0dff2e900c16ba021ef/lxml-6.0.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2593c77efde7bfea7f6389f1ab249b15ed4aa5bc5cb5131faa3b843c429fbedb", size = 5067343, upload-time = "2025-09-22T04:01:03.13Z" }, + { url = "https://files.pythonhosted.org/packages/1c/b0/bb8275ab5472f32b28cfbbcc6db7c9d092482d3439ca279d8d6fa02f7025/lxml-6.0.2-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:3e3cb08855967a20f553ff32d147e14329b3ae70ced6edc2f282b94afbc74b2a", size = 4725419, upload-time = "2025-09-22T04:01:05.013Z" }, + { url = "https://files.pythonhosted.org/packages/25/4c/7c222753bc72edca3b99dbadba1b064209bc8ed4ad448af990e60dcce462/lxml-6.0.2-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:2ed6c667fcbb8c19c6791bbf40b7268ef8ddf5a96940ba9404b9f9a304832f6c", size = 5275008, upload-time = "2025-09-22T04:01:07.327Z" }, + { url = "https://files.pythonhosted.org/packages/6c/8c/478a0dc6b6ed661451379447cdbec77c05741a75736d97e5b2b729687828/lxml-6.0.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:b8f18914faec94132e5b91e69d76a5c1d7b0c73e2489ea8929c4aaa10b76bbf7", size = 5248906, upload-time = "2025-09-22T04:01:09.452Z" }, + { url = "https://files.pythonhosted.org/packages/2d/d9/5be3a6ab2784cdf9accb0703b65e1b64fcdd9311c9f007630c7db0cfcce1/lxml-6.0.2-cp311-cp311-win32.whl", hash = "sha256:6605c604e6daa9e0d7f0a2137bdc47a2e93b59c60a65466353e37f8272f47c46", size = 3610357, upload-time = "2025-09-22T04:01:11.102Z" }, + { url = "https://files.pythonhosted.org/packages/e2/7d/ca6fb13349b473d5732fb0ee3eec8f6c80fc0688e76b7d79c1008481bf1f/lxml-6.0.2-cp311-cp311-win_amd64.whl", hash = "sha256:e5867f2651016a3afd8dd2c8238baa66f1e2802f44bc17e236f547ace6647078", size = 4036583, upload-time = "2025-09-22T04:01:12.766Z" }, + { url = "https://files.pythonhosted.org/packages/ab/a2/51363b5ecd3eab46563645f3a2c3836a2fc67d01a1b87c5017040f39f567/lxml-6.0.2-cp311-cp311-win_arm64.whl", hash = "sha256:4197fb2534ee05fd3e7afaab5d8bfd6c2e186f65ea7f9cd6a82809c887bd1285", size = 3680591, upload-time = "2025-09-22T04:01:14.874Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c8/8ff2bc6b920c84355146cd1ab7d181bc543b89241cfb1ebee824a7c81457/lxml-6.0.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a59f5448ba2ceccd06995c95ea59a7674a10de0810f2ce90c9006f3cbc044456", size = 8661887, upload-time = "2025-09-22T04:01:17.265Z" }, + { url = "https://files.pythonhosted.org/packages/37/6f/9aae1008083bb501ef63284220ce81638332f9ccbfa53765b2b7502203cf/lxml-6.0.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:e8113639f3296706fbac34a30813929e29247718e88173ad849f57ca59754924", size = 4667818, upload-time = "2025-09-22T04:01:19.688Z" }, + { url = "https://files.pythonhosted.org/packages/f1/ca/31fb37f99f37f1536c133476674c10b577e409c0a624384147653e38baf2/lxml-6.0.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a8bef9b9825fa8bc816a6e641bb67219489229ebc648be422af695f6e7a4fa7f", size = 4950807, upload-time = "2025-09-22T04:01:21.487Z" }, + { url = "https://files.pythonhosted.org/packages/da/87/f6cb9442e4bada8aab5ae7e1046264f62fdbeaa6e3f6211b93f4c0dd97f1/lxml-6.0.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:65ea18d710fd14e0186c2f973dc60bb52039a275f82d3c44a0e42b43440ea534", size = 5109179, upload-time = "2025-09-22T04:01:23.32Z" }, + { url = "https://files.pythonhosted.org/packages/c8/20/a7760713e65888db79bbae4f6146a6ae5c04e4a204a3c48896c408cd6ed2/lxml-6.0.2-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c371aa98126a0d4c739ca93ceffa0fd7a5d732e3ac66a46e74339acd4d334564", size = 5023044, upload-time = "2025-09-22T04:01:25.118Z" }, + { url = "https://files.pythonhosted.org/packages/a2/b0/7e64e0460fcb36471899f75831509098f3fd7cd02a3833ac517433cb4f8f/lxml-6.0.2-cp312-cp312-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:700efd30c0fa1a3581d80a748157397559396090a51d306ea59a70020223d16f", size = 5359685, upload-time = "2025-09-22T04:01:27.398Z" }, + { url = "https://files.pythonhosted.org/packages/b9/e1/e5df362e9ca4e2f48ed6411bd4b3a0ae737cc842e96877f5bf9428055ab4/lxml-6.0.2-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c33e66d44fe60e72397b487ee92e01da0d09ba2d66df8eae42d77b6d06e5eba0", size = 5654127, upload-time = "2025-09-22T04:01:29.629Z" }, + { url = "https://files.pythonhosted.org/packages/c6/d1/232b3309a02d60f11e71857778bfcd4acbdb86c07db8260caf7d008b08f8/lxml-6.0.2-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90a345bbeaf9d0587a3aaffb7006aa39ccb6ff0e96a57286c0cb2fd1520ea192", size = 5253958, upload-time = "2025-09-22T04:01:31.535Z" }, + { url = "https://files.pythonhosted.org/packages/35/35/d955a070994725c4f7d80583a96cab9c107c57a125b20bb5f708fe941011/lxml-6.0.2-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:064fdadaf7a21af3ed1dcaa106b854077fbeada827c18f72aec9346847cd65d0", size = 4711541, upload-time = "2025-09-22T04:01:33.801Z" }, + { url = "https://files.pythonhosted.org/packages/1e/be/667d17363b38a78c4bd63cfd4b4632029fd68d2c2dc81f25ce9eb5224dd5/lxml-6.0.2-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fbc74f42c3525ac4ffa4b89cbdd00057b6196bcefe8bce794abd42d33a018092", size = 5267426, upload-time = "2025-09-22T04:01:35.639Z" }, + { url = "https://files.pythonhosted.org/packages/ea/47/62c70aa4a1c26569bc958c9ca86af2bb4e1f614e8c04fb2989833874f7ae/lxml-6.0.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6ddff43f702905a4e32bc24f3f2e2edfe0f8fde3277d481bffb709a4cced7a1f", size = 5064917, upload-time = "2025-09-22T04:01:37.448Z" }, + { url = "https://files.pythonhosted.org/packages/bd/55/6ceddaca353ebd0f1908ef712c597f8570cc9c58130dbb89903198e441fd/lxml-6.0.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:6da5185951d72e6f5352166e3da7b0dc27aa70bd1090b0eb3f7f7212b53f1bb8", size = 4788795, upload-time = "2025-09-22T04:01:39.165Z" }, + { url = "https://files.pythonhosted.org/packages/cf/e8/fd63e15da5e3fd4c2146f8bbb3c14e94ab850589beab88e547b2dbce22e1/lxml-6.0.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:57a86e1ebb4020a38d295c04fc79603c7899e0df71588043eb218722dabc087f", size = 5676759, upload-time = "2025-09-22T04:01:41.506Z" }, + { url = "https://files.pythonhosted.org/packages/76/47/b3ec58dc5c374697f5ba37412cd2728f427d056315d124dd4b61da381877/lxml-6.0.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2047d8234fe735ab77802ce5f2297e410ff40f5238aec569ad7c8e163d7b19a6", size = 5255666, upload-time = "2025-09-22T04:01:43.363Z" }, + { url = "https://files.pythonhosted.org/packages/19/93/03ba725df4c3d72afd9596eef4a37a837ce8e4806010569bedfcd2cb68fd/lxml-6.0.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6f91fd2b2ea15a6800c8e24418c0775a1694eefc011392da73bc6cef2623b322", size = 5277989, upload-time = "2025-09-22T04:01:45.215Z" }, + { url = "https://files.pythonhosted.org/packages/c6/80/c06de80bfce881d0ad738576f243911fccf992687ae09fd80b734712b39c/lxml-6.0.2-cp312-cp312-win32.whl", hash = "sha256:3ae2ce7d6fedfb3414a2b6c5e20b249c4c607f72cb8d2bb7cc9c6ec7c6f4e849", size = 3611456, upload-time = "2025-09-22T04:01:48.243Z" }, + { url = "https://files.pythonhosted.org/packages/f7/d7/0cdfb6c3e30893463fb3d1e52bc5f5f99684a03c29a0b6b605cfae879cd5/lxml-6.0.2-cp312-cp312-win_amd64.whl", hash = "sha256:72c87e5ee4e58a8354fb9c7c84cbf95a1c8236c127a5d1b7683f04bed8361e1f", size = 4011793, upload-time = "2025-09-22T04:01:50.042Z" }, + { url = "https://files.pythonhosted.org/packages/ea/7b/93c73c67db235931527301ed3785f849c78991e2e34f3fd9a6663ffda4c5/lxml-6.0.2-cp312-cp312-win_arm64.whl", hash = "sha256:61cb10eeb95570153e0c0e554f58df92ecf5109f75eacad4a95baa709e26c3d6", size = 3672836, upload-time = "2025-09-22T04:01:52.145Z" }, + { url = "https://files.pythonhosted.org/packages/53/fd/4e8f0540608977aea078bf6d79f128e0e2c2bba8af1acf775c30baa70460/lxml-6.0.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:9b33d21594afab46f37ae58dfadd06636f154923c4e8a4d754b0127554eb2e77", size = 8648494, upload-time = "2025-09-22T04:01:54.242Z" }, + { url = "https://files.pythonhosted.org/packages/5d/f4/2a94a3d3dfd6c6b433501b8d470a1960a20ecce93245cf2db1706adf6c19/lxml-6.0.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6c8963287d7a4c5c9a432ff487c52e9c5618667179c18a204bdedb27310f022f", size = 4661146, upload-time = "2025-09-22T04:01:56.282Z" }, + { url = "https://files.pythonhosted.org/packages/25/2e/4efa677fa6b322013035d38016f6ae859d06cac67437ca7dc708a6af7028/lxml-6.0.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1941354d92699fb5ffe6ed7b32f9649e43c2feb4b97205f75866f7d21aa91452", size = 4946932, upload-time = "2025-09-22T04:01:58.989Z" }, + { url = "https://files.pythonhosted.org/packages/ce/0f/526e78a6d38d109fdbaa5049c62e1d32fdd70c75fb61c4eadf3045d3d124/lxml-6.0.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bb2f6ca0ae2d983ded09357b84af659c954722bbf04dea98030064996d156048", size = 5100060, upload-time = "2025-09-22T04:02:00.812Z" }, + { url = "https://files.pythonhosted.org/packages/81/76/99de58d81fa702cc0ea7edae4f4640416c2062813a00ff24bd70ac1d9c9b/lxml-6.0.2-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb2a12d704f180a902d7fa778c6d71f36ceb7b0d317f34cdc76a5d05aa1dd1df", size = 5019000, upload-time = "2025-09-22T04:02:02.671Z" }, + { url = "https://files.pythonhosted.org/packages/b5/35/9e57d25482bc9a9882cb0037fdb9cc18f4b79d85df94fa9d2a89562f1d25/lxml-6.0.2-cp313-cp313-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:6ec0e3f745021bfed19c456647f0298d60a24c9ff86d9d051f52b509663feeb1", size = 5348496, upload-time = "2025-09-22T04:02:04.904Z" }, + { url = "https://files.pythonhosted.org/packages/a6/8e/cb99bd0b83ccc3e8f0f528e9aa1f7a9965dfec08c617070c5db8d63a87ce/lxml-6.0.2-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:846ae9a12d54e368933b9759052d6206a9e8b250291109c48e350c1f1f49d916", size = 5643779, upload-time = "2025-09-22T04:02:06.689Z" }, + { url = "https://files.pythonhosted.org/packages/d0/34/9e591954939276bb679b73773836c6684c22e56d05980e31d52a9a8deb18/lxml-6.0.2-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ef9266d2aa545d7374938fb5c484531ef5a2ec7f2d573e62f8ce722c735685fd", size = 5244072, upload-time = "2025-09-22T04:02:08.587Z" }, + { url = "https://files.pythonhosted.org/packages/8d/27/b29ff065f9aaca443ee377aff699714fcbffb371b4fce5ac4ca759e436d5/lxml-6.0.2-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:4077b7c79f31755df33b795dc12119cb557a0106bfdab0d2c2d97bd3cf3dffa6", size = 4718675, upload-time = "2025-09-22T04:02:10.783Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9f/f756f9c2cd27caa1a6ef8c32ae47aadea697f5c2c6d07b0dae133c244fbe/lxml-6.0.2-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a7c5d5e5f1081955358533be077166ee97ed2571d6a66bdba6ec2f609a715d1a", size = 5255171, upload-time = "2025-09-22T04:02:12.631Z" }, + { url = "https://files.pythonhosted.org/packages/61/46/bb85ea42d2cb1bd8395484fd72f38e3389611aa496ac7772da9205bbda0e/lxml-6.0.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:8f8d0cbd0674ee89863a523e6994ac25fd5be9c8486acfc3e5ccea679bad2679", size = 5057175, upload-time = "2025-09-22T04:02:14.718Z" }, + { url = "https://files.pythonhosted.org/packages/95/0c/443fc476dcc8e41577f0af70458c50fe299a97bb6b7505bb1ae09aa7f9ac/lxml-6.0.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:2cbcbf6d6e924c28f04a43f3b6f6e272312a090f269eff68a2982e13e5d57659", size = 4785688, upload-time = "2025-09-22T04:02:16.957Z" }, + { url = "https://files.pythonhosted.org/packages/48/78/6ef0b359d45bb9697bc5a626e1992fa5d27aa3f8004b137b2314793b50a0/lxml-6.0.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:dfb874cfa53340009af6bdd7e54ebc0d21012a60a4e65d927c2e477112e63484", size = 5660655, upload-time = "2025-09-22T04:02:18.815Z" }, + { url = "https://files.pythonhosted.org/packages/ff/ea/e1d33808f386bc1339d08c0dcada6e4712d4ed8e93fcad5f057070b7988a/lxml-6.0.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:fb8dae0b6b8b7f9e96c26fdd8121522ce5de9bb5538010870bd538683d30e9a2", size = 5247695, upload-time = "2025-09-22T04:02:20.593Z" }, + { url = "https://files.pythonhosted.org/packages/4f/47/eba75dfd8183673725255247a603b4ad606f4ae657b60c6c145b381697da/lxml-6.0.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:358d9adae670b63e95bc59747c72f4dc97c9ec58881d4627fe0120da0f90d314", size = 5269841, upload-time = "2025-09-22T04:02:22.489Z" }, + { url = "https://files.pythonhosted.org/packages/76/04/5c5e2b8577bc936e219becb2e98cdb1aca14a4921a12995b9d0c523502ae/lxml-6.0.2-cp313-cp313-win32.whl", hash = "sha256:e8cd2415f372e7e5a789d743d133ae474290a90b9023197fd78f32e2dc6873e2", size = 3610700, upload-time = "2025-09-22T04:02:24.465Z" }, + { url = "https://files.pythonhosted.org/packages/fe/0a/4643ccc6bb8b143e9f9640aa54e38255f9d3b45feb2cbe7ae2ca47e8782e/lxml-6.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:b30d46379644fbfc3ab81f8f82ae4de55179414651f110a1514f0b1f8f6cb2d7", size = 4010347, upload-time = "2025-09-22T04:02:26.286Z" }, + { url = "https://files.pythonhosted.org/packages/31/ef/dcf1d29c3f530577f61e5fe2f1bd72929acf779953668a8a47a479ae6f26/lxml-6.0.2-cp313-cp313-win_arm64.whl", hash = "sha256:13dcecc9946dca97b11b7c40d29fba63b55ab4170d3c0cf8c0c164343b9bfdcf", size = 3671248, upload-time = "2025-09-22T04:02:27.918Z" }, + { url = "https://files.pythonhosted.org/packages/03/15/d4a377b385ab693ce97b472fe0c77c2b16ec79590e688b3ccc71fba19884/lxml-6.0.2-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:b0c732aa23de8f8aec23f4b580d1e52905ef468afb4abeafd3fec77042abb6fe", size = 8659801, upload-time = "2025-09-22T04:02:30.113Z" }, + { url = "https://files.pythonhosted.org/packages/c8/e8/c128e37589463668794d503afaeb003987373c5f94d667124ffd8078bbd9/lxml-6.0.2-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:4468e3b83e10e0317a89a33d28f7aeba1caa4d1a6fd457d115dd4ffe90c5931d", size = 4659403, upload-time = "2025-09-22T04:02:32.119Z" }, + { url = "https://files.pythonhosted.org/packages/00/ce/74903904339decdf7da7847bb5741fc98a5451b42fc419a86c0c13d26fe2/lxml-6.0.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:abd44571493973bad4598a3be7e1d807ed45aa2adaf7ab92ab7c62609569b17d", size = 4966974, upload-time = "2025-09-22T04:02:34.155Z" }, + { url = "https://files.pythonhosted.org/packages/1f/d3/131dec79ce61c5567fecf82515bd9bc36395df42501b50f7f7f3bd065df0/lxml-6.0.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:370cd78d5855cfbffd57c422851f7d3864e6ae72d0da615fca4dad8c45d375a5", size = 5102953, upload-time = "2025-09-22T04:02:36.054Z" }, + { url = "https://files.pythonhosted.org/packages/3a/ea/a43ba9bb750d4ffdd885f2cd333572f5bb900cd2408b67fdda07e85978a0/lxml-6.0.2-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:901e3b4219fa04ef766885fb40fa516a71662a4c61b80c94d25336b4934b71c0", size = 5055054, upload-time = "2025-09-22T04:02:38.154Z" }, + { url = "https://files.pythonhosted.org/packages/60/23/6885b451636ae286c34628f70a7ed1fcc759f8d9ad382d132e1c8d3d9bfd/lxml-6.0.2-cp314-cp314-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:a4bf42d2e4cf52c28cc1812d62426b9503cdb0c87a6de81442626aa7d69707ba", size = 5352421, upload-time = "2025-09-22T04:02:40.413Z" }, + { url = "https://files.pythonhosted.org/packages/48/5b/fc2ddfc94ddbe3eebb8e9af6e3fd65e2feba4967f6a4e9683875c394c2d8/lxml-6.0.2-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b2c7fdaa4d7c3d886a42534adec7cfac73860b89b4e5298752f60aa5984641a0", size = 5673684, upload-time = "2025-09-22T04:02:42.288Z" }, + { url = "https://files.pythonhosted.org/packages/29/9c/47293c58cc91769130fbf85531280e8cc7868f7fbb6d92f4670071b9cb3e/lxml-6.0.2-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:98a5e1660dc7de2200b00d53fa00bcd3c35a3608c305d45a7bbcaf29fa16e83d", size = 5252463, upload-time = "2025-09-22T04:02:44.165Z" }, + { url = "https://files.pythonhosted.org/packages/9b/da/ba6eceb830c762b48e711ded880d7e3e89fc6c7323e587c36540b6b23c6b/lxml-6.0.2-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:dc051506c30b609238d79eda75ee9cab3e520570ec8219844a72a46020901e37", size = 4698437, upload-time = "2025-09-22T04:02:46.524Z" }, + { url = "https://files.pythonhosted.org/packages/a5/24/7be3f82cb7990b89118d944b619e53c656c97dc89c28cfb143fdb7cd6f4d/lxml-6.0.2-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8799481bbdd212470d17513a54d568f44416db01250f49449647b5ab5b5dccb9", size = 5269890, upload-time = "2025-09-22T04:02:48.812Z" }, + { url = "https://files.pythonhosted.org/packages/1b/bd/dcfb9ea1e16c665efd7538fc5d5c34071276ce9220e234217682e7d2c4a5/lxml-6.0.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9261bb77c2dab42f3ecd9103951aeca2c40277701eb7e912c545c1b16e0e4917", size = 5097185, upload-time = "2025-09-22T04:02:50.746Z" }, + { url = "https://files.pythonhosted.org/packages/21/04/a60b0ff9314736316f28316b694bccbbabe100f8483ad83852d77fc7468e/lxml-6.0.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:65ac4a01aba353cfa6d5725b95d7aed6356ddc0a3cd734de00124d285b04b64f", size = 4745895, upload-time = "2025-09-22T04:02:52.968Z" }, + { url = "https://files.pythonhosted.org/packages/d6/bd/7d54bd1846e5a310d9c715921c5faa71cf5c0853372adf78aee70c8d7aa2/lxml-6.0.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:b22a07cbb82fea98f8a2fd814f3d1811ff9ed76d0fc6abc84eb21527596e7cc8", size = 5695246, upload-time = "2025-09-22T04:02:54.798Z" }, + { url = "https://files.pythonhosted.org/packages/fd/32/5643d6ab947bc371da21323acb2a6e603cedbe71cb4c99c8254289ab6f4e/lxml-6.0.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:d759cdd7f3e055d6bc8d9bec3ad905227b2e4c785dc16c372eb5b5e83123f48a", size = 5260797, upload-time = "2025-09-22T04:02:57.058Z" }, + { url = "https://files.pythonhosted.org/packages/33/da/34c1ec4cff1eea7d0b4cd44af8411806ed943141804ac9c5d565302afb78/lxml-6.0.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:945da35a48d193d27c188037a05fec5492937f66fb1958c24fc761fb9d40d43c", size = 5277404, upload-time = "2025-09-22T04:02:58.966Z" }, + { url = "https://files.pythonhosted.org/packages/82/57/4eca3e31e54dc89e2c3507e1cd411074a17565fa5ffc437c4ae0a00d439e/lxml-6.0.2-cp314-cp314-win32.whl", hash = "sha256:be3aaa60da67e6153eb15715cc2e19091af5dc75faef8b8a585aea372507384b", size = 3670072, upload-time = "2025-09-22T04:03:38.05Z" }, + { url = "https://files.pythonhosted.org/packages/e3/e0/c96cf13eccd20c9421ba910304dae0f619724dcf1702864fd59dd386404d/lxml-6.0.2-cp314-cp314-win_amd64.whl", hash = "sha256:fa25afbadead523f7001caf0c2382afd272c315a033a7b06336da2637d92d6ed", size = 4080617, upload-time = "2025-09-22T04:03:39.835Z" }, + { url = "https://files.pythonhosted.org/packages/d5/5d/b3f03e22b3d38d6f188ef044900a9b29b2fe0aebb94625ce9fe244011d34/lxml-6.0.2-cp314-cp314-win_arm64.whl", hash = "sha256:063eccf89df5b24e361b123e257e437f9e9878f425ee9aae3144c77faf6da6d8", size = 3754930, upload-time = "2025-09-22T04:03:41.565Z" }, + { url = "https://files.pythonhosted.org/packages/5e/5c/42c2c4c03554580708fc738d13414801f340c04c3eff90d8d2d227145275/lxml-6.0.2-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:6162a86d86893d63084faaf4ff937b3daea233e3682fb4474db07395794fa80d", size = 8910380, upload-time = "2025-09-22T04:03:01.645Z" }, + { url = "https://files.pythonhosted.org/packages/bf/4f/12df843e3e10d18d468a7557058f8d3733e8b6e12401f30b1ef29360740f/lxml-6.0.2-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:414aaa94e974e23a3e92e7ca5b97d10c0cf37b6481f50911032c69eeb3991bba", size = 4775632, upload-time = "2025-09-22T04:03:03.814Z" }, + { url = "https://files.pythonhosted.org/packages/e4/0c/9dc31e6c2d0d418483cbcb469d1f5a582a1cd00a1f4081953d44051f3c50/lxml-6.0.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:48461bd21625458dd01e14e2c38dd0aea69addc3c4f960c30d9f59d7f93be601", size = 4975171, upload-time = "2025-09-22T04:03:05.651Z" }, + { url = "https://files.pythonhosted.org/packages/e7/2b/9b870c6ca24c841bdd887504808f0417aa9d8d564114689266f19ddf29c8/lxml-6.0.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:25fcc59afc57d527cfc78a58f40ab4c9b8fd096a9a3f964d2781ffb6eb33f4ed", size = 5110109, upload-time = "2025-09-22T04:03:07.452Z" }, + { url = "https://files.pythonhosted.org/packages/bf/0c/4f5f2a4dd319a178912751564471355d9019e220c20d7db3fb8307ed8582/lxml-6.0.2-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5179c60288204e6ddde3f774a93350177e08876eaf3ab78aa3a3649d43eb7d37", size = 5041061, upload-time = "2025-09-22T04:03:09.297Z" }, + { url = "https://files.pythonhosted.org/packages/12/64/554eed290365267671fe001a20d72d14f468ae4e6acef1e179b039436967/lxml-6.0.2-cp314-cp314t-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:967aab75434de148ec80597b75062d8123cadf2943fb4281f385141e18b21338", size = 5306233, upload-time = "2025-09-22T04:03:11.651Z" }, + { url = "https://files.pythonhosted.org/packages/7a/31/1d748aa275e71802ad9722df32a7a35034246b42c0ecdd8235412c3396ef/lxml-6.0.2-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d100fcc8930d697c6561156c6810ab4a508fb264c8b6779e6e61e2ed5e7558f9", size = 5604739, upload-time = "2025-09-22T04:03:13.592Z" }, + { url = "https://files.pythonhosted.org/packages/8f/41/2c11916bcac09ed561adccacceaedd2bf0e0b25b297ea92aab99fd03d0fa/lxml-6.0.2-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2ca59e7e13e5981175b8b3e4ab84d7da57993eeff53c07764dcebda0d0e64ecd", size = 5225119, upload-time = "2025-09-22T04:03:15.408Z" }, + { url = "https://files.pythonhosted.org/packages/99/05/4e5c2873d8f17aa018e6afde417c80cc5d0c33be4854cce3ef5670c49367/lxml-6.0.2-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:957448ac63a42e2e49531b9d6c0fa449a1970dbc32467aaad46f11545be9af1d", size = 4633665, upload-time = "2025-09-22T04:03:17.262Z" }, + { url = "https://files.pythonhosted.org/packages/0f/c9/dcc2da1bebd6275cdc723b515f93edf548b82f36a5458cca3578bc899332/lxml-6.0.2-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b7fc49c37f1786284b12af63152fe1d0990722497e2d5817acfe7a877522f9a9", size = 5234997, upload-time = "2025-09-22T04:03:19.14Z" }, + { url = "https://files.pythonhosted.org/packages/9c/e2/5172e4e7468afca64a37b81dba152fc5d90e30f9c83c7c3213d6a02a5ce4/lxml-6.0.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e19e0643cc936a22e837f79d01a550678da8377d7d801a14487c10c34ee49c7e", size = 5090957, upload-time = "2025-09-22T04:03:21.436Z" }, + { url = "https://files.pythonhosted.org/packages/a5/b3/15461fd3e5cd4ddcb7938b87fc20b14ab113b92312fc97afe65cd7c85de1/lxml-6.0.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:1db01e5cf14345628e0cbe71067204db658e2fb8e51e7f33631f5f4735fefd8d", size = 4764372, upload-time = "2025-09-22T04:03:23.27Z" }, + { url = "https://files.pythonhosted.org/packages/05/33/f310b987c8bf9e61c4dd8e8035c416bd3230098f5e3cfa69fc4232de7059/lxml-6.0.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:875c6b5ab39ad5291588aed6925fac99d0097af0dd62f33c7b43736043d4a2ec", size = 5634653, upload-time = "2025-09-22T04:03:25.767Z" }, + { url = "https://files.pythonhosted.org/packages/70/ff/51c80e75e0bc9382158133bdcf4e339b5886c6ee2418b5199b3f1a61ed6d/lxml-6.0.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:cdcbed9ad19da81c480dfd6dd161886db6096083c9938ead313d94b30aadf272", size = 5233795, upload-time = "2025-09-22T04:03:27.62Z" }, + { url = "https://files.pythonhosted.org/packages/56/4d/4856e897df0d588789dd844dbed9d91782c4ef0b327f96ce53c807e13128/lxml-6.0.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:80dadc234ebc532e09be1975ff538d154a7fa61ea5031c03d25178855544728f", size = 5257023, upload-time = "2025-09-22T04:03:30.056Z" }, + { url = "https://files.pythonhosted.org/packages/0f/85/86766dfebfa87bea0ab78e9ff7a4b4b45225df4b4d3b8cc3c03c5cd68464/lxml-6.0.2-cp314-cp314t-win32.whl", hash = "sha256:da08e7bb297b04e893d91087df19638dc7a6bb858a954b0cc2b9f5053c922312", size = 3911420, upload-time = "2025-09-22T04:03:32.198Z" }, + { url = "https://files.pythonhosted.org/packages/fe/1a/b248b355834c8e32614650b8008c69ffeb0ceb149c793961dd8c0b991bb3/lxml-6.0.2-cp314-cp314t-win_amd64.whl", hash = "sha256:252a22982dca42f6155125ac76d3432e548a7625d56f5a273ee78a5057216eca", size = 4406837, upload-time = "2025-09-22T04:03:34.027Z" }, + { url = "https://files.pythonhosted.org/packages/92/aa/df863bcc39c5e0946263454aba394de8a9084dbaff8ad143846b0d844739/lxml-6.0.2-cp314-cp314t-win_arm64.whl", hash = "sha256:bb4c1847b303835d89d785a18801a883436cdfd5dc3d62947f9c49e24f0f5a2c", size = 3822205, upload-time = "2025-09-22T04:03:36.249Z" }, + { url = "https://files.pythonhosted.org/packages/e7/9c/780c9a8fce3f04690b374f72f41306866b0400b9d0fdf3e17aaa37887eed/lxml-6.0.2-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:e748d4cf8fef2526bb2a589a417eba0c8674e29ffcb570ce2ceca44f1e567bf6", size = 3939264, upload-time = "2025-09-22T04:04:32.892Z" }, + { url = "https://files.pythonhosted.org/packages/f5/5a/1ab260c00adf645d8bf7dec7f920f744b032f69130c681302821d5debea6/lxml-6.0.2-pp310-pypy310_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4ddb1049fa0579d0cbd00503ad8c58b9ab34d1254c77bc6a5576d96ec7853dba", size = 4216435, upload-time = "2025-09-22T04:04:34.907Z" }, + { url = "https://files.pythonhosted.org/packages/f2/37/565f3b3d7ffede22874b6d86be1a1763d00f4ea9fc5b9b6ccb11e4ec8612/lxml-6.0.2-pp310-pypy310_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cb233f9c95f83707dae461b12b720c1af9c28c2d19208e1be03387222151daf5", size = 4325913, upload-time = "2025-09-22T04:04:37.205Z" }, + { url = "https://files.pythonhosted.org/packages/22/ec/f3a1b169b2fb9d03467e2e3c0c752ea30e993be440a068b125fc7dd248b0/lxml-6.0.2-pp310-pypy310_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bc456d04db0515ce3320d714a1eac7a97774ff0849e7718b492d957da4631dd4", size = 4269357, upload-time = "2025-09-22T04:04:39.322Z" }, + { url = "https://files.pythonhosted.org/packages/77/a2/585a28fe3e67daa1cf2f06f34490d556d121c25d500b10082a7db96e3bcd/lxml-6.0.2-pp310-pypy310_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2613e67de13d619fd283d58bda40bff0ee07739f624ffee8b13b631abf33083d", size = 4412295, upload-time = "2025-09-22T04:04:41.647Z" }, + { url = "https://files.pythonhosted.org/packages/7b/d9/a57dd8bcebd7c69386c20263830d4fa72d27e6b72a229ef7a48e88952d9a/lxml-6.0.2-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:24a8e756c982c001ca8d59e87c80c4d9dcd4d9b44a4cbeb8d9be4482c514d41d", size = 3516913, upload-time = "2025-09-22T04:04:43.602Z" }, + { url = "https://files.pythonhosted.org/packages/0b/11/29d08bc103a62c0eba8016e7ed5aeebbf1e4312e83b0b1648dd203b0e87d/lxml-6.0.2-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:1c06035eafa8404b5cf475bb37a9f6088b0aca288d4ccc9d69389750d5543700", size = 3949829, upload-time = "2025-09-22T04:04:45.608Z" }, + { url = "https://files.pythonhosted.org/packages/12/b3/52ab9a3b31e5ab8238da241baa19eec44d2ab426532441ee607165aebb52/lxml-6.0.2-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c7d13103045de1bdd6fe5d61802565f1a3537d70cd3abf596aa0af62761921ee", size = 4226277, upload-time = "2025-09-22T04:04:47.754Z" }, + { url = "https://files.pythonhosted.org/packages/a0/33/1eaf780c1baad88224611df13b1c2a9dfa460b526cacfe769103ff50d845/lxml-6.0.2-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0a3c150a95fbe5ac91de323aa756219ef9cf7fde5a3f00e2281e30f33fa5fa4f", size = 4330433, upload-time = "2025-09-22T04:04:49.907Z" }, + { url = "https://files.pythonhosted.org/packages/7a/c1/27428a2ff348e994ab4f8777d3a0ad510b6b92d37718e5887d2da99952a2/lxml-6.0.2-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:60fa43be34f78bebb27812ed90f1925ec99560b0fa1decdb7d12b84d857d31e9", size = 4272119, upload-time = "2025-09-22T04:04:51.801Z" }, + { url = "https://files.pythonhosted.org/packages/f0/d0/3020fa12bcec4ab62f97aab026d57c2f0cfd480a558758d9ca233bb6a79d/lxml-6.0.2-pp311-pypy311_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:21c73b476d3cfe836be731225ec3421fa2f048d84f6df6a8e70433dff1376d5a", size = 4417314, upload-time = "2025-09-22T04:04:55.024Z" }, + { url = "https://files.pythonhosted.org/packages/6c/77/d7f491cbc05303ac6801651aabeb262d43f319288c1ea96c66b1d2692ff3/lxml-6.0.2-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:27220da5be049e936c3aca06f174e8827ca6445a4353a1995584311487fc4e3e", size = 3518768, upload-time = "2025-09-22T04:04:57.097Z" }, +] + [[package]] name = "magicattr" version = "0.1.6" @@ -4126,15 +4237,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/df/93/a7b983643d1253bb223234b5b226e69de6cda02b76cdca7770f684b795f5/ninja-1.13.0-py3-none-win_arm64.whl", hash = "sha256:3c0b40b1f0bba764644385319028650087b4c1b18cdfa6f45cb39a3669b81aa9", size = 290806, upload-time = "2025-08-11T15:10:18.018Z" }, ] -[[package]] -name = "nodeenv" -version = "1.10.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/24/bf/d1bda4f6168e0b2e9e5958945e01910052158313224ada5ce1fb2e1113b8/nodeenv-1.10.0.tar.gz", hash = "sha256:996c191ad80897d076bdfba80a41994c2b47c68e224c542b48feba42ba00f8bb", size = 55611, upload-time = "2025-12-20T14:08:54.006Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl", hash = "sha256:5bb13e3eed2923615535339b3c620e76779af4cb4c6a90deccc9e36b274d3827", size = 23438, upload-time = "2025-12-20T14:08:52.782Z" }, -] - [[package]] name = "numba" version = "0.61.2" @@ -4574,6 +4676,7 @@ source = { editable = "." } dependencies = [ { name = "click" }, { name = "datasets" }, + { name = "ddgs" }, { name = "httpx" }, { name = "openai" }, { name = "rich" }, @@ -4630,7 +4733,6 @@ dashboard = [ ] dev = [ { name = "maturin" }, - { name = "pre-commit" }, { name = "pytest" }, { name = "pytest-asyncio" }, { name = "pytest-cov" }, @@ -4740,6 +4842,7 @@ speech-deepgram = [ { name = "deepgram-sdk" }, ] tools-search = [ + { name = "ddgs" }, { name = "tavily-python" }, ] @@ -4758,6 +4861,8 @@ requires-dist = [ { name = "croniter", marker = "extra == 'scheduler'", specifier = ">=2.0" }, { name = "cryptography", marker = "extra == 'security-signing'", specifier = ">=43" }, { name = "datasets", specifier = ">=4.5.0" }, + { name = "ddgs", specifier = ">=9.11.4" }, + { name = "ddgs", marker = "extra == 'tools-search'", specifier = ">=9.11.4" }, { name = "deepgram-sdk", marker = "extra == 'speech-deepgram'", specifier = ">=3.0" }, { name = "discord-py", marker = "extra == 'channel-discord'", specifier = ">=2.3" }, { name = "docker", marker = "extra == 'sandbox-docker'", specifier = ">=7.0" }, @@ -4789,7 +4894,6 @@ requires-dist = [ { name = "pdfplumber", marker = "extra == 'pdf'", specifier = ">=0.10" }, { name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" }, { name = "praw", marker = "extra == 'channel-reddit'", specifier = ">=7.0" }, - { name = "pre-commit", marker = "extra == 'dev'", specifier = ">=3.0" }, { name = "pydantic", marker = "extra == 'server'", specifier = ">=2.0" }, { name = "pymessenger", marker = "extra == 'channel-messenger'", specifier = ">=0.0.7" }, { name = "pynostr", marker = "extra == 'channel-nostr'", specifier = ">=0.6" }, @@ -5496,19 +5600,41 @@ wheels = [ ] [[package]] -name = "pre-commit" -version = "4.5.1" +name = "primp" +version = "1.1.3" source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "cfgv" }, - { name = "identify" }, - { name = "nodeenv" }, - { name = "pyyaml" }, - { name = "virtualenv" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/40/f1/6d86a29246dfd2e9b6237f0b5823717f60cad94d47ddc26afa916d21f525/pre_commit-4.5.1.tar.gz", hash = "sha256:eb545fcff725875197837263e977ea257a402056661f09dae08e4b149b030a61", size = 198232, upload-time = "2025-12-16T21:14:33.552Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c4/0e/62ed44af95c66fd6fa8ad49c8bde815f64c7e976772d6979730be2b7cd97/primp-1.1.3.tar.gz", hash = "sha256:56adc3b8a5048cbd5f926b21fdff839195f3a9181512ca33f56ddc66f4c95897", size = 311356, upload-time = "2026-03-11T06:42:51.763Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/5d/19/fd3ef348460c80af7bb4669ea7926651d1f95c23ff2df18b9d24bab4f3fa/pre_commit-4.5.1-py2.py3-none-any.whl", hash = "sha256:3b3afd891e97337708c1674210f8eba659b52a38ea5f822ff142d10786221f77", size = 226437, upload-time = "2025-12-16T21:14:32.409Z" }, + { url = "https://files.pythonhosted.org/packages/ad/6b/36794b5758a0dd1251e67b6ab3ea946e53fa69745e0ecc29facc072ddf5b/primp-1.1.3-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:24383cfc267f620769be102b7fa4b64c7d47105f86bd21d047f1e07709e83c6e", size = 4000660, upload-time = "2026-03-11T06:42:58.092Z" }, + { url = "https://files.pythonhosted.org/packages/98/18/ebbe318a926d158c57f9e9cf49bbea70e8f0bd7f87e7675ed68e0d6ab433/primp-1.1.3-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:61bcb8c53b41e4bac43d04a1374b6ab7d8ded0f3517d32c5cdd5c30562756805", size = 3737318, upload-time = "2026-03-11T06:42:50.19Z" }, + { url = "https://files.pythonhosted.org/packages/a9/4c/430c9154284b53b771e6713a18dec4ad0159e4a501a20b222d67c730ced9/primp-1.1.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b0c6b9388578ee9d903f30549a792c5f391fdeb9d36b508da2ffb8e13c764954", size = 3881005, upload-time = "2026-03-11T06:43:12.894Z" }, + { url = "https://files.pythonhosted.org/packages/93/34/2466ef66386a1b50e6aaf7832f9f603628407bb33342378faf4b38c4aee8/primp-1.1.3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:09a8bfa870c92c81d76611846ec53b2520845e3ec5f4139f47604986bcf4bc25", size = 3514480, upload-time = "2026-03-11T06:43:06.058Z" }, + { url = "https://files.pythonhosted.org/packages/ff/42/ca7a71df6493dd6c1971c0cc3b20b8125e2547eb3bf88b4429715cb6ed81/primp-1.1.3-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ac372cb9959fff690b255fad91c5b3bc948c14065da9fc00ad80d139651515af", size = 3734658, upload-time = "2026-03-11T06:43:47.486Z" }, + { url = "https://files.pythonhosted.org/packages/bc/7c/0fb34db619e9935e11140929713c2c7b5323c1e8ba75cad6f0aade51c89d/primp-1.1.3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3034672a007f04e12b8fe7814c97ea172e8b9c5d45bd7b00cf6e7334fdd4222a", size = 4011898, upload-time = "2026-03-11T06:43:41.121Z" }, + { url = "https://files.pythonhosted.org/packages/da/8b/afd1bd8b14f38d58c5ebd0d45fc6b74914956907aa4e981bb2e5231626d3/primp-1.1.3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a07d5b7d7278dc63452a59f3bf851dc4d1f8ddc2aada7844cbdb68002256e2f4", size = 3910728, upload-time = "2026-03-11T06:43:01.819Z" }, + { url = "https://files.pythonhosted.org/packages/32/9e/1ec3a9678efcbb51e50d7b4886d9195f956c9fd7f4efcff13ccb152248b0/primp-1.1.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:08eec2f58abbcc1060032a2af81dabacec87a580a364a75862039f7422ac82e6", size = 4114189, upload-time = "2026-03-11T06:42:47.639Z" }, + { url = "https://files.pythonhosted.org/packages/28/d9/76de611027c0688be188d5a833be45b1e36d9c0c98baefab27bf6336ab9d/primp-1.1.3-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:9716d4cd36db2c175443fe1bbd54045a944fc9c49d01a385af8ada1fe9c948df", size = 4061973, upload-time = "2026-03-11T06:43:37.301Z" }, + { url = "https://files.pythonhosted.org/packages/37/3b/a30a5ea366705d0ece265b12ad089793d644bd5730b18201e3a0a7fa7b5f/primp-1.1.3-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:e19daca65dc6df369c33e711fa481ad2afe5d26c5bde926c069b3ab067c4fd45", size = 3747920, upload-time = "2026-03-11T06:43:10.403Z" }, + { url = "https://files.pythonhosted.org/packages/df/46/e3c323221c371cdfe6c2ed971f7a70e3b69f30b561977715c55230bd5fda/primp-1.1.3-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:ee357537712aa486364b0194cf403c5f9eaaa1354e23e9ac8322a22003f31e6b", size = 3861184, upload-time = "2026-03-11T06:43:49.391Z" }, + { url = "https://files.pythonhosted.org/packages/8a/7f/babaf00753daad7d80061003d7ae1bdfca64ea94c181cdea8d25c8a7226a/primp-1.1.3-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:06c53e77ebf6ac00633bc09e7e5a6d1a994592729d399ca8f065451a2574b92e", size = 4364610, upload-time = "2026-03-11T06:42:56.223Z" }, + { url = "https://files.pythonhosted.org/packages/03/48/c7bca8045c681f5f60972c180d2a20582c7a0857b3b07b12e0a0ee062ac4/primp-1.1.3-cp310-abi3-win32.whl", hash = "sha256:4b1ea3693c118bf04a6e05286f0a73637cf6fe5c9fd77fa1e29a01f190adf512", size = 3265160, upload-time = "2026-03-11T06:43:43.774Z" }, + { url = "https://files.pythonhosted.org/packages/45/3e/4a4b8a0f6f15734cded91e85439e68912b2bb8eafe7132420c13c2db8340/primp-1.1.3-cp310-abi3-win_amd64.whl", hash = "sha256:5ea386a4c8c4d8c1021d17182f4ee24dbb6f17c107c4e9ee5500b6372cf08f32", size = 3603953, upload-time = "2026-03-11T06:43:33.144Z" }, + { url = "https://files.pythonhosted.org/packages/70/46/1baf13a7f5fbed6052deb3e4822c69441a8d0fd990fe2a50e4cec802130b/primp-1.1.3-cp310-abi3-win_arm64.whl", hash = "sha256:63c7b1a1ccbcd07213f438375df186f807cdc5214bc2debb055737db9b5078de", size = 3619917, upload-time = "2026-03-11T06:42:44.76Z" }, + { url = "https://files.pythonhosted.org/packages/be/0c/a73cbe13f075e7ceaa5172b44ebc6f423713c6b4efe168114993a1710b26/primp-1.1.3-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:4b3d52f3233134584ef527e7e52f1b371a964ade1df0461f8187100e41d7fa84", size = 3987141, upload-time = "2026-03-11T06:43:24.904Z" }, + { url = "https://files.pythonhosted.org/packages/49/56/b70d7991fb1e07af53706b1f69f78a0b440a7b4b2a2999c44ab44afef1e7/primp-1.1.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:b3d947e2c1d15147e8f4736d027b9f3bef518d67da859ead1c54e028ff491bbb", size = 3735665, upload-time = "2026-03-11T06:43:29.347Z" }, + { url = "https://files.pythonhosted.org/packages/31/82/69efc663341c2bab55659ed221903a090e5c80255c2de2acc70f3726a3fc/primp-1.1.3-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c3ed2fee7d4758f6bb873b19a6759f54e0bc453213dad5ba7e52de7582921079", size = 3873695, upload-time = "2026-03-11T06:43:15.396Z" }, + { url = "https://files.pythonhosted.org/packages/07/7e/6b360742019ef8fb4ea036a420eb21b0a58d380ca09c68b075fc103cc043/primp-1.1.3-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b5aa717f256af9e4391fb1c4dc946d99d04652b4c57dad20c3947e839ab26769", size = 3512644, upload-time = "2026-03-11T06:43:08.368Z" }, + { url = "https://files.pythonhosted.org/packages/03/46/51d2ada6d5b53b8496eddf2c80392deab13698987412d0234f88e72390c1/primp-1.1.3-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:17f37fcacd97540f68b06f2b468b111ca7f2b142c48370db7344b522274fc0d6", size = 3733114, upload-time = "2026-03-11T06:43:22.838Z" }, + { url = "https://files.pythonhosted.org/packages/45/f5/5f5f5f4bef7e247ec3543e2fbdb670d8db8753a7693baf9c8b9fcf52cd43/primp-1.1.3-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d5f010d0b8ba111dd9a66f814c2cd56332e047c98f45d7714ffbf2b1cec5b073", size = 4005664, upload-time = "2026-03-11T06:43:20.824Z" }, + { url = "https://files.pythonhosted.org/packages/f2/bf/99cf4a5f179b3f13b0c2ba4d3ae8f8af19f0084308e76cb79a0cee03c31b/primp-1.1.3-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2e1e431915e4a7094d589213fc14e955243d93751031d889f4b359fa8ed54298", size = 3895746, upload-time = "2026-03-11T06:43:35.376Z" }, + { url = "https://files.pythonhosted.org/packages/c3/75/4c625e1cab37585365b0856ca44f31ad598e92a847d23561f454b7f36fca/primp-1.1.3-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aaffa22dae2f193d899d9f68cca109ea5d16cdf4c901c20cec186de89e7d5db4", size = 4109815, upload-time = "2026-03-11T06:43:04.059Z" }, + { url = "https://files.pythonhosted.org/packages/49/72/6197ea78779d359f307be1acc64659896fc960ed91c0bdc6e6e698e423e6/primp-1.1.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:f93bee50990884621ef482e8434e87f9fbb4eca6f4d47973c44c5d6393c35679", size = 4050839, upload-time = "2026-03-11T06:43:18.296Z" }, + { url = "https://files.pythonhosted.org/packages/a4/b2/cdd565b28bcf7ce555f4decdf89dafd16db8ed3ba8661890d3b9337abe45/primp-1.1.3-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:399dfb9ad01c3612c9e510a7034ac925af5524cade0961d8a019dedd90a46474", size = 3748397, upload-time = "2026-03-11T06:43:27.347Z" }, + { url = "https://files.pythonhosted.org/packages/62/6e/def3a90821b52589dbe1f57477c2c89bde7a5b26a7c166d7751930c06f98/primp-1.1.3-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:78ce595bbb9f339e83975efa9db2a81128842fad1a2fdafb78d72fcdc59590fc", size = 3861261, upload-time = "2026-03-11T06:43:39.292Z" }, + { url = "https://files.pythonhosted.org/packages/10/7d/3e610614d6a426502cfc6eccea21ef4557b39177d365df393c994945ca43/primp-1.1.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:7d709bdf520aa9401c0592b642730b3477c828629f01d2550977b77135b34e8d", size = 4358608, upload-time = "2026-03-11T06:43:45.606Z" }, + { url = "https://files.pythonhosted.org/packages/91/50/eb190cefe5eb05896825a5b3365d5650b9327161329cd1df4f7351b66ba9/primp-1.1.3-cp314-cp314t-win32.whl", hash = "sha256:6fe893eb87156dfb146dd666c7c8754670de82e38af0a27d82a47b7461ec2eea", size = 3259903, upload-time = "2026-03-11T06:42:59.922Z" }, + { url = "https://files.pythonhosted.org/packages/1f/a8/9e8534bc6d729a667f79b249fcdbf2230b0eb41214e277998cd6be900498/primp-1.1.3-cp314-cp314t-win_amd64.whl", hash = "sha256:ced76ef6669f31dc4af25e81e87914310645bcfc0892036bde084dafd6d00c3c", size = 3602569, upload-time = "2026-03-11T06:42:53.955Z" }, + { url = "https://files.pythonhosted.org/packages/9c/92/e18be996a01c7fd0e7dd7d198edefe42813cdfe1637bbbc80370ce656f62/primp-1.1.3-cp314-cp314t-win_arm64.whl", hash = "sha256:efadef0dfd10e733a254a949abf9ed05c668c28a68aa6513d811c0c6acd54cdb", size = 3611571, upload-time = "2026-03-11T06:43:31.249Z" }, ] [[package]] @@ -6431,19 +6557,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, ] -[[package]] -name = "python-discovery" -version = "1.1.3" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "filelock" }, - { name = "platformdirs" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/d7/7e/9f3b0dd3a074a6c3e1e79f35e465b1f2ee4b262d619de00cfce523cc9b24/python_discovery-1.1.3.tar.gz", hash = "sha256:7acca36e818cd88e9b2ba03e045ad7e93e1713e29c6bbfba5d90202310b7baa5", size = 56945, upload-time = "2026-03-10T15:08:15.038Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/e7/80/73211fc5bfbfc562369b4aa61dc1e4bf07dc7b34df7b317e4539316b809c/python_discovery-1.1.3-py3-none-any.whl", hash = "sha256:90e795f0121bc84572e737c9aa9966311b9fde44ffb88a5953b3ec9b31c6945e", size = 31485, upload-time = "2026-03-10T15:08:13.06Z" }, -] - [[package]] name = "python-dotenv" version = "1.2.1" @@ -8817,22 +8930,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5d/c1/f01846f10c8383b197a67b4cdfea6eea476aab678ed9f6f5d9ccc8c8dddf/viberbot-1.0.12-py3-none-any.whl", hash = "sha256:ca43fea2945d650c2ef2cbd777f3c546c795bf945278f6620ceda42f4101d801", size = 26164, upload-time = "2021-01-27T13:45:43.726Z" }, ] -[[package]] -name = "virtualenv" -version = "21.2.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "distlib" }, - { name = "filelock" }, - { name = "platformdirs" }, - { name = "python-discovery" }, - { name = "typing-extensions", marker = "python_full_version < '3.11'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/aa/92/58199fe10049f9703c2666e809c4f686c54ef0a68b0f6afccf518c0b1eb9/virtualenv-21.2.0.tar.gz", hash = "sha256:1720dc3a62ef5b443092e3f499228599045d7fea4c79199770499df8becf9098", size = 5840618, upload-time = "2026-03-09T17:24:38.013Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/c6/59/7d02447a55b2e55755011a647479041bc92a82e143f96a8195cb33bd0a1c/virtualenv-21.2.0-py3-none-any.whl", hash = "sha256:1bd755b504931164a5a496d217c014d098426cddc79363ad66ac78125f9d908f", size = 5825084, upload-time = "2026-03-09T17:24:35.378Z" }, -] - [[package]] name = "vllm" version = "0.17.1" From 806067a002e1e3d32375a4adb3c1c05084bacbf5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Teemu=20S=C3=A4ilynoja?= Date: Tue, 17 Mar 2026 10:27:06 +0200 Subject: [PATCH 28/92] fix: simplify web search error handling and improve test mocks - Catch any Exception from Tavily (not just specific error types) - Fall back to DuckDuckGo for any error, making the tool more robust - Fix test mocks to use builtins.__import__ for proper local import mocking - Simplify test_execute_tavily_error to test generic exception handling --- src/openjarvis/tools/web_search.py | 25 +---------- tests/tools/test_web_search.py | 72 +++++++++++++++++++++++++----- 2 files changed, 61 insertions(+), 36 deletions(-) diff --git a/src/openjarvis/tools/web_search.py b/src/openjarvis/tools/web_search.py index 60900f66..9b10bef3 100644 --- a/src/openjarvis/tools/web_search.py +++ b/src/openjarvis/tools/web_search.py @@ -158,14 +158,6 @@ class WebSearchTool(BaseTool): try: from tavily import TavilyClient - from tavily.errors import ( - BadRequestError, - ForbiddenError, - InvalidAPIKeyError, - MissingAPIKeyError, - TimeoutError, - UsageLimitExceededError, - ) client = TavilyClient(api_key=self._api_key) response = client.search(query, max_results=max_results) @@ -181,25 +173,10 @@ class WebSearchTool(BaseTool): success=True, metadata={"num_results": len(results), "engine": "tavily"}, ) - except ImportError: - logger.debug("Tavily not installed, falling back to DuckDuckGo") - except ( - MissingAPIKeyError, - InvalidAPIKeyError, - ForbiddenError, - UsageLimitExceededError, - TimeoutError, - BadRequestError, - ) as exc: + except Exception as exc: logger.debug( "Tavily error (%s), falling back to DuckDuckGo", type(exc).__name__ ) - except Exception as exc: - return ToolResult( - tool_name="web_search", - content=f"Tavily search failed: {exc}", - success=False, - ) try: formatted = self._duckduckgo_search(query, max_results) diff --git a/tests/tools/test_web_search.py b/tests/tools/test_web_search.py index 22ed2e6f..cdfdfc21 100644 --- a/tests/tools/test_web_search.py +++ b/tests/tools/test_web_search.py @@ -64,7 +64,21 @@ class TestWebSearchTool: } mock_tavily_module = MagicMock() mock_tavily_module.TavilyClient.return_value = mock_client - monkeypatch.setitem(sys.modules, "tavily", mock_tavily_module) + + import builtins + + original_import = builtins.__import__ + + def _mock_import(name, *args, **kwargs): + if name == "tavily": + return mock_tavily_module + if name == "tavily.errors": + mock_errors = MagicMock() + mock_errors.UsageLimitExceededError = Exception + return mock_errors + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", _mock_import) tool = WebSearchTool(api_key="test-key") result = tool.execute(query="test query") @@ -74,19 +88,27 @@ class TestWebSearchTool: assert result.metadata["num_results"] == 2 def test_execute_tavily_error(self, monkeypatch): - """When Tavily errors, falls back to DuckDuckGo.""" - from tavily.errors import UsageLimitExceededError + """When Tavily errors (any error), falls back to DuckDuckGo.""" + import builtins + from typing import Any + + original_import = builtins.__import__ + + class TavilyError(Exception): + def __init__(self, message: str): + super().__init__(message) mock_client = MagicMock() - mock_client.search.side_effect = UsageLimitExceededError( - "API rate limit exceeded" - ) + mock_client.search.side_effect = TavilyError("API error") mock_tavily_module = MagicMock() mock_tavily_module.TavilyClient.return_value = mock_client - mock_tavily_errors = MagicMock() - mock_tavily_errors.UsageLimitExceededError = UsageLimitExceededError - mock_tavily_module.errors = mock_tavily_errors - monkeypatch.setitem(sys.modules, "tavily", mock_tavily_module) + + def _mock_import(name: str, *args: Any, **kwargs: Any): + if name == "tavily": + return mock_tavily_module + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", _mock_import) tool = WebSearchTool(api_key="test-key") result = tool.execute(query="test query") @@ -127,11 +149,24 @@ class TestWebSearchTool: assert result.metadata["engine"] == "duckduckgo" def test_max_results_parameter(self, monkeypatch): + import builtins + + original_import = builtins.__import__ + mock_client = MagicMock() mock_client.search.return_value = {"results": []} mock_tavily_module = MagicMock() mock_tavily_module.TavilyClient.return_value = mock_client - monkeypatch.setitem(sys.modules, "tavily", mock_tavily_module) + mock_errors = MagicMock() + + def _mock_import(name, *args, **kwargs): + if name == "tavily": + return mock_tavily_module + if name == "tavily.errors": + return mock_errors + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", _mock_import) tool = WebSearchTool(api_key="test-key", max_results=3) tool.execute(query="test", max_results=7) @@ -164,11 +199,24 @@ class TestWebSearchTool: assert result.metadata["engine"] == "duckduckgo" def test_empty_results(self, monkeypatch): + import builtins + + original_import = builtins.__import__ + mock_client = MagicMock() mock_client.search.return_value = {"results": []} mock_tavily_module = MagicMock() mock_tavily_module.TavilyClient.return_value = mock_client - monkeypatch.setitem(sys.modules, "tavily", mock_tavily_module) + mock_errors = MagicMock() + + def _mock_import(name, *args, **kwargs): + if name == "tavily": + return mock_tavily_module + if name == "tavily.errors": + return mock_errors + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", _mock_import) tool = WebSearchTool(api_key="test-key") result = tool.execute(query="obscure query") From 62eaa70736dd5a3301f015e7cd0b9dd6779b7737 Mon Sep 17 00:00:00 2001 From: Octopus Date: Fri, 20 Mar 2026 06:16:33 -0500 Subject: [PATCH 29/92] feat: add MiniMax as cloud inference provider with M2.7 default (#85) * feat: add MiniMax as cloud inference provider Add MiniMax M2.5 and M2.5-highspeed as a 5th cloud provider alongside OpenAI, Anthropic, Google, and OpenRouter. Uses the OpenAI-compatible API at api.minimax.io/v1 via the existing openai SDK dependency. Changes: - Add MiniMax client init, generate, and streaming in CloudEngine - Add MiniMax models to model catalog with correct pricing - Add MINIMAX_API_KEY environment variable support - Add temperature clamping (0.01-1.0) per MiniMax API constraints - Add 19 unit tests and 3 integration tests - Update docs and README with MiniMax provider info * feat: upgrade MiniMax default model to M2.7 - Add MiniMax-M2.7 and MiniMax-M2.7-highspeed to model list - Set MiniMax-M2.7 as default model (first in list) - Keep all previous models (M2.5, M2.5-highspeed) as alternatives - Update pricing table with M2.7 entries - Update model catalog with M2.7 specs - Update docs to list all available MiniMax models - Update unit tests (23 passing) and integration tests (3 passing) --------- Co-authored-by: Octopus --- README.md | 2 +- docs/getting-started/configuration.md | 3 +- src/openjarvis/engine/cloud.py | 122 ++++++- src/openjarvis/intelligence/model_catalog.py | 63 ++++ tests/engine/test_cloud_minimax.py | 331 +++++++++++++++++++ tests/integration/test_minimax_cloud.py | 63 ++++ 6 files changed, 580 insertions(+), 4 deletions(-) create mode 100644 tests/engine/test_cloud_minimax.py create mode 100644 tests/integration/test_minimax_cloud.py diff --git a/README.md b/README.md index e7768888..39f3f0a5 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ uv sync # core framework uv sync --extra server # + FastAPI server ``` -You also need a local inference backend: [Ollama](https://ollama.com), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), or [llama.cpp](https://github.com/ggerganov/llama.cpp). +You also need a local inference backend: [Ollama](https://ollama.com), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), or [llama.cpp](https://github.com/ggerganov/llama.cpp). Alternatively, use the `cloud` engine with [OpenAI](https://openai.com), [Anthropic](https://anthropic.com), [Google Gemini](https://ai.google.dev), [OpenRouter](https://openrouter.ai), or [MiniMax](https://www.minimax.io) by setting the corresponding API key environment variable. ## Quick Start diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index a4f313c2..19083b35 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -140,7 +140,7 @@ max_tokens = 1024 | `checkpoint_path` | string | `""` | Path to a fine-tuned checkpoint or LoRA adapter directory. | | `quantization` | string | `"none"` | Quantization format. Accepted values: `none`, `fp8`, `int8`, `int4`, `gguf_q4`, `gguf_q8`. | | `preferred_engine` | string | `""` | Override engine for this model (e.g., `"vllm"`). Takes priority over `engine.default`. | -| `provider` | string | `""` | Model provider hint: `local`, `openai`, `anthropic`, `google`. Used by the Cloud engine to route API calls. | +| `provider` | string | `""` | Model provider hint: `local`, `openai`, `anthropic`, `google`, `minimax`. Used by the Cloud engine to route API calls. | **Generation default fields** (overridable per-call): @@ -972,6 +972,7 @@ OpenJarvis respects the following environment variables: | `OPENAI_API_KEY` | API key for OpenAI cloud inference. Required for the `cloud` engine with OpenAI models. | | `ANTHROPIC_API_KEY` | API key for Anthropic cloud inference. Required for the `cloud` engine with Claude models. | | `GOOGLE_API_KEY` | API key for Google Gemini inference. Required for the `google` engine. | +| `MINIMAX_API_KEY` | API key for MiniMax cloud inference. Required for the `cloud` engine with MiniMax models (MiniMax-M2.7, MiniMax-M2.7-highspeed, MiniMax-M2.5, MiniMax-M2.5-highspeed). | | `TAVILY_API_KEY` | API key for the Tavily web search tool. Required for the `web_search` tool. | ## Next Steps diff --git a/src/openjarvis/engine/cloud.py b/src/openjarvis/engine/cloud.py index 8181a4b7..e5324ab6 100644 --- a/src/openjarvis/engine/cloud.py +++ b/src/openjarvis/engine/cloud.py @@ -1,4 +1,4 @@ -"""Cloud inference engine — OpenAI, Anthropic, and Google API backends.""" +"""Cloud inference engine — OpenAI, Anthropic, Google, and MiniMax API backends.""" from __future__ import annotations @@ -38,6 +38,10 @@ PRICING: Dict[str, tuple[float, float]] = { "gemini-3.1-flash-lite-preview": (0.30, 2.50), "gemini-3-flash-preview": (0.50, 3.00), "claude-haiku-4-5-20251001": (1.00, 5.00), + "MiniMax-M2.7": (0.30, 1.20), + "MiniMax-M2.7-highspeed": (0.60, 2.40), + "MiniMax-M2.5": (0.30, 1.20), + "MiniMax-M2.5-highspeed": (0.60, 2.40), } # Well-known model IDs per provider @@ -62,6 +66,12 @@ _GOOGLE_MODELS = [ "gemini-3.1-flash-lite-preview", "gemini-3-flash-preview", ] +_MINIMAX_MODELS = [ + "MiniMax-M2.7", + "MiniMax-M2.7-highspeed", + "MiniMax-M2.5", + "MiniMax-M2.5-highspeed", +] # OpenRouter models — prefixed with "openrouter/" so they can be identified _OPENROUTER_POPULAR = [ @@ -76,6 +86,10 @@ _OPENROUTER_POPULAR = [ ] +def _is_minimax_model(model: str) -> bool: + return model.lower().startswith("minimax") + + def _is_openrouter_model(model: str) -> bool: return model.startswith("openrouter/") @@ -170,7 +184,7 @@ def _convert_tools_to_google( @EngineRegistry.register("cloud") class CloudEngine(InferenceEngine): - """Cloud inference via OpenAI, Anthropic, and Google SDKs.""" + """Cloud inference via OpenAI, Anthropic, Google, and MiniMax SDKs.""" engine_id = "cloud" @@ -179,6 +193,7 @@ class CloudEngine(InferenceEngine): self._anthropic_client: Any = None self._google_client: Any = None self._openrouter_client: Any = None + self._minimax_client: Any = None self._init_clients() def _init_clients(self) -> None: @@ -214,6 +229,16 @@ class CloudEngine(InferenceEngine): ) except ImportError: pass + minimax_key = os.environ.get("MINIMAX_API_KEY") + if minimax_key: + try: + import openai + self._minimax_client = openai.OpenAI( + base_url="https://api.minimax.io/v1", + api_key=minimax_key, + ) + except ImportError: + pass def _generate_openai( self, @@ -626,6 +651,59 @@ class CloudEngine(InferenceEngine): "ttft": elapsed, } + def _generate_minimax( + self, + messages: Sequence[Message], + *, + model: str, + temperature: float, + max_tokens: int, + **kwargs: Any, + ) -> Dict[str, Any]: + if self._minimax_client is None: + raise EngineConnectionError( + "MiniMax client not available — set MINIMAX_API_KEY" + ) + # MiniMax requires temperature in (0.0, 1.0]; clamp zero + temperature = max(temperature, 0.01) + temperature = min(temperature, 1.0) + kwargs.pop("response_format", None) + create_kwargs: Dict[str, Any] = { + "model": model, + "messages": messages_to_dicts(messages), + "max_tokens": max_tokens, + "temperature": temperature, + } + t0 = time.monotonic() + resp = self._minimax_client.chat.completions.create(**create_kwargs) + elapsed = time.monotonic() - t0 + choice = resp.choices[0] + usage = resp.usage + prompt_tokens = usage.prompt_tokens if usage else 0 + completion_tokens = usage.completion_tokens if usage else 0 + result: Dict[str, Any] = { + "content": choice.message.content or "", + "usage": { + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "total_tokens": (usage.total_tokens if usage else 0), + }, + "model": resp.model, + "finish_reason": choice.finish_reason or "stop", + "cost_usd": estimate_cost(model, prompt_tokens, completion_tokens), + "ttft": elapsed, + } + if hasattr(choice.message, "tool_calls") and choice.message.tool_calls: + result["tool_calls"] = [ + { + "id": tc.id, + "name": tc.function.name, + "arguments": tc.function.arguments, + } + for tc in choice.message.tool_calls + ] + return result + def generate( self, messages: Sequence[Message], @@ -643,6 +721,8 @@ class CloudEngine(InferenceEngine): ) if _is_openrouter_model(model): return self._generate_openrouter(messages, **kw) + if _is_minimax_model(model): + return self._generate_minimax(messages, **kw) if _is_anthropic_model(model): return self._generate_anthropic(messages, **kw) if _is_google_model(model): @@ -669,6 +749,11 @@ class CloudEngine(InferenceEngine): messages, **kw ): yield token + elif _is_minimax_model(model): + async for token in self._stream_minimax( + messages, **kw + ): + yield token elif _is_anthropic_model(model): async for token in self._stream_anthropic( messages, **kw @@ -804,6 +889,32 @@ class CloudEngine(InferenceEngine): if delta and delta.content: yield delta.content + async def _stream_minimax( + self, + messages: Sequence[Message], + *, + model: str, + temperature: float, + max_tokens: int, + **kwargs: Any, + ) -> AsyncIterator[str]: + if self._minimax_client is None: + raise EngineConnectionError("MiniMax client not available") + temperature = max(temperature, 0.01) + temperature = min(temperature, 1.0) + create_kwargs: Dict[str, Any] = { + "model": model, + "messages": messages_to_dicts(messages), + "max_tokens": max_tokens, + "temperature": temperature, + "stream": True, + } + resp = self._minimax_client.chat.completions.create(**create_kwargs) + for chunk in resp: + delta = chunk.choices[0].delta if chunk.choices else None + if delta and delta.content: + yield delta.content + def list_models(self) -> List[str]: models: List[str] = [] if self._openai_client is not None: @@ -814,6 +925,8 @@ class CloudEngine(InferenceEngine): models.extend(_GOOGLE_MODELS) if self._openrouter_client is not None: models.extend(_OPENROUTER_POPULAR) + if self._minimax_client is not None: + models.extend(_MINIMAX_MODELS) return models def health(self) -> bool: @@ -822,6 +935,7 @@ class CloudEngine(InferenceEngine): or self._anthropic_client is not None or self._google_client is not None or self._openrouter_client is not None + or self._minimax_client is not None ) def close(self) -> None: @@ -839,6 +953,10 @@ class CloudEngine(InferenceEngine): if hasattr(self._openrouter_client, "close"): self._openrouter_client.close() self._openrouter_client = None + if self._minimax_client is not None: + if hasattr(self._minimax_client, "close"): + self._minimax_client.close() + self._minimax_client = None __all__ = ["CloudEngine", "PRICING", "_annotate_anthropic_cache", "estimate_cost"] diff --git a/src/openjarvis/intelligence/model_catalog.py b/src/openjarvis/intelligence/model_catalog.py index 2dfcf65d..d77c1740 100644 --- a/src/openjarvis/intelligence/model_catalog.py +++ b/src/openjarvis/intelligence/model_catalog.py @@ -707,6 +707,69 @@ BUILTIN_MODELS: List[ModelSpec] = [ }, ), # ----------------------------------------------------------------------- + # Cloud models — MiniMax + # ----------------------------------------------------------------------- + ModelSpec( + model_id="MiniMax-M2.7", + name="MiniMax M2.7", + parameter_count_b=0.0, + context_length=204800, + supported_engines=("cloud",), + provider="minimax", + requires_api_key=True, + metadata={ + "architecture": "proprietary", + "pricing_input": 0.30, + "pricing_output": 1.20, + "url": "https://www.minimax.io", + }, + ), + ModelSpec( + model_id="MiniMax-M2.7-highspeed", + name="MiniMax M2.7 Highspeed", + parameter_count_b=0.0, + context_length=204800, + supported_engines=("cloud",), + provider="minimax", + requires_api_key=True, + metadata={ + "architecture": "proprietary", + "pricing_input": 0.60, + "pricing_output": 2.40, + "url": "https://www.minimax.io", + }, + ), + ModelSpec( + model_id="MiniMax-M2.5", + name="MiniMax M2.5", + parameter_count_b=0.0, + context_length=204800, + supported_engines=("cloud",), + provider="minimax", + requires_api_key=True, + metadata={ + "architecture": "proprietary", + "pricing_input": 0.30, + "pricing_output": 1.20, + "url": "https://www.minimax.io", + }, + ), + ModelSpec( + model_id="MiniMax-M2.5-highspeed", + name="MiniMax M2.5 Highspeed", + parameter_count_b=0.0, + context_length=204800, + supported_engines=("cloud",), + provider="minimax", + requires_api_key=True, + metadata={ + "architecture": "proprietary", + "pricing_input": 0.60, + "pricing_output": 2.40, + "url": "https://www.minimax.io", + }, + ), + # ----------------------------------------------------------------------- # Cloud models — Google # ----------------------------------------------------------------------- ModelSpec( diff --git a/tests/engine/test_cloud_minimax.py b/tests/engine/test_cloud_minimax.py new file mode 100644 index 00000000..5d1ef5d9 --- /dev/null +++ b/tests/engine/test_cloud_minimax.py @@ -0,0 +1,331 @@ +"""Tests for MiniMax cloud provider support.""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest import mock + +import pytest + +from openjarvis.core.registry import EngineRegistry +from openjarvis.core.types import Message, Role +from openjarvis.engine._base import EngineConnectionError +from openjarvis.engine.cloud import ( + _MINIMAX_MODELS, + PRICING, + CloudEngine, + _is_minimax_model, + estimate_cost, +) + + +def _make_cloud_engine(monkeypatch: pytest.MonkeyPatch) -> CloudEngine: + """Create a CloudEngine with all API keys cleared.""" + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + monkeypatch.delenv("GEMINI_API_KEY", raising=False) + monkeypatch.delenv("GOOGLE_API_KEY", raising=False) + monkeypatch.delenv("OPENROUTER_API_KEY", raising=False) + monkeypatch.delenv("MINIMAX_API_KEY", raising=False) + if not EngineRegistry.contains("cloud"): + EngineRegistry.register_value("cloud", CloudEngine) + return CloudEngine() + + +def _fake_minimax_response( + content: str = "Hello from MiniMax!", + model: str = "MiniMax-M2.5", + prompt_tokens: int = 10, + completion_tokens: int = 5, + tool_calls: list | None = None, +) -> SimpleNamespace: + usage = SimpleNamespace( + prompt_tokens=prompt_tokens, + completion_tokens=completion_tokens, + total_tokens=prompt_tokens + completion_tokens, + ) + message = SimpleNamespace(content=content, tool_calls=tool_calls) + choice = SimpleNamespace(message=message, finish_reason="stop") + return SimpleNamespace(choices=[choice], usage=usage, model=model) + + +# --------------------------------------------------------------------------- +# Routing tests +# --------------------------------------------------------------------------- + + +class TestMiniMaxRouting: + def test_is_minimax_model(self) -> None: + assert _is_minimax_model("MiniMax-M2.7") is True + assert _is_minimax_model("MiniMax-M2.7-highspeed") is True + assert _is_minimax_model("MiniMax-M2.5") is True + assert _is_minimax_model("MiniMax-M2.5-highspeed") is True + assert _is_minimax_model("minimax-m2.7") is True + assert _is_minimax_model("gpt-4o") is False + assert _is_minimax_model("claude-opus-4-6") is False + assert _is_minimax_model("gemini-3-pro") is False + + def test_minimax_models_list(self) -> None: + assert "MiniMax-M2.7" in _MINIMAX_MODELS + assert "MiniMax-M2.7-highspeed" in _MINIMAX_MODELS + assert "MiniMax-M2.5" in _MINIMAX_MODELS + assert "MiniMax-M2.5-highspeed" in _MINIMAX_MODELS + + def test_m27_is_first_in_list(self) -> None: + assert _MINIMAX_MODELS[0] == "MiniMax-M2.7" + assert _MINIMAX_MODELS[1] == "MiniMax-M2.7-highspeed" + + +# --------------------------------------------------------------------------- +# Init tests +# --------------------------------------------------------------------------- + + +class TestMiniMaxInit: + def test_init_with_api_key(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key") + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + fake_openai = mock.MagicMock() + with mock.patch.dict("sys.modules", {"openai": fake_openai}): + if not EngineRegistry.contains("cloud"): + EngineRegistry.register_value("cloud", CloudEngine) + engine = CloudEngine() + assert engine._minimax_client is not None + + def test_health_with_minimax_key(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("MINIMAX_API_KEY", "test-minimax-key") + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + fake_openai = mock.MagicMock() + with mock.patch.dict("sys.modules", {"openai": fake_openai}): + engine = CloudEngine() + assert engine.health() is True + + def test_no_minimax_key_no_client(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + assert engine._minimax_client is None + + +# --------------------------------------------------------------------------- +# Generate tests +# --------------------------------------------------------------------------- + + +class TestMiniMaxGenerate: + def test_m27_generate(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = _fake_minimax_response( + content="I am MiniMax M2.7", model="MiniMax-M2.7" + ) + engine._minimax_client = fake_client + + result = engine.generate( + [Message(role=Role.USER, content="Hi")], model="MiniMax-M2.7" + ) + assert result["content"] == "I am MiniMax M2.7" + assert result["model"] == "MiniMax-M2.7" + assert result["usage"]["prompt_tokens"] == 10 + assert result["usage"]["completion_tokens"] == 5 + + def test_m27_highspeed_generate(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = _fake_minimax_response( + content="I am MiniMax M2.7 Highspeed", model="MiniMax-M2.7-highspeed" + ) + engine._minimax_client = fake_client + + result = engine.generate( + [Message(role=Role.USER, content="Hi")], model="MiniMax-M2.7-highspeed" + ) + assert result["content"] == "I am MiniMax M2.7 Highspeed" + assert result["model"] == "MiniMax-M2.7-highspeed" + + def test_m25_generate(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = _fake_minimax_response( + content="I am MiniMax M2.5", model="MiniMax-M2.5" + ) + engine._minimax_client = fake_client + + result = engine.generate( + [Message(role=Role.USER, content="Hi")], model="MiniMax-M2.5" + ) + assert result["content"] == "I am MiniMax M2.5" + assert result["model"] == "MiniMax-M2.5" + assert result["usage"]["prompt_tokens"] == 10 + assert result["usage"]["completion_tokens"] == 5 + + def test_m25_highspeed_generate(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = _fake_minimax_response( + content="I am MiniMax M2.5 Highspeed", model="MiniMax-M2.5-highspeed" + ) + engine._minimax_client = fake_client + + result = engine.generate( + [Message(role=Role.USER, content="Hi")], model="MiniMax-M2.5-highspeed" + ) + assert result["content"] == "I am MiniMax M2.5 Highspeed" + assert result["model"] == "MiniMax-M2.5-highspeed" + + def test_temperature_clamped_above_zero( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """MiniMax requires temperature in (0.0, 1.0]; verify zero is clamped.""" + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = _fake_minimax_response() + engine._minimax_client = fake_client + + engine.generate( + [Message(role=Role.USER, content="Hi")], + model="MiniMax-M2.5", + temperature=0.0, + ) + call_kwargs = fake_client.chat.completions.create.call_args + actual_temp = call_kwargs.kwargs.get("temperature") or call_kwargs[1].get( + "temperature" + ) + assert actual_temp >= 0.01, "Temperature should be clamped above zero" + + def test_temperature_clamped_at_max( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """MiniMax requires temperature <= 1.0; verify high values are clamped.""" + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = _fake_minimax_response() + engine._minimax_client = fake_client + + engine.generate( + [Message(role=Role.USER, content="Hi")], + model="MiniMax-M2.5", + temperature=2.0, + ) + call_kwargs = fake_client.chat.completions.create.call_args + actual_temp = call_kwargs.kwargs.get("temperature") or call_kwargs[1].get( + "temperature" + ) + assert actual_temp <= 1.0, "Temperature should be clamped at 1.0" + + def test_tool_calls_extraction(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + fake_tool_call = SimpleNamespace( + id="call_minimax_123", + type="function", + function=SimpleNamespace(name="search", arguments='{"q":"test"}'), + ) + fake_resp = _fake_minimax_response(content="", model="MiniMax-M2.5") + fake_resp.choices[0].message.tool_calls = [fake_tool_call] + + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = fake_resp + engine._minimax_client = fake_client + + result = engine.generate( + [Message(role=Role.USER, content="Search")], model="MiniMax-M2.5" + ) + assert "tool_calls" in result + assert len(result["tool_calls"]) == 1 + tc = result["tool_calls"][0] + assert tc["id"] == "call_minimax_123" + assert tc["name"] == "search" + + def test_no_tool_calls_when_absent( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + fake_client.chat.completions.create.return_value = _fake_minimax_response( + content="Just text" + ) + engine._minimax_client = fake_client + + result = engine.generate( + [Message(role=Role.USER, content="Hi")], model="MiniMax-M2.5" + ) + assert "tool_calls" not in result + + def test_no_client_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + assert engine._minimax_client is None + + with pytest.raises( + EngineConnectionError, match="MiniMax client not available" + ): + engine.generate( + [Message(role=Role.USER, content="Hi")], model="MiniMax-M2.5" + ) + + +# --------------------------------------------------------------------------- +# Model discovery tests +# --------------------------------------------------------------------------- + + +class TestMiniMaxModelDiscovery: + def test_list_models_includes_minimax( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + engine = _make_cloud_engine(monkeypatch) + engine._minimax_client = mock.MagicMock() + models = engine.list_models() + for m in _MINIMAX_MODELS: + assert m in models + + def test_only_minimax_client(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + engine._minimax_client = mock.MagicMock() + models = engine.list_models() + assert set(models) == set(_MINIMAX_MODELS) + + +# --------------------------------------------------------------------------- +# Pricing tests +# --------------------------------------------------------------------------- + + +class TestMiniMaxPricing: + def test_minimax_models_in_pricing(self) -> None: + assert "MiniMax-M2.7" in PRICING + assert "MiniMax-M2.7-highspeed" in PRICING + assert "MiniMax-M2.5" in PRICING + assert "MiniMax-M2.5-highspeed" in PRICING + + def test_minimax_m27_cost_estimate(self) -> None: + # MiniMax-M2.7: $0.30/M in, $1.20/M out + cost = estimate_cost("MiniMax-M2.7", 1_000_000, 1_000_000) + assert cost == pytest.approx(1.50) + + def test_minimax_m27_highspeed_cost_estimate(self) -> None: + # MiniMax-M2.7-highspeed: $0.60/M in, $2.40/M out + cost = estimate_cost("MiniMax-M2.7-highspeed", 1_000_000, 1_000_000) + assert cost == pytest.approx(3.00) + + def test_minimax_m25_cost_estimate(self) -> None: + # MiniMax-M2.5: $0.30/M in, $1.20/M out + cost = estimate_cost("MiniMax-M2.5", 1_000_000, 1_000_000) + assert cost == pytest.approx(1.50) + + def test_zero_tokens_zero_cost(self) -> None: + assert estimate_cost("MiniMax-M2.7", 0, 0) == 0.0 + + +# --------------------------------------------------------------------------- +# Close tests +# --------------------------------------------------------------------------- + + +class TestMiniMaxClose: + def test_close_minimax_client(self, monkeypatch: pytest.MonkeyPatch) -> None: + engine = _make_cloud_engine(monkeypatch) + fake_client = mock.MagicMock() + engine._minimax_client = fake_client + engine.close() + assert engine._minimax_client is None + fake_client.close.assert_called_once() diff --git a/tests/integration/test_minimax_cloud.py b/tests/integration/test_minimax_cloud.py new file mode 100644 index 00000000..1cf0cec7 --- /dev/null +++ b/tests/integration/test_minimax_cloud.py @@ -0,0 +1,63 @@ +"""Integration test for MiniMax cloud provider. + +Requires MINIMAX_API_KEY environment variable to be set. +Run with: pytest tests/integration/test_minimax_cloud.py -v +""" + +from __future__ import annotations + +import os + +import pytest + +from openjarvis.core.registry import EngineRegistry +from openjarvis.core.types import Message, Role +from openjarvis.engine.cloud import CloudEngine + +_MINIMAX_KEY = os.environ.get("MINIMAX_API_KEY", "") +_skip_no_key = pytest.mark.skipif( + not _MINIMAX_KEY, + reason="MINIMAX_API_KEY not set", +) + + +@_skip_no_key +class TestMiniMaxCloudIntegration: + """Live integration tests against MiniMax Cloud API.""" + + @pytest.fixture() + def engine(self, monkeypatch: pytest.MonkeyPatch) -> CloudEngine: + monkeypatch.setenv("MINIMAX_API_KEY", _MINIMAX_KEY) + monkeypatch.delenv("OPENAI_API_KEY", raising=False) + monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) + monkeypatch.delenv("GEMINI_API_KEY", raising=False) + monkeypatch.delenv("GOOGLE_API_KEY", raising=False) + monkeypatch.delenv("OPENROUTER_API_KEY", raising=False) + if not EngineRegistry.contains("cloud"): + EngineRegistry.register_value("cloud", CloudEngine) + return CloudEngine() + + def test_m27_highspeed_basic_chat(self, engine: CloudEngine) -> None: + """Send a simple message via M2.7-highspeed and verify non-empty response.""" + result = engine.generate( + [Message(role=Role.USER, content="Reply with exactly: hello world")], + model="MiniMax-M2.7-highspeed", + temperature=0.01, + max_tokens=32, + ) + assert result["content"], "Expected non-empty content" + assert result["usage"]["prompt_tokens"] > 0 + assert result["usage"]["completion_tokens"] > 0 + assert result["finish_reason"] in ("stop", "length") + + def test_m27_highspeed_health(self, engine: CloudEngine) -> None: + """Engine health should be True when MINIMAX_API_KEY is set.""" + assert engine.health() is True + + def test_m27_highspeed_list_models(self, engine: CloudEngine) -> None: + """MiniMax models should appear in list_models.""" + models = engine.list_models() + assert "MiniMax-M2.7" in models + assert "MiniMax-M2.7-highspeed" in models + assert "MiniMax-M2.5" in models + assert "MiniMax-M2.5-highspeed" in models From c2756964a706072a46fc7383cdd9402d69ef7127 Mon Sep 17 00:00:00 2001 From: Prathap P <66202489+Prathap-P@users.noreply.github.com> Date: Sat, 21 Mar 2026 07:05:18 +0530 Subject: [PATCH 30/92] =?UTF-8?q?fix(channels):=20wire=20channel=E2=86=92a?= =?UTF-8?q?gent=20handler=20and=20fix=20Telegram=20send=20pipeline=20(#94)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(channels): wire channel→agent handler and fix Telegram send pipeline * format code * add supported tests --- pyproject.toml | 1 + src/openjarvis/channels/telegram.py | 20 +- src/openjarvis/cli/serve.py | 14 + src/openjarvis/sessions/session.py | 2 +- src/openjarvis/system.py | 79 ++++ tests/a2a/test_a2a.py | 141 ++++--- tests/agents/test_base_agent.py | 10 +- tests/agents/test_budget.py | 6 +- tests/agents/test_claude_code.py | 162 ++++---- tests/agents/test_continuation.py | 52 +-- tests/agents/test_executor_error_detail.py | 4 + tests/agents/test_learning_integration.py | 30 +- tests/agents/test_loop_guard.py | 59 +-- tests/agents/test_loop_guard_warn.py | 12 +- tests/agents/test_manager_recover.py | 1 + tests/agents/test_monitor_operative.py | 4 +- tests/agents/test_native_openhands.py | 53 +-- tests/agents/test_native_react.py | 83 ++-- tests/agents/test_orchestrator.py | 106 ++++-- tests/agents/test_rlm.py | 17 +- tests/agents/test_simple.py | 5 +- tests/agents/test_stall.py | 55 ++- tests/agents/test_suggest_action.py | 1 + tests/agents/test_trace_recording.py | 32 +- tests/bench/test_energy.py | 5 +- tests/bench/test_latency.py | 8 +- tests/channels/test_channel_config.py | 4 +- tests/channels/test_channel_registry.py | 24 +- tests/channels/test_channels_phase21.py | 146 +++++--- tests/channels/test_email_channel.py | 17 +- tests/channels/test_google_chat.py | 73 +++- tests/channels/test_irc_channel.py | 28 +- tests/channels/test_signal_channel.py | 22 +- tests/channels/test_stubs.py | 8 +- tests/channels/test_telegram.py | 122 ++++++ tests/channels/test_whatsapp.py | 22 +- tests/channels/test_whatsapp_baileys.py | 49 +-- tests/cli/test_add_cmd.py | 9 +- tests/cli/test_agent_cmd.py | 15 +- tests/cli/test_ask_agent.py | 29 +- tests/cli/test_ask_context.py | 10 +- tests/cli/test_ask_e2e.py | 57 +-- tests/cli/test_ask_router.py | 35 +- tests/cli/test_bench_cmd.py | 6 +- tests/cli/test_channel_cmd.py | 9 +- tests/cli/test_cli.py | 14 +- tests/cli/test_daemon_cmd.py | 20 +- tests/cli/test_doctor_cmd.py | 32 +- tests/cli/test_doctor_labels.py | 8 +- tests/cli/test_init_guidance.py | 32 +- tests/cli/test_log_config.py | 5 +- tests/cli/test_memory_cmd.py | 28 +- tests/cli/test_model_cmd.py | 34 +- tests/cli/test_quickstart.py | 71 ++-- tests/cli/test_serve_channel_wiring.py | 249 ++++++++++++ tests/cli/test_telemetry_cmd.py | 28 +- tests/cli/test_vault_cmd.py | 3 +- tests/conftest.py | 9 +- tests/core/test_config.py | 22 +- tests/core/test_config_phase4.py | 7 +- tests/core/test_credentials.py | 1 + tests/core/test_events.py | 2 + tests/core/test_rust_bridge.py | 7 + tests/engine/test_cloud_extended.py | 125 ++++--- tests/engine/test_discovery.py | 4 +- tests/engine/test_engine_model_matrix.py | 133 ++++--- tests/engine/test_llamacpp_models.py | 5 +- tests/engine/test_mlx.py | 4 +- tests/engine/test_ollama_models.py | 20 +- tests/engine/test_openai_compat.py | 7 +- tests/engine/test_openai_compat_tools.py | 231 +++++++----- tests/engine/test_structured_output.py | 20 +- tests/engine/test_vllm_models.py | 9 +- tests/evals/scorers/test_browser_assistant.py | 8 +- tests/evals/scorers/test_checklist.py | 5 +- tests/evals/scorers/test_coding_assistant.py | 6 +- tests/evals/scorers/test_doc_qa.py | 14 +- tests/evals/scorers/test_security_scanner.py | 11 +- tests/evals/test_agentic_runner.py | 8 +- tests/evals/test_ama_bench.py | 18 +- tests/evals/test_benchmark_datasets.py | 77 +++- tests/evals/test_deepplanning.py | 9 +- tests/evals/test_display.py | 13 +- tests/evals/test_episode_mode.py | 6 +- tests/evals/test_export.py | 131 ++++--- tests/evals/test_lifelong_agent.py | 143 ++++--- tests/evals/test_loghub.py | 27 +- tests/evals/test_paperarena.py | 9 +- tests/evals/test_trackers.py | 31 +- tests/evals/test_use_case_benchmarks.py | 79 ++-- tests/hardware/test_amd.py | 20 +- tests/hardware/test_apple.py | 6 +- tests/hardware/test_hardware_profiles.py | 4 +- tests/hardware/test_nvidia.py | 12 +- tests/integration/test_agent_manager_e2e.py | 8 +- tests/integration/test_integration.py | 86 +++-- .../integration/test_integration_extended.py | 108 +++--- tests/intelligence/test_intelligence_stubs.py | 3 +- .../test_model_catalog_extended.py | 38 +- tests/intelligence/test_router.py | 12 +- tests/learning/agents/test_dspy_optimizer.py | 50 +-- tests/learning/agents/test_gepa_optimizer.py | 36 +- tests/learning/routing/test_learned_router.py | 70 ++-- tests/learning/test_feedback_collector.py | 6 +- tests/learning/test_heuristic_reward.py | 102 +++-- tests/learning/test_llm_optimizer.py | 116 +++--- tests/learning/test_multi_bench_runner.py | 4 +- tests/learning/test_optimizer_engine.py | 81 ++-- tests/learning/test_pareto.py | 8 +- tests/learning/test_personal_synthesizer.py | 96 +++-- tests/learning/test_router.py | 40 +- tests/learning/test_router_stubs.py | 7 +- tests/learning/test_routing_models.py | 22 +- tests/learning/test_search_space.py | 41 +- tests/learning/test_skill_discovery.py | 38 +- tests/learning/test_trial_runner.py | 64 +++- tests/learning/training/test_data.py | 2 +- tests/mcp/test_protocol.py | 5 +- tests/mcp/test_transport.py | 18 +- tests/memory/test_bm25.py | 10 +- tests/memory/test_colbert.py | 32 +- tests/memory/test_context.py | 22 +- tests/memory/test_faiss.py | 20 +- tests/memory/test_hybrid.py | 40 +- tests/memory/test_retrieval_quality.py | 3 +- tests/memory/test_sqlite.py | 12 +- tests/memory/test_storage_suite.py | 11 +- tests/operators/test_operators.py | 116 ++++-- tests/prompt/test_builder.py | 6 + tests/recipes/test_loader.py | 8 +- tests/sandbox/test_runner.py | 96 +++-- tests/scheduler/test_scheduler.py | 19 +- tests/scheduler/test_store.py | 4 +- tests/security/test_audit.py | 60 +-- tests/security/test_capabilities.py | 13 +- tests/security/test_credential_stripper.py | 5 + tests/security/test_guardrails.py | 13 +- tests/security/test_security_wiring.py | 1 + tests/security/test_setup_security.py | 2 + tests/security/test_severity_policy.py | 4 + tests/security/test_signing.py | 7 + tests/security/test_subprocess_sandbox.py | 15 +- tests/server/test_agent_manager_routes.py | 31 +- tests/server/test_api_routes.py | 1 + tests/server/test_channel_routes.py | 3 +- tests/server/test_middleware.py | 3 + tests/server/test_model_management.py | 45 ++- tests/server/test_models_pydantic.py | 10 +- tests/server/test_routes.py | 161 +++++--- tests/server/test_tools_endpoint.py | 4 + tests/server/test_ws_bridge.py | 11 +- tests/sessions/test_session.py | 3 +- tests/skills/test_bundled_skills.py | 10 +- tests/skills/test_skills.py | 38 +- tests/speech/test_discovery.py | 12 +- tests/telemetry/test_aggregator.py | 100 +++-- tests/telemetry/test_derived_metrics.py | 16 +- tests/telemetry/test_energy_apple.py | 5 +- tests/telemetry/test_energy_monitor.py | 168 +++++---- tests/telemetry/test_energy_nvidia.py | 40 +- tests/telemetry/test_energy_rapl.py | 45 ++- tests/telemetry/test_energy_wiring.py | 353 +++++++++++------- tests/telemetry/test_gpu_monitor.py | 82 ++-- tests/telemetry/test_instrumented_engine.py | 11 +- tests/telemetry/test_itl_metrics.py | 26 +- tests/telemetry/test_itl_new.py | 9 +- tests/telemetry/test_phase_energy.py | 22 +- tests/telemetry/test_phase_metrics_new.py | 14 +- tests/telemetry/test_session.py | 25 +- tests/telemetry/test_steady_state.py | 10 +- tests/telemetry/test_vllm_metrics.py | 1 + tests/telemetry/test_wrapper.py | 6 +- tests/templates/test_agent_templates.py | 6 +- .../test_environment.py | 8 +- .../test_grpo_trainer.py | 1 + .../test_orchestrator_structured.py | 22 +- .../test_policy_model.py | 11 +- .../test_prompt_registry.py | 4 +- .../test_sft_trainer.py | 19 +- .../test_orchestrator_learning/test_types.py | 4 +- tests/tools/test_apply_patch.py | 22 +- tests/tools/test_audio_tool.py | 4 +- tests/tools/test_browser.py | 9 +- tests/tools/test_channel_tools.py | 16 +- tests/tools/test_file_write.py | 11 +- tests/tools/test_git_tool.py | 5 +- tests/tools/test_http_request.py | 3 +- tests/tools/test_knowledge_graph.py | 28 +- tests/tools/test_retrieval.py | 17 +- tests/tools/test_storage_stubs.py | 14 +- tests/tools/test_storage_tools.py | 4 +- tests/tools/test_templates.py | 132 ++++--- tests/tools/test_tool_descriptions.py | 3 +- tests/workflow/test_workflow.py | 12 +- uv.lock | 2 + 195 files changed, 4080 insertions(+), 2536 deletions(-) create mode 100644 tests/cli/test_serve_channel_wiring.py diff --git a/pyproject.toml b/pyproject.toml index 696885aa..d6adcaba 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,6 +14,7 @@ dependencies = [ "ddgs>=9.11.4", "httpx>=0.27", "openai>=1.30", + "python-telegram-bot>=22.6", "rich>=13", "tomli>=2.0; python_version < '3.11'", ] diff --git a/src/openjarvis/channels/telegram.py b/src/openjarvis/channels/telegram.py index 7097ece5..7cfa2146 100644 --- a/src/openjarvis/channels/telegram.py +++ b/src/openjarvis/channels/telegram.py @@ -109,14 +109,13 @@ class TelegramChannel(BaseChannel): import httpx url = f"https://api.telegram.org/bot{self._token}/sendMessage" + chat_id = conversation_id or channel payload: Dict[str, Any] = { - "chat_id": channel, + "chat_id": chat_id, "text": content, } if self._parse_mode: payload["parse_mode"] = self._parse_mode - if conversation_id: - payload["reply_to_message_id"] = conversation_id resp = httpx.post(url, json=payload, timeout=10.0) if resp.status_code < 300: @@ -164,6 +163,19 @@ class TelegramChannel(BaseChannel): message_id=str(msg.message_id), conversation_id=str(msg.chat.id), ) + # Enforce allow-list when configured + if self._allowed_chat_ids: + _allowed = { + cid.strip() + for cid in self._allowed_chat_ids.split(",") + if cid.strip() + } + if cm.conversation_id not in _allowed: + logger.debug( + "Ignoring message from unlisted chat %s", + cm.conversation_id, + ) + return for handler in self._handlers: try: handler(cm) @@ -181,7 +193,7 @@ class TelegramChannel(BaseChannel): ) app.add_handler(MessageHandler(filters.TEXT, _handle_msg)) - app.run_polling(stop_signals=None) + app.run_polling(stop_signals=None, drop_pending_updates=True) except Exception: logger.debug("Telegram poll loop error", exc_info=True) self._status = ChannelStatus.ERROR diff --git a/src/openjarvis/cli/serve.py b/src/openjarvis/cli/serve.py index d4b68798..bfd819a6 100644 --- a/src/openjarvis/cli/serve.py +++ b/src/openjarvis/cli/serve.py @@ -230,6 +230,20 @@ def serve( console.print(f"[yellow]Channel failed to start: {exc}[/yellow]") channel_bridge = None + # Wire channel messages → agent / engine (per-chat session isolation) + if channel_bridge is not None: + from openjarvis.system import JarvisSystem + + _wire_system = JarvisSystem( + config=config, + bus=bus, + engine=engine, + engine_key=engine_name, + model=model_name, + agent_name=agent_key or "", + ) + _wire_system.wire_channel(channel_bridge) + # Set up speech backend speech_backend = None try: diff --git a/src/openjarvis/sessions/session.py b/src/openjarvis/sessions/session.py index f546c964..2809113d 100644 --- a/src/openjarvis/sessions/session.py +++ b/src/openjarvis/sessions/session.py @@ -66,7 +66,7 @@ class SessionStore: ) -> None: self._db_path = Path(db_path) self._db_path.parent.mkdir(parents=True, exist_ok=True) - self._conn = sqlite3.connect(str(self._db_path)) + self._conn = sqlite3.connect(str(self._db_path), check_same_thread=False) self._max_age_hours = max_age_hours self._consolidation_threshold = consolidation_threshold self._create_tables() diff --git a/src/openjarvis/system.py b/src/openjarvis/system.py index d1514e81..023c4671 100644 --- a/src/openjarvis/system.py +++ b/src/openjarvis/system.py @@ -269,6 +269,85 @@ class JarvisSystem: logger.warning("Failed to build tool %r: %s", name, exc) return tools + def wire_channel(self, channel_bridge: Any) -> None: + """Register a message handler on *channel_bridge* that routes every + incoming message through this system (agent or engine) and replies. + + Sessions are isolated per ``":"`` key so + each chat retains its own history. + + Parameters + ---------- + channel_bridge: + A connected :class:`~openjarvis.channels._stubs.BaseChannel` + instance whose ``on_message`` method accepts a callable. + """ + from openjarvis.agents._stubs import AgentContext + from openjarvis.core.types import Message, Role + from openjarvis.sessions.session import SessionStore + + if self.session_store is None: + from pathlib import Path + self.session_store = SessionStore( + db_path=Path(self.config.sessions.db_path).expanduser(), + max_age_hours=self.config.sessions.max_age_hours, + consolidation_threshold=self.config.sessions.consolidation_threshold, + ) + + _system = self # capture for closure + + def _on_channel_message(cm) -> None: + session_key = f"{cm.channel}:{cm.conversation_id}" + session = _system.session_store.get_or_create( + session_key, + channel=cm.channel, + channel_user_id=cm.sender, + ) + + # Rebuild prior conversation turns into AgentContext + ctx = AgentContext() + for sm in session.messages: + try: + role = Role(sm.role) + except ValueError: + role = Role.USER + ctx.conversation.add(Message(role=role, content=sm.content)) + + reply = "" + try: + if _system.agent_name and _system.agent_name != "none": + result = _system.ask( + cm.content, context=False, + agent=_system.agent_name, + ) + reply = result.get("content", "") + else: + result = _system.ask(cm.content, context=False) + reply = result.get("content", "") + except Exception: + logger.exception("Channel message handler error") + reply = "Sorry, I encountered an error processing your message." + + try: + _system.session_store.save_message( + session.session_id, "user", cm.content, channel=cm.channel, + ) + _system.session_store.save_message( + session.session_id, "assistant", reply, channel=cm.channel, + ) + except Exception: + logger.debug("Session save error", exc_info=True) + + if reply: + try: + channel_bridge.send( + cm.channel, reply, conversation_id=cm.conversation_id, + ) + except Exception: + logger.exception("Channel send error") + + channel_bridge.on_message(_on_channel_message) + def close(self) -> None: """Release resources.""" if self.scheduler and hasattr(self.scheduler, "stop"): diff --git a/tests/a2a/test_a2a.py b/tests/a2a/test_a2a.py index e3560e2d..b17175d8 100644 --- a/tests/a2a/test_a2a.py +++ b/tests/a2a/test_a2a.py @@ -78,6 +78,7 @@ class TestA2AResponse: def test_from_json(self): import json + data = json.dumps({"jsonrpc": "2.0", "result": "ok", "id": "3"}) resp = A2AResponse.from_json(data) assert resp.result == "ok" @@ -88,24 +89,28 @@ class TestA2AServer: def test_task_send(self): card = AgentCard(name="Test") server = A2AServer(card, handler=lambda x: f"Echo: {x}") - response = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/send", - "params": {"input": "Hello"}, - "id": "1", - }) + response = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/send", + "params": {"input": "Hello"}, + "id": "1", + } + ) assert response["result"]["state"] == "completed" assert "Echo: Hello" in response["result"]["output"] def test_task_send_with_message_format(self): card = AgentCard(name="Test") server = A2AServer(card, handler=lambda x: f"Got: {x}") - response = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/send", - "params": {"message": {"role": "user", "parts": [{"text": "Hi"}]}}, - "id": "1", - }) + response = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/send", + "params": {"message": {"role": "user", "parts": [{"text": "Hi"}]}}, + "id": "1", + } + ) assert response["result"]["state"] == "completed" assert "Got: Hi" in response["result"]["output"] @@ -113,62 +118,74 @@ class TestA2AServer: card = AgentCard(name="Test") server = A2AServer(card, handler=lambda x: x) # First send a task - send_resp = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/send", - "params": {"input": "test"}, - "id": "1", - }) + send_resp = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/send", + "params": {"input": "test"}, + "id": "1", + } + ) task_id = send_resp["result"]["id"] # Now get it - get_resp = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/get", - "params": {"id": task_id}, - "id": "2", - }) + get_resp = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/get", + "params": {"id": task_id}, + "id": "2", + } + ) assert get_resp["result"]["id"] == task_id def test_task_get_not_found(self): card = AgentCard(name="Test") server = A2AServer(card) - response = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/get", - "params": {"id": "nonexistent"}, - "id": "1", - }) + response = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/get", + "params": {"id": "nonexistent"}, + "id": "1", + } + ) assert "error" in response def test_task_cancel(self): card = AgentCard(name="Test") server = A2AServer(card, handler=lambda x: x) - send_resp = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/send", - "params": {"input": "test"}, - "id": "1", - }) + send_resp = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/send", + "params": {"input": "test"}, + "id": "1", + } + ) task_id = send_resp["result"]["id"] - cancel_resp = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/cancel", - "params": {"id": task_id}, - "id": "2", - }) + cancel_resp = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/cancel", + "params": {"id": task_id}, + "id": "2", + } + ) assert cancel_resp["result"]["state"] == "canceled" def test_unknown_method(self): card = AgentCard(name="Test") server = A2AServer(card) - response = server.handle_request({ - "jsonrpc": "2.0", - "method": "unknown/method", - "params": {}, - "id": "1", - }) + response = server.handle_request( + { + "jsonrpc": "2.0", + "method": "unknown/method", + "params": {}, + "id": "1", + } + ) assert "error" in response assert response["error"]["code"] == -32601 @@ -176,12 +193,14 @@ class TestA2AServer: bus = EventBus(record_history=True) card = AgentCard(name="Test") server = A2AServer(card, handler=lambda x: x, bus=bus) - server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/send", - "params": {"input": "test"}, - "id": "1", - }) + server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/send", + "params": {"input": "test"}, + "id": "1", + } + ) event_types = {e.event_type for e in bus.history} assert EventType.A2A_TASK_RECEIVED in event_types assert EventType.A2A_TASK_COMPLETED in event_types @@ -193,10 +212,12 @@ class TestA2AServer: raise ValueError("boom") server = A2AServer(card, handler=bad_handler) - response = server.handle_request({ - "jsonrpc": "2.0", - "method": "tasks/send", - "params": {"input": "test"}, - "id": "1", - }) + response = server.handle_request( + { + "jsonrpc": "2.0", + "method": "tasks/send", + "params": {"input": "test"}, + "id": "1", + } + ) assert response["result"]["state"] == "failed" diff --git a/tests/agents/test_base_agent.py b/tests/agents/test_base_agent.py index b1c0313e..cbfa6cc5 100644 --- a/tests/agents/test_base_agent.py +++ b/tests/agents/test_base_agent.py @@ -87,7 +87,11 @@ class TestBaseAgentInit: bus = EventBus() engine = MagicMock() agent = _ConcreteAgent( - engine, "m", bus=bus, temperature=0.1, max_tokens=256, + engine, + "m", + bus=bus, + temperature=0.1, + max_tokens=256, ) assert agent._temperature == 0.1 assert agent._max_tokens == 256 @@ -176,7 +180,9 @@ class TestBuildMessages: conv.add(Message(role=Role.USER, content="prev")) ctx = AgentContext(conversation=conv) messages = agent._build_messages( - "new", ctx, system_prompt="System.", + "new", + ctx, + system_prompt="System.", ) assert len(messages) == 3 assert messages[0].role == Role.SYSTEM diff --git a/tests/agents/test_budget.py b/tests/agents/test_budget.py index 77325414..e8e35349 100644 --- a/tests/agents/test_budget.py +++ b/tests/agents/test_budget.py @@ -20,8 +20,7 @@ def test_budget_exceeded_sets_status(tmp_path): assert updated["status"] == "budget_exceeded" budget_events = [ - e for e in bus.history - if e.event_type == EventType.AGENT_BUDGET_EXCEEDED + e for e in bus.history if e.event_type == EventType.AGENT_BUDGET_EXCEEDED ] assert len(budget_events) == 1 mgr.close() @@ -54,7 +53,8 @@ def test_budget_unlimited_skips_check(tmp_path): mgr.start_tick(agent["id"]) result = AgentResult( - content="done", metadata={"cost": 999.99, "tokens_used": 1000000}, + content="done", + metadata={"cost": 999.99, "tokens_used": 1000000}, ) executor._finalize_tick(agent["id"], result, error=None, duration=1.0) diff --git a/tests/agents/test_claude_code.py b/tests/agents/test_claude_code.py index ec05e729..6eabfa37 100644 --- a/tests/agents/test_claude_code.py +++ b/tests/agents/test_claude_code.py @@ -141,16 +141,19 @@ class TestClaudeCodeRun: def test_successful_run(self): agent = self._make_agent() - output = _wrap_output({ - "content": "Hello from Claude Code!", - "tool_results": [], - "metadata": {"message_count": 3}, - }) + output = _wrap_output( + { + "content": "Hello from Claude Code!", + "tool_results": [], + "metadata": {"message_count": 3}, + } + ) proc = _mock_proc(stdout=output) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -165,22 +168,25 @@ class TestClaudeCodeRun: def test_run_with_tool_results(self): agent = self._make_agent() - output = _wrap_output({ - "content": "I read the file.", - "tool_results": [ - { - "tool_name": "Read", - "content": "file contents", - "success": True, - }, - ], - "metadata": {}, - }) + output = _wrap_output( + { + "content": "I read the file.", + "tool_results": [ + { + "tool_name": "Read", + "content": "file contents", + "success": True, + }, + ], + "metadata": {}, + } + ) proc = _mock_proc(stdout=output) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -200,20 +206,24 @@ class TestClaudeCodeRun: allowed_tools=["Read", "Write"], system_prompt="Be helpful.", ) - output = _wrap_output({ - "content": "ok", - "tool_results": [], - "metadata": {}, - }) + output = _wrap_output( + { + "content": "ok", + "tool_results": [], + "metadata": {}, + } + ) proc = _mock_proc(stdout=output) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch( - "subprocess.run", return_value=proc, + "subprocess.run", + return_value=proc, ) as mock_run, ): agent.run("Do something") @@ -230,12 +240,14 @@ class TestClaudeCodeRun: def test_timeout_handling(self): agent = self._make_agent(timeout=5) exc = subprocess.TimeoutExpired( - cmd="node", timeout=5, + cmd="node", + timeout=5, ) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", side_effect=exc), @@ -249,12 +261,14 @@ class TestClaudeCodeRun: def test_nonzero_exit_code(self): agent = self._make_agent() proc = _mock_proc( - returncode=1, stderr="ENOENT: module not found", + returncode=1, + stderr="ENOENT: module not found", ) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -273,7 +287,8 @@ class TestClaudeCodeRun: with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -291,7 +306,8 @@ class TestClaudeCodeRun: with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -312,18 +328,24 @@ class TestClaudeCodeEvents: engine = MagicMock() engine.engine_id = "mock" agent = ClaudeCodeAgent( - engine, "test-model", bus=bus, api_key="k", + engine, + "test-model", + bus=bus, + api_key="k", + ) + output = _wrap_output( + { + "content": "hi", + "tool_results": [], + "metadata": {}, + } ) - output = _wrap_output({ - "content": "hi", - "tool_results": [], - "metadata": {}, - }) proc = _mock_proc(stdout=output) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -339,18 +361,24 @@ class TestClaudeCodeEvents: engine = MagicMock() engine.engine_id = "mock" agent = ClaudeCodeAgent( - engine, "test-model", bus=bus, api_key="k", + engine, + "test-model", + bus=bus, + api_key="k", + ) + output = _wrap_output( + { + "content": "hi", + "tool_results": [], + "metadata": {}, + } ) - output = _wrap_output({ - "content": "hi", - "tool_results": [], - "metadata": {}, - }) proc = _mock_proc(stdout=output) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -358,8 +386,7 @@ class TestClaudeCodeEvents: agent.run("test input") start_events = [ - e for e in bus.history - if e.event_type == EventType.AGENT_TURN_START + e for e in bus.history if e.event_type == EventType.AGENT_TURN_START ] assert len(start_events) == 1 assert start_events[0].data["agent"] == "claude_code" @@ -370,13 +397,17 @@ class TestClaudeCodeEvents: engine = MagicMock() engine.engine_id = "mock" agent = ClaudeCodeAgent( - engine, "test-model", bus=bus, api_key="k", + engine, + "test-model", + bus=bus, + api_key="k", ) proc = _mock_proc(returncode=1, stderr="error") with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), @@ -449,11 +480,7 @@ class TestParseOutput: "tool_results": [], "metadata": {}, } - stdout = ( - "some debug output\n" - + _wrap_output(payload) - + "\nmore output" - ) + stdout = "some debug output\n" + _wrap_output(payload) + "\nmore output" content, tools, meta = ClaudeCodeAgent._parse_output( stdout, ) @@ -485,7 +512,9 @@ class TestClaudeCodeDefaults: engine = MagicMock() engine.engine_id = "mock" agent = ClaudeCodeAgent( - engine, "test-model", api_key="explicit-key", + engine, + "test-model", + api_key="explicit-key", ) assert agent._api_key == "explicit-key" @@ -499,7 +528,9 @@ class TestClaudeCodeDefaults: engine = MagicMock() engine.engine_id = "mock" agent = ClaudeCodeAgent( - engine, "test-model", timeout=60, + engine, + "test-model", + timeout=60, ) assert agent._timeout == 60 @@ -507,18 +538,23 @@ class TestClaudeCodeDefaults: engine = MagicMock() engine.engine_id = "mock" agent = ClaudeCodeAgent( - engine, "test-model", api_key="k", + engine, + "test-model", + api_key="k", + ) + output = _wrap_output( + { + "content": "ok", + "tool_results": [], + "metadata": {}, + } ) - output = _wrap_output({ - "content": "ok", - "tool_results": [], - "metadata": {}, - }) proc = _mock_proc(stdout=output) with ( patch.object( - agent, "_ensure_runner", + agent, + "_ensure_runner", return_value="/fake/runner", ), patch("subprocess.run", return_value=proc), diff --git a/tests/agents/test_continuation.py b/tests/agents/test_continuation.py index 0d596384..4bd3b131 100644 --- a/tests/agents/test_continuation.py +++ b/tests/agents/test_continuation.py @@ -43,39 +43,47 @@ class ContinuationAgent(BaseAgent): class TestContinuation: def test_no_continuation_needed(self): - engine = MockEngine([ - {"content": "Hello world", "finish_reason": "stop"}, - ]) + engine = MockEngine( + [ + {"content": "Hello world", "finish_reason": "stop"}, + ] + ) agent = ContinuationAgent(engine, "test-model") result = agent.run("Hi") assert result.content == "Hello world" def test_single_continuation(self): - engine = MockEngine([ - {"content": "Part 1...", "finish_reason": "length"}, - {"content": " Part 2.", "finish_reason": "stop"}, - ]) + engine = MockEngine( + [ + {"content": "Part 1...", "finish_reason": "length"}, + {"content": " Part 2.", "finish_reason": "stop"}, + ] + ) agent = ContinuationAgent(engine, "test-model") result = agent.run("Hi") assert result.content == "Part 1... Part 2." def test_multiple_continuations(self): - engine = MockEngine([ - {"content": "A", "finish_reason": "length"}, - {"content": "B", "finish_reason": "length"}, - {"content": "C", "finish_reason": "stop"}, - ]) + engine = MockEngine( + [ + {"content": "A", "finish_reason": "length"}, + {"content": "B", "finish_reason": "length"}, + {"content": "C", "finish_reason": "stop"}, + ] + ) agent = ContinuationAgent(engine, "test-model") result = agent.run("Hi") assert result.content == "ABC" def test_max_continuations_respected(self): - engine = MockEngine([ - {"content": "A", "finish_reason": "length"}, - {"content": "B", "finish_reason": "length"}, - {"content": "C", "finish_reason": "length"}, # 3rd continuation - {"content": "D", "finish_reason": "stop"}, - ]) + engine = MockEngine( + [ + {"content": "A", "finish_reason": "length"}, + {"content": "B", "finish_reason": "length"}, + {"content": "C", "finish_reason": "length"}, # 3rd continuation + {"content": "D", "finish_reason": "stop"}, + ] + ) agent = ContinuationAgent(engine, "test-model") # Default max_continuations=2, so should stop after 2 continuations messages = agent._build_messages("Hi") @@ -84,9 +92,11 @@ class TestContinuation: assert content == "ABC" # A + B + C, but not D def test_empty_finish_reason(self): - engine = MockEngine([ - {"content": "Done", "finish_reason": ""}, - ]) + engine = MockEngine( + [ + {"content": "Done", "finish_reason": ""}, + ] + ) agent = ContinuationAgent(engine, "test-model") result = agent.run("Hi") assert result.content == "Done" diff --git a/tests/agents/test_executor_error_detail.py b/tests/agents/test_executor_error_detail.py index aa4fbd81..87aa0bed 100644 --- a/tests/agents/test_executor_error_detail.py +++ b/tests/agents/test_executor_error_detail.py @@ -1,4 +1,5 @@ """Tests for structured error_detail in executor traces.""" + from openjarvis.agents.errors import EscalateError, FatalError, RetryableError from openjarvis.agents.executor import AgentExecutor from openjarvis.core.events import EventBus @@ -6,6 +7,7 @@ from openjarvis.core.events import EventBus def test_build_error_detail_fatal(tmp_path): from openjarvis.agents.manager import AgentManager + mgr = AgentManager(db_path=str(tmp_path / "agents.db")) exe = AgentExecutor(manager=mgr, event_bus=EventBus()) error = FatalError("401 unauthorized") @@ -17,6 +19,7 @@ def test_build_error_detail_fatal(tmp_path): def test_build_error_detail_retryable(tmp_path): from openjarvis.agents.manager import AgentManager + mgr = AgentManager(db_path=str(tmp_path / "agents.db")) exe = AgentExecutor(manager=mgr, event_bus=EventBus()) error = RetryableError("connection timed out") @@ -27,6 +30,7 @@ def test_build_error_detail_retryable(tmp_path): def test_build_error_detail_escalate(tmp_path): from openjarvis.agents.manager import AgentManager + mgr = AgentManager(db_path=str(tmp_path / "agents.db")) exe = AgentExecutor(manager=mgr, event_bus=EventBus()) error = EscalateError("agent needs help") diff --git a/tests/agents/test_learning_integration.py b/tests/agents/test_learning_integration.py index a8292e35..e889d9bf 100644 --- a/tests/agents/test_learning_integration.py +++ b/tests/agents/test_learning_integration.py @@ -50,11 +50,14 @@ def test_scheduler_tracks_tick_count_for_learning(tmp_path): executor = AgentExecutor(mgr, bus) scheduler = AgentScheduler(mgr, executor, event_bus=bus) - agent = mgr.create_agent("tick-counter", config={ - "learning_enabled": True, - "learning_schedule": "every_3_ticks", - "schedule_type": "manual", - }) + agent = mgr.create_agent( + "tick-counter", + config={ + "learning_enabled": True, + "learning_schedule": "every_3_ticks", + "schedule_type": "manual", + }, + ) # Simulate 3 ticks completing for _ in range(3): @@ -62,8 +65,7 @@ def test_scheduler_tracks_tick_count_for_learning(tmp_path): # Should have triggered learning learning_events = [ - e for e in bus.history - if e.event_type == EventType.AGENT_LEARNING_STARTED + e for e in bus.history if e.event_type == EventType.AGENT_LEARNING_STARTED ] assert len(learning_events) == 1 assert learning_events[0].data["agent_id"] == agent["id"] @@ -85,17 +87,19 @@ def test_scheduler_no_learning_when_disabled(tmp_path): executor = AgentExecutor(mgr, bus) scheduler = AgentScheduler(mgr, executor, event_bus=bus) - agent = mgr.create_agent("no-learning", config={ - "learning_enabled": False, - "learning_schedule": "every_3_ticks", - }) + agent = mgr.create_agent( + "no-learning", + config={ + "learning_enabled": False, + "learning_schedule": "every_3_ticks", + }, + ) for _ in range(5): scheduler._on_tick_completed(agent["id"]) learning_events = [ - e for e in bus.history - if e.event_type == EventType.AGENT_LEARNING_STARTED + e for e in bus.history if e.event_type == EventType.AGENT_LEARNING_STARTED ] assert len(learning_events) == 0 mgr.close() diff --git a/tests/agents/test_loop_guard.py b/tests/agents/test_loop_guard.py index 69ea11f6..3b15f3c0 100644 --- a/tests/agents/test_loop_guard.py +++ b/tests/agents/test_loop_guard.py @@ -8,6 +8,7 @@ from openjarvis.core.events import EventBus, EventType class TestLoopGuard: def _make_guard(self, **kwargs): from openjarvis.agents.loop_guard import LoopGuard, LoopGuardConfig + kwargs.setdefault("warn_before_block", False) config = LoopGuardConfig(**kwargs) bus = EventBus(record_history=True) @@ -55,8 +56,7 @@ class TestLoopGuard: guard.check_call("x", '{"a": 1}') guard.check_call("x", '{"a": 1}') events = [ - e for e in bus.history - if e.event_type == EventType.LOOP_GUARD_TRIGGERED + e for e in bus.history if e.event_type == EventType.LOOP_GUARD_TRIGGERED ] assert len(events) == 1 @@ -70,6 +70,7 @@ class TestLoopGuard: def test_context_compression_no_overflow(self): from openjarvis.core.types import Message, Role + guard, _ = self._make_guard(max_context_messages=100) messages = [Message(role=Role.USER, content=f"msg {i}") for i in range(10)] result = guard.compress_context(messages) @@ -77,42 +78,43 @@ class TestLoopGuard: def test_context_compression_with_overflow(self): from openjarvis.core.types import Message, Role + guard, _ = self._make_guard(max_context_messages=10) - messages = [ - Message(role=Role.SYSTEM, content="sys"), - ] + [ - Message(role=Role.USER, content=f"msg {i}") - for i in range(50) - ] + [ - Message(role=Role.TOOL, content=f"result {i}", tool_call_id=f"t{i}") - for i in range(50) - ] + messages = ( + [ + Message(role=Role.SYSTEM, content="sys"), + ] + + [Message(role=Role.USER, content=f"msg {i}") for i in range(50)] + + [ + Message(role=Role.TOOL, content=f"result {i}", tool_call_id=f"t{i}") + for i in range(50) + ] + ) result = guard.compress_context(messages) assert len(result) <= 10 def test_context_compression_stage4_uses_current_state(self): """Stage 4 should derive from compressed state.""" from openjarvis.core.types import Message, Role + guard, _ = self._make_guard(max_context_messages=5) - messages = [ - Message(role=Role.SYSTEM, content="sys"), - ] + [ - Message(role=Role.USER, content=f"msg {i}") - for i in range(100) - ] + [ - Message( - role=Role.TOOL, - content=f"result {i}", - tool_call_id=f"t{i}", - ) - for i in range(100) - ] + messages = ( + [ + Message(role=Role.SYSTEM, content="sys"), + ] + + [Message(role=Role.USER, content=f"msg {i}") for i in range(100)] + + [ + Message( + role=Role.TOOL, + content=f"result {i}", + tool_call_id=f"t{i}", + ) + for i in range(100) + ] + ) result = guard.compress_context(messages) assert len(result) == 5 - system_count = sum( - 1 for m in result - if getattr(m, 'role', None) == 'system' - ) + system_count = sum(1 for m in result if getattr(m, "role", None) == "system") assert system_count == 1 def test_check_response_returns_unblocked(self): @@ -122,6 +124,7 @@ class TestLoopGuard: def test_disabled_loop_guard(self): from openjarvis.agents.loop_guard import LoopGuard, LoopGuardConfig + config = LoopGuardConfig(enabled=False) guard = LoopGuard(config) # Even though we'd normally block, disabled guard shouldn't diff --git a/tests/agents/test_loop_guard_warn.py b/tests/agents/test_loop_guard_warn.py index 534082ad..24495766 100644 --- a/tests/agents/test_loop_guard_warn.py +++ b/tests/agents/test_loop_guard_warn.py @@ -7,7 +7,9 @@ from openjarvis.agents.loop_guard import LoopGuard, LoopGuardConfig, LoopVerdict def test_warn_before_block_first_cycle_warns(): config = LoopGuardConfig( - enabled=True, max_identical_calls=2, warn_before_block=True, + enabled=True, + max_identical_calls=2, + warn_before_block=True, ) guard = LoopGuard(config) # Simulate the Rust backend blocking on the second identical call @@ -25,7 +27,9 @@ def test_warn_before_block_first_cycle_warns(): def test_warn_before_block_second_cycle_blocks(): config = LoopGuardConfig( - enabled=True, max_identical_calls=2, warn_before_block=True, + enabled=True, + max_identical_calls=2, + warn_before_block=True, ) guard = LoopGuard(config) mock_rust = MagicMock() @@ -47,7 +51,9 @@ def test_warn_before_block_second_cycle_blocks(): def test_default_behavior_unchanged(): config = LoopGuardConfig( - enabled=True, max_identical_calls=2, warn_before_block=False, + enabled=True, + max_identical_calls=2, + warn_before_block=False, ) guard = LoopGuard(config) mock_rust = MagicMock() diff --git a/tests/agents/test_manager_recover.py b/tests/agents/test_manager_recover.py index 81a3defd..dfe1c830 100644 --- a/tests/agents/test_manager_recover.py +++ b/tests/agents/test_manager_recover.py @@ -1,4 +1,5 @@ """Tests for AgentManager.recover_agent() always resetting status.""" + import pytest from openjarvis.agents.manager import AgentManager diff --git a/tests/agents/test_monitor_operative.py b/tests/agents/test_monitor_operative.py index 48820969..dedb0826 100644 --- a/tests/agents/test_monitor_operative.py +++ b/tests/agents/test_monitor_operative.py @@ -23,6 +23,7 @@ class TestMonitorOperativeAgent: # Import triggers registration; re-register after autouse fixture # clears the registry (same pattern as test_monitor.py) import openjarvis.agents.monitor_operative # noqa: F401 + if not AgentRegistry.contains("monitor_operative"): AgentRegistry.register_value("monitor_operative", MonitorOperativeAgent) assert AgentRegistry.contains("monitor_operative") @@ -46,7 +47,8 @@ class TestMonitorOperativeAgent: def test_custom_strategies(self) -> None: engine = _make_engine() agent = MonitorOperativeAgent( - engine, "test-model", + engine, + "test-model", memory_extraction="scratchpad", observation_compression="none", retrieval_strategy="keyword", diff --git a/tests/agents/test_native_openhands.py b/tests/agents/test_native_openhands.py index 8a37f880..c4fdc36b 100644 --- a/tests/agents/test_native_openhands.py +++ b/tests/agents/test_native_openhands.py @@ -38,6 +38,7 @@ class _CodeInterpreterStub(BaseTool): # Simple simulation: if it contains print(), capture the content if "print(" in code: import re + match = re.search(r"print\((.+?)\)", code) if match: try: @@ -134,13 +135,12 @@ class TestNativeOpenHandsAgent: engine = MagicMock() engine.engine_id = "mock" engine.generate.side_effect = [ - _engine_response( - "Let me calculate:\n```python\nprint(2+2)\n```" - ), + _engine_response("Let me calculate:\n```python\nprint(2+2)\n```"), _engine_response("The result is 4."), ] agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub()], ) result = agent.run("What is 2+2?") @@ -159,7 +159,8 @@ class TestNativeOpenHandsAgent: _engine_response("First was 2, second was 9."), ] agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub()], ) result = agent.run("Two calculations") @@ -175,7 +176,8 @@ class TestNativeOpenHandsAgent: "More code:\n```python\nprint('hello')\n```" ) agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub()], max_turns=3, ) @@ -205,7 +207,8 @@ class TestNativeOpenHandsAgent: _engine_response("Done."), ] agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub()], bus=bus, ) @@ -238,13 +241,12 @@ class TestNativeOpenHandsAgent: engine = MagicMock() engine.engine_id = "mock" engine.generate.side_effect = [ - _engine_response( - 'Action: calculator\nAction Input: {"expression": "7*6"}' - ), + _engine_response('Action: calculator\nAction Input: {"expression": "7*6"}'), _engine_response("The answer is 42."), ] agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], ) result = agent.run("What is 7 times 6?") @@ -285,7 +287,8 @@ class TestNativeOpenHandsAgent: engine.engine_id = "mock" engine.generate.return_value = _engine_response("Ok") agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub(), _CalculatorStub()], ) agent.run("Hello") @@ -301,7 +304,8 @@ class TestNativeOpenHandsAgent: engine.engine_id = "mock" engine.generate.return_value = _engine_response("Ok") agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub(), _CalculatorStub()], ) agent.run("Hello") @@ -321,7 +325,8 @@ class TestNativeOpenHandsAgent: "Still working:\n```python\nx = 1\n```" ) agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub()], max_turns=2, ) @@ -339,7 +344,8 @@ class TestNativeOpenHandsAgent: _engine_response("Got 42."), ] agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CodeInterpreterStub()], ) agent.run("Print 42") @@ -358,8 +364,7 @@ class TestNativeOpenHandsAgent: agent = NativeOpenHandsAgent(engine, "test-model", bus=bus) agent.run("test input") start_events = [ - e for e in bus.history - if e.event_type == EventType.AGENT_TURN_START + e for e in bus.history if e.event_type == EventType.AGENT_TURN_START ] assert len(start_events) == 1 assert start_events[0].data["agent"] == "native_openhands" @@ -389,13 +394,12 @@ class TestNativeOpenHandsAgent: engine = MagicMock() engine.engine_id = "mock" engine.generate.side_effect = [ - _engine_response( - 'calculator\n$expression=7*6' - ), + _engine_response("calculator\n$expression=7*6"), _engine_response("The answer is 42."), ] agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], ) result = agent.run("What is 7 times 6?") @@ -408,9 +412,7 @@ class TestNativeOpenHandsAgent: engine = MagicMock() engine.engine_id = "mock" engine.generate.side_effect = [ - _engine_response( - 'Action: calculator\nAction Input: {"expression": "1+1"}' - ), + _engine_response('Action: calculator\nAction Input: {"expression": "1+1"}'), _engine_response("Done."), ] # Make calculator return very long output @@ -512,7 +514,8 @@ class TestUrlExpansion: import httpx monkeypatch.setattr( - httpx, "get", + httpx, + "get", MagicMock(side_effect=Exception("Connection error")), ) text, expanded = NativeOpenHandsAgent._expand_urls( diff --git a/tests/agents/test_native_react.py b/tests/agents/test_native_react.py index 8a3797b1..6ce065da 100644 --- a/tests/agents/test_native_react.py +++ b/tests/agents/test_native_react.py @@ -111,8 +111,8 @@ class TestNativeReActParsing: def test_parse_thought_action(self): parse = self._parser() text = ( - 'Thought: I need to calculate 2+2.\n' - 'Action: calculator\n' + "Thought: I need to calculate 2+2.\n" + "Action: calculator\n" 'Action Input: {"expression": "2+2"}' ) result = parse(text) @@ -157,8 +157,8 @@ class TestNativeReActParsing: def test_parse_case_insensitive_thought_action(self): parse = self._parser() text = ( - 'thought: I need to calculate 2+2.\n' - 'action: calculator\n' + "thought: I need to calculate 2+2.\n" + "action: calculator\n" 'action input: {"expression": "2+2"}' ) result = parse(text) @@ -199,18 +199,18 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'Thought: I need to calculate.\n' - 'Action: calculator\n' + "Thought: I need to calculate.\n" + "Action: calculator\n" 'Action Input: {"expression": "2+2"}' ), - _engine_response( - "Thought: The result is 4.\nFinal Answer: 4" - ), + _engine_response("Thought: The result is 4.\nFinal Answer: 4"), ] bus = EventBus(record_history=True) agent = NativeReActAgent( - engine, "test-model", - tools=[_CalculatorStub()], bus=bus, + engine, + "test-model", + tools=[_CalculatorStub()], + bus=bus, ) result = agent.run("What is 2+2?") assert result.content == "4" @@ -225,7 +225,7 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'Thought: Calculate.\nAction: calculator\n' + "Thought: Calculate.\nAction: calculator\n" 'Action Input: {"expression": "3*7"}' ), _engine_response("Thought: Done.\nFinal Answer: 21"), @@ -241,21 +241,22 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'Thought: Step 1.\nAction: calculator\n' + "Thought: Step 1.\nAction: calculator\n" 'Action Input: {"expression": "1+1"}' ), _engine_response( - 'Thought: Step 2.\nAction: calculator\n' + "Thought: Step 2.\nAction: calculator\n" 'Action Input: {"expression": "2+2"}' ), _engine_response( - 'Thought: Step 3.\nAction: think\n' + "Thought: Step 3.\nAction: think\n" 'Action Input: {"thought": "combining results"}' ), _engine_response("Thought: All done.\nFinal Answer: Complete."), ] agent = NativeReActAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub(), _ThinkStub()], ) result = agent.run("Multi step") @@ -268,11 +269,12 @@ class TestNativeReActAgent: engine = MagicMock() engine.engine_id = "mock" engine.generate.return_value = _engine_response( - 'Thought: Keep going.\nAction: calculator\n' + "Thought: Keep going.\nAction: calculator\n" 'Action Input: {"expression": "1+1"}' ) agent = NativeReActAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], max_turns=3, ) @@ -287,12 +289,10 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'Thought: Use a tool.\nAction: nonexistent\n' - 'Action Input: {}' + "Thought: Use a tool.\nAction: nonexistent\nAction Input: {}" ), _engine_response( - "Thought: Error occurred.\n" - "Final Answer: Could not run tool." + "Thought: Error occurred.\nFinal Answer: Could not run tool." ), ] agent = NativeReActAgent(engine, "test-model", tools=[_CalculatorStub()]) @@ -322,14 +322,16 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'Thought: Calc.\nAction: calculator\n' + "Thought: Calc.\nAction: calculator\n" 'Action Input: {"expression": "1+1"}' ), _engine_response("Thought: Done.\nFinal Answer: 2"), ] agent = NativeReActAgent( - engine, "test-model", - tools=[_CalculatorStub()], bus=bus, + engine, + "test-model", + tools=[_CalculatorStub()], + bus=bus, ) agent.run("Calc") event_types = [e.event_type for e in bus.history] @@ -365,7 +367,7 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'Thought: Let me reason.\nAction: think\n' + "Thought: Let me reason.\nAction: think\n" 'Action Input: {"thought": "The user wants a greeting"}' ), _engine_response("Thought: Now I know.\nFinal Answer: Greetings!"), @@ -403,7 +405,7 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'Thought: Calc.\nAction: calculator\n' + "Thought: Calc.\nAction: calculator\n" 'Action Input: {"expression": "5+5"}' ), _engine_response("Thought: Got it.\nFinal Answer: 10"), @@ -427,7 +429,8 @@ class TestNativeReActAgent: "Thought: Done.\nFinal Answer: ok" ) agent = NativeReActAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub(), _ThinkStub()], ) agent.run("Hello") @@ -455,11 +458,11 @@ class TestNativeReActAgent: engine = MagicMock() engine.engine_id = "mock" engine.generate.return_value = _engine_response( - 'Thought: Go.\nAction: calculator\n' - 'Action Input: {"expression": "1"}' + 'Thought: Go.\nAction: calculator\nAction Input: {"expression": "1"}' ) agent = NativeReActAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], max_turns=1, ) @@ -478,14 +481,12 @@ class TestNativeReActAgent: agent = NativeReActAgent(engine, "test-model", bus=bus) agent.run("test input") start_events = [ - e for e in bus.history - if e.event_type == EventType.AGENT_TURN_START + e for e in bus.history if e.event_type == EventType.AGENT_TURN_START ] assert len(start_events) == 1 assert start_events[0].data["agent"] == "native_react" assert start_events[0].data["input"] == "test input" - def test_system_prompt_enriched_descriptions(self): """System prompt should include parameter schemas, not just names.""" engine = MagicMock() @@ -494,7 +495,8 @@ class TestNativeReActAgent: "Thought: Done.\nFinal Answer: ok" ) agent = NativeReActAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub(), _ThinkStub()], ) agent.run("Hello") @@ -514,16 +516,15 @@ class TestNativeReActAgent: engine.engine_id = "mock" engine.generate.side_effect = [ _engine_response( - 'thought: I need to calculate.\n' - 'action: calculator\n' + "thought: I need to calculate.\n" + "action: calculator\n" 'action input: {"expression": "2+2"}' ), - _engine_response( - "thought: The result is 4.\nfinal answer: 4" - ), + _engine_response("thought: The result is 4.\nfinal answer: 4"), ] agent = NativeReActAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], ) result = agent.run("What is 2+2?") diff --git a/tests/agents/test_orchestrator.py b/tests/agents/test_orchestrator.py index e27f5027..f3c753d0 100644 --- a/tests/agents/test_orchestrator.py +++ b/tests/agents/test_orchestrator.py @@ -114,11 +114,13 @@ def _make_engine_multi_tool() -> MagicMock: "content": "", "tool_calls": [ { - "id": "call_1", "name": "calculator", + "id": "call_1", + "name": "calculator", "arguments": '{"expression":"2+2"}', }, { - "id": "call_2", "name": "think", + "id": "call_2", + "name": "think", "arguments": '{"thought":"thinking..."}', }, ], @@ -158,7 +160,9 @@ class TestOrchestratorAgent: def test_single_tool_call(self): engine = _make_engine_with_tool_call() agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) result = agent.run("What is 2+2?") assert result.content == "The answer is 4." @@ -170,7 +174,8 @@ class TestOrchestratorAgent: def test_multiple_tool_calls_same_turn(self): engine = _make_engine_multi_tool() agent = OrchestratorAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub(), _ThinkStub()], ) result = agent.run("Think and calculate.") @@ -193,7 +198,9 @@ class TestOrchestratorAgent: def test_tools_passed_to_engine(self): engine = _make_engine_no_tools() agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) agent.run("Hello") call_kwargs = engine.generate.call_args[1] @@ -221,7 +228,8 @@ class TestOrchestratorAgent: "finish_reason": "tool_calls", } agent = OrchestratorAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], max_turns=3, ) @@ -236,7 +244,9 @@ class TestOrchestratorAgent: final_content="Handled.", ) agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) result = agent.run("Use unknown tool") assert result.content == "Handled." @@ -281,7 +291,10 @@ class TestOrchestratorAgent: bus = EventBus(record_history=True) engine = _make_engine_with_tool_call() agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], bus=bus, + engine, + "test-model", + tools=[_CalculatorStub()], + bus=bus, ) agent.run("Calc 2+2") event_types = [e.event_type for e in bus.history] @@ -292,7 +305,9 @@ class TestOrchestratorAgent: """After tool call, messages include assistant + tool messages.""" engine = _make_engine_with_tool_call() agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) agent.run("What is 2+2?") # Second call should include accumulated messages @@ -305,7 +320,9 @@ class TestOrchestratorAgent: def test_tool_message_has_tool_call_id(self): engine = _make_engine_with_tool_call(tool_call_id="abc123") agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) agent.run("What is 2+2?") second_call = engine.generate.call_args_list[1] @@ -317,7 +334,9 @@ class TestOrchestratorAgent: def test_no_bus_works(self): engine = _make_engine_with_tool_call() agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) result = agent.run("What is 2+2?") assert result.content == "The answer is 4." @@ -335,10 +354,13 @@ class TestOrchestratorAgent: engine.generate.side_effect = [ { "content": "", - "tool_calls": [{ - "id": "c1", "name": "calculator", - "arguments": '{"expression":"2+2"}', - }], + "tool_calls": [ + { + "id": "c1", + "name": "calculator", + "arguments": '{"expression":"2+2"}', + } + ], "usage": { "prompt_tokens": 5, "completion_tokens": 3, @@ -349,10 +371,13 @@ class TestOrchestratorAgent: }, { "content": "", - "tool_calls": [{ - "id": "c2", "name": "calculator", - "arguments": '{"expression":"4*3"}', - }], + "tool_calls": [ + { + "id": "c2", + "name": "calculator", + "arguments": '{"expression":"4*3"}', + } + ], "usage": { "prompt_tokens": 15, "completion_tokens": 3, @@ -373,7 +398,9 @@ class TestOrchestratorAgent: }, ] agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) result = agent.run("Calculate") assert result.turns == 3 @@ -384,7 +411,9 @@ class TestOrchestratorAgent: def test_tool_result_latency_tracked(self): engine = _make_engine_with_tool_call() agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], ) result = agent.run("What is 2+2?") assert result.tool_results[0].latency_seconds >= 0 @@ -403,7 +432,8 @@ class TestOrchestratorAgent: "finish_reason": "tool_calls", } agent = OrchestratorAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], max_turns=1, ) @@ -433,7 +463,8 @@ class TestOrchestratorAgent: "finish_reason": "tool_calls", } agent = OrchestratorAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], max_turns=2, ) @@ -456,7 +487,9 @@ class TestOrchestratorStructuredMode: "finish_reason": "stop", } agent = OrchestratorAgent( - engine, "test-model", mode="structured", + engine, + "test-model", + mode="structured", ) result = agent.run("What is the capital of France?") assert result.content == "Paris" @@ -471,7 +504,7 @@ class TestOrchestratorStructuredMode: { "content": ( "THOUGHT: Need to calculate.\n" - 'TOOL: calculator\n' + "TOOL: calculator\n" 'INPUT: {"expression":"2+2"}' ), "usage": { @@ -483,10 +516,7 @@ class TestOrchestratorStructuredMode: "finish_reason": "stop", }, { - "content": ( - "THOUGHT: Got 4.\n" - "FINAL_ANSWER: The answer is 4." - ), + "content": ("THOUGHT: Got 4.\nFINAL_ANSWER: The answer is 4."), "usage": { "prompt_tokens": 20, "completion_tokens": 10, @@ -497,7 +527,8 @@ class TestOrchestratorStructuredMode: }, ] agent = OrchestratorAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], mode="structured", ) @@ -519,7 +550,8 @@ class TestOrchestratorStructuredMode: "finish_reason": "stop", } agent = OrchestratorAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub()], mode="structured", ) @@ -591,7 +623,10 @@ class TestOrchestratorParallelTools: ] agent = OrchestratorAgent( - engine, "test-model", tools=[_SlowTool()], parallel_tools=True, + engine, + "test-model", + tools=[_SlowTool()], + parallel_tools=True, ) t0 = time.time() result = agent.run("Do things") @@ -610,7 +645,8 @@ class TestOrchestratorParallelTools: """parallel_tools=False runs tools sequentially.""" engine = _make_engine_multi_tool() agent = OrchestratorAgent( - engine, "test-model", + engine, + "test-model", tools=[_CalculatorStub(), _ThinkStub()], parallel_tools=False, ) @@ -622,7 +658,9 @@ class TestOrchestratorParallelTools: """Single tool call should not use parallel path even if parallel_tools=True.""" engine = _make_engine_with_tool_call() agent = OrchestratorAgent( - engine, "test-model", tools=[_CalculatorStub()], + engine, + "test-model", + tools=[_CalculatorStub()], parallel_tools=True, ) result = agent.run("What is 2+2?") diff --git a/tests/agents/test_rlm.py b/tests/agents/test_rlm.py index 1e1c280d..132e6246 100644 --- a/tests/agents/test_rlm.py +++ b/tests/agents/test_rlm.py @@ -197,9 +197,7 @@ class TestRLMSubLMCalls: engine.generate.side_effect = [ { "content": ( - "```python\n" - "result = llm_query('What is 2+2?')\n" - "FINAL(result)\n```" + "```python\nresult = llm_query('What is 2+2?')\nFINAL(result)\n```" ), "usage": { "prompt_tokens": 5, @@ -236,10 +234,7 @@ class TestRLMMultiTurn: engine.generate.side_effect = [ # Turn 1: code that sets a variable { - "content": ( - "```python\n" - "x = 10\nprint(f'x = {x}')\n```" - ), + "content": ("```python\nx = 10\nprint(f'x = {x}')\n```"), "usage": { "prompt_tokens": 5, "completion_tokens": 10, @@ -250,9 +245,7 @@ class TestRLMMultiTurn: }, # Turn 2: code that uses the variable and terminates { - "content": ( - "```python\ny = x * 2\nFINAL(y)\n```" - ), + "content": ("```python\ny = x * 2\nFINAL(y)\n```"), "usage": { "prompt_tokens": 20, "completion_tokens": 10, @@ -335,9 +328,7 @@ class TestRLMSubLMWithTools: # Root LM: code that calls llm_query { "content": ( - "```python\n" - "result = llm_query('Calculate 2+2')\n" - "FINAL(result)\n```" + "```python\nresult = llm_query('Calculate 2+2')\nFINAL(result)\n```" ), "usage": { "prompt_tokens": 5, diff --git a/tests/agents/test_simple.py b/tests/agents/test_simple.py index fecf2fdd..e3195d3b 100644 --- a/tests/agents/test_simple.py +++ b/tests/agents/test_simple.py @@ -91,10 +91,7 @@ class TestSimpleAgent: agent = SimpleAgent(engine, "test-model", bus=bus) agent.run("test input") evts = bus.history - start = [ - e for e in evts - if e.event_type == EventType.AGENT_TURN_START - ][0] + start = [e for e in evts if e.event_type == EventType.AGENT_TURN_START][0] assert start.data["agent"] == "simple" assert start.data["input"] == "test input" diff --git a/tests/agents/test_stall.py b/tests/agents/test_stall.py index 3d9f59d4..18e932b1 100644 --- a/tests/agents/test_stall.py +++ b/tests/agents/test_stall.py @@ -18,10 +18,13 @@ def test_activity_tracking_updates_last_activity_at(tmp_path): agent = mgr.create_agent("stall-test") def fake_invoke(agent_dict): - bus.publish(EventType.TOOL_CALL_START, { - "agent": agent_dict["id"], - "tool": "web_search", - }) + bus.publish( + EventType.TOOL_CALL_START, + { + "agent": agent_dict["id"], + "tool": "web_search", + }, + ) return AgentResult(content="done", metadata={}) with patch.object(executor, "_invoke_agent", side_effect=fake_invoke): @@ -44,10 +47,13 @@ def test_activity_tracking_filters_by_agent_id(tmp_path): def fake_invoke(agent_dict): # Emit event for agent_b while agent_a is executing - bus.publish(EventType.TOOL_CALL_START, { - "agent": agent_b["id"], - "tool": "web_search", - }) + bus.publish( + EventType.TOOL_CALL_START, + { + "agent": agent_b["id"], + "tool": "web_search", + }, + ) return AgentResult(content="done", metadata={}) with patch.object(executor, "_invoke_agent", side_effect=fake_invoke): @@ -67,10 +73,13 @@ def test_reconcile_detects_stalled_agent(tmp_path): scheduler = AgentScheduler(mgr, executor, event_bus=bus) - agent = mgr.create_agent("stall-me", config={ - "timeout_seconds": 10, - "max_stall_retries": 3, - }) + agent = mgr.create_agent( + "stall-me", + config={ + "timeout_seconds": 10, + "max_stall_retries": 3, + }, + ) mgr.update_agent(agent["id"], status="running", last_activity_at=time.time() - 30) scheduler._reconcile() @@ -79,8 +88,7 @@ def test_reconcile_detects_stalled_agent(tmp_path): assert updated["stall_retries"] == 1 stall_events = [ - e for e in bus.history - if e.event_type == EventType.AGENT_STALL_DETECTED + e for e in bus.history if e.event_type == EventType.AGENT_STALL_DETECTED ] assert len(stall_events) == 1 mgr.close() @@ -114,12 +122,19 @@ def test_reconcile_retries_exhausted_sets_error(tmp_path): scheduler = AgentScheduler(mgr, executor, event_bus=bus) - agent = mgr.create_agent("exhausted", config={ - "timeout_seconds": 10, - "max_stall_retries": 2, - }) - mgr.update_agent(agent["id"], status="running", - last_activity_at=time.time() - 30, stall_retries=2) + agent = mgr.create_agent( + "exhausted", + config={ + "timeout_seconds": 10, + "max_stall_retries": 2, + }, + ) + mgr.update_agent( + agent["id"], + status="running", + last_activity_at=time.time() - 30, + stall_retries=2, + ) scheduler._reconcile() diff --git a/tests/agents/test_suggest_action.py b/tests/agents/test_suggest_action.py index 004ed738..aa816c2d 100644 --- a/tests/agents/test_suggest_action.py +++ b/tests/agents/test_suggest_action.py @@ -1,4 +1,5 @@ """Tests for suggest_action helper.""" + from openjarvis.agents.errors import FatalError, RetryableError, suggest_action diff --git a/tests/agents/test_trace_recording.py b/tests/agents/test_trace_recording.py index 51397ab9..e1c07b68 100644 --- a/tests/agents/test_trace_recording.py +++ b/tests/agents/test_trace_recording.py @@ -21,17 +21,23 @@ def test_executor_records_trace(tmp_path): agent = mgr.create_agent("trace-test") def fake_invoke(agent_dict): - bus.publish(EventType.TOOL_CALL_START, { - "agent": agent_dict["id"], - "tool": "web_search", - "args": {"query": "test"}, - }) - bus.publish(EventType.TOOL_CALL_END, { - "agent": agent_dict["id"], - "tool": "web_search", - "result": "search results...", - "duration": 0.5, - }) + bus.publish( + EventType.TOOL_CALL_START, + { + "agent": agent_dict["id"], + "tool": "web_search", + "args": {"query": "test"}, + }, + ) + bus.publish( + EventType.TOOL_CALL_END, + { + "agent": agent_dict["id"], + "tool": "web_search", + "result": "search results...", + "duration": 0.5, + }, + ) return AgentResult(content="found it", metadata={"tokens_used": 100}) with patch.object(executor, "_invoke_agent", side_effect=fake_invoke): @@ -62,7 +68,9 @@ def test_executor_records_error_trace(tmp_path): agent = mgr.create_agent("error-trace") with patch.object( - executor, "_invoke_agent", side_effect=FatalError("boom"), + executor, + "_invoke_agent", + side_effect=FatalError("boom"), ): executor.execute_tick(agent["id"]) diff --git a/tests/bench/test_energy.py b/tests/bench/test_energy.py index 37a3cdb5..d5602b23 100644 --- a/tests/bench/test_energy.py +++ b/tests/bench/test_energy.py @@ -78,7 +78,10 @@ class TestEnergyBenchmark: b = EnergyBenchmark() result = b.run( - engine, "test-model", num_samples=3, warmup_samples=0, + engine, + "test-model", + num_samples=3, + warmup_samples=0, energy_monitor=monitor, ) diff --git a/tests/bench/test_latency.py b/tests/bench/test_latency.py index 109da0f6..ea03a3df 100644 --- a/tests/bench/test_latency.py +++ b/tests/bench/test_latency.py @@ -56,8 +56,12 @@ class TestLatencyBenchmark: b = LatencyBenchmark() result = b.run(engine, "test-model", num_samples=3) expected_keys = { - "mean_latency", "p50_latency", "p95_latency", - "min_latency", "max_latency", "std_latency", + "mean_latency", + "p50_latency", + "p95_latency", + "min_latency", + "max_latency", + "std_latency", } assert set(result.metrics.keys()) == expected_keys diff --git a/tests/channels/test_channel_config.py b/tests/channels/test_channel_config.py index 005fa09a..0e7ad81d 100644 --- a/tests/channels/test_channel_config.py +++ b/tests/channels/test_channel_config.py @@ -66,7 +66,9 @@ class TestChannelConfig: class TestTomlLoading: def _write_toml(self, content: str) -> Path: f = tempfile.NamedTemporaryFile( - mode="w", suffix=".toml", delete=False, + mode="w", + suffix=".toml", + delete=False, ) f.write(content) f.flush() diff --git a/tests/channels/test_channel_registry.py b/tests/channels/test_channel_registry.py index d8e992c4..70f8a1df 100644 --- a/tests/channels/test_channel_registry.py +++ b/tests/channels/test_channel_registry.py @@ -25,8 +25,12 @@ class TestChannelRegistry: pass def send( - self, channel, content, - *, conversation_id="", metadata=None, + self, + channel, + content, + *, + conversation_id="", + metadata=None, ) -> bool: return True @@ -56,8 +60,12 @@ class TestChannelRegistry: pass def send( - self, channel, content, - *, conversation_id="", metadata=None, + self, + channel, + content, + *, + conversation_id="", + metadata=None, ) -> bool: return True @@ -99,8 +107,12 @@ class TestChannelRegistry: pass def send( - self, channel, content, - *, conversation_id="", metadata=None, + self, + channel, + content, + *, + conversation_id="", + metadata=None, ) -> bool: return True diff --git a/tests/channels/test_channels_phase21.py b/tests/channels/test_channels_phase21.py index 822a17ef..fe8d46ce 100644 --- a/tests/channels/test_channels_phase21.py +++ b/tests/channels/test_channels_phase21.py @@ -153,10 +153,13 @@ class TestChannelImportError: class TestLineChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "LINE_CHANNEL_ACCESS_TOKEN": "env-tok", - "LINE_CHANNEL_SECRET": "env-sec", - }): + with patch.dict( + "os.environ", + { + "LINE_CHANNEL_ACCESS_TOKEN": "env-tok", + "LINE_CHANNEL_SECRET": "env-sec", + }, + ): ch = LineChannel() assert ch._channel_access_token == "env-tok" assert ch._channel_secret == "env-sec" @@ -164,10 +167,13 @@ class TestLineChannel: class TestViberChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "VIBER_AUTH_TOKEN": "env-tok", - "VIBER_BOT_NAME": "TestBot", - }): + with patch.dict( + "os.environ", + { + "VIBER_AUTH_TOKEN": "env-tok", + "VIBER_BOT_NAME": "TestBot", + }, + ): ch = ViberChannel() assert ch._auth_token == "env-tok" assert ch._name == "TestBot" @@ -175,21 +181,27 @@ class TestViberChannel: class TestMessengerChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "MESSENGER_ACCESS_TOKEN": "env-tok", - }): + with patch.dict( + "os.environ", + { + "MESSENGER_ACCESS_TOKEN": "env-tok", + }, + ): ch = MessengerChannel() assert ch._access_token == "env-tok" class TestRedditChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "REDDIT_CLIENT_ID": "cid", - "REDDIT_CLIENT_SECRET": "csec", - "REDDIT_USERNAME": "user", - "REDDIT_PASSWORD": "pass", - }): + with patch.dict( + "os.environ", + { + "REDDIT_CLIENT_ID": "cid", + "REDDIT_CLIENT_SECRET": "csec", + "REDDIT_USERNAME": "user", + "REDDIT_PASSWORD": "pass", + }, + ): ch = RedditChannel() assert ch._client_id == "cid" assert ch._client_secret == "csec" @@ -199,10 +211,13 @@ class TestRedditChannel: class TestMastodonChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "MASTODON_API_BASE_URL": "https://m.social", - "MASTODON_ACCESS_TOKEN": "env-tok", - }): + with patch.dict( + "os.environ", + { + "MASTODON_API_BASE_URL": "https://m.social", + "MASTODON_ACCESS_TOKEN": "env-tok", + }, + ): ch = MastodonChannel() assert ch._api_base_url == "https://m.social" assert ch._access_token == "env-tok" @@ -210,12 +225,15 @@ class TestMastodonChannel: class TestXMPPChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "XMPP_JID": "bot@example.com", - "XMPP_PASSWORD": "pass", - "XMPP_SERVER": "xmpp.example.com", - "XMPP_PORT": "5223", - }): + with patch.dict( + "os.environ", + { + "XMPP_JID": "bot@example.com", + "XMPP_PASSWORD": "pass", + "XMPP_SERVER": "xmpp.example.com", + "XMPP_PORT": "5223", + }, + ): ch = XMPPChannel() assert ch._jid == "bot@example.com" assert ch._password == "pass" @@ -225,22 +243,28 @@ class TestXMPPChannel: class TestRocketChatChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "ROCKETCHAT_URL": "https://rc.example.com", - "ROCKETCHAT_USER": "bot", - "ROCKETCHAT_PASSWORD": "pass", - }): + with patch.dict( + "os.environ", + { + "ROCKETCHAT_URL": "https://rc.example.com", + "ROCKETCHAT_USER": "bot", + "ROCKETCHAT_PASSWORD": "pass", + }, + ): ch = RocketChatChannel() assert ch._url == "https://rc.example.com" assert ch._user == "bot" assert ch._password == "pass" def test_token_auth_env(self): - with patch.dict("os.environ", { - "ROCKETCHAT_URL": "https://rc.example.com", - "ROCKETCHAT_AUTH_TOKEN": "tok", - "ROCKETCHAT_USER_ID": "uid", - }): + with patch.dict( + "os.environ", + { + "ROCKETCHAT_URL": "https://rc.example.com", + "ROCKETCHAT_AUTH_TOKEN": "tok", + "ROCKETCHAT_USER_ID": "uid", + }, + ): ch = RocketChatChannel() assert ch._auth_token == "tok" assert ch._user_id == "uid" @@ -248,32 +272,41 @@ class TestRocketChatChannel: class TestZulipChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "ZULIP_EMAIL": "bot@zulip.com", - "ZULIP_API_KEY": "key", - "ZULIP_SITE": "https://z.com", - }): + with patch.dict( + "os.environ", + { + "ZULIP_EMAIL": "bot@zulip.com", + "ZULIP_API_KEY": "key", + "ZULIP_SITE": "https://z.com", + }, + ): ch = ZulipChannel() assert ch._email == "bot@zulip.com" assert ch._api_key == "key" assert ch._site == "https://z.com" def test_zuliprc_env(self): - with patch.dict("os.environ", { - "ZULIP_RC": "/path/to/zuliprc", - }): + with patch.dict( + "os.environ", + { + "ZULIP_RC": "/path/to/zuliprc", + }, + ): ch = ZulipChannel() assert ch._zuliprc == "/path/to/zuliprc" class TestTwitchChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "TWITCH_ACCESS_TOKEN": "env-tok", - "TWITCH_CLIENT_ID": "env-cid", - "TWITCH_NICK": "env-nick", - "TWITCH_CHANNELS": "chan1,chan2", - }): + with patch.dict( + "os.environ", + { + "TWITCH_ACCESS_TOKEN": "env-tok", + "TWITCH_CLIENT_ID": "env-cid", + "TWITCH_NICK": "env-nick", + "TWITCH_CHANNELS": "chan1,chan2", + }, + ): ch = TwitchChannel() assert ch._access_token == "env-tok" assert ch._client_id == "env-cid" @@ -283,10 +316,13 @@ class TestTwitchChannel: class TestNostrChannel: def test_env_fallback(self): - with patch.dict("os.environ", { - "NOSTR_PRIVATE_KEY": "aa" * 32, - "NOSTR_RELAYS": "wss://r1.example.com,wss://r2.example.com", - }): + with patch.dict( + "os.environ", + { + "NOSTR_PRIVATE_KEY": "aa" * 32, + "NOSTR_RELAYS": "wss://r1.example.com,wss://r2.example.com", + }, + ): ch = NostrChannel() assert ch._private_key == "aa" * 32 assert len(ch._relays) == 2 diff --git a/tests/channels/test_email_channel.py b/tests/channels/test_email_channel.py index fc515b98..0aa5e8b6 100644 --- a/tests/channels/test_email_channel.py +++ b/tests/channels/test_email_channel.py @@ -26,7 +26,8 @@ class TestRegistration: def test_channel_id(self): ch = EmailChannel( - smtp_host="smtp.example.com", username="user@example.com", + smtp_host="smtp.example.com", + username="user@example.com", ) assert ch.channel_id == "email" @@ -62,10 +63,13 @@ class TestInit: assert ch._use_tls is False def test_env_var_fallback(self): - with patch.dict(os.environ, { - "EMAIL_USERNAME": "env@example.com", - "EMAIL_PASSWORD": "env-pass", - }): + with patch.dict( + os.environ, + { + "EMAIL_USERNAME": "env@example.com", + "EMAIL_PASSWORD": "env-pass", + }, + ): ch = EmailChannel() assert ch._username == "env@example.com" assert ch._password == "env-pass" @@ -206,7 +210,8 @@ class TestOnMessage: class TestDisconnect: def test_disconnect(self): ch = EmailChannel( - smtp_host="smtp.example.com", username="user@example.com", + smtp_host="smtp.example.com", + username="user@example.com", ) ch._status = ChannelStatus.CONNECTED ch.disconnect() diff --git a/tests/channels/test_google_chat.py b/tests/channels/test_google_chat.py index b04ec86b..082cb7a1 100644 --- a/tests/channels/test_google_chat.py +++ b/tests/channels/test_google_chat.py @@ -25,7 +25,9 @@ class TestRegistration: assert ChannelRegistry.contains("google_chat") def test_channel_id(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) assert ch.channel_id == "google_chat" @@ -36,27 +38,48 @@ class TestInit: assert ch._status == ChannelStatus.DISCONNECTED def test_constructor_url(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") - assert ch._webhook_url == "https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) + assert ( + ch._webhook_url + == "https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) def test_env_var_fallback(self): - with patch.dict(os.environ, { - "GOOGLE_CHAT_WEBHOOK_URL": "https://chat.googleapis.com/v1/spaces/env/messages?key=env", - }): + with patch.dict( + os.environ, + { + "GOOGLE_CHAT_WEBHOOK_URL": "https://chat.googleapis.com/v1/spaces/env/messages?key=env", + }, + ): ch = GoogleChatChannel() - assert ch._webhook_url == "https://chat.googleapis.com/v1/spaces/env/messages?key=env" + assert ( + ch._webhook_url + == "https://chat.googleapis.com/v1/spaces/env/messages?key=env" + ) def test_constructor_overrides_env(self): - with patch.dict(os.environ, { - "GOOGLE_CHAT_WEBHOOK_URL": "https://chat.googleapis.com/v1/spaces/env/messages?key=env", - }): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/explicit/messages?key=explicit") - assert ch._webhook_url == "https://chat.googleapis.com/v1/spaces/explicit/messages?key=explicit" + with patch.dict( + os.environ, + { + "GOOGLE_CHAT_WEBHOOK_URL": "https://chat.googleapis.com/v1/spaces/env/messages?key=env", + }, + ): + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/explicit/messages?key=explicit" + ) + assert ( + ch._webhook_url + == "https://chat.googleapis.com/v1/spaces/explicit/messages?key=explicit" + ) class TestSend: def test_send_success(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) mock_response = MagicMock() mock_response.status_code = 200 @@ -67,7 +90,9 @@ class TestSend: mock_post.assert_called_once() def test_send_failure(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) mock_response = MagicMock() mock_response.status_code = 400 @@ -78,7 +103,9 @@ class TestSend: assert result is False def test_send_exception(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) with patch("httpx.post", side_effect=ConnectionError("refused")): result = ch.send("space", "Hello!") @@ -108,13 +135,17 @@ class TestSend: class TestListChannels: def test_list_channels(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) assert ch.list_channels() == ["google_chat"] class TestStatus: def test_disconnected_initially(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) assert ch.status() == ChannelStatus.DISCONNECTED def test_no_url_connect_error(self): @@ -125,7 +156,9 @@ class TestStatus: class TestOnMessage: def test_on_message(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) handler = MagicMock() ch.on_message(handler) assert handler in ch._handlers @@ -133,7 +166,9 @@ class TestOnMessage: class TestDisconnect: def test_disconnect(self): - ch = GoogleChatChannel(webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy") + ch = GoogleChatChannel( + webhook_url="https://chat.googleapis.com/v1/spaces/xxx/messages?key=yyy" + ) ch._status = ChannelStatus.CONNECTED ch.disconnect() assert ch.status() == ChannelStatus.DISCONNECTED diff --git a/tests/channels/test_irc_channel.py b/tests/channels/test_irc_channel.py index cc0392c8..e754f682 100644 --- a/tests/channels/test_irc_channel.py +++ b/tests/channels/test_irc_channel.py @@ -45,12 +45,15 @@ class TestInit: assert ch._password == "pass123" def test_env_var_fallback(self): - with patch.dict(os.environ, { - "IRC_SERVER": "irc.env.com", - "IRC_NICK": "envbot", - "IRC_PASSWORD": "envpass", - "IRC_PORT": "6697", - }): + with patch.dict( + os.environ, + { + "IRC_SERVER": "irc.env.com", + "IRC_NICK": "envbot", + "IRC_PASSWORD": "envpass", + "IRC_PORT": "6697", + }, + ): ch = IRCChannel() assert ch._server == "irc.env.com" assert ch._nick == "envbot" @@ -58,11 +61,14 @@ class TestInit: assert ch._port == 6697 def test_constructor_overrides_env(self): - with patch.dict(os.environ, { - "IRC_SERVER": "irc.env.com", - "IRC_NICK": "envbot", - "IRC_PASSWORD": "envpass", - }): + with patch.dict( + os.environ, + { + "IRC_SERVER": "irc.env.com", + "IRC_NICK": "envbot", + "IRC_PASSWORD": "envpass", + }, + ): ch = IRCChannel( server="irc.explicit.com", nick="explicit", diff --git a/tests/channels/test_signal_channel.py b/tests/channels/test_signal_channel.py index 38f6303d..34c83606 100644 --- a/tests/channels/test_signal_channel.py +++ b/tests/channels/test_signal_channel.py @@ -42,19 +42,25 @@ class TestInit: assert ch._phone_number == "+1234567890" def test_env_var_fallback(self): - with patch.dict(os.environ, { - "SIGNAL_API_URL": "http://env-signal:8080", - "SIGNAL_PHONE_NUMBER": "+9876543210", - }): + with patch.dict( + os.environ, + { + "SIGNAL_API_URL": "http://env-signal:8080", + "SIGNAL_PHONE_NUMBER": "+9876543210", + }, + ): ch = SignalChannel() assert ch._api_url == "http://env-signal:8080" assert ch._phone_number == "+9876543210" def test_constructor_overrides_env(self): - with patch.dict(os.environ, { - "SIGNAL_API_URL": "http://env-signal:8080", - "SIGNAL_PHONE_NUMBER": "+9876543210", - }): + with patch.dict( + os.environ, + { + "SIGNAL_API_URL": "http://env-signal:8080", + "SIGNAL_PHONE_NUMBER": "+9876543210", + }, + ): ch = SignalChannel( api_url="http://explicit:8080", phone_number="+1111111111", diff --git a/tests/channels/test_stubs.py b/tests/channels/test_stubs.py index bdc487bd..18e92184 100644 --- a/tests/channels/test_stubs.py +++ b/tests/channels/test_stubs.py @@ -95,8 +95,12 @@ class TestBaseChannel: pass def send( - self, channel, content, - *, conversation_id="", metadata=None, + self, + channel, + content, + *, + conversation_id="", + metadata=None, ) -> bool: return True diff --git a/tests/channels/test_telegram.py b/tests/channels/test_telegram.py index ae01dfb8..0e348060 100644 --- a/tests/channels/test_telegram.py +++ b/tests/channels/test_telegram.py @@ -140,3 +140,125 @@ class TestDisconnect: ch._status = ChannelStatus.CONNECTED ch.disconnect() assert ch.status() == ChannelStatus.DISCONNECTED + + +class TestAllowedChatIds: + """Tests for the allowed_chat_ids enforcement in _poll_loop.""" + + def _make_update(self, chat_id: str, text: str = "hello"): + """Build a minimal fake python-telegram-bot Update object.""" + msg = MagicMock() + msg.text = text + msg.message_id = 1 + msg.from_user.id = chat_id + msg.chat.id = chat_id + update = MagicMock() + update.message = msg + return update + + def _invoke_handle_msg(self, ch: TelegramChannel, chat_id: str, text: str = "hi"): + """Simulate _poll_loop dispatching a message without starting a thread.""" + from openjarvis.channels._stubs import ChannelMessage + + cm = ChannelMessage( + channel="telegram", + sender=chat_id, + content=text, + message_id="1", + conversation_id=chat_id, + ) + # Directly exercise the allow-list logic (mirrors _handle_msg body) + if ch._allowed_chat_ids: + _allowed = { + cid.strip() for cid in ch._allowed_chat_ids.split(",") if cid.strip() + } + if cm.conversation_id not in _allowed: + return False # would return inside _handle_msg + for handler in ch._handlers: + handler(cm) + return True + + def test_no_allowlist_accepts_any(self): + """When allowed_chat_ids is empty every chat is dispatched.""" + ch = TelegramChannel(bot_token="tok", allowed_chat_ids="") + handler = MagicMock() + ch.on_message(handler) + dispatched = self._invoke_handle_msg(ch, "99999") + assert dispatched is True + handler.assert_called_once() + + def test_allowlist_passes_listed_chat(self): + """A chat ID present in the allow-list is dispatched to handlers.""" + ch = TelegramChannel(bot_token="tok", allowed_chat_ids="111,222") + handler = MagicMock() + ch.on_message(handler) + dispatched = self._invoke_handle_msg(ch, "111") + assert dispatched is True + handler.assert_called_once() + + def test_allowlist_blocks_unlisted_chat(self): + """A chat ID not in the allow-list is silently dropped (not dispatched).""" + ch = TelegramChannel(bot_token="tok", allowed_chat_ids="111,222") + handler = MagicMock() + ch.on_message(handler) + dispatched = self._invoke_handle_msg(ch, "999") + assert dispatched is False + handler.assert_not_called() + + def test_allowlist_trims_whitespace(self): + """Spaces around IDs in the allow-list are handled gracefully.""" + ch = TelegramChannel(bot_token="tok", allowed_chat_ids=" 111 , 222 ") + handler = MagicMock() + ch.on_message(handler) + dispatched = self._invoke_handle_msg(ch, "111") + assert dispatched is True + handler.assert_called_once() + + +class TestChannelAgentWiring: + """Tests for the channel → agent handler wired in serve.py.""" + + def test_on_message_handler_invoked_on_message(self): + """on_message callback registered on a channel is called when a message + arrives.""" + ch = TelegramChannel(bot_token="tok") + received = [] + ch.on_message(lambda cm: received.append(cm)) + + from openjarvis.channels._stubs import ChannelMessage + + cm = ChannelMessage( + channel="telegram", + sender="42", + content="ping", + message_id="1", + conversation_id="42", + ) + for h in ch._handlers: + h(cm) + + assert len(received) == 1 + assert received[0].content == "ping" + + def test_multiple_handlers_all_invoked(self): + """Both handlers registered via on_message are called for the same message.""" + ch = TelegramChannel(bot_token="tok") + calls_a: list = [] + calls_b: list = [] + ch.on_message(lambda cm: calls_a.append(cm)) + ch.on_message(lambda cm: calls_b.append(cm)) + + from openjarvis.channels._stubs import ChannelMessage + + cm = ChannelMessage( + channel="telegram", + sender="1", + content="x", + message_id="1", + conversation_id="1", + ) + for h in ch._handlers: + h(cm) + + assert len(calls_a) == 1 + assert len(calls_b) == 1 diff --git a/tests/channels/test_whatsapp.py b/tests/channels/test_whatsapp.py index 3b0a2a30..a4e118f5 100644 --- a/tests/channels/test_whatsapp.py +++ b/tests/channels/test_whatsapp.py @@ -42,19 +42,25 @@ class TestInit: assert ch._phone_number_id == "12345" def test_env_var_fallback(self): - with patch.dict(os.environ, { - "WHATSAPP_ACCESS_TOKEN": "env-token", - "WHATSAPP_PHONE_NUMBER_ID": "env-id", - }): + with patch.dict( + os.environ, + { + "WHATSAPP_ACCESS_TOKEN": "env-token", + "WHATSAPP_PHONE_NUMBER_ID": "env-id", + }, + ): ch = WhatsAppChannel() assert ch._token == "env-token" assert ch._phone_number_id == "env-id" def test_constructor_overrides_env(self): - with patch.dict(os.environ, { - "WHATSAPP_ACCESS_TOKEN": "env-token", - "WHATSAPP_PHONE_NUMBER_ID": "env-id", - }): + with patch.dict( + os.environ, + { + "WHATSAPP_ACCESS_TOKEN": "env-token", + "WHATSAPP_PHONE_NUMBER_ID": "env-id", + }, + ): ch = WhatsAppChannel( access_token="explicit-token", phone_number_id="explicit-id", diff --git a/tests/channels/test_whatsapp_baileys.py b/tests/channels/test_whatsapp_baileys.py index 72e64035..9d969fce 100644 --- a/tests/channels/test_whatsapp_baileys.py +++ b/tests/channels/test_whatsapp_baileys.py @@ -259,13 +259,15 @@ class TestReaderLoop: bus = EventBus(record_history=True) ch = WhatsAppBaileysChannel(bus=bus) - ch._handle_bridge_event({ - "type": "message", - "jid": "123@s.whatsapp.net", - "sender": "456@s.whatsapp.net", - "text": "Bus test", - "message_id": "msg-002", - }) + ch._handle_bridge_event( + { + "type": "message", + "jid": "123@s.whatsapp.net", + "sender": "456@s.whatsapp.net", + "text": "Bus test", + "message_id": "msg-002", + } + ) event_types = [e.event_type for e in bus.history] assert EventType.CHANNEL_MESSAGE_RECEIVED in event_types @@ -276,13 +278,16 @@ class TestReaderLoop: lines = [ json.dumps({"type": "status", "status": "connected"}) + "\n", - json.dumps({ - "type": "message", - "jid": "j", - "sender": "s", - "text": "t", - "message_id": "m", - }) + "\n", + json.dumps( + { + "type": "message", + "jid": "j", + "sender": "s", + "text": "t", + "message_id": "m", + } + ) + + "\n", ] mock_proc = MagicMock() @@ -316,11 +321,13 @@ class TestReaderLoop: ch.on_message(bad_handler) # Should not raise. - ch._handle_bridge_event({ - "type": "message", - "jid": "j", - "sender": "s", - "text": "t", - "message_id": "m", - }) + ch._handle_bridge_event( + { + "type": "message", + "jid": "j", + "sender": "s", + "text": "t", + "message_id": "m", + } + ) bad_handler.assert_called_once() diff --git a/tests/cli/test_add_cmd.py b/tests/cli/test_add_cmd.py index 1348f2cf..06b8e2d4 100644 --- a/tests/cli/test_add_cmd.py +++ b/tests/cli/test_add_cmd.py @@ -42,7 +42,8 @@ class TestAddCmd: mcp_dir = tmp_path / "mcp" with mock.patch("openjarvis.cli.add_cmd._MCP_CONFIG_DIR", mcp_dir): result = CliRunner().invoke( - add, ["github", "--key", "test_token"], + add, + ["github", "--key", "test_token"], ) assert result.exit_code == 0 @@ -55,6 +56,6 @@ class TestAddCmd: def test_mcp_templates_complete(self) -> None: required_fields = {"command", "args", "env_key", "description"} for name, tmpl in _MCP_TEMPLATES.items(): - assert required_fields.issubset( - tmpl.keys() - ), f"Template '{name}' missing fields: {required_fields - tmpl.keys()}" + assert required_fields.issubset(tmpl.keys()), ( + f"Template '{name}' missing fields: {required_fields - tmpl.keys()}" + ) diff --git a/tests/cli/test_agent_cmd.py b/tests/cli/test_agent_cmd.py index 3e0fa94b..8a71c579 100644 --- a/tests/cli/test_agent_cmd.py +++ b/tests/cli/test_agent_cmd.py @@ -80,8 +80,19 @@ class TestNewAgentCommands: result = CliRunner().invoke(cli, ["agents", "--help"]) assert result.exit_code == 0 cmds = ( - "launch", "start", "stop", "run", "status", "logs", - "daemon", "watch", "recover", "errors", "ask", "instruct", "messages", + "launch", + "start", + "stop", + "run", + "status", + "logs", + "daemon", + "watch", + "recover", + "errors", + "ask", + "instruct", + "messages", ) for cmd in cmds: assert cmd in result.output, f"Missing command: {cmd}" diff --git a/tests/cli/test_ask_agent.py b/tests/cli/test_ask_agent.py index 577aa499..622c2910 100644 --- a/tests/cli/test_ask_agent.py +++ b/tests/cli/test_ask_agent.py @@ -123,7 +123,8 @@ def agent_setup(): patch.object(_ask_mod, "get_engine", return_value=("mock", engine)), patch.object(_ask_mod, "discover_engines", return_value=[("mock", engine)]), patch.object( - _ask_mod, "discover_models", + _ask_mod, + "discover_models", return_value={"mock": ["test-model"]}, ), patch.object(_ask_mod, "register_builtin_models"), @@ -152,6 +153,7 @@ def mock_setup(): patch.object(_ask_mod, "merge_discovered_models"), ): from openjarvis.core.config import JarvisConfig + mock_cfg.return_value = JarvisConfig() mock_ge.return_value = ("mock", engine) mock_de.return_value = [("mock", engine)] @@ -175,7 +177,8 @@ class TestAskAgentOption: def test_agent_orchestrator_no_tools(self, runner, mock_setup): result = runner.invoke( - cli, ["ask", "--agent", "orchestrator", "Hello"], + cli, + ["ask", "--agent", "orchestrator", "Hello"], ) assert result.exit_code == 0 @@ -183,8 +186,11 @@ class TestAskAgentOption: result = runner.invoke( cli, [ - "ask", "--agent", "orchestrator", - "--tools", "calculator,think", + "ask", + "--agent", + "orchestrator", + "--tools", + "calculator,think", "What is 2+2?", ], ) @@ -192,7 +198,8 @@ class TestAskAgentOption: def test_agent_json_output(self, runner, mock_setup): result = runner.invoke( - cli, ["ask", "--agent", "simple", "--json", "Hello"], + cli, + ["ask", "--agent", "simple", "--json", "Hello"], ) assert result.exit_code == 0 assert '"content"' in result.output @@ -200,7 +207,8 @@ class TestAskAgentOption: def test_unknown_agent(self, runner, mock_setup): result = runner.invoke( - cli, ["ask", "--agent", "nonexistent", "Hello"], + cli, + ["ask", "--agent", "nonexistent", "Hello"], ) assert result.exit_code != 0 @@ -211,13 +219,15 @@ class TestAskAgentOption: def test_agent_simple_with_model(self, runner, mock_setup): result = runner.invoke( - cli, ["ask", "--agent", "simple", "-m", "test-model", "Hello"], + cli, + ["ask", "--agent", "simple", "-m", "test-model", "Hello"], ) assert result.exit_code == 0 def test_agent_simple_with_temperature(self, runner, mock_setup): result = runner.invoke( - cli, ["ask", "--agent", "simple", "-t", "0.1", "Hello"], + cli, + ["ask", "--agent", "simple", "-t", "0.1", "Hello"], ) assert result.exit_code == 0 @@ -240,7 +250,8 @@ class TestAskAgentOption: agent_setup.config.agent.tools = agent_tools result = runner.invoke( - cli, ["ask", "--agent", "confirming_agent", "Hello"], + cli, + ["ask", "--agent", "confirming_agent", "Hello"], ) assert result.exit_code == 0 diff --git a/tests/cli/test_ask_context.py b/tests/cli/test_ask_context.py index afd3773b..5e4ed5ed 100644 --- a/tests/cli/test_ask_context.py +++ b/tests/cli/test_ask_context.py @@ -11,9 +11,7 @@ from openjarvis.cli import cli def test_ask_no_context_flag(): """The --no-context flag is accepted.""" - result = CliRunner().invoke( - cli, ["ask", "--no-context", "--help"] - ) + result = CliRunner().invoke(cli, ["ask", "--no-context", "--help"]) # --help should succeed regardless assert result.exit_code == 0 @@ -26,7 +24,8 @@ def test_ask_has_no_context_option(): def test_get_memory_backend_returns_none_when_empty( - tmp_path, monkeypatch, + tmp_path, + monkeypatch, ): """_get_memory_backend returns None when no docs indexed.""" from openjarvis.core.config import JarvisConfig, MemoryConfig @@ -47,7 +46,8 @@ def test_get_memory_backend_returns_none_when_empty( def test_get_memory_backend_returns_backend_with_docs( - tmp_path, monkeypatch, + tmp_path, + monkeypatch, ): """_get_memory_backend returns a backend when docs exist.""" from openjarvis.core.config import JarvisConfig, MemoryConfig diff --git a/tests/cli/test_ask_e2e.py b/tests/cli/test_ask_e2e.py index 18839a50..336f58b4 100644 --- a/tests/cli/test_ask_e2e.py +++ b/tests/cli/test_ask_e2e.py @@ -30,9 +30,7 @@ def _mock_engine_response(): } -def _patch_ask( - monkeypatch, tmp_path, *, engine_result=None, no_engine=False -): +def _patch_ask(monkeypatch, tmp_path, *, engine_result=None, no_engine=False): """Set up common mocks for ask tests.""" cfg = JarvisConfig() cfg.telemetry.db_path = str(tmp_path / "telemetry.db") @@ -40,74 +38,55 @@ def _patch_ask( monkeypatch.setattr(_ask_mod, "load_config", lambda: cfg) if no_engine: - monkeypatch.setattr( - _ask_mod, "get_engine", lambda *a, **kw: None - ) + monkeypatch.setattr(_ask_mod, "get_engine", lambda *a, **kw: None) else: fake_engine = mock.MagicMock() fake_engine.engine_id = "mock" fake_engine.health.return_value = True - fake_engine.generate.return_value = ( - engine_result or _mock_engine_response() - ) + fake_engine.generate.return_value = engine_result or _mock_engine_response() fake_engine.list_models.return_value = ["test-model"] monkeypatch.setattr( - _ask_mod, "get_engine", + _ask_mod, + "get_engine", lambda *a, **kw: ("mock", fake_engine), ) monkeypatch.setattr( - _ask_mod, "discover_engines", + _ask_mod, + "discover_engines", lambda c: [("mock", fake_engine)], ) monkeypatch.setattr( - _ask_mod, "discover_models", + _ask_mod, + "discover_models", lambda e: {"mock": ["test-model"]}, ) class TestAskCommand: - def test_basic_response( - self, monkeypatch, tmp_path: Path - ) -> None: + def test_basic_response(self, monkeypatch, tmp_path: Path) -> None: _patch_ask(monkeypatch, tmp_path) - result = CliRunner().invoke( - cli, ["ask", "What is 2+2?"] - ) + result = CliRunner().invoke(cli, ["ask", "What is 2+2?"]) assert result.exit_code == 0 assert "The answer is 4" in result.output - def test_no_engine_error( - self, monkeypatch, tmp_path: Path - ) -> None: + def test_no_engine_error(self, monkeypatch, tmp_path: Path) -> None: _patch_ask(monkeypatch, tmp_path, no_engine=True) - result = CliRunner().invoke( - cli, ["ask", "Hello"] - ) + result = CliRunner().invoke(cli, ["ask", "Hello"]) assert result.exit_code != 0 - def test_model_override( - self, monkeypatch, tmp_path: Path - ) -> None: + def test_model_override(self, monkeypatch, tmp_path: Path) -> None: _patch_ask(monkeypatch, tmp_path) - result = CliRunner().invoke( - cli, ["ask", "-m", "custom-model", "Hello"] - ) + result = CliRunner().invoke(cli, ["ask", "-m", "custom-model", "Hello"]) assert result.exit_code == 0 - def test_json_output( - self, monkeypatch, tmp_path: Path - ) -> None: + def test_json_output(self, monkeypatch, tmp_path: Path) -> None: _patch_ask(monkeypatch, tmp_path) - result = CliRunner().invoke( - cli, ["ask", "--json", "Hello"] - ) + result = CliRunner().invoke(cli, ["ask", "--json", "Hello"]) assert result.exit_code == 0 data = json.loads(result.output) assert "content" in data - def test_telemetry_recorded( - self, monkeypatch, tmp_path: Path - ) -> None: + def test_telemetry_recorded(self, monkeypatch, tmp_path: Path) -> None: _patch_ask(monkeypatch, tmp_path) CliRunner().invoke(cli, ["ask", "Hello"]) db_path = tmp_path / "telemetry.db" diff --git a/tests/cli/test_ask_router.py b/tests/cli/test_ask_router.py index 22f5b2e4..39758979 100644 --- a/tests/cli/test_ask_router.py +++ b/tests/cli/test_ask_router.py @@ -31,15 +31,18 @@ def _patch_engine(engine): """Return context managers that patch engine discovery to use our mock.""" return ( mock.patch.object( - _ask_mod, "get_engine", + _ask_mod, + "get_engine", return_value=("mock", engine), ), mock.patch.object( - _ask_mod, "discover_engines", + _ask_mod, + "discover_engines", return_value={"mock": engine}, ), mock.patch.object( - _ask_mod, "discover_models", + _ask_mod, + "discover_models", return_value={"mock": ["test-model"]}, ), mock.patch.object(_ask_mod, "register_builtin_models"), @@ -64,7 +67,8 @@ class TestAskModelResolution: patches = _patch_engine(engine) with patches[0], patches[1], patches[2], patches[3], patches[4], patches[5]: result = CliRunner().invoke( - cli, ["ask", "-m", "test-model", "Hello"], + cli, + ["ask", "-m", "test-model", "Hello"], ) assert result.exit_code == 0 assert "Hello!" in result.output @@ -74,9 +78,15 @@ class TestAskModelResolution: engine = _mock_engine() patches = _patch_engine(engine) with ( - patches[0], patches[1], patches[2], patches[3], patches[4], patches[5], + patches[0], + patches[1], + patches[2], + patches[3], + patches[4], + patches[5], mock.patch.object( - _ask_mod, "load_config", + _ask_mod, + "load_config", ) as mock_config, ): cfg = mock_config.return_value @@ -93,14 +103,19 @@ class TestAskModelResolution: patches = _patch_engine(engine) # Override discover_models to return empty list with ( - patches[0], patches[1], + patches[0], + patches[1], mock.patch.object( - _ask_mod, "discover_models", + _ask_mod, + "discover_models", return_value={"mock": []}, ), - patches[3], patches[4], patches[5], + patches[3], + patches[4], + patches[5], mock.patch.object( - _ask_mod, "load_config", + _ask_mod, + "load_config", ) as mock_config, ): cfg = mock_config.return_value diff --git a/tests/cli/test_bench_cmd.py b/tests/cli/test_bench_cmd.py index 0eed437c..c92d8600 100644 --- a/tests/cli/test_bench_cmd.py +++ b/tests/cli/test_bench_cmd.py @@ -56,7 +56,8 @@ class TestBenchCLI: return_value=("mock", engine), ): result = CliRunner().invoke( - cli, ["bench", "run", "-n", "2", "--json"], + cli, + ["bench", "run", "-n", "2", "--json"], ) assert result.exit_code == 0 assert "benchmark_count" in result.output @@ -76,7 +77,8 @@ class TestBenchCLI: return_value=("mock", engine), ): result = CliRunner().invoke( - cli, ["bench", "run", "-n", "2", "-o", str(out_file)], + cli, + ["bench", "run", "-n", "2", "-o", str(out_file)], ) assert result.exit_code == 0 assert out_file.exists() diff --git a/tests/cli/test_channel_cmd.py b/tests/cli/test_channel_cmd.py index 62ef5f97..f1ac6197 100644 --- a/tests/cli/test_channel_cmd.py +++ b/tests/cli/test_channel_cmd.py @@ -25,7 +25,8 @@ def _patch_channel( bridge_instance.status.return_value = status_return config_patch = mock.patch( - "openjarvis.core.config.load_config", return_value=cfg, + "openjarvis.core.config.load_config", + return_value=cfg, ) get_channel_patch = mock.patch( "openjarvis.cli.channel_cmd._get_channel", @@ -75,7 +76,8 @@ class TestChannelSend: config_p, getch_p, _ = _patch_channel(send_return=True) with config_p, getch_p: result = CliRunner().invoke( - cli, ["channel", "send", "slack", "Hello!"], + cli, + ["channel", "send", "slack", "Hello!"], ) assert result.exit_code == 0 assert "Message sent" in result.output @@ -84,7 +86,8 @@ class TestChannelSend: config_p, getch_p, _ = _patch_channel(send_return=False) with config_p, getch_p: result = CliRunner().invoke( - cli, ["channel", "send", "slack", "Hello!"], + cli, + ["channel", "send", "slack", "Hello!"], ) assert result.exit_code == 0 assert "Failed to send" in result.output diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py index d283cfb3..ce9e7f56 100644 --- a/tests/cli/test_cli.py +++ b/tests/cli/test_cli.py @@ -31,11 +31,7 @@ class TestCLI: # Either exits with error (no engine) or succeeds (deps missing) # Both are valid states for testing out = result.output.lower() - assert ( - result.exit_code != 0 - or "not installed" in out - or "no inference" in out - ) + assert result.exit_code != 0 or "not installed" in out or "no inference" in out def test_model_subcommands_exist(self) -> None: result = CliRunner().invoke(cli, ["model", "--help"]) @@ -82,12 +78,8 @@ class TestCLI: config_dir = tmp_path / ".openjarvis" config_path = config_dir / "config.toml" with ( - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir - ), - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path - ), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): result = CliRunner().invoke(cli, ["init", "--engine", "ollama"]) assert result.exit_code == 0 diff --git a/tests/cli/test_daemon_cmd.py b/tests/cli/test_daemon_cmd.py index fc2cf567..4df3b393 100644 --- a/tests/cli/test_daemon_cmd.py +++ b/tests/cli/test_daemon_cmd.py @@ -23,18 +23,14 @@ class TestDaemonCommands: def test_stop_no_server(self) -> None: """``jarvis stop`` when no PID file shows 'not running'.""" - with patch( - "openjarvis.cli.daemon_cmd._read_pid", return_value=None - ): + with patch("openjarvis.cli.daemon_cmd._read_pid", return_value=None): result = CliRunner().invoke(cli, ["stop"]) assert result.exit_code != 0 assert "No running server" in result.output def test_status_no_server(self) -> None: """``jarvis status`` when no PID file shows 'not running'.""" - with patch( - "openjarvis.cli.daemon_cmd._read_pid", return_value=None - ): + with patch("openjarvis.cli.daemon_cmd._read_pid", return_value=None): result = CliRunner().invoke(cli, ["status"]) assert result.exit_code == 0 assert "not running" in result.output @@ -52,9 +48,7 @@ class TestDaemonCommands: pid_file = tmp_path / "server.pid" with ( patch("openjarvis.cli.daemon_cmd._PID_FILE", pid_file), - patch( - "openjarvis.cli.daemon_cmd.DEFAULT_CONFIG_DIR", tmp_path - ), + patch("openjarvis.cli.daemon_cmd.DEFAULT_CONFIG_DIR", tmp_path), patch("os.kill", return_value=None), ): _write_pid(12345) @@ -68,9 +62,7 @@ class TestDaemonCommands: mock_config.server.port = 8000 with ( - patch( - "openjarvis.cli.daemon_cmd._read_pid", return_value=9999 - ), + patch("openjarvis.cli.daemon_cmd._read_pid", return_value=9999), patch( "openjarvis.cli.daemon_cmd.load_config", return_value=mock_config, @@ -83,9 +75,7 @@ class TestDaemonCommands: def test_start_already_running(self) -> None: """``jarvis start`` exits with error when a server is already running.""" - with patch( - "openjarvis.cli.daemon_cmd._read_pid", return_value=42 - ): + with patch("openjarvis.cli.daemon_cmd._read_pid", return_value=42): result = CliRunner().invoke(cli, ["start"]) assert result.exit_code != 0 assert "already running" in result.output diff --git a/tests/cli/test_doctor_cmd.py b/tests/cli/test_doctor_cmd.py index f6fe32d2..2c06c4f3 100644 --- a/tests/cli/test_doctor_cmd.py +++ b/tests/cli/test_doctor_cmd.py @@ -32,19 +32,13 @@ class TestDoctorRuns: mock_config.intelligence.default_model = "" with ( - patch( - "openjarvis.cli.doctor_cmd.load_config", return_value=mock_config - ), + patch("openjarvis.cli.doctor_cmd.load_config", return_value=mock_config), patch( "openjarvis.cli.doctor_cmd.DEFAULT_CONFIG_PATH", Path("/tmp/nonexistent/config.toml"), ), - patch( - "openjarvis.cli.doctor_cmd._check_engines", return_value=[] - ), - patch( - "openjarvis.cli.doctor_cmd._check_models", return_value=[] - ), + patch("openjarvis.cli.doctor_cmd._check_engines", return_value=[]), + patch("openjarvis.cli.doctor_cmd._check_models", return_value=[]), ): result = CliRunner().invoke(cli, ["doctor"]) assert result.exit_code == 0 @@ -58,19 +52,13 @@ class TestDoctorJsonOutput: mock_config.intelligence.default_model = "" with ( - patch( - "openjarvis.cli.doctor_cmd.load_config", return_value=mock_config - ), + patch("openjarvis.cli.doctor_cmd.load_config", return_value=mock_config), patch( "openjarvis.cli.doctor_cmd.DEFAULT_CONFIG_PATH", Path("/tmp/nonexistent/config.toml"), ), - patch( - "openjarvis.cli.doctor_cmd._check_engines", return_value=[] - ), - patch( - "openjarvis.cli.doctor_cmd._check_models", return_value=[] - ), + patch("openjarvis.cli.doctor_cmd._check_engines", return_value=[]), + patch("openjarvis.cli.doctor_cmd._check_models", return_value=[]), ): result = CliRunner().invoke(cli, ["doctor", "--json"]) assert result.exit_code == 0 @@ -129,13 +117,9 @@ class TestCheckEngineProbing: for key in sorted(keys): engine = mock_make_engine(key, mock_config) if engine.health(): - results.append( - CheckResult(f"Engine: {key}", "ok", "Reachable") - ) + results.append(CheckResult(f"Engine: {key}", "ok", "Reachable")) else: - results.append( - CheckResult(f"Engine: {key}", "warn", "Unreachable") - ) + results.append(CheckResult(f"Engine: {key}", "warn", "Unreachable")) names = [r.name for r in results] assert "Engine: ollama" in names diff --git a/tests/cli/test_doctor_labels.py b/tests/cli/test_doctor_labels.py index 3c0c0c94..0b570c80 100644 --- a/tests/cli/test_doctor_labels.py +++ b/tests/cli/test_doctor_labels.py @@ -15,10 +15,12 @@ _real_import = builtins.__import__ def _selective_import_blocker(*blocked: str): """Return an __import__ replacement that blocks specific packages.""" + def _import(name, *args, **kwargs): if name in blocked: raise ImportError(f"mocked: {name} not installed") return _real_import(name, *args, **kwargs) + return _import @@ -45,8 +47,7 @@ class TestDoctorOptionalLabels: result = runner.invoke(cli, ["doctor", "--json"]) data = json.loads(result.output) apple_checks = [ - c for c in data - if c["name"] == "Optional: Apple Silicon energy monitoring" + c for c in data if c["name"] == "Optional: Apple Silicon energy monitoring" ] assert len(apple_checks) == 1 assert "Not installed (openjarvis[energy-apple])" == apple_checks[0]["message"] @@ -59,8 +60,7 @@ class TestDoctorOptionalLabels: result = runner.invoke(cli, ["doctor", "--json"]) data = json.loads(result.output) nvidia_checks = [ - c for c in data - if c["name"] == "Optional: NVIDIA energy monitoring" + c for c in data if c["name"] == "Optional: NVIDIA energy monitoring" ] assert len(nvidia_checks) == 1 assert "Not installed (openjarvis[gpu-metrics])" == nvidia_checks[0]["message"] diff --git a/tests/cli/test_init_guidance.py b/tests/cli/test_init_guidance.py index 21ddbe9f..263d786d 100644 --- a/tests/cli/test_init_guidance.py +++ b/tests/cli/test_init_guidance.py @@ -17,12 +17,8 @@ class TestInitShowsNextSteps: config_dir = tmp_path / ".openjarvis" config_path = config_dir / "config.toml" with ( - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir - ), - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path - ), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) assert result.exit_code == 0 @@ -35,12 +31,8 @@ class TestInitShowsNextSteps: config_dir = tmp_path / ".openjarvis" config_path = config_dir / "config.toml" with ( - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir - ), - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path - ), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) assert result.exit_code == 0 @@ -98,12 +90,8 @@ class TestMinimalConfig: config_dir = tmp_path / ".openjarvis" config_path = config_dir / "config.toml" with ( - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir - ), - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path - ), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): result = CliRunner().invoke(cli, ["init", "--engine", "ollama"]) assert result.exit_code == 0 @@ -119,12 +107,8 @@ class TestMinimalConfig: config_dir = tmp_path / ".openjarvis" config_path = config_dir / "config.toml" with ( - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir - ), - mock.patch( - "openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path - ), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): result = CliRunner().invoke(cli, ["init", "--full", "--engine", "ollama"]) assert result.exit_code == 0 diff --git a/tests/cli/test_log_config.py b/tests/cli/test_log_config.py index 7bacfef0..3d554a2c 100644 --- a/tests/cli/test_log_config.py +++ b/tests/cli/test_log_config.py @@ -29,10 +29,7 @@ class TestSetupLogging: log_file = tmp_path / "cli.log" logger = setup_logging(verbose=True, quiet=False, log_file=log_file) # Should have at least one file handler - file_handlers = [ - h for h in logger.handlers - if hasattr(h, "baseFilename") - ] + file_handlers = [h for h in logger.handlers if hasattr(h, "baseFilename")] assert len(file_handlers) >= 1 # Clean up for h in logger.handlers[:]: diff --git a/tests/cli/test_memory_cmd.py b/tests/cli/test_memory_cmd.py index 9de534b2..dd86adaa 100644 --- a/tests/cli/test_memory_cmd.py +++ b/tests/cli/test_memory_cmd.py @@ -29,13 +29,12 @@ def test_memory_index_file(tmp_path: Path, monkeypatch): mod = importlib.import_module("openjarvis.cli.memory_cmd") monkeypatch.setattr( - mod, "_get_backend", + mod, + "_get_backend", lambda b=None: SQLiteMemory(db_path=db_path), ) - result = CliRunner().invoke( - cli, ["memory", "index", str(doc)] - ) + result = CliRunner().invoke(cli, ["memory", "index", str(doc)]) assert result.exit_code == 0 assert "Indexed" in result.output or "chunk" in result.output @@ -43,9 +42,7 @@ def test_memory_index_file(tmp_path: Path, monkeypatch): def test_memory_index_nonexistent(tmp_path: Path): """Indexing a nonexistent path should fail.""" _register_sqlite() - result = CliRunner().invoke( - cli, ["memory", "index", str(tmp_path / "nope")] - ) + result = CliRunner().invoke(cli, ["memory", "index", str(tmp_path / "nope")]) assert result.exit_code != 0 @@ -61,13 +58,12 @@ def test_memory_search_returns_results(tmp_path: Path, monkeypatch): mod = importlib.import_module("openjarvis.cli.memory_cmd") monkeypatch.setattr( - mod, "_get_backend", + mod, + "_get_backend", lambda b=None: SQLiteMemory(db_path=db_path), ) - result = CliRunner().invoke( - cli, ["memory", "search", "Python"] - ) + result = CliRunner().invoke(cli, ["memory", "search", "Python"]) assert result.exit_code == 0 assert "Python" in result.output backend.close() @@ -82,13 +78,12 @@ def test_memory_search_no_results(tmp_path: Path, monkeypatch): mod = importlib.import_module("openjarvis.cli.memory_cmd") monkeypatch.setattr( - mod, "_get_backend", + mod, + "_get_backend", lambda b=None: SQLiteMemory(db_path=db_path), ) - result = CliRunner().invoke( - cli, ["memory", "search", "quantum supercollider"] - ) + result = CliRunner().invoke(cli, ["memory", "search", "quantum supercollider"]) assert result.exit_code == 0 assert "No results" in result.output backend.close() @@ -104,7 +99,8 @@ def test_memory_stats_shows_count(tmp_path: Path, monkeypatch): mod = importlib.import_module("openjarvis.cli.memory_cmd") monkeypatch.setattr( - mod, "_get_backend", + mod, + "_get_backend", lambda b=None: SQLiteMemory(db_path=db_path), ) diff --git a/tests/cli/test_model_cmd.py b/tests/cli/test_model_cmd.py index 19c9e241..1cf93a9e 100644 --- a/tests/cli/test_model_cmd.py +++ b/tests/cli/test_model_cmd.py @@ -29,11 +29,13 @@ class TestModelList: monkeypatch.setattr(_model_mod, "load_config", lambda: cfg) fake = _mock_engine() monkeypatch.setattr( - _model_mod, "discover_engines", + _model_mod, + "discover_engines", lambda c: [("mock", fake)], ) monkeypatch.setattr( - _model_mod, "discover_models", + _model_mod, + "discover_models", lambda e: {"mock": ["model-a", "model-b"]}, ) result = CliRunner().invoke(cli, ["model", "list"]) @@ -43,9 +45,7 @@ class TestModelList: def test_no_engines_message(self, monkeypatch) -> None: cfg = JarvisConfig() monkeypatch.setattr(_model_mod, "load_config", lambda: cfg) - monkeypatch.setattr( - _model_mod, "discover_engines", lambda c: [] - ) + monkeypatch.setattr(_model_mod, "discover_engines", lambda c: []) result = CliRunner().invoke(cli, ["model", "list"]) assert result.exit_code == 0 assert "No inference engines" in result.output @@ -55,28 +55,16 @@ class TestModelInfo: def test_info_known_model(self, monkeypatch) -> None: cfg = JarvisConfig() monkeypatch.setattr(_model_mod, "load_config", lambda: cfg) - monkeypatch.setattr( - _model_mod, "discover_engines", lambda c: [] - ) - monkeypatch.setattr( - _model_mod, "discover_models", lambda e: {} - ) - result = CliRunner().invoke( - cli, ["model", "info", "qwen3:8b"] - ) + monkeypatch.setattr(_model_mod, "discover_engines", lambda c: []) + monkeypatch.setattr(_model_mod, "discover_models", lambda e: {}) + result = CliRunner().invoke(cli, ["model", "info", "qwen3:8b"]) assert result.exit_code == 0 assert "Qwen3 8B" in result.output def test_unknown_model_not_found(self, monkeypatch) -> None: cfg = JarvisConfig() monkeypatch.setattr(_model_mod, "load_config", lambda: cfg) - monkeypatch.setattr( - _model_mod, "discover_engines", lambda c: [] - ) - monkeypatch.setattr( - _model_mod, "discover_models", lambda e: {} - ) - result = CliRunner().invoke( - cli, ["model", "info", "nonexistent-model"] - ) + monkeypatch.setattr(_model_mod, "discover_engines", lambda c: []) + monkeypatch.setattr(_model_mod, "discover_models", lambda e: {}) + result = CliRunner().invoke(cli, ["model", "info", "nonexistent-model"]) assert result.exit_code != 0 diff --git a/tests/cli/test_quickstart.py b/tests/cli/test_quickstart.py index 19acf906..c80a1357 100644 --- a/tests/cli/test_quickstart.py +++ b/tests/cli/test_quickstart.py @@ -32,28 +32,23 @@ class TestQuickstartCommand: patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_PATH", config_path), patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_DIR", tmp_path), patch( - "openjarvis.cli.quickstart_cmd" - ".generate_default_toml", + "openjarvis.cli.quickstart_cmd.generate_default_toml", return_value="[engine]\n", ), patch( - "openjarvis.cli.quickstart_cmd" - ".recommend_engine", + "openjarvis.cli.quickstart_cmd.recommend_engine", return_value="ollama", ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_engine_health", + "openjarvis.cli.quickstart_cmd._check_engine_health", return_value=True, ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_model_available", + "openjarvis.cli.quickstart_cmd._check_model_available", return_value=True, ), patch( - "openjarvis.cli.quickstart_cmd" - "._test_query", + "openjarvis.cli.quickstart_cmd._test_query", return_value="Hello!", ), ): @@ -79,28 +74,23 @@ class TestQuickstartCommand: patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_PATH", config_path), patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_DIR", tmp_path), patch( - "openjarvis.cli.quickstart_cmd" - ".generate_default_toml", + "openjarvis.cli.quickstart_cmd.generate_default_toml", return_value="[engine]\n", ), patch( - "openjarvis.cli.quickstart_cmd" - ".recommend_engine", + "openjarvis.cli.quickstart_cmd.recommend_engine", return_value="ollama", ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_engine_health", + "openjarvis.cli.quickstart_cmd._check_engine_health", return_value=True, ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_model_available", + "openjarvis.cli.quickstart_cmd._check_model_available", return_value=True, ), patch( - "openjarvis.cli.quickstart_cmd" - "._test_query", + "openjarvis.cli.quickstart_cmd._test_query", return_value="Hello!", ), ): @@ -128,23 +118,19 @@ class TestQuickstartCommand: patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_PATH", config_path), patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_DIR", tmp_path), patch( - "openjarvis.cli.quickstart_cmd" - ".generate_default_toml", + "openjarvis.cli.quickstart_cmd.generate_default_toml", return_value="[engine]\nnew = true\n", ), patch( - "openjarvis.cli.quickstart_cmd" - ".recommend_engine", + "openjarvis.cli.quickstart_cmd.recommend_engine", return_value="ollama", ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_engine_health", + "openjarvis.cli.quickstart_cmd._check_engine_health", return_value=True, ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_model_available", + "openjarvis.cli.quickstart_cmd._check_model_available", return_value=True, ), patch("openjarvis.cli.quickstart_cmd._test_query", return_value="Hello!"), @@ -169,18 +155,15 @@ class TestQuickstartCommand: patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_PATH", config_path), patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_DIR", tmp_path), patch( - "openjarvis.cli.quickstart_cmd" - ".generate_default_toml", + "openjarvis.cli.quickstart_cmd.generate_default_toml", return_value="[engine]\n", ), patch( - "openjarvis.cli.quickstart_cmd" - ".recommend_engine", + "openjarvis.cli.quickstart_cmd.recommend_engine", return_value="ollama", ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_engine_health", + "openjarvis.cli.quickstart_cmd._check_engine_health", return_value=False, ), ): @@ -207,33 +190,27 @@ class TestQuickstartCommand: patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_PATH", config_path), patch("openjarvis.cli.quickstart_cmd.DEFAULT_CONFIG_DIR", tmp_path), patch( - "openjarvis.cli.quickstart_cmd" - ".generate_default_toml", - return_value="[engine]\ndefault = \"mlx\"\n", + "openjarvis.cli.quickstart_cmd.generate_default_toml", + return_value='[engine]\ndefault = "mlx"\n', ), patch( - "openjarvis.cli.quickstart_cmd" - ".recommend_engine", + "openjarvis.cli.quickstart_cmd.recommend_engine", return_value="mlx", ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_engine_health", + "openjarvis.cli.quickstart_cmd._check_engine_health", return_value=False, ), patch( - "openjarvis.cli.quickstart_cmd" - "._discover_healthy_engines", + "openjarvis.cli.quickstart_cmd._discover_healthy_engines", return_value=["ollama"], ), patch( - "openjarvis.cli.quickstart_cmd" - "._check_model_available", + "openjarvis.cli.quickstart_cmd._check_model_available", return_value=True, ), patch( - "openjarvis.cli.quickstart_cmd" - "._test_query", + "openjarvis.cli.quickstart_cmd._test_query", return_value="Hello!", ), ): diff --git a/tests/cli/test_serve_channel_wiring.py b/tests/cli/test_serve_channel_wiring.py new file mode 100644 index 00000000..81feb254 --- /dev/null +++ b/tests/cli/test_serve_channel_wiring.py @@ -0,0 +1,249 @@ +"""Tests for JarvisSystem.wire_channel() — channel → agent routing. + +These tests exercise wire_channel() on JarvisSystem directly. The serve.py +entrypoint now delegates all channel-wiring logic there. +""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +from openjarvis.channels._stubs import ChannelMessage +from openjarvis.core.config import JarvisConfig +from openjarvis.core.events import EventBus +from openjarvis.sessions.session import SessionStore +from openjarvis.system import JarvisSystem + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_channel_message( + channel: str = "telegram", + sender: str = "42", + content: str = "hello", + conversation_id: str = "42", +) -> ChannelMessage: + return ChannelMessage( + channel=channel, + sender=sender, + content=content, + message_id="1", + conversation_id=conversation_id, + ) + + +def _make_system(engine=None, agent_name="", tmp_path=None) -> JarvisSystem: + """Build a minimal JarvisSystem with mock engine for testing wire_channel.""" + config = JarvisConfig() + if tmp_path is not None: + config.sessions.db_path = str(tmp_path / "sessions.db") + mock_engine = engine or MagicMock() + return JarvisSystem( + config=config, + bus=EventBus(record_history=False), + engine=mock_engine, + engine_key="mock", + model="test-model", + agent_name=agent_name, + ) + + +def _fire(channel_mock, cm: ChannelMessage) -> None: + """Invoke all registered on_message handlers as if the channel received cm.""" + for handler in channel_mock.on_message.call_args_list: + handler[0][0](cm) + + +# --------------------------------------------------------------------------- +# Tests via JarvisSystem.wire_channel() +# --------------------------------------------------------------------------- + + +class TestWireChannelWithAgent: + """wire_channel routes incoming messages through the agent and replies.""" + + def test_ask_called_and_reply_sent(self, tmp_path): + system = _make_system(agent_name="simple", tmp_path=tmp_path) + # Patch ask() so we don't need a real engine/agent + system.ask = MagicMock(return_value={"content": "pong"}) + + mock_channel = MagicMock() + system.wire_channel(mock_channel) + + # Simulate an incoming message + cm = _make_channel_message(content="ping") + handler = mock_channel.on_message.call_args[0][0] + handler(cm) + + system.ask.assert_called_once() + assert system.ask.call_args[0][0] == "ping" + mock_channel.send.assert_called_once_with( + "telegram", + "pong", + conversation_id="42", + ) + + def test_session_store_created_lazily(self, tmp_path): + system = _make_system(tmp_path=tmp_path) + assert system.session_store is None + + mock_channel = MagicMock() + system.ask = MagicMock(return_value={"content": "ok"}) + system.wire_channel(mock_channel) + + handler = mock_channel.on_message.call_args[0][0] + handler(_make_channel_message()) + + assert system.session_store is not None + + def test_existing_session_store_reused(self, tmp_path): + system = _make_system(tmp_path=tmp_path) + existing_store = SessionStore(db_path=tmp_path / "sessions.db") + system.session_store = existing_store + system.ask = MagicMock(return_value={"content": "ok"}) + + mock_channel = MagicMock() + system.wire_channel(mock_channel) + + handler = mock_channel.on_message.call_args[0][0] + handler(_make_channel_message()) + + assert system.session_store is existing_store + + +class TestWireChannelWithEngine: + """wire_channel falls back to engine when no agent is set.""" + + def test_engine_path_used_when_no_agent(self, tmp_path): + system = _make_system(agent_name="", tmp_path=tmp_path) + system.ask = MagicMock(return_value={"content": "raw reply"}) + + mock_channel = MagicMock() + system.wire_channel(mock_channel) + + handler = mock_channel.on_message.call_args[0][0] + handler(_make_channel_message(content="hi")) + + mock_channel.send.assert_called_once_with( + "telegram", + "raw reply", + conversation_id="42", + ) + + +class TestWireChannelSessionIsolation: + """Separate conversation_ids get independent sessions.""" + + def test_two_chats_isolated(self, tmp_path): + system = _make_system(tmp_path=tmp_path) + replies = {"111": "reply-A", "222": "reply-B"} + system.ask = MagicMock( + side_effect=lambda q, **kw: {"content": replies.get(q, "")} + ) + + mock_channel = MagicMock() + system.wire_channel(mock_channel) + handler = mock_channel.on_message.call_args[0][0] + + handler(_make_channel_message(content="111", conversation_id="111")) + handler(_make_channel_message(content="222", conversation_id="222")) + + # Reload sessions — each chat must have only its own messages + s1 = system.session_store.get_or_create("telegram:111") + s2 = system.session_store.get_or_create("telegram:222") + c1 = {m.content for m in s1.messages} + c2 = {m.content for m in s2.messages} + + assert "111" in c1 and "222" not in c1 + assert "222" in c2 and "111" not in c2 + + def test_same_chat_accumulates_history(self, tmp_path): + system = _make_system(tmp_path=tmp_path) + system.ask = MagicMock(return_value={"content": "reply"}) + + mock_channel = MagicMock() + system.wire_channel(mock_channel) + handler = mock_channel.on_message.call_args[0][0] + + handler(_make_channel_message(content="first")) + handler(_make_channel_message(content="second")) + + session = system.session_store.get_or_create("telegram:42") + contents = [m.content for m in session.messages] + # user + assistant alternating for two turns + assert contents.count("first") == 1 + assert contents.count("second") == 1 + + +class TestWireChannelErrorHandling: + """Handler sends a user-visible error message when ask() raises.""" + + def test_error_reply_sent(self, tmp_path): + system = _make_system(tmp_path=tmp_path) + system.ask = MagicMock(side_effect=RuntimeError("boom")) + + mock_channel = MagicMock() + system.wire_channel(mock_channel) + handler = mock_channel.on_message.call_args[0][0] + handler(_make_channel_message()) + + mock_channel.send.assert_called_once() + sent_content = mock_channel.send.call_args[0][1] + assert "error" in sent_content.lower() + + +class TestPerChatSessionIsolation: + """Direct SessionStore isolation tests (not via wire_channel).""" + + def test_two_chats_have_separate_sessions(self, tmp_path): + store = SessionStore(db_path=tmp_path / "sessions.db") + + session_a = store.get_or_create( + "telegram:111", + channel="telegram", + channel_user_id="111", + ) + store.save_message(session_a.session_id, "user", "msg from A") + + session_b = store.get_or_create( + "telegram:222", + channel="telegram", + channel_user_id="222", + ) + store.save_message(session_b.session_id, "user", "msg from B") + + reloaded_a = store.get_or_create( + "telegram:111", + channel="telegram", + channel_user_id="111", + ) + reloaded_b = store.get_or_create( + "telegram:222", + channel="telegram", + channel_user_id="222", + ) + + contents_a = {m.content for m in reloaded_a.messages} + contents_b = {m.content for m in reloaded_b.messages} + + assert "msg from A" in contents_a and "msg from B" not in contents_a + assert "msg from B" in contents_b and "msg from A" not in contents_b + + def test_same_chat_accumulates_history(self, tmp_path): + store = SessionStore(db_path=tmp_path / "sessions.db") + session = store.get_or_create( + "telegram:42", + channel="telegram", + channel_user_id="42", + ) + store.save_message(session.session_id, "user", "first") + store.save_message(session.session_id, "assistant", "reply") + + reloaded = store.get_or_create( + "telegram:42", + channel="telegram", + channel_user_id="42", + ) + assert [m.content for m in reloaded.messages] == ["first", "reply"] diff --git a/tests/cli/test_telemetry_cmd.py b/tests/cli/test_telemetry_cmd.py index d51d1405..b6b3ee10 100644 --- a/tests/cli/test_telemetry_cmd.py +++ b/tests/cli/test_telemetry_cmd.py @@ -18,16 +18,18 @@ def _populate_db(db_path: Path, n: int = 3) -> None: """Create a telemetry DB with *n* records.""" store = TelemetryStore(db_path) for i in range(n): - store.record(TelemetryRecord( - timestamp=time.time() - (n - i), - model_id=f"model-{i % 2}", - engine="ollama", - prompt_tokens=10 * (i + 1), - completion_tokens=5 * (i + 1), - total_tokens=15 * (i + 1), - latency_seconds=0.5 * (i + 1), - cost_usd=0.001 * (i + 1), - )) + store.record( + TelemetryRecord( + timestamp=time.time() - (n - i), + model_id=f"model-{i % 2}", + engine="ollama", + prompt_tokens=10 * (i + 1), + completion_tokens=5 * (i + 1), + total_tokens=15 * (i + 1), + latency_seconds=0.5 * (i + 1), + cost_usd=0.001 * (i + 1), + ) + ) store.close() @@ -37,7 +39,8 @@ def _patch_config(tmp_path: Path): cfg = mock.MagicMock() cfg.telemetry.db_path = str(db_path) return mock.patch( - "openjarvis.cli.telemetry_cmd.load_config", return_value=cfg, + "openjarvis.cli.telemetry_cmd.load_config", + return_value=cfg, ), db_path @@ -115,7 +118,8 @@ class TestTelemetryExport: out_file = tmp_path / "export.json" with patch: result = CliRunner().invoke( - cli, ["telemetry", "export", "-o", str(out_file)], + cli, + ["telemetry", "export", "-o", str(out_file)], ) assert result.exit_code == 0 assert out_file.exists() diff --git a/tests/cli/test_vault_cmd.py b/tests/cli/test_vault_cmd.py index aaa01d2d..c839b25f 100644 --- a/tests/cli/test_vault_cmd.py +++ b/tests/cli/test_vault_cmd.py @@ -47,7 +47,8 @@ class TestVaultCmd: mock.patch("openjarvis.cli.vault_cmd._VAULT_FILE", vault_file), mock.patch("openjarvis.cli.vault_cmd._VAULT_KEY_FILE", key_file), mock.patch( - "openjarvis.cli.vault_cmd.DEFAULT_CONFIG_DIR", tmp_path, + "openjarvis.cli.vault_cmd.DEFAULT_CONFIG_DIR", + tmp_path, ), ): runner = CliRunner() diff --git a/tests/conftest.py b/tests/conftest.py index af05a657..cd709c1e 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -54,8 +54,10 @@ def nvidia_gpu() -> GpuInfo: def nvidia_consumer_gpu() -> GpuInfo: """NVIDIA consumer GPU fixture.""" return GpuInfo( - vendor="nvidia", name="NVIDIA GeForce RTX 4090", - vram_gb=24.0, count=1, + vendor="nvidia", + name="NVIDIA GeForce RTX 4090", + vram_gb=24.0, + count=1, ) @@ -147,6 +149,7 @@ def has_ollama() -> bool: """Check if Ollama is running locally.""" try: import httpx + resp = httpx.get("http://localhost:11434/api/tags", timeout=2.0) return resp.status_code == 200 except Exception: @@ -158,6 +161,7 @@ def has_vllm() -> bool: """Check if vLLM is running locally.""" try: import httpx + resp = httpx.get("http://localhost:8000/v1/models", timeout=2.0) return resp.status_code == 200 except Exception: @@ -169,6 +173,7 @@ def has_llamacpp() -> bool: """Check if llama.cpp server is running locally.""" try: import httpx + resp = httpx.get("http://localhost:8080/v1/models", timeout=2.0) return resp.status_code == 200 except Exception: diff --git a/tests/core/test_config.py b/tests/core/test_config.py index 28407d26..69c1ee14 100644 --- a/tests/core/test_config.py +++ b/tests/core/test_config.py @@ -200,9 +200,10 @@ class TestAgentConfigNew: def test_no_temperature_or_max_tokens(self) -> None: ac = AgentConfig() - assert not hasattr(ac.__class__, "temperature") or isinstance( - getattr(ac.__class__, "temperature", None), property - ) is False + assert ( + not hasattr(ac.__class__, "temperature") + or isinstance(getattr(ac.__class__, "temperature", None), property) is False + ) class TestNestedEngineConfig: @@ -288,9 +289,9 @@ class TestNestedLearningConfig: def test_loads_nested_toml(self, tmp_path: Path) -> None: toml_file = tmp_path / "config.toml" toml_file.write_text( - '[learning]\nenabled = true\nupdate_interval = 50\n\n' + "[learning]\nenabled = true\nupdate_interval = 50\n\n" '[learning.routing]\npolicy = "learned"\n\n' - '[learning.metrics]\nlatency_weight = 0.5\n' + "[learning.metrics]\nlatency_weight = 0.5\n" ) cfg = load_config(toml_file) assert cfg.learning.enabled is True @@ -315,16 +316,14 @@ class TestNestedLearningConfig: class TestBackwardCompatMigration: def test_agent_temperature_migrates_to_intelligence(self, tmp_path: Path) -> None: toml_file = tmp_path / "config.toml" - toml_file.write_text( - '[agent]\ntemperature = 0.3\nmax_tokens = 512\n' - ) + toml_file.write_text("[agent]\ntemperature = 0.3\nmax_tokens = 512\n") cfg = load_config(toml_file) assert cfg.intelligence.temperature == 0.3 assert cfg.intelligence.max_tokens == 512 def test_memory_context_injection_migrates_to_agent(self, tmp_path: Path) -> None: toml_file = tmp_path / "config.toml" - toml_file.write_text('[memory]\ncontext_injection = false\n') + toml_file.write_text("[memory]\ncontext_injection = false\n") cfg = load_config(toml_file) assert cfg.agent.context_from_memory is False @@ -400,8 +399,7 @@ class TestSandboxConfig: def test_loads_from_toml(self, tmp_path: Path) -> None: toml_file = tmp_path / "config.toml" toml_file.write_text( - '[sandbox]\nenabled = true\ntimeout = 600\n' - 'runtime = "podman"\n' + '[sandbox]\nenabled = true\ntimeout = 600\nruntime = "podman"\n' ) cfg = load_config(toml_file) assert cfg.sandbox.enabled is True @@ -435,7 +433,7 @@ class TestSchedulerConfig: def test_loads_from_toml(self, tmp_path: Path) -> None: toml_file = tmp_path / "config.toml" toml_file.write_text( - '[scheduler]\nenabled = true\npoll_interval = 30\n' + "[scheduler]\nenabled = true\npoll_interval = 30\n" 'db_path = "/tmp/sched.db"\n' ) cfg = load_config(toml_file) diff --git a/tests/core/test_config_phase4.py b/tests/core/test_config_phase4.py index c69c266f..4b2b4acf 100644 --- a/tests/core/test_config_phase4.py +++ b/tests/core/test_config_phase4.py @@ -45,8 +45,7 @@ class TestLearningConfig: def test_toml_loading_with_learning(self, tmp_path: Path) -> None: toml_file = tmp_path / "config.toml" toml_file.write_text( - '[learning]\ndefault_policy = "grpo"\n' - 'reward_weights = "latency=0.5"\n' + '[learning]\ndefault_policy = "grpo"\nreward_weights = "latency=0.5"\n' ) cfg = load_config(toml_file) assert cfg.learning.routing.policy == "grpo" @@ -55,9 +54,9 @@ class TestLearningConfig: def test_toml_loading_nested(self, tmp_path: Path) -> None: toml_file = tmp_path / "config.toml" toml_file.write_text( - '[learning]\nenabled = true\n\n' + "[learning]\nenabled = true\n\n" '[learning.routing]\npolicy = "learned"\n\n' - '[learning.metrics]\nlatency_weight = 0.5\n' + "[learning.metrics]\nlatency_weight = 0.5\n" ) cfg = load_config(toml_file) assert cfg.learning.enabled is True diff --git a/tests/core/test_credentials.py b/tests/core/test_credentials.py index e5679d47..2d8993fb 100644 --- a/tests/core/test_credentials.py +++ b/tests/core/test_credentials.py @@ -1,4 +1,5 @@ """Tests for credential persistence module.""" + import os import pytest diff --git a/tests/core/test_events.py b/tests/core/test_events.py index 3f846472..471b6270 100644 --- a/tests/core/test_events.py +++ b/tests/core/test_events.py @@ -95,12 +95,14 @@ class TestEventBus: class TestAgentEventTypes: def test_agent_tick_events_exist(self): from openjarvis.core.events import EventType + assert EventType.AGENT_TICK_START assert EventType.AGENT_TICK_END assert EventType.AGENT_TICK_ERROR def test_agent_operational_events_exist(self): from openjarvis.core.events import EventType + assert EventType.AGENT_BUDGET_EXCEEDED assert EventType.AGENT_STALL_DETECTED assert EventType.AGENT_MESSAGE_RECEIVED diff --git a/tests/core/test_rust_bridge.py b/tests/core/test_rust_bridge.py index 47ac6812..893250de 100644 --- a/tests/core/test_rust_bridge.py +++ b/tests/core/test_rust_bridge.py @@ -24,12 +24,14 @@ class TestScanResultFromJson: def test_empty_findings(self): from openjarvis._rust_bridge import scan_result_from_json + result = scan_result_from_json('{"findings": []}') assert result.clean assert result.findings == [] def test_with_findings(self): from openjarvis._rust_bridge import scan_result_from_json + data = { "findings": [ { @@ -54,6 +56,7 @@ class TestInjectionResultFromJson: def test_clean(self): from openjarvis._rust_bridge import injection_result_from_json + data = {"is_clean": True, "findings": [], "threat_level": "low"} result = injection_result_from_json(json.dumps(data)) assert result.is_clean @@ -61,6 +64,7 @@ class TestInjectionResultFromJson: def test_with_findings(self): from openjarvis._rust_bridge import injection_result_from_json + data = { "is_clean": False, "findings": [ @@ -86,11 +90,13 @@ class TestRetrievalResultsFromJson: def test_empty(self): from openjarvis._rust_bridge import retrieval_results_from_json + results = retrieval_results_from_json("[]") assert results == [] def test_with_items(self): from openjarvis._rust_bridge import retrieval_results_from_json + data = [ { "content": "hello world", @@ -108,6 +114,7 @@ class TestRetrievalResultsFromJson: def test_metadata_as_string(self): from openjarvis._rust_bridge import retrieval_results_from_json + data = [ { "content": "test", diff --git a/tests/engine/test_cloud_extended.py b/tests/engine/test_cloud_extended.py index e63c8dda..db946a80 100644 --- a/tests/engine/test_cloud_extended.py +++ b/tests/engine/test_cloud_extended.py @@ -324,8 +324,10 @@ class TestCloudAnthropic: model="claude-opus-4-6", ) call_kwargs = fake_client.messages.create.call_args - assert call_kwargs.kwargs.get("system") == "You are helpful" or \ - call_kwargs[1].get("system") == "You are helpful" + assert ( + call_kwargs.kwargs.get("system") == "You are helpful" + or call_kwargs[1].get("system") == "You are helpful" + ) # --------------------------------------------------------------------------- @@ -339,10 +341,13 @@ class TestCloudGemini: monkeypatch.delenv("OPENAI_API_KEY", raising=False) monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) fake_genai = mock.MagicMock() - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": fake_genai, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": fake_genai, + }, + ): if not EngineRegistry.contains("cloud"): EngineRegistry.register_value("cloud", CloudEngine) engine = CloudEngine() @@ -360,11 +365,14 @@ class TestCloudGemini: fake_config = mock.MagicMock() fake_types = mock.MagicMock() fake_types.GenerateContentConfig.return_value = fake_config - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": mock.MagicMock(), - "google.genai.types": fake_types, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": mock.MagicMock(), + "google.genai.types": fake_types, + }, + ): result = engine.generate( [Message(role=Role.USER, content="Hi")], model="gemini-2.5-pro" ) @@ -383,11 +391,14 @@ class TestCloudGemini: fake_types = mock.MagicMock() fake_types.GenerateContentConfig.return_value = mock.MagicMock() - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": mock.MagicMock(), - "google.genai.types": fake_types, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": mock.MagicMock(), + "google.genai.types": fake_types, + }, + ): result = engine.generate( [Message(role=Role.USER, content="Hi")], model="gemini-2.5-flash" ) @@ -403,11 +414,14 @@ class TestCloudGemini: fake_types = mock.MagicMock() fake_types.GenerateContentConfig.return_value = mock.MagicMock() - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": mock.MagicMock(), - "google.genai.types": fake_types, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": mock.MagicMock(), + "google.genai.types": fake_types, + }, + ): result = engine.generate( [Message(role=Role.USER, content="Hi")], model="gemini-3-pro" ) @@ -423,11 +437,14 @@ class TestCloudGemini: fake_types = mock.MagicMock() fake_types.GenerateContentConfig.return_value = mock.MagicMock() - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": mock.MagicMock(), - "google.genai.types": fake_types, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": mock.MagicMock(), + "google.genai.types": fake_types, + }, + ): result = engine.generate( [Message(role=Role.USER, content="Hi")], model="gemini-3-flash" ) @@ -441,16 +458,16 @@ class TestCloudGemini: fake_client = mock.MagicMock() # Build a response with a function_call part - text_part = SimpleNamespace( - text="Let me calculate.", function_call=None - ) + text_part = SimpleNamespace(text="Let me calculate.", function_call=None) fc = SimpleNamespace(name="calculator", args={"expression": "2+2"}) fc_part = SimpleNamespace(text=None, function_call=fc) content_obj = SimpleNamespace(parts=[text_part, fc_part]) candidate = SimpleNamespace(content=content_obj) usage = SimpleNamespace(prompt_token_count=10, candidates_token_count=8) fake_resp = SimpleNamespace( - candidates=[candidate], usage_metadata=usage, text=None, + candidates=[candidate], + usage_metadata=usage, + text=None, ) fake_client.models.generate_content.return_value = fake_resp engine._google_client = fake_client @@ -458,11 +475,14 @@ class TestCloudGemini: fake_types = mock.MagicMock() fake_config = mock.MagicMock() fake_types.GenerateContentConfig.return_value = fake_config - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": mock.MagicMock(), - "google.genai.types": fake_types, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": mock.MagicMock(), + "google.genai.types": fake_types, + }, + ): result = engine.generate( [Message(role=Role.USER, content="What is 2+2?")], model="gemini-3-pro", @@ -497,11 +517,14 @@ class TestCloudGemini: fake_types = mock.MagicMock() fake_types.GenerateContentConfig.return_value = mock.MagicMock() - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": mock.MagicMock(), - "google.genai.types": fake_types, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": mock.MagicMock(), + "google.genai.types": fake_types, + }, + ): result = engine.generate( [Message(role=Role.USER, content="Hi")], model="gemini-3-pro" ) @@ -538,11 +561,14 @@ class TestCloudGemini: fake_types = mock.MagicMock() fake_types.GenerateContentConfig.return_value = mock.MagicMock() - with mock.patch.dict("sys.modules", { - "google": mock.MagicMock(), - "google.genai": mock.MagicMock(), - "google.genai.types": fake_types, - }): + with mock.patch.dict( + "sys.modules", + { + "google": mock.MagicMock(), + "google.genai": mock.MagicMock(), + "google.genai.types": fake_types, + }, + ): with pytest.raises( EngineConnectionError, match="Google client not available", @@ -606,8 +632,13 @@ class TestPricingTable: def test_all_new_models_in_pricing(self) -> None: expected = [ "gpt-5-mini", - "claude-opus-4-6", "claude-sonnet-4-6", "claude-haiku-4-5", - "gemini-2.5-pro", "gemini-2.5-flash", "gemini-3-pro", "gemini-3-flash", + "claude-opus-4-6", + "claude-sonnet-4-6", + "claude-haiku-4-5", + "gemini-2.5-pro", + "gemini-2.5-flash", + "gemini-3-pro", + "gemini-3-flash", ] for model_id in expected: assert model_id in PRICING, f"{model_id} missing from PRICING dict" diff --git a/tests/engine/test_discovery.py b/tests/engine/test_discovery.py index fe3cb888..d111c169 100644 --- a/tests/engine/test_discovery.py +++ b/tests/engine/test_discovery.py @@ -54,9 +54,7 @@ class TestDiscoverEngines: cfg = JarvisConfig() with mock.patch( "openjarvis.engine._discovery._make_engine", - side_effect=lambda k, c: _FakeEngine( - healthy=(k == "healthy") - ), + side_effect=lambda k, c: _FakeEngine(healthy=(k == "healthy")), ): result = discover_engines(cfg) assert len(result) == 1 diff --git a/tests/engine/test_engine_model_matrix.py b/tests/engine/test_engine_model_matrix.py index 8e0cc36a..a8ad1554 100644 --- a/tests/engine/test_engine_model_matrix.py +++ b/tests/engine/test_engine_model_matrix.py @@ -40,15 +40,18 @@ def _api_prefix(engine_key: str) -> str: return "" return "/v1" -ENGINES_AND_HOSTS = [ - (key, host) for key, host, _ in _OPENAI_COMPAT_ENGINES -] + [ + +ENGINES_AND_HOSTS = [(key, host) for key, host, _ in _OPENAI_COMPAT_ENGINES] + [ ("ollama", "http://testhost:11434"), ] MODELS = [ - "gpt-oss:120b", "qwen3:8b", "glm-4.7-flash", "trinity-mini", - "qwen3.5:35b-a3b", "LiquidAI/LFM2.5-1.2B-Instruct-GGUF", + "gpt-oss:120b", + "qwen3:8b", + "glm-4.7-flash", + "trinity-mini", + "qwen3.5:35b-a3b", + "LiquidAI/LFM2.5-1.2B-Instruct-GGUF", ] _ENGINE_CLASSES = {key: cls for key, _, cls in _OPENAI_COMPAT_ENGINES} @@ -69,26 +72,34 @@ def _mock_simple_chat(respx_mock, engine_key: str, host: str, model: str): """Set up mock for a simple chat response.""" if engine_key == "ollama": respx_mock.post(f"{host}/api/chat").mock( - return_value=httpx.Response(200, json={ - "message": {"role": "assistant", "content": "Hello!"}, - "model": model, - "prompt_eval_count": 10, - "eval_count": 5, - "done": True, - }) + return_value=httpx.Response( + 200, + json={ + "message": {"role": "assistant", "content": "Hello!"}, + "model": model, + "prompt_eval_count": 10, + "eval_count": 5, + "done": True, + }, + ) ) else: # All OpenAI-compatible engines prefix = _api_prefix(engine_key) respx_mock.post(f"{host}{prefix}/chat/completions").mock( - return_value=httpx.Response(200, json={ - "choices": [ - {"message": {"content": "Hello!"}, "finish_reason": "stop"}, - ], - "usage": { - "prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15, + return_value=httpx.Response( + 200, + json={ + "choices": [ + {"message": {"content": "Hello!"}, "finish_reason": "stop"}, + ], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 5, + "total_tokens": 15, + }, + "model": model, }, - "model": model, - }) + ) ) @@ -96,39 +107,59 @@ def _mock_tool_call(respx_mock, engine_key: str, host: str, model: str): """Set up mock for a tool-call response.""" if engine_key == "ollama": respx_mock.post(f"{host}/api/chat").mock( - return_value=httpx.Response(200, json={ - "message": { - "content": "", - "tool_calls": [{ - "function": {"name": "calculator", "arguments": '{"x":1}'}, - }], + return_value=httpx.Response( + 200, + json={ + "message": { + "content": "", + "tool_calls": [ + { + "function": { + "name": "calculator", + "arguments": '{"x":1}', + }, + } + ], + }, + "model": model, + "prompt_eval_count": 10, + "eval_count": 8, + "done": True, }, - "model": model, - "prompt_eval_count": 10, - "eval_count": 8, - "done": True, - }) + ) ) else: # All OpenAI-compatible engines prefix = _api_prefix(engine_key) respx_mock.post(f"{host}{prefix}/chat/completions").mock( - return_value=httpx.Response(200, json={ - "choices": [{ - "message": { - "content": "", - "tool_calls": [{ - "id": "call_1", - "type": "function", - "function": {"name": "calculator", "arguments": '{"x":1}'}, - }], + return_value=httpx.Response( + 200, + json={ + "choices": [ + { + "message": { + "content": "", + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "calculator", + "arguments": '{"x":1}', + }, + } + ], + }, + "finish_reason": "tool_calls", + } + ], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 8, + "total_tokens": 18, }, - "finish_reason": "tool_calls", - }], - "usage": { - "prompt_tokens": 10, "completion_tokens": 8, "total_tokens": 18, + "model": model, }, - "model": model, - }) + ) ) @@ -176,9 +207,7 @@ class TestEngineScenarios: engine = _create_engine(engine_key, host) _mock_error(respx_mock, engine_key, host) with pytest.raises(EngineConnectionError): - engine.generate( - [Message(role=Role.USER, content="Hi")], model="qwen3:8b" - ) + engine.generate([Message(role=Role.USER, content="Hi")], model="qwen3:8b") # --------------------------------------------------------------------------- @@ -190,7 +219,11 @@ class TestEngineScenarios: @pytest.mark.parametrize("model_id", MODELS) class TestEngineModelMatrix: def test_generate_with_model( - self, respx_mock, engine_key: str, host: str, model_id: str, + self, + respx_mock, + engine_key: str, + host: str, + model_id: str, ) -> None: engine = _create_engine(engine_key, host) _mock_simple_chat(respx_mock, engine_key, host, model_id) diff --git a/tests/engine/test_llamacpp_models.py b/tests/engine/test_llamacpp_models.py index 4f740c82..03aa9b79 100644 --- a/tests/engine/test_llamacpp_models.py +++ b/tests/engine/test_llamacpp_models.py @@ -85,7 +85,9 @@ class TestLlamaCppGenerate: return_value=httpx.Response( 200, json=_openai_response( - content="", model=model_id, tool_calls=tool_calls, + content="", + model=model_id, + tool_calls=tool_calls, ), ) ) @@ -143,6 +145,7 @@ class TestLlamaCppGenerate: return tokens import asyncio + tokens = asyncio.run(collect()) assert tokens == ["Hi", " there"] diff --git a/tests/engine/test_mlx.py b/tests/engine/test_mlx.py index 2bd0e891..fb3c976f 100644 --- a/tests/engine/test_mlx.py +++ b/tests/engine/test_mlx.py @@ -51,9 +51,7 @@ class TestMLXGenerate: side_effect=httpx.ConnectError("refused") ) with pytest.raises(EngineConnectionError): - engine.generate( - [Message(role=Role.USER, content="Hi")], model="m" - ) + engine.generate([Message(role=Role.USER, content="Hi")], model="m") class TestMLXHealth: diff --git a/tests/engine/test_ollama_models.py b/tests/engine/test_ollama_models.py index 66a6214a..11cf54da 100644 --- a/tests/engine/test_ollama_models.py +++ b/tests/engine/test_ollama_models.py @@ -81,7 +81,9 @@ class TestOllamaGenerate: return_value=httpx.Response( 200, json=_ollama_response( - content="", model=model_id, tool_calls=tool_calls, + content="", + model=model_id, + tool_calls=tool_calls, ), ) ) @@ -104,7 +106,8 @@ class TestOllamaGenerate: return_value=httpx.Response( 200, json=_ollama_response( - content="", model=model_id, + content="", + model=model_id, tool_calls=tool_calls, ), ) @@ -136,6 +139,7 @@ class TestOllamaGenerate: return tokens import asyncio + tokens = asyncio.run(collect()) assert "Hello" in tokens @@ -222,9 +226,7 @@ class TestOllamaErrors: def capture(request): captured["body"] = json.loads(request.content) - return httpx.Response( - 200, json=_ollama_response(content="ok") - ) + return httpx.Response(200, json=_ollama_response(content="ok")) respx_mock.post(f"{OLLAMA_HOST}/api/chat").mock(side_effect=capture) engine.generate( @@ -241,12 +243,8 @@ class TestOllamaErrors: def capture(request): captured["body"] = json.loads(request.content) - return httpx.Response( - 200, json=_ollama_response(content="ok") - ) + return httpx.Response(200, json=_ollama_response(content="ok")) respx_mock.post(f"{OLLAMA_HOST}/api/chat").mock(side_effect=capture) - engine.generate( - [Message(role=Role.USER, content="Hello")], model="qwen3:8b" - ) + engine.generate([Message(role=Role.USER, content="Hello")], model="qwen3:8b") assert "tools" not in captured["body"] diff --git a/tests/engine/test_openai_compat.py b/tests/engine/test_openai_compat.py index 148594fd..31cc2af8 100644 --- a/tests/engine/test_openai_compat.py +++ b/tests/engine/test_openai_compat.py @@ -47,12 +47,11 @@ class TestOpenAICompatGenerate: assert result["usage"]["total_tokens"] == 9 def test_empty_choices_returns_graceful_fallback( - self, engine: VLLMEngine, + self, + engine: VLLMEngine, ) -> None: with respx.mock: - respx.post( - "http://testhost:8000/v1/chat/completions" - ).mock( + respx.post("http://testhost:8000/v1/chat/completions").mock( return_value=httpx.Response( 200, json={ diff --git a/tests/engine/test_openai_compat_tools.py b/tests/engine/test_openai_compat_tools.py index 18857748..3214d088 100644 --- a/tests/engine/test_openai_compat_tools.py +++ b/tests/engine/test_openai_compat_tools.py @@ -24,17 +24,20 @@ class TestOpenAICompatToolCalls: def test_no_tool_calls(self, respx_mock): """When no tool_calls in response, result has no tool_calls key.""" respx_mock.post("http://localhost:9999/v1/chat/completions").mock( - return_value=httpx.Response(200, json={ - "choices": [ - {"message": {"content": "Hi"}, "finish_reason": "stop"}, - ], - "usage": { - "prompt_tokens": 5, - "completion_tokens": 2, - "total_tokens": 7, + return_value=httpx.Response( + 200, + json={ + "choices": [ + {"message": {"content": "Hi"}, "finish_reason": "stop"}, + ], + "usage": { + "prompt_tokens": 5, + "completion_tokens": 2, + "total_tokens": 7, + }, + "model": "test", }, - "model": "test", - }) + ) ) engine = _TestEngine() result = engine.generate( @@ -47,28 +50,35 @@ class TestOpenAICompatToolCalls: def test_with_tool_calls(self, respx_mock): """Extract tool_calls from OpenAI-format response.""" respx_mock.post("http://localhost:9999/v1/chat/completions").mock( - return_value=httpx.Response(200, json={ - "choices": [{ - "message": { - "content": None, - "tool_calls": [{ - "id": "call_abc", - "type": "function", - "function": { - "name": "calculator", - "arguments": '{"expression":"2+2"}', + return_value=httpx.Response( + 200, + json={ + "choices": [ + { + "message": { + "content": None, + "tool_calls": [ + { + "id": "call_abc", + "type": "function", + "function": { + "name": "calculator", + "arguments": '{"expression":"2+2"}', + }, + } + ], }, - }], + "finish_reason": "tool_calls", + } + ], + "usage": { + "prompt_tokens": 5, + "completion_tokens": 10, + "total_tokens": 15, }, - "finish_reason": "tool_calls", - }], - "usage": { - "prompt_tokens": 5, - "completion_tokens": 10, - "total_tokens": 15, + "model": "test", }, - "model": "test", - }) + ) ) engine = _TestEngine() result = engine.generate( @@ -89,11 +99,16 @@ class TestOpenAICompatToolCalls: def capture(request): captured["body"] = json.loads(request.content) - return httpx.Response(200, json={ - "choices": [{"message": {"content": "ok"}, "finish_reason": "stop"}], - "usage": {}, - "model": "test", - }) + return httpx.Response( + 200, + json={ + "choices": [ + {"message": {"content": "ok"}, "finish_reason": "stop"} + ], + "usage": {}, + "model": "test", + }, + ) respx_mock.post("http://localhost:9999/v1/chat/completions").mock( side_effect=capture, @@ -108,26 +123,33 @@ class TestOpenAICompatToolCalls: def test_multiple_tool_calls(self, respx_mock): respx_mock.post("http://localhost:9999/v1/chat/completions").mock( - return_value=httpx.Response(200, json={ - "choices": [{ - "message": { - "content": "", - "tool_calls": [ - { - "id": "c1", "type": "function", - "function": {"name": "a", "arguments": "{}"}, + return_value=httpx.Response( + 200, + json={ + "choices": [ + { + "message": { + "content": "", + "tool_calls": [ + { + "id": "c1", + "type": "function", + "function": {"name": "a", "arguments": "{}"}, + }, + { + "id": "c2", + "type": "function", + "function": {"name": "b", "arguments": "{}"}, + }, + ], }, - { - "id": "c2", "type": "function", - "function": {"name": "b", "arguments": "{}"}, - }, - ], - }, - "finish_reason": "tool_calls", - }], - "usage": {}, - "model": "test", - }) + "finish_reason": "tool_calls", + } + ], + "usage": {}, + "model": "test", + }, + ) ) engine = _TestEngine() result = engine.generate( @@ -145,12 +167,15 @@ class TestOpenAICompatToolCalls: class TestOllamaToolCalls: def test_no_tool_calls(self, respx_mock): respx_mock.post("http://localhost:11434/api/chat").mock( - return_value=httpx.Response(200, json={ - "message": {"content": "Hi"}, - "model": "test", - "prompt_eval_count": 5, - "eval_count": 2, - }) + return_value=httpx.Response( + 200, + json={ + "message": {"content": "Hi"}, + "model": "test", + "prompt_eval_count": 5, + "eval_count": 2, + }, + ) ) engine = OllamaEngine() result = engine.generate( @@ -161,20 +186,25 @@ class TestOllamaToolCalls: def test_with_tool_calls(self, respx_mock): respx_mock.post("http://localhost:11434/api/chat").mock( - return_value=httpx.Response(200, json={ - "message": { - "content": "", - "tool_calls": [{ - "function": { - "name": "calculator", - "arguments": '{"expression":"3*3"}', - }, - }], + return_value=httpx.Response( + 200, + json={ + "message": { + "content": "", + "tool_calls": [ + { + "function": { + "name": "calculator", + "arguments": '{"expression":"3*3"}', + }, + } + ], + }, + "model": "test", + "prompt_eval_count": 5, + "eval_count": 3, }, - "model": "test", - "prompt_eval_count": 5, - "eval_count": 3, - }) + ) ) engine = OllamaEngine() result = engine.generate( @@ -189,10 +219,13 @@ class TestOllamaToolCalls: def capture(request): captured["body"] = json.loads(request.content) - return httpx.Response(200, json={ - "message": {"content": "ok"}, - "model": "test", - }) + return httpx.Response( + 200, + json={ + "message": {"content": "ok"}, + "model": "test", + }, + ) respx_mock.post("http://localhost:11434/api/chat").mock(side_effect=capture) engine = OllamaEngine() @@ -205,23 +238,26 @@ class TestOllamaToolCalls: def test_dict_arguments_serialized_to_json(self, respx_mock): """Ollama returns arguments as dict — engine must serialize.""" - respx_mock.post( - "http://localhost:11434/api/chat" - ).mock( - return_value=httpx.Response(200, json={ - "message": { - "content": "", - "tool_calls": [{ - "function": { - "name": "calculator", - "arguments": {"expression": "3*3"}, - }, - }], + respx_mock.post("http://localhost:11434/api/chat").mock( + return_value=httpx.Response( + 200, + json={ + "message": { + "content": "", + "tool_calls": [ + { + "function": { + "name": "calculator", + "arguments": {"expression": "3*3"}, + }, + } + ], + }, + "model": "test", + "prompt_eval_count": 5, + "eval_count": 3, }, - "model": "test", - "prompt_eval_count": 5, - "eval_count": 3, - }) + ) ) engine = OllamaEngine() result = engine.generate( @@ -240,10 +276,13 @@ class TestOllamaToolCalls: def capture(request): captured["body"] = json.loads(request.content) - return httpx.Response(200, json={ - "message": {"content": "ok"}, - "model": "test", - }) + return httpx.Response( + 200, + json={ + "message": {"content": "ok"}, + "model": "test", + }, + ) respx_mock.post("http://localhost:11434/api/chat").mock(side_effect=capture) engine = OllamaEngine() diff --git a/tests/engine/test_structured_output.py b/tests/engine/test_structured_output.py index 1ae964da..4ba8cd0c 100644 --- a/tests/engine/test_structured_output.py +++ b/tests/engine/test_structured_output.py @@ -180,9 +180,7 @@ class TestAnthropicStructuredOutput: json_tool = [t for t in call_kwargs["tools"] if t["name"] == "json_output"][0] assert json_tool["input_schema"] == schema - def test_appends_to_existing_tools( - self, monkeypatch: pytest.MonkeyPatch - ) -> None: + def test_appends_to_existing_tools(self, monkeypatch: pytest.MonkeyPatch) -> None: engine, fake_client = self._make_engine(monkeypatch) rf = ResponseFormat() existing_tools = [ @@ -228,12 +226,8 @@ class TestGoogleStructuredOutput: text='{"answer": 42}', function_call=None, ) - fake_candidate = SimpleNamespace( - content=SimpleNamespace(parts=[fake_part]) - ) - fake_um = SimpleNamespace( - prompt_token_count=10, candidates_token_count=5 - ) + fake_candidate = SimpleNamespace(content=SimpleNamespace(parts=[fake_part])) + fake_um = SimpleNamespace(prompt_token_count=10, candidates_token_count=5) fake_resp = SimpleNamespace( candidates=[fake_candidate], usage_metadata=fake_um, @@ -242,9 +236,7 @@ class TestGoogleStructuredOutput: fake_client.models.generate_content.return_value = fake_resp return engine, fake_client - def test_json_mode_sets_mime_type( - self, monkeypatch: pytest.MonkeyPatch - ) -> None: + def test_json_mode_sets_mime_type(self, monkeypatch: pytest.MonkeyPatch) -> None: engine, fake_client = self._make_engine(monkeypatch) rf = ResponseFormat() @@ -361,9 +353,7 @@ class TestOllamaStructuredOutput: sent_payload = json.loads(route.calls[0].request.content) assert sent_payload["format"] == "json" - def test_no_format_without_response_format( - self, engine: OllamaEngine - ) -> None: + def test_no_format_without_response_format(self, engine: OllamaEngine) -> None: with respx.mock: route = respx.post("http://testhost:11434/api/chat").mock( return_value=httpx.Response( diff --git a/tests/engine/test_vllm_models.py b/tests/engine/test_vllm_models.py index fd11a7ec..3ac27b43 100644 --- a/tests/engine/test_vllm_models.py +++ b/tests/engine/test_vllm_models.py @@ -84,7 +84,9 @@ class TestVLLMGenerate: return_value=httpx.Response( 200, json=_openai_response( - content="", model=model_id, tool_calls=tool_calls, + content="", + model=model_id, + tool_calls=tool_calls, ), ) ) @@ -112,9 +114,7 @@ class TestVLLMGenerate: 200, json=_openai_response(content="Fallback reply", model=model_id) ) - respx_mock.post(f"{VLLM_HOST}/v1/chat/completions").mock( - side_effect=handler - ) + respx_mock.post(f"{VLLM_HOST}/v1/chat/completions").mock(side_effect=handler) result = engine.generate( [Message(role=Role.USER, content="Hello")], model=model_id, @@ -144,6 +144,7 @@ class TestVLLMGenerate: return tokens import asyncio + tokens = asyncio.run(collect()) assert tokens == ["Hello", " world"] diff --git a/tests/evals/scorers/test_browser_assistant.py b/tests/evals/scorers/test_browser_assistant.py index dfccb90a..49ecb11a 100644 --- a/tests/evals/scorers/test_browser_assistant.py +++ b/tests/evals/scorers/test_browser_assistant.py @@ -11,9 +11,7 @@ def _make_record(exact_facts=None, semantic_facts=None): if exact_facts: all_facts += [{"fact": f, "type": "exact"} for f in exact_facts] if semantic_facts: - all_facts += [ - {"fact": f, "type": "semantic"} for f in semantic_facts - ] + all_facts += [{"fact": f, "type": "semantic"} for f in semantic_facts] return EvalRecord( record_id="test-ba-1", problem="Research this topic.", @@ -76,9 +74,7 @@ def test_sources_with_url(): record = _make_record(exact_facts=["5432"]) scorer = BrowserAssistantScorer() - answer = ( - "Port 5432. See https://www.postgresql.org/docs/" - ) + answer = "Port 5432. See https://www.postgresql.org/docs/" is_correct, meta = scorer.score(record, answer) assert meta["sources_cited"] is True diff --git a/tests/evals/scorers/test_checklist.py b/tests/evals/scorers/test_checklist.py index 00fae49b..51870ee4 100644 --- a/tests/evals/scorers/test_checklist.py +++ b/tests/evals/scorers/test_checklist.py @@ -64,10 +64,7 @@ def test_checklist_scorer_all_pass(): def test_checklist_scorer_partial(): - response = ( - "1. yes — Redis is mentioned\n" - "2. no — Port number not found\n" - ) + response = "1. yes — Redis is mentioned\n2. no — Port number not found\n" backend = FakeJudgeBackend(response) scorer = ChecklistScorer(backend, "test-model") score, details = scorer.score_checklist( diff --git a/tests/evals/scorers/test_coding_assistant.py b/tests/evals/scorers/test_coding_assistant.py index 5e4d4465..cb2ba861 100644 --- a/tests/evals/scorers/test_coding_assistant.py +++ b/tests/evals/scorers/test_coding_assistant.py @@ -28,7 +28,8 @@ def test_all_tests_fixed(): "def test_neg(): assert add(-1, 1) == 0\n" ) record = _make_record( - buggy, tests, + buggy, + tests, ["test_basic", "test_neg"], ["test_zero"], ) @@ -49,7 +50,8 @@ def test_partial_fix(): "def test_zero(): assert div(0, 1) == 0.0\n" ) record = _make_record( - buggy, tests, + buggy, + tests, ["test_zero_div"], # not actually testable here ["test_normal", "test_zero"], ) diff --git a/tests/evals/scorers/test_doc_qa.py b/tests/evals/scorers/test_doc_qa.py index 2f1c2b7d..aa290145 100644 --- a/tests/evals/scorers/test_doc_qa.py +++ b/tests/evals/scorers/test_doc_qa.py @@ -23,10 +23,8 @@ def _make_record(required_facts): def test_all_facts_with_citations(): facts = [ - {"fact": "reclaims storage from dead tuples", - "source_doc_index": 0}, - {"fact": "must run periodically", - "source_doc_index": 0}, + {"fact": "reclaims storage from dead tuples", "source_doc_index": 0}, + {"fact": "must run periodically", "source_doc_index": 0}, ] record = _make_record(facts) scorer = DocQAScorer() @@ -45,8 +43,7 @@ def test_all_facts_with_citations(): def test_facts_without_citations(): facts = [ - {"fact": "reclaims storage from dead tuples", - "source_doc_index": 0}, + {"fact": "reclaims storage from dead tuples", "source_doc_index": 0}, ] record = _make_record(facts) scorer = DocQAScorer() @@ -70,10 +67,7 @@ def test_partial_facts(): record = _make_record(facts) scorer = DocQAScorer() - answer = ( - "VACUUM reclaims storage [Doc 1]. " - "It also prevents table bloat [Doc 1]." - ) + answer = "VACUUM reclaims storage [Doc 1]. It also prevents table bloat [Doc 1]." is_correct, meta = scorer.score(record, answer) assert meta["facts_found"] == 2 assert 0.6 <= meta["fact_score"] <= 0.7 diff --git a/tests/evals/scorers/test_security_scanner.py b/tests/evals/scorers/test_security_scanner.py index ee406aa3..e81ac39b 100644 --- a/tests/evals/scorers/test_security_scanner.py +++ b/tests/evals/scorers/test_security_scanner.py @@ -62,8 +62,7 @@ def test_partial_detection(): scorer = SecurityScannerScorer() answer = ( - "Found SQL injection in app.py. " - "Severity: critical. Use parameterized queries." + "Found SQL injection in app.py. Severity: critical. Use parameterized queries." ) is_correct, meta = scorer.score(record, answer) assert meta["vulns_found"] == 1 @@ -107,10 +106,7 @@ def test_vuln_type_aliases(): record = _make_record(vulns) scorer = SecurityScannerScorer() - answer = ( - "app.py has a cross-site scripting " - "vulnerability. Severity: high." - ) + answer = "app.py has a cross-site scripting vulnerability. Severity: high." is_correct, meta = scorer.score(record, answer) assert meta["vulns_found"] == 1 assert meta["detection_rate"] == 1.0 @@ -120,7 +116,8 @@ def test_no_vulnerabilities(): record = _make_record([]) scorer = SecurityScannerScorer() is_correct, meta = scorer.score( - record, "Everything looks clean.", + record, + "Everything looks clean.", ) assert is_correct is None assert meta["reason"] == "no_vulnerabilities_in_manifest" diff --git a/tests/evals/test_agentic_runner.py b/tests/evals/test_agentic_runner.py index 188b2387..122d3fe8 100644 --- a/tests/evals/test_agentic_runner.py +++ b/tests/evals/test_agentic_runner.py @@ -121,9 +121,7 @@ class TestAgenticRunner: def test_artifacts_saved(self, tmp_path): records = [MockRecord(record_id="r1", problem="test")] dataset = MockDataset(records) - runner = AgenticRunner( - agent=MockAgent(), dataset=dataset, run_dir=tmp_path - ) + runner = AgenticRunner(agent=MockAgent(), dataset=dataset, run_dir=tmp_path) self._run_async(runner.run()) arts = tmp_path / "artifacts" @@ -137,9 +135,7 @@ class TestAgenticRunner: """Verify timeout is stored and runner accepts the parameter.""" records = [MockRecord(record_id="r1", problem="test")] dataset = MockDataset(records) - runner = AgenticRunner( - agent=MockAgent(), dataset=dataset, query_timeout=30.0 - ) + runner = AgenticRunner(agent=MockAgent(), dataset=dataset, query_timeout=30.0) assert runner._query_timeout == 30.0 diff --git a/tests/evals/test_ama_bench.py b/tests/evals/test_ama_bench.py index 51b05282..8c9affbf 100644 --- a/tests/evals/test_ama_bench.py +++ b/tests/evals/test_ama_bench.py @@ -91,6 +91,7 @@ class TestAMABenchDataset: def test_question_types_mapped(self) -> None: from openjarvis.evals.datasets.ama_bench import _QUESTION_TYPE_TO_SUBJECT + assert _QUESTION_TYPE_TO_SUBJECT["A"] == "recall" assert _QUESTION_TYPE_TO_SUBJECT["B"] == "causal_inference" assert _QUESTION_TYPE_TO_SUBJECT["C"] == "state_updating" @@ -111,8 +112,10 @@ class TestAMABenchScorer: def test_empty_response(self) -> None: s = AMABenchScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="test-1", problem="question", - reference="answer", category="agentic", + record_id="test-1", + problem="question", + reference="answer", + category="agentic", ) is_correct, meta = s.score(record, "") assert is_correct is False @@ -175,14 +178,10 @@ class TestJudgeParsing: assert _parse_judge_label("yes.") == "yes" def test_with_thinking_tags(self) -> None: - assert _parse_judge_label( - "Let me check...yes" - ) == "yes" + assert _parse_judge_label("Let me check...yes") == "yes" def test_with_thinking_tags_no(self) -> None: - assert _parse_judge_label( - "The answer doesn't match\nno" - ) == "no" + assert _parse_judge_label("The answer doesn't match\nno") == "no" def test_multiline(self) -> None: assert _parse_judge_label(" \n yes\n") == "yes" @@ -216,14 +215,17 @@ class TestTokenF1: class TestAMABenchCLI: def test_in_benchmarks_dict(self) -> None: from openjarvis.evals.cli import BENCHMARKS + assert "ama-bench" in BENCHMARKS def test_build_dataset(self) -> None: from openjarvis.evals.cli import _build_dataset + ds = _build_dataset("ama-bench") assert ds.dataset_id == "ama-bench" def test_build_scorer(self) -> None: from openjarvis.evals.cli import _build_scorer + s = _build_scorer("ama-bench", _mock_backend(), "test-model") assert s.scorer_id == "ama-bench" diff --git a/tests/evals/test_benchmark_datasets.py b/tests/evals/test_benchmark_datasets.py index 62753e97..ee5e73a3 100644 --- a/tests/evals/test_benchmark_datasets.py +++ b/tests/evals/test_benchmark_datasets.py @@ -24,84 +24,98 @@ class TestDatasetInstantiation: def test_supergpqa(self) -> None: from openjarvis.evals.datasets.supergpqa import SuperGPQADataset + ds = SuperGPQADataset() assert ds.dataset_id == "supergpqa" assert ds.dataset_name == "SuperGPQA" def test_gpqa(self) -> None: from openjarvis.evals.datasets.gpqa import GPQADataset + ds = GPQADataset() assert ds.dataset_id == "gpqa" assert ds.dataset_name == "GPQA" def test_mmlu_pro(self) -> None: from openjarvis.evals.datasets.mmlu_pro import MMLUProDataset + ds = MMLUProDataset() assert ds.dataset_id == "mmlu-pro" assert ds.dataset_name == "MMLU-Pro" def test_math500(self) -> None: from openjarvis.evals.datasets.math500 import MATH500Dataset + ds = MATH500Dataset() assert ds.dataset_id == "math500" assert ds.dataset_name == "MATH-500" def test_natural_reasoning(self) -> None: from openjarvis.evals.datasets.natural_reasoning import NaturalReasoningDataset + ds = NaturalReasoningDataset() assert ds.dataset_id == "natural-reasoning" assert ds.dataset_name == "Natural Reasoning" def test_hle(self) -> None: from openjarvis.evals.datasets.hle import HLEDataset + ds = HLEDataset() assert ds.dataset_id == "hle" assert ds.dataset_name == "HLE" def test_simpleqa(self) -> None: from openjarvis.evals.datasets.simpleqa import SimpleQADataset + ds = SimpleQADataset() assert ds.dataset_id == "simpleqa" assert ds.dataset_name == "SimpleQA" def test_wildchat(self) -> None: from openjarvis.evals.datasets.wildchat import WildChatDataset + ds = WildChatDataset() assert ds.dataset_id == "wildchat" assert ds.dataset_name == "WildChat" def test_ipw(self) -> None: from openjarvis.evals.datasets.ipw_mixed import IPWDataset + ds = IPWDataset() assert ds.dataset_id == "ipw" assert ds.dataset_name == "IPW" def test_gaia(self) -> None: from openjarvis.evals.datasets.gaia import GAIADataset + ds = GAIADataset() assert ds.dataset_id == "gaia" assert ds.dataset_name == "GAIA" def test_frames(self) -> None: from openjarvis.evals.datasets.frames import FRAMESDataset + ds = FRAMESDataset() assert ds.dataset_id == "frames" assert ds.dataset_name == "FRAMES" def test_swebench(self) -> None: from openjarvis.evals.datasets.swebench import SWEBenchDataset + ds = SWEBenchDataset() assert ds.dataset_id == "swebench" assert ds.dataset_name == "SWE-bench" def test_swefficiency(self) -> None: from openjarvis.evals.datasets.swefficiency import SWEfficiencyDataset + ds = SWEfficiencyDataset() assert ds.dataset_id == "swefficiency" assert ds.dataset_name == "SWEfficiency" def test_terminalbench(self) -> None: from openjarvis.evals.datasets.terminalbench import TerminalBenchDataset + ds = TerminalBenchDataset() assert ds.dataset_id == "terminalbench" assert ds.dataset_name == "TerminalBench" @@ -110,6 +124,7 @@ class TestDatasetInstantiation: from openjarvis.evals.datasets.terminalbench_native import ( TerminalBenchNativeDataset, ) + ds = TerminalBenchNativeDataset() assert ds.dataset_id == "terminalbench-native" assert ds.dataset_name == "TerminalBench Native" @@ -132,56 +147,67 @@ class TestScorerInstantiation: def test_supergpqa_scorer(self) -> None: from openjarvis.evals.scorers.supergpqa_mcq import SuperGPQAScorer + s = SuperGPQAScorer(_mock_backend(), "test-model") assert s.scorer_id == "supergpqa" def test_gpqa_scorer(self) -> None: from openjarvis.evals.scorers.gpqa_mcq import GPQAScorer + s = GPQAScorer(_mock_backend(), "test-model") assert s.scorer_id == "gpqa" def test_mmlu_pro_scorer(self) -> None: from openjarvis.evals.scorers.mmlu_pro_mcq import MMLUProScorer + s = MMLUProScorer(_mock_backend(), "test-model") assert s.scorer_id == "mmlu-pro" def test_reasoning_judge_scorer(self) -> None: from openjarvis.evals.scorers.reasoning_judge import ReasoningJudgeScorer + s = ReasoningJudgeScorer(_mock_backend(), "test-model") assert s.scorer_id == "reasoning_judge" def test_hle_scorer(self) -> None: from openjarvis.evals.scorers.hle_judge import HLEScorer + s = HLEScorer(_mock_backend(), "test-model") assert s.scorer_id == "hle" def test_simpleqa_scorer(self) -> None: from openjarvis.evals.scorers.simpleqa_judge import SimpleQAScorer + s = SimpleQAScorer(_mock_backend(), "test-model") assert s.scorer_id == "simpleqa" def test_wildchat_scorer(self) -> None: from openjarvis.evals.scorers.wildchat_judge import WildChatScorer + s = WildChatScorer(_mock_backend(), "test-model") assert s.scorer_id == "wildchat" def test_ipw_mixed_scorer(self) -> None: from openjarvis.evals.scorers.ipw_mixed import IPWMixedScorer + s = IPWMixedScorer(_mock_backend(), "test-model") assert s.scorer_id == "ipw" def test_gaia_scorer(self) -> None: from openjarvis.evals.scorers.gaia_exact import GAIAScorer + s = GAIAScorer(_mock_backend(), "test-model") assert s.scorer_id == "gaia" def test_frames_scorer(self) -> None: from openjarvis.evals.scorers.frames_judge import FRAMESScorer + s = FRAMESScorer(_mock_backend(), "test-model") assert s.scorer_id == "frames" def test_swebench_scorer(self) -> None: from openjarvis.evals.scorers.swebench_structural import SWEBenchScorer + s = SWEBenchScorer(_mock_backend(), "test-model") assert s.scorer_id == "swebench" @@ -189,11 +215,13 @@ class TestScorerInstantiation: from openjarvis.evals.scorers.swefficiency_structural import ( SWEfficiencyScorer, ) + s = SWEfficiencyScorer(_mock_backend(), "test-model") assert s.scorer_id == "swefficiency" def test_terminalbench_scorer(self) -> None: from openjarvis.evals.scorers.terminalbench_judge import TerminalBenchScorer + s = TerminalBenchScorer(_mock_backend(), "test-model") assert s.scorer_id == "terminalbench" @@ -201,6 +229,7 @@ class TestScorerInstantiation: from openjarvis.evals.scorers.terminalbench_native_structural import ( TerminalBenchNativeScorer, ) + s = TerminalBenchNativeScorer(_mock_backend(), "test-model") assert s.scorer_id == "terminalbench-native" @@ -211,9 +240,21 @@ class TestScorerInstantiation: ALL_BENCHMARKS = [ - "supergpqa", "gpqa", "mmlu-pro", "math500", "natural-reasoning", - "hle", "simpleqa", "wildchat", "ipw", "gaia", "frames", - "swebench", "swefficiency", "terminalbench", "terminalbench-native", + "supergpqa", + "gpqa", + "mmlu-pro", + "math500", + "natural-reasoning", + "hle", + "simpleqa", + "wildchat", + "ipw", + "gaia", + "frames", + "swebench", + "swefficiency", + "terminalbench", + "terminalbench-native", ] @@ -223,6 +264,7 @@ class TestCLIFactories: @pytest.mark.parametrize("benchmark", ALL_BENCHMARKS) def test_build_dataset(self, benchmark: str) -> None: from openjarvis.evals.cli import _build_dataset + ds = _build_dataset(benchmark) assert ds is not None assert hasattr(ds, "load") @@ -232,6 +274,7 @@ class TestCLIFactories: @pytest.mark.parametrize("benchmark", ALL_BENCHMARKS) def test_build_scorer(self, benchmark: str) -> None: from openjarvis.evals.cli import _build_scorer + scorer = _build_scorer(benchmark, _mock_backend(), "test-model") assert scorer is not None assert hasattr(scorer, "score") @@ -240,6 +283,7 @@ class TestCLIFactories: import click from openjarvis.evals.cli import _build_dataset + with pytest.raises(click.ClickException, match="Unknown benchmark"): _build_dataset("nonexistent") @@ -247,6 +291,7 @@ class TestCLIFactories: import click from openjarvis.evals.cli import _build_scorer + with pytest.raises(click.ClickException, match="Unknown benchmark"): _build_scorer("nonexistent", _mock_backend(), "test-model") @@ -261,11 +306,13 @@ class TestConfigBenchmarks: def test_all_benchmarks_known(self) -> None: from openjarvis.evals.core.config import KNOWN_BENCHMARKS + for b in ALL_BENCHMARKS: assert b in KNOWN_BENCHMARKS, f"{b} missing from KNOWN_BENCHMARKS" def test_benchmarks_count(self) -> None: from openjarvis.evals.core.config import KNOWN_BENCHMARKS + assert len(KNOWN_BENCHMARKS) == 25 @@ -280,9 +327,12 @@ class TestStructuralScorers: def test_swebench_empty_response(self) -> None: from openjarvis.evals.core.types import EvalRecord from openjarvis.evals.scorers.swebench_structural import SWEBenchScorer + scorer = SWEBenchScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="swe-1", problem="Fix bug", reference="patch", + record_id="swe-1", + problem="Fix bug", + reference="patch", category="agentic", ) is_correct, meta = scorer.score(record, "") @@ -292,9 +342,12 @@ class TestStructuralScorers: def test_swebench_with_diff(self) -> None: from openjarvis.evals.core.types import EvalRecord from openjarvis.evals.scorers.swebench_structural import SWEBenchScorer + scorer = SWEBenchScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="swe-2", problem="Fix bug", reference="patch", + record_id="swe-2", + problem="Fix bug", + reference="patch", category="agentic", ) answer = "--- a/file.py\n+++ b/file.py\n@@ -1 +1 @@\n-old\n+new" @@ -308,10 +361,13 @@ class TestStructuralScorers: from openjarvis.evals.scorers.terminalbench_native_structural import ( TerminalBenchNativeScorer, ) + scorer = TerminalBenchNativeScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="tb-1", problem="Run command", - reference="", category="agentic", + record_id="tb-1", + problem="Run command", + reference="", + category="agentic", ) is_correct, meta = scorer.score(record, "some output") assert is_correct is None @@ -322,10 +378,13 @@ class TestStructuralScorers: from openjarvis.evals.scorers.terminalbench_native_structural import ( TerminalBenchNativeScorer, ) + scorer = TerminalBenchNativeScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="tb-2", problem="Run command", - reference="", category="agentic", + record_id="tb-2", + problem="Run command", + reference="", + category="agentic", metadata={"is_resolved": True}, ) is_correct, meta = scorer.score(record, "output") diff --git a/tests/evals/test_deepplanning.py b/tests/evals/test_deepplanning.py index d4db7903..f0b9482c 100644 --- a/tests/evals/test_deepplanning.py +++ b/tests/evals/test_deepplanning.py @@ -63,8 +63,10 @@ class TestDeepPlanningScorer: def test_empty_response(self) -> None: s = DeepPlanningScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="dp-3", problem="task", - reference="answer", category="agentic", + record_id="dp-3", + problem="task", + reference="answer", + category="agentic", ) is_correct, meta = s.score(record, "") assert is_correct is False @@ -74,14 +76,17 @@ class TestDeepPlanningScorer: class TestDeepPlanningCLI: def test_in_benchmarks(self) -> None: from openjarvis.evals.cli import BENCHMARKS + assert "deepplanning" in BENCHMARKS def test_build_dataset(self) -> None: from openjarvis.evals.cli import _build_dataset + ds = _build_dataset("deepplanning") assert ds.dataset_id == "deepplanning" def test_build_scorer(self) -> None: from openjarvis.evals.cli import _build_scorer + s = _build_scorer("deepplanning", _mock_backend(), "test-model") assert s.scorer_id == "deepplanning" diff --git a/tests/evals/test_display.py b/tests/evals/test_display.py index 320f9660..1d307582 100644 --- a/tests/evals/test_display.py +++ b/tests/evals/test_display.py @@ -45,8 +45,14 @@ def _make_summary(**overrides) -> RunSummary: def _make_metric_stats(**kw) -> MetricStats: defaults = dict( - mean=1.0, median=0.9, min=0.1, max=2.5, - std=0.3, p90=2.0, p95=2.2, p99=2.4, + mean=1.0, + median=0.9, + min=0.1, + max=2.5, + std=0.3, + p90=2.0, + p95=2.2, + p99=2.4, ) defaults.update(kw) return MetricStats(**defaults) @@ -189,7 +195,8 @@ class TestPrintCompletion: summary = _make_summary() console, buf = _make_console() print_completion( - console, summary, + console, + summary, output_path=Path("results/test.jsonl"), traces_dir=Path("results/traces/supergpqa_qwen3-8b"), ) diff --git a/tests/evals/test_episode_mode.py b/tests/evals/test_episode_mode.py index 4a139267..c4d1214d 100644 --- a/tests/evals/test_episode_mode.py +++ b/tests/evals/test_episode_mode.py @@ -37,13 +37,17 @@ class TestDatasetProviderEpisodes: class TestRunConfigEpisodeMode: def test_episode_mode_field(self) -> None: from openjarvis.evals.core.types import RunConfig + cfg = RunConfig( - benchmark="test", backend="test", model="test", + benchmark="test", + backend="test", + model="test", episode_mode=True, ) assert cfg.episode_mode is True def test_episode_mode_default_false(self) -> None: from openjarvis.evals.core.types import RunConfig + cfg = RunConfig(benchmark="test", backend="test", model="test") assert cfg.episode_mode is False diff --git a/tests/evals/test_export.py b/tests/evals/test_export.py index 40605236..81df1e8e 100644 --- a/tests/evals/test_export.py +++ b/tests/evals/test_export.py @@ -17,25 +17,27 @@ from openjarvis.evals.core.trace import QueryTrace, TurnTrace def _make_traces(n=3): traces = [] for i in range(n): - traces.append(QueryTrace( - query_id=f"q{i:04d}", - workload_type="test", - query_text=f"Question {i}", - response_text=f"Answer {i}", - turns=[ - TurnTrace( - turn_index=0, - input_tokens=100 + i * 10, - output_tokens=50 + i * 5, - wall_clock_s=1.0 + i * 0.5, - gpu_energy_joules=5.0 + i, - cost_usd=0.01, - ), - ], - total_wall_clock_s=1.0 + i * 0.5, - completed=True, - is_resolved=i % 2 == 0, - )) + traces.append( + QueryTrace( + query_id=f"q{i:04d}", + workload_type="test", + query_text=f"Question {i}", + response_text=f"Answer {i}", + turns=[ + TurnTrace( + turn_index=0, + input_tokens=100 + i * 10, + output_tokens=50 + i * 5, + wall_clock_s=1.0 + i * 0.5, + gpu_energy_joules=5.0 + i, + cost_usd=0.01, + ), + ], + total_wall_clock_s=1.0 + i * 0.5, + completed=True, + is_resolved=i % 2 == 0, + ) + ) return traces @@ -89,11 +91,20 @@ class TestExportSummaryJson: summary = json.loads(path.read_text()) stats = summary["statistics"] expected_stat_keys = { - "wall_clock_s", "gpu_energy_joules", "cpu_energy_joules", - "gpu_power_watts", "cpu_power_watts", - "input_tokens", "output_tokens", "total_tokens", - "throughput_tokens_per_sec", "energy_per_token_joules", - "cost_usd", "turns", "tool_calls", "mbu_avg_pct", + "wall_clock_s", + "gpu_energy_joules", + "cpu_energy_joules", + "gpu_power_watts", + "cpu_power_watts", + "input_tokens", + "output_tokens", + "total_tokens", + "throughput_tokens_per_sec", + "energy_per_token_joules", + "cost_usd", + "turns", + "tool_calls", + "mbu_avg_pct", } assert set(stats.keys()) == expected_stat_keys @@ -152,8 +163,10 @@ class TestComputeEfficiency: def test_no_scored_traces(self): traces = [ QueryTrace( - query_id="q0", workload_type="test", - completed=True, is_resolved=None, + query_id="q0", + workload_type="test", + completed=True, + is_resolved=None, ), ] result = _compute_efficiency(traces, 5.0, 1.0) @@ -169,8 +182,10 @@ class TestComputeEfficiency: def test_with_gpu_power(self): traces = [ QueryTrace( - query_id="q0", workload_type="test", - completed=True, is_resolved=True, + query_id="q0", + workload_type="test", + completed=True, + is_resolved=True, query_gpu_power_avg_watts=100.0, ), ] @@ -351,35 +366,37 @@ class TestActionEnergyBreakdown: def test_action_energy_summary_in_export(self, tmp_path): traces = [] for i in range(2): - traces.append(QueryTrace( - query_id=f"q{i:04d}", - workload_type="test", - turns=[ - TurnTrace( - turn_index=0, - input_tokens=100, - output_tokens=50, - wall_clock_s=2.0, - gpu_energy_joules=5.0, - action_energy_breakdown=[ - { - "action_type": "lm_inference", - "duration_s": 1.5, - "gpu_energy_joules": 4.0, - "cpu_energy_joules": 0.3, - }, - { - "action_type": "tool_call:search", - "duration_s": 0.5, - "gpu_energy_joules": 1.0, - "cpu_energy_joules": 0.1, - }, - ], - ), - ], - total_wall_clock_s=2.0, - completed=True, - )) + traces.append( + QueryTrace( + query_id=f"q{i:04d}", + workload_type="test", + turns=[ + TurnTrace( + turn_index=0, + input_tokens=100, + output_tokens=50, + wall_clock_s=2.0, + gpu_energy_joules=5.0, + action_energy_breakdown=[ + { + "action_type": "lm_inference", + "duration_s": 1.5, + "gpu_energy_joules": 4.0, + "cpu_energy_joules": 0.3, + }, + { + "action_type": "tool_call:search", + "duration_s": 0.5, + "gpu_energy_joules": 1.0, + "cpu_energy_joules": 0.1, + }, + ], + ), + ], + total_wall_clock_s=2.0, + completed=True, + ) + ) path = tmp_path / "summary.json" export_summary_json(traces, {}, path) summary = json.loads(path.read_text()) diff --git a/tests/evals/test_lifelong_agent.py b/tests/evals/test_lifelong_agent.py index c26dbc99..bf10f964 100644 --- a/tests/evals/test_lifelong_agent.py +++ b/tests/evals/test_lifelong_agent.py @@ -55,13 +55,18 @@ def _db_record(direct=None, md5=None, sql="SELECT * FROM users", skills=None): answer_info = {"direct": direct, "md5": md5, "sql": sql} answer_type = "md5" if md5 else "direct" return EvalRecord( - record_id="test-1", problem="task", + record_id="test-1", + problem="task", reference=json.dumps(answer_info), - category="agentic", subject=f"db_{answer_type}", + category="agentic", + subject=f"db_{answer_type}", metadata={ - "answer_info": answer_info, "answer_type": answer_type, - "skills": skills or [], "table_info": _TABLE_INFO, - "table_name": "users", "subset": "db_bench", + "answer_info": answer_info, + "answer_type": answer_type, + "skills": skills or [], + "table_info": _TABLE_INFO, + "table_name": "users", + "subset": "db_bench", "sample_index": 0, }, ) @@ -69,9 +74,11 @@ def _db_record(direct=None, md5=None, sql="SELECT * FROM users", skills=None): def _kg_record(answer_list=None, skills=None, action_list=None): return EvalRecord( - record_id="test-kg-1", problem="question", + record_id="test-kg-1", + problem="question", reference=json.dumps(answer_list or []), - category="agentic", subject="knowledge_graph", + category="agentic", + subject="knowledge_graph", metadata={ "subset": "knowledge_graph", "question": "What is the answer?", @@ -86,9 +93,11 @@ def _kg_record(answer_list=None, skills=None, action_list=None): def _os_record(): return EvalRecord( - record_id="test-os-1", problem="task", + record_id="test-os-1", + problem="task", reference="{}", - category="agentic", subject="os_interaction", + category="agentic", + subject="os_interaction", metadata={ "subset": "os_interaction", "instruction": "Create a file", @@ -113,6 +122,7 @@ def _os_record(): # DB building # --------------------------------------------------------------------------- + class TestBuildDB: def test_creates_table_with_rows(self) -> None: conn = build_db(_TABLE_INFO) @@ -174,6 +184,7 @@ class TestTableStateComparison: # SQL extraction (original's Action: Operation format) # --------------------------------------------------------------------------- + class TestExtractSQL: def test_action_operation_format(self) -> None: text = "Action: Operation\n```sql\nSELECT * FROM users;\n```" @@ -211,6 +222,7 @@ class TestExtractSQL: # Text answer parsing (DirectTypeAnswerValidator format) # --------------------------------------------------------------------------- + class TestTextAnswerParsing: def test_tuple_list(self) -> None: text = "Final Answer: [(1, 'Alice', 95.5), (2, 'Bob', 87.0)]" @@ -249,6 +261,7 @@ class TestTextAnswerParsing: # DB scorer: direct (SELECT) # --------------------------------------------------------------------------- + class TestScorerDBDirect: def test_correct_sql(self) -> None: s = LifelongAgentScorer() @@ -265,7 +278,8 @@ class TestScorerDBDirect: direct=[[1, "Alice", 95.5], [2, "Bob", 87.0], [3, "Carol", 92.3]], ) ok, meta = s.score( - r, "Action: Operation\n```sql\nSELECT * FROM users\n```", + r, + "Action: Operation\n```sql\nSELECT * FROM users\n```", ) assert ok is True @@ -296,7 +310,8 @@ class TestScorerDBDirect: def test_no_sql_in_response(self) -> None: ok, meta = LifelongAgentScorer().score( - _db_record(direct=[[1]]), "I don't know", + _db_record(direct=[[1]]), + "I don't know", ) assert ok is False @@ -304,7 +319,8 @@ class TestScorerDBDirect: s = LifelongAgentScorer() r = _db_record(direct=[[1, "Alice", 95.5]]) ok, meta = s.score( - r, "Action: Answer\nFinal Answer: [(1, 'Alice', 95.5)]", + r, + "Action: Answer\nFinal Answer: [(1, 'Alice', 95.5)]", ) assert ok is True assert meta["strategy"] == "text_answer_parsing" @@ -322,6 +338,7 @@ class TestScorerDBDirect: # DB scorer: md5 (INSERT/UPDATE/DELETE) # --------------------------------------------------------------------------- + class TestScorerDBMD5: def test_correct_insert(self) -> None: s = LifelongAgentScorer() @@ -363,6 +380,7 @@ class TestScorerDBMD5: # KG scorer # --------------------------------------------------------------------------- + class TestScorerKG: def test_single_shot_unscorable(self) -> None: """KG tasks should be unscorable in single-shot mode.""" @@ -399,6 +417,7 @@ class TestScorerKG: # OS scorer # --------------------------------------------------------------------------- + class TestScorerOS: def test_returns_scorable_status(self) -> None: s = LifelongAgentScorer() @@ -413,6 +432,7 @@ class TestScorerOS: # KG answer extraction # --------------------------------------------------------------------------- + class TestExtractKGAnswers: def test_entity_id(self) -> None: assert extract_kg_answers("Final Answer: m.02h8b9t") == ["m.02h8b9t"] @@ -441,6 +461,7 @@ class TestExtractKGAnswers: # Bash command extraction # --------------------------------------------------------------------------- + class TestExtractBashCommands: def test_act_format(self) -> None: text = "Act: ```bash\nls -la /tmp\n```" @@ -469,6 +490,7 @@ class TestExtractBashCommands: # Value comparison # --------------------------------------------------------------------------- + class TestValueComparison: def test_int(self) -> None: assert values_match(42, 42) @@ -512,6 +534,7 @@ class TestTupleComparison: # Episode grouping # --------------------------------------------------------------------------- + class TestEpisodeGrouping: def test_single_subset_episode(self) -> None: ds = LifelongAgentDataset(subset="db_bench") @@ -536,14 +559,18 @@ class TestEpisodeGrouping: ds._records = [ EvalRecord( record_id="lifelong-db-0", - problem="task", reference="{}", - category="agentic", subject="db_direct", + problem="task", + reference="{}", + category="agentic", + subject="db_direct", metadata={"subset": "db_bench", "sample_index": 0}, ), EvalRecord( record_id="lifelong-kg-0", - problem="task", reference="{}", - category="agentic", subject="knowledge_graph", + problem="task", + reference="{}", + category="agentic", + subject="knowledge_graph", metadata={"subset": "knowledge_graph", "sample_index": 0}, ), ] @@ -555,8 +582,10 @@ class TestEpisodeGrouping: ds._records = [ EvalRecord( record_id=f"lifelong-db-{i}", - problem="task", reference="{}", - category="agentic", subject="db_direct", + problem="task", + reference="{}", + category="agentic", + subject="db_direct", metadata={"subset": "db_bench", "sample_index": i}, ) for i in range(5) @@ -571,6 +600,7 @@ class TestEpisodeGrouping: # Dataset # --------------------------------------------------------------------------- + class TestDataset: def test_instantiation_default(self) -> None: ds = LifelongAgentDataset() @@ -603,6 +633,7 @@ class TestDataset: # Multi-turn environments # --------------------------------------------------------------------------- + class TestDBEnvironment: def test_multi_turn_select(self) -> None: """DB environment should handle multi-turn SQL interaction.""" @@ -643,15 +674,15 @@ class TestDBEnvironment: from openjarvis.evals.environments.lifelong_agent_env import DBEnvironment record = _db_record( - md5="x", sql="INSERT INTO users VALUES (4, 'Dave', 88.0)", + md5="x", + sql="INSERT INTO users VALUES (4, 'Dave', 88.0)", ) env = DBEnvironment(use_mysql=False) env.reset(record) # Agent executes the correct INSERT obs, done = env.step( - "Action: Operation\n```sql\n" - "INSERT INTO users VALUES (4, 'Dave', 88.0)\n```" + "Action: Operation\n```sql\nINSERT INTO users VALUES (4, 'Dave', 88.0)\n```" ) assert not done assert "successfully" in obs.lower() or "Result" in obs @@ -814,20 +845,24 @@ class TestOSEnvironment: # CLI wiring # --------------------------------------------------------------------------- + class TestCLI: def test_in_benchmarks(self) -> None: from openjarvis.evals.cli import BENCHMARKS + assert "lifelong-agent" in BENCHMARKS assert BENCHMARKS["lifelong-agent"]["category"] == "agentic" def test_build_dataset(self) -> None: from openjarvis.evals.cli import _build_dataset + ds = _build_dataset("lifelong-agent") assert ds.dataset_id == "lifelong-agent" assert hasattr(ds, "create_task_env") def test_build_scorer(self) -> None: from openjarvis.evals.cli import _build_scorer + s = _build_scorer("lifelong-agent", None, "test-model") assert s.scorer_id == "lifelong-agent" @@ -836,9 +871,11 @@ class TestCLI: # Runner episode_mode integration # --------------------------------------------------------------------------- + class TestRunnerEpisodeMode: def test_episode_mode_field_exists(self) -> None: from openjarvis.evals.core.types import RunConfig + config = RunConfig( benchmark="lifelong-agent", backend="jarvis-direct", @@ -849,6 +886,7 @@ class TestRunnerEpisodeMode: def test_runner_has_episode_mode_method(self) -> None: from openjarvis.evals.core.runner import EvalRunner + assert hasattr(EvalRunner, "_run_episode_mode") assert hasattr(EvalRunner, "_process_interactive") assert hasattr(EvalRunner, "_inject_examples") @@ -858,8 +896,11 @@ class TestRunnerEpisodeMode: from openjarvis.evals.core.runner import EvalRunner record = EvalRecord( - record_id="test", problem="What is 2+2?", - reference="4", category="reasoning", subject="math", + record_id="test", + problem="What is 2+2?", + reference="4", + category="reasoning", + subject="math", metadata={}, ) # Call the static-ish method @@ -872,8 +913,11 @@ class TestRunnerEpisodeMode: from openjarvis.evals.core.runner import EvalRunner record = EvalRecord( - record_id="test", problem="What is 2+2?", - reference="4", category="reasoning", subject="math", + record_id="test", + problem="What is 2+2?", + reference="4", + category="reasoning", + subject="math", metadata={}, ) examples = [{"problem": "What is 1+1?", "answer": "2"}] @@ -888,20 +932,25 @@ class TestRunnerEpisodeMode: from openjarvis.evals.core.runner import EvalRunner record = EvalRecord( - record_id="test", problem="What is 3+3?", - reference="6", category="reasoning", subject="math", + record_id="test", + problem="What is 3+3?", + reference="6", + category="reasoning", + subject="math", metadata={}, ) - examples = [{ - "problem": "What is 1+1?", - "answer": "2", - "interaction_history": [ - {"role": "user", "content": "What is 1+1?"}, - {"role": "assistant", "content": "Action: compute(1+1)"}, - {"role": "user", "content": "Result: 2"}, - {"role": "assistant", "content": "Final Answer: 2"}, - ], - }] + examples = [ + { + "problem": "What is 1+1?", + "answer": "2", + "interaction_history": [ + {"role": "user", "content": "What is 1+1?"}, + {"role": "assistant", "content": "Action: compute(1+1)"}, + {"role": "user", "content": "Result: 2"}, + {"role": "assistant", "content": "Final Answer: 2"}, + ], + } + ] runner = EvalRunner.__new__(EvalRunner) result = runner._inject_examples(record, examples) assert "Previously Completed Tasks" in result.problem @@ -912,6 +961,7 @@ class TestRunnerEpisodeMode: def test_max_prior_examples_constant(self) -> None: """Runner should have a FIFO buffer size matching original default.""" from openjarvis.evals.core.runner import EvalRunner + assert EvalRunner._MAX_PRIOR_EXAMPLES == 3 @@ -919,14 +969,13 @@ class TestRunnerEpisodeMode: # KG variable reference resolution # --------------------------------------------------------------------------- + class TestKGVariableReference: def test_variable_ref_in_scorer(self) -> None: """Scorer should handle Final Answer: #N format.""" # When a variable ref is given and entity IDs are elsewhere in text result = extract_kg_answers( - "I found the answer.\n" - "The entity m.02h8b9t matches.\n" - "Final Answer: #2" + "I found the answer.\nThe entity m.02h8b9t matches.\nFinal Answer: #2" ) assert "m.02h8b9t" in result @@ -962,8 +1011,7 @@ class TestKGVariableReference: def test_variable_ref_with_var_keyword(self) -> None: """Should handle 'Final Answer: Variable #2' format.""" result = extract_kg_answers( - "Based on my analysis, m.001 is the answer.\n" - "Final Answer: Variable #2" + "Based on my analysis, m.001 is the answer.\nFinal Answer: Variable #2" ) assert "m.001" in result @@ -972,12 +1020,11 @@ class TestKGVariableReference: # OS action format # --------------------------------------------------------------------------- + class TestOSActionFormat: def test_original_format_act_bash(self) -> None: """Should parse original format: Act: bash\\n```bash\\n...\\n```""" - cmds = _extract_bash_commands( - "Act: bash\n```bash\nls -la /tmp\n```" - ) + cmds = _extract_bash_commands("Act: bash\n```bash\nls -la /tmp\n```") assert len(cmds) == 1 assert "ls -la" in cmds[0] @@ -995,12 +1042,14 @@ class TestOSActionFormat: # Per-subset max turns # --------------------------------------------------------------------------- + class TestMaxTurns: def test_db_max_turns(self) -> None: from openjarvis.evals.environments.lifelong_agent_env import ( MAX_TURNS_DB, DBEnvironment, ) + env = DBEnvironment(use_mysql=False) assert env.max_turns == MAX_TURNS_DB assert env.max_turns == 3 @@ -1010,6 +1059,7 @@ class TestMaxTurns: MAX_TURNS_KG, KGEnvironment, ) + env = KGEnvironment() assert env.max_turns == MAX_TURNS_KG assert env.max_turns == 15 @@ -1019,12 +1069,14 @@ class TestMaxTurns: MAX_TURNS_OS, OSEnvironment, ) + env = OSEnvironment() assert env.max_turns == MAX_TURNS_OS assert env.max_turns == 5 def test_base_default(self) -> None: from openjarvis.evals.environments.base import TaskEnvironment + # Can't instantiate ABC, but verify the property exists assert hasattr(TaskEnvironment, "max_turns") @@ -1033,6 +1085,7 @@ class TestMaxTurns: # Numeric tolerance # --------------------------------------------------------------------------- + class TestNumericTolerance: def test_abs_tol_near_zero(self) -> None: """abs_tol=1e-6 should make very small values match zero.""" diff --git a/tests/evals/test_loghub.py b/tests/evals/test_loghub.py index 06812f30..4b18e891 100644 --- a/tests/evals/test_loghub.py +++ b/tests/evals/test_loghub.py @@ -101,8 +101,10 @@ class TestLogHubScorer: def test_exact_match_anomaly(self) -> None: s = LogHubScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="test-1", problem="analyze logs", - reference="anomaly", category="agentic", + record_id="test-1", + problem="analyze logs", + reference="anomaly", + category="agentic", ) is_correct, meta = s.score(record, "ANOMALY\nThe logs show errors.") assert is_correct is True @@ -111,8 +113,10 @@ class TestLogHubScorer: def test_exact_match_normal(self) -> None: s = LogHubScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="test-2", problem="analyze logs", - reference="normal", category="agentic", + record_id="test-2", + problem="analyze logs", + reference="normal", + category="agentic", ) is_correct, meta = s.score(record, "NORMAL - no issues detected") assert is_correct is True @@ -120,8 +124,10 @@ class TestLogHubScorer: def test_empty_response(self) -> None: s = LogHubScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="test-3", problem="analyze logs", - reference="anomaly", category="agentic", + record_id="test-3", + problem="analyze logs", + reference="anomaly", + category="agentic", ) is_correct, meta = s.score(record, "") assert is_correct is False @@ -130,8 +136,10 @@ class TestLogHubScorer: def test_wrong_classification(self) -> None: s = LogHubScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="test-4", problem="analyze logs", - reference="anomaly", category="agentic", + record_id="test-4", + problem="analyze logs", + reference="anomaly", + category="agentic", ) is_correct, meta = s.score(record, "NORMAL - everything looks fine") assert is_correct is False @@ -140,17 +148,20 @@ class TestLogHubScorer: class TestLogHubCLI: def test_in_benchmarks_dict(self) -> None: from openjarvis.evals.cli import BENCHMARKS + assert "loghub" in BENCHMARKS assert BENCHMARKS["loghub"]["category"] == "agentic" def test_build_dataset(self) -> None: from openjarvis.evals.cli import _build_dataset + ds = _build_dataset("loghub") assert ds is not None assert ds.dataset_id == "loghub" def test_build_scorer(self) -> None: from openjarvis.evals.cli import _build_scorer + s = _build_scorer("loghub", _mock_backend(), "test-model") assert s is not None assert s.scorer_id == "loghub" diff --git a/tests/evals/test_paperarena.py b/tests/evals/test_paperarena.py index 022aeb9c..2339c41e 100644 --- a/tests/evals/test_paperarena.py +++ b/tests/evals/test_paperarena.py @@ -91,8 +91,10 @@ class TestPaperArenaScorer: def test_empty_response(self) -> None: s = PaperArenaScorer(_mock_backend(), "test-model") record = EvalRecord( - record_id="pa-5", problem="q", - reference="a", category="agentic", + record_id="pa-5", + problem="q", + reference="a", + category="agentic", ) is_correct, meta = s.score(record, "") assert is_correct is False @@ -102,14 +104,17 @@ class TestPaperArenaScorer: class TestPaperArenaCLI: def test_in_benchmarks(self) -> None: from openjarvis.evals.cli import BENCHMARKS + assert "paperarena" in BENCHMARKS def test_build_dataset(self) -> None: from openjarvis.evals.cli import _build_dataset + ds = _build_dataset("paperarena") assert ds.dataset_id == "paperarena" def test_build_scorer(self) -> None: from openjarvis.evals.cli import _build_scorer + s = _build_scorer("paperarena", _mock_backend(), "test-model") assert s.scorer_id == "paperarena" diff --git a/tests/evals/test_trackers.py b/tests/evals/test_trackers.py index 1654efa2..e8645873 100644 --- a/tests/evals/test_trackers.py +++ b/tests/evals/test_trackers.py @@ -15,6 +15,7 @@ from openjarvis.evals.core.types import EvalResult, RunConfig, RunSummary # Test double # --------------------------------------------------------------------------- + class RecordingTracker(ResultTracker): """Records all lifecycle calls for testing.""" @@ -58,6 +59,7 @@ class CrashingTracker(ResultTracker): # Helpers # --------------------------------------------------------------------------- + def _make_config(**overrides) -> RunConfig: defaults = dict(benchmark="test", backend="jarvis-direct", model="test-model") defaults.update(overrides) @@ -92,6 +94,7 @@ def _make_result(**overrides) -> EvalResult: # RecordingTracker through EvalRunner lifecycle # --------------------------------------------------------------------------- + class TestRecordingTrackerIntegration: """Test that trackers receive all lifecycle calls through EvalRunner.""" @@ -111,11 +114,13 @@ class TestRecordingTrackerIntegration: dataset.iter_records = MagicMock(return_value=[record]) backend = MagicMock() - backend.generate_full = MagicMock(return_value={ - "content": "2", - "usage": {"prompt_tokens": 10, "completion_tokens": 5}, - "latency_seconds": 0.5, - }) + backend.generate_full = MagicMock( + return_value={ + "content": "2", + "usage": {"prompt_tokens": 10, "completion_tokens": 5}, + "latency_seconds": 0.5, + } + ) scorer = MagicMock() scorer.score = MagicMock(return_value=(True, {})) @@ -152,11 +157,13 @@ class TestRecordingTrackerIntegration: dataset.iter_records = MagicMock(return_value=[record]) backend = MagicMock() - backend.generate_full = MagicMock(return_value={ - "content": "yes", - "usage": {}, - "latency_seconds": 0.1, - }) + backend.generate_full = MagicMock( + return_value={ + "content": "yes", + "usage": {}, + "latency_seconds": 0.1, + } + ) scorer = MagicMock() scorer.score = MagicMock(return_value=(True, {})) @@ -178,6 +185,7 @@ class TestRecordingTrackerIntegration: # WandbTracker unit tests # --------------------------------------------------------------------------- + class TestWandbTracker: """Unit tests for WandbTracker (mocked wandb module).""" @@ -185,6 +193,7 @@ class TestWandbTracker: """WandbTracker raises ImportError when wandb is not installed.""" with patch.dict(sys.modules, {"wandb": None}): import openjarvis.evals.trackers.wandb_tracker as wt_mod + original = wt_mod.wandb wt_mod.wandb = None try: @@ -277,12 +286,14 @@ class TestWandbTracker: # SheetsTracker unit tests # --------------------------------------------------------------------------- + class TestSheetsTracker: """Unit tests for SheetsTracker.""" def test_import_error_when_gspread_missing(self): """SheetsTracker raises ImportError when gspread not installed.""" import openjarvis.evals.trackers.sheets_tracker as st_mod + original = st_mod.gspread st_mod.gspread = None try: diff --git a/tests/evals/test_use_case_benchmarks.py b/tests/evals/test_use_case_benchmarks.py index f50ed24f..ec3ebbe3 100644 --- a/tests/evals/test_use_case_benchmarks.py +++ b/tests/evals/test_use_case_benchmarks.py @@ -25,12 +25,14 @@ import pytest class TestEmailTriageDataset: def test_instantiation(self) -> None: from openjarvis.evals.datasets.email_triage import EmailTriageDataset + ds = EmailTriageDataset() assert ds.dataset_id == "email_triage" assert ds.dataset_name == "Email Triage" def test_load(self) -> None: from openjarvis.evals.datasets.email_triage import EmailTriageDataset + ds = EmailTriageDataset() ds.load(max_samples=5, seed=42) assert ds.size() == 5 @@ -44,6 +46,7 @@ class TestEmailTriageDataset: def test_load_all(self) -> None: from openjarvis.evals.datasets.email_triage import EmailTriageDataset + ds = EmailTriageDataset() ds.load() assert ds.size() == 30 @@ -52,12 +55,14 @@ class TestEmailTriageDataset: class TestMorningBriefDataset: def test_instantiation(self) -> None: from openjarvis.evals.datasets.morning_brief import MorningBriefDataset + ds = MorningBriefDataset() assert ds.dataset_id == "morning_brief" assert ds.dataset_name == "Morning Brief" def test_load(self) -> None: from openjarvis.evals.datasets.morning_brief import MorningBriefDataset + ds = MorningBriefDataset() ds.load(max_samples=5, seed=42) assert ds.size() == 5 @@ -68,6 +73,7 @@ class TestMorningBriefDataset: def test_load_all(self) -> None: from openjarvis.evals.datasets.morning_brief import MorningBriefDataset + ds = MorningBriefDataset() ds.load() assert ds.size() == 15 @@ -76,12 +82,14 @@ class TestMorningBriefDataset: class TestResearchMiningDataset: def test_instantiation(self) -> None: from openjarvis.evals.datasets.research_mining import ResearchMiningDataset + ds = ResearchMiningDataset() assert ds.dataset_id == "research_mining" assert ds.dataset_name == "Research Mining" def test_load(self) -> None: from openjarvis.evals.datasets.research_mining import ResearchMiningDataset + ds = ResearchMiningDataset() ds.load(max_samples=5, seed=42) assert ds.size() == 5 @@ -92,6 +100,7 @@ class TestResearchMiningDataset: def test_load_all(self) -> None: from openjarvis.evals.datasets.research_mining import ResearchMiningDataset + ds = ResearchMiningDataset() ds.load() assert ds.size() == 31 @@ -100,12 +109,14 @@ class TestResearchMiningDataset: class TestKnowledgeBaseDataset: def test_instantiation(self) -> None: from openjarvis.evals.datasets.knowledge_base import KnowledgeBaseDataset + ds = KnowledgeBaseDataset() assert ds.dataset_id == "knowledge_base" assert ds.dataset_name == "Knowledge Base" def test_load(self) -> None: from openjarvis.evals.datasets.knowledge_base import KnowledgeBaseDataset + ds = KnowledgeBaseDataset() ds.load(max_samples=5, seed=42) assert ds.size() == 5 @@ -116,6 +127,7 @@ class TestKnowledgeBaseDataset: def test_load_all(self) -> None: from openjarvis.evals.datasets.knowledge_base import KnowledgeBaseDataset + ds = KnowledgeBaseDataset() ds.load() assert ds.size() == 30 @@ -124,12 +136,14 @@ class TestKnowledgeBaseDataset: class TestCodingTaskDataset: def test_instantiation(self) -> None: from openjarvis.evals.datasets.coding_task import CodingTaskDataset + ds = CodingTaskDataset() assert ds.dataset_id == "coding_task" assert ds.dataset_name == "Coding Task" def test_load(self) -> None: from openjarvis.evals.datasets.coding_task import CodingTaskDataset + ds = CodingTaskDataset() ds.load(max_samples=5, seed=42) assert ds.size() == 5 @@ -141,6 +155,7 @@ class TestCodingTaskDataset: def test_load_all(self) -> None: from openjarvis.evals.datasets.coding_task import CodingTaskDataset + ds = CodingTaskDataset() ds.load() assert ds.size() == 29 @@ -157,26 +172,31 @@ class TestScorerInstantiation: def test_email_triage_scorer(self) -> None: from openjarvis.evals.scorers.email_triage import EmailTriageScorer + scorer = EmailTriageScorer(self._mock_backend(), "gpt-5-mini") assert scorer.scorer_id == "email_triage" def test_morning_brief_scorer(self) -> None: from openjarvis.evals.scorers.morning_brief import MorningBriefScorer + scorer = MorningBriefScorer(self._mock_backend(), "gpt-5-mini") assert scorer.scorer_id == "morning_brief" def test_research_mining_scorer(self) -> None: from openjarvis.evals.scorers.research_mining import ResearchMiningScorer + scorer = ResearchMiningScorer(self._mock_backend(), "gpt-5-mini") assert scorer.scorer_id == "research_mining" def test_knowledge_base_scorer(self) -> None: from openjarvis.evals.scorers.knowledge_base import KnowledgeBaseScorer + scorer = KnowledgeBaseScorer(self._mock_backend(), "gpt-5-mini") assert scorer.scorer_id == "knowledge_base" def test_coding_task_scorer(self) -> None: from openjarvis.evals.scorers.coding_task import CodingTaskScorer + scorer = CodingTaskScorer(self._mock_backend(), "gpt-5-mini") assert scorer.scorer_id == "coding_task" @@ -208,10 +228,7 @@ class TestCodingTaskScoring: ), }, ) - answer = ( - "def is_palindrome(s):\n" - " return s == s[::-1]" - ) + answer = "def is_palindrome(s):\n return s == s[::-1]" is_correct, meta = scorer.score(record, answer) assert is_correct is True assert meta["tests_passed"] == 3 @@ -228,10 +245,7 @@ class TestCodingTaskScoring: reference="", category="use-case", metadata={ - "test_cases": ( - "assert add(1, 2) == 3\n" - "assert add(0, 0) == 0" - ), + "test_cases": ("assert add(1, 2) == 3\nassert add(0, 0) == 0"), }, ) # Wrong implementation @@ -285,27 +299,35 @@ class TestEmailTriageScoring: class TestCLIFactories: """Test that _build_dataset and _build_scorer work for new benchmarks.""" - @pytest.mark.parametrize("benchmark", [ - "email_triage", - "morning_brief", - "research_mining", - "knowledge_base", - "coding_task", - ]) + @pytest.mark.parametrize( + "benchmark", + [ + "email_triage", + "morning_brief", + "research_mining", + "knowledge_base", + "coding_task", + ], + ) def test_build_dataset(self, benchmark: str) -> None: from openjarvis.evals.cli import _build_dataset + ds = _build_dataset(benchmark) assert ds.dataset_id == benchmark - @pytest.mark.parametrize("benchmark", [ - "email_triage", - "morning_brief", - "research_mining", - "knowledge_base", - "coding_task", - ]) + @pytest.mark.parametrize( + "benchmark", + [ + "email_triage", + "morning_brief", + "research_mining", + "knowledge_base", + "coding_task", + ], + ) def test_build_scorer(self, benchmark: str) -> None: from openjarvis.evals.cli import _build_scorer + scorer = _build_scorer(benchmark, MagicMock(), "gpt-5-mini") assert scorer.scorer_id == benchmark @@ -320,6 +342,7 @@ class TestCostCalculator: def test_estimate_monthly_cost(self) -> None: from openjarvis.server.cost_calculator import estimate_monthly_cost + est = estimate_monthly_cost( calls_per_month=1000, avg_input_tokens=500, @@ -332,6 +355,7 @@ class TestCostCalculator: def test_estimate_scenario(self) -> None: from openjarvis.server.cost_calculator import estimate_scenario + estimates = estimate_scenario("daily_briefing") assert len(estimates) == 3 # 3 cloud providers for est in estimates: @@ -339,16 +363,19 @@ class TestCostCalculator: def test_estimate_all_scenarios(self) -> None: from openjarvis.server.cost_calculator import estimate_all_scenarios + all_est = estimate_all_scenarios() assert len(all_est) == 5 # 5 scenarios def test_unknown_provider(self) -> None: from openjarvis.server.cost_calculator import estimate_monthly_cost + with pytest.raises(ValueError, match="Unknown provider"): estimate_monthly_cost(100, 100, 100, "nonexistent") def test_unknown_scenario(self) -> None: from openjarvis.server.cost_calculator import estimate_scenario + with pytest.raises(ValueError, match="Unknown scenario"): estimate_scenario("nonexistent") @@ -363,6 +390,7 @@ class TestSavings: def test_compute_savings_basic(self) -> None: from openjarvis.server.savings import compute_savings + summary = compute_savings(1000, 500, total_calls=10) assert summary.total_calls == 10 assert summary.total_tokens == 1500 @@ -375,15 +403,20 @@ class TestSavings: import time from openjarvis.server.savings import compute_savings + start = time.time() - 3600 # 1 hour ago summary = compute_savings( - 100000, 50000, total_calls=100, session_start=start, + 100000, + 50000, + total_calls=100, + session_start=start, ) assert summary.session_duration_hours > 0 assert summary.monthly_projection # not empty def test_savings_to_dict(self) -> None: from openjarvis.server.savings import compute_savings, savings_to_dict + summary = compute_savings(1000, 500, total_calls=5) d = savings_to_dict(summary) assert isinstance(d, dict) diff --git a/tests/hardware/test_amd.py b/tests/hardware/test_amd.py index 87cfad9a..3eecdae8 100644 --- a/tests/hardware/test_amd.py +++ b/tests/hardware/test_amd.py @@ -28,9 +28,9 @@ class TestAMDDetection: @patch( "openjarvis.core.config._run_cmd", side_effect=[ - "AMD Instinct MI300X", # --showproductname + "AMD Instinct MI300X", # --showproductname "GPU[0] : vram Total Memory (B): 206158430208", # --showmeminfo vram - "GPU[0] : Some info", # --showallinfo + "GPU[0] : Some info", # --showallinfo ], ) def test_rocm_smi_parsing(self, mock_run, mock_which): @@ -48,8 +48,8 @@ class TestAMDDetection: "openjarvis.core.config._run_cmd", side_effect=[ "AMD Instinct MI250X\nAMD Instinct MI250X", # --showproductname - "", # --showmeminfo vram (empty) - "", # --showallinfo (empty) + "", # --showmeminfo vram (empty) + "", # --showallinfo (empty) ], ) def test_amd_gpu_model(self, mock_run, mock_which): @@ -140,8 +140,10 @@ class TestAMDEngineRecommendation: cpu_count=96, ram_gb=768.0, gpu=GpuInfo( - vendor="amd", name="AMD Instinct MI300X", - vram_gb=192.0, count=1, + vendor="amd", + name="AMD Instinct MI300X", + vram_gb=192.0, + count=1, ), ) assert recommend_engine(hw) == "vllm" @@ -163,8 +165,10 @@ class TestAMDEngineRecommendation: cpu_count=128, ram_gb=1024.0, gpu=GpuInfo( - vendor="amd", name="AMD Instinct MI300X", - vram_gb=192.0, count=4, + vendor="amd", + name="AMD Instinct MI300X", + vram_gb=192.0, + count=4, ), ) assert recommend_engine(hw) == "vllm" diff --git a/tests/hardware/test_apple.py b/tests/hardware/test_apple.py index f662ea00..e289d338 100644 --- a/tests/hardware/test_apple.py +++ b/tests/hardware/test_apple.py @@ -87,11 +87,7 @@ class TestAppleDetection: @patch("openjarvis.core.config.platform.system", return_value="Darwin") @patch( "openjarvis.core.config._run_cmd", - return_value=( - "Graphics/Displays:\n" - " Apple Silicon\n" - " Type: GPU\n" - ), + return_value=("Graphics/Displays:\n Apple Silicon\n Type: GPU\n"), ) def test_apple_no_chipset_line_falls_back(self, mock_run, mock_system): """When no 'Chipset Model' line exists, falls back to 'Apple Silicon'.""" diff --git a/tests/hardware/test_hardware_profiles.py b/tests/hardware/test_hardware_profiles.py index 8556e98a..d1686931 100644 --- a/tests/hardware/test_hardware_profiles.py +++ b/tests/hardware/test_hardware_profiles.py @@ -38,9 +38,9 @@ class TestDetectHardware: @patch( "openjarvis.core.config._run_cmd", side_effect=[ - "AMD Instinct MI300X", # --showproductname + "AMD Instinct MI300X", # --showproductname "GPU[0] : vram Total Memory (B): 206158430208", # --showmeminfo vram - "GPU[0] : Some info", # --showallinfo + "GPU[0] : Some info", # --showallinfo ], ) def test_detect_amd_gpu(self, mock_run, mock_which): diff --git a/tests/hardware/test_nvidia.py b/tests/hardware/test_nvidia.py index c8c94adb..d0a68e44 100644 --- a/tests/hardware/test_nvidia.py +++ b/tests/hardware/test_nvidia.py @@ -104,7 +104,8 @@ class TestNVIDIAEngineRecommendation: gpu=GpuInfo( vendor="nvidia", name="NVIDIA A100-SXM4-80GB", - vram_gb=80.0, count=1, + vram_gb=80.0, + count=1, ), ) assert recommend_engine(hw) == "vllm" @@ -118,7 +119,8 @@ class TestNVIDIAEngineRecommendation: gpu=GpuInfo( vendor="nvidia", name="NVIDIA H100 80GB HBM3", - vram_gb=80.0, count=1, + vram_gb=80.0, + count=1, ), ) assert recommend_engine(hw) == "vllm" @@ -132,7 +134,8 @@ class TestNVIDIAEngineRecommendation: gpu=GpuInfo( vendor="nvidia", name="NVIDIA Tesla V100-SXM2-32GB", - vram_gb=32.0, count=1, + vram_gb=32.0, + count=1, ), ) assert recommend_engine(hw) == "ollama" @@ -146,7 +149,8 @@ class TestNVIDIAEngineRecommendation: gpu=GpuInfo( vendor="nvidia", name="NVIDIA GeForce RTX 4090", - vram_gb=24.0, count=1, + vram_gb=24.0, + count=1, ), ) assert recommend_engine(hw) == "ollama" diff --git a/tests/integration/test_agent_manager_e2e.py b/tests/integration/test_agent_manager_e2e.py index b098c51f..c9b5d3a4 100644 --- a/tests/integration/test_agent_manager_e2e.py +++ b/tests/integration/test_agent_manager_e2e.py @@ -25,9 +25,7 @@ class TestResearchMonitorE2E: templates = manager.list_templates() assert any(t["id"] == "research_monitor" for t in templates) - agent = manager.create_from_template( - "research_monitor", "My Researcher" - ) + agent = manager.create_from_template("research_monitor", "My Researcher") assert agent["name"] == "My Researcher" assert agent["agent_type"] == "monitor_operative" assert agent["status"] == "idle" @@ -64,7 +62,9 @@ class TestResearchMonitorE2E: # Complete task manager.update_task( - t1["id"], status="completed", findings=["Paper A", "Paper B"], + t1["id"], + status="completed", + findings=["Paper A", "Paper B"], ) task = manager._get_task(t1["id"]) assert task["status"] == "completed" diff --git a/tests/integration/test_integration.py b/tests/integration/test_integration.py index ced76ecc..babbbb11 100644 --- a/tests/integration/test_integration.py +++ b/tests/integration/test_integration.py @@ -138,7 +138,8 @@ class TestOrchestratorWithCalculator: bus = EventBus(record_history=True) agent_cls = AgentRegistry.get("orchestrator") agent = agent_cls( - engine, "test-model", + engine, + "test-model", tools=[CalculatorTool()], bus=bus, ) @@ -250,8 +251,7 @@ class TestTelemetryThroughAgent: agent.run("Hello") telem_events = [ - e for e in bus.history - if e.event_type == EventType.TELEMETRY_RECORD + e for e in bus.history if e.event_type == EventType.TELEMETRY_RECORD ] assert len(telem_events) == 1 rec = telem_events[0].data["record"] @@ -286,14 +286,8 @@ class TestToolExecutorIntegration: assert think_result.content == "Step 1: solve" # Verify events - starts = [ - e for e in bus.history - if e.event_type == EventType.TOOL_CALL_START - ] - ends = [ - e for e in bus.history - if e.event_type == EventType.TOOL_CALL_END - ] + starts = [e for e in bus.history if e.event_type == EventType.TOOL_CALL_START] + ends = [e for e in bus.history if e.event_type == EventType.TOOL_CALL_END] assert len(starts) == 2 assert len(ends) == 2 @@ -321,7 +315,9 @@ class TestHeuristicRewardWithTelemetry: ) rf = HeuristicRewardFunction() score = rf.compute( - RoutingContext(query="test"), rec.model_id, "response", + RoutingContext(query="test"), + rec.model_id, + "response", latency_seconds=rec.latency_seconds, cost_usd=rec.cost_usd, prompt_tokens=rec.prompt_tokens, @@ -352,16 +348,30 @@ class TestTelemetryPipeline: db = tmp_path / "telemetry.db" store = TelemetryStore(db) - store.record(TelemetryRecord( - timestamp=time.time(), model_id="m1", engine="ollama", - prompt_tokens=10, completion_tokens=5, total_tokens=15, - latency_seconds=1.0, cost_usd=0.001, - )) - store.record(TelemetryRecord( - timestamp=time.time(), model_id="m2", engine="vllm", - prompt_tokens=20, completion_tokens=10, total_tokens=30, - latency_seconds=0.5, cost_usd=0.002, - )) + store.record( + TelemetryRecord( + timestamp=time.time(), + model_id="m1", + engine="ollama", + prompt_tokens=10, + completion_tokens=5, + total_tokens=15, + latency_seconds=1.0, + cost_usd=0.001, + ) + ) + store.record( + TelemetryRecord( + timestamp=time.time(), + model_id="m2", + engine="vllm", + prompt_tokens=20, + completion_tokens=10, + total_tokens=30, + latency_seconds=0.5, + cost_usd=0.002, + ) + ) store.close() agg = TelemetryAggregator(db) @@ -387,8 +397,12 @@ class TestEventBusTelemetryAggregator: # Publish a telemetry event rec = TelemetryRecord( - timestamp=1000.0, model_id="event-model", engine="test", - prompt_tokens=5, completion_tokens=3, total_tokens=8, + timestamp=1000.0, + model_id="event-model", + engine="test", + prompt_tokens=5, + completion_tokens=3, + total_tokens=8, latency_seconds=0.1, ) bus.publish(EventType.TELEMETRY_RECORD, {"record": rec}) @@ -435,11 +449,18 @@ class TestRewardTelemetryIntegration: db = tmp_path / "telemetry.db" store = TelemetryStore(db) - store.record(TelemetryRecord( - timestamp=time.time(), model_id="scored-model", engine="test", - prompt_tokens=50, completion_tokens=25, total_tokens=75, - latency_seconds=3.0, cost_usd=0.003, - )) + store.record( + TelemetryRecord( + timestamp=time.time(), + model_id="scored-model", + engine="test", + prompt_tokens=50, + completion_tokens=25, + total_tokens=75, + latency_seconds=3.0, + cost_usd=0.003, + ) + ) store.close() agg = TelemetryAggregator(db) @@ -448,7 +469,9 @@ class TestRewardTelemetryIntegration: rf = HeuristicRewardFunction() score = rf.compute( - RoutingContext(query="test"), ms.model_id, "response", + RoutingContext(query="test"), + ms.model_id, + "response", latency_seconds=ms.avg_latency, cost_usd=ms.total_cost, prompt_tokens=ms.prompt_tokens, @@ -581,7 +604,8 @@ class TestFullPipeline: def run(self, input, context=None, **kwargs): result = self.engine.generate( - [], model=self.model, + [], + model=self.model, ) return AgentResult(content=result["content"], turns=1) diff --git a/tests/integration/test_integration_extended.py b/tests/integration/test_integration_extended.py index 9687a70f..bec9b97d 100644 --- a/tests/integration/test_integration_extended.py +++ b/tests/integration/test_integration_extended.py @@ -87,16 +87,15 @@ class TestReActPipeline: "Action: calculator\n" 'Action Input: {"expression":"2+2"}' ), - _simple_response( - "Thought: The result is 4.\n" - "Final Answer: 2+2 equals 4." - ), + _simple_response("Thought: The result is 4.\nFinal Answer: 2+2 equals 4."), ] engine = _make_engine(responses) bus = EventBus(record_history=True) agent = NativeReActAgent( - engine, "test-model", - tools=[CalculatorTool()], bus=bus, + engine, + "test-model", + tools=[CalculatorTool()], + bus=bus, ) result = agent.run("What is 2+2?") @@ -118,13 +117,14 @@ class TestReActPipeline: 'Action Input: {"thought":"Step 1: analyze"}' ), _simple_response( - "Thought: I have my analysis.\n" - "Final Answer: The answer is clear." + "Thought: I have my analysis.\nFinal Answer: The answer is clear." ), ] engine = _make_engine(responses) agent = NativeReActAgent( - engine, "test-model", tools=[ThinkTool()], + engine, + "test-model", + tools=[ThinkTool()], ) result = agent.run("Analyze this.") assert result.turns == 2 @@ -136,10 +136,7 @@ class TestReActPipeline: from openjarvis.agents.native_react import NativeReActAgent engine = _make_engine( - _simple_response( - "Thought: This is simple.\n" - "Final Answer: Hello!" - ) + _simple_response("Thought: This is simple.\nFinal Answer: Hello!") ) agent = NativeReActAgent(engine, "test-model") result = agent.run("Say hello") @@ -155,14 +152,12 @@ class TestReActPipeline: _register_all() from openjarvis.agents.native_react import NativeReActAgent - engine = _make_engine( - _simple_response( - "Thought: done.\nFinal Answer: ok" - ) - ) + engine = _make_engine(_simple_response("Thought: done.\nFinal Answer: ok")) bus = EventBus(record_history=True) agent = NativeReActAgent( - engine, "test-model", bus=bus, + engine, + "test-model", + bus=bus, ) agent.run("Test") @@ -188,19 +183,18 @@ class TestOpenHandsPipeline: if not ToolRegistry.contains("code_interpreter"): ToolRegistry.register_value( - "code_interpreter", CodeInterpreterTool, + "code_interpreter", + CodeInterpreterTool, ) responses = [ - _simple_response( - "I'll calculate this:\n" - "```python\nprint(2 + 2)\n```" - ), + _simple_response("I'll calculate this:\n```python\nprint(2 + 2)\n```"), _simple_response("The result is 4."), ] engine = _make_engine(responses) agent = NativeOpenHandsAgent( - engine, "test-model", + engine, + "test-model", tools=[CodeInterpreterTool()], ) result = agent.run("What is 2+2?") @@ -216,9 +210,7 @@ class TestOpenHandsPipeline: _register_all() from openjarvis.agents.native_openhands import NativeOpenHandsAgent - engine = _make_engine( - _simple_response("Hello! How can I help?") - ) + engine = _make_engine(_simple_response("Hello! How can I help?")) agent = NativeOpenHandsAgent(engine, "test-model") result = agent.run("Say hello") assert result.content == "Hello! How can I help?" @@ -229,12 +221,12 @@ class TestOpenHandsPipeline: _register_all() from openjarvis.agents.native_openhands import NativeOpenHandsAgent - engine = _make_engine( - _simple_response("Direct answer.") - ) + engine = _make_engine(_simple_response("Direct answer.")) bus = EventBus(record_history=True) agent = NativeOpenHandsAgent( - engine, "test-model", bus=bus, + engine, + "test-model", + bus=bus, ) agent.run("Test") @@ -275,14 +267,16 @@ class TestMCPIntegration: # Call calculator result = client.call_tool( - "calculator", {"expression": "10*5"}, + "calculator", + {"expression": "10*5"}, ) assert result["content"][0]["text"] == "50.0" assert result["isError"] is False # Call think result = client.call_tool( - "think", {"thought": "reasoning step"}, + "think", + {"thought": "reasoning step"}, ) assert result["content"][0]["text"] == "reasoning step" @@ -325,7 +319,8 @@ class TestMCPIntegration: # 3. Call tool result = client.call_tool( - "calculator", {"expression": "7+3"}, + "calculator", + {"expression": "7+3"}, ) assert result["content"][0]["text"] == "10.0" @@ -372,15 +367,13 @@ class TestCrossEngineConsistency: "Action: calculator\n" 'Action Input: {"expression":"3*3"}' ), - _simple_response( - "Thought: got 9.\n" - "Final Answer: 9" - ), + _simple_response("Thought: got 9.\nFinal Answer: 9"), ] engine = _make_engine(responses) engine.engine_id = engine_name agent = NativeReActAgent( - engine, "test-model", + engine, + "test-model", tools=[CalculatorTool()], ) result = agent.run("What is 3*3?") @@ -453,13 +446,9 @@ class TestModelCatalogIntegration: BUILTIN_MODELS, ) - local = [ - s for s in BUILTIN_MODELS if not s.requires_api_key - ] + local = [s for s in BUILTIN_MODELS if not s.requires_api_key] for spec in local: - assert len(spec.supported_engines) >= 1, ( - f"{spec.model_id} has no engines" - ) + assert len(spec.supported_engines) >= 1, f"{spec.model_id} has no engines" def test_cloud_models_require_api_key(self): """All cloud models require an API key.""" @@ -478,9 +467,7 @@ class TestModelCatalogIntegration: "gemini-3-flash", ] for mid in cloud_ids: - matches = [ - s for s in BUILTIN_MODELS if s.model_id == mid - ] + matches = [s for s in BUILTIN_MODELS if s.model_id == mid] assert len(matches) == 1, f"Missing {mid}" assert matches[0].requires_api_key is True @@ -494,7 +481,8 @@ class TestAgentRoutingMatrix: """Agents run consistently across different configurations.""" @pytest.mark.parametrize( - "agent_key", ["native_react", "native_openhands"], + "agent_key", + ["native_react", "native_openhands"], ) def test_agent_returns_valid_result(self, agent_key): _register_all() @@ -515,7 +503,8 @@ class TestAgentRoutingMatrix: assert len(result.content) > 0 @pytest.mark.parametrize( - "agent_key", ["native_react", "native_openhands"], + "agent_key", + ["native_react", "native_openhands"], ) def test_agent_emits_events(self, agent_key): _register_all() @@ -543,15 +532,16 @@ class TestAgentRoutingMatrix: engine = _make_engine( _simple_response( - "Thought: I see the system message.\n" - "Final Answer: Got context." + "Thought: I see the system message.\nFinal Answer: Got context." ) ) conv = Conversation() - conv.add(Message( - role=Role.SYSTEM, - content="You are helpful.", - )) + conv.add( + Message( + role=Role.SYSTEM, + content="You are helpful.", + ) + ) ctx = AgentContext(conversation=conv) agent = NativeReActAgent(engine, "test-model") result = agent.run("Hello", context=ctx) @@ -561,6 +551,4 @@ class TestAgentRoutingMatrix: call_args = engine.generate.call_args msgs = call_args[0][0] # First message is ReAct system prompt, then context - assert any( - m.content == "You are helpful." for m in msgs - ) + assert any(m.content == "You are helpful." for m in msgs) diff --git a/tests/intelligence/test_intelligence_stubs.py b/tests/intelligence/test_intelligence_stubs.py index d483c81a..7f46cd2a 100644 --- a/tests/intelligence/test_intelligence_stubs.py +++ b/tests/intelligence/test_intelligence_stubs.py @@ -39,7 +39,8 @@ class TestBackwardCompatShims: ctx = build_routing_context("hello") assert ctx.query == "hello" router = HeuristicRouter( - available_models=[], default_model="m", + available_models=[], + default_model="m", ) assert router.select_model(ctx) == "m" diff --git a/tests/intelligence/test_model_catalog_extended.py b/tests/intelligence/test_model_catalog_extended.py index 3e6e5582..bb5e8d38 100644 --- a/tests/intelligence/test_model_catalog_extended.py +++ b/tests/intelligence/test_model_catalog_extended.py @@ -16,6 +16,7 @@ from openjarvis.intelligence.model_catalog import ( # Helpers # --------------------------------------------------------------------------- + def _get_spec(model_id: str) -> ModelSpec: """Lookup a spec from the BUILTIN_MODELS list (not registry).""" for spec in BUILTIN_MODELS: @@ -273,10 +274,19 @@ class TestModelDiscovery: def test_cloud_models_require_api_key(self) -> None: """All cloud models have requires_api_key=True.""" cloud_ids = { - "gpt-4o", "gpt-4o-mini", "gpt-5-mini", "gpt-5-mini-2025-08-07", - "claude-sonnet-4-20250514", "claude-opus-4-20250514", - "claude-opus-4-6", "claude-sonnet-4-6", "claude-haiku-4-5", - "gemini-2.5-pro", "gemini-2.5-flash", "gemini-3-pro", "gemini-3-flash", + "gpt-4o", + "gpt-4o-mini", + "gpt-5-mini", + "gpt-5-mini-2025-08-07", + "claude-sonnet-4-20250514", + "claude-opus-4-20250514", + "claude-opus-4-6", + "claude-sonnet-4-6", + "claude-haiku-4-5", + "gemini-2.5-pro", + "gemini-2.5-flash", + "gemini-3-pro", + "gemini-3-flash", } for spec in BUILTIN_MODELS: if spec.model_id in cloud_ids: @@ -287,9 +297,15 @@ class TestModelDiscovery: def test_moe_models_have_active_params(self) -> None: """MoE models have active_parameter_count_b set.""" moe_ids = { - "gpt-oss:120b", "glm-4.7-flash", "trinity-mini", - "qwen3.5:3b", "qwen3.5:8b", "qwen3.5:14b", - "qwen3.5:35b", "qwen3.5:122b", "qwen3.5:397b", + "gpt-oss:120b", + "glm-4.7-flash", + "trinity-mini", + "qwen3.5:3b", + "qwen3.5:8b", + "qwen3.5:14b", + "qwen3.5:35b", + "qwen3.5:122b", + "qwen3.5:397b", "granite4.0-h-small", } for spec in BUILTIN_MODELS: @@ -308,8 +324,12 @@ class TestModelDiscovery: """merge_discovered_models works for all new model IDs.""" register_builtin_models() new_ids = [ - "gpt-oss:120b", "glm-4.7-flash", "trinity-mini", - "gpt-5-mini", "claude-opus-4-6", "gemini-3-pro", + "gpt-oss:120b", + "glm-4.7-flash", + "trinity-mini", + "gpt-5-mini", + "claude-opus-4-6", + "gemini-3-pro", ] # Merging known IDs should not raise merge_discovered_models("vllm", new_ids) diff --git a/tests/intelligence/test_router.py b/tests/intelligence/test_router.py index b6ea9319..83c5f4d2 100644 --- a/tests/intelligence/test_router.py +++ b/tests/intelligence/test_router.py @@ -19,15 +19,19 @@ def _register_models() -> None: ModelRegistry.register_value( "small", ModelSpec( - model_id="small", name="Small", - parameter_count_b=3.0, context_length=4096, + model_id="small", + name="Small", + parameter_count_b=3.0, + context_length=4096, ), ) ModelRegistry.register_value( "large", ModelSpec( - model_id="large", name="Large", - parameter_count_b=70.0, context_length=131072, + model_id="large", + name="Large", + parameter_count_b=70.0, + context_length=131072, ), ) diff --git a/tests/learning/agents/test_dspy_optimizer.py b/tests/learning/agents/test_dspy_optimizer.py index e83d45f8..3904b75d 100644 --- a/tests/learning/agents/test_dspy_optimizer.py +++ b/tests/learning/agents/test_dspy_optimizer.py @@ -48,23 +48,27 @@ class TestDSPyOptimizerTraceConversion: now = time.time() traces = [] for i in range(5): - traces.append(Trace( - query=f"test query {i}", - agent="native_react", - model="qwen3:8b", - result=f"result {i}", - outcome="success", - feedback=0.9, - started_at=now, - ended_at=now + 1, - total_tokens=100, - total_latency_seconds=1.0, - steps=[TraceStep( - step_type=StepType.GENERATE, - timestamp=now, - duration_seconds=0.5, - )], - )) + traces.append( + Trace( + query=f"test query {i}", + agent="native_react", + model="qwen3:8b", + result=f"result {i}", + outcome="success", + feedback=0.9, + started_at=now, + ended_at=now + 1, + total_tokens=100, + total_latency_seconds=1.0, + steps=[ + TraceStep( + step_type=StepType.GENERATE, + timestamp=now, + duration_seconds=0.5, + ) + ], + ) + ) mock_store = MagicMock() mock_store.list_traces.return_value = traces @@ -73,10 +77,14 @@ class TestDSPyOptimizerTraceConversion: import openjarvis.learning.agents.dspy_optimizer as mod with patch.object(mod, "HAS_DSPY", True): - with patch.object(optimizer, "_run_dspy_optimization", return_value={ - "system_prompt": "You are a helpful assistant.", - "few_shot_examples": [{"input": "hi", "output": "hello"}], - }): + with patch.object( + optimizer, + "_run_dspy_optimization", + return_value={ + "system_prompt": "You are a helpful assistant.", + "few_shot_examples": [{"input": "hi", "output": "hello"}], + }, + ): result = optimizer.optimize(mock_store) assert result["status"] == "completed" assert "config_updates" in result diff --git a/tests/learning/agents/test_gepa_optimizer.py b/tests/learning/agents/test_gepa_optimizer.py index acb59198..db738b60 100644 --- a/tests/learning/agents/test_gepa_optimizer.py +++ b/tests/learning/agents/test_gepa_optimizer.py @@ -45,17 +45,27 @@ class TestGEPAOptimizerOptimize: optimizer = GEPAAgentOptimizer(cfg) now = time.time() - traces = [Trace( - query="test", agent="native_react", model="qwen3:8b", - result="result", outcome="success", feedback=0.9, - started_at=now, ended_at=now + 1, - total_tokens=100, total_latency_seconds=1.0, - steps=[TraceStep( - step_type=StepType.GENERATE, - timestamp=now, - duration_seconds=0.5, - )], - )] + traces = [ + Trace( + query="test", + agent="native_react", + model="qwen3:8b", + result="result", + outcome="success", + feedback=0.9, + started_at=now, + ended_at=now + 1, + total_tokens=100, + total_latency_seconds=1.0, + steps=[ + TraceStep( + step_type=StepType.GENERATE, + timestamp=now, + duration_seconds=0.5, + ) + ], + ) + ] mock_store = MagicMock() mock_store.list_traces.return_value = traces @@ -80,6 +90,8 @@ class TestOpenJarvisGEPAAdapter: mock_store = MagicMock() adapter = OpenJarvisGEPAAdapter( - mock_store, "native_react", GEPAOptimizerConfig(), + mock_store, + "native_react", + GEPAOptimizerConfig(), ) assert adapter.agent_name == "native_react" diff --git a/tests/learning/routing/test_learned_router.py b/tests/learning/routing/test_learned_router.py index 20dc4d71..574d2c5c 100644 --- a/tests/learning/routing/test_learned_router.py +++ b/tests/learning/routing/test_learned_router.py @@ -46,6 +46,7 @@ class TestLearnedRouterPolicy: def test_registered_as_learned(self) -> None: from openjarvis.core.registry import RouterPolicyRegistry from openjarvis.learning.routing.learned_router import ensure_registered + ensure_registered() assert RouterPolicyRegistry.contains("learned") @@ -66,19 +67,23 @@ class TestLearnedRouterPolicy: def test_update_from_traces(self, tmp_path: Path) -> None: store = TraceStore(tmp_path / "test.db") for _ in range(6): - store.save(_make_trace( - query="def foo(): pass", - model="codestral", - outcome="success", - feedback=0.9, - )) + store.save( + _make_trace( + query="def foo(): pass", + model="codestral", + outcome="success", + feedback=0.9, + ) + ) for _ in range(6): - store.save(_make_trace( - query="def bar(): return 1", - model="qwen3:8b", - outcome="failure", - feedback=0.3, - )) + store.save( + _make_trace( + query="def bar(): return 1", + model="qwen3:8b", + outcome="failure", + feedback=0.3, + ) + ) analyzer = TraceAnalyzer(store) policy = LearnedRouterPolicy( @@ -96,10 +101,13 @@ class TestLearnedRouterPolicy: def test_policy_map_readable(self, tmp_path: Path) -> None: store = TraceStore(tmp_path / "test.db") for _ in range(5): - store.save(_make_trace( - query="hello", model="small-model", - outcome="success", - )) + store.save( + _make_trace( + query="hello", + model="small-model", + outcome="success", + ) + ) analyzer = TraceAnalyzer(store) policy = LearnedRouterPolicy(analyzer=analyzer, default_model="default") @@ -126,24 +134,34 @@ class TestLearnedRouterPolicy: mock_store = MagicMock() mock_store.list_traces.return_value = [ _make_trace( - query="def foo(): pass", model="code-model", - outcome="success", feedback=0.9, + query="def foo(): pass", + model="code-model", + outcome="success", + feedback=0.9, ), _make_trace( - query="def bar(): pass", model="code-model", - outcome="success", feedback=0.85, + query="def bar(): pass", + model="code-model", + outcome="success", + feedback=0.85, ), _make_trace( - query="def baz(): pass", model="code-model", - outcome="success", feedback=0.88, + query="def baz(): pass", + model="code-model", + outcome="success", + feedback=0.88, ), _make_trace( - query="def qux(): pass", model="code-model", - outcome="success", feedback=0.92, + query="def qux(): pass", + model="code-model", + outcome="success", + feedback=0.92, ), _make_trace( - query="def quux(): pass", model="code-model", - outcome="success", feedback=0.87, + query="def quux(): pass", + model="code-model", + outcome="success", + feedback=0.87, ), ] result = policy.update(mock_store) diff --git a/tests/learning/test_feedback_collector.py b/tests/learning/test_feedback_collector.py index 93b2d50b..976c2fd7 100644 --- a/tests/learning/test_feedback_collector.py +++ b/tests/learning/test_feedback_collector.py @@ -222,9 +222,9 @@ class TestStats: def test_distribution_mixed(self) -> None: fc = FeedbackCollector() - fc.record_explicit("a", 0.1) # low - fc.record_explicit("b", 0.5) # medium - fc.record_explicit("c", 0.9) # high + fc.record_explicit("a", 0.1) # low + fc.record_explicit("b", 0.5) # medium + fc.record_explicit("c", 0.9) # high s = fc.stats() assert s["distribution"]["low"] == 1 assert s["distribution"]["medium"] == 1 diff --git a/tests/learning/test_heuristic_reward.py b/tests/learning/test_heuristic_reward.py index e4534eac..2217267a 100644 --- a/tests/learning/test_heuristic_reward.py +++ b/tests/learning/test_heuristic_reward.py @@ -12,9 +12,13 @@ class TestHeuristicRewardFunction: def test_perfect_score(self) -> None: rf = HeuristicRewardFunction() score = rf.compute( - RoutingContext(), "model-a", "response", - latency_seconds=0.0, cost_usd=0.0, - prompt_tokens=10, completion_tokens=10, + RoutingContext(), + "model-a", + "response", + latency_seconds=0.0, + cost_usd=0.0, + prompt_tokens=10, + completion_tokens=10, ) # latency=1.0, cost=1.0, efficiency=0.5 → 0.4*1 + 0.3*1 + 0.3*0.5 = 0.85 assert score == pytest.approx(0.85) @@ -22,45 +26,67 @@ class TestHeuristicRewardFunction: def test_worst_score(self) -> None: rf = HeuristicRewardFunction() score = rf.compute( - RoutingContext(), "model-a", "response", - latency_seconds=60.0, cost_usd=0.1, - prompt_tokens=100, completion_tokens=0, + RoutingContext(), + "model-a", + "response", + latency_seconds=60.0, + cost_usd=0.1, + prompt_tokens=100, + completion_tokens=0, ) # latency clamped to 0, cost clamped to 0, efficiency=0/100=0 assert score == pytest.approx(0.0) def test_latency_only_weight(self) -> None: rf = HeuristicRewardFunction( - weight_latency=1.0, weight_cost=0.0, weight_efficiency=0.0, + weight_latency=1.0, + weight_cost=0.0, + weight_efficiency=0.0, ) score = rf.compute( - RoutingContext(), "m", "", - latency_seconds=15.0, cost_usd=0.0, - prompt_tokens=10, completion_tokens=10, + RoutingContext(), + "m", + "", + latency_seconds=15.0, + cost_usd=0.0, + prompt_tokens=10, + completion_tokens=10, ) # 1 - 15/30 = 0.5 assert score == pytest.approx(0.5) def test_cost_only_weight(self) -> None: rf = HeuristicRewardFunction( - weight_latency=0.0, weight_cost=1.0, weight_efficiency=0.0, + weight_latency=0.0, + weight_cost=1.0, + weight_efficiency=0.0, ) score = rf.compute( - RoutingContext(), "m", "", - latency_seconds=0.0, cost_usd=0.005, - prompt_tokens=10, completion_tokens=10, + RoutingContext(), + "m", + "", + latency_seconds=0.0, + cost_usd=0.005, + prompt_tokens=10, + completion_tokens=10, ) # 1 - 0.005/0.01 = 0.5 assert score == pytest.approx(0.5) def test_efficiency_only_weight(self) -> None: rf = HeuristicRewardFunction( - weight_latency=0.0, weight_cost=0.0, weight_efficiency=1.0, + weight_latency=0.0, + weight_cost=0.0, + weight_efficiency=1.0, ) score = rf.compute( - RoutingContext(), "m", "", - latency_seconds=0.0, cost_usd=0.0, - prompt_tokens=25, completion_tokens=75, + RoutingContext(), + "m", + "", + latency_seconds=0.0, + cost_usd=0.0, + prompt_tokens=25, + completion_tokens=75, ) # 75/100 = 0.75 assert score == pytest.approx(0.75) @@ -69,18 +95,26 @@ class TestHeuristicRewardFunction: rf = HeuristicRewardFunction() # Even with extreme values, score should be in [0, 1] score = rf.compute( - RoutingContext(), "m", "", - latency_seconds=-10.0, cost_usd=-0.1, - prompt_tokens=0, completion_tokens=100, + RoutingContext(), + "m", + "", + latency_seconds=-10.0, + cost_usd=-0.1, + prompt_tokens=0, + completion_tokens=100, ) assert 0.0 <= score <= 1.0 def test_custom_max_values(self) -> None: rf = HeuristicRewardFunction(max_latency=10.0, max_cost=1.0) score = rf.compute( - RoutingContext(), "m", "", - latency_seconds=5.0, cost_usd=0.5, - prompt_tokens=50, completion_tokens=50, + RoutingContext(), + "m", + "", + latency_seconds=5.0, + cost_usd=0.5, + prompt_tokens=50, + completion_tokens=50, ) # latency: 1-5/10=0.5, cost: 1-0.5/1.0=0.5, efficiency: 50/100=0.5 # 0.4*0.5 + 0.3*0.5 + 0.3*0.5 = 0.5 @@ -89,9 +123,13 @@ class TestHeuristicRewardFunction: def test_zero_total_tokens_default(self) -> None: rf = HeuristicRewardFunction() score = rf.compute( - RoutingContext(), "m", "", - latency_seconds=0.0, cost_usd=0.0, - prompt_tokens=0, completion_tokens=0, + RoutingContext(), + "m", + "", + latency_seconds=0.0, + cost_usd=0.0, + prompt_tokens=0, + completion_tokens=0, ) # latency=1, cost=1, efficiency default=0.5 # 0.4*1 + 0.3*1 + 0.3*0.5 = 0.85 @@ -101,9 +139,13 @@ class TestHeuristicRewardFunction: rf = HeuristicRewardFunction() # Should not raise even with extra kwargs score = rf.compute( - RoutingContext(), "m", "", - latency_seconds=1.0, cost_usd=0.001, - prompt_tokens=10, completion_tokens=10, + RoutingContext(), + "m", + "", + latency_seconds=1.0, + cost_usd=0.001, + prompt_tokens=10, + completion_tokens=10, random_extra="ignored", ) assert 0.0 <= score <= 1.0 diff --git a/tests/learning/test_llm_optimizer.py b/tests/learning/test_llm_optimizer.py index 8e114c13..33375c3f 100644 --- a/tests/learning/test_llm_optimizer.py +++ b/tests/learning/test_llm_optimizer.py @@ -183,9 +183,7 @@ class TestInit: def test_stores_optimizer_backend(self) -> None: space = _make_search_space() backend = _make_mock_backend("") - opt = LLMOptimizer( - search_space=space, optimizer_backend=backend - ) + opt = LLMOptimizer(search_space=space, optimizer_backend=backend) assert opt.optimizer_backend is backend def test_default_backend_is_none(self) -> None: @@ -203,13 +201,15 @@ class TestProposeInitial: """Tests for LLMOptimizer.propose_initial.""" def test_returns_trial_config(self) -> None: - response = json.dumps({ - "params": { - "agent.type": "native_react", - "intelligence.temperature": 0.3, - }, - "reasoning": "Balanced starting point", - }) + response = json.dumps( + { + "params": { + "agent.type": "native_react", + "intelligence.temperature": 0.3, + }, + "reasoning": "Balanced starting point", + } + ) response = f"```json\n{response}\n```" backend = _make_mock_backend(response) opt = LLMOptimizer( @@ -265,14 +265,16 @@ class TestProposeNext: """Tests for LLMOptimizer.propose_next.""" def test_returns_trial_config_with_history(self) -> None: - response = json.dumps({ - "params": { - "agent.type": "native_react", - "intelligence.temperature": 0.2, - "agent.max_turns": 15, - }, - "reasoning": "Lower temp for better accuracy", - }) + response = json.dumps( + { + "params": { + "agent.type": "native_react", + "intelligence.temperature": 0.2, + "agent.max_turns": 15, + }, + "reasoning": "Lower temp for better accuracy", + } + ) response = f"```json\n{response}\n```" backend = _make_mock_backend(response) opt = LLMOptimizer( @@ -507,8 +509,7 @@ class TestParseConfigResponse: def test_raw_json(self) -> None: opt = self._make_optimizer() response = ( - 'I suggest: {"params": {"agent.max_turns": 10}, ' - '"reasoning": "More turns"}' + 'I suggest: {"params": {"agent.max_turns": 10}, "reasoning": "More turns"}' ) config = opt._parse_config_response(response) assert config.params["agent.max_turns"] == 10 @@ -666,10 +667,7 @@ class TestFormatTraces: def test_limits_to_last_10(self) -> None: opt = LLMOptimizer(search_space=_make_search_space()) - traces = [ - _make_trace(trace_id=f"trace-{i:03d}") - for i in range(20) - ] + traces = [_make_trace(trace_id=f"trace-{i:03d}") for i in range(20)] result = opt._format_traces(traces) # Should only include the last 10 (indices 10-19) assert "trace-010" in result @@ -740,26 +738,24 @@ class TestIntegration: """Simulate a two-step optimization loop.""" space = _make_search_space() initial_response = ( - '```json\n' + "```json\n" '{"params": {"agent.type": "orchestrator", ' '"intelligence.temperature": 0.5, "agent.max_turns": 10}, ' '"reasoning": "Balanced start"}\n' - '```' + "```" ) next_response = ( - '```json\n' + "```json\n" '{"params": {"agent.type": "native_react", ' '"intelligence.temperature": 0.2, "agent.max_turns": 15}, ' '"reasoning": "Switch to ReAct for better tool use"}\n' - '```' + "```" ) backend = MagicMock(spec=InferenceBackend) backend.backend_id = "mock" backend.generate.side_effect = [initial_response, next_response] - opt = LLMOptimizer( - search_space=space, optimizer_backend=backend - ) + opt = LLMOptimizer(search_space=space, optimizer_backend=backend) # Step 1: initial proposal config1 = opt.propose_initial() @@ -781,10 +777,10 @@ class TestIntegration: """Simulate propose -> evaluate -> analyze.""" space = _make_search_space() propose_response = ( - '```json\n' + "```json\n" '{"params": {"agent.type": "orchestrator"}, ' '"reasoning": "Start with orchestrator"}\n' - '```' + "```" ) analysis_response = ( "The orchestrator agent achieved moderate accuracy. " @@ -798,9 +794,7 @@ class TestIntegration: analysis_response, ] - opt = LLMOptimizer( - search_space=space, optimizer_backend=backend - ) + opt = LLMOptimizer(search_space=space, optimizer_backend=backend) config = opt.propose_initial() summary = _make_run_summary() @@ -833,13 +827,15 @@ class TestAnalyzeTrialStructured: def test_analyze_trial_returns_trial_feedback(self) -> None: """Mock backend returns structured JSON -> TrialFeedback.""" - feedback_json = json.dumps({ - "summary_text": "Good accuracy but high latency", - "failure_patterns": ["timeout on complex queries"], - "primitive_ratings": {"agent": "high", "intelligence": "medium"}, - "suggested_changes": ["reduce max_turns"], - "target_primitive": "agent", - }) + feedback_json = json.dumps( + { + "summary_text": "Good accuracy but high latency", + "failure_patterns": ["timeout on complex queries"], + "primitive_ratings": {"agent": "high", "intelligence": "medium"}, + "suggested_changes": ["reduce max_turns"], + "target_primitive": "agent", + } + ) response = f"```json\n{feedback_json}\n```" backend = _make_mock_backend(response) opt = LLMOptimizer( @@ -875,13 +871,15 @@ class TestAnalyzeTrialStructured: def test_analyze_trial_with_sample_scores(self) -> None: """Verify sample_scores are included in the prompt.""" - feedback_json = json.dumps({ - "summary_text": "Analysis with scores", - "failure_patterns": [], - "primitive_ratings": {}, - "suggested_changes": [], - "target_primitive": "", - }) + feedback_json = json.dumps( + { + "summary_text": "Analysis with scores", + "failure_patterns": [], + "primitive_ratings": {}, + "suggested_changes": [], + "target_primitive": "", + } + ) response = f"```json\n{feedback_json}\n```" backend = _make_mock_backend(response) opt = LLMOptimizer( @@ -909,14 +907,16 @@ class TestProposeTargeted: """Tests for LLMOptimizer.propose_targeted.""" def test_preserves_non_target_params(self) -> None: - response = json.dumps({ - "params": { - "agent.type": "native_react", - "agent.max_turns": 20, - "intelligence.temperature": 0.9, - }, - "reasoning": "Change agent only", - }) + response = json.dumps( + { + "params": { + "agent.type": "native_react", + "agent.max_turns": 20, + "intelligence.temperature": 0.9, + }, + "reasoning": "Change agent only", + } + ) response = f"```json\n{response}\n```" backend = _make_mock_backend(response) opt = LLMOptimizer( diff --git a/tests/learning/test_multi_bench_runner.py b/tests/learning/test_multi_bench_runner.py index aceef32a..8e6b21bc 100644 --- a/tests/learning/test_multi_bench_runner.py +++ b/tests/learning/test_multi_bench_runner.py @@ -74,7 +74,9 @@ class TestBenchmarkSpec: class TestBenchmarkScore: def test_creation(self): score = BenchmarkScore( - benchmark="hle", accuracy=0.75, weight=0.2, + benchmark="hle", + accuracy=0.75, + weight=0.2, ) assert score.benchmark == "hle" assert score.accuracy == 0.75 diff --git a/tests/learning/test_optimizer_engine.py b/tests/learning/test_optimizer_engine.py index 55694044..1634dee6 100644 --- a/tests/learning/test_optimizer_engine.py +++ b/tests/learning/test_optimizer_engine.py @@ -152,7 +152,8 @@ class TestOptimizationEngineRun: optimizer.optimizer_model = "test-model" runner.run_trial.return_value = _sample_trial_result( - "init", accuracy=0.8, + "init", + accuracy=0.8, ) runner.benchmark = "supergpqa" @@ -212,7 +213,8 @@ class TestOptimizationEngineRun: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t1", params={}, + trial_id="t1", + params={}, ) optimizer.analyze_trial.return_value = TrialFeedback( summary_text="detailed analysis", @@ -237,7 +239,8 @@ class TestOptimizationEngineRun: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t1", params={}, + trial_id="t1", + params={}, ) optimizer.optimizer_model = "m" @@ -276,11 +279,11 @@ class TestEarlyStopping: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t0", params={}, + trial_id="t0", + params={}, ) optimizer.propose_next.side_effect = [ - TrialConfig(trial_id=f"t{i}", params={}) - for i in range(1, 20) + TrialConfig(trial_id=f"t{i}", params={}) for i in range(1, 20) ] optimizer.analyze_trial.return_value = TrialFeedback(summary_text="ok") optimizer.optimizer_model = "m" @@ -311,11 +314,11 @@ class TestEarlyStopping: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t0", params={}, + trial_id="t0", + params={}, ) optimizer.propose_next.side_effect = [ - TrialConfig(trial_id=f"t{i}", params={}) - for i in range(1, 5) + TrialConfig(trial_id=f"t{i}", params={}) for i in range(1, 5) ] optimizer.analyze_trial.return_value = TrialFeedback(summary_text="ok") optimizer.optimizer_model = "m" @@ -357,19 +360,18 @@ class TestProgressCallback: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t0", params={}, + trial_id="t0", + params={}, ) optimizer.propose_next.side_effect = [ - TrialConfig(trial_id=f"t{i}", params={}) - for i in range(1, 3) + TrialConfig(trial_id=f"t{i}", params={}) for i in range(1, 3) ] optimizer.analyze_trial.return_value = TrialFeedback(summary_text="ok") optimizer.optimizer_model = "m" runner.benchmark = "b" results = [ - _sample_trial_result(f"t{i}", accuracy=0.5 + i * 0.1) - for i in range(3) + _sample_trial_result(f"t{i}", accuracy=0.5 + i * 0.1) for i in range(3) ] runner.run_trial.side_effect = results @@ -394,7 +396,8 @@ class TestProgressCallback: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t0", params={}, + trial_id="t0", + params={}, ) optimizer.analyze_trial.return_value = TrialFeedback(summary_text="ok") optimizer.optimizer_model = "m" @@ -513,7 +516,9 @@ class TestExportBestRecipe: config = TrialConfig(trial_id="min", params={}) best = TrialResult( - trial_id="min", config=config, accuracy=0.5, + trial_id="min", + config=config, + accuracy=0.5, ) run = OptimizationRun( run_id="run-min", @@ -544,10 +549,12 @@ class TestRunWithStore: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t0", params={}, + trial_id="t0", + params={}, ) optimizer.propose_next.return_value = TrialConfig( - trial_id="t1", params={}, + trial_id="t1", + params={}, ) optimizer.analyze_trial.return_value = TrialFeedback(summary_text="analysis") optimizer.optimizer_model = "m" @@ -584,7 +591,8 @@ class TestRunWithStore: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t0", params={}, + trial_id="t0", + params={}, ) optimizer.analyze_trial.return_value = TrialFeedback(summary_text="ok") optimizer.optimizer_model = "m" @@ -648,7 +656,7 @@ engine = "ollama" from openjarvis.optimize.config import load_optimize_config path = tmp_path / "test.toml" - path.write_bytes(b'[optimize]\nmax_trials = 5\n') + path.write_bytes(b"[optimize]\nmax_trials = 5\n") config = load_optimize_config(str(path)) assert config["optimize"]["max_trials"] == 5 @@ -662,10 +670,12 @@ class TestParetoFrontier: runner = MagicMock() optimizer.propose_initial.return_value = TrialConfig( - trial_id="t0", params={}, + trial_id="t0", + params={}, ) optimizer.propose_next.return_value = TrialConfig( - trial_id="t1", params={}, + trial_id="t1", + params={}, ) # analyze_trial returns TrialFeedback optimizer.analyze_trial.return_value = TrialFeedback( @@ -700,10 +710,7 @@ class TestTargetedAndMerge: optimizer = MagicMock() runner = MagicMock() - configs = [ - TrialConfig(trial_id=f"t{i}", params={}) - for i in range(5) - ] + configs = [TrialConfig(trial_id=f"t{i}", params={}) for i in range(5)] optimizer.propose_initial.return_value = configs[0] optimizer.propose_next.side_effect = configs[1:] optimizer.propose_targeted.return_value = configs[3] @@ -718,15 +725,14 @@ class TestTargetedAndMerge: ) optimizer.analyze_trial.side_effect = [ - fb_normal, # trial 0 - fb_normal, # trial 1 - fb_targeted, # trial 2 -> will trigger targeted on next - fb_normal, # trial 3 + fb_normal, # trial 0 + fb_normal, # trial 1 + fb_targeted, # trial 2 -> will trigger targeted on next + fb_normal, # trial 3 ] results = [ - _sample_trial_result(f"t{i}", accuracy=0.5 + i * 0.05) - for i in range(4) + _sample_trial_result(f"t{i}", accuracy=0.5 + i * 0.05) for i in range(4) ] runner.run_trial.side_effect = results @@ -745,10 +751,7 @@ class TestTargetedAndMerge: optimizer = MagicMock() runner = MagicMock() - configs = [ - TrialConfig(trial_id=f"t{i}", params={}) - for i in range(7) - ] + configs = [TrialConfig(trial_id=f"t{i}", params={}) for i in range(7)] optimizer.propose_initial.return_value = configs[0] optimizer.propose_next.side_effect = configs[1:] optimizer.propose_merge.return_value = configs[5] @@ -761,9 +764,9 @@ class TestTargetedAndMerge: # Create diverse results with genuine tradeoffs so # frontier has >= 2 members (high acc/high lat vs low acc/low lat) tradeoffs = [ - (0.9, 3.0, 0.05), # t0: high accuracy, high latency, high cost - (0.5, 0.5, 0.01), # t1: low accuracy, low latency, low cost - (0.7, 1.5, 0.03), # t2: dominated by t0+t1 combo + (0.9, 3.0, 0.05), # t0: high accuracy, high latency, high cost + (0.5, 0.5, 0.01), # t1: low accuracy, low latency, low cost + (0.7, 1.5, 0.03), # t2: dominated by t0+t1 combo (0.85, 2.5, 0.04), # t3: close to t0 tradeoff (0.6, 0.8, 0.015), # t4: close to t1 tradeoff (0.75, 1.0, 0.02), # t5: merged result diff --git a/tests/learning/test_pareto.py b/tests/learning/test_pareto.py index c0542daf..47fa9979 100644 --- a/tests/learning/test_pareto.py +++ b/tests/learning/test_pareto.py @@ -16,6 +16,7 @@ from openjarvis.optimize.types import ( # Helpers # --------------------------------------------------------------------------- + def _make_trial( trial_id: str, accuracy: float = 0.0, @@ -92,6 +93,7 @@ DEFAULT_OBJECTIVES = [ # Tests # --------------------------------------------------------------------------- + class TestComputeParetoFrontier: """Tests for :func:`compute_pareto_frontier`.""" @@ -185,7 +187,8 @@ class TestComputeParetoFrontier: ) frontier = compute_pareto_frontier( - [trial_a, trial_b, trial_c], objectives, + [trial_a, trial_b, trial_c], + objectives, ) ids = {t.trial_id for t in frontier} @@ -210,7 +213,8 @@ class TestComputeParetoFrontier: trial_c = _make_trial("C", accuracy=0.55, energy=120.0) frontier = compute_pareto_frontier( - [trial_a, trial_b, trial_c], objectives, + [trial_a, trial_b, trial_c], + objectives, ) ids = {t.trial_id for t in frontier} diff --git a/tests/learning/test_personal_synthesizer.py b/tests/learning/test_personal_synthesizer.py index 85dd3406..02bdc7ea 100644 --- a/tests/learning/test_personal_synthesizer.py +++ b/tests/learning/test_personal_synthesizer.py @@ -60,25 +60,33 @@ def trace_store(tmp_path: Path) -> TraceStore: class TestPersonalBenchmarkSampleDefaults: def test_default_category(self) -> None: sample = PersonalBenchmarkSample( - trace_id="t1", query="hello", reference_answer="world", + trace_id="t1", + query="hello", + reference_answer="world", ) assert sample.category == "chat" def test_default_agent_empty(self) -> None: sample = PersonalBenchmarkSample( - trace_id="t1", query="q", reference_answer="a", + trace_id="t1", + query="q", + reference_answer="a", ) assert sample.agent == "" def test_default_feedback_score_zero(self) -> None: sample = PersonalBenchmarkSample( - trace_id="t1", query="q", reference_answer="a", + trace_id="t1", + query="q", + reference_answer="a", ) assert sample.feedback_score == 0.0 def test_default_metadata_empty(self) -> None: sample = PersonalBenchmarkSample( - trace_id="t1", query="q", reference_answer="a", + trace_id="t1", + query="q", + reference_answer="a", ) assert sample.metadata == {} @@ -90,7 +98,8 @@ class TestPersonalBenchmarkSampleDefaults: class TestPersonalBenchmarkSynthesizer: def test_synthesize_creates_benchmark_from_traces( - self, trace_store: TraceStore, + self, + trace_store: TraceStore, ) -> None: trace_store.save(_make_trace(trace_id="t1", feedback=0.9)) trace_store.save(_make_trace(trace_id="t2", query="What is 3+3?", feedback=0.8)) @@ -119,38 +128,46 @@ class TestPersonalBenchmarkSynthesizer: def test_grouping_by_query_class(self, trace_store: TraceStore) -> None: """Same agent + same query prefix -> same group, so only one sample.""" - trace_store.save(_make_trace( - trace_id="t1", - query="What is 2+2?", - agent="simple", - feedback=0.8, - )) - trace_store.save(_make_trace( - trace_id="t2", - query="What is 2+2?", - agent="simple", - feedback=0.95, - )) + trace_store.save( + _make_trace( + trace_id="t1", + query="What is 2+2?", + agent="simple", + feedback=0.8, + ) + ) + trace_store.save( + _make_trace( + trace_id="t2", + query="What is 2+2?", + agent="simple", + feedback=0.95, + ) + ) synth = PersonalBenchmarkSynthesizer(trace_store) bm = synth.synthesize() # Should collapse into one sample (same group) assert len(bm.samples) == 1 def test_picks_highest_feedback_per_group(self, trace_store: TraceStore) -> None: - trace_store.save(_make_trace( - trace_id="t1", - query="Tell me a joke", - agent="simple", - feedback=0.7, - result="bad joke", - )) - trace_store.save(_make_trace( - trace_id="t2", - query="Tell me a joke", - agent="simple", - feedback=0.99, - result="great joke", - )) + trace_store.save( + _make_trace( + trace_id="t1", + query="Tell me a joke", + agent="simple", + feedback=0.7, + result="bad joke", + ) + ) + trace_store.save( + _make_trace( + trace_id="t2", + query="Tell me a joke", + agent="simple", + feedback=0.99, + result="great joke", + ) + ) synth = PersonalBenchmarkSynthesizer(trace_store) bm = synth.synthesize() assert len(bm.samples) == 1 @@ -162,12 +179,14 @@ class TestPersonalBenchmarkSynthesizer: trace_store.save( _make_trace(trace_id="t1", query="Hello", agent="simple", feedback=0.9), ) - trace_store.save(_make_trace( - trace_id="t2", - query="Hello", - agent="orchestrator", - feedback=0.8, - )) + trace_store.save( + _make_trace( + trace_id="t2", + query="Hello", + agent="orchestrator", + feedback=0.8, + ) + ) synth = PersonalBenchmarkSynthesizer(trace_store) bm = synth.synthesize() assert len(bm.samples) == 2 @@ -186,7 +205,8 @@ class TestPersonalBenchmarkSynthesizer: assert len(bm.samples) == 3 def test_empty_traces_returns_empty_benchmark( - self, trace_store: TraceStore, + self, + trace_store: TraceStore, ) -> None: synth = PersonalBenchmarkSynthesizer(trace_store) bm = synth.synthesize() diff --git a/tests/learning/test_router.py b/tests/learning/test_router.py index 5e242153..bf80b4ea 100644 --- a/tests/learning/test_router.py +++ b/tests/learning/test_router.py @@ -16,22 +16,28 @@ def _register_models() -> None: ModelRegistry.register_value( "small", ModelSpec( - model_id="small", name="Small", - parameter_count_b=3.0, context_length=4096, + model_id="small", + name="Small", + parameter_count_b=3.0, + context_length=4096, ), ) ModelRegistry.register_value( "large", ModelSpec( - model_id="large", name="Large", - parameter_count_b=70.0, context_length=131072, + model_id="large", + name="Large", + parameter_count_b=70.0, + context_length=131072, ), ) ModelRegistry.register_value( "coder", ModelSpec( - model_id="coder", name="DeepSeek Coder", - parameter_count_b=16.0, context_length=131072, + model_id="coder", + name="DeepSeek Coder", + parameter_count_b=16.0, + context_length=131072, ), ) @@ -71,7 +77,9 @@ class TestHeuristicRouter: available_models=["small", "large", "coder"], ) ctx = RoutingContext( - query="def foo():", query_length=10, has_code=True, + query="def foo():", + query_length=10, + has_code=True, ) assert router.select_model(ctx) == "coder" @@ -81,7 +89,9 @@ class TestHeuristicRouter: available_models=["small", "large", "coder"], ) ctx = RoutingContext( - query="solve x", query_length=7, has_math=True, + query="solve x", + query_length=7, + has_math=True, ) assert router.select_model(ctx) == "large" @@ -99,7 +109,9 @@ class TestHeuristicRouter: available_models=["small", "large", "coder"], ) ctx = RoutingContext( - query="x" * 501, query_length=501, urgency=0.9, + query="x" * 501, + query_length=501, + urgency=0.9, ) assert router.select_model(ctx) == "small" @@ -112,13 +124,15 @@ class TestHeuristicRouter: ) # Medium-length, no code/math, no reasoning → falls to default ctx = RoutingContext( - query="Tell me about cats", query_length=60, + query="Tell me about cats", + query_length=60, ) assert router.select_model(ctx) == "large" def test_no_available_models(self) -> None: router = HeuristicRouter( - available_models=[], default_model="fallback-model", + available_models=[], + default_model="fallback-model", ) ctx = RoutingContext(query="test", query_length=4) assert router.select_model(ctx) == "fallback-model" @@ -139,6 +153,8 @@ class TestHeuristicRouter: available_models=["small", "large"], ) ctx = RoutingContext( - query="def foo():", query_length=10, has_code=True, + query="def foo():", + query_length=10, + has_code=True, ) assert router.select_model(ctx) == "large" diff --git a/tests/learning/test_router_stubs.py b/tests/learning/test_router_stubs.py index 91ef44d6..41fd6c49 100644 --- a/tests/learning/test_router_stubs.py +++ b/tests/learning/test_router_stubs.py @@ -16,10 +16,13 @@ class _DummyRouter(RouterPolicy): class _DummyAnalyzer(QueryAnalyzer): def analyze( - self, query: str, **kwargs: object, + self, + query: str, + **kwargs: object, ) -> RoutingContext: return RoutingContext( - query=query, query_length=len(query), + query=query, + query_length=len(query), ) diff --git a/tests/learning/test_routing_models.py b/tests/learning/test_routing_models.py index 309004d0..febbd22c 100644 --- a/tests/learning/test_routing_models.py +++ b/tests/learning/test_routing_models.py @@ -13,10 +13,10 @@ from openjarvis.learning.routing.router import ( # New local model keys for testing NEW_LOCAL_MODELS = [ - "gpt-oss:120b", # 117B total, 5.1B active, MoE - "qwen3:8b", # 8.2B, dense + "gpt-oss:120b", # 117B total, 5.1B active, MoE + "qwen3:8b", # 8.2B, dense "glm-4.7-flash", # 30B total, 3.0B active, MoE - "trinity-mini", # 26B total, 3.0B active, MoE + "trinity-mini", # 26B total, 3.0B active, MoE ] # Cloud model keys @@ -114,8 +114,7 @@ class TestRouterWithNewModels: available_models=NEW_LOCAL_MODELS, ) ctx = build_routing_context( - "Please explain step by step how neural networks" - " learn" + "Please explain step by step how neural networks learn" ) selected = router.select_model(ctx) assert selected == "gpt-oss:120b" @@ -148,7 +147,9 @@ class TestRouterCloudFallback: _setup_models() router = HeuristicRouter(available_models=CLOUD_MODELS) ctx = RoutingContext( - query="solve x", query_length=7, has_math=True, + query="solve x", + query_length=7, + has_math=True, ) selected = router.select_model(ctx) assert selected in CLOUD_MODELS @@ -192,18 +193,21 @@ class TestRouterParameterized: @pytest.mark.parametrize("model_id", NEW_LOCAL_MODELS) def test_single_model_always_returns_it( - self, model_id: str, + self, + model_id: str, ) -> None: _setup_models() router = HeuristicRouter(available_models=[model_id]) ctx = RoutingContext( - query="hello world", query_length=11, + query="hello world", + query_length=11, ) assert router.select_model(ctx) == model_id @pytest.mark.parametrize("urgency", [0.85, 0.9, 1.0]) def test_high_urgency_always_smallest( - self, urgency: float, + self, + urgency: float, ) -> None: _setup_models() router = HeuristicRouter( diff --git a/tests/learning/test_search_space.py b/tests/learning/test_search_space.py index e220ff2c..4d28b4ba 100644 --- a/tests/learning/test_search_space.py +++ b/tests/learning/test_search_space.py @@ -36,8 +36,7 @@ class TestBuildSearchSpace: }, "constraints": { "rules": [ - "SimpleAgent should only have " - "max_turns = 1", + "SimpleAgent should only have max_turns = 1", ], }, }, @@ -350,9 +349,16 @@ class TestDefaultSearchSpace: dim = _find_dim("engine.backend") assert dim.dim_type == "categorical" expected = { - "ollama", "vllm", "sglang", - "llamacpp", "mlx", "lmstudio", - "exo", "nexa", "uzu", "apple_fm", + "ollama", + "vllm", + "sglang", + "llamacpp", + "mlx", + "lmstudio", + "exo", + "nexa", + "uzu", + "apple_fm", } assert set(dim.values) == expected @@ -360,8 +366,10 @@ class TestDefaultSearchSpace: dim = _find_dim("agent.type") assert dim.dim_type == "categorical" expected = { - "simple", "orchestrator", - "native_react", "native_openhands", + "simple", + "orchestrator", + "native_react", + "native_openhands", } assert set(dim.values) == expected @@ -388,15 +396,11 @@ class TestDefaultSearchSpace: def test_all_dimensions_have_descriptions(self) -> None: for dim in _DIMS: - assert dim.description != "", ( - f"Dimension {dim.name} has no description" - ) + assert dim.description != "", f"Dimension {dim.name} has no description" def test_all_dimensions_have_primitives(self) -> None: for dim in _DIMS: - assert dim.primitive != "", ( - f"Dimension {dim.name} has no primitive" - ) + assert dim.primitive != "", f"Dimension {dim.name} has no primitive" # --------------------------------------------------------------------------- @@ -415,15 +419,16 @@ class TestToPromptDescription: def test_all_dimensions_appear_in_description(self) -> None: desc = DEFAULT_SEARCH_SPACE.to_prompt_description() for dim in _DIMS: - assert dim.name in desc, ( - f"Dimension {dim.name} not in description" - ) + assert dim.name in desc, f"Dimension {dim.name} not in description" def test_all_primitive_headers_in_description(self) -> None: desc = DEFAULT_SEARCH_SPACE.to_prompt_description() for primitive in ( - "Intelligence", "Engine", "Agent", - "Tools", "Learning", + "Intelligence", + "Engine", + "Agent", + "Tools", + "Learning", ): assert f"## {primitive}" in desc, ( f"Primitive header {primitive} not in description" diff --git a/tests/learning/test_skill_discovery.py b/tests/learning/test_skill_discovery.py index 7873fc4c..0b9d61e4 100644 --- a/tests/learning/test_skill_discovery.py +++ b/tests/learning/test_skill_discovery.py @@ -11,6 +11,7 @@ from openjarvis.learning.agents.skill_discovery import SkillDiscovery # Helpers # --------------------------------------------------------------------------- + @dataclass class _Step: step_type: str = "tool_call" @@ -93,10 +94,7 @@ class TestSkillDiscovery: def test_outcome_threshold(self): """Low-outcome sequences should be filtered out.""" sd = SkillDiscovery(min_frequency=3, min_outcome=0.8) - traces = [ - _make_trace(["a", "b"], outcome=0.3) - for _ in range(5) - ] + traces = [_make_trace(["a", "b"], outcome=0.3) for _ in range(5)] result = sd.analyze_traces(traces) assert result == [] @@ -104,7 +102,9 @@ class TestSkillDiscovery: """Sequences shorter than min or longer than max should be excluded.""" # Only length-1 tools (below min_sequence_length=2) sd = SkillDiscovery( - min_frequency=2, min_sequence_length=2, max_sequence_length=3, + min_frequency=2, + min_sequence_length=2, + max_sequence_length=3, ) short_traces = [_make_trace(["a"], outcome=1.0) for _ in range(5)] result = sd.analyze_traces(short_traces) @@ -113,12 +113,11 @@ class TestSkillDiscovery: # Long sequence: with max_sequence_length=2, a 4-tool sequence # should only produce subsequences of length 2 sd2 = SkillDiscovery( - min_frequency=3, min_sequence_length=2, max_sequence_length=2, + min_frequency=3, + min_sequence_length=2, + max_sequence_length=2, ) - long_traces = [ - _make_trace(["a", "b", "c", "d"], outcome=1.0) - for _ in range(3) - ] + long_traces = [_make_trace(["a", "b", "c", "d"], outcome=1.0) for _ in range(3)] result2 = sd2.analyze_traces(long_traces) # All discovered skills should have exactly 2 tools for skill in result2: @@ -127,10 +126,7 @@ class TestSkillDiscovery: def test_to_skill_manifests(self): """Verify manifest dict format has expected keys.""" sd = SkillDiscovery(min_frequency=2, min_outcome=0.0) - traces = [ - _make_trace(["calc", "save"], outcome=0.9) - for _ in range(3) - ] + traces = [_make_trace(["calc", "save"], outcome=0.9) for _ in range(3)] sd.analyze_traces(traces) manifests = sd.to_skill_manifests() assert len(manifests) >= 1 @@ -150,15 +146,9 @@ class TestSkillDiscovery: """Higher frequency*outcome skills should come first.""" sd = SkillDiscovery(min_frequency=2, min_outcome=0.0) # Group A: high freq, high outcome - traces_a = [ - _make_trace(["alpha", "beta"], outcome=1.0) - for _ in range(10) - ] + traces_a = [_make_trace(["alpha", "beta"], outcome=1.0) for _ in range(10)] # Group B: low freq, low outcome - traces_b = [ - _make_trace(["gamma", "delta"], outcome=0.3) - for _ in range(2) - ] + traces_b = [_make_trace(["gamma", "delta"], outcome=0.3) for _ in range(2)] result = sd.analyze_traces(traces_a + traces_b) assert len(result) >= 2 # First should be the higher quality one @@ -172,7 +162,9 @@ class TestSkillDiscovery: sd = SkillDiscovery(min_frequency=3, min_outcome=0.5) traces = [ _make_dict_trace( - ["read", "compute", "write"], outcome=0.8, query=f"task {i}", + ["read", "compute", "write"], + outcome=0.8, + query=f"task {i}", ) for i in range(4) ] diff --git a/tests/learning/test_trial_runner.py b/tests/learning/test_trial_runner.py index c65c6d29..fd0fc042 100644 --- a/tests/learning/test_trial_runner.py +++ b/tests/learning/test_trial_runner.py @@ -185,8 +185,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_returns_trial_result( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: summary = self._make_summary() mock_runner_instance = MagicMock() @@ -218,8 +222,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_accuracy_from_summary( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: summary = self._make_summary(accuracy=0.92) mock_runner_cls.return_value.run.return_value = summary @@ -238,8 +246,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_tokens_summed( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: summary = self._make_summary( total_input_tokens=3000, @@ -261,8 +273,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_summary_attached( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: summary = self._make_summary() mock_runner_cls.return_value.run.return_value = summary @@ -281,8 +297,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_failure_modes_on_errors( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: summary = self._make_summary(errors=5) mock_runner_cls.return_value.run.return_value = summary @@ -302,8 +322,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_no_failure_modes_when_clean( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: summary = self._make_summary(errors=0) mock_runner_cls.return_value.run.return_value = summary @@ -322,8 +346,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_closes_backends( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: summary = self._make_summary() mock_runner_cls.return_value.run.return_value = summary @@ -345,8 +373,12 @@ class TestRunTrial: @patch("openjarvis.evals.cli._build_backend") @patch("openjarvis.evals.core.runner.EvalRunner") def test_run_trial_populates_sample_scores( - self, mock_runner_cls, mock_build_backend, mock_build_dataset, - mock_build_judge, mock_build_scorer, + self, + mock_runner_cls, + mock_build_backend, + mock_build_dataset, + mock_build_judge, + mock_build_scorer, ) -> None: from openjarvis.evals.core.types import EvalResult diff --git a/tests/learning/training/test_data.py b/tests/learning/training/test_data.py index f5dc8725..086a4763 100644 --- a/tests/learning/training/test_data.py +++ b/tests/learning/training/test_data.py @@ -78,7 +78,7 @@ class FakeTraceStore: self._traces = traces or [] def list_traces(self, *, limit: int = 10000, **kwargs: Any) -> list[Trace]: - return self._traces[: limit] + return self._traces[:limit] # --------------------------------------------------------------------------- diff --git a/tests/mcp/test_protocol.py b/tests/mcp/test_protocol.py index f397cc0a..b6d3b727 100644 --- a/tests/mcp/test_protocol.py +++ b/tests/mcp/test_protocol.py @@ -91,7 +91,10 @@ class TestMCPResponse: def test_error_response_with_data(self): resp = MCPResponse.error_response( - 1, INTERNAL_ERROR, "Oops", data={"detail": "stack"}, + 1, + INTERNAL_ERROR, + "Oops", + data={"detail": "stack"}, ) assert resp.error["data"] == {"detail": "stack"} diff --git a/tests/mcp/test_transport.py b/tests/mcp/test_transport.py index e43e0ec3..f1cb545e 100644 --- a/tests/mcp/test_transport.py +++ b/tests/mcp/test_transport.py @@ -71,7 +71,8 @@ class TestStdioTransport: def test_send_receive(self, tmp_path): """Use a simple Python echo script as the subprocess.""" script = tmp_path / "echo_server.py" - script.write_text(textwrap.dedent("""\ + script.write_text( + textwrap.dedent("""\ import sys import json for line in sys.stdin: @@ -86,7 +87,8 @@ class TestStdioTransport: } sys.stdout.write(json.dumps(resp) + "\\n") sys.stdout.flush() - """)) + """) + ) transport = StdioTransport([sys.executable, str(script)]) try: @@ -101,7 +103,8 @@ class TestStdioTransport: def test_multiple_requests(self, tmp_path): """Send multiple requests to the subprocess.""" script = tmp_path / "echo_server.py" - script.write_text(textwrap.dedent("""\ + script.write_text( + textwrap.dedent("""\ import sys import json for line in sys.stdin: @@ -116,7 +119,8 @@ class TestStdioTransport: } sys.stdout.write(json.dumps(resp) + "\\n") sys.stdout.flush() - """)) + """) + ) transport = StdioTransport([sys.executable, str(script)]) try: @@ -129,11 +133,13 @@ class TestStdioTransport: def test_close_terminates_process(self, tmp_path): script = tmp_path / "sleep_server.py" - script.write_text(textwrap.dedent("""\ + script.write_text( + textwrap.dedent("""\ import sys import time time.sleep(300) - """)) + """) + ) transport = StdioTransport([sys.executable, str(script)]) proc = transport._process diff --git a/tests/memory/test_bm25.py b/tests/memory/test_bm25.py index 6209104c..6aae1269 100644 --- a/tests/memory/test_bm25.py +++ b/tests/memory/test_bm25.py @@ -130,10 +130,7 @@ def test_event_bus_store(): mod.get_event_bus = lambda: bus try: backend.store("test event emission") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_STORE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_STORE] assert len(events) == 1 assert events[0].data["backend"] == "bm25" assert "doc_id" in events[0].data @@ -151,10 +148,7 @@ def test_event_bus_retrieve(): mod.get_event_bus = lambda: bus try: backend.retrieve("searchable") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_RETRIEVE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_RETRIEVE] assert len(events) == 1 assert events[0].data["backend"] == "bm25" assert events[0].data["num_results"] >= 1 diff --git a/tests/memory/test_colbert.py b/tests/memory/test_colbert.py index 598ca52c..1053fa87 100644 --- a/tests/memory/test_colbert.py +++ b/tests/memory/test_colbert.py @@ -64,16 +64,20 @@ def test_maxsim_scoring(): """Test the _maxsim function directly with synthetic tensors.""" backend = _make_backend() # 2 query tokens, dim 4 - q = torch.tensor([ - [1.0, 0.0, 0.0, 0.0], - [0.0, 1.0, 0.0, 0.0], - ]) + q = torch.tensor( + [ + [1.0, 0.0, 0.0, 0.0], + [0.0, 1.0, 0.0, 0.0], + ] + ) # 3 doc tokens, dim 4 - d = torch.tensor([ - [1.0, 0.0, 0.0, 0.0], - [0.0, 0.0, 1.0, 0.0], - [0.0, 1.0, 0.0, 0.0], - ]) + d = torch.tensor( + [ + [1.0, 0.0, 0.0, 0.0], + [0.0, 0.0, 1.0, 0.0], + [0.0, 1.0, 0.0, 0.0], + ] + ) score = backend._maxsim(q, d) # query[0] max sim: max(1.0, 0.0, 0.0) = 1.0 # query[1] max sim: max(0.0, 0.0, 1.0) = 1.0 @@ -134,10 +138,7 @@ def test_event_bus_store(): mod.get_event_bus = lambda: bus try: backend.store("test event emission") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_STORE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_STORE] assert len(events) == 1 assert events[0].data["backend"] == "colbert" assert "doc_id" in events[0].data @@ -155,10 +156,7 @@ def test_event_bus_retrieve(): mod.get_event_bus = lambda: bus try: backend.retrieve("searchable") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_RETRIEVE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_RETRIEVE] assert len(events) == 1 assert events[0].data["backend"] == "colbert" assert events[0].data["num_results"] >= 0 diff --git a/tests/memory/test_context.py b/tests/memory/test_context.py index 5f7f0a15..8aa036ad 100644 --- a/tests/memory/test_context.py +++ b/tests/memory/test_context.py @@ -17,6 +17,7 @@ from openjarvis.tools.storage.context import ( # -- Fake backend for testing ------------------------------------------------ + class _FakeMemory(MemoryBackend): """In-memory backend that returns pre-set results.""" @@ -113,7 +114,10 @@ def test_inject_context_filters_low_score(): messages = [Message(role=Role.USER, content="hello")] cfg = ContextConfig(min_score=0.1) augmented = inject_context( - "query", messages, backend, config=cfg, + "query", + messages, + backend, + config=cfg, ) # Low score filtered out — no context added assert len(augmented) == 1 @@ -130,7 +134,10 @@ def test_inject_context_respects_max_tokens(): messages = [Message(role=Role.USER, content="test")] cfg = ContextConfig(max_context_tokens=150) augmented = inject_context( - "query", messages, backend, config=cfg, + "query", + messages, + backend, + config=cfg, ) assert len(augmented) == 2 # system + user # Only one source should be cited @@ -145,7 +152,10 @@ def test_inject_context_disabled(): messages = [Message(role=Role.USER, content="hello")] cfg = ContextConfig(enabled=False) augmented = inject_context( - "query", messages, backend, config=cfg, + "query", + messages, + backend, + config=cfg, ) assert len(augmented) == 1 @@ -166,14 +176,12 @@ def test_inject_context_publishes_event(): messages = [Message(role=Role.USER, content="hello")] import openjarvis.tools.storage.context as mod + original = mod.get_event_bus mod.get_event_bus = lambda: bus try: inject_context("query", messages, backend) - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_RETRIEVE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_RETRIEVE] assert len(events) == 1 assert events[0].data["context_injection"] is True finally: diff --git a/tests/memory/test_faiss.py b/tests/memory/test_faiss.py index a7100ec1..6d0e37a9 100644 --- a/tests/memory/test_faiss.py +++ b/tests/memory/test_faiss.py @@ -30,14 +30,10 @@ class _FakeEmbedder(Embedder): def embed(self, texts: list[str]) -> np.ndarray: # type: ignore[override] results = [] for text in texts: - rng = np.random.RandomState( - hash(text) % 2**31 - ) + rng = np.random.RandomState(hash(text) % 2**31) vec = rng.randn(64).astype(np.float32) results.append(vec) - return np.array(results) if results else np.empty( - (0, 64), dtype=np.float32 - ) + return np.array(results) if results else np.empty((0, 64), dtype=np.float32) def dim(self) -> int: return 64 @@ -174,11 +170,7 @@ def test_event_bus_store(): mod.get_event_bus = lambda: bus try: backend.store("event test document") - events = [ - e - for e in bus.history - if e.event_type == EventType.MEMORY_STORE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_STORE] assert len(events) == 1 assert events[0].data["backend"] == "faiss" assert "doc_id" in events[0].data @@ -198,11 +190,7 @@ def test_event_bus_retrieve(): mod.get_event_bus = lambda: bus try: backend.retrieve("searchable") - events = [ - e - for e in bus.history - if e.event_type == EventType.MEMORY_RETRIEVE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_RETRIEVE] assert len(events) == 1 assert events[0].data["backend"] == "faiss" assert events[0].data["num_results"] >= 0 diff --git a/tests/memory/test_hybrid.py b/tests/memory/test_hybrid.py index dc2f5223..70326d78 100644 --- a/tests/memory/test_hybrid.py +++ b/tests/memory/test_hybrid.py @@ -46,12 +46,14 @@ class _FakeBackend(MemoryBackend): results = [] for doc_id, (content, source, meta) in self._docs.items(): if query.lower() in content.lower(): - results.append(RetrievalResult( - content=content, - score=1.0, - source=source, - metadata=meta, - )) + results.append( + RetrievalResult( + content=content, + score=1.0, + source=source, + metadata=meta, + ) + ) return results[:top_k] def delete(self, doc_id: str) -> bool: @@ -127,7 +129,9 @@ def test_rrf_custom_weights(): list1 = [RetrievalResult(content="A", score=1.0)] list2 = [RetrievalResult(content="B", score=1.0)] fused = reciprocal_rank_fusion( - [list1, list2], k=60, weights=[2.0, 1.0], + [list1, list2], + k=60, + weights=[2.0, 1.0], ) scores = {r.content: r.score for r in fused} # A has weight 2, B has weight 1, both at rank 0 @@ -191,19 +195,14 @@ def test_event_bus_store(): bus = EventBus(record_history=True) hybrid = _make_hybrid() import openjarvis.tools.storage.hybrid as mod + original = mod.get_event_bus mod.get_event_bus = lambda: bus try: hybrid.store("event test content") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_STORE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_STORE] assert len(events) >= 1 - assert any( - e.data.get("backend") == "hybrid" - for e in events - ) + assert any(e.data.get("backend") == "hybrid" for e in events) finally: mod.get_event_bus = original @@ -213,18 +212,13 @@ def test_event_bus_retrieve(): hybrid = _make_hybrid() hybrid.store("retrievable content here") import openjarvis.tools.storage.hybrid as mod + original = mod.get_event_bus mod.get_event_bus = lambda: bus try: hybrid.retrieve("retrievable") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_RETRIEVE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_RETRIEVE] assert len(events) >= 1 - assert any( - e.data.get("backend") == "hybrid" - for e in events - ) + assert any(e.data.get("backend") == "hybrid" for e in events) finally: mod.get_event_bus = original diff --git a/tests/memory/test_retrieval_quality.py b/tests/memory/test_retrieval_quality.py index d520c378..d49ce7f2 100644 --- a/tests/memory/test_retrieval_quality.py +++ b/tests/memory/test_retrieval_quality.py @@ -39,7 +39,8 @@ def _make_sqlite(tmp_path): def _make_bm25(): bm25_mod = pytest.importorskip( - "openjarvis.tools.storage.bm25", exc_type=ImportError, + "openjarvis.tools.storage.bm25", + exc_type=ImportError, ) BM25Memory = bm25_mod.BM25Memory if not MemoryRegistry.contains("bm25"): diff --git a/tests/memory/test_sqlite.py b/tests/memory/test_sqlite.py index bbbf2dc7..987c6cb4 100644 --- a/tests/memory/test_sqlite.py +++ b/tests/memory/test_sqlite.py @@ -134,14 +134,12 @@ def test_event_bus_integration_store(tmp_path: Path): backend = _make_backend(tmp_path) # Monkey-patch the global bus for this test import openjarvis.tools.storage.sqlite as mod + original = mod.get_event_bus mod.get_event_bus = lambda: bus try: backend.store("test event emission") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_STORE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_STORE] assert len(events) == 1 assert events[0].data["backend"] == "sqlite" finally: @@ -154,14 +152,12 @@ def test_event_bus_integration_retrieve(tmp_path: Path): backend = _make_backend(tmp_path) backend.store("searchable content for events") import openjarvis.tools.storage.sqlite as mod + original = mod.get_event_bus mod.get_event_bus = lambda: bus try: backend.retrieve("searchable") - events = [ - e for e in bus.history - if e.event_type == EventType.MEMORY_RETRIEVE - ] + events = [e for e in bus.history if e.event_type == EventType.MEMORY_RETRIEVE] assert len(events) == 1 assert events[0].data["backend"] == "sqlite" finally: diff --git a/tests/memory/test_storage_suite.py b/tests/memory/test_storage_suite.py index ae5bbbcf..9adbee6f 100644 --- a/tests/memory/test_storage_suite.py +++ b/tests/memory/test_storage_suite.py @@ -20,7 +20,8 @@ def _make_sqlite(tmp_path): def _make_bm25(): bm25_mod = pytest.importorskip( - "openjarvis.tools.storage.bm25", exc_type=ImportError, + "openjarvis.tools.storage.bm25", + exc_type=ImportError, ) BM25Memory = bm25_mod.BM25Memory if not MemoryRegistry.contains("bm25"): @@ -171,9 +172,7 @@ class TestStorageSuiteOptional: def test_delete_document(self, backend_key, tmp_path): if backend_key == "hybrid": - pytest.skip( - "HybridMemory sub-backend BM25 lacks delete()" - ) + pytest.skip("HybridMemory sub-backend BM25 lacks delete()") backend = _make_backend(backend_key, tmp_path) doc_id = backend.store("content to delete") assert backend.delete(doc_id) is True @@ -181,9 +180,7 @@ class TestStorageSuiteOptional: def test_clear_all(self, backend_key, tmp_path): if backend_key == "hybrid": - pytest.skip( - "HybridMemory sub-backend BM25 lacks clear()" - ) + pytest.skip("HybridMemory sub-backend BM25 lacks clear()") backend = _make_backend(backend_key, tmp_path) backend.store("first document") backend.store("second document") diff --git a/tests/operators/test_operators.py b/tests/operators/test_operators.py index b67bffe5..065dc596 100644 --- a/tests/operators/test_operators.py +++ b/tests/operators/test_operators.py @@ -314,8 +314,11 @@ name = "Discovered" mgr = OperatorManager(system) m = OperatorManifest( - id="test_op", name="Test", - tools=["think"], schedule_type="interval", schedule_value="60", + id="test_op", + name="Test", + tools=["think"], + schedule_type="interval", + schedule_value="60", ) mgr.register(m) task_id = mgr.activate("test_op") @@ -352,8 +355,10 @@ name = "Discovered" mgr = OperatorManager(system) m = OperatorManifest( - id="meta_test", name="Meta", - system_prompt="Do stuff", temperature=0.5, + id="meta_test", + name="Meta", + system_prompt="Do stuff", + temperature=0.5, ) mgr.register(m) mgr.activate("meta_test") @@ -452,8 +457,10 @@ name = "Discovered" mgr = OperatorManager(system) m = OperatorManifest( - id="run_test", name="Run", - system_prompt="Test prompt", tools=["think"], + id="run_test", + name="Run", + system_prompt="Test prompt", + tools=["think"], ) mgr.register(m) @@ -494,7 +501,8 @@ class TestOperativeAgent: engine = FakeEngine([{"content": "Response with prompt."}]) agent = OperativeAgent( - engine, "test-model", + engine, + "test-model", system_prompt="You are a test operator.", ) result = agent.run("Execute tick") @@ -504,8 +512,7 @@ class TestOperativeAgent: call = engine.calls[0] messages = call["messages"] assert any( - m.role.value == "system" and "test operator" in m.content - for m in messages + m.role.value == "system" and "test operator" in m.content for m in messages ) def test_run_loads_session(self): @@ -521,7 +528,8 @@ class TestOperativeAgent: engine = FakeEngine([{"content": "New tick done."}]) agent = OperativeAgent( - engine, "test-model", + engine, + "test-model", operator_id="test_op", session_store=session_store, ) @@ -535,7 +543,8 @@ class TestOperativeAgent: engine = FakeEngine([{"content": "Tick response."}]) agent = OperativeAgent( - engine, "test-model", + engine, + "test-model", operator_id="save_test", session_store=session_store, ) @@ -555,7 +564,8 @@ class TestOperativeAgent: engine = FakeEngine([{"content": "State recalled."}]) agent = OperativeAgent( - engine, "test-model", + engine, + "test-model", operator_id="recall_test", memory_backend=memory, ) @@ -583,22 +593,25 @@ class TestOperativeAgent: tool.spec.taint_labels = [] tool.run = MagicMock(return_value="Thought result") - engine = FakeEngine([ - { - "content": "", - "tool_calls": [ - { - "id": "call_1", - "name": "think", - "arguments": '{"thought": "test"}', - }, - ], - }, - {"content": "Final answer after tool use."}, - ]) + engine = FakeEngine( + [ + { + "content": "", + "tool_calls": [ + { + "id": "call_1", + "name": "think", + "arguments": '{"thought": "test"}', + }, + ], + }, + {"content": "Final answer after tool use."}, + ] + ) agent = OperativeAgent( - engine, "test-model", + engine, + "test-model", tools=[tool], ) result = agent.run("Do something") @@ -620,7 +633,8 @@ class TestOperativeAgent: engine = FakeEngine([{"content": "Tick complete."}]) agent = OperativeAgent( - engine, "test-model", + engine, + "test-model", operator_id="persist_test", memory_backend=memory, ) @@ -660,7 +674,8 @@ class TestOperativeAgent: tool.run = MagicMock(return_value="thought") agent = OperativeAgent( - engine, "test-model", + engine, + "test-model", tools=[tool], max_turns=3, ) @@ -685,15 +700,26 @@ class TestSystemAskPassthrough: # Patch _run_agent to capture kwargs captured = {} - def patched_run_agent(self, query, messages, agent_name, tool_names, - temperature, max_tokens, **kwargs): + def patched_run_agent( + self, + query, + messages, + agent_name, + tool_names, + temperature, + max_tokens, + **kwargs, + ): captured.update(kwargs) return {"content": "OK"} with patch.object(JarvisSystem, "_run_agent", patched_run_agent): real_system = JarvisSystem( - config=system.config, bus=system.bus, - engine=engine, engine_key="test", model="test-model", + config=system.config, + bus=system.bus, + engine=engine, + engine_key="test", + model="test-model", agent_name="operative", ) real_system.ask("test", system_prompt="Custom prompt") @@ -709,15 +735,26 @@ class TestSystemAskPassthrough: captured = {} - def patched_run_agent(self, query, messages, agent_name, tool_names, - temperature, max_tokens, **kwargs): + def patched_run_agent( + self, + query, + messages, + agent_name, + tool_names, + temperature, + max_tokens, + **kwargs, + ): captured.update(kwargs) return {"content": "OK"} with patch.object(JarvisSystem, "_run_agent", patched_run_agent): real_system = JarvisSystem( - config=system.config, bus=system.bus, - engine=engine, engine_key="test", model="test-model", + config=system.config, + bus=system.bus, + engine=engine, + engine_key="test", + model="test-model", agent_name="operative", ) real_system.ask("test", operator_id="my_op") @@ -762,8 +799,10 @@ class TestSchedulerOperatorExecution: mock_system.ask.assert_called_once() call_kwargs = mock_system.ask.call_args # The scheduler passes agent and tools - assert call_kwargs.kwargs.get("agent") == "operative" or \ - call_kwargs[1].get("agent") == "operative" + assert ( + call_kwargs.kwargs.get("agent") == "operative" + or call_kwargs[1].get("agent") == "operative" + ) # --------------------------------------------------------------------------- @@ -815,6 +854,7 @@ class TestAgentRegistration: # Re-register if cleared by another test if not AgentRegistry.contains("operative"): from openjarvis.agents.operative import OperativeAgent + AgentRegistry.register_value("operative", OperativeAgent) assert AgentRegistry.contains("operative") diff --git a/tests/prompt/test_builder.py b/tests/prompt/test_builder.py index 13cc20ac..14b1d0f9 100644 --- a/tests/prompt/test_builder.py +++ b/tests/prompt/test_builder.py @@ -20,6 +20,7 @@ def memory_dir(tmp_path: Path) -> Path: def test_build_frozen_prefix(memory_dir: Path): from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( agent_template="You are Jarvis.", memory_files_config=MemoryFilesConfig( @@ -38,6 +39,7 @@ def test_build_frozen_prefix(memory_dir: Path): def test_frozen_prefix_stability(memory_dir: Path): from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( agent_template="You are Jarvis.", memory_files_config=MemoryFilesConfig( @@ -55,6 +57,7 @@ def test_frozen_prefix_stability(memory_dir: Path): def test_char_limit_truncation(memory_dir: Path): from openjarvis.prompt.builder import SystemPromptBuilder + (memory_dir / "SOUL.md").write_text("x" * 10000) builder = SystemPromptBuilder( agent_template="You are Jarvis.", @@ -72,6 +75,7 @@ def test_char_limit_truncation(memory_dir: Path): def test_skill_index_in_prompt(memory_dir: Path): from openjarvis.prompt.builder import SystemPromptBuilder + skills = [("api_health_check", "Check API health across all endpoints")] builder = SystemPromptBuilder( agent_template="You are Jarvis.", @@ -90,6 +94,7 @@ def test_skill_index_in_prompt(memory_dir: Path): def test_dynamic_section_appended(memory_dir: Path): from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( agent_template="You are Jarvis.", memory_files_config=MemoryFilesConfig( @@ -106,6 +111,7 @@ def test_dynamic_section_appended(memory_dir: Path): def test_missing_files_handled(tmp_path: Path): from openjarvis.prompt.builder import SystemPromptBuilder + builder = SystemPromptBuilder( agent_template="You are Jarvis.", memory_files_config=MemoryFilesConfig( diff --git a/tests/recipes/test_loader.py b/tests/recipes/test_loader.py index 02fa024b..6a04d700 100644 --- a/tests/recipes/test_loader.py +++ b/tests/recipes/test_loader.py @@ -74,7 +74,7 @@ class TestLoadRecipe: def test_load_recipe_defaults(self, tmp_path: Path) -> None: """Minimal TOML should yield sensible defaults.""" toml_file = tmp_path / "minimal.toml" - toml_file.write_text("[recipe]\nname = \"minimal\"\n") + toml_file.write_text('[recipe]\nname = "minimal"\n') recipe = load_recipe(toml_file) @@ -87,7 +87,7 @@ class TestLoadRecipe: def test_load_recipe_name_from_filename(self, tmp_path: Path) -> None: """When [recipe] has no name, use the file stem.""" toml_file = tmp_path / "my_recipe.toml" - toml_file.write_text("[recipe]\ndescription = \"no name\"\n") + toml_file.write_text('[recipe]\ndescription = "no name"\n') recipe = load_recipe(toml_file) assert recipe.name == "my_recipe" @@ -104,9 +104,7 @@ class TestDiscoverRecipes: def test_discover_extra_dirs(self, tmp_path: Path) -> None: toml_file = tmp_path / "custom.toml" - toml_file.write_text( - '[recipe]\nname = "custom"\ndescription = "extra"\n' - ) + toml_file.write_text('[recipe]\nname = "custom"\ndescription = "extra"\n') recipes = discover_recipes(extra_dirs=[tmp_path]) names = {r.name for r in recipes} assert "custom" in names diff --git a/tests/sandbox/test_runner.py b/tests/sandbox/test_runner.py index 308dd84d..308ab272 100644 --- a/tests/sandbox/test_runner.py +++ b/tests/sandbox/test_runner.py @@ -70,7 +70,9 @@ class TestBuildDockerArgs: runner = ContainerRunner() with patch("shutil.which", return_value="/usr/bin/docker"): args = runner._build_docker_args( - "test-container", [], None, + "test-container", + [], + None, ) assert "/usr/bin/docker" in args assert "run" in args @@ -85,7 +87,9 @@ class TestBuildDockerArgs: runner = ContainerRunner() with patch("shutil.which", return_value="/usr/bin/docker"): args = runner._build_docker_args( - "test-container", ["/data/project"], None, + "test-container", + ["/data/project"], + None, ) assert "-v" in args idx = args.index("-v") @@ -95,7 +99,9 @@ class TestBuildDockerArgs: runner = ContainerRunner() with patch("shutil.which", return_value="/usr/bin/docker"): args = runner._build_docker_args( - "test-container", [], {"FOO": "bar"}, + "test-container", + [], + {"FOO": "bar"}, ) assert "-e" in args idx = args.index("-e") @@ -106,40 +112,62 @@ class TestContainerRunnerRun: def test_successful_run(self): runner = ContainerRunner() output = _wrap_output({"content": "Hello!"}) - with patch("shutil.which", return_value="/usr/bin/docker"), \ - patch("subprocess.run", return_value=_mock_proc( - stdout=output, - )): + with ( + patch("shutil.which", return_value="/usr/bin/docker"), + patch( + "subprocess.run", + return_value=_mock_proc( + stdout=output, + ), + ), + ): result = runner.run({"prompt": "test"}) assert result["content"] == "Hello!" def test_timeout(self): runner = ContainerRunner(timeout=5) - with patch("shutil.which", return_value="/usr/bin/docker"), \ - patch("subprocess.run", side_effect=subprocess.TimeoutExpired( - cmd=["docker"], timeout=5, - )), \ - patch.object(runner, "stop"): + with ( + patch("shutil.which", return_value="/usr/bin/docker"), + patch( + "subprocess.run", + side_effect=subprocess.TimeoutExpired( + cmd=["docker"], + timeout=5, + ), + ), + patch.object(runner, "stop"), + ): result = runner.run({"prompt": "test"}) assert result["error"] is True assert result["error_type"] == "timeout" def test_nonzero_exit(self): runner = ContainerRunner() - with patch("shutil.which", return_value="/usr/bin/docker"), \ - patch("subprocess.run", return_value=_mock_proc( - returncode=1, stderr="OOM killed", - )): + with ( + patch("shutil.which", return_value="/usr/bin/docker"), + patch( + "subprocess.run", + return_value=_mock_proc( + returncode=1, + stderr="OOM killed", + ), + ), + ): result = runner.run({"prompt": "test"}) assert result["error"] is True assert "OOM killed" in result["content"] def test_no_sentinel_output(self): runner = ContainerRunner() - with patch("shutil.which", return_value="/usr/bin/docker"), \ - patch("subprocess.run", return_value=_mock_proc( - stdout="plain text output", - )): + with ( + patch("shutil.which", return_value="/usr/bin/docker"), + patch( + "subprocess.run", + return_value=_mock_proc( + stdout="plain text output", + ), + ), + ): result = runner.run({"prompt": "test"}) assert result["content"] == "plain text output" @@ -155,8 +183,10 @@ class TestContainerRunnerRuntimeCheck: class TestCleanupOrphans: def test_cleanup_orphans(self): runner = ContainerRunner() - with patch("shutil.which", return_value="/usr/bin/docker"), \ - patch("subprocess.run") as mock_run: + with ( + patch("shutil.which", return_value="/usr/bin/docker"), + patch("subprocess.run") as mock_run, + ): mock_run.return_value = _mock_proc(stdout="abc123\ndef456") runner.cleanup_orphans() assert mock_run.call_count == 2 # ps + rm @@ -198,8 +228,10 @@ class TestSandboxedAgentInit: runner = MagicMock(spec=ContainerRunner) agent = SandboxedAgent( - inner, runner, - engine=inner._engine, model="test-model", + inner, + runner, + engine=inner._engine, + model="test-model", ) assert agent._wrapped_agent is inner assert agent._runner is runner @@ -219,7 +251,10 @@ class TestSandboxedAgentRun: engine = MagicMock() agent = SandboxedAgent( - inner, runner, engine=engine, model="test-model", + inner, + runner, + engine=engine, + model="test-model", ) result = agent.run("hello") @@ -246,7 +281,10 @@ class TestSandboxedAgentRun: engine = MagicMock() agent = SandboxedAgent( - inner, runner, engine=engine, model="m", + inner, + runner, + engine=engine, + model="m", ) result = agent.run("compute") @@ -264,7 +302,11 @@ class TestSandboxedAgentRun: bus = EventBus(record_history=True) engine = MagicMock() agent = SandboxedAgent( - inner, runner, engine=engine, model="m", bus=bus, + inner, + runner, + engine=engine, + model="m", + bus=bus, ) agent.run("test") diff --git a/tests/scheduler/test_scheduler.py b/tests/scheduler/test_scheduler.py index abdddeda..0643bc15 100644 --- a/tests/scheduler/test_scheduler.py +++ b/tests/scheduler/test_scheduler.py @@ -164,15 +164,20 @@ class TestComputeNextRun: def test_once_not_yet_run(self, scheduler): target = "2099-06-15T12:00:00+00:00" task = ScheduledTask( - id="t", prompt="p", schedule_type="once", - schedule_value=target, last_run=None, + id="t", + prompt="p", + schedule_type="once", + schedule_value=target, + last_run=None, ) next_run = scheduler._compute_next_run(task) assert next_run == target def test_once_already_run(self, scheduler): task = ScheduledTask( - id="t", prompt="p", schedule_type="once", + id="t", + prompt="p", + schedule_type="once", schedule_value="2099-06-15T12:00:00+00:00", last_run="2099-06-15T12:01:00+00:00", ) @@ -181,7 +186,9 @@ class TestComputeNextRun: def test_cron_fallback(self, scheduler): task = ScheduledTask( - id="t", prompt="p", schedule_type="cron", + id="t", + prompt="p", + schedule_type="cron", schedule_value="30 2 * * *", ) next_run = scheduler._compute_next_run(task) @@ -266,7 +273,9 @@ class TestExecuteTask: sched = TaskScheduler(store, system=mock_system, poll_interval=1) task = sched.create_task( - "what is 2+2?", "once", "2026-01-01T00:00:00+00:00", + "what is 2+2?", + "once", + "2026-01-01T00:00:00+00:00", tools="calculator,think", ) sched._execute_task(task) diff --git a/tests/scheduler/test_store.py b/tests/scheduler/test_store.py index 00fc8f46..cb4098be 100644 --- a/tests/scheduler/test_store.py +++ b/tests/scheduler/test_store.py @@ -152,8 +152,8 @@ class TestRunLogs: for i in range(20): store.log_run( task_id="t1", - started_at=f"2026-01-{i+1:02d}T00:00:00+00:00", - finished_at=f"2026-01-{i+1:02d}T00:01:00+00:00", + started_at=f"2026-01-{i + 1:02d}T00:00:00+00:00", + finished_at=f"2026-01-{i + 1:02d}T00:01:00+00:00", success=True, ) logs = store.get_run_logs("t1", limit=5) diff --git a/tests/security/test_audit.py b/tests/security/test_audit.py index b358e53e..6e86e8d5 100644 --- a/tests/security/test_audit.py +++ b/tests/security/test_audit.py @@ -61,18 +61,24 @@ class TestAuditLogger: def test_query_by_type(self, tmp_path: Path) -> None: logger = AuditLogger(db_path=tmp_path / "audit.db") - logger.log(SecurityEvent( - event_type=SecurityEventType.SECRET_DETECTED, - timestamp=time.time(), - )) - logger.log(SecurityEvent( - event_type=SecurityEventType.PII_DETECTED, - timestamp=time.time(), - )) - logger.log(SecurityEvent( - event_type=SecurityEventType.SECRET_DETECTED, - timestamp=time.time(), - )) + logger.log( + SecurityEvent( + event_type=SecurityEventType.SECRET_DETECTED, + timestamp=time.time(), + ) + ) + logger.log( + SecurityEvent( + event_type=SecurityEventType.PII_DETECTED, + timestamp=time.time(), + ) + ) + logger.log( + SecurityEvent( + event_type=SecurityEventType.SECRET_DETECTED, + timestamp=time.time(), + ) + ) secrets = logger.query(event_type="secret_detected") assert len(secrets) == 2 @@ -87,14 +93,18 @@ class TestAuditLogger: t1 = time.time() - 100 t2 = time.time() - logger.log(SecurityEvent( - event_type=SecurityEventType.SECRET_DETECTED, - timestamp=t1, - )) - logger.log(SecurityEvent( - event_type=SecurityEventType.SECRET_DETECTED, - timestamp=t2, - )) + logger.log( + SecurityEvent( + event_type=SecurityEventType.SECRET_DETECTED, + timestamp=t1, + ) + ) + logger.log( + SecurityEvent( + event_type=SecurityEventType.SECRET_DETECTED, + timestamp=t2, + ) + ) recent = logger.query(since=t2 - 1) assert len(recent) == 1 @@ -146,10 +156,12 @@ class TestAuditLogger: def test_close_and_reopen(self, tmp_path: Path) -> None: db_path = tmp_path / "audit.db" logger = AuditLogger(db_path=db_path) - logger.log(SecurityEvent( - event_type=SecurityEventType.SECRET_DETECTED, - timestamp=time.time(), - )) + logger.log( + SecurityEvent( + event_type=SecurityEventType.SECRET_DETECTED, + timestamp=time.time(), + ) + ) logger.close() logger2 = AuditLogger(db_path=db_path) diff --git a/tests/security/test_capabilities.py b/tests/security/test_capabilities.py index 3b1a592e..958f2328 100644 --- a/tests/security/test_capabilities.py +++ b/tests/security/test_capabilities.py @@ -18,9 +18,16 @@ class TestCapability: def test_all_capabilities_exist(self): expected = { - "file:read", "file:write", "network:fetch", "code:execute", - "memory:read", "memory:write", "channel:send", "tool:invoke", - "schedule:create", "system:admin", + "file:read", + "file:write", + "network:fetch", + "code:execute", + "memory:read", + "memory:write", + "channel:send", + "tool:invoke", + "schedule:create", + "system:admin", } actual = {c.value for c in Capability} assert expected == actual diff --git a/tests/security/test_credential_stripper.py b/tests/security/test_credential_stripper.py index 01f7f781..b7a65d2f 100644 --- a/tests/security/test_credential_stripper.py +++ b/tests/security/test_credential_stripper.py @@ -3,6 +3,7 @@ from __future__ import annotations def test_strips_openai_key(): from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() text = ( "Error: auth failed with key " @@ -15,6 +16,7 @@ def test_strips_openai_key(): def test_strips_aws_key(): from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() text = "Using credentials AKIAIOSFODNN7EXAMPLE for access" result = stripper.strip(text) @@ -24,6 +26,7 @@ def test_strips_aws_key(): def test_strips_github_token(): from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() text = "Token: ghp_ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghij" result = stripper.strip(text) @@ -32,6 +35,7 @@ def test_strips_github_token(): def test_preserves_normal_text(): from openjarvis.security.credential_stripper import CredentialStripper + stripper = CredentialStripper() text = "The function returned 42 results." result = stripper.strip(text) @@ -40,6 +44,7 @@ def test_preserves_normal_text(): def test_tool_output_wrapping(): from openjarvis.security.credential_stripper import wrap_tool_output + content = "Search results: found 3 items" wrapped = wrap_tool_output("web_search", content, success=True) assert '' in wrapped diff --git a/tests/security/test_guardrails.py b/tests/security/test_guardrails.py index 5eee6dc8..67fa72e0 100644 --- a/tests/security/test_guardrails.py +++ b/tests/security/test_guardrails.py @@ -111,7 +111,10 @@ class TestGuardrailsEngineInputScanning: bus = EventBus(record_history=True) mock = _make_mock_engine("OK") ge = GuardrailsEngine( - mock, mode=RedactionMode.WARN, scan_input=True, bus=bus, + mock, + mode=RedactionMode.WARN, + scan_input=True, + bus=bus, ) secret = "my key sk-abc123def456ghi789jkl012" @@ -129,7 +132,9 @@ class TestGuardrailsEngineInputScanning: """REDACT mode on input must send redacted messages.""" mock = _make_mock_engine("OK") ge = GuardrailsEngine( - mock, mode=RedactionMode.REDACT, scan_input=True, + mock, + mode=RedactionMode.REDACT, + scan_input=True, ) secret = "my key sk-abc123def456ghi789jkl012" @@ -147,7 +152,9 @@ class TestGuardrailsEngineInputScanning: bus = EventBus(record_history=True) mock = _make_mock_engine("OK") ge = GuardrailsEngine( - mock, mode=RedactionMode.WARN, scan_input=False, + mock, + mode=RedactionMode.WARN, + scan_input=False, bus=bus, ) diff --git a/tests/security/test_security_wiring.py b/tests/security/test_security_wiring.py index 9651bd4f..eca55808 100644 --- a/tests/security/test_security_wiring.py +++ b/tests/security/test_security_wiring.py @@ -1,4 +1,5 @@ """Verify security wiring reaches agents and ToolExecutor.""" + from __future__ import annotations from unittest.mock import MagicMock diff --git a/tests/security/test_setup_security.py b/tests/security/test_setup_security.py index 754dc73f..2ad1c181 100644 --- a/tests/security/test_setup_security.py +++ b/tests/security/test_setup_security.py @@ -1,4 +1,5 @@ """Tests for setup_security() helper.""" + from __future__ import annotations from unittest.mock import MagicMock @@ -34,6 +35,7 @@ def _make_config(*, enabled: bool = True, caps_enabled: bool = False) -> JarvisC def _has_rust() -> bool: try: import openjarvis_rust # noqa: F401 + return True except ImportError: return False diff --git a/tests/security/test_severity_policy.py b/tests/security/test_severity_policy.py index 907ca826..37c34c00 100644 --- a/tests/security/test_severity_policy.py +++ b/tests/security/test_severity_policy.py @@ -4,6 +4,7 @@ from __future__ import annotations def test_severity_policy_block(): from openjarvis.security.severity_policy import SeverityPolicy from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() assert policy.action_for(ThreatLevel.CRITICAL) == "block" @@ -11,6 +12,7 @@ def test_severity_policy_block(): def test_severity_policy_warn(): from openjarvis.security.severity_policy import SeverityPolicy from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() assert policy.action_for(ThreatLevel.HIGH) == "warn" @@ -18,6 +20,7 @@ def test_severity_policy_warn(): def test_severity_policy_sanitize(): from openjarvis.security.severity_policy import SeverityPolicy from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() assert policy.action_for(ThreatLevel.MEDIUM) == "sanitize" @@ -25,5 +28,6 @@ def test_severity_policy_sanitize(): def test_severity_policy_log(): from openjarvis.security.severity_policy import SeverityPolicy from openjarvis.security.types import ThreatLevel + policy = SeverityPolicy() assert policy.action_for(ThreatLevel.LOW) == "log" diff --git a/tests/security/test_signing.py b/tests/security/test_signing.py index 2562c92e..5562448f 100644 --- a/tests/security/test_signing.py +++ b/tests/security/test_signing.py @@ -16,6 +16,7 @@ class TestSigning: def test_generate_keypair(self): _skip_if_no_cryptography() from openjarvis.security.signing import generate_keypair + kp = generate_keypair() assert len(kp.private_key) == 32 assert len(kp.public_key) == 32 @@ -23,6 +24,7 @@ class TestSigning: def test_sign_and_verify(self): _skip_if_no_cryptography() from openjarvis.security.signing import generate_keypair, sign, verify + kp = generate_keypair() data = b"hello world" sig = sign(data, kp.private_key) @@ -32,6 +34,7 @@ class TestSigning: def test_verify_wrong_data(self): _skip_if_no_cryptography() from openjarvis.security.signing import generate_keypair, sign, verify + kp = generate_keypair() sig = sign(b"hello", kp.private_key) assert not verify(b"wrong", sig, kp.public_key) @@ -39,6 +42,7 @@ class TestSigning: def test_verify_wrong_key(self): _skip_if_no_cryptography() from openjarvis.security.signing import generate_keypair, sign, verify + kp1 = generate_keypair() kp2 = generate_keypair() sig = sign(b"data", kp1.private_key) @@ -47,6 +51,7 @@ class TestSigning: def test_sign_b64(self): _skip_if_no_cryptography() from openjarvis.security.signing import generate_keypair, sign_b64, verify_b64 + kp = generate_keypair() data = b"test data" sig_b64 = sign_b64(data, kp.private_key) @@ -56,6 +61,7 @@ class TestSigning: def test_verify_b64_invalid(self): _skip_if_no_cryptography() from openjarvis.security.signing import generate_keypair, verify_b64 + kp = generate_keypair() assert not verify_b64(b"data", "invalid-base64!!!", kp.public_key) @@ -68,6 +74,7 @@ class TestSigning: sign, verify, ) + kp = generate_keypair() priv_path = str(tmp_path / "private.key") pub_path = str(tmp_path / "public.key") diff --git a/tests/security/test_subprocess_sandbox.py b/tests/security/test_subprocess_sandbox.py index 1bca470c..92fe2d95 100644 --- a/tests/security/test_subprocess_sandbox.py +++ b/tests/security/test_subprocess_sandbox.py @@ -21,8 +21,16 @@ class TestBuildSafeEnv: env = build_safe_env() # All keys should be from the safe set safe_keys = { - "PATH", "HOME", "USER", "LANG", "TERM", "SHELL", - "LC_ALL", "LC_CTYPE", "TMPDIR", "TZ", + "PATH", + "HOME", + "USER", + "LANG", + "TERM", + "SHELL", + "LC_ALL", + "LC_CTYPE", + "TMPDIR", + "TZ", } for key in env: assert key in safe_keys @@ -78,7 +86,8 @@ class TestRunSandboxed: os.environ["TEST_SECRET"] = "super_secret_value" try: result = run_sandboxed( - 'echo "val=$TEST_SECRET"', timeout=10.0, + 'echo "val=$TEST_SECRET"', + timeout=10.0, ) assert result.returncode == 0 assert "super_secret_value" not in result.stdout diff --git a/tests/server/test_agent_manager_routes.py b/tests/server/test_agent_manager_routes.py index b694edd5..498bc167 100644 --- a/tests/server/test_agent_manager_routes.py +++ b/tests/server/test_agent_manager_routes.py @@ -49,10 +49,13 @@ class TestAgentManagerRoutes: assert resp.json()["agents"] == [] def test_create_agent(self, client): - resp = client.post("/v1/managed-agents", json={ - "name": "researcher", - "agent_type": "monitor_operative", - }) + resp = client.post( + "/v1/managed-agents", + json={ + "name": "researcher", + "agent_type": "monitor_operative", + }, + ) assert resp.status_code == 200 data = resp.json() assert data["name"] == "researcher" @@ -95,9 +98,12 @@ class TestAgentManagerRoutes: def test_create_task(self, client): create_resp = client.post("/v1/managed-agents", json={"name": "worker"}) agent_id = create_resp.json()["id"] - resp = client.post(f"/v1/managed-agents/{agent_id}/tasks", json={ - "description": "Find papers on reasoning", - }) + resp = client.post( + f"/v1/managed-agents/{agent_id}/tasks", + json={ + "description": "Find papers on reasoning", + }, + ) assert resp.status_code == 200 assert resp.json()["description"] == "Find papers on reasoning" @@ -113,10 +119,13 @@ class TestAgentManagerRoutes: create_resp = client.post("/v1/managed-agents", json={"name": "slacker"}) agent_id = create_resp.json()["id"] # Bind - bind_resp = client.post(f"/v1/managed-agents/{agent_id}/channels", json={ - "channel_type": "slack", - "config": {"channel": "#research"}, - }) + bind_resp = client.post( + f"/v1/managed-agents/{agent_id}/channels", + json={ + "channel_type": "slack", + "config": {"channel": "#research"}, + }, + ) assert bind_resp.status_code == 200 binding_id = bind_resp.json()["id"] # List diff --git a/tests/server/test_api_routes.py b/tests/server/test_api_routes.py index d2856f21..d846c348 100644 --- a/tests/server/test_api_routes.py +++ b/tests/server/test_api_routes.py @@ -1,4 +1,5 @@ """Tests for extended API routes.""" + import pytest fastapi = pytest.importorskip("fastapi") diff --git a/tests/server/test_channel_routes.py b/tests/server/test_channel_routes.py index c3da6ecd..eab9210b 100644 --- a/tests/server/test_channel_routes.py +++ b/tests/server/test_channel_routes.py @@ -47,7 +47,8 @@ def app_with_bridge(mock_engine, mock_bridge): from openjarvis.server.app import create_app return create_app( - mock_engine, "test-model", + mock_engine, + "test-model", channel_bridge=mock_bridge, ) diff --git a/tests/server/test_middleware.py b/tests/server/test_middleware.py index 07dac292..d2016082 100644 --- a/tests/server/test_middleware.py +++ b/tests/server/test_middleware.py @@ -48,6 +48,7 @@ class TestSecurityHeaders: if middleware_cls is None: # starlette not installed -- skip import pytest + pytest.skip("starlette not available") assert middleware_cls is not None assert callable(middleware_cls) @@ -55,6 +56,7 @@ class TestSecurityHeaders: def test_middleware_adds_headers(self) -> None: """Middleware adds all security headers to responses.""" import pytest + fastapi = pytest.importorskip("fastapi") from fastapi.testclient import TestClient @@ -80,6 +82,7 @@ class TestSecurityHeaders: def test_middleware_skips_options(self) -> None: """OPTIONS requests pass through without security headers.""" import pytest + fastapi = pytest.importorskip("fastapi") from fastapi.middleware.cors import CORSMiddleware from fastapi.testclient import TestClient diff --git a/tests/server/test_model_management.py b/tests/server/test_model_management.py index 7c5f99ef..7e300135 100644 --- a/tests/server/test_model_management.py +++ b/tests/server/test_model_management.py @@ -71,7 +71,6 @@ class TestModelPull: engine = _make_ollama_engine() client = TestClient(_app(engine, engine_name="ollama")) - mock_resp = MagicMock() mock_resp.status_code = 200 mock_resp.raise_for_status = MagicMock() @@ -157,11 +156,14 @@ class TestStreamingResilience: app = create_app(engine, "test-model") client = TestClient(app) - resp = client.post("/v1/chat/completions", json={ - "model": "bad-model", - "messages": [{"role": "user", "content": "Hello"}], - "stream": True, - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "bad-model", + "messages": [{"role": "user", "content": "Hello"}], + "stream": True, + }, + ) assert resp.status_code == 200 # Should contain partial content + error message + [DONE] @@ -176,11 +178,14 @@ class TestStreamingResilience: app = create_app(engine, "test-model") client = TestClient(app) - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - "stream": True, - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + "stream": True, + }, + ) assert resp.status_code == 200 # Collect tokens @@ -203,18 +208,22 @@ class TestStreamingResilience: agent = MagicMock() agent.agent_id = "simple" agent.run.return_value = AgentResult( - content="agent response", turns=1, + content="agent response", + turns=1, ) app = create_app(engine, "test-model", agent=agent) client = TestClient(app) - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - "stream": True, - # No tools — should use direct engine stream - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + "stream": True, + # No tools — should use direct engine stream + }, + ) assert resp.status_code == 200 # Should get engine tokens, not agent response diff --git a/tests/server/test_models_pydantic.py b/tests/server/test_models_pydantic.py index dc80bbe5..f3757fb0 100644 --- a/tests/server/test_models_pydantic.py +++ b/tests/server/test_models_pydantic.py @@ -83,10 +83,12 @@ class TestModelListResponse: assert resp.data == [] def test_with_models(self): - resp = ModelListResponse(data=[ - ModelObject(id="model-a"), - ModelObject(id="model-b"), - ]) + resp = ModelListResponse( + data=[ + ModelObject(id="model-a"), + ModelObject(id="model-b"), + ] + ) assert len(resp.data) == 2 assert resp.data[0].id == "model-a" assert resp.data[0].object == "model" diff --git a/tests/server/test_routes.py b/tests/server/test_routes.py index 8880d4e2..f82d08d5 100644 --- a/tests/server/test_routes.py +++ b/tests/server/test_routes.py @@ -31,8 +31,12 @@ def _make_engine(content="Hello from server", models=None): # Set up async stream async def mock_stream( - messages, *, model, temperature=0.7, - max_tokens=1024, **kwargs, + messages, + *, + model, + temperature=0.7, + max_tokens=1024, + **kwargs, ): for token in ["Hello", " ", "world"]: yield token @@ -43,6 +47,7 @@ def _make_engine(content="Hello from server", models=None): def _make_agent(content="Hello from agent"): from openjarvis.agents._stubs import AgentResult + agent = MagicMock() agent.agent_id = "mock" agent.run.return_value = AgentResult(content=content, turns=1) @@ -71,47 +76,62 @@ def client_with_agent(): class TestChatCompletions: def test_basic_completion(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + }, + ) assert resp.status_code == 200 data = resp.json() assert data["object"] == "chat.completion" assert data["choices"][0]["message"]["content"] == "Hello from server" def test_completion_has_usage(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + }, + ) data = resp.json() assert data["usage"]["total_tokens"] == 8 def test_completion_has_id(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + }, + ) data = resp.json() assert data["id"].startswith("chatcmpl-") def test_custom_temperature(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - "temperature": 0.1, - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + "temperature": 0.1, + }, + ) assert resp.status_code == 200 def test_with_system_message(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [ - {"role": "system", "content": "Be helpful"}, - {"role": "user", "content": "Hello"}, - ], - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [ + {"role": "system", "content": "Be helpful"}, + {"role": "user", "content": "Hello"}, + ], + }, + ) assert resp.status_code == 200 def test_with_tools(self): @@ -127,40 +147,52 @@ class TestChatCompletions: } app = create_app(engine, "test-model") client = TestClient(app) - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Calc"}], - "tools": [{"type": "function", "function": {"name": "calc"}}], - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Calc"}], + "tools": [{"type": "function", "function": {"name": "calc"}}], + }, + ) assert resp.status_code == 200 data = resp.json() assert data["choices"][0]["message"]["tool_calls"] is not None def test_agent_mode(self, client_with_agent): - resp = client_with_agent.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - }) + resp = client_with_agent.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + }, + ) assert resp.status_code == 200 data = resp.json() assert data["choices"][0]["message"]["content"] == "Hello from agent" def test_agent_with_conversation(self, client_with_agent): - resp = client_with_agent.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [ - {"role": "system", "content": "Be helpful"}, - {"role": "user", "content": "Hello"}, - ], - }) + resp = client_with_agent.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [ + {"role": "system", "content": "Be helpful"}, + {"role": "user", "content": "Hello"}, + ], + }, + ) assert resp.status_code == 200 def test_streaming(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - "stream": True, - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + "stream": True, + }, + ) assert resp.status_code == 200 assert "text/event-stream" in resp.headers.get("content-type", "") # Parse SSE events @@ -171,29 +203,40 @@ class TestChatCompletions: assert data_lines[-1].strip() == "data: [DONE]" def test_streaming_content(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - "stream": True, - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + "stream": True, + }, + ) # Collect content tokens from stream content = "" for line in resp.text.strip().split("\n"): if line.startswith("data:") and "[DONE]" not in line: data = json.loads(line[5:].strip()) choices = data.get("choices", [{}]) - delta_content = choices[0].get( - "delta", {}, - ).get("content") + delta_content = ( + choices[0] + .get( + "delta", + {}, + ) + .get("content") + ) if delta_content: content += delta_content assert content == "Hello world" def test_finish_reason_default(self, client): - resp = client.post("/v1/chat/completions", json={ - "model": "test-model", - "messages": [{"role": "user", "content": "Hello"}], - }) + resp = client.post( + "/v1/chat/completions", + json={ + "model": "test-model", + "messages": [{"role": "user", "content": "Hello"}], + }, + ) data = resp.json() assert data["choices"][0]["finish_reason"] == "stop" diff --git a/tests/server/test_tools_endpoint.py b/tests/server/test_tools_endpoint.py index 414b194d..4f061209 100644 --- a/tests/server/test_tools_endpoint.py +++ b/tests/server/test_tools_endpoint.py @@ -1,4 +1,5 @@ """Tests for GET /v1/tools endpoint.""" + import pytest try: @@ -14,6 +15,7 @@ pytestmark = pytest.mark.skipif( def test_tools_endpoint_returns_list(): from openjarvis.server.agent_manager_routes import build_tools_list + tools = build_tools_list() assert isinstance(tools, list) assert len(tools) > 0 @@ -29,6 +31,7 @@ def test_tools_endpoint_returns_list(): def test_tools_includes_channels(): from openjarvis.server.agent_manager_routes import build_tools_list + tools = build_tools_list() names = {t["name"] for t in tools} channel_names = {"slack", "telegram", "discord", "email"} @@ -37,6 +40,7 @@ def test_tools_includes_channels(): def test_browser_meta_group(): from openjarvis.server.agent_manager_routes import build_tools_list + tools = build_tools_list() names = {t["name"] for t in tools} assert "browser" in names diff --git a/tests/server/test_ws_bridge.py b/tests/server/test_ws_bridge.py index 9e714efe..1727c1e0 100644 --- a/tests/server/test_ws_bridge.py +++ b/tests/server/test_ws_bridge.py @@ -38,10 +38,13 @@ class TestWSBridge: def test_websocket_receives_events(self, app, event_bus): client = TestClient(app) with client.websocket_connect("/v1/agents/events") as ws: - event_bus.publish(EventType.AGENT_TICK_START, { - "agent_id": "test-123", - "agent_name": "test", - }) + event_bus.publish( + EventType.AGENT_TICK_START, + { + "agent_id": "test-123", + "agent_name": "test", + }, + ) time.sleep(0.05) # Let call_soon_threadsafe deliver to queue data = ws.receive_json() assert data["type"] == "agent_tick_start" diff --git a/tests/sessions/test_session.py b/tests/sessions/test_session.py index 8368f3f1..ca6b7d53 100644 --- a/tests/sessions/test_session.py +++ b/tests/sessions/test_session.py @@ -29,7 +29,8 @@ class TestSession: class TestSessionIdentity: def test_create_identity(self): identity = SessionIdentity( - user_id="u1", display_name="Alice", + user_id="u1", + display_name="Alice", channel_ids={"telegram": "t123"}, ) assert identity.user_id == "u1" diff --git a/tests/skills/test_bundled_skills.py b/tests/skills/test_bundled_skills.py index 44940e4a..56b645e5 100644 --- a/tests/skills/test_bundled_skills.py +++ b/tests/skills/test_bundled_skills.py @@ -10,11 +10,7 @@ from openjarvis.skills.loader import load_skill # Resolve the skills/builtin/ directory relative to the project root. BUILTIN_DIR = ( - Path(__file__).resolve().parents[2] - / "src" - / "openjarvis" - / "skills" - / "data" + Path(__file__).resolve().parents[2] / "src" / "openjarvis" / "skills" / "data" ) # Collect all TOML files once so parametrized IDs are readable. @@ -98,6 +94,4 @@ class TestStepsHaveToolNames: def test_steps_have_tool_names(self, toml_path: Path): manifest = load_skill(toml_path) for i, step in enumerate(manifest.steps): - assert step.tool_name, ( - f"{toml_path.name} step {i} has empty tool_name" - ) + assert step.tool_name, f"{toml_path.name} step {i} has empty tool_name" diff --git a/tests/skills/test_skills.py b/tests/skills/test_skills.py index ec2dace2..b151558e 100644 --- a/tests/skills/test_skills.py +++ b/tests/skills/test_skills.py @@ -11,6 +11,7 @@ from openjarvis.tools._stubs import BaseTool, ToolExecutor, ToolSpec class EchoTool(BaseTool): """Simple tool that echoes input.""" + tool_id = "echo" @property @@ -27,6 +28,7 @@ class EchoTool(BaseTool): class UpperTool(BaseTool): """Simple tool that uppercases input.""" + tool_id = "upper" @property @@ -68,11 +70,13 @@ class TestSkillExecutor: executor = self._make_executor() manifest = SkillManifest( name="single", - steps=[SkillStep( - tool_name="echo", - arguments_template='{"text": "hello"}', - output_key="result", - )], + steps=[ + SkillStep( + tool_name="echo", + arguments_template='{"text": "hello"}', + output_key="result", + ) + ], ) result = executor.run(manifest) assert result.success @@ -116,11 +120,13 @@ class TestSkillExecutor: executor = self._make_executor() manifest = SkillManifest( name="with_ctx", - steps=[SkillStep( - tool_name="echo", - arguments_template='{"text": "{greeting}"}', - output_key="result", - )], + steps=[ + SkillStep( + tool_name="echo", + arguments_template='{"text": "{greeting}"}', + output_key="result", + ) + ], ) result = executor.run(manifest, initial_context={"greeting": "hi"}) assert result.success @@ -152,11 +158,13 @@ class TestSkillTool: manifest = SkillManifest( name="tool_skill", description="A skill exposed as a tool", - steps=[SkillStep( - tool_name="echo", - arguments_template='{"text": "{input}"}', - output_key="result", - )], + steps=[ + SkillStep( + tool_name="echo", + arguments_template='{"text": "{input}"}', + output_key="result", + ) + ], ) skill_tool = SkillTool(manifest, executor) assert skill_tool.spec.name == "skill_tool_skill" diff --git a/tests/speech/test_discovery.py b/tests/speech/test_discovery.py index 523b73cc..50f44220 100644 --- a/tests/speech/test_discovery.py +++ b/tests/speech/test_discovery.py @@ -13,10 +13,14 @@ def test_get_speech_backend_explicit(): config.speech.backend = "faster-whisper" with patch("openjarvis.speech._discovery._create_backend") as mock_create: - mock_backend = type("MockBackend", (), { - "backend_id": "faster-whisper", - "health": lambda self: True, - })() + mock_backend = type( + "MockBackend", + (), + { + "backend_id": "faster-whisper", + "health": lambda self: True, + }, + )() mock_create.return_value = mock_backend result = get_speech_backend(config) diff --git a/tests/telemetry/test_aggregator.py b/tests/telemetry/test_aggregator.py index d71d7e4c..ed3bb74a 100644 --- a/tests/telemetry/test_aggregator.py +++ b/tests/telemetry/test_aggregator.py @@ -71,11 +71,14 @@ class TestTelemetryAggregator: agg.close() def test_multiple_models_grouped(self, tmp_path: Path) -> None: - agg = _setup(tmp_path, [ - _make_record(model_id="m1"), - _make_record(model_id="m1"), - _make_record(model_id="m2"), - ]) + agg = _setup( + tmp_path, + [ + _make_record(model_id="m1"), + _make_record(model_id="m1"), + _make_record(model_id="m2"), + ], + ) stats = agg.per_model_stats() assert len(stats) == 2 # Ordered by call_count DESC @@ -86,11 +89,14 @@ class TestTelemetryAggregator: agg.close() def test_per_engine_stats(self, tmp_path: Path) -> None: - agg = _setup(tmp_path, [ - _make_record(engine="ollama"), - _make_record(engine="vllm"), - _make_record(engine="vllm"), - ]) + agg = _setup( + tmp_path, + [ + _make_record(engine="ollama"), + _make_record(engine="vllm"), + _make_record(engine="vllm"), + ], + ) stats = agg.per_engine_stats() assert len(stats) == 2 assert stats[0].engine == "vllm" @@ -105,22 +111,28 @@ class TestTelemetryAggregator: agg.close() def test_top_models_ordering(self, tmp_path: Path) -> None: - agg = _setup(tmp_path, [ - _make_record(model_id="rare"), - _make_record(model_id="popular"), - _make_record(model_id="popular"), - _make_record(model_id="popular"), - ]) + agg = _setup( + tmp_path, + [ + _make_record(model_id="rare"), + _make_record(model_id="popular"), + _make_record(model_id="popular"), + _make_record(model_id="popular"), + ], + ) top = agg.top_models(n=2) assert top[0].model_id == "popular" assert top[0].call_count == 3 agg.close() def test_summary_totals(self, tmp_path: Path) -> None: - agg = _setup(tmp_path, [ - _make_record(prompt_tokens=10, completion_tokens=5, cost=0.001), - _make_record(prompt_tokens=20, completion_tokens=10, cost=0.002), - ]) + agg = _setup( + tmp_path, + [ + _make_record(prompt_tokens=10, completion_tokens=5, cost=0.001), + _make_record(prompt_tokens=20, completion_tokens=10, cost=0.002), + ], + ) s = agg.summary() assert s.total_calls == 2 assert s.total_tokens == 45 # (10+5) + (20+10) @@ -136,11 +148,14 @@ class TestTelemetryAggregator: def test_time_range_since(self, tmp_path: Path) -> None: now = time.time() - agg = _setup(tmp_path, [ - _make_record(ts=now - 100), - _make_record(ts=now - 10), - _make_record(ts=now), - ]) + agg = _setup( + tmp_path, + [ + _make_record(ts=now - 100), + _make_record(ts=now - 10), + _make_record(ts=now), + ], + ) stats = agg.per_model_stats(since=now - 50) total = sum(s.call_count for s in stats) assert total == 2 @@ -148,10 +163,13 @@ class TestTelemetryAggregator: def test_time_range_until(self, tmp_path: Path) -> None: now = time.time() - agg = _setup(tmp_path, [ - _make_record(ts=now - 100), - _make_record(ts=now), - ]) + agg = _setup( + tmp_path, + [ + _make_record(ts=now - 100), + _make_record(ts=now), + ], + ) stats = agg.per_model_stats(until=now - 50) total = sum(s.call_count for s in stats) assert total == 1 @@ -159,11 +177,14 @@ class TestTelemetryAggregator: def test_time_range_since_and_until(self, tmp_path: Path) -> None: now = time.time() - agg = _setup(tmp_path, [ - _make_record(ts=now - 200), - _make_record(ts=now - 100), - _make_record(ts=now), - ]) + agg = _setup( + tmp_path, + [ + _make_record(ts=now - 200), + _make_record(ts=now - 100), + _make_record(ts=now), + ], + ) stats = agg.per_model_stats(since=now - 150, until=now - 50) total = sum(s.call_count for s in stats) assert total == 1 @@ -178,10 +199,13 @@ class TestTelemetryAggregator: def test_export_records_filtered(self, tmp_path: Path) -> None: now = time.time() - agg = _setup(tmp_path, [ - _make_record(ts=now - 100), - _make_record(ts=now), - ]) + agg = _setup( + tmp_path, + [ + _make_record(ts=now - 100), + _make_record(ts=now), + ], + ) records = agg.export_records(since=now - 50) assert len(records) == 1 agg.close() diff --git a/tests/telemetry/test_derived_metrics.py b/tests/telemetry/test_derived_metrics.py index c2f5ddb6..50e66a8a 100644 --- a/tests/telemetry/test_derived_metrics.py +++ b/tests/telemetry/test_derived_metrics.py @@ -176,13 +176,15 @@ class TestDerivedMetricsInStore: def test_summary_weighted_averages(self, tmp_path): store = TelemetryStore(tmp_path / "test.db") for i in range(3): - store.record(TelemetryRecord( - timestamp=time.time() + i, - model_id="m1", - engine="e1", - energy_per_output_token_joules=0.1 * (i + 1), - throughput_per_watt=1.0 * (i + 1), - )) + store.record( + TelemetryRecord( + timestamp=time.time() + i, + model_id="m1", + engine="e1", + energy_per_output_token_joules=0.1 * (i + 1), + throughput_per_watt=1.0 * (i + 1), + ) + ) agg = TelemetryAggregator(tmp_path / "test.db") summary = agg.summary() assert summary.avg_energy_per_output_token_joules > 0 diff --git a/tests/telemetry/test_energy_apple.py b/tests/telemetry/test_energy_apple.py index c9c48b4f..4b1c0268 100644 --- a/tests/telemetry/test_energy_apple.py +++ b/tests/telemetry/test_energy_apple.py @@ -50,8 +50,9 @@ class TestAvailable: orig = mod._ZEUS_APPLE_AVAILABLE mod._ZEUS_APPLE_AVAILABLE = False try: - with patch("platform.system", return_value="Darwin"), patch( - "platform.machine", return_value="arm64" + with ( + patch("platform.system", return_value="Darwin"), + patch("platform.machine", return_value="arm64"), ): assert mod.AppleEnergyMonitor.available() is True monitor = mod.AppleEnergyMonitor.__new__(mod.AppleEnergyMonitor) diff --git a/tests/telemetry/test_energy_monitor.py b/tests/telemetry/test_energy_monitor.py index 848b095b..07d753b6 100644 --- a/tests/telemetry/test_energy_monitor.py +++ b/tests/telemetry/test_energy_monitor.py @@ -77,40 +77,51 @@ class TestEnergyMonitorABC: class TestCreateEnergyMonitor: def test_returns_none_when_nothing_available(self): - with patch( - "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", - return_value=False, + with ( + patch( + "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", + return_value=False, + ), ): result = create_energy_monitor() assert result is None def test_prefer_vendor_parameter(self): """When prefer_vendor is set, that vendor is tried first.""" - with patch( - "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", - return_value=True, - ), patch( - "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.__init__", - return_value=None, - ) as mock_init: + with ( + patch( + "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", + return_value=True, + ), + patch( + "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.__init__", + return_value=None, + ) as mock_init, + ): create_energy_monitor(prefer_vendor="cpu_rapl") # RaplEnergyMonitor was available and preferred mock_init.assert_called_once_with(poll_interval_ms=50) @@ -127,21 +138,27 @@ class TestCreateEnergyMonitor: call_order.append("amd") return True - with patch( - "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", - side_effect=nvidia_available, - ), patch( - "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", - side_effect=amd_available, - ), patch( - "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.__init__", - return_value=None, + with ( + patch( + "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", + side_effect=nvidia_available, + ), + patch( + "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", + side_effect=amd_available, + ), + patch( + "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.__init__", + return_value=None, + ), ): create_energy_monitor() # NVIDIA was tried first and returned True @@ -159,21 +176,27 @@ class TestCreateEnergyMonitor: call_order.append("amd") return True - with patch( - "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", - side_effect=nvidia_available, - ), patch( - "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", - side_effect=amd_available, - ), patch( - "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", - return_value=False, - ), patch( - "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.__init__", - return_value=None, + with ( + patch( + "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", + side_effect=nvidia_available, + ), + patch( + "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", + side_effect=amd_available, + ), + patch( + "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", + return_value=False, + ), + patch( + "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.__init__", + return_value=None, + ), ): create_energy_monitor() assert call_order == ["nvidia", "amd"] @@ -198,18 +221,23 @@ class TestCreateEnergyMonitor: call_order.append("apple") return False - with patch( - "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", - side_effect=nvidia_available, - ), patch( - "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", - side_effect=amd_available, - ), patch( - "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", - side_effect=apple_available, - ), patch( - "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", - side_effect=rapl_available, + with ( + patch( + "openjarvis.telemetry.energy_nvidia.NvidiaEnergyMonitor.available", + side_effect=nvidia_available, + ), + patch( + "openjarvis.telemetry.energy_amd.AmdEnergyMonitor.available", + side_effect=amd_available, + ), + patch( + "openjarvis.telemetry.energy_apple.AppleEnergyMonitor.available", + side_effect=apple_available, + ), + patch( + "openjarvis.telemetry.energy_rapl.RaplEnergyMonitor.available", + side_effect=rapl_available, + ), ): result = create_energy_monitor(prefer_vendor="cpu_rapl") assert result is None diff --git a/tests/telemetry/test_energy_nvidia.py b/tests/telemetry/test_energy_nvidia.py index 264748a1..f9f4264d 100644 --- a/tests/telemetry/test_energy_nvidia.py +++ b/tests/telemetry/test_energy_nvidia.py @@ -34,14 +34,10 @@ def _make_fake_pynvml(device_count: int = 1, power_mw: int = 300_000): mod.nvmlInit = MagicMock() mod.nvmlShutdown = MagicMock() mod.nvmlDeviceGetCount = MagicMock(return_value=device_count) - mod.nvmlDeviceGetHandleByIndex = MagicMock( - side_effect=lambda i: f"handle-{i}" - ) + mod.nvmlDeviceGetHandleByIndex = MagicMock(side_effect=lambda i: f"handle-{i}") mod.nvmlDeviceGetName = MagicMock(return_value="NVIDIA A100-SXM") mod.nvmlDeviceGetPowerUsage = MagicMock(return_value=power_mw) - mod.nvmlDeviceGetUtilizationRates = MagicMock( - return_value=_FakeUtilization() - ) + mod.nvmlDeviceGetUtilizationRates = MagicMock(return_value=_FakeUtilization()) mod.nvmlDeviceGetMemoryInfo = MagicMock(return_value=_FakeMemInfo()) mod.nvmlDeviceGetTemperature = MagicMock(return_value=65) mod.nvmlDeviceGetTotalEnergyConsumption = MagicMock(return_value=5000.0) @@ -115,8 +111,8 @@ class TestHwCounterProbe: def test_probe_fails_on_pre_volta(self): """nvmlDeviceGetTotalEnergyConsumption raises => polling fallback.""" fake_pynvml = _make_fake_pynvml(device_count=1) - fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = ( - RuntimeError("Not supported") + fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = RuntimeError( + "Not supported" ) with patch.dict(sys.modules, {"pynvml": fake_pynvml}): @@ -155,8 +151,8 @@ class TestEnergyMethod: def test_returns_polling_when_no_hw_counter(self): fake_pynvml = _make_fake_pynvml(device_count=1) - fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = ( - RuntimeError("Not supported") + fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = RuntimeError( + "Not supported" ) with patch.dict(sys.modules, {"pynvml": fake_pynvml}): @@ -238,8 +234,8 @@ class TestSamplePolling: """Fallback mode uses trapezoidal integration of power readings.""" fake_pynvml = _make_fake_pynvml(device_count=1, power_mw=300_000) # Make hw counter probe fail => polling mode - fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = ( - RuntimeError("Not supported") + fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = RuntimeError( + "Not supported" ) with patch.dict(sys.modules, {"pynvml": fake_pynvml}): @@ -276,16 +272,18 @@ class TestSampleMultiGpu: # 2 devices: __init__ probe reads device 0 once. # Then sample() reads start (dev0, dev1), end (dev0, dev1). - readings = iter([ - 1000.0, # probe: device 0 - 2000.0, # sample start: device 0 - 3000.0, # sample start: device 1 - 5000.0, # sample end: device 0 - 7000.0, # sample end: device 1 - ]) + readings = iter( + [ + 1000.0, # probe: device 0 + 2000.0, # sample start: device 0 + 3000.0, # sample start: device 1 + 5000.0, # sample end: device 0 + 7000.0, # sample end: device 1 + ] + ) - fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = ( - lambda h: next(readings) + fake_pynvml.nvmlDeviceGetTotalEnergyConsumption.side_effect = lambda h: next( + readings ) with patch.dict(sys.modules, {"pynvml": fake_pynvml}): diff --git a/tests/telemetry/test_energy_rapl.py b/tests/telemetry/test_energy_rapl.py index 8695c779..2bb80b08 100644 --- a/tests/telemetry/test_energy_rapl.py +++ b/tests/telemetry/test_energy_rapl.py @@ -33,9 +33,7 @@ def _create_rapl_domain( domain_dir.mkdir(parents=True, exist_ok=True) (domain_dir / "name").write_text(name) (domain_dir / "energy_uj").write_text(str(energy_uj)) - (domain_dir / "max_energy_range_uj").write_text( - str(max_energy_uj) - ) + (domain_dir / "max_energy_range_uj").write_text(str(max_energy_uj)) return domain_dir @@ -50,12 +48,18 @@ def _build_fake_sysfs(tmp_path: Path) -> Path: rapl_base.mkdir() _create_rapl_domain( - rapl_base, "package-0", "intel-rapl:0", - energy_uj=500000, max_energy_uj=262143328850, + rapl_base, + "package-0", + "intel-rapl:0", + energy_uj=500000, + max_energy_uj=262143328850, ) _create_rapl_domain( - rapl_base, "dram", "intel-rapl:0/intel-rapl:0:0", - energy_uj=100000, max_energy_uj=65535999603, + rapl_base, + "dram", + "intel-rapl:0/intel-rapl:0:0", + energy_uj=100000, + max_energy_uj=65535999603, ) return rapl_base @@ -127,16 +131,15 @@ class TestSampleNormalDelta: with patch(_PLAT, return_value="Linux"): monitor = RaplEnergyMonitor( - poll_interval_ms=50, rapl_base=rapl_base, + poll_interval_ms=50, + rapl_base=rapl_base, ) assert monitor._initialized is True assert len(monitor._domains) == 2 # Set start values pkg_energy = rapl_base / "intel-rapl:0" / "energy_uj" - dram_energy = ( - rapl_base / "intel-rapl:0" / "intel-rapl:0:0" / "energy_uj" - ) + dram_energy = rapl_base / "intel-rapl:0" / "intel-rapl:0:0" / "energy_uj" pkg_energy.write_text("500000") dram_energy.write_text("100000") @@ -169,13 +172,17 @@ class TestSampleWrapAround: max_energy = 1000000 _create_rapl_domain( - rapl_base, "package-0", "intel-rapl:0", - energy_uj=900000, max_energy_uj=max_energy, + rapl_base, + "package-0", + "intel-rapl:0", + energy_uj=900000, + max_energy_uj=max_energy, ) with patch(_PLAT, return_value="Linux"): monitor = RaplEnergyMonitor( - poll_interval_ms=50, rapl_base=rapl_base, + poll_interval_ms=50, + rapl_base=rapl_base, ) assert monitor._initialized is True @@ -205,14 +212,13 @@ class TestSampleDomainCategorization: with patch(_PLAT, return_value="Linux"): monitor = RaplEnergyMonitor( - poll_interval_ms=50, rapl_base=rapl_base, + poll_interval_ms=50, + rapl_base=rapl_base, ) # Set start values pkg_energy = rapl_base / "intel-rapl:0" / "energy_uj" - dram_energy = ( - rapl_base / "intel-rapl:0" / "intel-rapl:0:0" / "energy_uj" - ) + dram_energy = rapl_base / "intel-rapl:0" / "intel-rapl:0:0" / "energy_uj" pkg_energy.write_text("1000") dram_energy.write_text("2000") @@ -237,7 +243,8 @@ class TestClose: with patch(_PLAT, return_value="Linux"): monitor = RaplEnergyMonitor( - poll_interval_ms=50, rapl_base=rapl_base, + poll_interval_ms=50, + rapl_base=rapl_base, ) assert len(monitor._domains) == 2 assert monitor._initialized is True diff --git a/tests/telemetry/test_energy_wiring.py b/tests/telemetry/test_energy_wiring.py index 1ceb9b7f..399b717d 100644 --- a/tests/telemetry/test_energy_wiring.py +++ b/tests/telemetry/test_energy_wiring.py @@ -138,25 +138,32 @@ class TestCliAskWiring: engine = _mock_engine() monkeypatch.setattr( - _ask_mod, "get_engine", + _ask_mod, + "get_engine", lambda *a, **kw: ("mock", engine), ) monkeypatch.setattr( - _ask_mod, "discover_engines", + _ask_mod, + "discover_engines", lambda c: [("mock", engine)], ) monkeypatch.setattr( - _ask_mod, "discover_models", + _ask_mod, + "discover_models", lambda e: {"mock": ["test-model"]}, ) return cfg, engine def test_engine_wrapped_with_instrumented( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """InstrumentedEngine wraps engine, not instrumented_generate.""" cfg, engine = self._patch_ask( - monkeypatch, tmp_path, gpu_metrics=False, + monkeypatch, + tmp_path, + gpu_metrics=False, ) result = CliRunner().invoke(cli, ["ask", "Hello"]) assert result.exit_code == 0 @@ -165,11 +172,15 @@ class TestCliAskWiring: engine.generate.assert_called_once() def test_energy_monitor_created_when_gpu_metrics_on( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """Energy monitor is created when gpu_metrics=True.""" cfg, engine = self._patch_ask( - monkeypatch, tmp_path, gpu_metrics=True, + monkeypatch, + tmp_path, + gpu_metrics=True, ) mock_monitor = _mock_energy_monitor() with patch( @@ -182,22 +193,30 @@ class TestCliAskWiring: mock_monitor.close.assert_called_once() def test_no_energy_monitor_when_gpu_metrics_off( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """No energy monitor when gpu_metrics=False.""" cfg, engine = self._patch_ask( - monkeypatch, tmp_path, gpu_metrics=False, + monkeypatch, + tmp_path, + gpu_metrics=False, ) # Should not attempt to import create_energy_monitor result = CliRunner().invoke(cli, ["ask", "Hello"]) assert result.exit_code == 0 def test_telemetry_events_published( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """InstrumentedEngine publishes TELEMETRY_RECORD events.""" cfg, engine = self._patch_ask( - monkeypatch, tmp_path, gpu_metrics=False, + monkeypatch, + tmp_path, + gpu_metrics=False, ) result = CliRunner().invoke(cli, ["ask", "Hello"]) assert result.exit_code == 0 @@ -209,11 +228,15 @@ class TestCliAskWiring: agg.close() def test_energy_data_in_telemetry_record( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """Energy data flows into TelemetryRecord in SQLite.""" cfg, engine = self._patch_ask( - monkeypatch, tmp_path, gpu_metrics=True, + monkeypatch, + tmp_path, + gpu_metrics=True, ) mock_monitor = _mock_energy_monitor() with patch( @@ -236,11 +259,15 @@ class TestCliAskWiring: agg.close() def test_agent_mode_uses_instrumented_engine( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """Agent mode passes InstrumentedEngine to agent.""" cfg, engine = self._patch_ask( - monkeypatch, tmp_path, gpu_metrics=False, + monkeypatch, + tmp_path, + gpu_metrics=False, ) # Register a trivial agent that calls engine.generate @@ -262,11 +289,13 @@ class TestCliAskWiring: return AgentResult(content="Agent OK", turns=1) AgentRegistry.register_value( - "test-wiring-agent", _TestAgent, + "test-wiring-agent", + _TestAgent, ) result = CliRunner().invoke( - cli, ["ask", "--agent", "test-wiring-agent", "Hi"], + cli, + ["ask", "--agent", "test-wiring-agent", "Hi"], ) assert result.exit_code == 0 assert "Agent OK" in result.output @@ -309,12 +338,15 @@ class TestSdkWiring: cfg = _energy_config(tmp_path, gpu_metrics=True) mock_monitor = _mock_energy_monitor() - with patch( - "openjarvis.sdk.get_engine", - return_value=("mock", engine), - ), patch( - "openjarvis.telemetry.energy_monitor.create_energy_monitor", - return_value=mock_monitor, + with ( + patch( + "openjarvis.sdk.get_engine", + return_value=("mock", engine), + ), + patch( + "openjarvis.telemetry.energy_monitor.create_energy_monitor", + return_value=mock_monitor, + ), ): j = Jarvis(config=cfg, model="test-model") j._ensure_engine() @@ -347,12 +379,15 @@ class TestSdkWiring: cfg = _energy_config(tmp_path, gpu_metrics=True) mock_monitor = _mock_energy_monitor() - with patch( - "openjarvis.sdk.get_engine", - return_value=("mock", engine), - ), patch( - "openjarvis.telemetry.energy_monitor.create_energy_monitor", - return_value=mock_monitor, + with ( + patch( + "openjarvis.sdk.get_engine", + return_value=("mock", engine), + ), + patch( + "openjarvis.telemetry.energy_monitor.create_energy_monitor", + return_value=mock_monitor, + ), ): j = Jarvis(config=cfg, model="test-model") result = j.ask_full("Hello") @@ -376,12 +411,15 @@ class TestSdkWiring: cfg.telemetry.gpu_metrics = True mock_monitor = _mock_energy_monitor() - with patch( - "openjarvis.sdk.get_engine", - return_value=("mock", engine), - ), patch( - "openjarvis.telemetry.energy_monitor.create_energy_monitor", - return_value=mock_monitor, + with ( + patch( + "openjarvis.sdk.get_engine", + return_value=("mock", engine), + ), + patch( + "openjarvis.telemetry.energy_monitor.create_energy_monitor", + return_value=mock_monitor, + ), ): j = Jarvis(config=cfg, model="test-model") j._ensure_engine() @@ -398,12 +436,15 @@ class TestSdkWiring: cfg.telemetry.gpu_metrics = True mock_monitor = _mock_energy_monitor() - with patch( - "openjarvis.sdk.get_engine", - return_value=("mock", engine), - ), patch( - "openjarvis.telemetry.energy_monitor.create_energy_monitor", - return_value=mock_monitor, + with ( + patch( + "openjarvis.sdk.get_engine", + return_value=("mock", engine), + ), + patch( + "openjarvis.telemetry.energy_monitor.create_energy_monitor", + return_value=mock_monitor, + ), ): j = Jarvis(config=cfg, model="test-model") j._ensure_engine() @@ -426,7 +467,9 @@ class TestInstrumentedEngineEnergy: monitor = _mock_energy_monitor() ie = InstrumentedEngine( - engine, bus, energy_monitor=monitor, + engine, + bus, + energy_monitor=monitor, ) messages = [Message(role=Role.USER, content="Hi")] result = ie.generate(messages, model="test") @@ -434,8 +477,7 @@ class TestInstrumentedEngineEnergy: assert result["content"] == "Test response" # Verify energy data is in the telemetry record tel_events = [ - e for e in bus.history - if e.event_type == EventType.TELEMETRY_RECORD + e for e in bus.history if e.event_type == EventType.TELEMETRY_RECORD ] assert len(tel_events) == 1 rec = tel_events[0].data["record"] @@ -452,7 +494,9 @@ class TestInstrumentedEngineEnergy: monitor = _mock_energy_monitor() ie = InstrumentedEngine( - engine, bus, energy_monitor=monitor, + engine, + bus, + energy_monitor=monitor, ) messages = [Message(role=Role.USER, content="Hi")] result = ie.generate(messages, model="test") @@ -477,10 +521,7 @@ class TestInstrumentedEngineEnergy: result = ie.generate(messages, model="test") assert result["content"] == "Test response" - tel = [ - e for e in bus.history - if e.event_type == EventType.TELEMETRY_RECORD - ] + tel = [e for e in bus.history if e.event_type == EventType.TELEMETRY_RECORD] rec = tel[0].data["record"] assert rec.energy_joules == 0.0 assert rec.energy_method == "" @@ -499,7 +540,9 @@ class TestInstrumentedEngineEnergy: monitor.sample = _broken_sample ie = InstrumentedEngine( - engine, bus, energy_monitor=monitor, + engine, + bus, + energy_monitor=monitor, ) messages = [Message(role=Role.USER, content="Hi")] # Should not crash — energy is best-effort. @@ -537,18 +580,23 @@ class TestBenchWiring: cfg.telemetry.gpu_metrics = True mock_monitor = _mock_energy_monitor() - with patch( - "openjarvis.cli.bench_cmd.get_engine", - return_value=("mock", engine), - ), patch( - "openjarvis.cli.bench_cmd.load_config", - return_value=cfg, - ), patch( - "openjarvis.telemetry.energy_monitor.create_energy_monitor", - return_value=mock_monitor, - ) as mock_create: + with ( + patch( + "openjarvis.cli.bench_cmd.get_engine", + return_value=("mock", engine), + ), + patch( + "openjarvis.cli.bench_cmd.load_config", + return_value=cfg, + ), + patch( + "openjarvis.telemetry.energy_monitor.create_energy_monitor", + return_value=mock_monitor, + ) as mock_create, + ): result = CliRunner().invoke( - cli, ["bench", "run", "-n", "2"], + cli, + ["bench", "run", "-n", "2"], ) assert result.exit_code == 0 @@ -572,15 +620,19 @@ class TestBenchWiring: cfg = JarvisConfig() cfg.telemetry.gpu_metrics = False - with patch( - "openjarvis.cli.bench_cmd.get_engine", - return_value=("mock", engine), - ), patch( - "openjarvis.cli.bench_cmd.load_config", - return_value=cfg, + with ( + patch( + "openjarvis.cli.bench_cmd.get_engine", + return_value=("mock", engine), + ), + patch( + "openjarvis.cli.bench_cmd.load_config", + return_value=cfg, + ), ): result = CliRunner().invoke( - cli, ["bench", "run", "-n", "2"], + cli, + ["bench", "run", "-n", "2"], ) assert result.exit_code == 0 @@ -602,15 +654,19 @@ class TestBenchWiring: cfg = JarvisConfig() cfg.telemetry.gpu_metrics = False - with patch( - "openjarvis.cli.bench_cmd.get_engine", - return_value=("mock", engine), - ), patch( - "openjarvis.cli.bench_cmd.load_config", - return_value=cfg, + with ( + patch( + "openjarvis.cli.bench_cmd.get_engine", + return_value=("mock", engine), + ), + patch( + "openjarvis.cli.bench_cmd.load_config", + return_value=cfg, + ), ): result = CliRunner().invoke( - cli, ["bench", "run", "-n", "2", "-w", "3"], + cli, + ["bench", "run", "-n", "2", "-w", "3"], ) assert result.exit_code == 0 @@ -625,14 +681,16 @@ def _populate_energy_db(db_path: Path, n: int = 3) -> None: """Create a telemetry DB with energy-enriched records.""" store = TelemetryStore(db_path) for i in range(n): - store.record(_make_energy_record( - model_id=f"model-{i % 2}", - energy_joules=10.0 * (i + 1), - throughput=100.0 + i * 10, - gpu_util=70.0 + i * 5, - power=200.0 + i * 25, - ts=time.time() - (n - i), - )) + store.record( + _make_energy_record( + model_id=f"model-{i % 2}", + energy_joules=10.0 * (i + 1), + throughput=100.0 + i * 10, + gpu_util=70.0 + i * 5, + power=200.0 + i * 25, + ts=time.time() - (n - i), + ) + ) store.close() @@ -656,7 +714,8 @@ class TestTelemetryStatsEnergy: _populate_energy_db(db_path) with p: result = CliRunner().invoke( - cli, ["telemetry", "stats"], + cli, + ["telemetry", "stats"], ) assert result.exit_code == 0 assert "Total Energy (J)" in result.output @@ -674,21 +733,24 @@ class TestTelemetryStatsEnergy: # Populate with non-energy records store = TelemetryStore(db_path) for i in range(3): - store.record(TelemetryRecord( - timestamp=time.time(), - model_id="model-0", - engine="ollama", - prompt_tokens=10, - completion_tokens=5, - total_tokens=15, - latency_seconds=0.5, - cost_usd=0.001, - )) + store.record( + TelemetryRecord( + timestamp=time.time(), + model_id="model-0", + engine="ollama", + prompt_tokens=10, + completion_tokens=5, + total_tokens=15, + latency_seconds=0.5, + cost_usd=0.001, + ) + ) store.close() with p: result = CliRunner().invoke( - cli, ["telemetry", "stats"], + cli, + ["telemetry", "stats"], ) assert result.exit_code == 0 assert "Total Calls" in result.output @@ -702,7 +764,8 @@ class TestTelemetryStatsEnergy: _populate_energy_db(db_path, n=1) with p: result = CliRunner().invoke( - cli, ["telemetry", "export", "-f", "json"], + cli, + ["telemetry", "export", "-f", "json"], ) assert result.exit_code == 0 data = json.loads(result.output) @@ -726,7 +789,8 @@ class TestTelemetryStatsEnergy: _populate_energy_db(db_path, n=1) with p: result = CliRunner().invoke( - cli, ["telemetry", "export", "-f", "csv"], + cli, + ["telemetry", "export", "-f", "csv"], ) assert result.exit_code == 0 header = result.output.strip().splitlines()[0] @@ -755,24 +819,30 @@ class TestAggregatorEnergy: """summary() sums energy and computes weighted averages.""" db_path = tmp_path / "telemetry.db" store = TelemetryStore(db_path) - store.record(_make_energy_record( - model_id="m1", - energy_joules=10.0, - throughput=100.0, - gpu_util=80.0, - )) - store.record(_make_energy_record( - model_id="m1", - energy_joules=20.0, - throughput=120.0, - gpu_util=90.0, - )) - store.record(_make_energy_record( - model_id="m2", - energy_joules=30.0, - throughput=80.0, - gpu_util=60.0, - )) + store.record( + _make_energy_record( + model_id="m1", + energy_joules=10.0, + throughput=100.0, + gpu_util=80.0, + ) + ) + store.record( + _make_energy_record( + model_id="m1", + energy_joules=20.0, + throughput=120.0, + gpu_util=90.0, + ) + ) + store.record( + _make_energy_record( + model_id="m2", + energy_joules=30.0, + throughput=80.0, + gpu_util=60.0, + ) + ) store.close() agg = TelemetryAggregator(db_path) @@ -784,7 +854,8 @@ class TestAggregatorEnergy: assert s.avg_throughput_tok_per_sec == pytest.approx(100.0) # Weighted avg GPU util: (85*2 + 60*1) / 3 = 76.67 assert s.avg_gpu_utilization_pct == pytest.approx( - 76.666, rel=0.01, + 76.666, + rel=0.01, ) agg.close() @@ -792,9 +863,12 @@ class TestAggregatorEnergy: """per_model_stats includes energy fields.""" db_path = tmp_path / "telemetry.db" store = TelemetryStore(db_path) - store.record(_make_energy_record( - model_id="m1", energy_joules=50.0, - )) + store.record( + _make_energy_record( + model_id="m1", + energy_joules=50.0, + ) + ) store.close() agg = TelemetryAggregator(db_path) @@ -811,9 +885,12 @@ class TestAggregatorEnergy: """per_engine_stats includes energy fields.""" db_path = tmp_path / "telemetry.db" store = TelemetryStore(db_path) - store.record(_make_energy_record( - engine="vllm", energy_joules=25.0, - )) + store.record( + _make_energy_record( + engine="vllm", + energy_joules=25.0, + ) + ) store.close() agg = TelemetryAggregator(db_path) @@ -845,7 +922,9 @@ class TestEndToEndPipeline: """Full pipeline: ask → InstrumentedEngine → energy → SQLite → stats.""" def test_ask_to_stats_with_energy( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """Full flow: ask records energy, stats displays it.""" cfg = _energy_config(tmp_path, gpu_metrics=True) @@ -853,15 +932,18 @@ class TestEndToEndPipeline: monkeypatch.setattr(_ask_mod, "load_config", lambda: cfg) monkeypatch.setattr( - _ask_mod, "get_engine", + _ask_mod, + "get_engine", lambda *a, **kw: ("mock", engine), ) monkeypatch.setattr( - _ask_mod, "discover_engines", + _ask_mod, + "discover_engines", lambda c: [("mock", engine)], ) monkeypatch.setattr( - _ask_mod, "discover_models", + _ask_mod, + "discover_models", lambda e: {"mock": ["test-model"]}, ) @@ -880,7 +962,8 @@ class TestEndToEndPipeline: return_value=telem_cfg, ): result = CliRunner().invoke( - cli, ["telemetry", "stats"], + cli, + ["telemetry", "stats"], ) assert result.exit_code == 0 @@ -888,7 +971,9 @@ class TestEndToEndPipeline: assert "42.50" in result.output # energy_joules value def test_ask_to_export_with_energy( - self, monkeypatch, tmp_path, + self, + monkeypatch, + tmp_path, ): """Full flow: ask records energy, export includes it.""" cfg = _energy_config(tmp_path, gpu_metrics=True) @@ -896,15 +981,18 @@ class TestEndToEndPipeline: monkeypatch.setattr(_ask_mod, "load_config", lambda: cfg) monkeypatch.setattr( - _ask_mod, "get_engine", + _ask_mod, + "get_engine", lambda *a, **kw: ("mock", engine), ) monkeypatch.setattr( - _ask_mod, "discover_engines", + _ask_mod, + "discover_engines", lambda c: [("mock", engine)], ) monkeypatch.setattr( - _ask_mod, "discover_models", + _ask_mod, + "discover_models", lambda e: {"mock": ["test-model"]}, ) @@ -923,7 +1011,8 @@ class TestEndToEndPipeline: return_value=telem_cfg, ): result = CliRunner().invoke( - cli, ["telemetry", "export", "-f", "json"], + cli, + ["telemetry", "export", "-f", "json"], ) data = json.loads(result.output) diff --git a/tests/telemetry/test_gpu_monitor.py b/tests/telemetry/test_gpu_monitor.py index 2ff34b6b..e4b0f1dc 100644 --- a/tests/telemetry/test_gpu_monitor.py +++ b/tests/telemetry/test_gpu_monitor.py @@ -28,32 +28,22 @@ class _FakeMemInfo: free: int = 12 * 1024**3 -def _make_fake_pynvml( - device_count: int = 1, power_mw: int = 300_000 -): +def _make_fake_pynvml(device_count: int = 1, power_mw: int = 300_000): """Return a fake pynvml module object.""" mod = types.ModuleType("pynvml") mod.nvmlInit = MagicMock() mod.nvmlShutdown = MagicMock() mod.nvmlDeviceGetCount = MagicMock(return_value=device_count) - mod.nvmlDeviceGetHandleByIndex = MagicMock( - side_effect=lambda i: f"handle-{i}" - ) + mod.nvmlDeviceGetHandleByIndex = MagicMock(side_effect=lambda i: f"handle-{i}") mod.nvmlDeviceGetPowerUsage = MagicMock(return_value=power_mw) - mod.nvmlDeviceGetUtilizationRates = MagicMock( - return_value=_FakeUtilization() - ) - mod.nvmlDeviceGetMemoryInfo = MagicMock( - return_value=_FakeMemInfo() - ) + mod.nvmlDeviceGetUtilizationRates = MagicMock(return_value=_FakeUtilization()) + mod.nvmlDeviceGetMemoryInfo = MagicMock(return_value=_FakeMemInfo()) mod.nvmlDeviceGetTemperature = MagicMock(return_value=65) mod.NVML_TEMPERATURE_GPU = 0 return mod -def _snap( - power=300, util=80, mem=12, temp=65, dev=0 -): +def _snap(power=300, util=80, mem=12, temp=65, dev=0): """Shorthand to build a GpuSnapshot.""" from openjarvis.telemetry.gpu_monitor import GpuSnapshot @@ -105,12 +95,18 @@ class TestGpuHardwareSpec: expected = { "B200-SXM", - "A100-SXM", "A100-PCIE", - "H100-SXM", "H100-PCIE", - "L40S", "A10", - "RTX 4090", "RTX 3090", - "MI300X", "MI250X", - "M4 Max", "M2 Ultra", + "A100-SXM", + "A100-PCIE", + "H100-SXM", + "H100-PCIE", + "L40S", + "A10", + "RTX 4090", + "RTX 3090", + "MI300X", + "MI250X", + "M4 Max", + "M2 Ultra", } assert set(GPU_SPECS.keys()) == expected @@ -132,15 +128,10 @@ class TestEnergyIntegration: """Constant 300W for 10 seconds = 3000 J.""" from openjarvis.telemetry.gpu_monitor import GpuMonitor - snapshots = [ - [_snap(power=300)] - for _ in range(11) - ] + snapshots = [[_snap(power=300)] for _ in range(11)] timestamps = [float(i) for i in range(11)] - sample = GpuMonitor._aggregate( - snapshots, timestamps, wall_duration=10.0 - ) + sample = GpuMonitor._aggregate(snapshots, timestamps, wall_duration=10.0) assert sample.energy_joules == pytest.approx(3000.0, rel=1e-6) assert sample.mean_power_watts == pytest.approx(300.0, rel=1e-6) assert sample.peak_power_watts == pytest.approx(300.0, rel=1e-6) @@ -152,14 +143,11 @@ class TestEnergyIntegration: from openjarvis.telemetry.gpu_monitor import GpuMonitor snapshots = [ - [_snap(power=100.0 * i, util=50, mem=8, temp=60)] - for i in range(5) + [_snap(power=100.0 * i, util=50, mem=8, temp=60)] for i in range(5) ] timestamps = [float(i) for i in range(5)] - sample = GpuMonitor._aggregate( - snapshots, timestamps, wall_duration=4.0 - ) + sample = GpuMonitor._aggregate(snapshots, timestamps, wall_duration=4.0) assert sample.energy_joules == pytest.approx(800.0, rel=1e-6) assert sample.mean_power_watts == pytest.approx(200.0, rel=1e-6) assert sample.peak_power_watts == pytest.approx(400.0) @@ -177,14 +165,10 @@ class TestEnergyIntegration: """One snapshot: energy is zero (no interval).""" from openjarvis.telemetry.gpu_monitor import GpuMonitor - snapshots = [ - [_snap(power=250, util=90, mem=16, temp=70)] - ] + snapshots = [[_snap(power=250, util=90, mem=16, temp=70)]] timestamps = [0.0] - sample = GpuMonitor._aggregate( - snapshots, timestamps, wall_duration=0.05 - ) + sample = GpuMonitor._aggregate(snapshots, timestamps, wall_duration=0.05) assert sample.energy_joules == 0.0 assert sample.num_snapshots == 1 assert sample.mean_power_watts == pytest.approx(250.0) @@ -206,9 +190,7 @@ class TestGpuSampleAggregation: ] timestamps = [0.0, 1.0, 2.0] - sample = GpuMonitor._aggregate( - snapshots, timestamps, wall_duration=2.0 - ) + sample = GpuMonitor._aggregate(snapshots, timestamps, wall_duration=2.0) assert sample.peak_power_watts == pytest.approx(400.0) assert sample.peak_utilization_pct == pytest.approx(95.0) assert sample.peak_memory_used_gb == pytest.approx(20.0) @@ -224,9 +206,7 @@ class TestGpuSampleAggregation: ] timestamps = [0.0, 1.0, 2.0] - sample = GpuMonitor._aggregate( - snapshots, timestamps, wall_duration=2.0 - ) + sample = GpuMonitor._aggregate(snapshots, timestamps, wall_duration=2.0) assert sample.mean_power_watts == pytest.approx(200.0) assert sample.mean_utilization_pct == pytest.approx(60.0) assert sample.mean_memory_used_gb == pytest.approx(12.0) @@ -250,9 +230,7 @@ class TestMultiGpu: snapshots = [tick, tick] timestamps = [0.0, 1.0] - sample = GpuMonitor._aggregate( - snapshots, timestamps, wall_duration=1.0 - ) + sample = GpuMonitor._aggregate(snapshots, timestamps, wall_duration=1.0) assert sample.energy_joules == pytest.approx(500.0) assert sample.mean_power_watts == pytest.approx(500.0) assert sample.mean_utilization_pct == pytest.approx(85.0) @@ -275,9 +253,7 @@ class TestMultiGpu: ] timestamps = [0.0, 2.0] - sample = GpuMonitor._aggregate( - snapshots, timestamps, wall_duration=2.0 - ) + sample = GpuMonitor._aggregate(snapshots, timestamps, wall_duration=2.0) # Tick powers: 200, 600 => 0.5*(200+600)*2 = 800 assert sample.energy_joules == pytest.approx(800.0) assert sample.peak_power_watts == pytest.approx(600.0) @@ -291,9 +267,7 @@ class TestMultiGpu: class TestContextManager: def test_sample_context_manager(self): """sample() starts/stops polling and populates result.""" - fake_pynvml = _make_fake_pynvml( - device_count=1, power_mw=300_000 - ) + fake_pynvml = _make_fake_pynvml(device_count=1, power_mw=300_000) import openjarvis.telemetry.gpu_monitor as mod diff --git a/tests/telemetry/test_instrumented_engine.py b/tests/telemetry/test_instrumented_engine.py index 2795ad9e..50e23965 100644 --- a/tests/telemetry/test_instrumented_engine.py +++ b/tests/telemetry/test_instrumented_engine.py @@ -63,8 +63,7 @@ class TestInstrumentedEngine: ie.generate(messages, model="test") tel_events = [ - e for e in bus.history - if e.event_type == EventType.TELEMETRY_RECORD + e for e in bus.history if e.event_type == EventType.TELEMETRY_RECORD ] assert len(tel_events) == 1 record = tel_events[0].data["record"] @@ -92,9 +91,8 @@ class TestInstrumentedEngine: messages = [Message(role=Role.USER, content="Hi")] ie.generate(messages, model="test", temperature=0.5, max_tokens=100) call_kwargs = mock_engine.generate.call_args - temp = ( - call_kwargs.kwargs.get("temperature") - or call_kwargs[1].get("temperature") + temp = call_kwargs.kwargs.get("temperature") or call_kwargs[1].get( + "temperature" ) assert temp == 0.5 @@ -130,8 +128,7 @@ class TestTokensPerJoule: ie.generate(messages, model="test") tel_events = [ - e for e in bus.history - if e.event_type == EventType.TELEMETRY_RECORD + e for e in bus.history if e.event_type == EventType.TELEMETRY_RECORD ] record = tel_events[0].data["record"] assert record.tokens_per_joule == 0.0 diff --git a/tests/telemetry/test_itl_metrics.py b/tests/telemetry/test_itl_metrics.py index ff067575..0bd8aeb8 100644 --- a/tests/telemetry/test_itl_metrics.py +++ b/tests/telemetry/test_itl_metrics.py @@ -330,18 +330,20 @@ class TestItlStorage: def test_store_and_query_itl(self, tmp_path): store = TelemetryStore(tmp_path / "test.db") - store.record(TelemetryRecord( - timestamp=time.time(), - model_id="m1", - engine="mock", - mean_itl_ms=15.0, - median_itl_ms=14.0, - p90_itl_ms=20.0, - p95_itl_ms=25.0, - p99_itl_ms=30.0, - std_itl_ms=5.0, - is_streaming=True, - )) + store.record( + TelemetryRecord( + timestamp=time.time(), + model_id="m1", + engine="mock", + mean_itl_ms=15.0, + median_itl_ms=14.0, + p90_itl_ms=20.0, + p95_itl_ms=25.0, + p99_itl_ms=30.0, + std_itl_ms=5.0, + is_streaming=True, + ) + ) agg = TelemetryAggregator(tmp_path / "test.db") stats = agg.per_model_stats() diff --git a/tests/telemetry/test_itl_new.py b/tests/telemetry/test_itl_new.py index 66f1ec00..b553b20d 100644 --- a/tests/telemetry/test_itl_new.py +++ b/tests/telemetry/test_itl_new.py @@ -57,7 +57,12 @@ class TestComputeItlStats: def test_all_keys_present(self): result = compute_itl_stats([0.0, 5.0, 10.0]) expected_keys = { - "p50_ms", "p90_ms", "p95_ms", "p99_ms", - "mean_ms", "min_ms", "max_ms", + "p50_ms", + "p90_ms", + "p95_ms", + "p99_ms", + "mean_ms", + "min_ms", + "max_ms", } assert set(result.keys()) == expected_keys diff --git a/tests/telemetry/test_phase_energy.py b/tests/telemetry/test_phase_energy.py index 2878e28c..40f539b0 100644 --- a/tests/telemetry/test_phase_energy.py +++ b/tests/telemetry/test_phase_energy.py @@ -85,9 +85,7 @@ class TestDecodeLatency: ie.generate([Message(role=Role.USER, content="hi")], model="m") rec = records[0] # decode_latency = latency - ttft - assert rec.decode_latency_seconds == pytest.approx( - rec.latency_seconds - 0.1 - ) + assert rec.decode_latency_seconds == pytest.approx(rec.latency_seconds - 0.1) def test_decode_latency_zero_when_no_ttft(self): bus = EventBus() @@ -213,14 +211,16 @@ class TestPhaseEnergyStorage: def test_store_and_aggregate(self, tmp_path): store = TelemetryStore(tmp_path / "test.db") - store.record(TelemetryRecord( - timestamp=time.time(), - model_id="m1", - engine="mock", - energy_joules=10.0, - prefill_energy_joules=3.0, - decode_energy_joules=7.0, - )) + store.record( + TelemetryRecord( + timestamp=time.time(), + model_id="m1", + engine="mock", + energy_joules=10.0, + prefill_energy_joules=3.0, + decode_energy_joules=7.0, + ) + ) agg = TelemetryAggregator(tmp_path / "test.db") stats = agg.per_model_stats() diff --git a/tests/telemetry/test_phase_metrics_new.py b/tests/telemetry/test_phase_metrics_new.py index d4c5f23f..569b88a8 100644 --- a/tests/telemetry/test_phase_metrics_new.py +++ b/tests/telemetry/test_phase_metrics_new.py @@ -14,11 +14,13 @@ class TestComputePhaseMetrics: session = TelemetrySession(monitor=None) # Manually push samples into the buffer for i in range(11): - session._buffer.push(TelemetrySample( - timestamp_ns=i * 100_000_000, - gpu_power_w=100.0, - cpu_power_w=50.0, - )) + session._buffer.push( + TelemetrySample( + timestamp_ns=i * 100_000_000, + gpu_power_w=100.0, + cpu_power_w=50.0, + ) + ) return session def test_basic_metrics(self): @@ -38,7 +40,7 @@ class TestComputePhaseMetrics: prefill, decode = split_at_ttft( session, start_ns=0, - ttft_ns=500_000_000, # 500ms + ttft_ns=500_000_000, # 500ms end_ns=1_000_000_000, # 1s input_tokens=50, output_tokens=100, diff --git a/tests/telemetry/test_session.py b/tests/telemetry/test_session.py index 7226b27e..24b15801 100644 --- a/tests/telemetry/test_session.py +++ b/tests/telemetry/test_session.py @@ -40,14 +40,16 @@ class TestPythonRingBuffer: buf = _PythonRingBuffer(capacity=100) # 100W GPU, 50W CPU for 1 second (10 samples, 100ms apart) for i in range(11): - buf.push(TelemetrySample( - timestamp_ns=i * 100_000_000, # 0ms, 100ms, ..., 1000ms - gpu_power_w=100.0, - cpu_power_w=50.0, - )) + buf.push( + TelemetrySample( + timestamp_ns=i * 100_000_000, # 0ms, 100ms, ..., 1000ms + gpu_power_w=100.0, + cpu_power_w=50.0, + ) + ) gpu_j, cpu_j = buf.compute_energy_delta(0, 1_000_000_000) assert abs(gpu_j - 100.0) < 1.0 # 100W * 1s = 100J - assert abs(cpu_j - 50.0) < 1.0 # 50W * 1s = 50J + assert abs(cpu_j - 50.0) < 1.0 # 50W * 1s = 50J def test_energy_delta_insufficient_samples(self): buf = _PythonRingBuffer(capacity=100) @@ -59,10 +61,13 @@ class TestPythonRingBuffer: def test_avg_power(self): buf = _PythonRingBuffer(capacity=100) buf.push(TelemetrySample(timestamp_ns=0, gpu_power_w=100.0, cpu_power_w=40.0)) - buf.push(TelemetrySample( - timestamp_ns=500_000_000, gpu_power_w=200.0, - cpu_power_w=60.0, - )) + buf.push( + TelemetrySample( + timestamp_ns=500_000_000, + gpu_power_w=200.0, + cpu_power_w=60.0, + ) + ) gpu_w, cpu_w = buf.compute_avg_power(0, 1_000_000_000) assert gpu_w == 150.0 assert cpu_w == 50.0 diff --git a/tests/telemetry/test_steady_state.py b/tests/telemetry/test_steady_state.py index 0420afc4..34eaa307 100644 --- a/tests/telemetry/test_steady_state.py +++ b/tests/telemetry/test_steady_state.py @@ -68,7 +68,10 @@ class TestSteadyStateDetector: def test_erratic_values_no_steady_state(self): """Highly variable values should not reach steady state.""" cfg = SteadyStateConfig( - warmup_samples=2, window_size=3, cv_threshold=0.01, min_steady_samples=3, + warmup_samples=2, + window_size=3, + cv_threshold=0.01, + min_steady_samples=3, ) detector = SteadyStateDetector(cfg) @@ -96,7 +99,10 @@ class TestSteadyStateDetector: def test_cv_calculation_correctness(self): """Verify CV-based detection with known values.""" cfg = SteadyStateConfig( - warmup_samples=2, window_size=3, cv_threshold=0.05, min_steady_samples=1, + warmup_samples=2, + window_size=3, + cv_threshold=0.05, + min_steady_samples=1, ) detector = SteadyStateDetector(cfg) diff --git a/tests/telemetry/test_vllm_metrics.py b/tests/telemetry/test_vllm_metrics.py index eb651e58..89dcf4ef 100644 --- a/tests/telemetry/test_vllm_metrics.py +++ b/tests/telemetry/test_vllm_metrics.py @@ -166,6 +166,7 @@ class TestVLLMMetricsScraper: class FakeResp: status_code = 200 text = SAMPLE_METRICS + def raise_for_status(self): pass diff --git a/tests/telemetry/test_wrapper.py b/tests/telemetry/test_wrapper.py index d280c661..63462214 100644 --- a/tests/telemetry/test_wrapper.py +++ b/tests/telemetry/test_wrapper.py @@ -73,8 +73,7 @@ class TestInstrumentedGenerate: bus=bus, ) telem_events = [ - e for e in bus.history - if e.event_type == EventType.TELEMETRY_RECORD + e for e in bus.history if e.event_type == EventType.TELEMETRY_RECORD ] assert len(telem_events) == 1 rec = telem_events[0].data["record"] @@ -91,8 +90,7 @@ class TestInstrumentedGenerate: bus=bus, ) telem_events = [ - e for e in bus.history - if e.event_type == EventType.TELEMETRY_RECORD + e for e in bus.history if e.event_type == EventType.TELEMETRY_RECORD ] rec = telem_events[0].data["record"] assert rec.latency_seconds >= 0 diff --git a/tests/templates/test_agent_templates.py b/tests/templates/test_agent_templates.py index 5877af44..5f703393 100644 --- a/tests/templates/test_agent_templates.py +++ b/tests/templates/test_agent_templates.py @@ -13,11 +13,7 @@ from openjarvis.templates.agent_templates import ( ) TEMPLATES_DIR = ( - Path(__file__).resolve().parents[2] - / "src" - / "openjarvis" - / "templates" - / "data" + Path(__file__).resolve().parents[2] / "src" / "openjarvis" / "templates" / "data" ) VALID_AGENT_TYPES = {"simple", "orchestrator", "native_react", "monitor"} diff --git a/tests/test_orchestrator_learning/test_environment.py b/tests/test_orchestrator_learning/test_environment.py index eee9b532..bcce6ac2 100644 --- a/tests/test_orchestrator_learning/test_environment.py +++ b/tests/test_orchestrator_learning/test_environment.py @@ -86,9 +86,7 @@ class TestOrchestratorEnvironment: assert env.is_done(state) is True def test_is_done_on_max_turns(self): - env = OrchestratorEnvironment( - tools=[_MockCalculator()], max_turns=2 - ) + env = OrchestratorEnvironment(tools=[_MockCalculator()], max_turns=2) state = env.reset("q") for _ in range(2): action = OrchestratorAction( @@ -107,9 +105,7 @@ class TestOrchestratorEnvironment: env.step(state, action) def test_max_turns_exceeded_raises(self): - env = OrchestratorEnvironment( - tools=[_MockCalculator()], max_turns=1 - ) + env = OrchestratorEnvironment(tools=[_MockCalculator()], max_turns=1) state = env.reset("q") action = OrchestratorAction( thought="go", tool_name="calculator", tool_input="1+1" diff --git a/tests/test_orchestrator_learning/test_grpo_trainer.py b/tests/test_orchestrator_learning/test_grpo_trainer.py index 066e5c24..4f122640 100644 --- a/tests/test_orchestrator_learning/test_grpo_trainer.py +++ b/tests/test_orchestrator_learning/test_grpo_trainer.py @@ -99,4 +99,5 @@ class TestGRPORegistration: def test_registered_in_learning_registry(self): import openjarvis.learning.intelligence.orchestrator.grpo_trainer # noqa: F401 from openjarvis.core.registry import LearningRegistry + assert LearningRegistry.contains("orchestrator_grpo") diff --git a/tests/test_orchestrator_learning/test_orchestrator_structured.py b/tests/test_orchestrator_learning/test_orchestrator_structured.py index d96ef7bb..63643935 100644 --- a/tests/test_orchestrator_learning/test_orchestrator_structured.py +++ b/tests/test_orchestrator_learning/test_orchestrator_structured.py @@ -55,9 +55,7 @@ class _MockTool(BaseTool): def execute(self, **params) -> ToolResult: expr = params.get("expression", "") - return ToolResult( - tool_name="calculator", content=str(expr), success=True - ) + return ToolResult(tool_name="calculator", content=str(expr), success=True) # -- Tests ------------------------------------------------------------------- @@ -66,10 +64,12 @@ class _MockTool(BaseTool): class TestStructuredMode: def test_thought_tool_input_then_final_answer(self): """Test full structured loop: TOOL call -> FINAL_ANSWER.""" - engine = _MockEngine([ - "THOUGHT: I need to calculate\nTOOL: calculator\nINPUT: 2+2", - "THOUGHT: Got result\nFINAL_ANSWER: 4", - ]) + engine = _MockEngine( + [ + "THOUGHT: I need to calculate\nTOOL: calculator\nINPUT: 2+2", + "THOUGHT: Got result\nFINAL_ANSWER: 4", + ] + ) agent = OrchestratorAgent( engine=engine, model="test", @@ -83,9 +83,11 @@ class TestStructuredMode: def test_direct_final_answer(self): """Test that FINAL_ANSWER on first turn works.""" - engine = _MockEngine([ - "THOUGHT: Easy\nFINAL_ANSWER: Paris", - ]) + engine = _MockEngine( + [ + "THOUGHT: Easy\nFINAL_ANSWER: Paris", + ] + ) agent = OrchestratorAgent( engine=engine, model="test", diff --git a/tests/test_orchestrator_learning/test_policy_model.py b/tests/test_orchestrator_learning/test_policy_model.py index 59890251..09795353 100644 --- a/tests/test_orchestrator_learning/test_policy_model.py +++ b/tests/test_orchestrator_learning/test_policy_model.py @@ -22,11 +22,7 @@ class TestParseOutput: def test_valid_thought_tool_input(self): m = self._model() - text = ( - "THOUGHT: I need to calculate 2+2\n" - "TOOL: calculator\n" - "INPUT: 2+2" - ) + text = "THOUGHT: I need to calculate 2+2\nTOOL: calculator\nINPUT: 2+2" po = m._parse_output(text, ["calculator", "think"]) assert po.thought == "I need to calculate 2+2" assert po.tool_name == "calculator" @@ -35,10 +31,7 @@ class TestParseOutput: def test_final_answer(self): m = self._model() - text = ( - "THOUGHT: I have the result\n" - "FINAL_ANSWER: 42" - ) + text = "THOUGHT: I have the result\nFINAL_ANSWER: 42" po = m._parse_output(text, ["calculator"]) assert po.is_final_answer is True assert po.tool_input == "42" diff --git a/tests/test_orchestrator_learning/test_prompt_registry.py b/tests/test_orchestrator_learning/test_prompt_registry.py index 30028ebc..2d1c527b 100644 --- a/tests/test_orchestrator_learning/test_prompt_registry.py +++ b/tests/test_orchestrator_learning/test_prompt_registry.py @@ -60,9 +60,7 @@ class TestBuildSystemPrompt: assert "custom_tool_xyz" in prompt def test_with_memory_tools(self): - prompt = build_system_prompt( - ["calculator", "memory_search", "memory_store"] - ) + prompt = build_system_prompt(["calculator", "memory_search", "memory_store"]) assert "memory_search" in prompt def test_with_llm_tool(self): diff --git a/tests/test_orchestrator_learning/test_sft_trainer.py b/tests/test_orchestrator_learning/test_sft_trainer.py index b9768eab..3e2dfbcf 100644 --- a/tests/test_orchestrator_learning/test_sft_trainer.py +++ b/tests/test_orchestrator_learning/test_sft_trainer.py @@ -97,15 +97,15 @@ class TestOrchestratorSFTDataset: trace_file = tmp_path / "traces.jsonl" traces = [] for i in range(5): - traces.append({ - "conversations": [ - {"role": "user", "content": f"q{i}"}, - {"role": "assistant", "content": f"a{i}"}, - ] - }) - trace_file.write_text( - "\n".join(json.dumps(t) for t in traces) + "\n" - ) + traces.append( + { + "conversations": [ + {"role": "user", "content": f"q{i}"}, + {"role": "assistant", "content": f"a{i}"}, + ] + } + ) + trace_file.write_text("\n".join(json.dumps(t) for t in traces) + "\n") tok = MagicMock() tok.eos_token = "" @@ -128,4 +128,5 @@ class TestSFTRegistration: # Import to trigger registration import openjarvis.learning.intelligence.orchestrator.sft_trainer # noqa: F401 from openjarvis.core.registry import LearningRegistry + assert LearningRegistry.contains("orchestrator_sft") diff --git a/tests/test_orchestrator_learning/test_types.py b/tests/test_orchestrator_learning/test_types.py index 63930f60..f330f7d2 100644 --- a/tests/test_orchestrator_learning/test_types.py +++ b/tests/test_orchestrator_learning/test_types.py @@ -147,9 +147,7 @@ class TestEpisodeState: def test_add_turn(self): state = EpisodeState(initial_prompt="q") - action = OrchestratorAction( - thought="t", tool_name="calc", tool_input="1+1" - ) + action = OrchestratorAction(thought="t", tool_name="calc", tool_input="1+1") obs = OrchestratorObservation(content="2") state.add_turn(action, obs) assert state.num_turns() == 1 diff --git a/tests/tools/test_apply_patch.py b/tests/tools/test_apply_patch.py index b606b66e..734c0b81 100644 --- a/tests/tools/test_apply_patch.py +++ b/tests/tools/test_apply_patch.py @@ -127,13 +127,7 @@ class TestApplyPatchTool: def test_blocks_sensitive_files(self, tmp_path): f = tmp_path / ".env" f.write_text("SECRET=foo\n", encoding="utf-8") - patch = ( - "--- a/.env\n" - "+++ b/.env\n" - "@@ -1 +1 @@\n" - "-SECRET=foo\n" - "+SECRET=bar\n" - ) + patch = "--- a/.env\n+++ b/.env\n@@ -1 +1 @@\n-SECRET=foo\n+SECRET=bar\n" tool = ApplyPatchTool() result = tool.execute(patch=patch, path=str(f)) assert result.success is False @@ -143,13 +137,7 @@ class TestApplyPatchTool: f = tmp_path / "auto.txt" f.write_text("one\ntwo\nthree\n", encoding="utf-8") patch = ( - "--- a/auto.txt\n" - f"+++ b/{f}\n" - "@@ -1,3 +1,3 @@\n" - " one\n" - "-two\n" - "+TWO\n" - " three\n" + f"--- a/auto.txt\n+++ b/{f}\n@@ -1,3 +1,3 @@\n one\n-two\n+TWO\n three\n" ) tool = ApplyPatchTool() # No explicit path — should auto-detect from +++ header @@ -167,11 +155,7 @@ class TestApplyPatchTool: def test_file_not_found(self): tool = ApplyPatchTool() patch = ( - "--- a/nonexistent.txt\n" - "+++ b/nonexistent.txt\n" - "@@ -1 +1 @@\n" - "-old\n" - "+new\n" + "--- a/nonexistent.txt\n+++ b/nonexistent.txt\n@@ -1 +1 @@\n-old\n+new\n" ) result = tool.execute(patch=patch, path="/nonexistent/path/file.txt") assert result.success is False diff --git a/tests/tools/test_audio_tool.py b/tests/tools/test_audio_tool.py index d9cee4cc..bef3ccba 100644 --- a/tests/tools/test_audio_tool.py +++ b/tests/tools/test_audio_tool.py @@ -169,9 +169,7 @@ class TestAudioTranscribeTool: monkeypatch.setenv("OPENAI_API_KEY", "test-key") mock_client = MagicMock() - mock_client.audio.transcriptions.create.side_effect = RuntimeError( - "API error" - ) + mock_client.audio.transcriptions.create.side_effect = RuntimeError("API error") mock_openai = MagicMock() mock_openai.OpenAI.return_value = mock_client diff --git a/tests/tools/test_browser.py b/tests/tools/test_browser.py index df810a9a..3f658924 100644 --- a/tests/tools/test_browser.py +++ b/tests/tools/test_browser.py @@ -48,8 +48,7 @@ def _make_import_error_session(): session = MagicMock() type(session).page = PropertyMock( side_effect=ImportError( - "playwright not installed. Install with: " - "uv sync --extra browser" + "playwright not installed. Install with: uv sync --extra browser" ) ) return session @@ -145,6 +144,7 @@ class TestBrowserNavigateTool: # Make the ssrf import fail inside execute import builtins + original_import = builtins.__import__ def _mock_import(name, *args, **kwargs): @@ -202,9 +202,7 @@ class TestBrowserNavigateTool: tool = BrowserNavigateTool() tool.execute(url="https://example.com", wait_for="invalid") - page.goto.assert_called_once_with( - "https://example.com", wait_until="load" - ) + page.goto.assert_called_once_with("https://example.com", wait_until="load") def test_execute_content_truncation(self): from openjarvis.tools.browser import BrowserNavigateTool @@ -840,6 +838,7 @@ class TestBrowserSession: session = _BrowserSession() import builtins + original_import = builtins.__import__ def _mock_import(name, *args, **kwargs): diff --git a/tests/tools/test_channel_tools.py b/tests/tools/test_channel_tools.py index 07ea94fb..2c9b1907 100644 --- a/tests/tools/test_channel_tools.py +++ b/tests/tools/test_channel_tools.py @@ -34,11 +34,13 @@ class _MockChannel(BaseChannel): self._status = ChannelStatus.DISCONNECTED def send(self, channel, content, *, conversation_id="", metadata=None) -> bool: - self._sent.append({ - "channel": channel, - "content": content, - "conversation_id": conversation_id, - }) + self._sent.append( + { + "channel": channel, + "content": content, + "conversation_id": conversation_id, + } + ) return True def status(self) -> ChannelStatus: @@ -95,7 +97,9 @@ class TestChannelSendTool: def test_send_with_conversation_id(self, channel): tool = ChannelSendTool(channel) result = tool.execute( - channel="chat-123", content="Reply", conversation_id="conv-1", + channel="chat-123", + content="Reply", + conversation_id="conv-1", ) assert result.success is True assert channel._sent[0]["conversation_id"] == "conv-1" diff --git a/tests/tools/test_file_write.py b/tests/tools/test_file_write.py index 643f5035..002de3e7 100644 --- a/tests/tools/test_file_write.py +++ b/tests/tools/test_file_write.py @@ -45,7 +45,9 @@ class TestFileWriteTool: f = tmp_path / "sub" / "deep" / "test.txt" tool = FileWriteTool() result = tool.execute( - path=str(f), content="nested", create_dirs=True, + path=str(f), + content="nested", + create_dirs=True, ) assert result.success is True assert f.read_text(encoding="utf-8") == "nested" @@ -68,7 +70,8 @@ class TestFileWriteTool: f = tmp_path / "server.pem" tool = FileWriteTool() result = tool.execute( - path=str(f), content="-----BEGIN CERTIFICATE-----", + path=str(f), + content="-----BEGIN CERTIFICATE-----", ) assert result.success is False assert "sensitive" in result.content.lower() @@ -124,7 +127,9 @@ class TestFileWriteTool: f = tmp_path / "test.txt" tool = FileWriteTool() result = tool.execute( - path=str(f), content="data", mode="invalid", + path=str(f), + content="data", + mode="invalid", ) assert result.success is False assert "Invalid mode" in result.content diff --git a/tests/tools/test_git_tool.py b/tests/tools/test_git_tool.py index 79c72144..e3cc5b80 100644 --- a/tests/tools/test_git_tool.py +++ b/tests/tools/test_git_tool.py @@ -497,10 +497,7 @@ class TestGitLogTool: assert result.success is True # Rust ignores the count param (reads "n", gets "count") and # defaults to 10, so all 6 commits are returned. - lines = [ - line for line in result.content.strip().splitlines() - if line.strip() - ] + lines = [line for line in result.content.strip().splitlines() if line.strip()] assert len(lines) == 6 def test_default_count_is_10(self): diff --git a/tests/tools/test_http_request.py b/tests/tools/test_http_request.py index 544e4289..04f68805 100644 --- a/tests/tools/test_http_request.py +++ b/tests/tools/test_http_request.py @@ -83,8 +83,7 @@ class TestHttpRequestTool: tool = HttpRequestTool() with patch("openjarvis.tools.http_request.check_ssrf") as mock_ssrf: mock_ssrf.return_value = ( - "Blocked host: 169.254.169.254" - " (cloud metadata endpoint)" + "Blocked host: 169.254.169.254 (cloud metadata endpoint)" ) result = tool.execute(url="http://169.254.169.254/latest/meta-data/") assert result.success is False diff --git a/tests/tools/test_knowledge_graph.py b/tests/tools/test_knowledge_graph.py index 79d97c6f..7b094e71 100644 --- a/tests/tools/test_knowledge_graph.py +++ b/tests/tools/test_knowledge_graph.py @@ -16,8 +16,10 @@ class TestKnowledgeGraph: def test_add_and_get_entity(self, tmp_path): kg = self._make_kg(tmp_path) entity = Entity( - entity_id="e1", entity_type="concept", - name="Machine Learning", properties={"field": "AI"}, + entity_id="e1", + entity_type="concept", + name="Machine Learning", + properties={"field": "AI"}, ) kg.add_entity(entity) result = kg.get_entity("e1") @@ -35,10 +37,13 @@ class TestKnowledgeGraph: kg = self._make_kg(tmp_path) kg.add_entity(Entity(entity_id="a", entity_type="concept", name="A")) kg.add_entity(Entity(entity_id="b", entity_type="concept", name="B")) - kg.add_relation(Relation( - source_id="a", target_id="b", - relation_type="depends_on", - )) + kg.add_relation( + Relation( + source_id="a", + target_id="b", + relation_type="depends_on", + ) + ) assert kg.relation_count() == 1 kg.close() @@ -61,10 +66,13 @@ class TestKnowledgeGraph: kg.add_entity(Entity(entity_id="b", entity_type="concept", name="B")) kg.add_entity(Entity(entity_id="c", entity_type="concept", name="C")) kg.add_relation(Relation(source_id="a", target_id="b", relation_type="uses")) - kg.add_relation(Relation( - source_id="a", target_id="c", - relation_type="produces", - )) + kg.add_relation( + Relation( + source_id="a", + target_id="c", + relation_type="produces", + ) + ) neighbors = kg.neighbors("a", relation_type="uses", direction="out") assert len(neighbors) == 1 assert neighbors[0].name == "B" diff --git a/tests/tools/test_retrieval.py b/tests/tools/test_retrieval.py index eaac1847..1219defa 100644 --- a/tests/tools/test_retrieval.py +++ b/tests/tools/test_retrieval.py @@ -17,13 +17,20 @@ class _FakeBackend(MemoryBackend): self._results = results or [] def store( - self, content: str, *, source: str = "", + self, + content: str, + *, + source: str = "", metadata: Optional[Dict[str, Any]] = None, ) -> str: return "fake-id" def retrieve( - self, query: str, *, top_k: int = 5, **kwargs: Any, + self, + query: str, + *, + top_k: int = 5, + **kwargs: Any, ) -> List[RetrievalResult]: return self._results[:top_k] @@ -36,7 +43,11 @@ class _FakeBackend(MemoryBackend): class _ErrorBackend(_FakeBackend): def retrieve( - self, query: str, *, top_k: int = 5, **kwargs: Any, + self, + query: str, + *, + top_k: int = 5, + **kwargs: Any, ) -> List[RetrievalResult]: raise RuntimeError("backend error") diff --git a/tests/tools/test_storage_stubs.py b/tests/tools/test_storage_stubs.py index ba63793f..96ad2470 100644 --- a/tests/tools/test_storage_stubs.py +++ b/tests/tools/test_storage_stubs.py @@ -24,11 +24,13 @@ class _DummyStorage(MemoryBackend): results = [] for doc_id, doc in self._data.items(): if query.lower() in doc["content"].lower(): - results.append(RetrievalResult( - content=doc["content"], - score=1.0, - source=doc["source"], - )) + results.append( + RetrievalResult( + content=doc["content"], + score=1.0, + source=doc["source"], + ) + ) return results[:top_k] def delete(self, doc_id): @@ -67,12 +69,14 @@ class TestStorageStubs: """Memory imports should still work via shim.""" from openjarvis.tools.storage._stubs import MemoryBackend as MB from openjarvis.tools.storage._stubs import RetrievalResult as RR + assert MB is MemoryBackend assert RR is RetrievalResult def test_canonical_import(self) -> None: """Canonical import from tools.storage should work.""" from openjarvis.tools.storage._stubs import MemoryBackend as MB + assert MB is MemoryBackend def test_sqlite_backward_compat(self) -> None: diff --git a/tests/tools/test_storage_tools.py b/tests/tools/test_storage_tools.py index 6d56d1df..558ac143 100644 --- a/tests/tools/test_storage_tools.py +++ b/tests/tools/test_storage_tools.py @@ -181,7 +181,9 @@ class TestMemoryIndexTool: def test_index_file(self, backend): with tempfile.NamedTemporaryFile( - mode="w", suffix=".txt", delete=False, + mode="w", + suffix=".txt", + delete=False, ) as f: f.write("This is test content for indexing.") f.flush() diff --git a/tests/tools/test_templates.py b/tests/tools/test_templates.py index 2ccf6af5..ae504c99 100644 --- a/tests/tools/test_templates.py +++ b/tests/tools/test_templates.py @@ -7,107 +7,129 @@ from openjarvis.tools.templates.loader import ToolTemplate, discover_templates class TestToolTemplate: def test_create_template(self): - template = ToolTemplate({ - "name": "test_tool", - "description": "A test tool", - "action": {"type": "transform", "transform": "upper"}, - }) + template = ToolTemplate( + { + "name": "test_tool", + "description": "A test tool", + "action": {"type": "transform", "transform": "upper"}, + } + ) assert template.spec.name == "test_tool" def test_execute_upper_transform(self): - template = ToolTemplate({ - "name": "upper", - "description": "Uppercase", - "action": {"type": "transform", "transform": "upper"}, - }) + template = ToolTemplate( + { + "name": "upper", + "description": "Uppercase", + "action": {"type": "transform", "transform": "upper"}, + } + ) result = template.execute(input="hello") assert result.success assert result.content == "HELLO" def test_execute_lower_transform(self): - template = ToolTemplate({ - "name": "lower", - "description": "Lowercase", - "action": {"type": "transform", "transform": "lower"}, - }) + template = ToolTemplate( + { + "name": "lower", + "description": "Lowercase", + "action": {"type": "transform", "transform": "lower"}, + } + ) result = template.execute(input="HELLO") assert result.success assert result.content == "hello" def test_execute_reverse_transform(self): - template = ToolTemplate({ - "name": "reverse", - "description": "Reverse", - "action": {"type": "transform", "transform": "reverse"}, - }) + template = ToolTemplate( + { + "name": "reverse", + "description": "Reverse", + "action": {"type": "transform", "transform": "reverse"}, + } + ) result = template.execute(input="hello") assert result.success assert result.content == "olleh" def test_execute_length_transform(self): - template = ToolTemplate({ - "name": "length", - "description": "Length", - "action": {"type": "transform", "transform": "length"}, - }) + template = ToolTemplate( + { + "name": "length", + "description": "Length", + "action": {"type": "transform", "transform": "length"}, + } + ) result = template.execute(input="hello") assert result.success assert result.content == "5" def test_execute_json_pretty_transform(self): - template = ToolTemplate({ - "name": "json", - "description": "JSON pretty", - "action": {"type": "transform", "transform": "json_pretty"}, - }) + template = ToolTemplate( + { + "name": "json", + "description": "JSON pretty", + "action": {"type": "transform", "transform": "json_pretty"}, + } + ) result = template.execute(input='{"a":1}') assert result.success assert '"a": 1' in result.content def test_execute_json_pretty_invalid(self): - template = ToolTemplate({ - "name": "json", - "description": "JSON pretty", - "action": {"type": "transform", "transform": "json_pretty"}, - }) + template = ToolTemplate( + { + "name": "json", + "description": "JSON pretty", + "action": {"type": "transform", "transform": "json_pretty"}, + } + ) result = template.execute(input="not json") assert not result.success def test_execute_python_action(self): - template = ToolTemplate({ - "name": "py", - "description": "Python", - "action": {"type": "python", "expression": "str(len(input))"}, - }) + template = ToolTemplate( + { + "name": "py", + "description": "Python", + "action": {"type": "python", "expression": "str(len(input))"}, + } + ) result = template.execute(input="hello") assert result.success assert result.content == "5" def test_execute_identity_transform(self): - template = ToolTemplate({ - "name": "identity", - "description": "Identity", - "action": {"type": "transform", "transform": "identity"}, - }) + template = ToolTemplate( + { + "name": "identity", + "description": "Identity", + "action": {"type": "transform", "transform": "identity"}, + } + ) result = template.execute(input="unchanged") assert result.success assert result.content == "unchanged" def test_unknown_action_type(self): - template = ToolTemplate({ - "name": "bad", - "description": "Bad", - "action": {"type": "unknown"}, - }) + template = ToolTemplate( + { + "name": "bad", + "description": "Bad", + "action": {"type": "unknown"}, + } + ) result = template.execute() assert not result.success def test_template_metadata(self): - template = ToolTemplate({ - "name": "test", - "description": "test", - "action": {"type": "transform"}, - }) + template = ToolTemplate( + { + "name": "test", + "description": "test", + "action": {"type": "transform"}, + } + ) assert template.spec.metadata.get("template") is True diff --git a/tests/tools/test_tool_descriptions.py b/tests/tools/test_tool_descriptions.py index 96bd8b2b..3c7ce965 100644 --- a/tests/tools/test_tool_descriptions.py +++ b/tests/tools/test_tool_descriptions.py @@ -28,8 +28,7 @@ class _CalcTool(BaseTool): "expression": { "type": "string", "description": ( - "Math expression to evaluate" - " (e.g. '2+3*4', 'sqrt(16)')" + "Math expression to evaluate (e.g. '2+3*4', 'sqrt(16)')" ), }, }, diff --git a/tests/workflow/test_workflow.py b/tests/workflow/test_workflow.py index 87a2ae29..841d31eb 100644 --- a/tests/workflow/test_workflow.py +++ b/tests/workflow/test_workflow.py @@ -138,22 +138,14 @@ class TestWorkflowEngine: def test_run_transform_node(self): bus = EventBus(record_history=True) engine = WorkflowEngine(bus=bus) - wf = ( - WorkflowBuilder("test") - .add_transform("t", transform="concatenate") - .build() - ) + wf = WorkflowBuilder("test").add_transform("t", transform="concatenate").build() result = engine.run(wf, initial_input="hello") assert result.success def test_events_emitted(self): bus = EventBus(record_history=True) engine = WorkflowEngine(bus=bus) - wf = ( - WorkflowBuilder("test") - .add_transform("t", transform="concatenate") - .build() - ) + wf = WorkflowBuilder("test").add_transform("t", transform="concatenate").build() engine.run(wf, initial_input="hello") event_types = {e.event_type for e in bus.history} assert EventType.WORKFLOW_START in event_types diff --git a/uv.lock b/uv.lock index 5629a0a3..da90bd67 100644 --- a/uv.lock +++ b/uv.lock @@ -4679,6 +4679,7 @@ dependencies = [ { name = "ddgs" }, { name = "httpx" }, { name = "openai" }, + { name = "python-telegram-bot" }, { name = "rich" }, { name = "tomli", marker = "python_full_version < '3.11'" }, ] @@ -4903,6 +4904,7 @@ requires-dist = [ { name = "pytest-asyncio", marker = "extra == 'dev'", specifier = ">=0.24" }, { name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=5" }, { name = "python-multipart", marker = "extra == 'server'", specifier = ">=0.0.9" }, + { name = "python-telegram-bot", specifier = ">=22.6" }, { name = "python-telegram-bot", marker = "extra == 'channel-telegram'", specifier = ">=21.0" }, { name = "rank-bm25", marker = "extra == 'memory-bm25'", specifier = ">=0.2.2" }, { name = "respx", marker = "extra == 'dev'", specifier = ">=0.22" }, From 8468f08c41b59fde124c8ac84dcb3abd81274d13 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Fri, 20 Mar 2026 18:52:18 -0700 Subject: [PATCH 31/92] =?UTF-8?q?fix:=20correct=20O(N=C2=B2)=20energy/FLOP?= =?UTF-8?q?s=20savings=20calculation=20(#95)=20(#97)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The energy_wh_saved and flops_saved values were orders of magnitude too high because a scaling factor that grows linearly with N was applied to the energy calculation, making it scale as O(N³) instead of O(N²). Replace the buggy scale-factor approach with a direct FLOP-to-energy conversion using each provider's per-token constants. Add regression tests to prevent recurrence. Closes #95 Co-authored-by: Claude Opus 4.6 (1M context) --- docs/javascripts/leaderboard.js | 2 ++ src/openjarvis/server/savings.py | 40 +++++++++++-------------- tests/evals/test_use_case_benchmarks.py | 37 +++++++++++++++++++++++ 3 files changed, 57 insertions(+), 22 deletions(-) diff --git a/docs/javascripts/leaderboard.js b/docs/javascripts/leaderboard.js index 60d64c90..742b398a 100644 --- a/docs/javascripts/leaderboard.js +++ b/docs/javascripts/leaderboard.js @@ -10,6 +10,8 @@ var currentPage = 0; // No-KV-cache formula: FLOPs = params_b * 1e9 * N * (N+1) + // Reference values only (GPT-5.3 constants) — recompute functions below + // are defined but not called; the leaderboard displays database values directly. var DEFAULT_PARAMS_B = 137; var ENERGY_WH_PER_FLOP = 0.4 / (1000 * 3e12); diff --git a/src/openjarvis/server/savings.py b/src/openjarvis/server/savings.py index 3f6144a2..993af063 100644 --- a/src/openjarvis/server/savings.py +++ b/src/openjarvis/server/savings.py @@ -101,30 +101,26 @@ def compute_savings( # No-KV-cache FLOPs: P * N * (N+1) params_b = pricing.get("params_b", 200.0) params = params_b * 1e9 - flops = ( - params * total_tokens * (total_tokens + 1) - if total_tokens > 0 else 0.0 - ) - # Scale energy by same ratio as FLOPs (energy ∝ compute) - flops_with_cache = ( - total_tokens * pricing.get("flops_per_token", 3e12) - if total_tokens > 0 else 0.0 - ) - scale = (flops / flops_with_cache) if flops_with_cache > 0 else 1.0 - energy_wh = ( - (total_tokens / 1000) * pricing["energy_wh_per_1k_tokens"] * scale + flops = params * total_tokens * (total_tokens + 1) if total_tokens > 0 else 0.0 + # Derive Wh-per-FLOP from the provider's per-token constants: + # energy_wh_per_1k_tokens / (1000 * flops_per_token) = Wh per FLOP + wh_per_flop = pricing["energy_wh_per_1k_tokens"] / ( + 1000 * pricing.get("flops_per_token", 3e12) ) + energy_wh = flops * wh_per_flop - providers.append(ProviderSavings( - provider=key, - label=pricing["label"], - input_cost=input_cost, - output_cost=output_cost, - total_cost=total_cost, - energy_wh=energy_wh, - energy_joules=energy_wh * 3600, # 1 Wh = 3600 J - flops=flops, - )) + providers.append( + ProviderSavings( + provider=key, + label=pricing["label"], + input_cost=input_cost, + output_cost=output_cost, + total_cost=total_cost, + energy_wh=energy_wh, + energy_joules=energy_wh * 3600, # 1 Wh = 3600 J + flops=flops, + ) + ) # Monthly projection: extrapolate current spend to 720 hours/month if session_duration_hours > 0: diff --git a/tests/evals/test_use_case_benchmarks.py b/tests/evals/test_use_case_benchmarks.py index ec3ebbe3..5bddb4d1 100644 --- a/tests/evals/test_use_case_benchmarks.py +++ b/tests/evals/test_use_case_benchmarks.py @@ -422,3 +422,40 @@ class TestSavings: assert isinstance(d, dict) assert "per_provider" in d assert "total_calls" in d + + def test_energy_scales_linearly(self) -> None: + """Energy should scale with FLOPs (quadratic in N), not N^3.""" + from openjarvis.server.savings import compute_savings + + s1 = compute_savings(1000, 0) + s10 = compute_savings(10000, 0) + # FLOPs ~ N^2, so 10x tokens => ~100x FLOPs => ~100x energy + for p1, p10 in zip(s1.per_provider, s10.per_provider): + ratio = p10.energy_wh / p1.energy_wh + # Allow some tolerance for the (N+1) factor + assert 90 < ratio < 110, ( + f"{p1.provider}: energy ratio {ratio:.1f}, expected ~100" + ) + + def test_energy_wh_matches_direct_formula(self) -> None: + """Energy must equal flops * wh_per_flop for known constants.""" + from openjarvis.server.savings import CLOUD_PRICING, compute_savings + + summary = compute_savings(10000, 0) + for p in summary.per_provider: + pricing = CLOUD_PRICING[p.provider] + wh_per_flop = pricing["energy_wh_per_1k_tokens"] / ( + 1000 * pricing.get("flops_per_token", 3e12) + ) + expected = p.flops * wh_per_flop + assert abs(p.energy_wh - expected) < 1e-6, ( + f"{p.provider}: energy_wh={p.energy_wh}, expected={expected}" + ) + + def test_energy_not_zero(self) -> None: + """Energy must be positive for non-zero token counts.""" + from openjarvis.server.savings import compute_savings + + summary = compute_savings(500, 500) + for p in summary.per_provider: + assert p.energy_wh > 0, f"{p.provider}: energy_wh should be > 0" From 04c28ef7ed7e89a32b82cc01cb7275fa57284a7e Mon Sep 17 00:00:00 2001 From: geodab Date: Mon, 23 Mar 2026 04:38:23 -0400 Subject: [PATCH 32/92] feat: add CLI commands: config, registry, tool (#98) * feat: add CLI commands: config, registry, tool * fix: address review feedback on CLI config/registry/tool commands - Replace non-existent `create_config_template` with `generate_default_toml` to fix ImportError when config file is missing - Deduplicate registry maps into shared `_load_registry_map()` helper - Remove dead `_get_registry_class()` function and its tests - Route JSON output to stdout for pipeability (`jarvis config show json | jq`) Co-Authored-By: Claude Opus 4.6 (1M context) --------- Co-authored-by: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/__init__.py | 6 + src/openjarvis/cli/config_cmd.py | 277 +++++++++++++++++++++++++++++ src/openjarvis/cli/registry_cmd.py | 146 +++++++++++++++ src/openjarvis/cli/tool_cmd.py | 107 +++++++++++ tests/cli/test_config_cmd.py | 210 ++++++++++++++++++++++ tests/cli/test_registry_cmd.py | 154 ++++++++++++++++ tests/cli/test_tool_cmd.py | 238 +++++++++++++++++++++++++ 7 files changed, 1138 insertions(+) create mode 100644 src/openjarvis/cli/config_cmd.py create mode 100644 src/openjarvis/cli/registry_cmd.py create mode 100644 src/openjarvis/cli/tool_cmd.py create mode 100644 tests/cli/test_config_cmd.py create mode 100644 tests/cli/test_registry_cmd.py create mode 100644 tests/cli/test_tool_cmd.py diff --git a/src/openjarvis/cli/__init__.py b/src/openjarvis/cli/__init__.py index 678ff1fa..10a70a94 100644 --- a/src/openjarvis/cli/__init__.py +++ b/src/openjarvis/cli/__init__.py @@ -12,6 +12,7 @@ from openjarvis.cli.bench_cmd import bench from openjarvis.cli.channel_cmd import channel from openjarvis.cli.chat_cmd import chat from openjarvis.cli.compose_cmd import compose +from openjarvis.cli.config_cmd import config from openjarvis.cli.daemon_cmd import restart, start, status, stop from openjarvis.cli.doctor_cmd import doctor from openjarvis.cli.eval_cmd import eval_group @@ -24,10 +25,12 @@ from openjarvis.cli.model import model from openjarvis.cli.operators_cmd import operators from openjarvis.cli.optimize_cmd import optimize_group from openjarvis.cli.quickstart_cmd import quickstart +from openjarvis.cli.registry_cmd import registry from openjarvis.cli.scheduler_cmd import scheduler from openjarvis.cli.serve import serve from openjarvis.cli.skill_cmd import skill from openjarvis.cli.telemetry_cmd import telemetry +from openjarvis.cli.tool_cmd import tool from openjarvis.cli.vault_cmd import vault from openjarvis.cli.workflow_cmd import workflow @@ -81,6 +84,9 @@ cli.add_command(optimize_group, "optimize") cli.add_command(feedback_group, "feedback") cli.add_command(compose, "compose") cli.add_command(gateway, "gateway") +cli.add_command(tool, "tool") +cli.add_command(registry, "registry") +cli.add_command(config, "config") def main() -> None: diff --git a/src/openjarvis/cli/config_cmd.py b/src/openjarvis/cli/config_cmd.py new file mode 100644 index 00000000..324a348a --- /dev/null +++ b/src/openjarvis/cli/config_cmd.py @@ -0,0 +1,277 @@ +"""``jarvis config`` — configuration inspection commands.""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import click +from rich.console import Console +from rich.panel import Panel +from rich.syntax import Syntax +from rich.table import Table + + +@click.group() +def config() -> None: + """Inspect configuration — show loaded settings, hardware, and config files.""" + + +def _get_config_path(path: str | None) -> Path: + """Determine the config path from argument or environment.""" + from openjarvis.core.config import DEFAULT_CONFIG_PATH + + if path: + return Path(path) + return Path(os.environ.get("OPENJARVIS_CONFIG", DEFAULT_CONFIG_PATH)) + + +def _show_hardware_info(console: Console, show_recommendations: bool = True) -> None: + """Display detected hardware information.""" + from openjarvis.core.config import detect_hardware, recommend_engine + + hardware = detect_hardware() + + console.print("\n[bold]Detected Hardware[/bold]") + + hardware_table = Table(show_header=True, header_style="cyan") + hardware_table.add_column("Component", style="green") + hardware_table.add_column("Value", style="white") + + # Platform + hardware_table.add_row("Platform", str(hardware.platform)) + + # CPU + if hardware.cpu_brand: + hardware_table.add_row("CPU", hardware.cpu_brand) + hardware_table.add_row("CPU Count", str(hardware.cpu_count)) + hardware_table.add_row("RAM", f"{hardware.ram_gb:.1f} GB") + + # GPU + if hardware.gpu: + hardware_table.add_row("GPU Vendor", hardware.gpu.vendor) + hardware_table.add_row("GPU Model", hardware.gpu.name) + hardware_table.add_row("GPU VRAM", f"{hardware.gpu.vram_gb:.1f} GB") + hardware_table.add_row("GPU Count", str(hardware.gpu.count)) + + console.print(hardware_table) + + # Show recommended engine + if show_recommendations: + recommended = recommend_engine(hardware) + console.print(f"\n[bold]Recommended Engine:[/bold] [cyan]{recommended}[/cyan]") + + +def _show_config_template(console: Console, config_path: Path) -> None: + """Show default config template when config file doesn't exist.""" + from openjarvis.core.config import ( + DEFAULT_CONFIG_DIR, + detect_hardware, + generate_default_toml, + ) + + console.print(f"[yellow]Config file not found: {config_path}[/yellow]") + config_location = str(DEFAULT_CONFIG_DIR / "config.toml") + msg = f"Create one at {config_location} or use --path to specify a location." + console.print(f"[dim]{msg}[/dim]") + + console.print("\n[bold]Default Configuration Template:[/bold]") + hw = detect_hardware() + template = generate_default_toml(hw) + syntax = Syntax(template, "toml", theme="monokai", line_numbers=True) + console.print(Panel(syntax, border_style="dim")) + + +def _show_loaded_config(console: Console, config_path: Path, as_json: bool) -> None: + """Show the loaded effective configuration from config.toml.""" + from openjarvis.core.config import load_config + + console.print(f"[dim]Loading config from: {config_path}[/dim]") + + if config_path.exists(): + config = load_config(config_path) + + if as_json: + # Convert the dataclass to a dict and output as JSON + from dataclasses import fields, is_dataclass + + def convert(obj): + if obj is None: + return None + if isinstance(obj, (str, int, float, bool)): + return obj + if isinstance(obj, (list, tuple)): + return [convert(item) for item in obj] + if isinstance(obj, dict): + return {k: convert(v) for k, v in obj.items()} + if is_dataclass(obj): + result = {} + for field in fields(obj): + field_value = getattr(obj, field.name) + if field_value is not None: + result[field.name] = convert(field_value) + return result + # Fallback for other types + return str(obj) + + config_dict = convert(config) + # Write JSON to stdout so it is pipeable + stdout_console = Console() + stdout_console.print_json(json.dumps(config_dict, indent=2, default=str)) + else: + # Show as formatted table + console.print("[bold]Engine Configuration[/bold]") + console.print(f" Default Engine: [cyan]{config.engine.default}[/cyan]") + + console.print("\n[bold]Intelligence Configuration[/bold]") + console.print( + f" Default Model: [cyan]{config.intelligence.default_model}[/cyan]" + ) + fallback = config.intelligence.fallback_model or "N/A" + console.print(f" Fallback Model: [cyan]{fallback}[/cyan]") + console.print( + f" Temperature: [cyan]{config.intelligence.temperature}[/cyan]" + ) + console.print( + f" Max Tokens: [cyan]{config.intelligence.max_tokens}[/cyan]" + ) + + console.print("\n[bold]Agent Configuration[/bold]") + console.print(f" Default Agent: [cyan]{config.agent.default_agent}[/cyan]") + console.print(f" Max Turns: [cyan]{config.agent.max_turns}[/cyan]") + console.print(f" Tools: [cyan]{config.agent.tools or 'none'}[/cyan]") + ctx_mem = config.agent.context_from_memory + console.print(f" Context from Memory: [cyan]{ctx_mem}[/cyan]") + + # Show hardware info + _show_hardware_info(console) + + else: + _show_config_template(console, config_path) + + +def _show_toml_config(console: Console, config_path: Path) -> None: + """Show the raw TOML configuration file content with syntax highlighting.""" + console.print(f"[dim]Loading config from: {config_path}[/dim]") + + if config_path.exists(): + config_content = config_path.read_text() + syntax = Syntax(config_content, "toml", theme="monokai", line_numbers=True) + console.print(Panel(syntax, title="Config File", border_style="cyan")) + else: + _show_config_template(console, config_path) + + +def _show_json_config(console: Console, config_path: Path) -> None: + """Show the parsed TOML configuration as JSON.""" + console.print(f"[dim]Loading config from: {config_path}[/dim]") + + if config_path.exists(): + config_content = config_path.read_text() + + try: + import tomllib # Python 3.11+ + except ModuleNotFoundError: + import tomli as tomllib # type: ignore[no-redef] + + config_dict = tomllib.loads(config_content) + # Write JSON to stdout so it is pipeable + stdout_console = Console() + stdout_console.print_json(json.dumps(config_dict, indent=2)) + else: + _show_config_template(console, config_path) + + +def _show_hardware(console: Console) -> None: + """Show detected hardware information with recommended engine and model.""" + from openjarvis.core.config import ( + detect_hardware, + recommend_engine, + recommend_model, + ) + + hardware = detect_hardware() + _show_hardware_info(console, show_recommendations=False) + + # Show recommended engine + recommended_engine = recommend_engine(hardware) + console.print( + f"\n[bold]Recommended Engine:[/bold] [cyan]{recommended_engine}[/cyan]" + ) + + # Show recommended model + console.print("\n[bold]Model Recommendations[/bold]") + recommended_model = recommend_model(hardware, recommended_engine) + console.print(f" Recommended Model: [cyan]{recommended_model}[/cyan]") + + +# Nested group for show sub-commands +@click.group(invoke_without_command=True) +@click.option("--path", "-p", default=None, help="Explicit config file path") +@click.option("--json", "as_json", is_flag=True, default=False, help="Output as JSON") +@click.pass_context +def show_group(ctx: click.Context, path: str | None, as_json: bool) -> None: + """Show configuration details.""" + # Default to 'loaded' if no subcommand is invoked + if ctx.invoked_subcommand is None: + console = Console(stderr=True) + try: + config_path = _get_config_path(path) + _show_loaded_config(console, config_path, as_json) + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +@show_group.command() +@click.option("--path", "-p", default=None, help="Explicit config file path") +@click.option("--json", "as_json", is_flag=True, default=False, help="Output as JSON") +def loaded(path: str | None, as_json: bool) -> None: + """Show the loaded effective configuration from config.toml.""" + console = Console(stderr=True) + try: + config_path = _get_config_path(path) + _show_loaded_config(console, config_path, as_json) + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +@show_group.command() +@click.option("--path", "-p", default=None, help="Explicit config file path") +def toml(path: str | None) -> None: + """Show the raw TOML configuration file content with syntax highlighting.""" + console = Console(stderr=True) + try: + config_path = _get_config_path(path) + _show_toml_config(console, config_path) + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +@show_group.command("json") +@click.option("--path", "-p", default=None, help="Explicit config file path") +def as_json(path: str | None) -> None: + """Show the parsed TOML configuration as JSON.""" + console = Console(stderr=True) + try: + config_path = _get_config_path(path) + _show_json_config(console, config_path) + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +@show_group.command() +def hardware() -> None: + """Show detected hardware information with recommended engine and model.""" + console = Console(stderr=True) + try: + _show_hardware(console) + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +# Register the show group under config +config.add_command(show_group, "show") + + +__all__ = ["config"] diff --git a/src/openjarvis/cli/registry_cmd.py b/src/openjarvis/cli/registry_cmd.py new file mode 100644 index 00000000..9558ea77 --- /dev/null +++ b/src/openjarvis/cli/registry_cmd.py @@ -0,0 +1,146 @@ +"""``jarvis registry`` — registry inspection commands.""" + +from __future__ import annotations + +import click +from rich.console import Console +from rich.table import Table + + +def _load_registry_map() -> tuple[dict[str, object], dict[str, object]]: + """Import all registries and return (by_name, aliases) lookup dicts.""" + from openjarvis.core.registry import ( + AgentRegistry, + BenchmarkRegistry, + ChannelRegistry, + CompressionRegistry, + EngineRegistry, + LearningRegistry, + MemoryRegistry, + ModelRegistry, + RouterPolicyRegistry, + SkillRegistry, + SpeechRegistry, + ToolRegistry, + ) + + by_name = { + "ToolRegistry": ToolRegistry, + "AgentRegistry": AgentRegistry, + "EngineRegistry": EngineRegistry, + "MemoryRegistry": MemoryRegistry, + "ModelRegistry": ModelRegistry, + "ChannelRegistry": ChannelRegistry, + "LearningRegistry": LearningRegistry, + "SkillRegistry": SkillRegistry, + "BenchmarkRegistry": BenchmarkRegistry, + "RouterPolicyRegistry": RouterPolicyRegistry, + "SpeechRegistry": SpeechRegistry, + "CompressionRegistry": CompressionRegistry, + } + + aliases: dict[str, object] = {} + _alias_map = { + "ToolRegistry": ("tool", "tools"), + "AgentRegistry": ("agent", "agents"), + "EngineRegistry": ("engine", "engines"), + "MemoryRegistry": ("memory", "memories"), + "ModelRegistry": ("model", "models"), + "ChannelRegistry": ("channel", "channels"), + "LearningRegistry": ("learning", "learnings"), + "SkillRegistry": ("skill", "skills"), + "BenchmarkRegistry": ("benchmark", "benchmarks"), + "RouterPolicyRegistry": ("router", "routers"), + "SpeechRegistry": ("speech", "speeches"), + "CompressionRegistry": ("compression", "compressions"), + } + for class_name, cls in by_name.items(): + aliases[class_name] = cls + for alias in _alias_map[class_name]: + aliases[alias] = cls + + return by_name, aliases + + +@click.group() +def registry() -> None: + """Inspect registered components — list registries, show entries.""" + + +@registry.command("list") +def list_registries() -> None: + """List all available registries.""" + console = Console(stderr=True) + + table = Table(title="Available Registries") + table.add_column("Registry", style="cyan") + table.add_column("Module", style="green") + table.add_column("Entry Count", style="yellow") + + try: + by_name, _ = _load_registry_map() + except Exception as exc: + console.print(f"[red]Error loading registries: {exc}[/red]") + return + + module_path = "openjarvis.core.registry" + for reg_name, registry_cls in by_name.items(): + try: + count = len(registry_cls.keys()) + table.add_row(reg_name, module_path, str(count)) + except Exception as exc: + table.add_row(reg_name, module_path, f"[red]Error: {exc}[/red]") + + console.print(table) + + +@registry.command() +@click.argument("registry_name") +@click.option( + "--verbose", "-v", is_flag=True, default=False, help="Show full entry details" +) +def show(registry_name: str, verbose: bool) -> None: + """Show entries in a specific registry.""" + console = Console(stderr=True) + + try: + _, aliases = _load_registry_map() + + registry_cls = aliases.get(registry_name) + if registry_cls is None: + console.print(f"[red]Unknown registry: {registry_name}[/red]") + console.print( + "[dim]Run 'jarvis registry list' to see available registries.[/dim]" + ) + return + + keys = registry_cls.keys() + if not keys: + console.print(f"[dim]{registry_name} is empty.[/dim]") + return + + console.print(f"[bold]{registry_name}[/bold] — {len(keys)} entry/entries") + + if verbose: + for key in keys: + entry = registry_cls.get(key) + console.print(f"\n [cyan]{key}[/cyan]") + console.print(f" Type: {type(entry).__name__}") + console.print(f" Value: {entry}") + else: + table = Table() + table.add_column("Key", style="cyan") + table.add_column("Type", style="green") + table.add_column("Value", style="white", max_width=80) + for key in keys: + entry = registry_cls.get(key) + entry_type = type(entry).__name__ + entry_value = str(entry) + table.add_row(key, entry_type, entry_value) + console.print(table) + + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +__all__ = ["registry"] diff --git a/src/openjarvis/cli/tool_cmd.py b/src/openjarvis/cli/tool_cmd.py new file mode 100644 index 00000000..80960b38 --- /dev/null +++ b/src/openjarvis/cli/tool_cmd.py @@ -0,0 +1,107 @@ +"""``jarvis tool`` — tool management commands.""" + +from __future__ import annotations + +import click +from rich.console import Console +from rich.table import Table + + +@click.group() +def tool() -> None: + """Manage tools — list, inspect.""" + + +@tool.command("list") +def list_tools() -> None: + """List all registered tools with their descriptions.""" + console = Console(stderr=True) + try: + # Trigger tool registration by importing the tools module + import openjarvis.tools # noqa: F401 + from openjarvis.core.registry import ToolRegistry + + keys = sorted(ToolRegistry.keys()) + if not keys: + console.print("[dim]No tools registered.[/dim]") + return + + table = Table(title="Registered Tools") + table.add_column("Name", style="cyan") + table.add_column("Description", style="green", max_width=60) + table.add_column("Category", style="yellow") + + for key in keys: + tool_cls = ToolRegistry.get(key) + description = "" + category = "" + + # Try to get spec from the class or an instance + try: + # Some tools may require initialization, so try with default init + tool_instance = tool_cls() if callable(tool_cls) else tool_cls + if hasattr(tool_instance, "spec"): + spec = tool_instance.spec + description = getattr(spec, "description", "")[:60] + category = getattr(spec, "category", "") + except Exception: + # If instantiation fails, just show the key + description = "N/A (instantiation error)" + + table.add_row(key, description, category) + + console.print(table) + console.print(f"\n[dim]Total: {len(keys)} tool(s)[/dim]") + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +@tool.command() +@click.argument("tool_name") +def inspect(tool_name: str) -> None: + """Show detailed information about a specific tool.""" + console = Console(stderr=True) + try: + # Trigger tool registration by importing the tools module + import openjarvis.tools # noqa: F401 + from openjarvis.core.registry import ToolRegistry + + if not ToolRegistry.contains(tool_name): + console.print(f"[red]Tool not found: {tool_name}[/red]") + console.print("[dim]Run 'jarvis tool list' to see available tools.[/dim]") + return + + tool_cls = ToolRegistry.get(tool_name) + console.print(f"[bold]{tool_name}[/bold]") + + try: + tool_instance = tool_cls() if callable(tool_cls) else tool_cls + if hasattr(tool_instance, "spec"): + spec = tool_instance.spec + console.print(f" [cyan]Name:[/cyan] {getattr(spec, 'name', 'N/A')}") + console.print( + f" [cyan]Description:[/cyan] {getattr(spec, 'description', 'N/A')}" + ) + category = getattr(spec, "category", "N/A") or "none" + console.print(f" [cyan]Category:[/cyan] {category}") + + params = getattr(spec, "parameters", {}) + if params: + console.print(" [cyan]Parameters:[/cyan]") + if isinstance(params, dict) and "properties" in params: + for param_name, param_info in params["properties"].items(): + param_type = param_info.get("type", "any") + param_desc = param_info.get("description", "") + console.print( + f" • {param_name}: {param_type} — {param_desc}" + ) + else: + console.print(f" {params}") + except Exception as e: + console.print(f"[yellow]Note: Could not instantiate tool: {e}[/yellow]") + + except Exception as exc: + console.print(f"[red]Error: {exc}[/red]") + + +__all__ = ["tool"] diff --git a/tests/cli/test_config_cmd.py b/tests/cli/test_config_cmd.py new file mode 100644 index 00000000..d2e9d249 --- /dev/null +++ b/tests/cli/test_config_cmd.py @@ -0,0 +1,210 @@ +"""Tests for the ``jarvis config`` CLI commands.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from click.testing import CliRunner + +from openjarvis.cli import cli + + +class TestConfigCmd: + """Test cases for the jarvis config CLI group.""" + + def test_config_group_help(self) -> None: + """Test that the config group help displays correctly.""" + result = CliRunner().invoke(cli, ["config", "--help"]) + assert result.exit_code == 0 + assert "config" in result.output.lower() + + def test_config_show_help(self) -> None: + """Test that the config show help displays correctly.""" + result = CliRunner().invoke(cli, ["config", "show", "--help"]) + assert result.exit_code == 0 + assert "show" in result.output.lower() + + def test_config_show_loaded_help(self) -> None: + """Test that the config show loaded help displays correctly.""" + result = CliRunner().invoke(cli, ["config", "show", "loaded", "--help"]) + assert result.exit_code == 0 + assert "loaded" in result.output.lower() + + def test_config_show_toml_help(self) -> None: + """Test that the config show toml help displays correctly.""" + result = CliRunner().invoke(cli, ["config", "show", "toml", "--help"]) + assert result.exit_code == 0 + assert "toml" in result.output.lower() + + def test_config_show_json_help(self) -> None: + """Test that the config show json help displays correctly.""" + result = CliRunner().invoke(cli, ["config", "show", "json", "--help"]) + assert result.exit_code == 0 + assert "json" in result.output.lower() + + def test_config_show_hardware_help(self) -> None: + """Test that the config show hardware help displays correctly.""" + result = CliRunner().invoke(cli, ["config", "show", "hardware", "--help"]) + assert result.exit_code == 0 + assert "hardware" in result.output.lower() + + def test_config_show_loaded_displays_config(self, tmp_path: Path) -> None: + """Test that config show loaded displays the configuration.""" + # Create a temporary config file + config_file = tmp_path / "test_config.toml" + config_file.write_text( + """ +[engine] +default = "ollama" + +[intelligence] +default_model = "test-model" +temperature = 0.7 +max_tokens = 1024 + +[agent] +default_agent = "simple" +max_turns = 5 +""" + ) + + result = CliRunner().invoke( + cli, ["config", "show", "loaded", "--path", str(config_file)] + ) + + assert result.exit_code == 0 + assert "ollama" in result.output + assert "test-model" in result.output + + def test_config_show_loaded_json_output(self, tmp_path: Path) -> None: + """Test that config show loaded --json outputs valid JSON.""" + # Create a temporary config file + config_file = tmp_path / "test_config.toml" + config_file.write_text( + """ +[engine] +default = "ollama" + +[intelligence] +default_model = "test-model" +temperature = 0.7 +""" + ) + + result = CliRunner().invoke( + cli, ["config", "show", "loaded", "--path", str(config_file), "--json"] + ) + + assert result.exit_code == 0 + # Try to parse the output as JSON (may have prefix lines) + # Find the JSON part by looking for the opening brace + json_start = result.output.find("{") + assert json_start >= 0, f"Could not find JSON in output: {result.output}" + try: + json_data = json.loads(result.output[json_start:]) + assert "engine" in json_data + assert json_data["engine"]["default"] == "ollama" + except json.JSONDecodeError: + pytest.fail(f"Output is not valid JSON: {result.output}") + + def test_config_show_toml_displays_raw_content(self, tmp_path: Path) -> None: + """Test that config show toml displays the raw TOML content.""" + # Create a temporary config file + config_file = tmp_path / "test_config.toml" + config_file.write_text('[engine]\ndefault = "ollama"\n') + + result = CliRunner().invoke( + cli, ["config", "show", "toml", "--path", str(config_file)] + ) + + assert result.exit_code == 0 + assert "[engine]" in result.output + assert "ollama" in result.output + + def test_config_show_json_displays_parsed_content(self, tmp_path: Path) -> None: + """Test that config show json displays parsed TOML as JSON.""" + # Create a temporary config file + config_file = tmp_path / "test_config.toml" + config_file.write_text('[engine]\ndefault = "ollama"\n') + + result = CliRunner().invoke( + cli, ["config", "show", "json", "--path", str(config_file)] + ) + + assert result.exit_code == 0 + # The output should be valid JSON (may have prefix lines) + json_start = result.output.find("{") + assert json_start >= 0, f"Could not find JSON in output: {result.output}" + try: + json_data = json.loads(result.output[json_start:]) + assert json_data["engine"]["default"] == "ollama" + except json.JSONDecodeError: + pytest.fail(f"Output is not valid JSON: {result.output}") + + def test_config_show_hardware_displays_info(self) -> None: + """Test that config show hardware displays hardware information.""" + result = CliRunner().invoke(cli, ["config", "show", "hardware"]) + assert result.exit_code == 0 + # Should display hardware-related content + output = result.output.lower() + assert "hardware" in output or "cpu" in output or "ram" in output + + def test_config_show_default_to_loaded(self, tmp_path: Path) -> None: + """Test that config show (no subcommand) defaults to loaded.""" + # Create a temporary config file + config_file = tmp_path / "test_config.toml" + config_file.write_text('[engine]\ndefault = "ollama"\n') + + result = CliRunner().invoke(cli, ["config", "show", "--path", str(config_file)]) + + assert result.exit_code == 0 + # Default behavior should show loaded config + assert "ollama" in result.output + + def test_config_show_no_config_file(self, tmp_path: Path) -> None: + """Test that config show handles missing config file gracefully.""" + # Create a path for a non-existent config file + config_file = tmp_path / "nonexistent_config.toml" + + result = CliRunner().invoke( + cli, ["config", "show", "toml", "--path", str(config_file)] + ) + + # Should exit 0 and show a message about missing config + assert result.exit_code == 0 + assert "not found" in result.output.lower() or "config" in result.output.lower() + + def test_config_show_path_option(self, tmp_path: Path) -> None: + """Test that the --path option works for all subcommands.""" + # Create a custom config file + custom_config = tmp_path / "custom.toml" + custom_config.write_text('[engine]\ndefault = "custom-engine"\n') + + # Test various subcommands with custom path + result = CliRunner().invoke( + cli, ["config", "show", "toml", "--path", str(custom_config)] + ) + assert result.exit_code == 0 + assert "custom-engine" in result.output + + result = CliRunner().invoke( + cli, ["config", "show", "json", "--path", str(custom_config)] + ) + assert result.exit_code == 0 + assert "custom-engine" in result.output + + def test_config_show_invalid_subcommand(self) -> None: + """Test that an invalid subcommand shows an error.""" + result = CliRunner().invoke(cli, ["config", "show", "invalid_subcommand_xyz"]) + # Should show error about unknown command + assert result.exit_code != 0 + assert "no such" in result.output.lower() or "unknown" in result.output.lower() + + def test_config_subcommands_not_available_standalone(self) -> None: + """Test that config subcommands are not available directly under config.""" + # These should not be available as direct children of config + result = CliRunner().invoke(cli, ["config", "loaded"]) + assert result.exit_code != 0 + assert "no such" in result.output.lower() or "unknown" in result.output.lower() diff --git a/tests/cli/test_registry_cmd.py b/tests/cli/test_registry_cmd.py new file mode 100644 index 00000000..17e2a848 --- /dev/null +++ b/tests/cli/test_registry_cmd.py @@ -0,0 +1,154 @@ +"""Tests for the ``jarvis registry`` CLI commands.""" + +from __future__ import annotations + +from click.testing import CliRunner + +from openjarvis.cli import cli +from openjarvis.core.registry import ( + ToolRegistry, +) + + +class TestRegistryCmd: + """Test cases for the jarvis registry CLI group.""" + + def test_registry_group_help(self) -> None: + """Test that the registry group help displays correctly.""" + result = CliRunner().invoke(cli, ["registry", "--help"]) + assert result.exit_code == 0 + assert "registry" in result.output.lower() + + def test_registry_list_help(self) -> None: + """Test that the registry list help displays correctly.""" + result = CliRunner().invoke(cli, ["registry", "list", "--help"]) + assert result.exit_code == 0 + + def test_registry_show_help(self) -> None: + """Test that the registry show help displays correctly.""" + result = CliRunner().invoke(cli, ["registry", "show", "--help"]) + assert result.exit_code == 0 + + def test_registry_list_shows_all_registries(self) -> None: + """Test that registry list displays all available registries.""" + result = CliRunner().invoke(cli, ["registry", "list"]) + assert result.exit_code == 0 + # Should show registry-related content + output = result.output.lower() + assert "registry" in output + + def test_registry_show_unknown_registry(self) -> None: + """Test that showing an unknown registry shows an error.""" + result = CliRunner().invoke(cli, ["registry", "show", "unknown_registry_xyz"]) + assert result.exit_code == 0 # CLI still exits 0, just shows error message + assert ( + "unknown" in result.output.lower() or "not found" in result.output.lower() + ) + + def test_registry_show_tool_registry(self) -> None: + """Test that showing the tool registry displays entries.""" + # Trigger tool registration + import openjarvis.tools # noqa: F401 + + result = CliRunner().invoke(cli, ["registry", "show", "tool"]) + assert result.exit_code == 0 + # Should show tool-related content + assert "tool" in result.output.lower() or "key" in result.output.lower() + + def test_registry_show_tool_registry_verbose(self) -> None: + """Test that showing the tool registry with verbose flag shows details.""" + # Trigger tool registration + import openjarvis.tools # noqa: F401 + + result = CliRunner().invoke(cli, ["registry", "show", "tool", "-v"]) + assert result.exit_code == 0 + + def test_registry_show_empty_registry(self) -> None: + """Test registry show behavior with an empty registry.""" + result = CliRunner().invoke(cli, ["registry", "show", "agent"]) + assert result.exit_code == 0 + # Should handle empty registries gracefully + assert "agent" in result.output.lower() or "empty" in result.output.lower() + + def test_registry_show_accepts_aliases(self) -> None: + """Test that registry show accepts various aliases.""" + # Trigger tool registration + import openjarvis.tools # noqa: F401 + + # Test with 'tools' alias + result = CliRunner().invoke(cli, ["registry", "show", "tools"]) + assert result.exit_code == 0 + + def test_registry_keys_match_actual_registries(self) -> None: + """Test that the list command shows all expected registry names.""" + result = CliRunner().invoke(cli, ["registry", "list"]) + assert result.exit_code == 0 + output = result.output + + # Check that key registries are mentioned + assert "ToolRegistry" in output or "tool" in output.lower() + assert "EngineRegistry" in output or "engine" in output.lower() + assert "MemoryRegistry" in output or "memory" in output.lower() + assert "ChannelRegistry" in output or "channel" in output.lower() + + def test_registry_show_nonexistent_key(self) -> None: + """Test that showing a nonexistent key in a registry is handled.""" + result = CliRunner().invoke(cli, ["registry", "show", "nonexistent"]) + assert result.exit_code == 0 + assert ( + "unknown" in result.output.lower() or "not found" in result.output.lower() + ) + + def test_registry_list_handles_import_error(self) -> None: + """Test that registry list handles import errors gracefully.""" + # This tests the exception handling path in list_registries + result = CliRunner().invoke(cli, ["registry", "list"]) + assert result.exit_code == 0 + # Should still complete even if some registries have issues + + def test_registry_show_handles_exception(self) -> None: + """Test that registry show handles exceptions gracefully.""" + # Patch the registry to raise an exception during keys() call + from unittest.mock import patch + + with patch( + "openjarvis.core.registry.ToolRegistry.keys", + side_effect=Exception("Test error"), + ): + result = CliRunner().invoke(cli, ["registry", "show", "tool"]) + assert result.exit_code == 0 + assert "error" in result.output.lower() + + def test_registry_list_handles_registry_error(self) -> None: + """Test that registry list handles errors for specific registries.""" + from unittest.mock import patch + + # Make one of the registry classes raise an error during keys() + with patch.object(ToolRegistry, "keys", side_effect=Exception("Import error")): + result = CliRunner().invoke(cli, ["registry", "list"]) + assert result.exit_code == 0 + # Should still complete and show other registries + + def test_registry_show_handles_error_during_import(self) -> None: + """Test that registry show handles exceptions during import.""" + from unittest.mock import patch + + # Patch the entire import to raise an exception + with patch.dict("sys.modules", {"openjarvis.core.registry": None}): + # This tests the outer exception handler + result = CliRunner().invoke(cli, ["registry", "show", "tool"]) + assert result.exit_code == 0 + + def test_registry_show_handles_error_during_iteration(self) -> None: + """Test that registry show handles exceptions during entry iteration.""" + from unittest.mock import patch + + # Patch keys to return a fake entry so iteration happens, + # then patch get to raise + with patch.object(ToolRegistry, "keys", return_value=["fake_tool"]): + with patch.object( + ToolRegistry, "get", side_effect=Exception("Iteration error") + ): + result = CliRunner().invoke(cli, ["registry", "show", "tool"]) + assert result.exit_code == 0 + assert "error" in result.output.lower() diff --git a/tests/cli/test_tool_cmd.py b/tests/cli/test_tool_cmd.py new file mode 100644 index 00000000..fbb8351c --- /dev/null +++ b/tests/cli/test_tool_cmd.py @@ -0,0 +1,238 @@ +"""Tests for the ``jarvis tool`` CLI commands.""" + +from __future__ import annotations + +from click.testing import CliRunner + +from openjarvis.cli import cli +from openjarvis.core.registry import ToolRegistry + + +class TestToolCmd: + """Test cases for the jarvis tool CLI group.""" + + def test_tool_group_help(self) -> None: + """Test that the tool group help displays correctly.""" + result = CliRunner().invoke(cli, ["tool", "--help"]) + assert result.exit_code == 0 + assert "tool" in result.output.lower() or "list" in result.output + + def test_tool_list_help(self) -> None: + """Test that the tool list help displays correctly.""" + result = CliRunner().invoke(cli, ["tool", "list", "--help"]) + assert result.exit_code == 0 + assert "list" in result.output.lower() or "registered" in result.output.lower() + + def test_tool_inspect_help(self) -> None: + """Test that the tool inspect help displays correctly.""" + result = CliRunner().invoke(cli, ["tool", "inspect", "--help"]) + assert result.exit_code == 0 + assert "tool" in result.output.lower() or "inspect" in result.output.lower() + + def test_tool_list_shows_registered_tools(self) -> None: + """Test that tool list displays registered tools.""" + result = CliRunner().invoke(cli, ["tool", "list"]) + assert result.exit_code == 0 + # Should show at least some tools if they're registered + output = result.output.lower() + # The output should contain a table header or tool-related content + assert "tool" in output or "name" in output or "description" in output + + def test_tool_inspect_unknown_tool(self) -> None: + """Test that inspecting an unknown tool shows an error.""" + result = CliRunner().invoke(cli, ["tool", "inspect", "nonexistent_tool_xyz"]) + assert result.exit_code == 0 # CLI still exits 0, just shows error message + assert "not found" in result.output.lower() or "error" in result.output.lower() + + def test_tool_inspect_known_tool(self) -> None: + """Test that inspecting a known tool shows details.""" + # First, trigger tool registration + import openjarvis.tools # noqa: F401 + + # Get a known tool name + registered_tools = ToolRegistry.keys() + if registered_tools: + tool_name = registered_tools[0] + result = CliRunner().invoke(cli, ["tool", "inspect", tool_name]) + assert result.exit_code == 0 + assert tool_name in result.output + + def test_tool_list_empty_registry(self) -> None: + """Test tool list behavior with empty registry.""" + # This test verifies the command runs even if no tools are registered + result = CliRunner().invoke(cli, ["tool", "list"]) + assert result.exit_code == 0 + + def test_tool_inspect_requires_tool_name(self) -> None: + """Test that inspect command requires a tool name argument.""" + # Running inspect without a tool name should show help or error + result = CliRunner().invoke(cli, ["tool", "inspect"]) + # Either shows help (exit 0) or error (exit non-zero) + assert result.exit_code in (0, 2) # 0 for help, 2 for missing argument + + def test_tool_list_with_registered_tools(self) -> None: + """Test that tool list shows details for registered tools.""" + # Trigger tool registration + import openjarvis.tools # noqa: F401 + + result = CliRunner().invoke(cli, ["tool", "list"]) + assert result.exit_code == 0 + # Should either show tools or indicate no tools are registered + output = result.output + assert ( + "Registered Tools" in output + or "No tools registered" in output + or "Total:" in output + ) + + def test_tool_list_handles_instantiation_error(self) -> None: + """Test that tool list handles tool instantiation errors gracefully.""" + from unittest.mock import patch + + # Mock a tool class that raises an exception during instantiation + with patch.object(ToolRegistry, "keys", return_value=["mock_tool"]): + with patch.object(ToolRegistry, "get", return_value=object): + result = CliRunner().invoke(cli, ["tool", "list"]) + assert result.exit_code == 0 + # Should still complete even if tools have instantiation issues + assert "mock_tool" in result.output or "Total:" in result.output + + def test_tool_inspect_with_spec_details(self) -> None: + """Test that inspect shows full spec details for tools with specs.""" + import openjarvis.tools # noqa: F401 + + registered_tools = ToolRegistry.keys() + if registered_tools: + tool_name = registered_tools[0] + result = CliRunner().invoke(cli, ["tool", "inspect", tool_name]) + assert result.exit_code == 0 + # Should show tool details including name, description, category + output = result.output + assert tool_name in output + + def test_tool_inspect_handles_instantiation_error(self) -> None: + """Test that inspect handles tool instantiation errors gracefully.""" + from unittest.mock import patch + + # Mock a tool that exists but fails to instantiate + with patch.object(ToolRegistry, "contains", return_value=True): + with patch.object(ToolRegistry, "get", return_value=object): + result = CliRunner().invoke(cli, ["tool", "inspect", "mock_tool"]) + assert result.exit_code == 0 + # Should show error note but still complete + assert "mock_tool" in result.output + + def test_tool_list_catches_registry_exception(self) -> None: + """Test that tool list handles exceptions during registry access.""" + from unittest.mock import patch + + # Mock ToolRegistry.keys to raise an exception + with patch.object( + ToolRegistry, "keys", side_effect=Exception("Registry error") + ): + result = CliRunner().invoke(cli, ["tool", "list"]) + assert result.exit_code == 0 + # Should catch exception and display error message + assert "error" in result.output.lower() + + def test_tool_inspect_catches_registry_exception(self) -> None: + """Test that tool inspect handles exceptions during registry access.""" + from unittest.mock import patch + + # Mock ToolRegistry.contains to raise an exception + with patch.object( + ToolRegistry, "contains", side_effect=Exception("Registry error") + ): + result = CliRunner().invoke(cli, ["tool", "inspect", "mock_tool"]) + assert result.exit_code == 0 + # Should catch exception and display error message + assert "error" in result.output.lower() + + def test_tool_list_shows_tool_spec(self) -> None: + """Test that tool list displays spec details when available.""" + from unittest.mock import MagicMock, patch + + # Create a mock tool class with a spec + mock_spec = MagicMock() + mock_spec.description = "Test tool description" + mock_spec.category = "test-category" + + mock_tool_cls = MagicMock() + mock_tool_instance = MagicMock() + mock_tool_instance.spec = mock_spec + mock_tool_cls.return_value = mock_tool_instance + + with patch.object(ToolRegistry, "keys", return_value=["mock_tool"]): + with patch.object(ToolRegistry, "get", return_value=mock_tool_cls): + result = CliRunner().invoke(cli, ["tool", "list"]) + assert result.exit_code == 0 + # Should show spec details + output = result.output + assert "mock_tool" in output + assert "Test tool description" in output + + def test_tool_inspect_shows_full_spec_details(self) -> None: + """Test that inspect displays full spec details including parameters.""" + from unittest.mock import MagicMock, patch + + # Create a mock tool with full spec including parameters + mock_spec = MagicMock() + mock_spec.name = "MockTool" + mock_spec.description = "A mock tool for testing" + mock_spec.category = "testing" + mock_spec.parameters = { + "properties": { + "input": {"type": "string", "description": "Input parameter"}, + "count": {"type": "integer", "description": "Count value"}, + } + } + + mock_tool_cls = MagicMock() + mock_tool_instance = MagicMock() + mock_tool_instance.spec = mock_spec + mock_tool_cls.return_value = mock_tool_instance + + with patch.object(ToolRegistry, "contains", return_value=True): + with patch.object(ToolRegistry, "get", return_value=mock_tool_cls): + result = CliRunner().invoke(cli, ["tool", "inspect", "mock_tool"]) + assert result.exit_code == 0 + output = result.output + assert "mock_tool" in output + assert "MockTool" in output + assert "A mock tool for testing" in output + assert "testing" in output + assert "Parameters" in output + assert "input" in output + assert "count" in output + + def test_tool_inspect_handles_tool_without_spec(self) -> None: + """Test that inspect handles tools without spec attribute.""" + from unittest.mock import MagicMock, patch + + # Mock a tool class without spec attribute + mock_tool_cls = MagicMock() + del mock_tool_cls.spec # Remove spec attribute + + with patch.object(ToolRegistry, "contains", return_value=True): + with patch.object(ToolRegistry, "get", return_value=mock_tool_cls): + result = CliRunner().invoke(cli, ["tool", "inspect", "mock_tool"]) + assert result.exit_code == 0 + # Should handle gracefully + assert "mock_tool" in result.output + + def test_tool_list_handles_tool_without_spec(self) -> None: + """Test that tool list handles tools without spec attribute.""" + from unittest.mock import patch + + # Create a simple class without spec attribute + class ToolWithoutSpec: + pass + + mock_tool_cls = ToolWithoutSpec + + with patch.object(ToolRegistry, "keys", return_value=["mock_tool"]): + with patch.object(ToolRegistry, "get", return_value=mock_tool_cls): + result = CliRunner().invoke(cli, ["tool", "list"]) + assert result.exit_code == 0 + # Should handle gracefully and show the tool name + assert "mock_tool" in result.output or "Total" in result.output From 16f601c7d7a16bfc27e2942fbdebf9e1cfddac4d Mon Sep 17 00:00:00 2001 From: mattiasghodsian Date: Mon, 23 Mar 2026 09:53:13 +0100 Subject: [PATCH 33/92] feat: NVIDIA GPU Docker support with compose override (#93) * feat: nvidia gpu config for docker * refactor: split NVIDIA GPU support into compose override Address review feedback: - Revert --engine ollama from base Dockerfile CMD (breaks non-Ollama users) - Keep bookworm pin and OLLAMA_HOST fix in base config - Move GPU-specific config (/proc, /sys mounts, deploy.resources.reservations) into new docker-compose.gpu.nvidia.yml override, matching the existing ROCm pattern (docker-compose.gpu.rocm.yml) - Restore Ollama port to standard 11434 - Remove commented-out GPU blocks from base docker-compose.yml - Add Ollama healthcheck with depends_on condition to base compose - Remove run.sh from repo root - Rewrite README Docker section to document CPU, NVIDIA, and ROCm patterns Co-Authored-By: Claude Opus 4.6 (1M context) --------- Co-authored-by: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) --- README.md | 22 +++++++++++++++ deploy/docker/Dockerfile | 4 +-- deploy/docker/docker-compose.gpu.nvidia.yml | 31 +++++++++++++++++++++ deploy/docker/docker-compose.yml | 15 ++++++---- 4 files changed, 65 insertions(+), 7 deletions(-) create mode 100644 deploy/docker/docker-compose.gpu.nvidia.yml diff --git a/README.md b/README.md index 39f3f0a5..0b885e24 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,28 @@ uv run jarvis doctor `jarvis init` auto-detects your hardware and recommends the best engine. After init, it prints engine-specific next steps. Run `uv run jarvis doctor` at any time to diagnose configuration or connectivity issues. +## Docker + +A Docker Compose configuration bundles Jarvis with Ollama: + +```bash +# CPU-only (default) +docker compose -f deploy/docker/docker-compose.yml up -d + +# NVIDIA GPU (requires NVIDIA Container Toolkit) +docker compose -f deploy/docker/docker-compose.yml \ + -f deploy/docker/docker-compose.gpu.nvidia.yml up -d + +# AMD GPU (requires ROCm) +docker compose -f deploy/docker/docker-compose.yml \ + -f deploy/docker/docker-compose.gpu.rocm.yml up -d + +# Pull a model into Ollama +docker compose -f deploy/docker/docker-compose.yml exec ollama ollama pull qwen3:8b +``` + +Services: Jarvis on `:8000`, Ollama on `:11434`. + ## Development From source, you need to make sure Rust is installed on System: diff --git a/deploy/docker/Dockerfile b/deploy/docker/Dockerfile index 4f5b8291..6e1f1e50 100644 --- a/deploy/docker/Dockerfile +++ b/deploy/docker/Dockerfile @@ -8,7 +8,7 @@ COPY frontend/ . RUN npm run build # Stage 2: Build Python package -FROM python:3.12-slim AS builder +FROM python:3.12-slim-bookworm AS builder WORKDIR /app COPY pyproject.toml README.md ./ @@ -21,7 +21,7 @@ RUN pip install --no-cache-dir uv && \ uv pip install --system ".[server]" # Stage 3: Runtime -FROM python:3.12-slim +FROM python:3.12-slim-bookworm COPY --from=builder /usr/local /usr/local COPY --from=builder /app /app diff --git a/deploy/docker/docker-compose.gpu.nvidia.yml b/deploy/docker/docker-compose.gpu.nvidia.yml new file mode 100644 index 00000000..22dcb450 --- /dev/null +++ b/deploy/docker/docker-compose.gpu.nvidia.yml @@ -0,0 +1,31 @@ +# NVIDIA GPU override — use with: +# docker compose -f deploy/docker/docker-compose.yml -f deploy/docker/docker-compose.gpu.nvidia.yml up + +services: + jarvis: + build: + context: ../.. + dockerfile: deploy/docker/Dockerfile.gpu + volumes: + - /proc:/proc:ro + - /sys:/sys:ro + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] + + ollama: + image: ollama/ollama:latest + environment: + - NVIDIA_VISIBLE_DEVICES=all + - NVIDIA_DRIVER_CAPABILITIES=compute,utility + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] diff --git a/deploy/docker/docker-compose.yml b/deploy/docker/docker-compose.yml index a19268ef..284f0897 100644 --- a/deploy/docker/docker-compose.yml +++ b/deploy/docker/docker-compose.yml @@ -1,5 +1,3 @@ -version: "3.9" - services: jarvis: build: @@ -9,17 +7,24 @@ services: - "8000:8000" environment: - OPENJARVIS_ENGINE_DEFAULT=ollama - - OPENJARVIS_OLLAMA_HOST=http://ollama:11434 + - OLLAMA_HOST=http://ollama:11434 depends_on: - - ollama + ollama: + condition: service_healthy restart: unless-stopped ollama: - image: ollama/ollama + image: ollama/ollama:latest ports: - "11434:11434" volumes: - ollama-models:/root/.ollama + healthcheck: + test: ["CMD", "ollama", "list"] + interval: 5s + timeout: 3s + retries: 10 + start_period: 10s restart: unless-stopped volumes: From f1aaed089573ab1cf55fb60229049c39ac8855a6 Mon Sep 17 00:00:00 2001 From: Jana Bergant Date: Mon, 23 Mar 2026 14:36:19 +0100 Subject: [PATCH 34/92] fix: wire TraceStore into AgentExecutor when traces.enabled is true SystemBuilder.build() constructed AgentExecutor without a trace_store, so traces were never recorded even when config.traces.enabled = true. The traces.enabled flag was parsed but never read during executor construction. Now reads config.traces.enabled and creates a TraceStore instance that is passed to AgentExecutor, enabling trace recording for jarvis agents ask/run and the agent scheduler daemon. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/system.py | 128 ++++++++++++++++++++++++++------------- 1 file changed, 86 insertions(+), 42 deletions(-) diff --git a/src/openjarvis/system.py b/src/openjarvis/system.py index 023c4671..2764323a 100644 --- a/src/openjarvis/system.py +++ b/src/openjarvis/system.py @@ -84,7 +84,10 @@ class JarvisSystem: max_context_tokens=self.config.memory.context_max_tokens, ) messages = inject_context( - query, messages, self.memory_backend, config=ctx_cfg, + query, + messages, + self.memory_backend, + config=ctx_cfg, ) except Exception as exc: logger.warning("Failed to inject memory context: %s", exc) @@ -93,14 +96,22 @@ class JarvisSystem: use_agent = agent or self.agent_name if use_agent and use_agent != "none": return self._run_agent( - query, messages, use_agent, tools, temperature, max_tokens, - system_prompt=system_prompt, operator_id=operator_id, + query, + messages, + use_agent, + tools, + temperature, + max_tokens, + system_prompt=system_prompt, + operator_id=operator_id, ) # Direct engine mode result = self.engine.generate( - messages, model=self.model, - temperature=temperature, max_tokens=max_tokens, + messages, + model=self.model, + temperature=temperature, + max_tokens=max_tokens, ) return { "content": result.get("content", ""), @@ -110,8 +121,16 @@ class JarvisSystem: } def _run_agent( - self, query, messages, agent_name, tool_names, temperature, max_tokens, - *, system_prompt=None, operator_id=None, + self, + query, + messages, + agent_name, + tool_names, + temperature, + max_tokens, + *, + system_prompt=None, + operator_id=None, ) -> Dict[str, Any]: """Run through an agent.""" from openjarvis.agents._stubs import AgentContext @@ -202,29 +221,22 @@ class JarvisSystem: "ttft": telemetry_events[0].get("ttft", 0.0), "energy_joules": total_energy, "power_watts": ( - sum(power_vals) / len(power_vals) - if power_vals else 0.0 + sum(power_vals) / len(power_vals) if power_vals else 0.0 ), "gpu_utilization_pct": ( - sum(util_vals) / len(util_vals) - if util_vals else 0.0 + sum(util_vals) / len(util_vals) if util_vals else 0.0 ), "throughput_tok_per_sec": ( sum(throughput_vals) / len(throughput_vals) - if throughput_vals else 0.0 + if throughput_vals + else 0.0 ), "gpu_memory_used_gb": max( - ( - e.get("gpu_memory_used_gb", 0.0) - for e in telemetry_events - ), + (e.get("gpu_memory_used_gb", 0.0) for e in telemetry_events), default=0.0, ), "gpu_temperature_c": max( - ( - e.get("gpu_temperature_c", 0.0) - for e in telemetry_events - ), + (e.get("gpu_temperature_c", 0.0) for e in telemetry_events), default=0.0, ), "inference_calls": len(telemetry_events), @@ -288,6 +300,7 @@ class JarvisSystem: if self.session_store is None: from pathlib import Path + self.session_store = SessionStore( db_path=Path(self.config.sessions.db_path).expanduser(), max_age_hours=self.config.sessions.max_age_hours, @@ -317,7 +330,8 @@ class JarvisSystem: try: if _system.agent_name and _system.agent_name != "none": result = _system.ask( - cm.content, context=False, + cm.content, + context=False, agent=_system.agent_name, ) reply = result.get("content", "") @@ -330,10 +344,16 @@ class JarvisSystem: try: _system.session_store.save_message( - session.session_id, "user", cm.content, channel=cm.channel, + session.session_id, + "user", + cm.content, + channel=cm.channel, ) _system.session_store.save_message( - session.session_id, "assistant", reply, channel=cm.channel, + session.session_id, + "assistant", + reply, + channel=cm.channel, ) except Exception: logger.debug("Session save error", exc_info=True) @@ -341,7 +361,9 @@ class JarvisSystem: if reply: try: channel_bridge.send( - cm.channel, reply, conversation_id=cm.conversation_id, + cm.channel, + reply, + conversation_id=cm.conversation_id, ) except Exception: logger.exception("Channel send error") @@ -470,8 +492,7 @@ class SystemBuilder: # Compute telemetry_enabled once telemetry_enabled = ( - self._telemetry if self._telemetry is not None - else config.telemetry.enabled + self._telemetry if self._telemetry is not None else config.telemetry.enabled ) gpu_monitor = None energy_monitor = None @@ -503,6 +524,7 @@ class SystemBuilder: # Apply security guardrails FIRST (innermost wrapper) from openjarvis.security import setup_security + sec = setup_security(config, engine, bus) engine = sec.engine @@ -513,7 +535,8 @@ class SystemBuilder: ) engine = InstrumentedEngine( - engine, bus, + engine, + bus, gpu_monitor=gpu_monitor, energy_monitor=energy_monitor, ) @@ -531,7 +554,11 @@ class SystemBuilder: # Resolve tools tool_list = self._resolve_tools( - config, engine, model, memory_backend, channel_backend, + config, + engine, + model, + memory_backend, + channel_backend, ) # Build tool executor @@ -581,9 +608,23 @@ class SystemBuilder: from openjarvis.agents.executor import AgentExecutor from openjarvis.agents.scheduler import AgentScheduler + # Wire TraceStore into executor when tracing is enabled + _trace_store = None + if config.traces.enabled: + try: + from openjarvis.traces.store import TraceStore + + _trace_store = TraceStore(config.traces.db_path) + except Exception: + logger.warning( + "Failed to initialize TraceStore", + exc_info=True, + ) + agent_executor = AgentExecutor( manager=agent_manager, event_bus=bus, + trace_store=_trace_store, ) agent_scheduler = AgentScheduler( manager=agent_manager, @@ -598,6 +639,7 @@ class SystemBuilder: if speech_enabled: try: from openjarvis.speech._discovery import get_speech_backend + speech_backend = get_speech_backend(config) except Exception as exc: logger.warning("Failed to initialize speech backend: %s", exc) @@ -814,8 +856,9 @@ class SystemBuilder: logger.warning("Failed to resolve channel backend %r: %s", key, exc) return None - def _resolve_tools(self, config, engine, model, memory_backend, - channel_backend=None): + def _resolve_tools( + self, config, engine, model, memory_backend, channel_backend=None + ): """Resolve tool instances via MCPServer (primary) + external MCP servers.""" from openjarvis.mcp.server import MCPServer @@ -851,6 +894,7 @@ class SystemBuilder: if config.tools.mcp.servers: try: import json + server_list = json.loads(config.tools.mcp.servers) if isinstance(server_list, list): for server_cfg in server_list: @@ -858,13 +902,15 @@ class SystemBuilder: external_tools = self._discover_external_mcp(server_cfg) if tool_names: external_tools = [ - t for t in external_tools + t + for t in external_tools if t.spec.name in tool_names ] tools.extend(external_tools) except Exception as exc: logger.warning( - "Failed to discover external MCP tools: %s", exc, + "Failed to discover external MCP tools: %s", + exc, ) except (json.JSONDecodeError, TypeError) as exc: logger.warning("Failed to parse MCP server config: %s", exc) @@ -890,8 +936,10 @@ class SystemBuilder: if hasattr(tool, "_channel"): tool._channel = channel_backend elif name in ( - "schedule_task", "list_scheduled_tasks", - "pause_scheduled_task", "resume_scheduled_task", + "schedule_task", + "list_scheduled_tasks", + "pause_scheduled_task", + "resume_scheduled_task", "cancel_scheduled_task", ): pass # scheduler injection handled post-build @@ -899,8 +947,7 @@ class SystemBuilder: def _setup_sandbox(self, config): """Set up container sandbox runner if enabled.""" sandbox_enabled = ( - self._sandbox if self._sandbox is not None - else config.sandbox.enabled + self._sandbox if self._sandbox is not None else config.sandbox.enabled ) if not sandbox_enabled: return None @@ -921,8 +968,7 @@ class SystemBuilder: def _setup_scheduler(self, config, bus): """Set up task scheduler if enabled.""" scheduler_enabled = ( - self._scheduler if self._scheduler is not None - else config.scheduler.enabled + self._scheduler if self._scheduler is not None else config.scheduler.enabled ) if not scheduler_enabled: return None, None @@ -954,8 +1000,7 @@ class SystemBuilder: def _setup_workflow(self, config, bus): """Set up workflow engine if enabled.""" workflow_enabled = ( - self._workflow if self._workflow is not None - else config.workflow.enabled + self._workflow if self._workflow is not None else config.workflow.enabled ) if not workflow_enabled: return None @@ -974,8 +1019,7 @@ class SystemBuilder: def _setup_sessions(self, config): """Set up session store if enabled.""" sessions_enabled = ( - self._sessions if self._sessions is not None - else config.sessions.enabled + self._sessions if self._sessions is not None else config.sessions.enabled ) if not sessions_enabled: return None From b7ad2d2c12d78339e277f1f6fee25e5438e2affa Mon Sep 17 00:00:00 2001 From: "manuel.richarz" Date: Tue, 24 Mar 2026 14:56:58 +0100 Subject: [PATCH 35/92] feat: add SSE streaming support for managed agent messages Add `stream: bool` parameter to `POST /v1/managed-agents/{id}/messages`. When `stream=true`, the agent processes the message synchronously and returns an SSE stream (OpenAI-compatible format) with token-by-token response, tool result events, and usage metadata. This enables real-time voice assistants and chat UIs to receive agent responses as they are generated, rather than polling for completion. - Extend SendMessageRequest with `stream` field (default: false) - Add _stream_managed_agent() helper using asyncio.to_thread() - Build AgentContext from conversation history for multi-turn support - Emit tool_results as named SSE events - Persist agent response in DB after streaming completes - Add 6 new tests covering streaming behavior - Update agents.md documentation with streaming examples --- docs/user-guide/agents.md | 46 ++++ src/openjarvis/server/agent_manager_routes.py | 199 +++++++++++++++++- tests/server/test_agent_manager_routes.py | 167 +++++++++++++++ 3 files changed, 410 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/agents.md b/docs/user-guide/agents.md index ae27abf5..9f21a46f 100644 --- a/docs/user-guide/agents.md +++ b/docs/user-guide/agents.md @@ -621,3 +621,49 @@ All agents publish events on the `EventBus` when a bus is provided: `INFERENCE_START` / `INFERENCE_END` events are published by the `InstrumentedEngine` wrapper, not by agents directly. This keeps telemetry opt-in and transparent to agent code. These events enable the telemetry and trace systems to record detailed interaction data automatically. + +--- + +## Managed Agent Streaming + +The Managed Agent API (`/v1/managed-agents/{id}/messages`) supports real-time SSE streaming. Send a message with `stream: true` to receive the agent's response as a Server-Sent Events stream instead of the default asynchronous queue mode. + +### Streaming Messages + +```bash +curl -N -X POST http://localhost:8000/v1/managed-agents/{id}/messages \ + -H "Content-Type: application/json" \ + -d '{"content": "What is 2+2?", "stream": true}' +``` + +The response follows the OpenAI SSE format: + +1. **Content chunks** -- `data: {"choices": [{"delta": {"content": "token"}}]}` +2. **Tool results** (if the agent used tools) -- `event: tool_results\ndata: {"results": [...]}` +3. **Final chunk** -- `data: {"choices": [{"delta": {}, "finish_reason": "stop"}]}` +4. **Done sentinel** -- `data: [DONE]` + +When `stream: false` (the default), the endpoint behaves exactly as before -- the message is queued and the agent must be triggered separately via `/run`. + +### Behavior Details + +- The user message is always stored in the database before the agent runs. +- After streaming completes, the full agent response is persisted as an `agent_to_user` message. +- The agent is instantiated from the managed agent's stored `agent_type` and `config`. +- Conversation history from prior messages is automatically loaded as context. +- If the engine is not available on the server, a `503` error is returned. + +### Python Example + +```python +import httpx + +with httpx.stream( + "POST", + "http://localhost:8000/v1/managed-agents/{id}/messages", + json={"content": "Summarize today's news", "stream": True}, +) as response: + for line in response.iter_lines(): + if line.startswith("data:") and "[DONE]" not in line: + print(line[5:].strip()) +``` diff --git a/src/openjarvis/server/agent_manager_routes.py b/src/openjarvis/server/agent_manager_routes.py index 45a93f1a..d5515bd0 100644 --- a/src/openjarvis/server/agent_manager_routes.py +++ b/src/openjarvis/server/agent_manager_routes.py @@ -2,16 +2,20 @@ from __future__ import annotations +import logging from typing import Any, Dict, List, Optional, Tuple from openjarvis.agents.manager import AgentManager try: from fastapi import APIRouter, HTTPException, Request + from fastapi.responses import StreamingResponse from pydantic import BaseModel except ImportError: raise ImportError("fastapi and pydantic are required for server routes") +logger = logging.getLogger("openjarvis.server.agent_manager") + class CreateAgentRequest(BaseModel): name: str @@ -46,6 +50,7 @@ class BindChannelRequest(BaseModel): class SendMessageRequest(BaseModel): content: str mode: str = "queued" + stream: bool = False # SSE streaming mode class FeedbackRequest(BaseModel): @@ -202,6 +207,171 @@ def build_tools_list() -> List[Dict[str, Any]]: return items +async def _stream_managed_agent( + *, + manager: AgentManager, + agent_record: Dict[str, Any], + user_content: str, + message_id: str, + engine: Any, + bus: Any, +) -> StreamingResponse: + """Run a managed agent and stream the response as SSE. + + Instantiates the agent from its stored config, builds conversation + context from message history, executes the agent in a background + thread, and yields SSE-formatted chunks. After completion the + full response is persisted via ``manager.store_agent_response()``. + """ + import asyncio + import json + import uuid + + from openjarvis.agents._stubs import AgentContext + from openjarvis.core.registry import AgentRegistry + from openjarvis.core.types import Message, Role + + agent_id = agent_record["id"] + config = agent_record.get("config", {}) + agent_type = agent_record.get("agent_type", "orchestrator") + model = config.get("model", getattr(engine, "_model", "")) + + # Resolve the agent class from registry + agent_cls = AgentRegistry.get(agent_type) + if agent_cls is None: + # Fallback to orchestrator if the type is not registered + agent_cls = AgentRegistry.get("orchestrator") + if agent_cls is None: + raise HTTPException( + status_code=500, detail=f"Agent type '{agent_type}' not found in registry", + ) + + # Build agent constructor kwargs from config + agent_kwargs: Dict[str, Any] = { + "engine": engine, + "model": model, + } + if bus is not None: + agent_kwargs["bus"] = bus + if config.get("system_prompt"): + agent_kwargs["system_prompt"] = config["system_prompt"] + if config.get("temperature") is not None: + agent_kwargs["temperature"] = config["temperature"] + if config.get("max_tokens") is not None: + agent_kwargs["max_tokens"] = config["max_tokens"] + if config.get("max_turns") is not None: + agent_kwargs["max_turns"] = config["max_turns"] + + try: + agent = agent_cls(**agent_kwargs) + except TypeError as exc: + logger.warning("Agent instantiation failed with all kwargs, retrying minimal: %s", exc) + agent = agent_cls(engine=engine, model=model) + + # Build conversation context from existing messages + ctx = AgentContext() + messages = manager.list_messages(agent_id, limit=50) + # Messages come in DESC order, reverse for chronological + for m in reversed(messages): + # Skip the message we just stored (it will be the input) + if m["id"] == message_id: + continue + if m["direction"] == "user_to_agent": + ctx.conversation.add(Message(role=Role.USER, content=m["content"])) + elif m["direction"] == "agent_to_user": + ctx.conversation.add(Message(role=Role.ASSISTANT, content=m["content"])) + + # Mark the user message as delivered + manager.mark_message_delivered(message_id) + + chunk_id = f"chatcmpl-{uuid.uuid4().hex[:12]}" + + async def generate(): + """Async generator yielding SSE-formatted chunks.""" + collected_content = "" + + # Run agent.run() in a background thread + try: + result = await asyncio.to_thread(agent.run, user_content, context=ctx) + except Exception as exc: + logger.error("Managed agent stream error: %s", exc, exc_info=True) + error_data = { + "id": chunk_id, + "object": "chat.completion.chunk", + "model": model, + "choices": [{ + "index": 0, + "delta": {"content": f"Error: {exc}"}, + "finish_reason": "stop", + }], + } + yield f"data: {json.dumps(error_data)}\n\n" + yield "data: [DONE]\n\n" + return + + content = result.content or "" + collected_content = content + + # Emit tool results metadata if any + if result.tool_results: + tool_data = [] + for tr in result.tool_results: + tool_data.append({ + "tool_name": tr.tool_name, + "success": tr.success, + "output": tr.content, + "latency_ms": tr.latency_seconds * 1000, + }) + yield f"event: tool_results\ndata: {json.dumps({'results': tool_data})}\n\n" + + # Stream content word-by-word for real-time feel + if content: + words = content.split(" ") + for i, word in enumerate(words): + token = word if i == 0 else " " + word + chunk_data = { + "id": chunk_id, + "object": "chat.completion.chunk", + "model": model, + "choices": [{ + "index": 0, + "delta": {"content": token}, + "finish_reason": None, + }], + } + yield f"data: {json.dumps(chunk_data)}\n\n" + await asyncio.sleep(0.012) + + # Final chunk with finish_reason + final_data = { + "id": chunk_id, + "object": "chat.completion.chunk", + "model": model, + "choices": [{ + "index": 0, + "delta": {}, + "finish_reason": "stop", + }], + } + yield f"data: {json.dumps(final_data)}\n\n" + yield "data: [DONE]\n\n" + + # Persist agent response in DB after streaming completes + if collected_content: + try: + manager.store_agent_response(agent_id, collected_content) + except Exception as store_exc: + logger.error( + "Failed to store agent response: %s", store_exc, exc_info=True, + ) + + return StreamingResponse( + generate(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "Connection": "keep-alive"}, + ) + + def create_agent_manager_router( manager: AgentManager, ) -> Tuple[APIRouter, APIRouter, APIRouter, APIRouter]: @@ -399,9 +569,34 @@ def create_agent_manager_router( return {"messages": manager.list_messages(agent_id)} @agents_router.post("/{agent_id}/messages") - def send_message(agent_id: str, req: SendMessageRequest): + async def send_message(agent_id: str, req: SendMessageRequest, request: Request): + agent_record = manager.get_agent(agent_id) + if not agent_record: + raise HTTPException(status_code=404, detail="Agent not found") + + # Store user message in DB (always, regardless of stream mode) msg = manager.send_message(agent_id, req.content, mode=req.mode) - return msg + + if not req.stream: + return msg + + # --- Streaming mode: run agent and return SSE response --- + engine = getattr(request.app.state, "engine", None) + bus = getattr(request.app.state, "bus", None) + if engine is None: + raise HTTPException( + status_code=503, + detail="Engine not available for streaming", + ) + + return await _stream_managed_agent( + manager=manager, + agent_record=agent_record, + user_content=req.content, + message_id=msg["id"], + engine=engine, + bus=bus, + ) # ── State inspection ───────────────────────────────────── diff --git a/tests/server/test_agent_manager_routes.py b/tests/server/test_agent_manager_routes.py index 498bc167..48eea9b1 100644 --- a/tests/server/test_agent_manager_routes.py +++ b/tests/server/test_agent_manager_routes.py @@ -2,8 +2,10 @@ from __future__ import annotations +import json import tempfile from pathlib import Path +from unittest.mock import MagicMock import pytest @@ -200,3 +202,168 @@ class TestAgentManagerRoutes: assert "channels" in state assert "messages" in state assert "checkpoint" in state + + def test_send_message_non_stream_unchanged(self, manager, client): + """stream=False (default) returns a normal JSON message, not SSE.""" + agent = manager.create_agent(name="basic", agent_type="simple") + res = client.post( + f"/v1/managed-agents/{agent['id']}/messages", + json={"content": "hello", "stream": False}, + ) + assert res.status_code == 200 + data = res.json() + assert data["content"] == "hello" + assert data["direction"] == "user_to_agent" + + def test_send_message_stream_not_found(self, manager, client): + """Streaming to a non-existent agent returns 404.""" + res = client.post( + "/v1/managed-agents/nonexistent/messages", + json={"content": "hello", "stream": True}, + ) + assert res.status_code == 404 + + +@pytest.mark.skipif(not HAS_FASTAPI, reason="fastapi not installed") +class TestAgentManagerStreaming: + """Tests for the SSE streaming mode of the managed-agent messages endpoint.""" + + @pytest.fixture + def _mock_engine(self): + engine = MagicMock() + engine.engine_id = "mock" + engine._model = "test-model" + engine.health.return_value = True + return engine + + @pytest.fixture + def _mock_agent_cls(self): + """Register a mock agent class in the AgentRegistry for testing.""" + from openjarvis.agents._stubs import AgentResult + from openjarvis.core.registry import AgentRegistry + + class _MockStreamAgent: + agent_id = "mock_stream" + + def __init__(self, engine, model, **kwargs): + self._engine = engine + self._model = model + + def run(self, input_text, context=None, **kwargs): + return AgentResult(content=f"Echo: {input_text}", turns=1) + + # Register under a unique key for test isolation + AgentRegistry._entries()["_test_stream"] = _MockStreamAgent + yield _MockStreamAgent + AgentRegistry._entries().pop("_test_stream", None) + + @pytest.fixture + def stream_client(self, manager, _mock_engine, _mock_agent_cls): + from fastapi import FastAPI + + from openjarvis.server.agent_manager_routes import create_agent_manager_router + + app = FastAPI() + app.state.engine = _mock_engine + app.state.bus = None + + routers = create_agent_manager_router(manager) + agents_router, templates_router, global_router, tools_router = routers + app.include_router(agents_router) + app.include_router(templates_router) + app.include_router(global_router) + app.include_router(tools_router) + return TestClient(app) + + def test_send_message_stream(self, manager, stream_client, _mock_agent_cls): + """Test streaming mode returns SSE response with [DONE] sentinel.""" + agent = manager.create_agent( + name="streamer", agent_type="_test_stream", + ) + resp = stream_client.post( + f"/v1/managed-agents/{agent['id']}/messages", + json={"content": "What is 2+2?", "stream": True}, + ) + assert resp.status_code == 200 + assert "text/event-stream" in resp.headers.get("content-type", "") + + # Parse SSE events + lines = resp.text.strip().split("\n") + data_lines = [ln for ln in lines if ln.startswith("data:")] + assert len(data_lines) > 0 + # Last data line must be [DONE] + assert data_lines[-1].strip() == "data: [DONE]" + + def test_send_message_stream_content(self, manager, stream_client, _mock_agent_cls): + """Test streaming returns the correct agent response content.""" + agent = manager.create_agent( + name="streamer2", agent_type="_test_stream", + ) + resp = stream_client.post( + f"/v1/managed-agents/{agent['id']}/messages", + json={"content": "Hello world", "stream": True}, + ) + assert resp.status_code == 200 + + # Collect content tokens from stream + content = "" + for line in resp.text.strip().split("\n"): + if line.startswith("data:") and "[DONE]" not in line: + raw = line[5:].strip() + try: + data = json.loads(raw) + except json.JSONDecodeError: + continue + choices = data.get("choices", [{}]) + delta_content = choices[0].get("delta", {}).get("content") + if delta_content: + content += delta_content + + assert content == "Echo: Hello world" + + def test_send_message_stream_stores_response( + self, manager, stream_client, _mock_agent_cls, + ): + """After streaming, agent response is persisted in the DB.""" + agent = manager.create_agent( + name="streamer3", agent_type="_test_stream", + ) + resp = stream_client.post( + f"/v1/managed-agents/{agent['id']}/messages", + json={"content": "persist me", "stream": True}, + ) + assert resp.status_code == 200 + + # Check messages in DB + messages = manager.list_messages(agent["id"]) + # Should have both the user message and the agent response + assert len(messages) == 2 + directions = {m["direction"] for m in messages} + assert "user_to_agent" in directions + assert "agent_to_user" in directions + agent_msg = next(m for m in messages if m["direction"] == "agent_to_user") + assert "persist me" in agent_msg["content"] + + def test_send_message_stream_finish_reason( + self, manager, stream_client, _mock_agent_cls, + ): + """The final chunk before [DONE] has finish_reason='stop'.""" + agent = manager.create_agent( + name="streamer4", agent_type="_test_stream", + ) + resp = stream_client.post( + f"/v1/managed-agents/{agent['id']}/messages", + json={"content": "check finish", "stream": True}, + ) + # Collect all data chunks (excluding [DONE]) + chunks = [] + for line in resp.text.strip().split("\n"): + if line.startswith("data:") and "[DONE]" not in line: + raw = line[5:].strip() + try: + chunks.append(json.loads(raw)) + except json.JSONDecodeError: + continue + + # Last chunk should have finish_reason="stop" + assert chunks[-1]["choices"][0]["finish_reason"] == "stop" From 555e982f51960baae8f166649ad33d06f504b9f7 Mon Sep 17 00:00:00 2001 From: Jana Bergant Date: Tue, 24 Mar 2026 23:27:23 +0100 Subject: [PATCH 36/92] feat: query complexity analyzer for local/cloud routing (#102) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add query complexity analyzer with CLI and UI integration Classify incoming queries by difficulty (trivial→very_complex) to suggest appropriate token budgets for local vs. cloud routing. - Add score_complexity() with weighted signals (length, code, math, reasoning, multi-step, creative) and token tier mapping - Wire into `jarvis ask`: auto-suggest max_tokens when not set by user, show complexity in --profile output, log at DEBUG level - Add complexity metadata to /v1/chat/completions API response - Display complexity tier and score in frontend XRayFooter - Extend RoutingContext with complexity_score, suggested_max_tokens, has_reasoning fields - Update HeuristicRouter to route on complexity_score instead of raw query_length - Remove duplicated regex patterns from router.py (use complexity module as single source of truth) - Add 30 unit tests for complexity module - Fix existing router tests for new complexity-based routing Co-Authored-By: Claude Opus 4.6 (1M context) * fix: pass temperature and max_tokens from UI settings to backend The settings page stores temperature and max_tokens in the frontend store, but these values were never included in the chat API request. The backend Pydantic model defaults max_tokens to 1024 when the field is absent, which is too low for thinking models like qwen3.5 — they consume all tokens on reasoning and return empty content. Co-Authored-By: Claude Opus 4.6 (1M context) * fix: pass UI settings to backend and auto-bump max_tokens from complexity - Pass temperature and max_tokens from frontend Settings store to the backend API (cherry-picked from fix/ui-max-tokens-passthrough) - Server-side: bump max_tokens when the complexity analyzer suggests a higher budget (e.g. for thinking models on complex queries), never reduce below the client-requested value - Fixes empty responses with thinking models (e.g. qwen3.5) that consumed all tokens on internal reasoning Co-Authored-By: Claude Opus 4.6 (1M context) * fix: raise token budget tiers to prevent empty responses on thinking models Double all tier budgets (trivial: 512→1024, simple: 1024→2048, etc.) so that thinking models like qwen3.5 have enough headroom for internal chain-of-thought plus visible output, even on simple queries. Co-Authored-By: Claude Opus 4.6 (1M context) * fix: CI lint and test failures - Break long lines in routes.py to satisfy 88-char limit (E501) - Add intelligence.max_tokens and temperature to mocked config in test_ask_router.py so complexity analyzer can compare against int instead of MagicMock Co-Authored-By: Claude Opus 4.6 (1M context) * fix: line too long in test_complexity.py (E501) Co-Authored-By: Claude Opus 4.6 (1M context) --------- Co-authored-by: Claude Opus 4.6 (1M context) --- frontend/src/components/Chat/InputArea.tsx | 9 +- frontend/src/components/Chat/XRayFooter.tsx | 10 + frontend/src/lib/sse.ts | 2 + frontend/src/types/index.ts | 3 + src/openjarvis/cli/ask.py | 40 +++ src/openjarvis/core/types.py | 3 + src/openjarvis/learning/__init__.py | 6 + src/openjarvis/learning/routing/complexity.py | 261 ++++++++++++++++++ src/openjarvis/learning/routing/router.py | 65 +++-- src/openjarvis/server/models.py | 8 + src/openjarvis/server/routes.py | 54 +++- tests/cli/test_ask_router.py | 4 + tests/learning/routing/test_complexity.py | 183 ++++++++++++ tests/learning/test_router.py | 7 +- tests/learning/test_routing_models.py | 11 +- 15 files changed, 626 insertions(+), 40 deletions(-) create mode 100644 src/openjarvis/learning/routing/complexity.py create mode 100644 tests/learning/routing/test_complexity.py diff --git a/frontend/src/components/Chat/InputArea.tsx b/frontend/src/components/Chat/InputArea.tsx index a0a0cb2e..c395e71c 100644 --- a/frontend/src/components/Chat/InputArea.tsx +++ b/frontend/src/components/Chat/InputArea.tsx @@ -18,6 +18,8 @@ export function InputArea() { const streamState = useAppStore((s) => s.streamState); const messages = useAppStore((s) => s.messages); const speechEnabled = useAppStore((s) => s.settings.speechEnabled); + const maxTokens = useAppStore((s) => s.settings.maxTokens); + const temperature = useAppStore((s) => s.settings.temperature); const createConversation = useAppStore((s) => s.createConversation); const addMessage = useAppStore((s) => s.addMessage); const updateLastAssistant = useAppStore((s) => s.updateLastAssistant); @@ -127,6 +129,7 @@ export function InputArea() { let accumulatedContent = ''; let usage: TokenUsage | undefined; + let complexity: { score: number; tier: string; suggested_max_tokens: number } | undefined; const toolCalls: ToolCallInfo[] = []; let lastFlush = 0; let ttftMs: number | undefined; @@ -147,7 +150,7 @@ export function InputArea() { try { for await (const sseEvent of streamChat( - { model: selectedModel, messages: apiMessages, stream: true }, + { model: selectedModel, messages: apiMessages, stream: true, temperature, max_tokens: maxTokens }, controller.signal, )) { const eventName = sseEvent.event; @@ -202,6 +205,7 @@ export function InputArea() { const data = JSON.parse(sseEvent.data); const delta = data.choices?.[0]?.delta; if (data.usage) usage = data.usage; + if (data.complexity) complexity = data.complexity; if (delta?.content) { if (!ttftMs) ttftMs = Date.now() - startTime; accumulatedContent += delta.content; @@ -248,6 +252,9 @@ export function InputArea() { tokens_per_sec: usage?.completion_tokens ? usage.completion_tokens / (totalMs / 1000) : undefined, + complexity_score: complexity?.score, + complexity_tier: complexity?.tier, + suggested_max_tokens: complexity?.suggested_max_tokens, }; updateLastAssistant( convId, diff --git a/frontend/src/components/Chat/XRayFooter.tsx b/frontend/src/components/Chat/XRayFooter.tsx index 4aad1e58..b13126dc 100644 --- a/frontend/src/components/Chat/XRayFooter.tsx +++ b/frontend/src/components/Chat/XRayFooter.tsx @@ -18,6 +18,7 @@ export function XRayFooter({ usage, telemetry }: Props) { const parts: string[] = []; if (telemetry?.engine) parts.push(telemetry.engine); if (telemetry?.model_id) parts.push(telemetry.model_id); + if (telemetry?.complexity_tier) parts.push(telemetry.complexity_tier); if (telemetry?.total_ms) parts.push(formatMs(telemetry.total_ms)); if (usage && (usage.prompt_tokens || usage.completion_tokens)) { parts.push(`${usage.prompt_tokens} input tokens`); @@ -50,6 +51,15 @@ export function XRayFooter({ usage, telemetry }: Props) { } rows.push({ label: 'Tokens', value: tokenParts.join(' \u00B7 ') }); } + if (telemetry?.complexity_tier) { + rows.push({ + label: 'Complexity', + value: `${telemetry.complexity_tier} (${telemetry.complexity_score?.toFixed(2)})`, + }); + } + if (telemetry?.suggested_max_tokens) { + rows.push({ label: 'Token budget', value: `${telemetry.suggested_max_tokens}` }); + } if (telemetry?.tokens_per_sec) { rows.push({ label: 'Speed', value: `${Math.round(telemetry.tokens_per_sec)} tok/s` }); } diff --git a/frontend/src/lib/sse.ts b/frontend/src/lib/sse.ts index addd10b9..d00fae92 100644 --- a/frontend/src/lib/sse.ts +++ b/frontend/src/lib/sse.ts @@ -5,6 +5,8 @@ export interface ChatRequest { model: string; messages: Array<{ role: string; content: string }>; stream: true; + temperature?: number; + max_tokens?: number; } export async function* streamChat( diff --git a/frontend/src/types/index.ts b/frontend/src/types/index.ts index fa186d32..a01205b3 100644 --- a/frontend/src/types/index.ts +++ b/frontend/src/types/index.ts @@ -56,6 +56,9 @@ export interface MessageTelemetry { tokens_per_sec?: number; ttft_ms?: number; total_ms?: number; + complexity_score?: number; + complexity_tier?: string; + suggested_max_tokens?: number; } export interface ChatMessage { diff --git a/src/openjarvis/cli/ask.py b/src/openjarvis/cli/ask.py index 85455506..43014909 100644 --- a/src/openjarvis/cli/ask.py +++ b/src/openjarvis/cli/ask.py @@ -165,6 +165,7 @@ def _print_profile( engine_name: str, model_name: str, console: Console, + complexity_result=None, ) -> None: """Print an inference telemetry profile table from EventBus history.""" # Collect all INFERENCE_END events (agents may fire multiple) @@ -227,6 +228,12 @@ def _print_profile( def _row(label: str, val: str) -> None: table.add_row(label, val) + if complexity_result is not None: + _row("Complexity score", f"{complexity_result.score:.3f}") + _row("Complexity tier", complexity_result.tier) + _row("Suggested max tokens", str(complexity_result.suggested_max_tokens)) + _row("", "") # separator + _row("Wall time", f"{wall_seconds:.3f} s") _row("Inference calls", str(total_calls)) _row("Total latency", f"{total_latency:.3f} s") @@ -324,12 +331,30 @@ def ask( # Load config config = load_config() + # Track whether the user explicitly set --max-tokens + user_set_max_tokens = max_tokens is not None + # Fall back to config values for generation params if temperature is None: temperature = config.intelligence.temperature if max_tokens is None: max_tokens = config.intelligence.max_tokens + # Run complexity analysis on the query + from openjarvis.learning.routing.complexity import ( + ComplexityResult, + adjust_tokens_for_model, + score_complexity, + ) + + complexity_result: ComplexityResult = score_complexity(query_text) + logger.debug( + "Complexity analysis: score=%.3f tier=%s suggested_max_tokens=%d", + complexity_result.score, + complexity_result.tier, + complexity_result.suggested_max_tokens, + ) + # Set up telemetry bus = EventBus(record_history=True) telem_store: TelemetryStore | None = None @@ -397,6 +422,19 @@ def ask( console.print("[red]No model available on engine.[/red]") sys.exit(1) + # Apply complexity-suggested token budget when user didn't override. + # Use at least the config default so we never reduce tokens below what + # the user would have gotten without the analyzer. + if not user_set_max_tokens: + suggested = adjust_tokens_for_model( + complexity_result.suggested_max_tokens, model_name, + ) + max_tokens = max(suggested, config.intelligence.max_tokens) + logger.debug( + "Using complexity-suggested max_tokens=%d (model=%s)", + max_tokens, model_name, + ) + # Agent mode if agent_name is not None: parsed_tools = resolve_tool_names( @@ -435,6 +473,7 @@ def ask( _print_profile( bus, time.monotonic() - wall_start, engine_name, model_name, console, + complexity_result=complexity_result, ) if telem_store is not None: @@ -494,6 +533,7 @@ def ask( _print_profile( bus, time.monotonic() - wall_start, engine_name, model_name, console, + complexity_result=complexity_result, ) # Cleanup diff --git a/src/openjarvis/core/types.py b/src/openjarvis/core/types.py index 1d354ccf..a8629752 100644 --- a/src/openjarvis/core/types.py +++ b/src/openjarvis/core/types.py @@ -235,8 +235,11 @@ class RoutingContext: query_length: int = 0 has_code: bool = False has_math: bool = False + has_reasoning: bool = False language: str = "en" urgency: float = 0.5 + complexity_score: float = 0.0 # 0.0 (trivial) to 1.0 (very complex) + suggested_max_tokens: int = 1024 metadata: Dict[str, Any] = field(default_factory=dict) diff --git a/src/openjarvis/learning/__init__.py b/src/openjarvis/learning/__init__.py index ec1c8315..9623c340 100644 --- a/src/openjarvis/learning/__init__.py +++ b/src/openjarvis/learning/__init__.py @@ -13,6 +13,10 @@ from openjarvis.learning.learning_orchestrator import LearningOrchestrator from openjarvis.learning.optimize.llm_optimizer import LLMOptimizer from openjarvis.learning.optimize.optimizer import OptimizationEngine from openjarvis.learning.optimize.store import OptimizationStore +from openjarvis.learning.routing.complexity import ( + ComplexityQueryAnalyzer, + score_complexity, +) from openjarvis.learning.routing.heuristic_reward import HeuristicRewardFunction from openjarvis.learning.routing.router import ( HeuristicRouter, @@ -59,6 +63,7 @@ def ensure_registered() -> None: __all__ = [ "AgentConfigEvolver", + "ComplexityQueryAnalyzer", "HAS_TORCH", "HeuristicRewardFunction", "HeuristicRouter", @@ -75,4 +80,5 @@ __all__ = [ "TrainingDataMiner", "build_routing_context", "ensure_registered", + "score_complexity", ] diff --git a/src/openjarvis/learning/routing/complexity.py b/src/openjarvis/learning/routing/complexity.py new file mode 100644 index 00000000..7ce77137 --- /dev/null +++ b/src/openjarvis/learning/routing/complexity.py @@ -0,0 +1,261 @@ +"""Query complexity analyzer — scores queries and suggests token budgets. + +Produces a numeric complexity score (0.0–1.0) and a suggested +``max_tokens`` budget based on query characteristics such as length, +domain signals (code, math, multi-step reasoning), and whether the +target model is a thinking model that needs extra headroom. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Optional + +from openjarvis.core.types import RoutingContext +from openjarvis.learning._stubs import QueryAnalyzer + +# --------------------------------------------------------------------------- +# Signal patterns +# --------------------------------------------------------------------------- + +_CODE_PATTERNS = re.compile( + r"```|`[^`]+`|\bdef\s|\bclass\s|\bimport\s|\bfunction\s|\bconst\s|\bvar\s|\blet\s|" + r"\bif\s*\(|->|=>|\{\s*\}|\bfor\s+\w+\s+in\s|#include|System\.out", + re.IGNORECASE, +) +_MATH_PATTERNS = re.compile( + r"\bsolve\b|\bintegral\b|\bequation\b|\bproof\b|\bderivative\b|\bmatrix\b|" + r"\btheorem\b|\bcalculate\b|\bcompute\b|\bsigma\b|\bsum\b|\blimit\b|\bprobability\b", + re.IGNORECASE, +) +_REASONING_PATTERNS = re.compile( + r"\bexplain\b|\banalyze\b|\bcompare\b|\bwhy\b" + r"|\bstep[- ]by[- ]step\b|\breason\b|\bthink\b" + r"|\bpros\s+and\s+cons\b|\btrade-?\s*offs?\b|\bevaluate\b", + re.IGNORECASE, +) +_MULTI_STEP_PATTERNS = re.compile( + r"\bthen\b.*\bthen\b|\bfirst\b.*\bnext\b|\bstep\s*\d" + r"|\b(?:and\s+also|additionally|furthermore)\b" + r"|\b\d+\.\s", + re.IGNORECASE | re.DOTALL, +) +_CREATIVE_PATTERNS = re.compile( + r"\bwrite\b.*\b(?:essay|story|article|report|poem)\b" + r"|\bgenerate\b.*\b(?:code|script|program)\b" + r"|\bcreate\b|\bdesign\b|\bdraft\b|\bcompose\b", + re.IGNORECASE, +) + +# Models known to use internal chain-of-thought that consumes output tokens. +_THINKING_MODEL_PATTERNS = re.compile( + r"qwen3\.5|qwq|deepseek-r1|o1-|o3-|o4-", re.IGNORECASE +) + +# --------------------------------------------------------------------------- +# Token budget tiers +# --------------------------------------------------------------------------- + +_TOKEN_TIERS = { + "trivial": 1024, # greetings, yes/no, factoid lookups + "simple": 2048, # short answers, definitions + "moderate": 4096, # explanations, summaries + "complex": 8192, # analysis, code generation, multi-step + "very_complex": 16384, # long-form, multi-part reasoning +} + +# Thinking models need extra headroom for internal chain-of-thought. +_THINKING_TOKEN_MULTIPLIER = 2 + + +# --------------------------------------------------------------------------- +# Complexity scoring +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class ComplexityResult: + """Output of the complexity analysis.""" + + score: float # 0.0–1.0 + tier: str + suggested_max_tokens: int + signals: dict + + +def _count_questions(query: str) -> int: + """Count the number of question marks in the query.""" + return query.count("?") + + +def _count_sub_tasks(query: str) -> int: + """Estimate the number of sub-tasks or enumerated items.""" + numbered = len(re.findall(r"^\s*\d+[.)]\s", query, re.MULTILINE)) + bulleted = len(re.findall(r"^\s*[-*]\s", query, re.MULTILINE)) + return numbered + bulleted + + +def score_complexity(query: str) -> ComplexityResult: + """Score a query's complexity from 0.0 (trivial) to 1.0 (very complex). + + The score is a weighted combination of independent signals, each + contributing a fraction between 0 and 1. Weights reflect how much + each signal correlates with the amount of reasoning and output + tokens a model will need. + """ + signals: dict = {} + score = 0.0 + + # --- Length signal (0–0.20) --- + length = len(query) + if length < 20: + length_score = 0.0 + elif length < 100: + length_score = 0.3 + elif length < 300: + length_score = 0.6 + elif length < 800: + length_score = 0.8 + else: + length_score = 1.0 + signals["length"] = length_score + score += 0.20 * length_score + + # --- Domain signals (0–0.25) --- + has_code = bool(_CODE_PATTERNS.search(query)) + has_math = bool(_MATH_PATTERNS.search(query)) + domain_score = 0.0 + if has_code: + domain_score = max(domain_score, 0.7) + if has_math: + domain_score = max(domain_score, 0.8) + if has_code and has_math: + domain_score = 1.0 + signals["domain"] = domain_score + signals["has_code"] = has_code + signals["has_math"] = has_math + score += 0.25 * domain_score + + # --- Reasoning signal (0–0.25) --- + has_reasoning = bool(_REASONING_PATTERNS.search(query)) + has_multi_step = bool(_MULTI_STEP_PATTERNS.search(query)) + reasoning_score = 0.0 + if has_reasoning: + reasoning_score = 0.6 + if has_multi_step: + reasoning_score = max(reasoning_score, 0.8) + if has_reasoning and has_multi_step: + reasoning_score = 1.0 + signals["reasoning"] = reasoning_score + signals["has_reasoning"] = has_reasoning + signals["has_multi_step"] = has_multi_step + score += 0.25 * reasoning_score + + # --- Question / sub-task count (0–0.15) --- + n_questions = _count_questions(query) + n_subtasks = _count_sub_tasks(query) + multi_part = n_questions + n_subtasks + if multi_part <= 1: + multi_score = 0.0 + elif multi_part <= 3: + multi_score = 0.5 + else: + multi_score = 1.0 + signals["multi_part"] = multi_score + signals["n_questions"] = n_questions + signals["n_subtasks"] = n_subtasks + score += 0.15 * multi_score + + # --- Creative / generative signal (0–0.15) --- + has_creative = bool(_CREATIVE_PATTERNS.search(query)) + creative_score = 0.7 if has_creative else 0.0 + signals["creative"] = creative_score + signals["has_creative"] = has_creative + score += 0.15 * creative_score + + # Clamp + score = max(0.0, min(1.0, score)) + + # Map to tier + if score < 0.15: + tier = "trivial" + elif score < 0.30: + tier = "simple" + elif score < 0.55: + tier = "moderate" + elif score < 0.80: + tier = "complex" + else: + tier = "very_complex" + + suggested_max_tokens = _TOKEN_TIERS[tier] + + return ComplexityResult( + score=round(score, 3), + tier=tier, + suggested_max_tokens=suggested_max_tokens, + signals=signals, + ) + + +def is_thinking_model(model_name: str) -> bool: + """Return True if the model is known to use internal chain-of-thought.""" + return bool(_THINKING_MODEL_PATTERNS.search(model_name)) + + +def adjust_tokens_for_model( + suggested: int, model_name: Optional[str] = None +) -> int: + """Multiply the token budget when the target is a thinking model.""" + if model_name and is_thinking_model(model_name): + return suggested * _THINKING_TOKEN_MULTIPLIER + return suggested + + +# --------------------------------------------------------------------------- +# QueryAnalyzer implementation +# --------------------------------------------------------------------------- + + +class ComplexityQueryAnalyzer(QueryAnalyzer): + """Query analyzer that produces a complexity-aware RoutingContext. + + Drop-in replacement for ``DefaultQueryAnalyzer`` — adds + ``complexity_score``, ``has_reasoning``, and ``suggested_max_tokens`` + to the returned ``RoutingContext``. + """ + + def analyze( + self, query: str, **kwargs: object + ) -> RoutingContext: + urgency = kwargs.get("urgency", 0.5) + if not isinstance(urgency, (int, float)): + urgency = 0.5 + model_name = kwargs.get("model") + if not isinstance(model_name, str): + model_name = None + + result = score_complexity(query) + tokens = adjust_tokens_for_model(result.suggested_max_tokens, model_name) + + return RoutingContext( + query=query, + query_length=len(query), + has_code=result.signals.get("has_code", False), + has_math=result.signals.get("has_math", False), + has_reasoning=result.signals.get("has_reasoning", False), + urgency=float(urgency), + complexity_score=result.score, + suggested_max_tokens=tokens, + metadata={"complexity_tier": result.tier, "signals": result.signals}, + ) + + +__all__ = [ + "ComplexityQueryAnalyzer", + "ComplexityResult", + "adjust_tokens_for_model", + "is_thinking_model", + "score_complexity", +] diff --git a/src/openjarvis/learning/routing/router.py b/src/openjarvis/learning/routing/router.py index 3bc4744b..4862c01f 100644 --- a/src/openjarvis/learning/routing/router.py +++ b/src/openjarvis/learning/routing/router.py @@ -3,7 +3,6 @@ from __future__ import annotations import logging -import re from typing import List, Optional from openjarvis.core.registry import ModelRegistry @@ -12,32 +11,33 @@ from openjarvis.learning._stubs import QueryAnalyzer, RouterPolicy logger = logging.getLogger(__name__) -# Detection patterns -_CODE_PATTERNS = re.compile( - r"```|`[^`]+`|\bdef\s|\bclass\s|\bimport\s|\bfunction\s|\bconst\s|\bvar\s|\blet\s|" - r"\bif\s*\(|->|=>|\{\s*\}|\bfor\s+\w+\s+in\s|#include|System\.out", - re.IGNORECASE, -) -_MATH_PATTERNS = re.compile( - r"\bsolve\b|\bintegral\b|\bequation\b|\bproof\b|\bderivative\b|\bmatrix\b|" - r"\btheorem\b|\bcalculate\b|\bcompute\b|\bsigma\b|\bsum\b|\blimit\b|\bprobability\b", - re.IGNORECASE, -) -_REASONING_KEYWORDS = re.compile( - r"\bexplain\b|\banalyze\b|\bcompare\b|\bwhy\b" - r"|\bstep[- ]by[- ]step\b|\breason\b|\bthink\b", - re.IGNORECASE, -) +def build_routing_context( + query: str, *, urgency: float = 0.5, model: str | None = None +) -> RoutingContext: + """Populate a ``RoutingContext`` from a raw query string. + + When *model* is provided, the suggested token budget is adjusted + for thinking models that need extra headroom. + """ + from openjarvis.learning.routing.complexity import ( + adjust_tokens_for_model, + score_complexity, + ) + + result = score_complexity(query) + tokens = adjust_tokens_for_model(result.suggested_max_tokens, model) -def build_routing_context(query: str, *, urgency: float = 0.5) -> RoutingContext: - """Populate a ``RoutingContext`` from a raw query string.""" return RoutingContext( query=query, query_length=len(query), - has_code=bool(_CODE_PATTERNS.search(query)), - has_math=bool(_MATH_PATTERNS.search(query)), + has_code=result.signals.get("has_code", False), + has_math=result.signals.get("has_math", False), + has_reasoning=result.signals.get("has_reasoning", False), urgency=urgency, + complexity_score=result.score, + suggested_max_tokens=tokens, + metadata={"complexity_tier": result.tier, "signals": result.signals}, ) @@ -94,8 +94,8 @@ class HeuristicRouter(RouterPolicy): Rules (applied in order): 1. Code detected → prefer model with "code"/"coder" in name 2. Math detected → prefer larger model - 3. Short query (<50 chars, no code/math) → prefer smaller/faster model - 4. Long/complex query (>500 chars OR reasoning keywords) → prefer larger model + 3. Low complexity (score < 0.20) → prefer smaller/faster model + 4. High complexity (score >= 0.55 OR reasoning keywords) → prefer larger model 5. High urgency (>0.8) → override to smaller model 6. Default fallback → default_model → fallback_model → first available """ @@ -139,12 +139,12 @@ class HeuristicRouter(RouterPolicy): if context.has_math: return _largest_model(available) or available[0] - # Rule 3: Short simple query → prefer smaller model - if context.query_length < 50: + # Rule 3: Low complexity → prefer smaller model + if context.complexity_score < 0.20: return _smallest_model(available) or available[0] - # Rule 4: Long/complex query → prefer larger model - if context.query_length > 500 or _REASONING_KEYWORDS.search(context.query): + # Rule 4: High complexity or reasoning → prefer larger model + if context.complexity_score >= 0.55 or context.has_reasoning: return _largest_model(available) or available[0] # Rule 6: Default fallback @@ -162,7 +162,14 @@ class DefaultQueryAnalyzer(QueryAnalyzer): urgency = kwargs.get("urgency", 0.5) if not isinstance(urgency, (int, float)): urgency = 0.5 - return build_routing_context(query, urgency=urgency) + model = kwargs.get("model") + if not isinstance(model, str): + model = None + return build_routing_context(query, urgency=urgency, model=model) -__all__ = ["DefaultQueryAnalyzer", "HeuristicRouter", "build_routing_context"] +__all__ = [ + "DefaultQueryAnalyzer", + "HeuristicRouter", + "build_routing_context", +] diff --git a/src/openjarvis/server/models.py b/src/openjarvis/server/models.py index 45140767..8086f078 100644 --- a/src/openjarvis/server/models.py +++ b/src/openjarvis/server/models.py @@ -53,6 +53,12 @@ class Choice(BaseModel): finish_reason: str = "stop" +class ComplexityInfo(BaseModel): + score: float + tier: str + suggested_max_tokens: int + + class ChatCompletionResponse(BaseModel): id: str = Field(default_factory=lambda: f"chatcmpl-{uuid.uuid4().hex[:12]}") object: str = "chat.completion" @@ -60,6 +66,7 @@ class ChatCompletionResponse(BaseModel): model: str = "" choices: List[Choice] = Field(default_factory=list) usage: UsageInfo = Field(default_factory=UsageInfo) + complexity: Optional[ComplexityInfo] = None # --------------------------------------------------------------------------- @@ -110,6 +117,7 @@ __all__ = [ "ChatMessage", "Choice", "ChoiceMessage", + "ComplexityInfo", "DeltaMessage", "ModelListResponse", "ModelObject", diff --git a/src/openjarvis/server/routes.py b/src/openjarvis/server/routes.py index c6625cb4..bf935bef 100644 --- a/src/openjarvis/server/routes.py +++ b/src/openjarvis/server/routes.py @@ -16,6 +16,7 @@ from openjarvis.server.models import ( ChatCompletionResponse, Choice, ChoiceMessage, + ComplexityInfo, DeltaMessage, ModelListResponse, ModelObject, @@ -94,6 +95,38 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request "Memory context injection failed", exc_info=True, ) + # Run complexity analysis on the last user message + complexity_info = None + query_text_for_complexity = "" + for m in reversed(request_body.messages): + if m.role == "user" and m.content: + query_text_for_complexity = m.content + break + if query_text_for_complexity: + try: + from openjarvis.learning.routing.complexity import ( + adjust_tokens_for_model, + score_complexity, + ) + + cr = score_complexity(query_text_for_complexity) + suggested = adjust_tokens_for_model( + cr.suggested_max_tokens, model, + ) + complexity_info = ComplexityInfo( + score=cr.score, + tier=cr.tier, + suggested_max_tokens=suggested, + ) + # Bump max_tokens when complexity suggests more than what + # the client requested — never reduce below the request value. + if suggested > request_body.max_tokens: + request_body.max_tokens = suggested + except Exception: + logging.getLogger("openjarvis.server").debug( + "Complexity analysis failed", exc_info=True, + ) + if request_body.stream: bus = getattr(request.app.state, "bus", None) # Use the agent stream bridge only when tools are present (the @@ -102,18 +135,22 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request # directly from the engine for true token-by-token output. if agent is not None and bus is not None and request_body.tools: return await _handle_agent_stream(agent, bus, model, request_body) - return await _handle_stream(engine, model, request_body) + return await _handle_stream(engine, model, request_body, complexity_info) # Non-streaming: use agent if available, otherwise direct engine call if agent is not None: - return _handle_agent(agent, model, request_body) + return _handle_agent(agent, model, request_body, complexity_info) bus = getattr(request.app.state, "bus", None) - return _handle_direct(engine, model, request_body, bus=bus) + return _handle_direct( + engine, model, request_body, + bus=bus, complexity_info=complexity_info, + ) def _handle_direct( engine, model: str, req: ChatCompletionRequest, bus=None, + complexity_info=None, ) -> ChatCompletionResponse: """Direct engine call without agent.""" messages = _to_messages(req.messages) @@ -166,11 +203,13 @@ def _handle_direct( completion_tokens=usage.get("completion_tokens", 0), total_tokens=usage.get("total_tokens", 0), ), + complexity=complexity_info, ) def _handle_agent( agent, model: str, req: ChatCompletionRequest, + complexity_info=None, ) -> ChatCompletionResponse: """Run through agent.""" from openjarvis.agents._stubs import AgentContext @@ -207,6 +246,7 @@ def _handle_agent( finish_reason="stop", )], usage=usage, + complexity=complexity_info, ) @@ -217,7 +257,10 @@ async def _handle_agent_stream(agent, bus, model, req): return await create_agent_stream(agent, bus, model, req) -async def _handle_stream(engine, model: str, req: ChatCompletionRequest): +async def _handle_stream( + engine, model: str, req: ChatCompletionRequest, + complexity_info=None, +): """Stream response using SSE format.""" messages = _to_messages(req.messages) chunk_id = f"chatcmpl-{uuid.uuid4().hex[:12]}" @@ -294,6 +337,9 @@ async def _handle_stream(engine, model: str, req: ChatCompletionRequest): if isinstance(stream_usage, dict) and stream_usage.get("total_tokens", 0) > 0: finish_dict["usage"] = stream_usage + if complexity_info is not None: + finish_dict["complexity"] = complexity_info.model_dump() + yield f"data: {_json.dumps(finish_dict)}\n\n" yield "data: [DONE]\n\n" diff --git a/tests/cli/test_ask_router.py b/tests/cli/test_ask_router.py index 39758979..a4f5fbbe 100644 --- a/tests/cli/test_ask_router.py +++ b/tests/cli/test_ask_router.py @@ -93,6 +93,8 @@ class TestAskModelResolution: cfg.telemetry.enabled = False cfg.intelligence.default_model = "" cfg.intelligence.fallback_model = "" + cfg.intelligence.temperature = 0.7 + cfg.intelligence.max_tokens = 1024 cfg.agent.context_from_memory = False result = CliRunner().invoke(cli, ["ask", "Hello"]) assert result.exit_code == 0 @@ -122,6 +124,8 @@ class TestAskModelResolution: cfg.telemetry.enabled = False cfg.intelligence.default_model = "" cfg.intelligence.fallback_model = "fallback-model" + cfg.intelligence.temperature = 0.7 + cfg.intelligence.max_tokens = 1024 cfg.agent.context_from_memory = False result = CliRunner().invoke(cli, ["ask", "Hello"]) assert result.exit_code == 0 diff --git a/tests/learning/routing/test_complexity.py b/tests/learning/routing/test_complexity.py new file mode 100644 index 00000000..999eaf1e --- /dev/null +++ b/tests/learning/routing/test_complexity.py @@ -0,0 +1,183 @@ +"""Tests for query complexity analyzer.""" + +from __future__ import annotations + +from openjarvis.learning.routing.complexity import ( + ComplexityQueryAnalyzer, + ComplexityResult, + adjust_tokens_for_model, + is_thinking_model, + score_complexity, +) + + +class TestScoreComplexity: + def test_trivial_query(self) -> None: + result = score_complexity("Hi") + assert result.tier == "trivial" + assert result.score < 0.15 + + def test_simple_query(self) -> None: + result = score_complexity("What is the capital of France?") + assert result.tier in ("trivial", "simple") + assert result.score < 0.30 + + def test_code_signal(self) -> None: + result = score_complexity("def hello():\n return 'world'") + assert result.signals["has_code"] is True + assert result.score > 0.0 + + def test_math_signal(self) -> None: + result = score_complexity("Solve the integral of x^2 dx") + assert result.signals["has_math"] is True + assert result.score > 0.0 + + def test_code_and_math_gives_high_domain(self) -> None: + query = "```python\nimport numpy\n```\nSolve the integral of x^2" + result = score_complexity(query) + assert result.signals["has_code"] is True + assert result.signals["has_math"] is True + assert result.signals["domain"] == 1.0 + + def test_reasoning_signal(self) -> None: + result = score_complexity("Explain why the sky is blue step by step") + assert result.signals["has_reasoning"] is True + + def test_multi_step_signal(self) -> None: + result = score_complexity("First do X, then do Y, then do Z") + assert result.signals["has_multi_step"] is True + + def test_reasoning_and_multi_step_combined(self) -> None: + result = score_complexity( + "Explain why X works, then analyze Y, then compare them" + ) + assert result.signals["has_reasoning"] is True + assert result.signals["has_multi_step"] is True + assert result.signals["reasoning"] == 1.0 + + def test_multi_part_questions(self) -> None: + query = "What is X? What is Y? What is Z? What is W?" + result = score_complexity(query) + assert result.signals["n_questions"] == 4 + assert result.signals["multi_part"] == 1.0 + + def test_creative_signal(self) -> None: + result = score_complexity("Write an essay about climate change") + assert result.signals["has_creative"] is True + + def test_very_complex_query(self) -> None: + query = ( + "Explain step by step how to solve the integral of x^2, " + "then write Python code to compute it numerically. " + "1. Derive the analytical solution\n" + "2. Implement numerical integration\n" + "3. Compare the results and analyze the error" + ) + result = score_complexity(query) + assert result.tier in ("complex", "very_complex") + assert result.score >= 0.55 + + def test_score_clamped_to_unit_interval(self) -> None: + result = score_complexity("x" * 2000) + assert 0.0 <= result.score <= 1.0 + + def test_result_type(self) -> None: + result = score_complexity("hello") + assert isinstance(result, ComplexityResult) + assert isinstance(result.score, float) + assert isinstance(result.tier, str) + assert isinstance(result.suggested_max_tokens, int) + assert isinstance(result.signals, dict) + + def test_token_tiers_increase_with_complexity(self) -> None: + trivial = score_complexity("Hi") + complex_ = score_complexity( + "Explain step by step how to solve the integral of x^2 " + "and write code to compute the derivative of the matrix equation" + ) + assert complex_.suggested_max_tokens >= trivial.suggested_max_tokens + + def test_subtask_counting(self) -> None: + query = "1. First task\n2. Second task\n- Bullet item" + result = score_complexity(query) + assert result.signals["n_subtasks"] == 3 + + +class TestIsThinkingModel: + def test_deepseek_r1(self) -> None: + assert is_thinking_model("deepseek-r1-32b") is True + + def test_o1_model(self) -> None: + assert is_thinking_model("o1-preview") is True + + def test_o3_model(self) -> None: + assert is_thinking_model("o3-mini") is True + + def test_qwq(self) -> None: + assert is_thinking_model("qwq-32b") is True + + def test_regular_model(self) -> None: + assert is_thinking_model("llama-3.1-8b") is False + + def test_gpt4(self) -> None: + assert is_thinking_model("gpt-4o") is False + + +class TestAdjustTokensForModel: + def test_thinking_model_doubles(self) -> None: + assert adjust_tokens_for_model(1024, "deepseek-r1-32b") == 2048 + + def test_regular_model_unchanged(self) -> None: + assert adjust_tokens_for_model(1024, "llama-3.1-8b") == 1024 + + def test_no_model_unchanged(self) -> None: + assert adjust_tokens_for_model(1024) == 1024 + + def test_none_model_unchanged(self) -> None: + assert adjust_tokens_for_model(1024, None) == 1024 + + +class TestComplexityQueryAnalyzer: + def test_returns_routing_context(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx = analyzer.analyze("Hello world") + assert ctx.query == "Hello world" + assert ctx.query_length == len("Hello world") + + def test_complexity_score_populated(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx = analyzer.analyze("Explain step by step how photosynthesis works") + assert ctx.complexity_score > 0.0 + assert ctx.has_reasoning is True + + def test_code_detection(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx = analyzer.analyze("def foo(): pass") + assert ctx.has_code is True + + def test_math_detection(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx = analyzer.analyze("solve the equation x^2 = 4") + assert ctx.has_math is True + + def test_urgency_passthrough(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx = analyzer.analyze("test", urgency=0.9) + assert ctx.urgency == 0.9 + + def test_invalid_urgency_defaults(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx = analyzer.analyze("test", urgency="invalid") + assert ctx.urgency == 0.5 + + def test_thinking_model_adjusts_tokens(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx_normal = analyzer.analyze("Hello", model="llama-3.1-8b") + ctx_thinking = analyzer.analyze("Hello", model="deepseek-r1-32b") + assert ctx_thinking.suggested_max_tokens == ctx_normal.suggested_max_tokens * 2 + + def test_metadata_contains_tier(self) -> None: + analyzer = ComplexityQueryAnalyzer() + ctx = analyzer.analyze("Hi") + assert "complexity_tier" in ctx.metadata + assert "signals" in ctx.metadata diff --git a/tests/learning/test_router.py b/tests/learning/test_router.py index bf80b4ea..736a7f8c 100644 --- a/tests/learning/test_router.py +++ b/tests/learning/test_router.py @@ -95,12 +95,12 @@ class TestHeuristicRouter: ) assert router.select_model(ctx) == "large" - def test_long_query_prefers_large(self) -> None: + def test_high_complexity_prefers_large(self) -> None: _register_models() router = HeuristicRouter( available_models=["small", "large", "coder"], ) - ctx = RoutingContext(query="x" * 501, query_length=501) + ctx = RoutingContext(query="x" * 501, query_length=501, complexity_score=0.7) assert router.select_model(ctx) == "large" def test_high_urgency_overrides_to_small(self) -> None: @@ -122,10 +122,11 @@ class TestHeuristicRouter: default_model="large", fallback_model="small", ) - # Medium-length, no code/math, no reasoning → falls to default + # Medium complexity, no code/math, no reasoning → falls to default ctx = RoutingContext( query="Tell me about cats", query_length=60, + complexity_score=0.35, ) assert router.select_model(ctx) == "large" diff --git a/tests/learning/test_routing_models.py b/tests/learning/test_routing_models.py index febbd22c..ebe4917a 100644 --- a/tests/learning/test_routing_models.py +++ b/tests/learning/test_routing_models.py @@ -85,12 +85,12 @@ class TestRouterWithNewModels: selected = router.select_model(ctx) assert selected == "gpt-oss:120b" - def test_long_context_routes_to_largest(self) -> None: + def test_high_complexity_routes_to_largest(self) -> None: _setup_models() router = HeuristicRouter( available_models=NEW_LOCAL_MODELS, ) - ctx = RoutingContext(query="x" * 501, query_length=501) + ctx = RoutingContext(query="x" * 501, query_length=501, complexity_score=0.7) selected = router.select_model(ctx) assert selected == "gpt-oss:120b" @@ -128,6 +128,7 @@ class TestRouterWithNewModels: ctx = RoutingContext( query="Tell me about the weather today", query_length=60, + complexity_score=0.35, ) selected = router.select_model(ctx) assert selected == "glm-4.7-flash" @@ -172,7 +173,11 @@ class TestRouterParameterized: ("hi", False), ("solve the integral of sin(x)", True), ("def foo(): pass", True), - ("x" * 501, True), + ( + "Explain step by step how to solve the integral of x^2 " + "and then write code to compute the derivative", + True, + ), ], ) def test_query_type_selects_expected_size( From 601942c9cec9b7d728347f508b5dc9a9d2a8796a Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 15:52:14 -0700 Subject: [PATCH 37/92] style: wrap long logger.warning line to fix E501 lint error Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/server/agent_manager_routes.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/openjarvis/server/agent_manager_routes.py b/src/openjarvis/server/agent_manager_routes.py index d5515bd0..714a168d 100644 --- a/src/openjarvis/server/agent_manager_routes.py +++ b/src/openjarvis/server/agent_manager_routes.py @@ -265,7 +265,10 @@ async def _stream_managed_agent( try: agent = agent_cls(**agent_kwargs) except TypeError as exc: - logger.warning("Agent instantiation failed with all kwargs, retrying minimal: %s", exc) + logger.warning( + "Agent instantiation failed with all kwargs, retrying minimal: %s", + exc, + ) agent = agent_cls(engine=engine, model=model) # Build conversation context from existing messages From 406c115a5280cc6dfcb464c9b47556c941e41ace Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:12:36 -0700 Subject: [PATCH 38/92] fix: register all tool modules so they appear in web UI (#99) (#107) Eight tool modules (file_write, apply_patch, git_tool, db_query, pdf_tool, image_tool, audio_tool, knowledge_tools) were missing from openjarvis/tools/__init__.py. Their @ToolRegistry.register() decorators never fired, so the /v1/tools endpoint never returned them and the web UI agent wizard showed an incomplete tool list. Add the missing imports and a regression test that checks all expected tool names are in the registry after package import. Co-authored-by: Claude Opus 4.6 (1M context) --- src/openjarvis/tools/__init__.py | 40 ++++++++++ tests/tools/test_tool_registration.py | 102 ++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100644 tests/tools/test_tool_registration.py diff --git a/src/openjarvis/tools/__init__.py b/src/openjarvis/tools/__init__.py index 0c407a3a..522e5fde 100644 --- a/src/openjarvis/tools/__init__.py +++ b/src/openjarvis/tools/__init__.py @@ -91,4 +91,44 @@ try: except ImportError: pass +try: + import openjarvis.tools.file_write # noqa: F401 +except ImportError: + pass + +try: + import openjarvis.tools.apply_patch # noqa: F401 +except ImportError: + pass + +try: + import openjarvis.tools.git_tool # noqa: F401 +except ImportError: + pass + +try: + import openjarvis.tools.db_query # noqa: F401 +except ImportError: + pass + +try: + import openjarvis.tools.pdf_tool # noqa: F401 +except ImportError: + pass + +try: + import openjarvis.tools.image_tool # noqa: F401 +except ImportError: + pass + +try: + import openjarvis.tools.audio_tool # noqa: F401 +except ImportError: + pass + +try: + import openjarvis.tools.knowledge_tools # noqa: F401 +except ImportError: + pass + __all__ = ["BaseTool", "ToolExecutor", "ToolSpec"] diff --git a/tests/tools/test_tool_registration.py b/tests/tools/test_tool_registration.py new file mode 100644 index 00000000..1369678b --- /dev/null +++ b/tests/tools/test_tool_registration.py @@ -0,0 +1,102 @@ +"""Verify that importing openjarvis.tools registers all built-in tools.""" + +from __future__ import annotations + +import importlib +import sys + +from openjarvis.core.registry import ToolRegistry + +# Every tool name that should be registered after importing the package. +EXPECTED_TOOLS = { + # calculator.py + "calculator", + # think.py + "think", + # retrieval.py + "retrieval", + # llm_tool.py + "llm", + # file_read.py + "file_read", + # file_write.py + "file_write", + # web_search.py + "web_search", + # code_interpreter.py + "code_interpreter", + # code_interpreter_docker.py + "code_interpreter_docker", + # repl.py + "repl", + # storage_tools.py + "memory_store", + "memory_retrieve", + "memory_search", + "memory_index", + # channel_tools.py + "channel_send", + "channel_list", + "channel_status", + # http_request.py + "http_request", + # shell_exec.py + "shell_exec", + # memory_manage.py + "memory_manage", + # user_profile_manage.py + "user_profile_manage", + # skill_manage.py + "skill_manage", + # apply_patch.py + "apply_patch", + # git_tool.py + "git_status", + "git_diff", + "git_commit", + "git_log", + # db_query.py + "db_query", + # pdf_tool.py + "pdf_extract", + # image_tool.py + "image_generate", + # audio_tool.py + "audio_transcribe", + # knowledge_tools.py + "kg_add_entity", + "kg_add_relation", + "kg_query", + "kg_neighbors", +} + + +def _reload_tool_modules() -> None: + """Reload all openjarvis.tools.* submodules to re-trigger @register decorators. + + The autouse ``_clean_registries`` fixture clears all registries before each + test. A plain ``import openjarvis.tools`` won't re-register because the + submodules are already cached in ``sys.modules``. We must reload the + individual submodules so their class-level ``@ToolRegistry.register`` + decorators execute again. + """ + for mod_name in list(sys.modules): + if ( + mod_name.startswith("openjarvis.tools.") + and not mod_name.endswith("_stubs") + and not mod_name.endswith("agent_tools") + ): + try: + importlib.reload(sys.modules[mod_name]) + except Exception: + pass + + +def test_all_builtin_tools_registered(): + _reload_tool_modules() + + registered = set(ToolRegistry.keys()) + missing = EXPECTED_TOOLS - registered + assert not missing, ( + f"Tools not registered (missing import in __init__.py?): {sorted(missing)}" + ) From 0481f32129833d57b254e726985c3ee27f0494d2 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:12:47 -0700 Subject: [PATCH 39/92] perf: cache load_config() with lru_cache to avoid repeated hardware detection Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/core/config.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 3db5bc53..76ca3c8d 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -7,6 +7,7 @@ found in the TOML file. from __future__ import annotations +import functools import os import platform import shutil @@ -1189,6 +1190,7 @@ def _migrate_toml_data(data: Dict[str, Any], cfg: "JarvisConfig") -> None: ) +@functools.lru_cache(maxsize=1) def load_config(path: Optional[Path] = None) -> JarvisConfig: """Detect hardware, build defaults, overlay TOML overrides. From 7b5f078b4b6f519f722ff109b9ee86ee786ef5cb Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:14:31 -0700 Subject: [PATCH 40/92] test: add failing tests for agent config-based default resolution (issue #103) Co-Authored-By: Claude Opus 4.6 (1M context) --- tests/agents/test_config_defaults.py | 167 +++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 tests/agents/test_config_defaults.py diff --git a/tests/agents/test_config_defaults.py b/tests/agents/test_config_defaults.py new file mode 100644 index 00000000..95493bdd --- /dev/null +++ b/tests/agents/test_config_defaults.py @@ -0,0 +1,167 @@ +"""Tests for agent constructor config-based default resolution.""" + +from __future__ import annotations + +from unittest.mock import MagicMock, patch + +from openjarvis.agents._stubs import AgentResult, BaseAgent, ToolUsingAgent + + +class _TestAgent(BaseAgent): + agent_id = "test_cfg" + + def run(self, input, context=None, **kwargs): + return AgentResult(content="ok", turns=1) + + +class _TestToolAgent(ToolUsingAgent): + agent_id = "test_cfg_tool" + + def run(self, input, context=None, **kwargs): + return AgentResult(content="ok", turns=1) + + +class _TestToolAgentWithDefaults(ToolUsingAgent): + """Agent with class-level defaults (like MonitorOperativeAgent).""" + + agent_id = "test_cfg_tool_defaults" + _default_temperature = 0.3 + _default_max_tokens = 4096 + _default_max_turns = 25 + + def run(self, input, context=None, **kwargs): + return AgentResult(content="ok", turns=1) + + +class TestBaseAgentConfigResolution: + """BaseAgent resolves None params from config.intelligence.""" + + def test_none_temperature_reads_config(self): + """When temperature is not passed, it should come from config.""" + engine = MagicMock() + with patch("openjarvis.agents._stubs.load_config") as mock_cfg: + mock_cfg.return_value.intelligence.temperature = 0.2 + mock_cfg.return_value.intelligence.max_tokens = 512 + agent = _TestAgent(engine, "m") + assert agent._temperature == 0.2 + + def test_none_max_tokens_reads_config(self): + """When max_tokens is not passed, it should come from config.""" + engine = MagicMock() + with patch("openjarvis.agents._stubs.load_config") as mock_cfg: + mock_cfg.return_value.intelligence.temperature = 0.7 + mock_cfg.return_value.intelligence.max_tokens = 512 + agent = _TestAgent(engine, "m") + assert agent._max_tokens == 512 + + def test_explicit_temperature_overrides_config(self): + """Caller-provided temperature takes precedence over config.""" + engine = MagicMock() + agent = _TestAgent(engine, "m", temperature=0.9) + assert agent._temperature == 0.9 + + def test_explicit_max_tokens_overrides_config(self): + """Caller-provided max_tokens takes precedence over config.""" + engine = MagicMock() + agent = _TestAgent(engine, "m", max_tokens=2048) + assert agent._max_tokens == 2048 + + def test_partial_override_temperature_only(self): + """Providing only temperature still reads max_tokens from config.""" + engine = MagicMock() + with patch("openjarvis.agents._stubs.load_config") as mock_cfg: + mock_cfg.return_value.intelligence.temperature = 0.2 + mock_cfg.return_value.intelligence.max_tokens = 512 + agent = _TestAgent(engine, "m", temperature=0.9) + assert agent._temperature == 0.9 + assert agent._max_tokens == 512 + + def test_config_load_failure_uses_hardcoded_fallback(self): + """When config loading fails, fall back to class defaults then 0.7/1024.""" + engine = MagicMock() + with patch( + "openjarvis.agents._stubs.load_config", + side_effect=Exception("boom"), + ): + agent = _TestAgent(engine, "m") + assert agent._temperature == 0.7 + assert agent._max_tokens == 1024 + + +class TestToolUsingAgentConfigResolution: + """ToolUsingAgent resolves None max_turns from config.agent.""" + + def test_none_max_turns_reads_config(self): + """When max_turns is not passed, it should come from config.""" + engine = MagicMock() + with patch("openjarvis.agents._stubs.load_config") as mock_cfg: + mock_cfg.return_value.intelligence.temperature = 0.7 + mock_cfg.return_value.intelligence.max_tokens = 1024 + mock_cfg.return_value.agent.max_turns = 15 + agent = _TestToolAgent(engine, "m") + assert agent._max_turns == 15 + + def test_explicit_max_turns_overrides_config(self): + """Caller-provided max_turns takes precedence over config.""" + engine = MagicMock() + agent = _TestToolAgent(engine, "m", max_turns=5) + assert agent._max_turns == 5 + + def test_temperature_and_max_tokens_forwarded_to_base(self): + """ToolUsingAgent also resolves temperature/max_tokens from config.""" + engine = MagicMock() + with patch("openjarvis.agents._stubs.load_config") as mock_cfg: + mock_cfg.return_value.intelligence.temperature = 0.1 + mock_cfg.return_value.intelligence.max_tokens = 256 + mock_cfg.return_value.agent.max_turns = 10 + agent = _TestToolAgent(engine, "m") + assert agent._temperature == 0.1 + assert agent._max_tokens == 256 + + def test_config_load_failure_max_turns_fallback(self): + """When config loading fails, max_turns falls back to class default then 10.""" + engine = MagicMock() + with patch( + "openjarvis.agents._stubs.load_config", + side_effect=Exception("boom"), + ): + agent = _TestToolAgent(engine, "m") + assert agent._max_turns == 10 + + +class TestClassLevelDefaults: + """Agents with class-level _default_* attributes use them as fallback.""" + + def test_class_default_used_when_config_fails(self): + """Agent class defaults are used when config is unavailable.""" + engine = MagicMock() + with patch( + "openjarvis.agents._stubs.load_config", + side_effect=Exception("boom"), + ): + agent = _TestToolAgentWithDefaults(engine, "m") + assert agent._temperature == 0.3 + assert agent._max_tokens == 4096 + assert agent._max_turns == 25 + + def test_config_overrides_class_default(self): + """User config takes precedence over class-level defaults.""" + engine = MagicMock() + with patch("openjarvis.agents._stubs.load_config") as mock_cfg: + mock_cfg.return_value.intelligence.temperature = 0.5 + mock_cfg.return_value.intelligence.max_tokens = 2048 + mock_cfg.return_value.agent.max_turns = 12 + agent = _TestToolAgentWithDefaults(engine, "m") + assert agent._temperature == 0.5 + assert agent._max_tokens == 2048 + assert agent._max_turns == 12 + + def test_explicit_overrides_everything(self): + """Caller-provided values override both config and class defaults.""" + engine = MagicMock() + agent = _TestToolAgentWithDefaults( + engine, "m", temperature=0.9, max_tokens=100, max_turns=2, + ) + assert agent._temperature == 0.9 + assert agent._max_tokens == 100 + assert agent._max_turns == 2 From cc4621a840670b83e07642afab7e590e5b533393 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:15:59 -0700 Subject: [PATCH 41/92] fix: resolve agent defaults from config with class-level fallbacks (issue #103) Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/agents/_stubs.py | 48 +++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 8 deletions(-) diff --git a/src/openjarvis/agents/_stubs.py b/src/openjarvis/agents/_stubs.py index 9a91c90b..359be7f7 100644 --- a/src/openjarvis/agents/_stubs.py +++ b/src/openjarvis/agents/_stubs.py @@ -13,6 +13,7 @@ from abc import ABC, abstractmethod from dataclasses import dataclass, field from typing import Any, Dict, List, Optional +from openjarvis.core.config import load_config from openjarvis.core.events import EventBus, EventType from openjarvis.core.types import Conversation, Message, Role, ToolResult from openjarvis.engine._stubs import InferenceEngine @@ -63,17 +64,40 @@ class BaseAgent(ABC): model: str, *, bus: Optional[EventBus] = None, - temperature: float = 0.7, - max_tokens: int = 1024, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, prompt_builder: Optional[Any] = None, ) -> None: self._engine = engine self._model = model self._bus = bus - self._temperature = temperature - self._max_tokens = max_tokens self._prompt_builder = prompt_builder + # Three-tier resolution: explicit arg > config > class default > hardcoded + if temperature is not None and max_tokens is not None: + self._temperature = temperature + self._max_tokens = max_tokens + else: + try: + cfg = load_config() + self._temperature = ( + temperature if temperature is not None + else cfg.intelligence.temperature + ) + self._max_tokens = ( + max_tokens if max_tokens is not None + else cfg.intelligence.max_tokens + ) + except Exception: + self._temperature = ( + temperature if temperature is not None + else getattr(self, "_default_temperature", 0.7) + ) + self._max_tokens = ( + max_tokens if max_tokens is not None + else getattr(self, "_default_max_tokens", 1024) + ) + # ------------------------------------------------------------------ # Concrete helpers # ------------------------------------------------------------------ @@ -230,9 +254,9 @@ class ToolUsingAgent(BaseAgent): *, tools: Optional[List["BaseTool"]] = None, # noqa: F821 bus: Optional[EventBus] = None, - max_turns: int = 10, - temperature: float = 0.7, - max_tokens: int = 1024, + max_turns: Optional[int] = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, loop_guard_config: Optional[Any] = None, capability_policy: Optional[Any] = None, agent_id: Optional[str] = None, @@ -254,7 +278,15 @@ class ToolUsingAgent(BaseAgent): interactive=interactive, confirm_callback=confirm_callback, ) - self._max_turns = max_turns + # Resolve max_turns: explicit arg > config > class default > 10 + if max_turns is not None: + self._max_turns = max_turns + else: + try: + cfg = load_config() + self._max_turns = cfg.agent.max_turns + except Exception: + self._max_turns = getattr(self, "_default_max_turns", 10) # Loop guard self._loop_guard = None From 59dad5eddfe66719aa7ab49df1cccc9f5e6b1f6d Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:29:26 -0700 Subject: [PATCH 42/92] fix: address installation failures and SDK bugs (#100) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bug 1 — Document Rust toolchain requirement: - Add `maturin develop` step to README, installation docs (all 3 sections), and quickstart.sh - Note PYO3_USE_ABI3_FORWARD_COMPATIBILITY for Python 3.14+ Bug 2 — Fix namespace package conflict: - Move force-include targets from openjarvis/ to _node_modules/ in pyproject.toml to prevent editable-install namespace shadowing - Add fallback path resolution in claude_code.py and whatsapp_baileys.py for wheel installs Bug 3 — Fix trace debugger "No traces yet": - Enable traces by default (TracesConfig.enabled = True) - Fix api_routes.py: use store.list_traces() not .recent(), dataclasses.asdict() not .to_dict(), get store from app.state - Wire TraceStore into app.state in app.py Bug 4 — Fix thinking models returning empty responses: - Remove hardcoded enable_thinking: False from _openai_compat.py that suppressed Qwen3/DeepSeek-R1 output on OpenAI-compatible engines (vLLM, SGLang, llama.cpp) Closes #100 Co-authored-by: Claude Opus 4.6 (1M context) --- README.md | 5 +++++ docs/getting-started/installation.md | 11 ++++++++++ pyproject.toml | 4 ++-- scripts/quickstart.sh | 6 ++++++ src/openjarvis/agents/claude_code.py | 10 ++++++++- src/openjarvis/channels/whatsapp_baileys.py | 8 +++++++ src/openjarvis/core/config.py | 2 +- src/openjarvis/engine/_openai_compat.py | 1 - src/openjarvis/server/api_routes.py | 23 ++++++++++++--------- src/openjarvis/server/app.py | 12 +++++++++++ 10 files changed, 67 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 0b885e24..54e79dcd 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,13 @@ git clone https://github.com/open-jarvis/OpenJarvis.git cd OpenJarvis uv sync # core framework uv sync --extra server # + FastAPI server + +# Build the Rust extension (requires Rust: https://rustup.rs/) +uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml ``` +> **Python 3.14+:** set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command. + You also need a local inference backend: [Ollama](https://ollama.com), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), or [llama.cpp](https://github.com/ggerganov/llama.cpp). Alternatively, use the `cloud` engine with [OpenAI](https://openai.com), [Anthropic](https://anthropic.com), [Google Gemini](https://ai.google.dev), [OpenRouter](https://openrouter.ai), or [MiniMax](https://www.minimax.io) by setting the corresponding API key environment variable. ## Quick Start diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 9324dbe0..20598dd7 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -42,9 +42,14 @@ If you prefer to run each step yourself: git clone https://github.com/open-jarvis/OpenJarvis.git cd OpenJarvis uv sync --extra server + uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml cd frontend && npm install && cd .. ``` + !!! note "Prerequisites" + Requires [Rust](https://rustup.rs/) (`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`). + On Python 3.14+, set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command. + === "Step 2: Start Ollama" ```bash @@ -131,8 +136,11 @@ programmatically. Every feature is accessible from the terminal. git clone https://github.com/open-jarvis/OpenJarvis.git cd OpenJarvis uv sync +uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml ``` +Requires [Rust](https://rustup.rs/). On Python 3.14+, set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command. + ### Verify ```bash @@ -172,8 +180,11 @@ For programmatic access, the `Jarvis` class provides a high-level sync API. git clone https://github.com/open-jarvis/OpenJarvis.git cd OpenJarvis uv sync +uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml ``` +Requires [Rust](https://rustup.rs/). On Python 3.14+, set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command. + ### Quick example ```python diff --git a/pyproject.toml b/pyproject.toml index d6adcaba..f13232dc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -108,8 +108,8 @@ jarvis = "openjarvis.cli:main" packages = ["src/openjarvis"] [tool.hatch.build.targets.wheel.force-include] -"src/openjarvis/agents/claude_code_runner" = "openjarvis/agents/claude_code_runner" -"src/openjarvis/channels/whatsapp_baileys_bridge" = "openjarvis/channels/whatsapp_baileys_bridge" +"src/openjarvis/agents/claude_code_runner" = "_node_modules/claude_code_runner" +"src/openjarvis/channels/whatsapp_baileys_bridge" = "_node_modules/whatsapp_baileys_bridge" [tool.pytest.ini_options] testpaths = ["tests"] diff --git a/scripts/quickstart.sh b/scripts/quickstart.sh index 1a89b9df..157bf0b6 100755 --- a/scripts/quickstart.sh +++ b/scripts/quickstart.sh @@ -146,6 +146,12 @@ info "Installing Python dependencies..." uv sync --extra server --quiet 2>/dev/null || uv sync --extra server ok "Python dependencies installed" +# ── 7b. Build Rust extension ────────────────────────────────────── +info "Building Rust extension..." +uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml --quiet 2>/dev/null \ + || uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml +ok "Rust extension built" + # ── 8. Install frontend dependencies ──────────────────────────────── info "Installing frontend dependencies..." (cd frontend && npm install --silent 2>/dev/null || npm install) diff --git a/src/openjarvis/agents/claude_code.py b/src/openjarvis/agents/claude_code.py index 060c226f..59aaefc1 100644 --- a/src/openjarvis/agents/claude_code.py +++ b/src/openjarvis/agents/claude_code.py @@ -29,8 +29,16 @@ logger = logging.getLogger(__name__) _OUTPUT_START = "---OPENJARVIS_OUTPUT_START---" _OUTPUT_END = "---OPENJARVIS_OUTPUT_END---" -# Path to the bundled runner source (relative to this module) +# Path to the bundled runner source (relative to this module). +# In editable installs this lives next to this file; in wheel installs +# it is placed under _node_modules/ to avoid namespace package conflicts. _RUNNER_SRC = Path(__file__).resolve().parent / "claude_code_runner" +if not _RUNNER_SRC.exists(): + _RUNNER_SRC = ( + Path(__file__).resolve().parents[2] + / "_node_modules" + / "claude_code_runner" + ) @AgentRegistry.register("claude_code") diff --git a/src/openjarvis/channels/whatsapp_baileys.py b/src/openjarvis/channels/whatsapp_baileys.py index 0bb182be..bfbdd704 100644 --- a/src/openjarvis/channels/whatsapp_baileys.py +++ b/src/openjarvis/channels/whatsapp_baileys.py @@ -27,7 +27,15 @@ from openjarvis.core.registry import ChannelRegistry logger = logging.getLogger(__name__) # Path to the bundled bridge shipped inside the package. +# In editable installs this lives next to this file; in wheel installs +# it is placed under _node_modules/ to avoid namespace package conflicts. _BRIDGE_SRC = Path(__file__).resolve().parent / "whatsapp_baileys_bridge" +if not _BRIDGE_SRC.exists(): + _BRIDGE_SRC = ( + Path(__file__).resolve().parents[2] + / "_node_modules" + / "whatsapp_baileys_bridge" + ) # Default runtime directory (npm install + auth state). _DEFAULT_RUNTIME_DIR = Path.home() / ".openjarvis" / "whatsapp_baileys_bridge" diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 3db5bc53..5ca8fa77 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -762,7 +762,7 @@ class TelemetryConfig: class TracesConfig: """Trace system settings.""" - enabled: bool = False + enabled: bool = True db_path: str = str(DEFAULT_CONFIG_DIR / "traces.db") diff --git a/src/openjarvis/engine/_openai_compat.py b/src/openjarvis/engine/_openai_compat.py index fcc08034..ba7d3870 100644 --- a/src/openjarvis/engine/_openai_compat.py +++ b/src/openjarvis/engine/_openai_compat.py @@ -68,7 +68,6 @@ class _OpenAICompatibleEngine(InferenceEngine): "temperature": temperature, "max_tokens": max_tokens, "stream": False, - "chat_template_kwargs": {"enable_thinking": False}, **kwargs, } try: diff --git a/src/openjarvis/server/api_routes.py b/src/openjarvis/server/api_routes.py index 24b5a7fe..ca6c93c8 100644 --- a/src/openjarvis/server/api_routes.py +++ b/src/openjarvis/server/api_routes.py @@ -179,13 +179,13 @@ traces_router = APIRouter(prefix="/v1/traces", tags=["traces"]) async def list_traces(request: Request, limit: int = 20): """List recent traces.""" try: - from openjarvis.traces.store import TraceStore - store = TraceStore() - traces = store.recent(limit=limit) - items = [ - t.to_dict() if hasattr(t, "to_dict") else str(t) - for t in traces - ] + from dataclasses import asdict + + store = getattr(request.app.state, "trace_store", None) + if store is None: + return {"traces": []} + traces = store.list_traces(limit=limit) + items = [asdict(t) for t in traces] return {"traces": items} except Exception as exc: return {"traces": [], "error": str(exc)} @@ -194,12 +194,15 @@ async def list_traces(request: Request, limit: int = 20): async def get_trace(trace_id: str, request: Request): """Get a specific trace by ID.""" try: - from openjarvis.traces.store import TraceStore - store = TraceStore() + from dataclasses import asdict + + store = getattr(request.app.state, "trace_store", None) + if store is None: + raise HTTPException(status_code=404, detail="Trace not found") trace = store.get(trace_id) if trace is None: raise HTTPException(status_code=404, detail="Trace not found") - return trace.to_dict() if hasattr(trace, 'to_dict') else {"id": trace_id} + return asdict(trace) except HTTPException: raise except Exception as exc: diff --git a/src/openjarvis/server/app.py b/src/openjarvis/server/app.py index dce15f4f..e00f0945 100644 --- a/src/openjarvis/server/app.py +++ b/src/openjarvis/server/app.py @@ -110,6 +110,18 @@ def create_app( app.state.agent_scheduler = agent_scheduler app.state.session_start = time.time() + # Wire up trace store if traces are enabled + app.state.trace_store = None + try: + from openjarvis.core.config import load_config + from openjarvis.traces.store import TraceStore + + cfg = config if config is not None else load_config() + if cfg.traces.enabled: + app.state.trace_store = TraceStore(db_path=cfg.traces.db_path) + except Exception: + pass # traces are optional; don't block server startup + app.include_router(router) app.include_router(dashboard_router) app.include_router(comparison_router) From 1fe75b28077c9a73a5e5d1156c8926ab3b48b81c Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:33:33 -0700 Subject: [PATCH 43/92] refactor: move agent defaults to class attrs, accept Optional params (issue #103) Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/agents/claude_code.py | 6 ++++-- src/openjarvis/agents/monitor_operative.py | 9 ++++++--- src/openjarvis/agents/native_openhands.py | 9 ++++++--- src/openjarvis/agents/native_react.py | 9 ++++++--- src/openjarvis/agents/openhands.py | 6 ++++-- src/openjarvis/agents/operative.py | 9 ++++++--- src/openjarvis/agents/orchestrator.py | 9 ++++++--- src/openjarvis/agents/rlm.py | 9 ++++++--- 8 files changed, 44 insertions(+), 22 deletions(-) diff --git a/src/openjarvis/agents/claude_code.py b/src/openjarvis/agents/claude_code.py index 060c226f..9800bd81 100644 --- a/src/openjarvis/agents/claude_code.py +++ b/src/openjarvis/agents/claude_code.py @@ -47,6 +47,8 @@ class ClaudeCodeAgent(BaseAgent): agent_id = "claude_code" accepts_tools = False + _default_temperature = 0.7 + _default_max_tokens = 1024 def __init__( self, @@ -54,8 +56,8 @@ class ClaudeCodeAgent(BaseAgent): model: str, *, bus: Optional[EventBus] = None, - temperature: float = 0.7, - max_tokens: int = 1024, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, api_key: str = "", workspace: str = "", session_id: str = "", diff --git a/src/openjarvis/agents/monitor_operative.py b/src/openjarvis/agents/monitor_operative.py index a764ca35..a5acf288 100644 --- a/src/openjarvis/agents/monitor_operative.py +++ b/src/openjarvis/agents/monitor_operative.py @@ -84,6 +84,9 @@ class MonitorOperativeAgent(ToolUsingAgent): agent_id = "monitor_operative" accepts_tools = True + _default_temperature = 0.3 + _default_max_tokens = 4096 + _default_max_turns = 25 def __init__( self, @@ -92,9 +95,9 @@ class MonitorOperativeAgent(ToolUsingAgent): *, tools: Optional[List[BaseTool]] = None, bus: Optional[EventBus] = None, - max_turns: int = 25, - temperature: float = 0.3, - max_tokens: int = 4096, + max_turns: Optional[int] = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, system_prompt: Optional[str] = None, # Strategy parameters memory_extraction: str = "causality_graph", diff --git a/src/openjarvis/agents/native_openhands.py b/src/openjarvis/agents/native_openhands.py index 46570d53..b80663bd 100644 --- a/src/openjarvis/agents/native_openhands.py +++ b/src/openjarvis/agents/native_openhands.py @@ -54,6 +54,9 @@ class NativeOpenHandsAgent(ToolUsingAgent): """Native CodeAct agent -- generates and executes Python code.""" agent_id = "native_openhands" + _default_temperature = 0.7 + _default_max_tokens = 2048 + _default_max_turns = 3 def __init__( self, @@ -62,9 +65,9 @@ class NativeOpenHandsAgent(ToolUsingAgent): *, tools: Optional[List[BaseTool]] = None, bus: Optional[EventBus] = None, - max_turns: int = 3, - temperature: float = 0.7, - max_tokens: int = 2048, + max_turns: Optional[int] = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, interactive: bool = False, confirm_callback=None, ) -> None: diff --git a/src/openjarvis/agents/native_react.py b/src/openjarvis/agents/native_react.py index cfcc5023..49e8c37a 100644 --- a/src/openjarvis/agents/native_react.py +++ b/src/openjarvis/agents/native_react.py @@ -36,6 +36,9 @@ class NativeReActAgent(ToolUsingAgent): """ReAct agent: Thought -> Action -> Observation loop.""" agent_id = "native_react" + _default_temperature = 0.7 + _default_max_tokens = 1024 + _default_max_turns = 10 def __init__( self, @@ -44,9 +47,9 @@ class NativeReActAgent(ToolUsingAgent): *, tools: Optional[List[BaseTool]] = None, bus: Optional[EventBus] = None, - max_turns: int = 10, - temperature: float = 0.7, - max_tokens: int = 1024, + max_turns: Optional[int] = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, interactive: bool = False, confirm_callback=None, ) -> None: diff --git a/src/openjarvis/agents/openhands.py b/src/openjarvis/agents/openhands.py index 87dad601..997f0844 100644 --- a/src/openjarvis/agents/openhands.py +++ b/src/openjarvis/agents/openhands.py @@ -25,6 +25,8 @@ class OpenHandsAgent(BaseAgent): """ agent_id = "openhands" + _default_temperature = 0.7 + _default_max_tokens = 1024 def __init__( self, @@ -32,8 +34,8 @@ class OpenHandsAgent(BaseAgent): model: str, *, bus: Optional[EventBus] = None, - temperature: float = 0.7, - max_tokens: int = 1024, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, workspace: Optional[str] = None, api_key: Optional[str] = None, ) -> None: diff --git a/src/openjarvis/agents/operative.py b/src/openjarvis/agents/operative.py index 7f88f840..707934b7 100644 --- a/src/openjarvis/agents/operative.py +++ b/src/openjarvis/agents/operative.py @@ -38,6 +38,9 @@ class OperativeAgent(ToolUsingAgent): agent_id = "operative" accepts_tools = True + _default_temperature = 0.3 + _default_max_tokens = 2048 + _default_max_turns = 20 def __init__( self, @@ -46,9 +49,9 @@ class OperativeAgent(ToolUsingAgent): *, tools: Optional[List[BaseTool]] = None, bus: Optional[EventBus] = None, - max_turns: int = 20, - temperature: float = 0.3, - max_tokens: int = 2048, + max_turns: Optional[int] = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, system_prompt: Optional[str] = None, operator_id: Optional[str] = None, session_store: Optional[Any] = None, diff --git a/src/openjarvis/agents/orchestrator.py b/src/openjarvis/agents/orchestrator.py index ffbff90e..90125ae1 100644 --- a/src/openjarvis/agents/orchestrator.py +++ b/src/openjarvis/agents/orchestrator.py @@ -41,6 +41,9 @@ class OrchestratorAgent(ToolUsingAgent): """ agent_id = "orchestrator" + _default_temperature = 0.7 + _default_max_tokens = 1024 + _default_max_turns = 10 def __init__( self, @@ -49,9 +52,9 @@ class OrchestratorAgent(ToolUsingAgent): *, tools: Optional[List[BaseTool]] = None, bus: Optional[EventBus] = None, - max_turns: int = 10, - temperature: float = 0.7, - max_tokens: int = 1024, + max_turns: Optional[int] = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, mode: str = "function_calling", system_prompt: Optional[str] = None, parallel_tools: bool = True, diff --git a/src/openjarvis/agents/rlm.py b/src/openjarvis/agents/rlm.py index 0cc6e3a4..43488b8c 100644 --- a/src/openjarvis/agents/rlm.py +++ b/src/openjarvis/agents/rlm.py @@ -84,6 +84,9 @@ class RLMAgent(ToolUsingAgent): """ agent_id = "rlm" + _default_temperature = 0.7 + _default_max_tokens = 2048 + _default_max_turns = 10 def __init__( self, @@ -92,9 +95,9 @@ class RLMAgent(ToolUsingAgent): *, tools: Optional[List[BaseTool]] = None, bus: Optional[EventBus] = None, - max_turns: int = 10, - temperature: float = 0.7, - max_tokens: int = 2048, + max_turns: Optional[int] = None, + temperature: Optional[float] = None, + max_tokens: Optional[int] = None, sub_model: Optional[str] = None, sub_temperature: float = 0.3, sub_max_tokens: int = 1024, From 9624f53c1858c70c67fd43a243d2ad92862eca90 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:43:51 -0700 Subject: [PATCH 44/92] style: apply ruff format to changed files Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/agents/_stubs.py | 26 ++++-- src/openjarvis/agents/claude_code.py | 12 ++- src/openjarvis/agents/monitor_operative.py | 80 ++++++++++++------- src/openjarvis/agents/native_openhands.py | 36 +++++---- src/openjarvis/agents/native_react.py | 14 +++- src/openjarvis/agents/openhands.py | 7 +- src/openjarvis/agents/operative.py | 61 ++++++++------ src/openjarvis/agents/orchestrator.py | 93 ++++++++++++---------- src/openjarvis/agents/rlm.py | 51 +++++++----- src/openjarvis/core/config.py | 84 +++++++++++-------- tests/agents/test_config_defaults.py | 6 +- 11 files changed, 285 insertions(+), 185 deletions(-) diff --git a/src/openjarvis/agents/_stubs.py b/src/openjarvis/agents/_stubs.py index 359be7f7..66b8755e 100644 --- a/src/openjarvis/agents/_stubs.py +++ b/src/openjarvis/agents/_stubs.py @@ -81,20 +81,24 @@ class BaseAgent(ABC): try: cfg = load_config() self._temperature = ( - temperature if temperature is not None + temperature + if temperature is not None else cfg.intelligence.temperature ) self._max_tokens = ( - max_tokens if max_tokens is not None + max_tokens + if max_tokens is not None else cfg.intelligence.max_tokens ) except Exception: self._temperature = ( - temperature if temperature is not None + temperature + if temperature is not None else getattr(self, "_default_temperature", 0.7) ) self._max_tokens = ( - max_tokens if max_tokens is not None + max_tokens + if max_tokens is not None else getattr(self, "_default_max_tokens", 1024) ) @@ -221,7 +225,9 @@ class BaseAgent(ABC): """ # Full ... blocks text = re.sub( - r".*?\s*", "", text, + r".*?\s*", + "", + text, flags=re.DOTALL | re.IGNORECASE, ) # Leading content before a bare (no opening tag) @@ -264,15 +270,19 @@ class ToolUsingAgent(BaseAgent): confirm_callback: Optional[Any] = None, ) -> None: super().__init__( - engine, model, bus=bus, - temperature=temperature, max_tokens=max_tokens, + engine, + model, + bus=bus, + temperature=temperature, + max_tokens=max_tokens, ) from openjarvis.tools._stubs import ToolExecutor self._tools = tools or [] _aid = agent_id or getattr(self, "agent_id", "") self._executor = ToolExecutor( - self._tools, bus=bus, + self._tools, + bus=bus, capability_policy=capability_policy, agent_id=_aid, interactive=interactive, diff --git a/src/openjarvis/agents/claude_code.py b/src/openjarvis/agents/claude_code.py index 9800bd81..5eb61b6d 100644 --- a/src/openjarvis/agents/claude_code.py +++ b/src/openjarvis/agents/claude_code.py @@ -66,8 +66,11 @@ class ClaudeCodeAgent(BaseAgent): timeout: int = 300, ) -> None: super().__init__( - engine, model, bus=bus, - temperature=temperature, max_tokens=max_tokens, + engine, + model, + bus=bus, + temperature=temperature, + max_tokens=max_tokens, ) self._api_key = api_key or os.environ.get("ANTHROPIC_API_KEY", "") self._workspace = workspace or os.getcwd() @@ -172,7 +175,8 @@ class ClaudeCodeAgent(BaseAgent): stderr = proc.stderr.strip() if proc.stderr else "Unknown error" logger.error( "claude_code_runner exited with code %d: %s", - proc.returncode, stderr, + proc.returncode, + stderr, ) self._emit_turn_end(turns=1, error=True) return AgentResult( @@ -211,7 +215,7 @@ class ClaudeCodeAgent(BaseAgent): # No sentinels -- treat entire stdout as plain content return stdout.strip(), [], {} - json_str = stdout[start + len(_OUTPUT_START):end].strip() + json_str = stdout[start + len(_OUTPUT_START) : end].strip() try: data = json.loads(json_str) diff --git a/src/openjarvis/agents/monitor_operative.py b/src/openjarvis/agents/monitor_operative.py index a5acf288..3965e3d4 100644 --- a/src/openjarvis/agents/monitor_operative.py +++ b/src/openjarvis/agents/monitor_operative.py @@ -113,10 +113,15 @@ class MonitorOperativeAgent(ToolUsingAgent): **kwargs: Any, ) -> None: super().__init__( - engine, model, tools=tools, bus=bus, - max_turns=max_turns, temperature=temperature, + engine, + model, + tools=tools, + bus=bus, + max_turns=max_turns, + temperature=temperature, max_tokens=max_tokens, - interactive=interactive, confirm_callback=confirm_callback, + interactive=interactive, + confirm_callback=confirm_callback, ) # Validate strategies if memory_extraction not in VALID_MEMORY_EXTRACTION: @@ -194,7 +199,8 @@ class MonitorOperativeAgent(ToolUsingAgent): # 4. Build messages messages = self._build_operative_messages( - input, context, + input, + context, system_prompt=system_prompt, session_messages=session_messages, ) @@ -241,18 +247,21 @@ class MonitorOperativeAgent(ToolUsingAgent): ] # Append assistant message with tool calls - messages.append(Message( - role=Role.ASSISTANT, - content=content, - tool_calls=tool_calls, - )) + messages.append( + Message( + role=Role.ASSISTANT, + content=content, + tool_calls=tool_calls, + ) + ) # Execute each tool for tc in tool_calls: # Loop guard check if self._loop_guard: verdict = self._loop_guard.check_call( - tc.name, tc.arguments, + tc.name, + tc.arguments, ) if verdict.blocked: tool_result = ToolResult( @@ -261,12 +270,14 @@ class MonitorOperativeAgent(ToolUsingAgent): success=False, ) all_tool_results.append(tool_result) - messages.append(Message( - role=Role.TOOL, - content=tool_result.content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=tool_result.content, + tool_call_id=tc.id, + name=tc.name, + ) + ) continue tool_result = self._executor.execute(tc) @@ -282,18 +293,21 @@ class MonitorOperativeAgent(ToolUsingAgent): except (json.JSONDecodeError, TypeError) as exc: logger.debug( "Failed to parse tool call arguments" - " for state tracking: %s", exc, + " for state tracking: %s", + exc, ) # Compress observation if strategy requires it observation_content = self._compress_observation(tool_result.content) - messages.append(Message( - role=Role.TOOL, - content=observation_content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=observation_content, + tool_call_id=tc.id, + name=tc.name, + ) + ) # Extract and store findings based on memory strategy self._extract_and_store(tc.name, tool_result.content) @@ -301,7 +315,9 @@ class MonitorOperativeAgent(ToolUsingAgent): # Max turns exceeded self._save_session(input, content) return self._max_turns_result( - all_tool_results, turns, content=content, + all_tool_results, + turns, + content=content, metadata=total_usage, ) @@ -348,6 +364,7 @@ class MonitorOperativeAgent(ToolUsingAgent): if not self._tools: return "" from openjarvis.tools._stubs import build_tool_descriptions + return build_tool_descriptions(self._tools) # ------------------------------------------------------------------ @@ -461,11 +478,13 @@ class MonitorOperativeAgent(ToolUsingAgent): self._memory_backend.store(key, value) except Exception as exc: logger.debug( - "Failed to store causality relation in memory: %s", exc, + "Failed to store causality relation in memory: %s", + exc, ) except (json.JSONDecodeError, Exception): logger.debug( - "Causality extraction failed for tool %s output", tool_name, + "Causality extraction failed for tool %s output", + tool_name, ) def _store_scratchpad(self, tool_name: str, content: str) -> None: @@ -506,7 +525,8 @@ class MonitorOperativeAgent(ToolUsingAgent): except Exception as exc: logger.debug( "Failed to store structured data for tool %s: %s", - tool_name, exc, + tool_name, + exc, ) # ------------------------------------------------------------------ @@ -560,10 +580,12 @@ class MonitorOperativeAgent(ToolUsingAgent): session_id = f"monitor_operative:{self._operator_id}" try: self._session_store.save_message( - session_id, {"role": "user", "content": input_text}, + session_id, + {"role": "user", "content": input_text}, ) self._session_store.save_message( - session_id, {"role": "assistant", "content": response}, + session_id, + {"role": "assistant", "content": response}, ) except Exception: logger.debug( diff --git a/src/openjarvis/agents/native_openhands.py b/src/openjarvis/agents/native_openhands.py index b80663bd..cf5579cd 100644 --- a/src/openjarvis/agents/native_openhands.py +++ b/src/openjarvis/agents/native_openhands.py @@ -72,10 +72,15 @@ class NativeOpenHandsAgent(ToolUsingAgent): confirm_callback=None, ) -> None: super().__init__( - engine, model, tools=tools, bus=bus, - max_turns=max_turns, temperature=temperature, + engine, + model, + tools=tools, + bus=bus, + max_turns=max_turns, + temperature=temperature, max_tokens=max_tokens, - interactive=interactive, confirm_callback=confirm_callback, + interactive=interactive, + confirm_callback=confirm_callback, ) @staticmethod @@ -96,9 +101,7 @@ class NativeOpenHandsAgent(ToolUsingAgent): content = WebSearchTool._fetch_url(url, max_chars=4000) header = f"\n\n--- Content from {url} ---\n" footer = "\n--- End of content ---\n" - expanded = text.replace( - url, f"{header}{content}{footer}" - ) + expanded = text.replace(url, f"{header}{content}{footer}") return expanded, True except Exception: return text, False @@ -124,9 +127,7 @@ class NativeOpenHandsAgent(ToolUsingAgent): messages[i] = Message( role=Role.USER, content=( - truncated - + "\n\n[Input truncated" - " to fit context window]" + truncated + "\n\n[Input truncated to fit context window]" ), ) break @@ -138,7 +139,9 @@ class NativeOpenHandsAgent(ToolUsingAgent): # Remove Action: ... Action Input: ... blocks text = re.sub( r"Action:\s*.+?(?:Action Input:\s*.+?)?(?=\n\n|\Z)", - "", text, flags=re.DOTALL | re.IGNORECASE, + "", + text, + flags=re.DOTALL | re.IGNORECASE, ) # Remove ... or blocks text = re.sub(r".*?", "", text, flags=re.DOTALL) @@ -245,7 +248,9 @@ class NativeOpenHandsAgent(ToolUsingAgent): usage = result.get("usage", {}) self._emit_turn_end(turns=1) return AgentResult( - content=content, tool_results=[], turns=1, + content=content, + tool_results=[], + turns=1, metadata={ "prompt_tokens": usage.get("prompt_tokens", 0), "completion_tokens": usage.get("completion_tokens", 0), @@ -261,10 +266,7 @@ class NativeOpenHandsAgent(ToolUsingAgent): "Please try a shorter message." ) else: - error_msg = ( - "The model returned an error: " - + error_str - ) + error_msg = "The model returned an error: " + error_str self._emit_turn_end(turns=1, error=True) return AgentResult( content=error_msg, @@ -361,7 +363,9 @@ class NativeOpenHandsAgent(ToolUsingAgent): content = self._strip_tool_call_text(content) self._emit_turn_end(turns=turns) return AgentResult( - content=content, tool_results=all_tool_results, turns=turns, + content=content, + tool_results=all_tool_results, + turns=turns, metadata=total_usage, ) diff --git a/src/openjarvis/agents/native_react.py b/src/openjarvis/agents/native_react.py index 49e8c37a..75cd8efc 100644 --- a/src/openjarvis/agents/native_react.py +++ b/src/openjarvis/agents/native_react.py @@ -54,10 +54,15 @@ class NativeReActAgent(ToolUsingAgent): confirm_callback=None, ) -> None: super().__init__( - engine, model, tools=tools, bus=bus, - max_turns=max_turns, temperature=temperature, + engine, + model, + tools=tools, + bus=bus, + max_turns=max_turns, + temperature=temperature, max_tokens=max_tokens, - interactive=interactive, confirm_callback=confirm_callback, + interactive=interactive, + confirm_callback=confirm_callback, ) def _parse_response(self, text: str) -> dict: @@ -164,7 +169,8 @@ class NativeReActAgent(ToolUsingAgent): # Loop guard check before execution if self._loop_guard: verdict = self._loop_guard.check_call( - tool_call.name, tool_call.arguments, + tool_call.name, + tool_call.arguments, ) if verdict.blocked: tool_result = ToolResult( diff --git a/src/openjarvis/agents/openhands.py b/src/openjarvis/agents/openhands.py index 997f0844..4432b655 100644 --- a/src/openjarvis/agents/openhands.py +++ b/src/openjarvis/agents/openhands.py @@ -40,8 +40,11 @@ class OpenHandsAgent(BaseAgent): api_key: Optional[str] = None, ) -> None: super().__init__( - engine, model, bus=bus, - temperature=temperature, max_tokens=max_tokens, + engine, + model, + bus=bus, + temperature=temperature, + max_tokens=max_tokens, ) self._workspace = workspace or os.getcwd() self._api_key = api_key or os.environ.get("LLM_API_KEY", "") diff --git a/src/openjarvis/agents/operative.py b/src/openjarvis/agents/operative.py index 707934b7..02c1d725 100644 --- a/src/openjarvis/agents/operative.py +++ b/src/openjarvis/agents/operative.py @@ -61,10 +61,15 @@ class OperativeAgent(ToolUsingAgent): **kwargs: Any, ) -> None: super().__init__( - engine, model, tools=tools, bus=bus, - max_turns=max_turns, temperature=temperature, + engine, + model, + tools=tools, + bus=bus, + max_turns=max_turns, + temperature=temperature, max_tokens=max_tokens, - interactive=interactive, confirm_callback=confirm_callback, + interactive=interactive, + confirm_callback=confirm_callback, ) self._system_prompt = system_prompt or "" self._operator_id = operator_id @@ -97,7 +102,9 @@ class OperativeAgent(ToolUsingAgent): # 4. Build messages messages = self._build_operative_messages( - input, context, system_prompt=system_prompt, + input, + context, + system_prompt=system_prompt, session_messages=session_messages, ) @@ -143,11 +150,13 @@ class OperativeAgent(ToolUsingAgent): for i, tc in enumerate(raw_tool_calls) ] - messages.append(Message( - role=Role.ASSISTANT, - content=content, - tool_calls=tool_calls, - )) + messages.append( + Message( + role=Role.ASSISTANT, + content=content, + tool_calls=tool_calls, + ) + ) for tc in tool_calls: # Loop guard check @@ -160,12 +169,14 @@ class OperativeAgent(ToolUsingAgent): success=False, ) all_tool_results.append(tool_result) - messages.append(Message( - role=Role.TOOL, - content=tool_result.content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=tool_result.content, + tool_call_id=tc.id, + name=tc.name, + ) + ) continue tool_result = self._executor.execute(tc) @@ -181,12 +192,14 @@ class OperativeAgent(ToolUsingAgent): except (json.JSONDecodeError, TypeError): pass - messages.append(Message( - role=Role.TOOL, - content=tool_result.content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=tool_result.content, + tool_call_id=tc.id, + name=tc.name, + ) + ) else: # Max turns exceeded self._save_session(input, content) @@ -277,10 +290,12 @@ class OperativeAgent(ToolUsingAgent): session_id = f"operator:{self._operator_id}" try: self._session_store.save_message( - session_id, {"role": "user", "content": input_text}, + session_id, + {"role": "user", "content": input_text}, ) self._session_store.save_message( - session_id, {"role": "assistant", "content": response}, + session_id, + {"role": "assistant", "content": response}, ) except Exception: logger.debug("Could not save session for operator %s", self._operator_id) diff --git a/src/openjarvis/agents/orchestrator.py b/src/openjarvis/agents/orchestrator.py index 90125ae1..cca03d2c 100644 --- a/src/openjarvis/agents/orchestrator.py +++ b/src/openjarvis/agents/orchestrator.py @@ -62,10 +62,15 @@ class OrchestratorAgent(ToolUsingAgent): confirm_callback=None, ) -> None: super().__init__( - engine, model, tools=tools, bus=bus, - max_turns=max_turns, temperature=temperature, + engine, + model, + tools=tools, + bus=bus, + max_turns=max_turns, + temperature=temperature, max_tokens=max_tokens, - interactive=interactive, confirm_callback=confirm_callback, + interactive=interactive, + confirm_callback=confirm_callback, ) self._mode = mode self._system_prompt = system_prompt @@ -100,6 +105,7 @@ class OrchestratorAgent(ToolUsingAgent): from openjarvis.learning.intelligence.orchestrator.prompt_registry import ( build_system_prompt, ) + sys_prompt = build_system_prompt(tools=self._tools) messages = self._build_messages(input, context, system_prompt=sys_prompt) @@ -129,9 +135,7 @@ class OrchestratorAgent(ToolUsingAgent): # TOOL -> execute if parsed["tool"]: - messages.append( - Message(role=Role.ASSISTANT, content=content) - ) + messages.append(Message(role=Role.ASSISTANT, content=content)) tool_call = ToolCall( id=f"orch_{turns}", @@ -141,12 +145,8 @@ class OrchestratorAgent(ToolUsingAgent): tool_result = self._executor.execute(tool_call) all_tool_results.append(tool_result) - observation = ( - f"Observation: {tool_result.content}" - ) - messages.append( - Message(role=Role.USER, content=observation) - ) + observation = f"Observation: {tool_result.content}" + messages.append(Message(role=Role.USER, content=observation)) continue # Neither -> treat content as final answer @@ -187,9 +187,7 @@ class OrchestratorAgent(ToolUsingAgent): result["final_answer"] = final_match.group(1).strip() return result - tool_match = re.search( - r"TOOL:\s*(.+)", text, re.IGNORECASE - ) + tool_match = re.search(r"TOOL:\s*(.+)", text, re.IGNORECASE) if tool_match: result["tool"] = tool_match.group(1).strip() @@ -274,11 +272,13 @@ class OrchestratorAgent(ToolUsingAgent): ] # Append assistant message with tool calls - messages.append(Message( - role=Role.ASSISTANT, - content=content, - tool_calls=tool_calls, - )) + messages.append( + Message( + role=Role.ASSISTANT, + content=content, + tool_calls=tool_calls, + ) + ) # Execute each tool (with loop guard check) and append results if self._parallel_tools and len(tool_calls) > 1: @@ -286,7 +286,8 @@ class OrchestratorAgent(ToolUsingAgent): def _exec_tool(tc: ToolCall) -> tuple: if self._loop_guard: verdict = self._loop_guard.check_call( - tc.name, tc.arguments, + tc.name, + tc.arguments, ) if verdict.blocked: return tc, ToolResult( @@ -299,10 +300,7 @@ class OrchestratorAgent(ToolUsingAgent): with concurrent.futures.ThreadPoolExecutor( max_workers=len(tool_calls), ) as pool: - futures = { - pool.submit(_exec_tool, tc): tc - for tc in tool_calls - } + futures = {pool.submit(_exec_tool, tc): tc for tc in tool_calls} results_map: dict[int, tuple] = {} for future in concurrent.futures.as_completed(futures): tc_orig = futures[future] @@ -312,19 +310,22 @@ class OrchestratorAgent(ToolUsingAgent): for tc in tool_calls: _, tool_result = results_map[id(tc)] all_tool_results.append(tool_result) - messages.append(Message( - role=Role.TOOL, - content=tool_result.content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=tool_result.content, + tool_call_id=tc.id, + name=tc.name, + ) + ) else: # Sequential execution for tc in tool_calls: # Loop guard check before execution if self._loop_guard: verdict = self._loop_guard.check_call( - tc.name, tc.arguments, + tc.name, + tc.arguments, ) if verdict.blocked: tool_result = ToolResult( @@ -333,24 +334,28 @@ class OrchestratorAgent(ToolUsingAgent): success=False, ) all_tool_results.append(tool_result) - messages.append(Message( - role=Role.TOOL, - content=tool_result.content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=tool_result.content, + tool_call_id=tc.id, + name=tc.name, + ) + ) continue tool_result = self._executor.execute(tc) all_tool_results.append(tool_result) # Append tool response message - messages.append(Message( - role=Role.TOOL, - content=tool_result.content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=tool_result.content, + tool_call_id=tc.id, + name=tc.name, + ) + ) # Max turns exceeded final_content = self._strip_think_tags(content) if content else "" diff --git a/src/openjarvis/agents/rlm.py b/src/openjarvis/agents/rlm.py index 43488b8c..cd8c0c37 100644 --- a/src/openjarvis/agents/rlm.py +++ b/src/openjarvis/agents/rlm.py @@ -35,8 +35,8 @@ RLM_SYSTEM_PROMPT = ( "final answer.\n" "- `FINAL_VAR(var_name: str)` — Terminate and return the " "value of variable `var_name`.\n" - "- `answer` dict — Set `answer[\"value\"] = ...` and " - "`answer[\"ready\"] = True` to terminate.\n\n" + '- `answer` dict — Set `answer["value"] = ...` and ' + '`answer["ready"] = True` to terminate.\n\n' "{tool_section}" "## Available Modules\n\n" "json, re, math, collections, itertools, functools, " @@ -52,7 +52,7 @@ RLM_SYSTEM_PROMPT = ( "and use `llm_query()` on each chunk.\n" "3. Combine sub-results programmatically.\n" "4. When you have the final answer, call " - "`FINAL(answer_value)` or `FINAL_VAR(\"var_name\")`.\n" + '`FINAL(answer_value)` or `FINAL_VAR("var_name")`.\n' "5. If you can answer directly without code, just respond " "with text (no code block).\n\n" "## Strategy Tips\n\n" @@ -107,10 +107,15 @@ class RLMAgent(ToolUsingAgent): confirm_callback=None, ) -> None: super().__init__( - engine, model, tools=tools, bus=bus, - max_turns=max_turns, temperature=temperature, + engine, + model, + tools=tools, + bus=bus, + max_turns=max_turns, + temperature=temperature, max_tokens=max_tokens, - interactive=interactive, confirm_callback=confirm_callback, + interactive=interactive, + confirm_callback=confirm_callback, ) # Override executor: RLM only creates one if tools are provided if not self._tools: @@ -171,7 +176,9 @@ class RLMAgent(ToolUsingAgent): # Build conversation messages = self._build_messages( - input, context, system_prompt=system_prompt, + input, + context, + system_prompt=system_prompt, ) all_tool_results: list[ToolResult] = [] @@ -236,9 +243,7 @@ class RLMAgent(ToolUsingAgent): # Feed output back as user message messages.append(Message(role=Role.ASSISTANT, content=content)) feedback = ( - f"REPL Output: {output}" - if output - else "REPL Output: (no output)" + f"REPL Output: {output}" if output else "REPL Output: (no output)" ) messages.append(Message(role=Role.USER, content=feedback)) @@ -284,19 +289,23 @@ class RLMAgent(ToolUsingAgent): ) for i, tc in enumerate(raw_tool_calls) ] - messages.append(Message( - role=Role.ASSISTANT, - content=content, - tool_calls=tool_calls, - )) + messages.append( + Message( + role=Role.ASSISTANT, + content=content, + tool_calls=tool_calls, + ) + ) for tc in tool_calls: tr = self._executor.execute(tc) - messages.append(Message( - role=Role.TOOL, - content=tr.content, - tool_call_id=tc.id, - name=tc.name, - )) + messages.append( + Message( + role=Role.TOOL, + content=tr.content, + tool_call_id=tc.id, + name=tc.name, + ) + ) followup = self._engine.generate( messages, model=self._sub_model, diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 76ca3c8d..ee1b36da 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -60,7 +60,10 @@ def _run_cmd(cmd: list[str]) -> str: """Run a command and return stripped stdout, or empty string on failure.""" try: result = subprocess.run( - cmd, capture_output=True, text=True, timeout=10, # noqa: S603 + cmd, + capture_output=True, + text=True, + timeout=10, # noqa: S603 ) return result.stdout.strip() except (FileNotFoundError, subprocess.TimeoutExpired, OSError): @@ -70,11 +73,13 @@ def _run_cmd(cmd: list[str]) -> str: def _detect_nvidia_gpu() -> Optional[GpuInfo]: if not shutil.which("nvidia-smi"): return None - raw = _run_cmd([ - "nvidia-smi", - "--query-gpu=name,memory.total,count", - "--format=csv,noheader,nounits", - ]) + raw = _run_cmd( + [ + "nvidia-smi", + "--query-gpu=name,memory.total,count", + "--format=csv,noheader,nounits", + ] + ) if not raw: return None try: @@ -118,6 +123,7 @@ def _detect_amd_gpu() -> Optional[GpuInfo]: try: allinfo_raw = _run_cmd(["rocm-smi", "--showallinfo"]) import re + gpu_ids = set(re.findall(r"GPU\[(\d+)\]", allinfo_raw)) if gpu_ids: count = len(gpu_ids) @@ -449,26 +455,26 @@ class IntelligenceConfig: default_model: str = "" fallback_model: str = "" - model_path: str = "" # Local weights (HF repo, GGUF file, etc.) - checkpoint_path: str = "" # Checkpoint/adapter path - quantization: str = "none" # none, fp8, int8, int4, gguf_q4, gguf_q8 - preferred_engine: str = "" # Override engine for this model (e.g., "vllm") - provider: str = "" # local, openai, anthropic, google + model_path: str = "" # Local weights (HF repo, GGUF file, etc.) + checkpoint_path: str = "" # Checkpoint/adapter path + quantization: str = "none" # none, fp8, int8, int4, gguf_q4, gguf_q8 + preferred_engine: str = "" # Override engine for this model (e.g., "vllm") + provider: str = "" # local, openai, anthropic, google # Generation defaults (overridable per-call) temperature: float = 0.7 max_tokens: int = 1024 top_p: float = 0.9 top_k: int = 40 repetition_penalty: float = 1.0 - stop_sequences: str = "" # Comma-separated stop strings + stop_sequences: str = "" # Comma-separated stop strings @dataclass(slots=True) class RoutingLearningConfig: """Routing sub-policy config within Learning.""" - policy: str = "heuristic" # heuristic | learned - min_samples: int = 5 # Min traces before trusting learned routing + policy: str = "heuristic" # heuristic | learned + min_samples: int = 5 # Min traces before trusting learned routing @dataclass(slots=True) @@ -717,9 +723,9 @@ class AgentConfig: default_agent: str = "simple" max_turns: int = 10 - tools: str = "" # comma-separated tool names - objective: str = "" # concise purpose for routing/learning/docs - system_prompt: str = "" # inline system prompt (takes precedence if set) + tools: str = "" # comma-separated tool names + objective: str = "" # concise purpose for routing/learning/docs + system_prompt: str = "" # inline system prompt (takes precedence if set) system_prompt_path: str = "" # path to system prompt file (.txt, .md) context_from_memory: bool = True # inject relevant memory context into prompts @@ -899,7 +905,7 @@ class BlueBubblesChannelConfig: class WhatsAppBaileysChannelConfig: """Per-channel config for WhatsApp via Baileys protocol.""" - auth_dir: str = "" # Defaults to ~/.openjarvis/whatsapp_auth + auth_dir: str = "" # Defaults to ~/.openjarvis/whatsapp_auth assistant_name: str = "Jarvis" assistant_has_own_number: bool = False @@ -1177,7 +1183,8 @@ def _migrate_toml_data(data: Dict[str, Any], cfg: "JarvisConfig") -> None: src = data.get(src_section, {}) if isinstance(src, dict) and "context_injection" in src: data.setdefault("agent", {}).setdefault( - "context_from_memory", src.pop("context_injection"), + "context_from_memory", + src.pop("context_injection"), ) if "tools" in data: @@ -1186,7 +1193,8 @@ def _migrate_toml_data(data: Dict[str, Any], cfg: "JarvisConfig") -> None: storage_sub = tools_data.get("storage", {}) if isinstance(storage_sub, dict) and "context_injection" in storage_sub: data.setdefault("agent", {}).setdefault( - "context_from_memory", storage_sub.pop("context_injection"), + "context_from_memory", + storage_sub.pop("context_injection"), ) @@ -1220,16 +1228,31 @@ def load_config(path: Optional[Path] = None) -> JarvisConfig: # All top-level sections — recursive _apply_toml_section handles # nested sub-configs (engine.ollama, learning.routing, channel.*, etc.) top_sections = ( - "engine", "intelligence", "learning", "agent", - "server", "telemetry", "traces", "security", - "channel", "tools", "sandbox", "scheduler", - "workflow", "sessions", "a2a", "operators", - "speech", "optimize", "agent_manager", + "engine", + "intelligence", + "learning", + "agent", + "server", + "telemetry", + "traces", + "security", + "channel", + "tools", + "sandbox", + "scheduler", + "workflow", + "sessions", + "a2a", + "operators", + "speech", + "optimize", + "agent_manager", ) for section_name in top_sections: if section_name in data: _apply_toml_section( - getattr(cfg, section_name), data[section_name], + getattr(cfg, section_name), + data[section_name], ) # Memory: accept [memory] (old) → maps to tools.storage @@ -1250,13 +1273,8 @@ def generate_minimal_toml(hw: HardwareInfo, engine: str | None = None) -> str: model = recommend_model(hw, engine) gpu_comment = "" if hw.gpu: - mem_label = ( - "unified memory" if hw.gpu.vendor == "apple" else "VRAM" - ) - gpu_comment = ( - f"\n# GPU: {hw.gpu.name}" - f" ({hw.gpu.vram_gb} GB {mem_label})" - ) + mem_label = "unified memory" if hw.gpu.vendor == "apple" else "VRAM" + gpu_comment = f"\n# GPU: {hw.gpu.name} ({hw.gpu.vram_gb} GB {mem_label})" return f"""\ # OpenJarvis configuration # Hardware: {hw.cpu_brand} ({hw.cpu_count} cores, {hw.ram_gb} GB RAM){gpu_comment} diff --git a/tests/agents/test_config_defaults.py b/tests/agents/test_config_defaults.py index 95493bdd..d3f2b117 100644 --- a/tests/agents/test_config_defaults.py +++ b/tests/agents/test_config_defaults.py @@ -160,7 +160,11 @@ class TestClassLevelDefaults: """Caller-provided values override both config and class defaults.""" engine = MagicMock() agent = _TestToolAgentWithDefaults( - engine, "m", temperature=0.9, max_tokens=100, max_turns=2, + engine, + "m", + temperature=0.9, + max_tokens=100, + max_turns=2, ) assert agent._temperature == 0.9 assert agent._max_tokens == 100 From 3777086d3aebd28be809adc74c7c50accbda3991 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:55:26 -0700 Subject: [PATCH 45/92] fix: count full prompt tokens without KV-cache assumption Ollama's prompt_eval_count may exclude KV-cached tokens (system prompt, prior conversation turns), under-counting the prompt size used for cost/FLOPs/energy calculations and leaderboard submissions. Engine changes: - Add estimate_prompt_tokens() helper in engine/_base.py that computes cache-agnostic token count from message content - Ollama + OpenAI-compat engines now use max(reported, estimated) to ensure system prompt and full context are always counted Savings/leaderboard changes: - Add TOKEN_COUNTING_VERSION=2 to savings.py so frontends can tag Supabase submissions (old=v1, corrected=v2) - Desktop + web frontends forward version in leaderboard submissions - Add POST /v1/telemetry/reset endpoint to clear stale telemetry Active users auto-correct via upsert on next session; no manual Supabase migration required. --- desktop/src/components/SavingsDashboard.tsx | 2 ++ frontend/src/App.tsx | 1 + frontend/src/lib/api.ts | 1 + frontend/src/types/index.ts | 1 + src/openjarvis/engine/_base.py | 26 +++++++++++++++++++- src/openjarvis/engine/_openai_compat.py | 14 ++++++++--- src/openjarvis/engine/ollama.py | 27 +++++++++++++++------ src/openjarvis/server/routes.py | 24 ++++++++++++++++++ src/openjarvis/server/savings.py | 20 ++++++++++++++- src/openjarvis/telemetry/flops.py | 4 +++ 10 files changed, 107 insertions(+), 13 deletions(-) diff --git a/desktop/src/components/SavingsDashboard.tsx b/desktop/src/components/SavingsDashboard.tsx index df67050b..3f5937e2 100644 --- a/desktop/src/components/SavingsDashboard.tsx +++ b/desktop/src/components/SavingsDashboard.tsx @@ -34,6 +34,7 @@ interface SavingsData { heavy_low: number; heavy_high: number; }; + token_counting_version?: number; } // --------------------------------------------------------------------------- @@ -335,6 +336,7 @@ export function SavingsDashboard({ apiUrl }: { apiUrl: string }) { dollar_savings: dollarSavings, energy_wh_saved: energySaved, flops_saved: flopsSaved, + token_counting_version: data.token_counting_version ?? 1, }, }).catch(() => {}); }, [data, optInEnabled, displayName, anonId]); diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 617a034f..f619379a 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -88,6 +88,7 @@ export default function App() { dollar_savings: dollarSavings, energy_wh_saved: energySaved, flops_saved: flopsSaved, + token_counting_version: data.token_counting_version ?? 1, }); } }) diff --git a/frontend/src/lib/api.ts b/frontend/src/lib/api.ts index 0bc2cfe4..201e9c7c 100644 --- a/frontend/src/lib/api.ts +++ b/frontend/src/lib/api.ts @@ -589,6 +589,7 @@ export interface SavingsSubmission { dollar_savings: number; energy_wh_saved: number; flops_saved: number; + token_counting_version?: number; } export async function submitSavings(data: SavingsSubmission): Promise { diff --git a/frontend/src/types/index.ts b/frontend/src/types/index.ts index a01205b3..747e1d25 100644 --- a/frontend/src/types/index.ts +++ b/frontend/src/types/index.ts @@ -123,6 +123,7 @@ export interface SavingsData { total_tokens: number; local_cost: number; per_provider: ProviderSavings[]; + token_counting_version?: number; } export interface ServerInfo { diff --git a/src/openjarvis/engine/_base.py b/src/openjarvis/engine/_base.py index 4207ba65..3b12f34f 100644 --- a/src/openjarvis/engine/_base.py +++ b/src/openjarvis/engine/_base.py @@ -38,4 +38,28 @@ def messages_to_dicts(messages: Sequence[Message]) -> List[Dict[str, Any]]: return out -__all__ = ["EngineConnectionError", "InferenceEngine", "messages_to_dicts"] +def estimate_prompt_tokens(messages: Sequence[Message]) -> int: + """Estimate full prompt token count from message content. + + Ollama's ``prompt_eval_count`` may report only *newly evaluated* + tokens when KV-cache hits occur, under-counting the system prompt + and earlier conversation turns. This helper provides a + cache-agnostic estimate so that downstream cost / FLOPs / energy + calculations reflect the true prompt size — matching what a cloud + provider would charge. + + Uses ~4 characters per token (standard BPE average for English) plus + a small per-message overhead for role markers and separators. + """ + total_chars = sum(len(m.content) for m in messages) + # ~4 tokens overhead per message for role markers / separators + overhead = len(messages) * 4 + return max(1, total_chars // 4 + overhead) + + +__all__ = [ + "EngineConnectionError", + "InferenceEngine", + "estimate_prompt_tokens", + "messages_to_dicts", +] diff --git a/src/openjarvis/engine/_openai_compat.py b/src/openjarvis/engine/_openai_compat.py index ba7d3870..bbc3cf00 100644 --- a/src/openjarvis/engine/_openai_compat.py +++ b/src/openjarvis/engine/_openai_compat.py @@ -13,6 +13,7 @@ from openjarvis.core.types import Message from openjarvis.engine._base import ( EngineConnectionError, InferenceEngine, + estimate_prompt_tokens, messages_to_dicts, ) @@ -93,12 +94,19 @@ class _OpenAICompatibleEngine(InferenceEngine): } choice = choices[0] usage = data.get("usage", {}) + # Ensure prompt_tokens reflects the full prompt size (including + # system prompt and all conversation history) without assuming + # KV-cache savings. + reported_prompt = usage.get("prompt_tokens", 0) + estimated_prompt = estimate_prompt_tokens(messages) + prompt_tokens = max(reported_prompt, estimated_prompt) + completion_tokens = usage.get("completion_tokens", 0) result: Dict[str, Any] = { "content": choice["message"].get("content") or "", "usage": { - "prompt_tokens": usage.get("prompt_tokens", 0), - "completion_tokens": usage.get("completion_tokens", 0), - "total_tokens": usage.get("total_tokens", 0), + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "total_tokens": prompt_tokens + completion_tokens, }, "model": data.get("model", model), "finish_reason": choice.get("finish_reason", "stop"), diff --git a/src/openjarvis/engine/ollama.py b/src/openjarvis/engine/ollama.py index 5e95ef9a..1b219368 100644 --- a/src/openjarvis/engine/ollama.py +++ b/src/openjarvis/engine/ollama.py @@ -15,6 +15,7 @@ from openjarvis.core.types import Message from openjarvis.engine._base import ( EngineConnectionError, InferenceEngine, + estimate_prompt_tokens, messages_to_dicts, ) @@ -105,7 +106,13 @@ class OllamaEngine(InferenceEngine): f"Ollama returned {exc.response.status_code}: {body}" ) from exc data = resp.json() - prompt_tokens = data.get("prompt_eval_count", 0) + # Ollama's prompt_eval_count may exclude KV-cached tokens + # (system prompt, prior turns). Use the larger of the reported + # count and a cache-agnostic estimate so that cost / FLOPs / + # energy calculations reflect the full prompt size. + reported_prompt = data.get("prompt_eval_count", 0) + estimated_prompt = estimate_prompt_tokens(messages) + prompt_tokens = max(reported_prompt, estimated_prompt) completion_tokens = data.get("eval_count", 0) content = data.get("message", {}).get("content", "") result: Dict[str, Any] = { @@ -176,14 +183,18 @@ class OllamaEngine(InferenceEngine): if content: yield content if chunk.get("done", False): - # Capture usage from final chunk + # Capture usage from final chunk. Use the + # cache-agnostic estimate for prompt tokens to + # avoid under-counting when Ollama's KV cache + # suppresses prompt_eval_count. + reported_prompt = chunk.get("prompt_eval_count", 0) + est_prompt = estimate_prompt_tokens(messages) + full_prompt = max(reported_prompt, est_prompt) + comp = chunk.get("eval_count", 0) self._last_stream_usage = { - "prompt_tokens": chunk.get("prompt_eval_count", 0), - "completion_tokens": chunk.get("eval_count", 0), - "total_tokens": ( - chunk.get("prompt_eval_count", 0) - + chunk.get("eval_count", 0) - ), + "prompt_tokens": full_prompt, + "completion_tokens": comp, + "total_tokens": full_prompt + comp, } break except (httpx.ConnectError, httpx.TimeoutException) as exc: diff --git a/src/openjarvis/server/routes.py b/src/openjarvis/server/routes.py index bf935bef..6c163f65 100644 --- a/src/openjarvis/server/routes.py +++ b/src/openjarvis/server/routes.py @@ -476,6 +476,30 @@ async def savings(request: Request): agg.close() +@router.post("/v1/telemetry/reset") +async def reset_telemetry(): + """Clear all stored telemetry records. + + Useful after updating token-counting methodology — clears + historical records that were computed under the old rules so + that the savings dashboard and leaderboard submissions start + fresh with corrected values. + """ + from openjarvis.core.config import DEFAULT_CONFIG_DIR + from openjarvis.telemetry.aggregator import TelemetryAggregator + + db_path = DEFAULT_CONFIG_DIR / "telemetry.db" + if not db_path.exists(): + return {"status": "ok", "records_cleared": 0} + + agg = TelemetryAggregator(db_path) + try: + count = agg.clear() + finally: + agg.close() + return {"status": "ok", "records_cleared": count} + + @router.get("/v1/info") async def server_info(request: Request): """Return server configuration: model, agent, engine.""" diff --git a/src/openjarvis/server/savings.py b/src/openjarvis/server/savings.py index 993af063..0c6fbc0d 100644 --- a/src/openjarvis/server/savings.py +++ b/src/openjarvis/server/savings.py @@ -10,6 +10,12 @@ import time from dataclasses import asdict, dataclass, field from typing import Any, Dict, List +# Bump when token-counting methodology changes so that the leaderboard +# can distinguish entries computed under different rules. +# v1 = original (Ollama prompt_eval_count, may under-count due to KV cache) +# v2 = full prompt token count, no KV-cache assumption, system prompt always counted +TOKEN_COUNTING_VERSION: int = 2 + # --------------------------------------------------------------------------- # Cloud provider pricing (USD per 1M tokens) # params_b: model size in billions, for no-KV-cache FLOPs @@ -75,6 +81,7 @@ class SavingsSummary: session_duration_hours: float = 0.0 avg_cost_per_query: Dict[str, float] = field(default_factory=dict) cloud_agent_equivalent: Dict[str, int] = field(default_factory=dict) + token_counting_version: int = TOKEN_COUNTING_VERSION def compute_savings( @@ -83,7 +90,18 @@ def compute_savings( total_calls: int = 0, session_start: float = 0.0, ) -> SavingsSummary: - """Compute savings vs cloud providers given token counts.""" + """Compute savings vs cloud providers given token counts. + + Token counting assumptions (these match cloud provider billing): + - **System prompt included**: ``prompt_tokens`` includes system + prompt tokens for every API call, matching what cloud providers + charge. + - **No KV-cache discount**: Each call's ``prompt_tokens`` reflects + the full context size (system prompt + conversation history). + Even though local engines may use KV caching internally, the + savings comparison uses the uncached token count because cloud + providers bill for all input tokens on every request. + """ total_tokens = prompt_tokens + completion_tokens providers: List[ProviderSavings] = [] diff --git a/src/openjarvis/telemetry/flops.py b/src/openjarvis/telemetry/flops.py index a9527dcb..67e7b06b 100644 --- a/src/openjarvis/telemetry/flops.py +++ b/src/openjarvis/telemetry/flops.py @@ -49,6 +49,10 @@ def estimate_flops( Uses the 2 * P * T approximation where P = params, T = total tokens. Returns (total_flops, flops_per_token). + + ``input_tokens`` must include system-prompt tokens and must *not* + be reduced for KV-cache reuse — it should represent the full prompt + size that was sent to the engine. """ params_b = _get_params_b(model) total_tokens = input_tokens + output_tokens From 4cdc4639bc52f4dc73c995073216e0448ec4bcd8 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:50:05 -0700 Subject: [PATCH 46/92] docs: add REVIEW.md with PR review instructions for Claude --- REVIEW.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 REVIEW.md diff --git a/REVIEW.md b/REVIEW.md new file mode 100644 index 00000000..d80561e7 --- /dev/null +++ b/REVIEW.md @@ -0,0 +1,42 @@ +# OpenJarvis PR Review Instructions + +You are reviewing pull requests for OpenJarvis, a local-first personal AI agent framework built with Python, Rust (PyO3), and TypeScript. + +## Review Checklist + +Evaluate every PR against these criteria: + +### 1. Relevance +Is this PR doing something useful? Valid contributions include: bug fixes, new features, feature expansions, documentation improvements, test coverage, and performance improvements. Flag PRs that appear to be empty, auto-generated without substance, or unrelated to the project. + +### 2. Completeness +Does the code actually implement what the PR title and description claim? If the PR says "fix X", verify X is actually fixed. If it says "add Y", verify Y is fully added and functional — not partially implemented or stubbed out. + +### 3. Correctness +Check for logic errors, edge cases, and off-by-one errors. Pay particular attention to: +- **Rust-Python bridge (PyO3) boundaries** — type conversions, error propagation, GIL handling +- **Async/await patterns** — missing awaits, unclosed resources, blocking calls in async contexts +- **Registry pattern compliance** — new components (engines, tools, agents, channels) must register via `ToolRegistry`, `EngineRegistry`, `AgentRegistry`, `ChannelRegistry`, etc. in `src/openjarvis/core/registry.py` +- **Event bus integration** — new lifecycle events should use `EventBus` from `src/openjarvis/core/events.py` + +### 4. Testing +Does the PR include tests for new code paths? Are existing tests expected to still pass? New tools, engines, agents, and channels should have corresponding test files in `tests/` mirroring the `src/` structure. + +### 5. Security +Check for: hardcoded API keys or secrets, missing input validation at system boundaries (user input, external APIs), and anything that compromises local-first data isolation. + +## Do NOT Comment On + +- Formatting or style — Ruff handles this automatically in CI +- Code in unchanged files outside the PR diff +- Subjective naming preferences +- Adding docstrings or comments to code the PR did not modify + +## Output Format + +- Post **inline comments** on specific lines for actionable issues +- Post a **summary comment** containing: what the PR does, whether it achieves its stated goal, and any blocking concerns +- Use severity levels: + - `blocking` — must fix before merge + - `suggestion` — consider fixing + - `nit` — take it or leave it From 620788bee15e72074d7bfe2e3a4f5067f9fefb71 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:53:26 -0700 Subject: [PATCH 47/92] ci: add Claude PR review workflow --- .github/workflows/claude-review.yml | 50 +++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 .github/workflows/claude-review.yml diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml new file mode 100644 index 00000000..42732cc9 --- /dev/null +++ b/.github/workflows/claude-review.yml @@ -0,0 +1,50 @@ +name: Claude PR Review + +on: + pull_request: + types: [opened, synchronize] + issue_comment: + types: [created] + pull_request_review_comment: + types: [created] + workflow_dispatch: + +concurrency: + group: claude-review-${{ github.event.pull_request.number || github.event.issue.number || github.run_id }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: write + issues: write + +jobs: + review: + runs-on: ubuntu-latest + timeout-minutes: 30 + if: | + github.event_name == 'pull_request' || + github.event_name == 'workflow_dispatch' || + (github.event_name == 'issue_comment' && + github.event.issue.pull_request && + contains(github.event.comment.body, '@claude') && + github.actor != 'claude[bot]') || + (github.event_name == 'pull_request_review_comment' && + contains(github.event.comment.body, '@claude') && + github.actor != 'claude[bot]') + steps: + - uses: actions/checkout@v4 + + - name: Read review instructions + id: review + run: | + EOF=$(dd if=/dev/urandom bs=15 count=1 status=none | base64) + echo "instructions<<$EOF" >> "$GITHUB_OUTPUT" + cat REVIEW.md >> "$GITHUB_OUTPUT" + echo "$EOF" >> "$GITHUB_OUTPUT" + + - uses: anthropics/claude-code-action@v1 + with: + anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} + model: claude-sonnet-4-6 + review_instructions: ${{ steps.review.outputs.instructions }} From 4ccec017849ef2010420ae85993bfc2bc3fa8a52 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 16:53:58 -0700 Subject: [PATCH 48/92] ci: add Claude issue fixer workflow --- .github/workflows/claude-issues.yml | 74 +++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 .github/workflows/claude-issues.yml diff --git a/.github/workflows/claude-issues.yml b/.github/workflows/claude-issues.yml new file mode 100644 index 00000000..0026b1d5 --- /dev/null +++ b/.github/workflows/claude-issues.yml @@ -0,0 +1,74 @@ +name: Claude Issue Fixer + +on: + issues: + types: [opened, labeled] + issue_comment: + types: [created] + workflow_dispatch: + +concurrency: + group: claude-issues-${{ github.event.issue.number || github.run_id }} + cancel-in-progress: true + +permissions: + contents: write + pull-requests: write + issues: write + +jobs: + fix: + runs-on: ubuntu-latest + timeout-minutes: 60 + if: | + github.event_name == 'workflow_dispatch' || + (github.event_name == 'issues' && + (contains(github.event.issue.labels.*.name, 'bug') || + contains(github.event.issue.labels.*.name, 'autofix'))) || + (github.event_name == 'issue_comment' && + !github.event.issue.pull_request && + contains(github.event.comment.body, '@claude') && + github.actor != 'claude[bot]') + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - uses: anthropics/claude-code-action@v1 + with: + anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} + model: claude-sonnet-4-6 + direct_prompt: | + You are an automated issue fixer for the OpenJarvis repository. Follow these steps in order: + + ## Step 1: Diagnose + Read the issue thoroughly. Explore the codebase to understand the problem. If this is a bug report, attempt to reproduce it. Identify the root cause and affected files. + + ## Step 2: Comment your plan + BEFORE making any code changes, post a comment on this issue describing: + - Your root cause analysis + - Which files need to change and why + - Your implementation approach + + ## Step 3: Implement + Create a branch named `claude/issue-${{ github.event.issue.number }}` and make the changes. Use conventional commit messages that reference the issue, e.g.: + `fix: handle empty tool responses in orchestrator (fixes #${{ github.event.issue.number }})` + + ## Step 4: Test + Run these commands and ensure both pass: + - `uv run ruff check src/ tests/` (linting) + - `uv run pytest tests/ -v --tb=short` (tests) + + If tests fail, fix the issues before proceeding. If you added new functionality, add corresponding tests in `tests/` mirroring the `src/` directory structure. + + ## Step 5: Open PR + Create a pull request that: + - Links back to this issue (include `Fixes #${{ github.event.issue.number }}` in the PR body) + - Describes what was changed and why + - Includes a summary of test results + + ## If you cannot fix it + If you cannot reproduce the issue, cannot determine the root cause, or the fix is beyond your capabilities, post a comment explaining: + - What you investigated + - What you found (or didn't find) + - What additional information you need from the reporter From 9dbd172950a5ef04e1a2bf3ce5893fc9b2137bde Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 17:09:43 -0700 Subject: [PATCH 49/92] fix: mock config in test_init_defaults to test class-level defaults The test_init_defaults test expected OperativeAgent class defaults (temperature=0.3) but didn't mock load_config(), so when config loaded successfully it returned the global default (0.7) instead. Fix: mock load_config to raise, so the test properly validates the class-level _default_temperature/max_tokens/max_turns fallback path. Co-Authored-By: Claude Opus 4.6 (1M context) --- tests/operators/test_operators.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/operators/test_operators.py b/tests/operators/test_operators.py index 065dc596..87e51eb4 100644 --- a/tests/operators/test_operators.py +++ b/tests/operators/test_operators.py @@ -485,7 +485,11 @@ class TestOperativeAgent: from openjarvis.agents.operative import OperativeAgent engine = FakeEngine() - agent = OperativeAgent(engine, "test-model") + with patch( + "openjarvis.agents._stubs.load_config", + side_effect=Exception("no config"), + ): + agent = OperativeAgent(engine, "test-model") assert agent.agent_id == "operative" assert agent._temperature == 0.3 assert agent._max_tokens == 2048 From 3c2d9462ab233bfe1b4911d0ecb5fc416361523e Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 17:36:33 -0700 Subject: [PATCH 50/92] fix: use correct claude-code-action input params and add id-token permission - Replace invalid `model` input with default (action auto-selects) - Replace `review_instructions`/`direct_prompt` with `prompt` (valid input) - Add `id-token: write` permission required for OIDC token fetching Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/claude-issues.yml | 4 ++-- .github/workflows/claude-review.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/claude-issues.yml b/.github/workflows/claude-issues.yml index 0026b1d5..dfbd3ff7 100644 --- a/.github/workflows/claude-issues.yml +++ b/.github/workflows/claude-issues.yml @@ -15,6 +15,7 @@ permissions: contents: write pull-requests: write issues: write + id-token: write jobs: fix: @@ -37,8 +38,7 @@ jobs: - uses: anthropics/claude-code-action@v1 with: anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} - model: claude-sonnet-4-6 - direct_prompt: | + prompt: | You are an automated issue fixer for the OpenJarvis repository. Follow these steps in order: ## Step 1: Diagnose diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml index 42732cc9..2da7cd5b 100644 --- a/.github/workflows/claude-review.yml +++ b/.github/workflows/claude-review.yml @@ -17,6 +17,7 @@ permissions: contents: read pull-requests: write issues: write + id-token: write jobs: review: @@ -46,5 +47,4 @@ jobs: - uses: anthropics/claude-code-action@v1 with: anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} - model: claude-sonnet-4-6 - review_instructions: ${{ steps.review.outputs.instructions }} + prompt: ${{ steps.review.outputs.instructions }} From b502441293595b2ab3bb4d51b183eba24a1b845c Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 17:42:32 -0700 Subject: [PATCH 51/92] docs: add design spec for init model onboarding and privacy scanner Covers two features based on user feedback: 1. Interactive model download in `jarvis init` with MLX catalog fix 2. New `jarvis scan` privacy environment audit command Co-Authored-By: Claude Opus 4.6 (1M context) --- ...t-onboarding-and-privacy-scanner-design.md | 249 ++++++++++++++++++ 1 file changed, 249 insertions(+) create mode 100644 docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md diff --git a/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md b/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md new file mode 100644 index 00000000..3f93ed5f --- /dev/null +++ b/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md @@ -0,0 +1,249 @@ +# Design: Init Model Onboarding & Privacy Environment Scanner + +**Date:** 2026-03-24 +**Status:** Draft +**Author:** Jon Saad-Falcon + Claude + +## Problem Statement + +Two issues reported by a user (Ali Shahkar) after setting up OpenJarvis on macOS with llama.cpp: + +1. **`jarvis init` sets a default model with no way to download it.** The config is written with a recommended model, but there's no download prompt, no validation that it's available, and `jarvis doctor` immediately warns it can't be found. On Apple Silicon with MLX, the bug is worse: `recommend_model()` returns an empty string because no Qwen3.5 model lists MLX as a supported engine. + +2. **No environment privacy auditing.** OpenJarvis positions itself as privacy-first local AI, but `jarvis init` doesn't verify the local environment is actually private. Cloud sync agents, MDM profiles, disk encryption status, and network exposure can all undermine the privacy guarantee without the user knowing. + +## Solution Overview + +### Issue 1: Interactive Model Download in `jarvis init` + +**Bug fix:** Add `"mlx"` to `supported_engines` for Qwen3.5 models (3b, 4b, 8b, 14b) in the model catalog. + +**Feature:** After writing the config and displaying the recommended model, `jarvis init` prompts: + +``` + Recommended model: qwen3.5:8b (~4.4 GB) + Download now? [y/n]: +``` + +Download logic is engine-specific: +- **Ollama:** Reuse existing `model pull` code (HTTP stream to `POST /api/pull`). +- **llama.cpp:** Shell out to `huggingface-cli download` with GGUF filename from catalog metadata. +- **MLX:** Shell out to `huggingface-cli download` for pre-quantized MLX repo from catalog metadata. +- **vLLM/SGLang:** Inform the user the model downloads automatically on first serve. +- **LM Studio/Exo/Nexa:** Show manual download instructions (these have their own UIs). + +**Empty model fallback:** If `recommend_model()` returns `""`, display: + +``` + ! Not enough memory to run any local model. + Consider a cloud engine or a machine with more RAM. +``` + +### Issue 2: Privacy Environment Scanner + +**New CLI command:** `jarvis scan` performs a full environment privacy audit. + +**Init hook:** At the end of `jarvis init`, run a lightweight subset (disk encryption + cloud sync overlap with `~/.openjarvis/`) and show a compact summary pointing to `jarvis scan` for the full audit. + +## Detailed Design + +### Model Catalog Changes + +Add `"mlx"` to `supported_engines` and download metadata for Qwen3.5 models in `src/openjarvis/intelligence/model_catalog.py`. + +**Important:** The catalog has TWO Qwen3.5 sections. The first (lines ~42-121) contains the original entries (`qwen3.5:3b`, `qwen3.5:8b`, `qwen3.5:14b`, `qwen3.5:35b`, etc. with `context_length=131072`). The second (lines ~219-274) contains newer entries (`qwen3.5:4b`, `qwen3.5:35b-a3b`, etc. with `context_length=262144`). Both sections' models are candidates for `recommend_model()`. MLX must be added to entries in BOTH sections. + +Models getting MLX support added to `supported_engines`: + +| model_id | Section | Current context_length | Add MLX | +|----------|---------|----------------------|---------| +| `qwen3.5:3b` | First | 131072 | Yes | +| `qwen3.5:4b` | Second | 262144 | Yes | +| `qwen3.5:8b` | First | 131072 | Yes | +| `qwen3.5:14b` | First | 131072 | Yes | + +Larger models (35b+) are excluded because MLX quantized weights aren't widely available and would exceed typical Apple Silicon memory. + +Download metadata to add to each model's `metadata` dict: + +| model_id | `gguf_file` | `mlx_repo` | +|----------|-------------|------------| +| `qwen3.5:3b` | `qwen3.5-3b-q4_k_m.gguf` | `mlx-community/Qwen3.5-3B-4bit` | +| `qwen3.5:4b` | `qwen3.5-4b-q4_k_m.gguf` | `mlx-community/Qwen3.5-4B-4bit` | +| `qwen3.5:8b` | `qwen3.5-8b-q4_k_m.gguf` | `mlx-community/Qwen3.5-8B-4bit` | +| `qwen3.5:14b` | `qwen3.5-14b-q4_k_m.gguf` | `mlx-community/Qwen3.5-14B-4bit` | + +Example updated entry: + +```python +ModelSpec( + model_id="qwen3.5:8b", + name="Qwen3.5 8B", + parameter_count_b=8.0, + active_parameter_count_b=1.0, + context_length=131072, + supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"), # added mlx + provider="alibaba", + metadata={ + "architecture": "moe", + "hf_repo": "Qwen/Qwen3.5-8B", + "gguf_file": "qwen3.5-8b-q4_k_m.gguf", + "mlx_repo": "mlx-community/Qwen3.5-8B-4bit", + }, +) +``` + +### Init Command Changes (`src/openjarvis/cli/init_cmd.py`) + +After the config is written and the "Getting Started" panel is shown, add: + +1. **Size estimate display:** Compute `parameter_count_b * 0.5 * 1.1` and display alongside the model name. Note: this is an estimate based on Q4_K_M quantization. Actual download size may differ slightly by engine/format. The prompt labels it as "~X GB estimated". + +2. **Download prompt:** `click.confirm(f"Download {model} (~{size:.1f} GB estimated) now?", default=True)`. A new `--no-download` flag on the `init` command skips this prompt (for CI/automated environments). + +3. **Engine-specific download functions:** + - `_download_ollama(model, host, console)` — calls a new shared helper `ollama_pull(host, model_name, console) -> bool` extracted from `cli/model.py`. Both the `model pull` CLI command and `init_cmd.py` import this helper, avoiding duplication. + - `_download_llamacpp(model, spec, console)` — `subprocess.run(["huggingface-cli", "download", repo, gguf_file, "--local-dir", cache_dir])`. If `huggingface-cli` is not found (`FileNotFoundError`), prints: `"Install huggingface-cli: pip install huggingface_hub"` and shows the manual download URL as fallback. + - `_download_mlx(model, spec, console)` — `subprocess.run(["huggingface-cli", "download", mlx_repo, "--local-dir", cache_dir])`. Same `FileNotFoundError` handling as llama.cpp. + - `_download_auto(model, engine, console)` — print info message that the model auto-downloads on first serve (vLLM, SGLang). + - `_download_manual(model, engine, console)` — print engine-specific manual instructions (LM Studio, Exo, Nexa). + +4. **Empty model fallback:** Check `if not model:` and print the "not enough memory" warning before the next-steps panel. + +5. **Privacy hook:** After download (or skip), call `_quick_privacy_check()` that runs disk encryption + cloud sync checks and prints a compact summary. + +6. **Next-steps coverage:** Add entries to `_next_steps_text()` for `exo` and `nexa` engines so the Ollama fallback is eliminated. + +### `jarvis model pull` Enhancement (`src/openjarvis/cli/model.py`) + +**Refactor:** Extract the Ollama streaming pull logic (currently lines 165-196) into a reusable function: + +```python +def ollama_pull(host: str, model_name: str, console: Console) -> bool: + """Pull a model via Ollama API. Returns True on success.""" +``` + +The existing `pull` Click command becomes a thin wrapper around this function. + +**Extend** the `pull` command to support engines beyond Ollama: +- Add an `--engine` flag (optional, defaults to configured engine). +- Look up the model in `BUILTIN_MODELS` or `ModelRegistry` to get metadata. +- Dispatch to engine-specific download based on the engine flag. +- Fallback to Ollama pull if no engine specified and model is in Ollama format (name:tag). +- llama.cpp and MLX paths use `huggingface-cli download` with metadata from the catalog, with `FileNotFoundError` handling. + +### Privacy Scanner (`src/openjarvis/cli/scan_cmd.py` — new file) + +#### Data model + +```python +@dataclass +class ScanResult: + name: str # e.g. "FileVault" + status: str # "ok" | "warn" | "fail" | "skip" + message: str # Human-readable explanation + platform: str # "darwin" | "linux" | "all" +``` + +#### `PrivacyScanner` class + +Contains a list of check methods. Each check: +- Is decorated or tagged with the platform it applies to. +- Calls a subprocess and parses the output. +- Returns a `ScanResult`. +- Never raises — catches all exceptions and returns `status="skip"` with the error. + +#### macOS checks + +| Check | Implementation | Status mapping | +|-------|---------------|----------------| +| `check_filevault()` | `subprocess.run(["fdesetup", "status"])`, parse "FileVault is On/Off" | On = ok, Off = fail | +| `check_mdm()` | `subprocess.run(["profiles", "status", "-type", "enrollment"])`, parse for "MDM enrollment" | Not enrolled = ok, enrolled = warn | +| `check_icloud_sync()` | Read `defaults read MobileMeAccounts` for Desktop/Documents sync; check if `~/.openjarvis/` resolves to a path under `~/Library/Mobile Documents/`. Note: `defaults read` output format varies across macOS versions — parsing should be lenient and return `skip` on unexpected output rather than raising. | Not syncing = ok, syncing = warn | +| `check_cloud_sync_agents()` | Check for Dropbox, OneDrive, Google Drive processes via `pgrep`; check if their known sync dirs overlap with `~/.openjarvis/` | No overlap = ok, overlap = warn | +| `check_network_exposure()` | `lsof -iTCP -sTCP:LISTEN -nP`, filter for known engine ports, check bind address | 127.0.0.1 = ok, 0.0.0.0 = warn | +| `check_screen_recording()` | `pgrep` for TeamViewer, AnyDesk, ScreenConnect, vnc | Not found = ok, found = warn | + +#### Linux checks + +| Check | Implementation | Status mapping | +|-------|---------------|----------------| +| `check_luks()` | `lsblk -o NAME,TYPE,FSTYPE -J`, look for `crypto_LUKS` on root device | Found = ok, not found = fail | +| `check_cloud_sync_agents()` | `pgrep` for rclone, dropbox, insync, onedriver | Not found = ok, found = warn | +| `check_network_exposure()` | `ss -tlnp` or `lsof`, same port-check logic | 127.0.0.1 = ok, 0.0.0.0 = warn | +| `check_remote_access()` | `pgrep` for xrdp, x11vnc, vncserver, AnyDesk | Not found = ok, found = warn | + +#### CLI command + +``` +@click.command() +def scan(): + """Audit your environment for privacy and security risks.""" +``` + +Output format: +``` + Privacy & Environment Audit + ──────────────────────────── + ✓ FileVault: enabled + ✓ Network: inference ports bound to localhost only + ! iCloud Drive: Desktop & Documents sync is active + ✗ MDM: enterprise management profile detected + + 1 warning, 1 issue found. +``` + +Symbols: `✓` for ok, `!` for warn, `✗` for fail. Skipped checks are hidden. + +#### Init integration + +A function `_quick_privacy_check()` in `init_cmd.py` that: +1. Instantiates `PrivacyScanner`. +2. Runs only disk encryption and cloud sync overlap checks. +3. Prints a compact 2-3 line summary. +4. Always prints `Run 'jarvis scan' for a full environment audit.` + +### File changes summary + +| File | Type | Description | +|------|------|-------------| +| `src/openjarvis/intelligence/model_catalog.py` | Edit | Add MLX to supported_engines, add gguf_file/mlx_repo metadata | +| `src/openjarvis/core/config.py` | Edit | Add `estimated_download_gb(spec: ModelSpec) -> float` helper (computes `parameter_count_b * 0.5 * 1.1`) | +| `src/openjarvis/cli/init_cmd.py` | Edit | Add download prompt, engine-specific downloaders, empty-model fallback, privacy hook | +| `src/openjarvis/cli/model.py` | Edit | Extend `pull` for llama.cpp and MLX engines | +| `src/openjarvis/cli/scan_cmd.py` | New | `PrivacyScanner` class, check functions, `jarvis scan` command | +| `src/openjarvis/cli/__init__.py` | Edit | Register `scan` command | + +## Testing + +### Issue 1 — Model onboarding + +| Test | File | What it verifies | +|------|------|-----------------| +| MLX model recommendation | `tests/core/test_recommend_model.py` | Apple Silicon 8/16/32/64GB returns valid model with MLX engine (8GB should get `qwen3.5:4b`, 16GB should get `qwen3.5:14b`) | +| Empty model fallback message | `tests/cli/test_init_guidance.py` | `recommend_model` returning `""` shows "not enough memory" | +| Download prompt shown | `tests/cli/test_init_guidance.py` | Init output includes download prompt when model recommended | +| Engine-specific download dispatch | `tests/cli/test_init_guidance.py` | Ollama triggers pull, llamacpp shows HF download, vllm shows auto-download | +| `model pull` multi-engine | `tests/cli/test_model_pull.py` | llama.cpp and MLX pull paths work (mocked subprocess) | + +### Issue 2 — Privacy scanner + +| Test | File | What it verifies | +|------|------|-----------------| +| FileVault check | `tests/cli/test_scan.py` | Parses `fdesetup status` for on/off | +| MDM check | `tests/cli/test_scan.py` | Parses `profiles status` for enrollment | +| iCloud sync detection | `tests/cli/test_scan.py` | Detects overlap with `~/.openjarvis/` | +| LUKS check | `tests/cli/test_scan.py` | Parses `lsblk` for crypto_LUKS | +| Network exposure | `tests/cli/test_scan.py` | Parses `lsof`/`ss` for 0.0.0.0 binds | +| Platform filtering | `tests/cli/test_scan.py` | macOS checks return skip on Linux, vice versa | +| Init privacy hook | `tests/cli/test_init_guidance.py` | Init output includes privacy summary | + +All tests mock subprocess calls via `monkeypatch`. No real system state required. + +## Out of Scope + +- Windows support (future work). +- Post-download inference validation (trying a test query). +- Rust implementation of scanner checks. +- DNS leak detection (complex, unreliable heuristics). +- Automatic remediation (e.g., moving `~/.openjarvis/` out of iCloud). From d97e1dd266ac7b54c517c3d365911f8d36863dfa Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon Date: Wed, 25 Mar 2026 00:33:17 +0000 Subject: [PATCH 52/92] revert: remove _fix_tool_call_arguments from OpenAI-compatible engines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts the tool_call arguments string→dict conversion added in PR #69. The OpenAI API spec requires tool_call arguments as JSON strings, not dicts. vLLM with --enable-auto-tool-choice validates this and returns 400 errors when arguments are dicts. The original 400 errors were caused by missing --enable-auto-tool-choice/--tool-call-parser flags on the vLLM server, not by the arguments format. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/engine/_openai_compat.py | 26 ++----------------------- 1 file changed, 2 insertions(+), 24 deletions(-) diff --git a/src/openjarvis/engine/_openai_compat.py b/src/openjarvis/engine/_openai_compat.py index ba7d3870..b859bfd9 100644 --- a/src/openjarvis/engine/_openai_compat.py +++ b/src/openjarvis/engine/_openai_compat.py @@ -32,26 +32,6 @@ class _OpenAICompatibleEngine(InferenceEngine): # -- InferenceEngine interface ------------------------------------------ - @staticmethod - def _fix_tool_call_arguments(msg_dicts: list) -> list: - """Ensure tool_call arguments are dicts, not JSON strings. - - OpenAI-compatible servers (vLLM, SGLang, llama.cpp, etc.) expect - tool_call arguments as JSON objects. ``messages_to_dicts`` may - serialize them as strings, which causes 400 errors on multi-turn - tool-calling conversations. - """ - for md in msg_dicts: - for tc in md.get("tool_calls", []): - fn = tc.get("function", {}) - args = fn.get("arguments") - if isinstance(args, str): - try: - fn["arguments"] = json.loads(args) - except (json.JSONDecodeError, TypeError): - pass - return msg_dicts - def generate( self, messages: Sequence[Message], @@ -61,10 +41,9 @@ class _OpenAICompatibleEngine(InferenceEngine): max_tokens: int = 1024, **kwargs: Any, ) -> Dict[str, Any]: - msg_dicts = self._fix_tool_call_arguments(messages_to_dicts(messages)) payload: Dict[str, Any] = { "model": model, - "messages": msg_dicts, + "messages": messages_to_dicts(messages), "temperature": temperature, "max_tokens": max_tokens, "stream": False, @@ -125,10 +104,9 @@ class _OpenAICompatibleEngine(InferenceEngine): max_tokens: int = 1024, **kwargs: Any, ) -> AsyncIterator[str]: - msg_dicts = self._fix_tool_call_arguments(messages_to_dicts(messages)) payload: Dict[str, Any] = { "model": model, - "messages": msg_dicts, + "messages": messages_to_dicts(messages), "temperature": temperature, "max_tokens": max_tokens, "stream": True, From b2a88a5514c8908c5b4a85744058b530fc3e82aa Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 17:56:31 -0700 Subject: [PATCH 53/92] docs: add implementation plan for init onboarding and privacy scanner 6 tasks with TDD steps, covering model catalog fixes, multi-engine pull, interactive download in init, privacy scanner, and CLI registration. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...nit-onboarding-and-privacy-scanner-plan.md | 1413 +++++++++++++++++ 1 file changed, 1413 insertions(+) create mode 100644 docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md diff --git a/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md b/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md new file mode 100644 index 00000000..acb292da --- /dev/null +++ b/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md @@ -0,0 +1,1413 @@ +# Init Model Onboarding & Privacy Scanner Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Fix the broken model recommendation for MLX users, add interactive model download to `jarvis init`, and create a `jarvis scan` privacy environment audit command. + +**Architecture:** Two independent features sharing only the init command integration point. Feature 1 modifies the model catalog and init flow. Feature 2 creates a new `PrivacyScanner` class with platform-specific checks exposed via `jarvis scan` CLI command and a lightweight hook in init. + +**Tech Stack:** Python 3.10+, Click (CLI), Rich (terminal UI), httpx (Ollama API), subprocess (system checks) + +**Spec:** `docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md` + +--- + +### Task 1: Fix MLX support in model catalog + +**Files:** +- Modify: `src/openjarvis/intelligence/model_catalog.py:42-54` (qwen3.5:3b) +- Modify: `src/openjarvis/intelligence/model_catalog.py:55-67` (qwen3.5:8b) +- Modify: `src/openjarvis/intelligence/model_catalog.py:68-80` (qwen3.5:14b) +- Modify: `src/openjarvis/intelligence/model_catalog.py:219-232` (qwen3.5:4b) +- Test: `tests/core/test_recommend_model.py` + +- [ ] **Step 1: Write failing tests for MLX model recommendation** + +Add a new test class to `tests/core/test_recommend_model.py`: + +```python +class TestRecommendModelMlx: + """Apple Silicon (MLX) model recommendation.""" + + def test_apple_silicon_8gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=8.0, + gpu=GpuInfo(vendor="apple", name="Apple M1", vram_gb=8.0, count=1), + ) + result = recommend_model(hw, "mlx") + # available = 8 * 0.9 = 7.2 GB + # 8B * 0.5 * 1.1 = 4.4 → fits, but check 14B first: 7.7 → too big + # 4B * 0.5 * 1.1 = 2.2 → fits, but 8B also fits → pick 8B + assert result == "qwen3.5:8b" + + def test_apple_silicon_16gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=16.0, + gpu=GpuInfo(vendor="apple", name="Apple M2", vram_gb=16.0, count=1), + ) + result = recommend_model(hw, "mlx") + # available = 16 * 0.9 = 14.4 GB + # 14B * 0.5 * 1.1 = 7.7 → fits + assert result == "qwen3.5:14b" + + def test_apple_silicon_32gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=32.0, + gpu=GpuInfo(vendor="apple", name="Apple M2 Pro", vram_gb=32.0, count=1), + ) + result = recommend_model(hw, "mlx") + # available = 32 * 0.9 = 28.8 GB + # 14B * 0.5 * 1.1 = 7.7 → fits (14b is the largest with mlx support) + assert result == "qwen3.5:14b" + + def test_apple_silicon_64gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=64.0, + gpu=GpuInfo(vendor="apple", name="Apple M2 Max", vram_gb=64.0, count=1), + ) + result = recommend_model(hw, "mlx") + # available = 64 * 0.9 = 57.6 GB — but 14b is the largest MLX model + assert result == "qwen3.5:14b" +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/core/test_recommend_model.py::TestRecommendModelMlx -v` +Expected: FAIL — all 4 tests fail because no Qwen3.5 model has `"mlx"` in `supported_engines`. + +- [ ] **Step 3: Add MLX to supported_engines and download metadata** + +In `src/openjarvis/intelligence/model_catalog.py`, update the four Qwen3.5 model entries: + +For `qwen3.5:3b` (line 48): change `supported_engines` from `("ollama", "vllm", "llamacpp", "sglang")` to `("ollama", "vllm", "llamacpp", "sglang", "mlx")`. Add `"gguf_file": "qwen3.5-3b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-3B-4bit"` to metadata. + +For `qwen3.5:8b` (line 61): change `supported_engines` from `("ollama", "vllm", "llamacpp", "sglang")` to `("ollama", "vllm", "llamacpp", "sglang", "mlx")`. Add `"gguf_file": "qwen3.5-8b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-8B-4bit"` to metadata. + +For `qwen3.5:14b` (line 74): change `supported_engines` from `("ollama", "vllm", "llamacpp", "sglang")` to `("ollama", "vllm", "llamacpp", "sglang", "mlx")`. Add `"gguf_file": "qwen3.5-14b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-14B-4bit"` to metadata. + +For `qwen3.5:4b` (line 226): change `supported_engines` from `("ollama", "vllm", "sglang", "llamacpp")` to `("ollama", "vllm", "sglang", "llamacpp", "mlx")`. Add `"gguf_file": "qwen3.5-4b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-4B-4bit"` to metadata. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/core/test_recommend_model.py -v` +Expected: ALL 15 tests pass (11 existing + 4 new). + +- [ ] **Step 5: Add `estimated_download_gb` helper to config.py** + +In `src/openjarvis/core/config.py`, after the `recommend_model` function (after line 248), add: + +```python +def estimated_download_gb(parameter_count_b: float) -> float: + """Estimate download size in GB for Q4_K_M quantized model.""" + return parameter_count_b * 0.5 * 1.1 +``` + +- [ ] **Step 6: Commit** + +```bash +git add src/openjarvis/intelligence/model_catalog.py src/openjarvis/core/config.py tests/core/test_recommend_model.py +git commit -m "fix: add MLX engine support to Qwen3.5 model catalog entries + +Fixes recommend_model() returning empty string on Apple Silicon +when MLX is the recommended engine. Also adds gguf_file and +mlx_repo download metadata, and estimated_download_gb helper." +``` + +--- + +### Task 2: Extract Ollama pull helper and extend `jarvis model pull` + +**Files:** +- Modify: `src/openjarvis/cli/model.py:152-197` +- Test: `tests/cli/test_model_pull.py` (new) + +- [ ] **Step 1: Write failing tests for multi-engine pull** + +Create `tests/cli/test_model_pull.py`: + +```python +"""Tests for ``jarvis model pull`` multi-engine support.""" + +from __future__ import annotations + +from unittest import mock + +import pytest +from click.testing import CliRunner +from rich.console import Console + +from openjarvis.cli.model import ollama_pull + + +class TestOllamaPull: + """Test the extracted ollama_pull helper.""" + + def test_ollama_pull_success(self) -> None: + import io + console = Console(file=io.StringIO()) + mock_lines = [ + '{"status": "pulling manifest"}', + '{"status": "downloading", "total": 100, "completed": 100}', + '{"status": "success"}', + ] + mock_resp = mock.MagicMock() + mock_resp.raise_for_status = mock.MagicMock() + mock_resp.iter_lines.return_value = iter(mock_lines) + mock_resp.__enter__ = mock.MagicMock(return_value=mock_resp) + mock_resp.__exit__ = mock.MagicMock(return_value=False) + + with mock.patch("httpx.stream", return_value=mock_resp): + result = ollama_pull("http://localhost:11434", "qwen3.5:3b", console) + assert result is True + + def test_ollama_pull_connect_error(self) -> None: + import io + import httpx + + console = Console(file=io.StringIO()) + with mock.patch("httpx.stream", side_effect=httpx.ConnectError("refused")): + result = ollama_pull("http://localhost:11434", "qwen3.5:3b", console) + assert result is False + + +class TestPullCliMultiEngine: + """Test the pull CLI command dispatches to correct engine.""" + + def test_pull_llamacpp_uses_huggingface_cli(self) -> None: + from openjarvis.cli import cli + + runner = CliRunner() + with ( + mock.patch("openjarvis.cli.model.load_config") as mock_cfg, + mock.patch("subprocess.run") as mock_run, + ): + mock_cfg.return_value.engine.default = "llamacpp" + mock_cfg.return_value.engine.ollama_host = None + mock_run.return_value = mock.MagicMock(returncode=0) + + result = runner.invoke( + cli, ["model", "pull", "qwen3.5:8b", "--engine", "llamacpp"] + ) + + assert result.exit_code == 0 + mock_run.assert_called_once() + call_args = mock_run.call_args[0][0] + assert "huggingface-cli" in call_args + assert "qwen3.5-8b-q4_k_m.gguf" in call_args + + def test_pull_mlx_uses_huggingface_cli(self) -> None: + from openjarvis.cli import cli + + runner = CliRunner() + with ( + mock.patch("openjarvis.cli.model.load_config") as mock_cfg, + mock.patch("subprocess.run") as mock_run, + ): + mock_cfg.return_value.engine.default = "mlx" + mock_cfg.return_value.engine.ollama_host = None + mock_run.return_value = mock.MagicMock(returncode=0) + + result = runner.invoke( + cli, ["model", "pull", "qwen3.5:8b", "--engine", "mlx"] + ) + + assert result.exit_code == 0 + mock_run.assert_called_once() + call_args = mock_run.call_args[0][0] + assert "huggingface-cli" in call_args + assert "mlx-community/Qwen3.5-8B-4bit" in call_args + + def test_pull_llamacpp_huggingface_cli_not_found(self) -> None: + from openjarvis.cli import cli + + runner = CliRunner() + with ( + mock.patch("openjarvis.cli.model.load_config") as mock_cfg, + mock.patch("subprocess.run", side_effect=FileNotFoundError), + ): + mock_cfg.return_value.engine.default = "llamacpp" + mock_cfg.return_value.engine.ollama_host = None + + result = runner.invoke( + cli, ["model", "pull", "qwen3.5:8b", "--engine", "llamacpp"] + ) + + assert result.exit_code != 0 + assert "huggingface_hub" in result.output or "pip install" in result.output +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/cli/test_model_pull.py -v` +Expected: FAIL — `ollama_pull` function doesn't exist yet, `--engine` flag not recognized. + +- [ ] **Step 3: Refactor model.py — extract ollama_pull and add multi-engine support** + +In `src/openjarvis/cli/model.py`: + +1. Add `import subprocess` at the top. +2. Add `from openjarvis.intelligence.model_catalog import BUILTIN_MODELS` at the top. +3. Extract the Ollama pull logic into a standalone function: + +```python +def ollama_pull(host: str, model_name: str, console: Console) -> bool: + """Pull a model via Ollama API. Returns True on success.""" + console.print(f"Pulling [cyan]{model_name}[/cyan] via Ollama...") + try: + with httpx.stream( + "POST", + f"{host}/api/pull", + json={"name": model_name, "stream": True}, + timeout=600.0, + ) as resp: + resp.raise_for_status() + import json + + for line in resp.iter_lines(): + if not line.strip(): + continue + try: + data = json.loads(line) + except Exception: + continue + status = data.get("status", "") + if "total" in data and "completed" in data: + total = data["total"] + done = data["completed"] + pct = int(done / total * 100) if total else 0 + console.print(f" {status}: {pct}%", end="\r") + elif status: + console.print(f" {status}") + console.print(f"\n[green]Successfully pulled {model_name}[/green]") + return True + except httpx.ConnectError: + console.print("[red]Cannot connect to Ollama.[/red] Is it running?") + return False + except httpx.HTTPStatusError as exc: + console.print(f"[red]Ollama error:[/red] {exc.response.status_code}") + return False +``` + +4. Add helper to find model spec: + +```python +def find_model_spec(model_name: str): + """Look up a model in the builtin catalog. Returns None if not found.""" + for spec in BUILTIN_MODELS: + if spec.model_id == model_name: + return spec + return None +``` + +5. Add HuggingFace download helper: + +```python +def hf_download(repo: str, filename: str | None, console: Console) -> bool: + """Download from HuggingFace via huggingface-cli. Returns True on success.""" + cmd = ["huggingface-cli", "download", repo] + if filename: + cmd.append(filename) + try: + result = subprocess.run(cmd, check=True) + console.print(f"[green]Download complete.[/green]") + return True + except FileNotFoundError: + console.print( + "[red]huggingface-cli not found.[/red]\n" + "Install it: [cyan]pip install huggingface_hub[/cyan]\n" + f"Or download manually: https://huggingface.co/{repo}" + ) + return False + except subprocess.CalledProcessError: + console.print(f"[red]Download failed.[/red]") + return False +``` + +6. Replace the existing `pull` command with a multi-engine version: + +```python +@model.command() +@click.argument("model_name") +@click.option("--engine", default=None, help="Engine to download for (ollama, llamacpp, mlx).") +def pull(model_name: str, engine: str | None) -> None: + """Download a model.""" + console = Console() + config = load_config() + + engine = engine or config.engine.default or "ollama" + + if engine == "ollama": + host = ( + config.engine.ollama_host + or os.environ.get("OLLAMA_HOST") + or "http://localhost:11434" + ).rstrip("/") + if not ollama_pull(host, model_name, console): + sys.exit(1) + elif engine in ("llamacpp", "mlx"): + spec = find_model_spec(model_name) + if not spec: + console.print(f"[red]Model not in catalog:[/red] {model_name}") + sys.exit(1) + if engine == "llamacpp": + repo = spec.metadata.get("hf_repo", "") + gguf = spec.metadata.get("gguf_file", "") + if not repo or not gguf: + console.print(f"[red]No GGUF download info for {model_name}[/red]") + sys.exit(1) + console.print(f"Downloading [cyan]{gguf}[/cyan] from {repo}...") + if not hf_download(repo, gguf, console): + sys.exit(1) + else: # mlx + mlx_repo = spec.metadata.get("mlx_repo", "") + if not mlx_repo: + console.print(f"[red]No MLX repo info for {model_name}[/red]") + sys.exit(1) + console.print(f"Downloading [cyan]{mlx_repo}[/cyan]...") + if not hf_download(mlx_repo, None, console): + sys.exit(1) + elif engine in ("vllm", "sglang"): + console.print( + f"[cyan]{model_name}[/cyan] will download automatically when " + f"{engine} starts serving it." + ) + else: + console.print( + f"Manual download required for engine [cyan]{engine}[/cyan].\n" + f"Check the engine documentation for instructions." + ) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/cli/test_model_pull.py -v` +Expected: ALL tests pass. + +- [ ] **Step 5: Run existing tests to verify no regressions** + +Run: `uv run pytest tests/core/test_recommend_model.py tests/cli/test_init_guidance.py -v` +Expected: ALL pass. + +- [ ] **Step 6: Commit** + +```bash +git add src/openjarvis/cli/model.py tests/cli/test_model_pull.py +git commit -m "feat: extract ollama_pull helper and add multi-engine model pull + +Refactors model pull into reusable ollama_pull() function. Adds +--engine flag to support llamacpp (GGUF) and mlx (HuggingFace) +downloads via huggingface-cli, with FileNotFoundError handling." +``` + +--- + +### Task 3: Add interactive download and empty-model fallback to `jarvis init` + +**Files:** +- Modify: `src/openjarvis/cli/init_cmd.py:139-299` +- Test: `tests/cli/test_init_guidance.py` + +- [ ] **Step 1: Write failing tests for download prompt and empty-model fallback** + +Add to `tests/cli/test_init_guidance.py`: + +```python +class TestInitDownloadPrompt: + """Interactive download prompt during init.""" + + def test_init_shows_download_prompt(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + ): + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama"], input="n\n" + ) + assert result.exit_code == 0 + assert "Download" in result.output and "now?" in result.output + + def test_init_no_download_flag_skips_prompt(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + ): + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama", "--no-download"] + ) + assert result.exit_code == 0 + assert "Download" not in result.output or "now?" not in result.output + + +class TestInitEmptyModelFallback: + """Empty model recommendation shows helpful message.""" + + def test_init_no_model_shows_warning(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch( + "openjarvis.cli.init_cmd.recommend_model", return_value="" + ), + ): + result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) + assert result.exit_code == 0 + assert "Not enough memory" in result.output or "not enough memory" in result.output + + +class TestNextStepsExoNexa: + """Exo and Nexa have their own next-steps text.""" + + def test_next_steps_exo(self) -> None: + text = _next_steps_text("exo") + assert "exo" in text.lower() + assert "jarvis ask" in text + # Should NOT fall back to Ollama instructions + assert "ollama" not in text.lower() + + def test_next_steps_nexa(self) -> None: + text = _next_steps_text("nexa") + assert "nexa" in text.lower() + assert "jarvis ask" in text + assert "ollama" not in text.lower() +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/cli/test_init_guidance.py::TestInitDownloadPrompt tests/cli/test_init_guidance.py::TestInitEmptyModelFallback tests/cli/test_init_guidance.py::TestNextStepsExoNexa -v` +Expected: FAIL — no `--no-download` flag, no "Not enough memory" message, no exo/nexa next-steps. + +- [ ] **Step 3: Update init_cmd.py** + +In `src/openjarvis/cli/init_cmd.py`: + +1. Add imports at the top: + +```python +from openjarvis.cli.model import ollama_pull, find_model_spec, hf_download +from openjarvis.core.config import estimated_download_gb +``` + +2. Add `exo` and `nexa` entries to `_next_steps_text()` dict (before the closing `}`): + +```python + "exo": ( + "Next steps:\n" + "\n" + " 1. Install and start Exo:\n" + " pip install exo\n" + " exo\n" + "\n" + " 2. Try it out:\n" + " jarvis ask \"Hello\"\n" + "\n" + " Run `jarvis doctor` to verify your setup." + ), + "nexa": ( + "Next steps:\n" + "\n" + " 1. Install and start Nexa:\n" + " pip install nexaai\n" + " nexa server\n" + "\n" + " 2. Try it out:\n" + " jarvis ask \"Hello\"\n" + "\n" + " Run `jarvis doctor` to verify your setup." + ), +``` + +3. Add `--no-download` flag to the `@click.command()` decorator chain (after `--engine`): + +```python +@click.option( + "--no-download", + is_flag=True, + default=False, + help="Skip the model download prompt.", +) +``` + +4. Update the `init()` function signature to accept `no_download: bool = False`. + +5. Replace the block at lines 287-298 (the model recommendation + next-steps panel) with: + +```python + selected_engine = engine or recommend_engine(hw) + model = recommend_model(hw, selected_engine) + + if not model: + console.print( + "\n [yellow]! Not enough memory to run any local model.[/yellow]\n" + " Consider a cloud engine or a machine with more RAM." + ) + else: + spec = find_model_spec(model) + size_gb = estimated_download_gb(spec.parameter_count_b) if spec else 0 + console.print(f"\n [bold]Recommended model:[/bold] {model} (~{size_gb:.1f} GB estimated)") + + if not no_download and spec: + if click.confirm(f" Download {model} (~{size_gb:.1f} GB estimated) now?", default=True): + _do_download(selected_engine, model, spec, console) + + console.print() + console.print( + Panel( + _next_steps_text(selected_engine, model), + title="Getting Started", + border_style="cyan", + ) + ) +``` + +6. Add the `_do_download` helper function: + +```python +def _do_download(engine: str, model: str, spec, console: Console) -> None: + """Dispatch model download based on engine type.""" + import os + + if engine == "ollama": + host = os.environ.get("OLLAMA_HOST", "http://localhost:11434").rstrip("/") + ollama_pull(host, model, console) + elif engine == "llamacpp": + repo = spec.metadata.get("hf_repo", "") + gguf = spec.metadata.get("gguf_file", "") + if repo and gguf: + console.print(f" Downloading [cyan]{gguf}[/cyan] from {repo}...") + hf_download(repo, gguf, console) + else: + console.print(f" [yellow]No GGUF download info for {model}[/yellow]") + elif engine == "mlx": + mlx_repo = spec.metadata.get("mlx_repo", "") + if mlx_repo: + console.print(f" Downloading [cyan]{mlx_repo}[/cyan]...") + hf_download(mlx_repo, None, console) + else: + console.print(f" [yellow]No MLX repo info for {model}[/yellow]") + elif engine in ("vllm", "sglang"): + console.print( + f" [cyan]{model}[/cyan] will download automatically when " + f"{engine} starts serving it." + ) + else: + console.print( + f" Download {model} through the {engine} interface." + ) +``` + +- [ ] **Step 4: Update existing init tests to add `--no-download`** + +The download prompt breaks existing tests because `click.confirm` will try to download a model. Update all existing init CLI tests in `tests/cli/test_init_guidance.py` to add `"--no-download"` to their invocation args: + +- `test_init_shows_next_steps`: change `["init", "--engine", "llamacpp"]` to `["init", "--engine", "llamacpp", "--no-download"]` +- `test_init_output_shows_toml_sections_literally`: same change +- `test_init_generates_minimal_by_default`: change `["init", "--engine", "ollama"]` to `["init", "--engine", "ollama", "--no-download"]` +- `test_init_full_generates_verbose_config`: change `["init", "--full", "--engine", "ollama"]` to `["init", "--full", "--engine", "ollama", "--no-download"]` + +- [ ] **Step 5: Add download dispatch tests** + +Add to `tests/cli/test_init_guidance.py`: + +```python +class TestInitDownloadDispatch: + """Verify download dispatches correctly for each engine.""" + + def test_init_ollama_download_calls_ollama_pull(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.ollama_pull", return_value=True) as mock_pull, + ): + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama"], input="y\n" + ) + assert result.exit_code == 0 + mock_pull.assert_called_once() + + def test_init_vllm_shows_auto_download_message(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + ): + result = CliRunner().invoke( + cli, ["init", "--engine", "vllm"], input="y\n" + ) + assert result.exit_code == 0 + assert "automatically" in result.output +``` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `uv run pytest tests/cli/test_init_guidance.py -v` +Expected: ALL tests pass (10 existing + 7 new). + +- [ ] **Step 7: Lint** + +Run: `uv run ruff check src/openjarvis/cli/init_cmd.py --fix` +Expected: Clean or auto-fixed. + +- [ ] **Step 8: Commit** + +```bash +git add src/openjarvis/cli/init_cmd.py tests/cli/test_init_guidance.py +git commit -m "feat: add interactive model download and empty-model fallback to init + +Prompts user to download recommended model during jarvis init. +Adds --no-download flag for CI. Shows helpful message when no +model fits available memory. Adds exo/nexa next-steps text." +``` + +--- + +### Task 4: Create privacy scanner — ScanResult and check functions + +**Files:** +- Create: `src/openjarvis/cli/scan_cmd.py` +- Test: `tests/cli/test_scan.py` (new) + +- [ ] **Step 1: Write failing tests for individual scanner checks** + +Create `tests/cli/test_scan.py`: + +```python +"""Tests for ``jarvis scan`` privacy environment audit.""" + +from __future__ import annotations + +import subprocess +import sys +from unittest import mock + +import pytest + +from openjarvis.cli.scan_cmd import PrivacyScanner, ScanResult + + +class TestScanResultDataclass: + def test_scan_result_fields(self) -> None: + r = ScanResult(name="Test", status="ok", message="All good", platform="all") + assert r.name == "Test" + assert r.status == "ok" + assert r.message == "All good" + assert r.platform == "all" + + +class TestFileVault: + def test_filevault_enabled(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout="FileVault is On.", returncode=0 + ) + result = scanner.check_filevault() + assert result.status == "ok" + assert "enabled" in result.message.lower() + + def test_filevault_disabled(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout="FileVault is Off.", returncode=0 + ) + result = scanner.check_filevault() + assert result.status == "fail" + + def test_filevault_command_not_found(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run", side_effect=FileNotFoundError): + result = scanner.check_filevault() + assert result.status == "skip" + + +class TestMDM: + def test_mdm_not_enrolled(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout="This machine is not enrolled", returncode=0 + ) + result = scanner.check_mdm() + assert result.status == "ok" + + def test_mdm_enrolled(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout="Enrolled via DEP: Yes\nMDM server: example.com", + returncode=0, + ) + result = scanner.check_mdm() + assert result.status == "warn" + + +class TestCloudSync: + def test_no_cloud_sync_agents(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + # pgrep returns exit code 1 when no match + mock_run.return_value = mock.MagicMock(stdout="", returncode=1) + result = scanner.check_cloud_sync_agents() + assert result.status == "ok" + + def test_dropbox_running(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + def side_effect(cmd, **kwargs): + m = mock.MagicMock() + # Match on the process name in the pgrep pattern + if any("Dropbox" in str(c) for c in cmd): + m.stdout = "12345" + m.returncode = 0 + else: + m.stdout = "" + m.returncode = 1 + return m + mock_run.side_effect = side_effect + result = scanner.check_cloud_sync_agents() + assert result.status == "warn" + assert "dropbox" in result.message.lower() + + +class TestNetworkExposure: + def test_no_exposed_ports(self) -> None: + scanner = PrivacyScanner() + lsof_output = ( + "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n" + "ollama 12345 user 5u IPv4 0x1234 0t0 TCP 127.0.0.1:11434 (LISTEN)\n" + ) + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout=lsof_output, returncode=0 + ) + result = scanner.check_network_exposure() + assert result.status == "ok" + + def test_exposed_port(self) -> None: + scanner = PrivacyScanner() + lsof_output = ( + "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n" + "ollama 12345 user 5u IPv4 0x1234 0t0 TCP *:11434 (LISTEN)\n" + ) + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout=lsof_output, returncode=0 + ) + result = scanner.check_network_exposure() + assert result.status == "warn" + assert "11434" in result.message + + +class TestLUKS: + def test_luks_encrypted(self) -> None: + scanner = PrivacyScanner() + lsblk_json = '{"blockdevices": [{"name": "sda", "type": "disk", "fstype": null, "children": [{"name": "sda1", "type": "part", "fstype": "crypto_LUKS"}]}]}' + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout=lsblk_json, returncode=0 + ) + result = scanner.check_luks() + assert result.status == "ok" + + def test_luks_not_encrypted(self) -> None: + scanner = PrivacyScanner() + lsblk_json = '{"blockdevices": [{"name": "sda", "type": "disk", "fstype": null, "children": [{"name": "sda1", "type": "part", "fstype": "ext4"}]}]}' + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock( + stdout=lsblk_json, returncode=0 + ) + result = scanner.check_luks() + assert result.status == "fail" + + +class TestScreenRecording: + def test_no_screen_recording(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + mock_run.return_value = mock.MagicMock(stdout="", returncode=1) + result = scanner.check_screen_recording() + assert result.status == "ok" + + def test_teamviewer_running(self) -> None: + scanner = PrivacyScanner() + with mock.patch("subprocess.run") as mock_run: + def side_effect(cmd, **kwargs): + m = mock.MagicMock() + if any("TeamViewer" in str(c) for c in cmd): + m.stdout = "12345" + m.returncode = 0 + else: + m.stdout = "" + m.returncode = 1 + return m + mock_run.side_effect = side_effect + result = scanner.check_screen_recording() + assert result.status == "warn" + + +class TestPlatformFiltering: + def test_run_all_returns_only_current_platform(self) -> None: + scanner = PrivacyScanner() + with mock.patch.object(scanner, "_get_all_checks") as mock_checks: + mock_checks.return_value = [ + lambda: ScanResult("Test1", "ok", "ok", "darwin"), + lambda: ScanResult("Test2", "ok", "ok", "linux"), + lambda: ScanResult("Test3", "ok", "ok", "all"), + ] + results = scanner.run_all() + current = sys.platform + for r in results: + assert r.platform in (current, "all") +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run pytest tests/cli/test_scan.py -v` +Expected: FAIL — `scan_cmd` module doesn't exist. + +- [ ] **Step 3: Implement PrivacyScanner class** + +Create `src/openjarvis/cli/scan_cmd.py`: + +```python +"""``jarvis scan`` — privacy and environment audit.""" + +from __future__ import annotations + +import json +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Callable, List + +import click +from rich.console import Console + + +@dataclass(slots=True) +class ScanResult: + """Result of a single privacy check.""" + + name: str + status: str # "ok" | "warn" | "fail" | "skip" + message: str + platform: str # "darwin" | "linux" | "all" + + +# Engine ports to check for network exposure +_ENGINE_PORTS = { + 11434: "Ollama", + 8080: "llama.cpp / MLX", + 8000: "vLLM", + 30000: "SGLang", + 1234: "LM Studio", + 52415: "Exo", + 18181: "Nexa", +} + +# Cloud sync process names to check +_CLOUD_SYNC_PROCESSES = ["Dropbox", "OneDrive", "Google Drive", "iCloudDrive"] + +# Screen recording / remote access process names +_SCREEN_RECORDING_PROCESSES_MACOS = [ + "TeamViewer", "AnyDesk", "ScreenConnect", "VNC", +] +_REMOTE_ACCESS_PROCESSES_LINUX = [ + "xrdp", "x11vnc", "vncserver", "AnyDesk", "TeamViewer", +] + + +class PrivacyScanner: + """Run platform-specific privacy and environment checks.""" + + def _run(self, cmd: list[str], **kwargs) -> subprocess.CompletedProcess: + """Run a subprocess, capturing stdout as text.""" + return subprocess.run( + cmd, capture_output=True, text=True, timeout=10, **kwargs + ) + + # ------------------------------------------------------------------ + # macOS checks + # ------------------------------------------------------------------ + + def check_filevault(self) -> ScanResult: + """Check macOS FileVault disk encryption status.""" + try: + proc = self._run(["fdesetup", "status"]) + if "On" in proc.stdout: + return ScanResult("FileVault", "ok", "FileVault enabled", "darwin") + return ScanResult( + "FileVault", "fail", + "FileVault is disabled — disk is not encrypted", "darwin" + ) + except (FileNotFoundError, subprocess.TimeoutExpired): + return ScanResult("FileVault", "skip", "fdesetup not available", "darwin") + + def check_mdm(self) -> ScanResult: + """Check for MDM / enterprise management enrollment.""" + try: + proc = self._run(["profiles", "status", "-type", "enrollment"]) + output = proc.stdout + proc.stderr + if "MDM" in output or "Enrolled" in output: + return ScanResult( + "MDM", "warn", + "Enterprise management profile detected — this device may be monitored", + "darwin", + ) + return ScanResult("MDM", "ok", "No MDM enrollment detected", "darwin") + except (FileNotFoundError, subprocess.TimeoutExpired): + return ScanResult("MDM", "skip", "profiles command not available", "darwin") + + def check_icloud_sync(self) -> ScanResult: + """Check if ~/.openjarvis/ could be synced by iCloud Drive.""" + try: + oj_path = Path.home() / ".openjarvis" + # Check if the path is under iCloud's mobile documents + mobile_docs = Path.home() / "Library" / "Mobile Documents" + try: + resolved = oj_path.resolve() + if str(resolved).startswith(str(mobile_docs)): + return ScanResult( + "iCloud Drive", "warn", + "~/.openjarvis/ is inside iCloud Drive sync path", + "darwin", + ) + except OSError: + pass + + # Check if Desktop & Documents sync is enabled + proc = self._run( + ["defaults", "read", "com.apple.bird", "optimize-storage"] + ) + # Also check the broader MobileMeAccounts for sync status + proc2 = self._run( + ["defaults", "read", "MobileMeAccounts"] + ) + output = proc.stdout + proc2.stdout + if "MOBILE_DOCUMENTS" in output or "Desktop" in output: + return ScanResult( + "iCloud Drive", "warn", + "iCloud Desktop & Documents sync may be active — " + "~/.openjarvis/ could be synced to Apple servers", + "darwin", + ) + return ScanResult( + "iCloud Drive", "ok", + "No iCloud sync overlap detected", "darwin" + ) + except (FileNotFoundError, subprocess.TimeoutExpired): + return ScanResult( + "iCloud Drive", "skip", + "Could not determine iCloud sync status", "darwin" + ) + except Exception: + return ScanResult( + "iCloud Drive", "skip", + "Could not determine iCloud sync status", "darwin" + ) + + # ------------------------------------------------------------------ + # Linux checks + # ------------------------------------------------------------------ + + def check_luks(self) -> ScanResult: + """Check for LUKS disk encryption on Linux.""" + try: + proc = self._run(["lsblk", "-o", "NAME,TYPE,FSTYPE", "-J"]) + data = json.loads(proc.stdout) + + def _has_luks(devices: list) -> bool: + for dev in devices: + if dev.get("fstype") == "crypto_LUKS": + return True + if _has_luks(dev.get("children", [])): + return True + return False + + if _has_luks(data.get("blockdevices", [])): + return ScanResult( + "Disk Encryption", "ok", + "LUKS encryption detected", "linux" + ) + return ScanResult( + "Disk Encryption", "fail", + "No LUKS encryption detected — disk may not be encrypted", + "linux", + ) + except (FileNotFoundError, subprocess.TimeoutExpired): + return ScanResult( + "Disk Encryption", "skip", + "lsblk not available", "linux" + ) + except (json.JSONDecodeError, KeyError): + return ScanResult( + "Disk Encryption", "skip", + "Could not parse lsblk output", "linux" + ) + + def check_remote_access(self) -> ScanResult: + """Check for remote access tools on Linux.""" + return self._check_processes( + _REMOTE_ACCESS_PROCESSES_LINUX, + "Remote Access", + "Remote access tool detected", + "linux", + ) + + # ------------------------------------------------------------------ + # Cross-platform checks + # ------------------------------------------------------------------ + + def check_cloud_sync_agents(self) -> ScanResult: + """Check for running cloud sync agents.""" + platform = "darwin" if sys.platform == "darwin" else "linux" + return self._check_processes( + _CLOUD_SYNC_PROCESSES, + "Cloud Sync", + "Cloud sync agent detected", + platform, + ) + + def check_network_exposure(self) -> ScanResult: + """Check if inference engine ports are bound to 0.0.0.0.""" + platform = "darwin" if sys.platform == "darwin" else "linux" + try: + if sys.platform == "darwin": + proc = self._run(["lsof", "-iTCP", "-sTCP:LISTEN", "-nP"]) + else: + proc = self._run(["ss", "-tlnp"]) + + exposed = [] + for line in proc.stdout.splitlines(): + for port, engine_name in _ENGINE_PORTS.items(): + port_str = str(port) + if port_str in line and ("*:" + port_str in line or "0.0.0.0:" + port_str in line): + exposed.append(f"{engine_name} (port {port})") + + if exposed: + return ScanResult( + "Network Exposure", "warn", + f"Inference ports exposed to network: {', '.join(exposed)}", + platform, + ) + return ScanResult( + "Network Exposure", "ok", + "Inference ports bound to localhost only", platform + ) + except (FileNotFoundError, subprocess.TimeoutExpired): + return ScanResult( + "Network Exposure", "skip", + "Could not check listening ports", platform + ) + + def check_screen_recording(self) -> ScanResult: + """Check for screen recording / remote desktop tools (macOS).""" + return self._check_processes( + _SCREEN_RECORDING_PROCESSES_MACOS, + "Screen Recording", + "Screen recording or remote access tool detected", + "darwin", + ) + + # ------------------------------------------------------------------ + # Helpers + # ------------------------------------------------------------------ + + def _check_processes( + self, + process_names: list[str], + check_name: str, + warn_message: str, + platform: str, + ) -> ScanResult: + """Check if any of the named processes are running.""" + found = [] + for name in process_names: + try: + proc = self._run(["pgrep", "-i", name]) + if proc.returncode == 0 and proc.stdout.strip(): + found.append(name) + except (FileNotFoundError, subprocess.TimeoutExpired): + continue + if found: + return ScanResult( + check_name, "warn", + f"{warn_message}: {', '.join(found)}", platform + ) + return ScanResult( + check_name, "ok", + f"No {check_name.lower()} detected", platform + ) + + def _get_all_checks(self) -> List[Callable[[], ScanResult]]: + """Return all check methods.""" + return [ + self.check_filevault, + self.check_mdm, + self.check_icloud_sync, + self.check_luks, + self.check_cloud_sync_agents, + self.check_network_exposure, + self.check_screen_recording, + self.check_remote_access, + ] + + def run_all(self) -> list[ScanResult]: + """Run all checks, filtering to current platform.""" + results = [] + for check in self._get_all_checks(): + result = check() + if result.platform in (sys.platform, "all"): + if result.status != "skip": + results.append(result) + return results + + def run_quick(self) -> list[ScanResult]: + """Run only critical checks (for init hook).""" + checks = [self.check_filevault, self.check_luks, self.check_icloud_sync, + self.check_cloud_sync_agents] + results = [] + for check in checks: + result = check() + if result.platform in (sys.platform, "all"): + if result.status != "skip": + results.append(result) + return results + + +# ------------------------------------------------------------------ +# CLI command +# ------------------------------------------------------------------ + +_STATUS_SYMBOLS = {"ok": "\u2713", "warn": "!", "fail": "\u2717"} + + +@click.command() +def scan() -> None: + """Audit your environment for privacy and security risks.""" + console = Console() + scanner = PrivacyScanner() + results = scanner.run_all() + + console.print() + console.print(" [bold]Privacy & Environment Audit[/bold]") + console.print(" " + "\u2500" * 28) + + warns = 0 + fails = 0 + for r in results: + symbol = _STATUS_SYMBOLS.get(r.status, "?") + if r.status == "ok": + console.print(f" [green]{symbol}[/green] {r.name}: {r.message}") + elif r.status == "warn": + console.print(f" [yellow]{symbol}[/yellow] {r.name}: {r.message}") + warns += 1 + elif r.status == "fail": + console.print(f" [red]{symbol}[/red] {r.name}: {r.message}") + fails += 1 + + console.print() + if warns == 0 and fails == 0: + console.print(" [green]No issues found.[/green]") + else: + parts = [] + if warns: + parts.append(f"{warns} warning{'s' if warns != 1 else ''}") + if fails: + parts.append(f"{fails} issue{'s' if fails != 1 else ''}") + console.print(f" {', '.join(parts)} found.") + console.print() +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run pytest tests/cli/test_scan.py -v` +Expected: ALL tests pass. + +- [ ] **Step 5: Lint** + +Run: `uv run ruff check src/openjarvis/cli/scan_cmd.py --fix` +Expected: Clean or auto-fixed. + +- [ ] **Step 6: Commit** + +```bash +git add src/openjarvis/cli/scan_cmd.py tests/cli/test_scan.py +git commit -m "feat: add privacy environment scanner with platform-specific checks + +New PrivacyScanner class with checks for disk encryption (FileVault/LUKS), +MDM profiles, cloud sync agents, network exposure, and screen recording. +Supports macOS and Linux with graceful skip on missing tools." +``` + +--- + +### Task 5: Register `jarvis scan` and integrate privacy hook into init + +**Files:** +- Modify: `src/openjarvis/cli/__init__.py:34` (add import) +- Modify: `src/openjarvis/cli/__init__.py:89` (register command) +- Modify: `src/openjarvis/cli/init_cmd.py` (add privacy hook) +- Test: `tests/cli/test_init_guidance.py` + +- [ ] **Step 1: Write failing test for init privacy hook** + +Add to `tests/cli/test_init_guidance.py`: + +```python +class TestInitPrivacyHook: + """Init shows a lightweight privacy summary.""" + + def test_init_shows_privacy_summary(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner") as MockScanner, + ): + from openjarvis.cli.scan_cmd import ScanResult + instance = MockScanner.return_value + instance.run_quick.return_value = [ + ScanResult("FileVault", "ok", "FileVault enabled", "darwin"), + ] + result = CliRunner().invoke( + cli, ["init", "--engine", "llamacpp", "--no-download"] + ) + assert result.exit_code == 0 + assert "jarvis scan" in result.output +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/cli/test_init_guidance.py::TestInitPrivacyHook -v` +Expected: FAIL — no `PrivacyScanner` import in init_cmd, no privacy output. + +- [ ] **Step 3: Update all existing init tests to mock PrivacyScanner** + +After adding the privacy hook to init, all existing init CLI tests will call real system commands (`fdesetup`, `pgrep`, etc.). Add `mock.patch("openjarvis.cli.init_cmd.PrivacyScanner")` to every existing init CLI test's context manager block in `tests/cli/test_init_guidance.py`. This includes: + +- `test_init_shows_next_steps` +- `test_init_output_shows_toml_sections_literally` +- `test_init_generates_minimal_by_default` +- `test_init_full_generates_verbose_config` +- `test_init_shows_download_prompt` +- `test_init_no_download_flag_skips_prompt` +- `test_init_no_model_shows_warning` +- `test_init_ollama_download_calls_ollama_pull` +- `test_init_vllm_shows_auto_download_message` + +For each, add inside the `with (...)` block: +```python +mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), +``` + +- [ ] **Step 4: Register scan command and add init privacy hook** + +In `src/openjarvis/cli/__init__.py`, add after line 33: + +```python +from openjarvis.cli.scan_cmd import scan +``` + +After line 89 (the last `cli.add_command` call), add: + +```python +cli.add_command(scan, "scan") +``` + +In `src/openjarvis/cli/init_cmd.py`, add import: + +```python +from openjarvis.cli.scan_cmd import PrivacyScanner +``` + +Add `_quick_privacy_check` function: + +```python +def _quick_privacy_check(console: Console) -> None: + """Run critical privacy checks and print compact summary.""" + scanner = PrivacyScanner() + results = scanner.run_quick() + + if results: + console.print(" [bold]Privacy check:[/bold]") + for r in results: + if r.status == "ok": + console.print(f" [green]\u2713[/green] {r.message}") + elif r.status == "warn": + console.print(f" [yellow]![/yellow] {r.message}") + elif r.status == "fail": + console.print(f" [red]\u2717[/red] {r.message}") + + console.print() + console.print(" Run [cyan]jarvis scan[/cyan] for a full environment audit.") +``` + +Call `_quick_privacy_check(console)` at the end of the `init()` function, just before the final "Getting Started" panel. + +- [ ] **Step 4: Run all tests to verify everything passes** + +Run: `uv run pytest tests/cli/test_init_guidance.py tests/cli/test_scan.py tests/core/test_recommend_model.py tests/cli/test_model_pull.py -v` +Expected: ALL tests pass. + +- [ ] **Step 5: Lint all changed files** + +Run: `uv run ruff check src/openjarvis/cli/ --fix` +Expected: Clean. + +- [ ] **Step 6: Commit** + +```bash +git add src/openjarvis/cli/__init__.py src/openjarvis/cli/init_cmd.py tests/cli/test_init_guidance.py +git commit -m "feat: register jarvis scan command and add init privacy hook + +Registers the new scan command in the CLI. Adds a lightweight +privacy check at the end of jarvis init that runs disk encryption +and cloud sync checks, with pointer to jarvis scan for full audit." +``` + +--- + +### Task 6: Final integration test and cleanup + +**Files:** +- All modified files from tasks 1-5 + +- [ ] **Step 1: Run full test suite for all touched modules** + +Run: `uv run pytest tests/core/test_recommend_model.py tests/cli/test_init_guidance.py tests/cli/test_model_pull.py tests/cli/test_scan.py -v` +Expected: ALL pass. + +- [ ] **Step 2: Run linter on all source files** + +Run: `uv run ruff check src/openjarvis/cli/ src/openjarvis/intelligence/model_catalog.py src/openjarvis/core/config.py` +Expected: Clean. + +- [ ] **Step 3: Verify the MLX bug is fixed** + +Run: `uv run python3 -c "from openjarvis.core.config import HardwareInfo, GpuInfo, recommend_model; hw = HardwareInfo(platform='darwin', ram_gb=16.0, gpu=GpuInfo(vendor='apple', name='Apple M1', vram_gb=16.0, count=1)); print(f'MLX model: {recommend_model(hw, \"mlx\")}')"` +Expected output: `MLX model: qwen3.5:14b` + +- [ ] **Step 4: Verify the CLI scan command is registered** + +Run: `uv run jarvis scan --help` +Expected: Shows help text for the scan command. + +- [ ] **Step 5: Commit if any cleanup was needed** + +Only if changes were made during cleanup. Otherwise skip. From 7d3b4e5ea3bb893e726e2a89c9726dc7e79b893d Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:04:25 -0700 Subject: [PATCH 54/92] fix: add MLX engine support to Qwen3.5 model catalog entries Fixes recommend_model() returning empty string on Apple Silicon when MLX is the recommended engine. Also adds gguf_file and mlx_repo download metadata, and estimated_download_gb helper. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/core/config.py | 5 +++ src/openjarvis/intelligence/model_catalog.py | 16 ++++++-- tests/core/test_recommend_model.py | 40 ++++++++++++++++++++ 3 files changed, 57 insertions(+), 4 deletions(-) diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 5ca8fa77..33160232 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -248,6 +248,11 @@ def recommend_model(hw: HardwareInfo, engine: str) -> str: return "" +def estimated_download_gb(parameter_count_b: float) -> float: + """Estimate download size in GB for Q4_K_M quantized model.""" + return parameter_count_b * 0.5 * 1.1 + + # --------------------------------------------------------------------------- # Configuration hierarchy # --------------------------------------------------------------------------- diff --git a/src/openjarvis/intelligence/model_catalog.py b/src/openjarvis/intelligence/model_catalog.py index d77c1740..ee36976f 100644 --- a/src/openjarvis/intelligence/model_catalog.py +++ b/src/openjarvis/intelligence/model_catalog.py @@ -45,11 +45,13 @@ BUILTIN_MODELS: List[ModelSpec] = [ parameter_count_b=3.0, active_parameter_count_b=0.6, context_length=131072, - supported_engines=("ollama", "vllm", "llamacpp", "sglang"), + supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"), provider="alibaba", metadata={ "architecture": "moe", "hf_repo": "Qwen/Qwen3.5-3B", + "gguf_file": "qwen3.5-3b-q4_k_m.gguf", + "mlx_repo": "mlx-community/Qwen3.5-3B-4bit", }, ), ModelSpec( @@ -58,11 +60,13 @@ BUILTIN_MODELS: List[ModelSpec] = [ parameter_count_b=8.0, active_parameter_count_b=1.0, context_length=131072, - supported_engines=("ollama", "vllm", "llamacpp", "sglang"), + supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"), provider="alibaba", metadata={ "architecture": "moe", "hf_repo": "Qwen/Qwen3.5-8B", + "gguf_file": "qwen3.5-8b-q4_k_m.gguf", + "mlx_repo": "mlx-community/Qwen3.5-8B-4bit", }, ), ModelSpec( @@ -71,11 +75,13 @@ BUILTIN_MODELS: List[ModelSpec] = [ parameter_count_b=14.0, active_parameter_count_b=2.0, context_length=131072, - supported_engines=("ollama", "vllm", "llamacpp", "sglang"), + supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"), provider="alibaba", metadata={ "architecture": "moe", "hf_repo": "Qwen/Qwen3.5-14B", + "gguf_file": "qwen3.5-14b-q4_k_m.gguf", + "mlx_repo": "mlx-community/Qwen3.5-14B-4bit", }, ), ModelSpec( @@ -223,11 +229,13 @@ BUILTIN_MODELS: List[ModelSpec] = [ active_parameter_count_b=0.5, context_length=262144, min_vram_gb=3.0, - supported_engines=("ollama", "vllm", "sglang", "llamacpp"), + supported_engines=("ollama", "vllm", "sglang", "llamacpp", "mlx"), provider="alibaba", metadata={ "architecture": "moe", "hf_repo": "Qwen/Qwen3.5-4B", + "gguf_file": "qwen3.5-4b-q4_k_m.gguf", + "mlx_repo": "mlx-community/Qwen3.5-4B-4bit", }, ), ModelSpec( diff --git a/tests/core/test_recommend_model.py b/tests/core/test_recommend_model.py index 8f03d430..61600b69 100644 --- a/tests/core/test_recommend_model.py +++ b/tests/core/test_recommend_model.py @@ -115,3 +115,43 @@ class TestRecommendModelEdgeCases: # With ollama, 397b is excluded (only vllm, sglang) result = recommend_model(hw, "ollama") assert result == "qwen3.5:122b" + + +class TestRecommendModelMlx: + """Apple Silicon (MLX) model recommendation.""" + + def test_apple_silicon_8gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=8.0, + gpu=GpuInfo(vendor="apple", name="Apple M1", vram_gb=8.0, count=1), + ) + result = recommend_model(hw, "mlx") + assert result == "qwen3.5:8b" + + def test_apple_silicon_16gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=16.0, + gpu=GpuInfo(vendor="apple", name="Apple M2", vram_gb=16.0, count=1), + ) + result = recommend_model(hw, "mlx") + assert result == "qwen3.5:14b" + + def test_apple_silicon_32gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=32.0, + gpu=GpuInfo(vendor="apple", name="Apple M2 Pro", vram_gb=32.0, count=1), + ) + result = recommend_model(hw, "mlx") + assert result == "qwen3.5:14b" + + def test_apple_silicon_64gb_mlx(self) -> None: + hw = HardwareInfo( + platform="darwin", + ram_gb=64.0, + gpu=GpuInfo(vendor="apple", name="Apple M2 Max", vram_gb=64.0, count=1), + ) + result = recommend_model(hw, "mlx") + assert result == "qwen3.5:14b" From ced38bcba33514173fbb51a64f1329f4faccac8d Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:07:54 -0700 Subject: [PATCH 55/92] feat: add privacy environment scanner with platform-specific checks New PrivacyScanner class with checks for disk encryption (FileVault/LUKS), MDM profiles, cloud sync agents, network exposure, and screen recording. Supports macOS and Linux with graceful skip on missing tools. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/scan_cmd.py | 386 +++++++++++++++++++++++++++++++++ tests/cli/test_scan.py | 256 ++++++++++++++++++++++ 2 files changed, 642 insertions(+) create mode 100644 src/openjarvis/cli/scan_cmd.py create mode 100644 tests/cli/test_scan.py diff --git a/src/openjarvis/cli/scan_cmd.py b/src/openjarvis/cli/scan_cmd.py new file mode 100644 index 00000000..e8d83e08 --- /dev/null +++ b/src/openjarvis/cli/scan_cmd.py @@ -0,0 +1,386 @@ +"""``jarvis scan`` — audit your environment for privacy and security risks.""" + +from __future__ import annotations + +import json +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Callable, List + +import click + +# Engine ports that should only be listening on localhost. +_ENGINE_PORTS = {11434, 8080, 8000, 30000, 1234, 52415, 18181} + +# Processes associated with cloud-sync agents. +_CLOUD_SYNC_PROCS = ["Dropbox", "OneDrive", "Google Drive", "iCloudDrive"] + +# Screen-recording / remote-access processes (macOS). +_SCREEN_RECORDING_PROCS = [ + "TeamViewer", "AnyDesk", "ScreenConnect", "vncviewer", "Vine" +] + +# Remote-access processes (Linux). +_REMOTE_ACCESS_PROCS = ["xrdp", "x11vnc", "vncserver", "AnyDesk"] + + +# --------------------------------------------------------------------------- +# Data model +# --------------------------------------------------------------------------- + + +@dataclass +class ScanResult: + """Result of a single privacy/security check.""" + + name: str + status: str # "ok" | "warn" | "fail" | "skip" + message: str + platform: str # "darwin" | "linux" | "all" + + +# --------------------------------------------------------------------------- +# Scanner +# --------------------------------------------------------------------------- + + +class PrivacyScanner: + """Collection of environment privacy checks.""" + + # -- Subprocess helper --------------------------------------------------- + + def _run(self, cmd: list[str]) -> subprocess.CompletedProcess: # type: ignore[type-arg] + return subprocess.run(cmd, capture_output=True, text=True, timeout=10) + + # -- Individual checks --------------------------------------------------- + + def check_filevault(self) -> ScanResult: + """Check whether FileVault disk encryption is enabled (macOS).""" + try: + proc = self._run(["fdesetup", "status"]) + if "On" in proc.stdout: + return ScanResult( + name="FileVault", + status="ok", + message="FileVault is enabled.", + platform="darwin", + ) + return ScanResult( + name="FileVault", + status="fail", + message="FileVault is NOT enabled. Enable full-disk encryption.", + platform="darwin", + ) + except Exception: + return ScanResult( + name="FileVault", + status="skip", + message="fdesetup not available.", + platform="darwin", + ) + + def check_mdm(self) -> ScanResult: + """Check whether the device is enrolled in an MDM profile (macOS).""" + try: + proc = self._run(["profiles", "status", "-type", "enrollment"]) + output = proc.stdout + proc.stderr + lower = output.lower() + # "not enrolled" is a strong negative signal — check it first. + not_enrolled = "not enrolled" in lower or "no" in lower + # Positive signals: explicit Yes/enrolled without a negation. + enrolled_yes = ( + "mdm enrollment: yes" in lower + or "enrolled via dep: yes" in lower + or ("enrolled" in lower and not not_enrolled) + ) + if enrolled_yes: + return ScanResult( + name="MDM Enrollment", + status="warn", + message="Device appears to be enrolled in an MDM profile.", + platform="darwin", + ) + return ScanResult( + name="MDM Enrollment", + status="ok", + message="Device is not enrolled in an MDM profile.", + platform="darwin", + ) + except Exception: + return ScanResult( + name="MDM Enrollment", + status="skip", + message="profiles command not available.", + platform="darwin", + ) + + def check_icloud_sync(self) -> ScanResult: + """Check whether ~/.openjarvis is inside iCloud Drive sync scope.""" + try: + config_path = Path("~/.openjarvis").expanduser().resolve() + icloud_path = Path("~/Library/Mobile Documents/").expanduser().resolve() + if str(config_path).startswith(str(icloud_path)): + return ScanResult( + name="iCloud Sync", + status="warn", + message="~/.openjarvis may be synced to iCloud.", + platform="darwin", + ) + # Also probe defaults for com.apple.bird (iCloud daemon) + try: + proc = self._run( + ["defaults", "read", "com.apple.bird", "optout_preference"] + ) + val = proc.stdout.strip() + if val == "0": + return ScanResult( + name="iCloud Sync", + status="warn", + message="iCloud Desktop/Documents sync may be active.", + platform="darwin", + ) + except Exception: + pass + return ScanResult( + name="iCloud Sync", + status="ok", + message="~/.openjarvis is not inside iCloud Drive.", + platform="darwin", + ) + except Exception: + return ScanResult( + name="iCloud Sync", + status="skip", + message="Could not determine iCloud sync status.", + platform="darwin", + ) + + def check_luks(self) -> ScanResult: + """Check whether any block device uses LUKS encryption (Linux).""" + try: + proc = self._run(["lsblk", "-o", "NAME,TYPE,FSTYPE", "-J"]) + data = json.loads(proc.stdout) + except Exception: + return ScanResult( + name="LUKS Encryption", + status="skip", + message="lsblk not available or returned unexpected output.", + platform="linux", + ) + + def _has_luks(devices: list) -> bool: # type: ignore[type-arg] + for dev in devices: + if dev.get("fstype") == "crypto_LUKS": + return True + children = dev.get("children") or [] + if _has_luks(children): + return True + return False + + try: + devices = data.get("blockdevices", []) + if _has_luks(devices): + return ScanResult( + name="LUKS Encryption", + status="ok", + message="At least one LUKS-encrypted device found.", + platform="linux", + ) + return ScanResult( + name="LUKS Encryption", + status="fail", + message="No LUKS-encrypted block devices found.", + platform="linux", + ) + except Exception: + return ScanResult( + name="LUKS Encryption", + status="skip", + message="Could not parse lsblk output.", + platform="linux", + ) + + def _check_processes( + self, + names: list[str], + check_name: str, + warn_msg: str, + platform: str, + ) -> ScanResult: + """Shared helper: pgrep for any of the given process names.""" + try: + for name in names: + try: + proc = self._run(["pgrep", "-x", name]) + if proc.returncode == 0: + return ScanResult( + name=check_name, + status="warn", + message=warn_msg.format(name=name), + platform=platform, + ) + except Exception: + continue + return ScanResult( + name=check_name, + status="ok", + message=f"No {check_name.lower()} processes detected.", + platform=platform, + ) + except Exception: + return ScanResult( + name=check_name, + status="skip", + message="pgrep not available.", + platform=platform, + ) + + def check_cloud_sync_agents(self) -> ScanResult: + """Check for running cloud-sync agent processes.""" + return self._check_processes( + names=_CLOUD_SYNC_PROCS, + check_name="Cloud Sync Agents", + warn_msg="{name} sync agent is running — weights may be uploaded to cloud.", + platform="all", + ) + + def check_network_exposure(self) -> ScanResult: + """Check if engine ports are exposed on 0.0.0.0 rather than localhost.""" + try: + if sys.platform == "darwin": + proc = self._run(["lsof", "-iTCP", "-sTCP:LISTEN", "-n", "-P"]) + else: + proc = self._run(["ss", "-tlnp"]) + output = proc.stdout + + exposed: list[int] = [] + for port in _ENGINE_PORTS: + # Look for patterns like *:PORT or 0.0.0.0:PORT + for token in (f"*:{port}", f"0.0.0.0:{port}", f":::{ port}"): + if token.replace(" ", "") in output.replace(" ", ""): + exposed.append(port) + break + + if exposed: + ports_str = ", ".join(str(p) for p in sorted(exposed)) + return ScanResult( + name="Network Exposure", + status="warn", + message=f"Engine port(s) {ports_str} exposed on all interfaces.", + platform="all", + ) + return ScanResult( + name="Network Exposure", + status="ok", + message="All engine ports appear to be bound to localhost only.", + platform="all", + ) + except Exception: + return ScanResult( + name="Network Exposure", + status="skip", + message="Could not determine network exposure (lsof/ss unavailable).", + platform="all", + ) + + def check_screen_recording(self) -> ScanResult: + """Check for running screen-recording / remote-desktop processes (macOS).""" + return self._check_processes( + names=_SCREEN_RECORDING_PROCS, + check_name="Screen Recording", + warn_msg="{name} is running — screen may be accessible remotely.", + platform="darwin", + ) + + def check_remote_access(self) -> ScanResult: + """Check for running remote-access processes (Linux).""" + return self._check_processes( + names=_REMOTE_ACCESS_PROCS, + check_name="Remote Access", + warn_msg="{name} is running — system may be accessible remotely.", + platform="linux", + ) + + # -- Orchestration ------------------------------------------------------- + + def _get_all_checks(self) -> list[Callable[[], ScanResult]]: + return [ + self.check_filevault, + self.check_mdm, + self.check_icloud_sync, + self.check_cloud_sync_agents, + self.check_network_exposure, + self.check_luks, + self.check_screen_recording, + self.check_remote_access, + ] + + def run_all(self) -> list[ScanResult]: + """Run all checks, filter to the current platform, hide 'skip' results.""" + current_plat = "darwin" if sys.platform == "darwin" else "linux" + results: list[ScanResult] = [] + for check_fn in self._get_all_checks(): + result = check_fn() + if result.platform not in (current_plat, "all"): + continue + if result.status == "skip": + continue + results.append(result) + return results + + def run_quick(self) -> list[ScanResult]: + """Run only critical checks: disk encryption + cloud sync agents.""" + current_plat = "darwin" if sys.platform == "darwin" else "linux" + quick_checks: list[Callable[[], ScanResult]] + if current_plat == "darwin": + quick_checks = [self.check_filevault, self.check_cloud_sync_agents] + else: + quick_checks = [self.check_luks, self.check_cloud_sync_agents] + results = [] + for check_fn in quick_checks: + result = check_fn() + if result.status != "skip": + results.append(result) + return results + + +# --------------------------------------------------------------------------- +# CLI command +# --------------------------------------------------------------------------- + +_STATUS_ICONS = {"ok": "✓", "warn": "!", "fail": "✗", "skip": "-"} + + +@click.command() +@click.option("--quick", is_flag=True, default=False, help="Run only critical checks.") +def scan(quick: bool) -> None: + """Audit your environment for privacy and security risks.""" + scanner = PrivacyScanner() + results: List[ScanResult] = scanner.run_quick() if quick else scanner.run_all() + + if not results: + click.echo("No applicable checks for this platform.") + return + + warnings = 0 + failures = 0 + for r in results: + icon = _STATUS_ICONS.get(r.status, "?") + click.echo(f" [{icon}] {r.name}: {r.message}") + if r.status == "warn": + warnings += 1 + elif r.status == "fail": + failures += 1 + + click.echo("") + parts = [] + if warnings: + parts.append(f"{warnings} warning(s)") + if failures: + parts.append(f"{failures} issue(s)") + if parts: + click.echo("Summary: " + ", ".join(parts) + ".") + else: + click.echo("Summary: all checks passed.") diff --git a/tests/cli/test_scan.py b/tests/cli/test_scan.py new file mode 100644 index 00000000..9b8eea7e --- /dev/null +++ b/tests/cli/test_scan.py @@ -0,0 +1,256 @@ +"""Tests for ``jarvis scan`` privacy scanner CLI command.""" + +from __future__ import annotations + +import json +from subprocess import CompletedProcess +from unittest.mock import MagicMock, patch + +import pytest + +from openjarvis.cli.scan_cmd import ScanResult, PrivacyScanner + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_proc(stdout: str = "", stderr: str = "", returncode: int = 0) -> CompletedProcess: + return CompletedProcess(args=[], returncode=returncode, stdout=stdout, stderr=stderr) + + +# --------------------------------------------------------------------------- +# TestScanResultDataclass +# --------------------------------------------------------------------------- + + +class TestScanResultDataclass: + def test_fields_exist(self) -> None: + r = ScanResult(name="test", status="ok", message="all good", platform="all") + assert r.name == "test" + assert r.status == "ok" + assert r.message == "all good" + assert r.platform == "all" + + def test_status_values(self) -> None: + for status in ("ok", "warn", "fail", "skip"): + r = ScanResult(name="x", status=status, message="", platform="all") + assert r.status == status + + def test_platform_values(self) -> None: + for plat in ("darwin", "linux", "all"): + r = ScanResult(name="x", status="ok", message="", platform=plat) + assert r.platform == plat + + +# --------------------------------------------------------------------------- +# TestFileVault +# --------------------------------------------------------------------------- + + +class TestFileVault: + def test_filevault_enabled(self) -> None: + scanner = PrivacyScanner() + with patch("subprocess.run", return_value=_make_proc(stdout="FileVault is On.")): + result = scanner.check_filevault() + assert result.status == "ok" + + def test_filevault_disabled(self) -> None: + scanner = PrivacyScanner() + with patch("subprocess.run", return_value=_make_proc(stdout="FileVault is Off.")): + result = scanner.check_filevault() + assert result.status == "fail" + + def test_command_not_found(self) -> None: + scanner = PrivacyScanner() + with patch("subprocess.run", side_effect=FileNotFoundError): + result = scanner.check_filevault() + assert result.status == "skip" + + +# --------------------------------------------------------------------------- +# TestMDM +# --------------------------------------------------------------------------- + + +class TestMDM: + def test_not_enrolled(self) -> None: + scanner = PrivacyScanner() + with patch( + "subprocess.run", + return_value=_make_proc(stdout="Enrolled via DEP: No\nMDM enrollment: not enrolled"), + ): + result = scanner.check_mdm() + assert result.status == "ok" + + def test_enrolled(self) -> None: + scanner = PrivacyScanner() + with patch( + "subprocess.run", + return_value=_make_proc(stdout="MDM enrollment: Yes\nEnrolled via DEP: Yes"), + ): + result = scanner.check_mdm() + assert result.status == "warn" + + +# --------------------------------------------------------------------------- +# TestCloudSync +# --------------------------------------------------------------------------- + + +class TestCloudSync: + def test_no_agents(self) -> None: + """pgrep returns non-zero (process not found) → no sync agents running.""" + scanner = PrivacyScanner() + with patch("subprocess.run", return_value=_make_proc(returncode=1, stdout="")): + result = scanner.check_cloud_sync_agents() + assert result.status == "ok" + + def test_dropbox_running(self) -> None: + """pgrep returns 0 when Dropbox is in the command.""" + scanner = PrivacyScanner() + + def _pgrep_side_effect(cmd, **kwargs): + if "Dropbox" in cmd: + return _make_proc(returncode=0, stdout="1234") + return _make_proc(returncode=1, stdout="") + + with patch("subprocess.run", side_effect=_pgrep_side_effect): + result = scanner.check_cloud_sync_agents() + assert result.status == "warn" + + +# --------------------------------------------------------------------------- +# TestNetworkExposure +# --------------------------------------------------------------------------- + + +class TestNetworkExposure: + def test_no_exposed_ports(self) -> None: + """Only localhost bindings → ok.""" + scanner = PrivacyScanner() + # lsof / ss output showing only 127.0.0.1 + lsof_output = "ollama 1234 user IPv4 TCP 127.0.0.1:11434 (LISTEN)\n" + with patch("subprocess.run", return_value=_make_proc(stdout=lsof_output)): + result = scanner.check_network_exposure() + assert result.status == "ok" + + def test_exposed_port(self) -> None: + """A port bound to 0.0.0.0 or * → warn with port in message.""" + scanner = PrivacyScanner() + lsof_output = "ollama 1234 user IPv4 TCP *:11434 (LISTEN)\n" + with patch("subprocess.run", return_value=_make_proc(stdout=lsof_output)): + result = scanner.check_network_exposure() + assert result.status == "warn" + assert "11434" in result.message + + +# --------------------------------------------------------------------------- +# TestLUKS +# --------------------------------------------------------------------------- + + +class TestLUKS: + def test_encrypted(self) -> None: + """lsblk JSON contains crypto_LUKS → ok.""" + scanner = PrivacyScanner() + lsblk_data = { + "blockdevices": [ + { + "name": "sda", + "type": "disk", + "fstype": None, + "children": [ + {"name": "sda1", "type": "part", "fstype": "crypto_LUKS"} + ], + } + ] + } + with patch( + "subprocess.run", + return_value=_make_proc(stdout=json.dumps(lsblk_data)), + ): + result = scanner.check_luks() + assert result.status == "ok" + + def test_not_encrypted(self) -> None: + """lsblk JSON has only ext4 → fail.""" + scanner = PrivacyScanner() + lsblk_data = { + "blockdevices": [ + {"name": "sda", "type": "disk", "fstype": None}, + {"name": "sda1", "type": "part", "fstype": "ext4"}, + ] + } + with patch( + "subprocess.run", + return_value=_make_proc(stdout=json.dumps(lsblk_data)), + ): + result = scanner.check_luks() + assert result.status == "fail" + + +# --------------------------------------------------------------------------- +# TestScreenRecording +# --------------------------------------------------------------------------- + + +class TestScreenRecording: + def test_none_running(self) -> None: + """No screen-recording processes → ok.""" + scanner = PrivacyScanner() + with patch("subprocess.run", return_value=_make_proc(returncode=1, stdout="")): + result = scanner.check_screen_recording() + assert result.status == "ok" + + def test_teamviewer_running(self) -> None: + """TeamViewer found → warn.""" + scanner = PrivacyScanner() + + def _pgrep_side_effect(cmd, **kwargs): + if "TeamViewer" in cmd: + return _make_proc(returncode=0, stdout="5678") + return _make_proc(returncode=1, stdout="") + + with patch("subprocess.run", side_effect=_pgrep_side_effect): + result = scanner.check_screen_recording() + assert result.status == "warn" + + +# --------------------------------------------------------------------------- +# TestPlatformFiltering +# --------------------------------------------------------------------------- + + +class TestPlatformFiltering: + def test_run_all_filters_to_current_platform(self) -> None: + """run_all() should only call checks tagged 'all' or current platform.""" + scanner = PrivacyScanner() + called_platforms: list[str] = [] + + # Patch every check method to record its platform tag instead of running + checks = scanner._get_all_checks() + for check_fn in checks: + # Run the real method but capture which ones get included + pass + + import sys + + current_plat = "darwin" if sys.platform == "darwin" else "linux" + other_plat = "linux" if current_plat == "darwin" else "darwin" + + # Make every check return immediately with a known platform + with patch.object(scanner, "_get_all_checks") as mock_get: + darwin_check = MagicMock(return_value=ScanResult("d", "ok", "msg", "darwin")) + linux_check = MagicMock(return_value=ScanResult("l", "ok", "msg", "linux")) + all_check = MagicMock(return_value=ScanResult("a", "ok", "msg", "all")) + + mock_get.return_value = [darwin_check, linux_check, all_check] + results = scanner.run_all() + + # The check for the other platform should not appear in results + other_result_platform = "linux" if current_plat == "darwin" else "darwin" + result_platforms = {r.platform for r in results} + assert other_result_platform not in result_platforms + assert "all" in result_platforms or current_plat in result_platforms From 26a4e2e65eb7e0627bd3721023e08f5ae34d2885 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:12:13 -0700 Subject: [PATCH 56/92] feat: KV-cache-aware FLOPs/energy with full-count dollar cost MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thread prompt_tokens_evaluated through the telemetry stack so savings calculations use the right token count for each metric: - Dollar cost: full prompt_tokens (what cloud providers charge) - FLOPs/energy: prompt_tokens_evaluated (actual compute with KV cache — subsequent turns only re-evaluate new tokens) Ollama reports both: prompt_eval_count (cache-aware) and we estimate full count from messages. OpenAI-compat engines report full count only (KV caching is transparent in their API). Changes: TelemetryRecord, store schema, aggregator, engines, savings calculation, /v1/savings route. --- src/openjarvis/core/types.py | 1 + src/openjarvis/engine/_openai_compat.py | 6 ++-- src/openjarvis/engine/ollama.py | 22 +++++++----- src/openjarvis/server/routes.py | 3 ++ src/openjarvis/server/savings.py | 35 +++++++++++++------ src/openjarvis/telemetry/aggregator.py | 11 ++++++ .../telemetry/instrumented_engine.py | 2 ++ src/openjarvis/telemetry/store.py | 7 ++-- 8 files changed, 64 insertions(+), 23 deletions(-) diff --git a/src/openjarvis/core/types.py b/src/openjarvis/core/types.py index a8629752..bfde1a41 100644 --- a/src/openjarvis/core/types.py +++ b/src/openjarvis/core/types.py @@ -132,6 +132,7 @@ class TelemetryRecord: timestamp: float model_id: str prompt_tokens: int = 0 + prompt_tokens_evaluated: int = 0 # KV-cache-aware: actual tokens processed completion_tokens: int = 0 total_tokens: int = 0 latency_seconds: float = 0.0 diff --git a/src/openjarvis/engine/_openai_compat.py b/src/openjarvis/engine/_openai_compat.py index bbc3cf00..0eb430f3 100644 --- a/src/openjarvis/engine/_openai_compat.py +++ b/src/openjarvis/engine/_openai_compat.py @@ -95,8 +95,9 @@ class _OpenAICompatibleEngine(InferenceEngine): choice = choices[0] usage = data.get("usage", {}) # Ensure prompt_tokens reflects the full prompt size (including - # system prompt and all conversation history) without assuming - # KV-cache savings. + # system prompt and all conversation history). + # OpenAI-compat APIs (vLLM, SGLang) report full counts — KV + # caching is transparent, so evaluated == full. reported_prompt = usage.get("prompt_tokens", 0) estimated_prompt = estimate_prompt_tokens(messages) prompt_tokens = max(reported_prompt, estimated_prompt) @@ -105,6 +106,7 @@ class _OpenAICompatibleEngine(InferenceEngine): "content": choice["message"].get("content") or "", "usage": { "prompt_tokens": prompt_tokens, + "prompt_tokens_evaluated": reported_prompt or prompt_tokens, "completion_tokens": completion_tokens, "total_tokens": prompt_tokens + completion_tokens, }, diff --git a/src/openjarvis/engine/ollama.py b/src/openjarvis/engine/ollama.py index 1b219368..a757293e 100644 --- a/src/openjarvis/engine/ollama.py +++ b/src/openjarvis/engine/ollama.py @@ -106,19 +106,24 @@ class OllamaEngine(InferenceEngine): f"Ollama returned {exc.response.status_code}: {body}" ) from exc data = resp.json() - # Ollama's prompt_eval_count may exclude KV-cached tokens - # (system prompt, prior turns). Use the larger of the reported - # count and a cache-agnostic estimate so that cost / FLOPs / - # energy calculations reflect the full prompt size. + # prompt_eval_count = tokens actually evaluated (KV-cache-aware). + # estimate_prompt_tokens = full prompt size (for cost comparison). + # We report both so downstream can use the right one: + # prompt_tokens → full size (what cloud would charge) + # prompt_tokens_evaluated → actual compute (with KV cache) reported_prompt = data.get("prompt_eval_count", 0) estimated_prompt = estimate_prompt_tokens(messages) prompt_tokens = max(reported_prompt, estimated_prompt) + prompt_tokens_evaluated = ( + reported_prompt if reported_prompt > 0 else prompt_tokens + ) completion_tokens = data.get("eval_count", 0) content = data.get("message", {}).get("content", "") result: Dict[str, Any] = { "content": content, "usage": { "prompt_tokens": prompt_tokens, + "prompt_tokens_evaluated": prompt_tokens_evaluated, "completion_tokens": completion_tokens, "total_tokens": prompt_tokens + completion_tokens, }, @@ -183,16 +188,17 @@ class OllamaEngine(InferenceEngine): if content: yield content if chunk.get("done", False): - # Capture usage from final chunk. Use the - # cache-agnostic estimate for prompt tokens to - # avoid under-counting when Ollama's KV cache - # suppresses prompt_eval_count. reported_prompt = chunk.get("prompt_eval_count", 0) est_prompt = estimate_prompt_tokens(messages) full_prompt = max(reported_prompt, est_prompt) + evaluated = ( + reported_prompt if reported_prompt > 0 + else full_prompt + ) comp = chunk.get("eval_count", 0) self._last_stream_usage = { "prompt_tokens": full_prompt, + "prompt_tokens_evaluated": evaluated, "completion_tokens": comp, "total_tokens": full_prompt + comp, } diff --git a/src/openjarvis/server/routes.py b/src/openjarvis/server/routes.py index 6c163f65..50fb645b 100644 --- a/src/openjarvis/server/routes.py +++ b/src/openjarvis/server/routes.py @@ -470,6 +470,9 @@ async def savings(request: Request): ), total_calls=sum(m.call_count for m in local_models), session_start=session_start if session_start else 0.0, + prompt_tokens_evaluated=sum( + m.prompt_tokens_evaluated for m in local_models + ), ) return savings_to_dict(result) finally: diff --git a/src/openjarvis/server/savings.py b/src/openjarvis/server/savings.py index 0c6fbc0d..ae807139 100644 --- a/src/openjarvis/server/savings.py +++ b/src/openjarvis/server/savings.py @@ -89,20 +89,28 @@ def compute_savings( completion_tokens: int, total_calls: int = 0, session_start: float = 0.0, + prompt_tokens_evaluated: int = 0, ) -> SavingsSummary: """Compute savings vs cloud providers given token counts. - Token counting assumptions (these match cloud provider billing): - - **System prompt included**: ``prompt_tokens`` includes system - prompt tokens for every API call, matching what cloud providers - charge. - - **No KV-cache discount**: Each call's ``prompt_tokens`` reflects - the full context size (system prompt + conversation history). - Even though local engines may use KV caching internally, the - savings comparison uses the uncached token count because cloud - providers bill for all input tokens on every request. + Two token counts are used: + + - ``prompt_tokens`` — full prompt size (system prompt + all history). + Used for **dollar cost** comparison since cloud providers bill for + every input token on every request. + - ``prompt_tokens_evaluated`` — actual tokens processed (KV-cache- + aware). In multi-turn conversations, subsequent turns only + evaluate new tokens; the system prompt and prior context are + served from KV cache. Used for **FLOPs** and **energy** + calculations since these reflect actual compute. + + If ``prompt_tokens_evaluated`` is 0 (e.g. old telemetry without the + column), it falls back to ``prompt_tokens``. """ + if prompt_tokens_evaluated <= 0: + prompt_tokens_evaluated = prompt_tokens total_tokens = prompt_tokens + completion_tokens + total_tokens_evaluated = prompt_tokens_evaluated + completion_tokens providers: List[ProviderSavings] = [] now = time.time() @@ -116,10 +124,15 @@ def compute_savings( output_cost = (completion_tokens / 1_000_000) * pricing["output_per_1m"] total_cost = input_cost + output_cost - # No-KV-cache FLOPs: P * N * (N+1) + # KV-cache-aware FLOPs: 2 * P * T_evaluated + # Only the actually-evaluated tokens require compute; cached + # tokens from prior turns are served from KV cache. params_b = pricing.get("params_b", 200.0) params = params_b * 1e9 - flops = params * total_tokens * (total_tokens + 1) if total_tokens > 0 else 0.0 + flops = ( + 2.0 * params * total_tokens_evaluated + if total_tokens_evaluated > 0 else 0.0 + ) # Derive Wh-per-FLOP from the provider's per-token constants: # energy_wh_per_1k_tokens / (1000 * flops_per_token) = Wh per FLOP wh_per_flop = pricing["energy_wh_per_1k_tokens"] / ( diff --git a/src/openjarvis/telemetry/aggregator.py b/src/openjarvis/telemetry/aggregator.py index c055a51f..31754888 100644 --- a/src/openjarvis/telemetry/aggregator.py +++ b/src/openjarvis/telemetry/aggregator.py @@ -19,6 +19,7 @@ class ModelStats: call_count: int = 0 total_tokens: int = 0 prompt_tokens: int = 0 + prompt_tokens_evaluated: int = 0 completion_tokens: int = 0 total_latency: float = 0.0 avg_latency: float = 0.0 @@ -128,11 +129,17 @@ class TelemetryAggregator: # Build optional columns for new fields (graceful on old DBs) extra_cols = "" + has_pte = self._safe_col("prompt_tokens_evaluated") has_tpj = self._safe_col("tokens_per_joule") has_derived = self._safe_col("energy_per_output_token_joules") has_phase = self._safe_col("prefill_energy_joules") has_itl = self._safe_col("mean_itl_ms") + if has_pte: + extra_cols += ( + ", SUM(prompt_tokens_evaluated)" + " AS prompt_tokens_evaluated" + ) if has_tpj: extra_cols += ( ", AVG(tokens_per_joule) AS avg_tokens_per_joule" @@ -189,6 +196,10 @@ class TelemetryAggregator: avg_gpu_utilization_pct=r["avg_gpu_utilization_pct"] or 0.0, avg_throughput_tok_per_sec=r["avg_throughput_tok_per_sec"] or 0.0, ) + if has_pte: + ms.prompt_tokens_evaluated = ( + r["prompt_tokens_evaluated"] or 0 + ) if has_tpj: ms.avg_tokens_per_joule = r["avg_tokens_per_joule"] or 0.0 if has_derived: diff --git a/src/openjarvis/telemetry/instrumented_engine.py b/src/openjarvis/telemetry/instrumented_engine.py index e8934681..1ebe5d06 100644 --- a/src/openjarvis/telemetry/instrumented_engine.py +++ b/src/openjarvis/telemetry/instrumented_engine.py @@ -109,6 +109,7 @@ class InstrumentedEngine(InferenceEngine): usage = result.get("usage", {}) completion_tokens = usage.get("completion_tokens", 0) + prompt_tokens_evaluated = usage.get("prompt_tokens_evaluated", 0) ttft = result.get("ttft", 0.0) throughput = completion_tokens / latency if latency > 0 else 0.0 @@ -186,6 +187,7 @@ class InstrumentedEngine(InferenceEngine): timestamp=t0, model_id=model, prompt_tokens=prompt_tok, + prompt_tokens_evaluated=prompt_tokens_evaluated or prompt_tok, completion_tokens=completion_tokens, total_tokens=prompt_tok + completion_tokens, latency_seconds=latency, diff --git a/src/openjarvis/telemetry/store.py b/src/openjarvis/telemetry/store.py index 8b4702c7..eea89b84 100644 --- a/src/openjarvis/telemetry/store.py +++ b/src/openjarvis/telemetry/store.py @@ -20,6 +20,7 @@ CREATE TABLE IF NOT EXISTS telemetry ( engine TEXT NOT NULL DEFAULT '', agent TEXT NOT NULL DEFAULT '', prompt_tokens INTEGER NOT NULL DEFAULT 0, + prompt_tokens_evaluated INTEGER NOT NULL DEFAULT 0, completion_tokens INTEGER NOT NULL DEFAULT 0, total_tokens INTEGER NOT NULL DEFAULT 0, latency_seconds REAL NOT NULL DEFAULT 0.0, @@ -59,7 +60,7 @@ CREATE TABLE IF NOT EXISTS telemetry ( _INSERT = """\ INSERT INTO telemetry ( timestamp, model_id, engine, agent, - prompt_tokens, completion_tokens, total_tokens, + prompt_tokens, prompt_tokens_evaluated, completion_tokens, total_tokens, latency_seconds, ttft, cost_usd, energy_joules, power_watts, gpu_utilization_pct, gpu_memory_used_gb, gpu_temperature_c, throughput_tok_per_sec, prefill_latency_seconds, decode_latency_seconds, @@ -75,7 +76,7 @@ INSERT INTO telemetry ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, - ?, ?, ?, ?, ?, ?, ?, ? + ?, ?, ?, ?, ?, ?, ?, ?, ? ) """ @@ -105,6 +106,7 @@ _MIGRATE_COLUMNS = [ ("p99_itl_ms", "REAL NOT NULL DEFAULT 0.0"), ("std_itl_ms", "REAL NOT NULL DEFAULT 0.0"), ("is_streaming", "INTEGER NOT NULL DEFAULT 0"), + ("prompt_tokens_evaluated", "INTEGER NOT NULL DEFAULT 0"), ] @@ -139,6 +141,7 @@ class TelemetryStore: rec.engine, rec.agent, rec.prompt_tokens, + rec.prompt_tokens_evaluated, rec.completion_tokens, rec.total_tokens, rec.latency_seconds, From a05c2b7df641f77b03d299ad8bd8b64cb24b1f95 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:12:55 -0700 Subject: [PATCH 57/92] feat: extract ollama_pull helper and add multi-engine model pull Refactors model pull into reusable ollama_pull() function. Adds --engine flag to support llamacpp (GGUF) and mlx (HuggingFace) downloads via huggingface-cli, with FileNotFoundError handling. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/model.py | 102 ++++++++++++++++++++++++++++----- tests/cli/test_model_pull.py | 108 +++++++++++++++++++++++++++++++++++ 2 files changed, 195 insertions(+), 15 deletions(-) create mode 100644 tests/cli/test_model_pull.py diff --git a/src/openjarvis/cli/model.py b/src/openjarvis/cli/model.py index 4a6fc9d1..d3243311 100644 --- a/src/openjarvis/cli/model.py +++ b/src/openjarvis/cli/model.py @@ -3,6 +3,7 @@ from __future__ import annotations import os +import subprocess import sys import click @@ -15,6 +16,7 @@ from openjarvis.core.config import load_config from openjarvis.core.registry import ModelRegistry from openjarvis.engine import discover_engines, discover_models from openjarvis.intelligence import merge_discovered_models, register_builtin_models +from openjarvis.intelligence.model_catalog import BUILTIN_MODELS @click.group() @@ -149,18 +151,8 @@ def info(model_name: str) -> None: console.print(Panel("\n".join(lines), title=spec.name, border_style="blue")) -@model.command() -@click.argument("model_name") -def pull(model_name: str) -> None: - """Download a model (Ollama only).""" - console = Console() - config = load_config() - host = ( - config.engine.ollama_host - or os.environ.get("OLLAMA_HOST") - or "http://localhost:11434" - ).rstrip("/") - +def ollama_pull(host: str, model_name: str, console: Console) -> bool: + """Pull a model via Ollama API. Returns True on success.""" console.print(f"Pulling [cyan]{model_name}[/cyan] via Ollama...") try: with httpx.stream( @@ -171,7 +163,6 @@ def pull(model_name: str) -> None: ) as resp: resp.raise_for_status() import json - for line in resp.iter_lines(): if not line.strip(): continue @@ -188,9 +179,90 @@ def pull(model_name: str) -> None: elif status: console.print(f" {status}") console.print(f"\n[green]Successfully pulled {model_name}[/green]") + return True except httpx.ConnectError: console.print("[red]Cannot connect to Ollama.[/red] Is it running?") - sys.exit(1) + return False except httpx.HTTPStatusError as exc: console.print(f"[red]Ollama error:[/red] {exc.response.status_code}") - sys.exit(1) + return False + + +def find_model_spec(model_name: str): + """Look up a model in the builtin catalog. Returns None if not found.""" + for spec in BUILTIN_MODELS: + if spec.model_id == model_name: + return spec + return None + + +def hf_download(repo: str, filename: str | None, console: Console) -> bool: + """Download from HuggingFace via huggingface-cli. Returns True on success.""" + cmd = ["huggingface-cli", "download", repo] + if filename: + cmd.append(filename) + try: + subprocess.run(cmd, check=True) + console.print("[green]Download complete.[/green]") + return True + except FileNotFoundError: + console.print( + "[red]huggingface-cli not found.[/red]\n" + "Install it: [cyan]pip install huggingface_hub[/cyan]\n" + f"Or download manually: https://huggingface.co/{repo}" + ) + return False + except subprocess.CalledProcessError: + console.print("[red]Download failed.[/red]") + return False + + +@model.command() +@click.argument("model_name") +@click.option("--engine", default=None, help="Engine to download for (ollama, llamacpp, mlx).") +def pull(model_name: str, engine: str | None) -> None: + """Download a model.""" + console = Console() + config = load_config() + engine = engine or config.engine.default or "ollama" + + if engine == "ollama": + host = ( + config.engine.ollama_host + or os.environ.get("OLLAMA_HOST") + or "http://localhost:11434" + ).rstrip("/") + if not ollama_pull(host, model_name, console): + sys.exit(1) + elif engine in ("llamacpp", "mlx"): + spec = find_model_spec(model_name) + if not spec: + console.print(f"[red]Model not in catalog:[/red] {model_name}") + sys.exit(1) + if engine == "llamacpp": + repo = spec.metadata.get("hf_repo", "") + gguf = spec.metadata.get("gguf_file", "") + if not repo or not gguf: + console.print(f"[red]No GGUF download info for {model_name}[/red]") + sys.exit(1) + console.print(f"Downloading [cyan]{gguf}[/cyan] from {repo}...") + if not hf_download(repo, gguf, console): + sys.exit(1) + else: # mlx + mlx_repo = spec.metadata.get("mlx_repo", "") + if not mlx_repo: + console.print(f"[red]No MLX repo info for {model_name}[/red]") + sys.exit(1) + console.print(f"Downloading [cyan]{mlx_repo}[/cyan]...") + if not hf_download(mlx_repo, None, console): + sys.exit(1) + elif engine in ("vllm", "sglang"): + console.print( + f"[cyan]{model_name}[/cyan] will download automatically when " + f"{engine} starts serving it." + ) + else: + console.print( + f"Manual download required for engine [cyan]{engine}[/cyan].\n" + f"Check the engine documentation for instructions." + ) diff --git a/tests/cli/test_model_pull.py b/tests/cli/test_model_pull.py new file mode 100644 index 00000000..a87a55ae --- /dev/null +++ b/tests/cli/test_model_pull.py @@ -0,0 +1,108 @@ +"""Tests for ``jarvis model pull`` multi-engine support.""" + +from __future__ import annotations + +from unittest import mock + +import pytest +from click.testing import CliRunner +from rich.console import Console + +from openjarvis.cli.model import ollama_pull + + +class TestOllamaPull: + """Test the extracted ollama_pull helper.""" + + def test_ollama_pull_success(self) -> None: + import io + console = Console(file=io.StringIO()) + mock_lines = [ + '{"status": "pulling manifest"}', + '{"status": "downloading", "total": 100, "completed": 100}', + '{"status": "success"}', + ] + mock_resp = mock.MagicMock() + mock_resp.raise_for_status = mock.MagicMock() + mock_resp.iter_lines.return_value = iter(mock_lines) + mock_resp.__enter__ = mock.MagicMock(return_value=mock_resp) + mock_resp.__exit__ = mock.MagicMock(return_value=False) + + with mock.patch("httpx.stream", return_value=mock_resp): + result = ollama_pull("http://localhost:11434", "qwen3.5:3b", console) + assert result is True + + def test_ollama_pull_connect_error(self) -> None: + import io + import httpx + + console = Console(file=io.StringIO()) + with mock.patch("httpx.stream", side_effect=httpx.ConnectError("refused")): + result = ollama_pull("http://localhost:11434", "qwen3.5:3b", console) + assert result is False + + +class TestPullCliMultiEngine: + """Test the pull CLI command dispatches to correct engine.""" + + def test_pull_llamacpp_uses_huggingface_cli(self) -> None: + from openjarvis.cli import cli + + runner = CliRunner() + with ( + mock.patch("openjarvis.cli.model.load_config") as mock_cfg, + mock.patch("subprocess.run") as mock_run, + ): + mock_cfg.return_value.engine.default = "llamacpp" + mock_cfg.return_value.engine.ollama_host = None + mock_run.return_value = mock.MagicMock(returncode=0) + + result = runner.invoke( + cli, ["model", "pull", "qwen3.5:8b", "--engine", "llamacpp"] + ) + + assert result.exit_code == 0 + mock_run.assert_called_once() + call_args = mock_run.call_args[0][0] + assert "huggingface-cli" in call_args + assert "qwen3.5-8b-q4_k_m.gguf" in call_args + + def test_pull_mlx_uses_huggingface_cli(self) -> None: + from openjarvis.cli import cli + + runner = CliRunner() + with ( + mock.patch("openjarvis.cli.model.load_config") as mock_cfg, + mock.patch("subprocess.run") as mock_run, + ): + mock_cfg.return_value.engine.default = "mlx" + mock_cfg.return_value.engine.ollama_host = None + mock_run.return_value = mock.MagicMock(returncode=0) + + result = runner.invoke( + cli, ["model", "pull", "qwen3.5:8b", "--engine", "mlx"] + ) + + assert result.exit_code == 0 + mock_run.assert_called_once() + call_args = mock_run.call_args[0][0] + assert "huggingface-cli" in call_args + assert "mlx-community/Qwen3.5-8B-4bit" in call_args + + def test_pull_llamacpp_huggingface_cli_not_found(self) -> None: + from openjarvis.cli import cli + + runner = CliRunner() + with ( + mock.patch("openjarvis.cli.model.load_config") as mock_cfg, + mock.patch("subprocess.run", side_effect=FileNotFoundError), + ): + mock_cfg.return_value.engine.default = "llamacpp" + mock_cfg.return_value.engine.ollama_host = None + + result = runner.invoke( + cli, ["model", "pull", "qwen3.5:8b", "--engine", "llamacpp"] + ) + + assert result.exit_code != 0 + assert "huggingface_hub" in result.output or "pip install" in result.output From 00f71ae504dea78c65b6cd92f7cc3e475208b949 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:16:11 -0700 Subject: [PATCH 58/92] fix: update energy scaling test for linear KV-cache FLOPs model The FLOPs formula changed from quadratic (P*N*(N+1)) to linear (2*P*T_evaluated) with KV-cache awareness. Update the test expectation from ~100x to ~10x for 10x token increase. --- tests/evals/test_use_case_benchmarks.py | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/tests/evals/test_use_case_benchmarks.py b/tests/evals/test_use_case_benchmarks.py index 5bddb4d1..8e06d727 100644 --- a/tests/evals/test_use_case_benchmarks.py +++ b/tests/evals/test_use_case_benchmarks.py @@ -424,17 +424,16 @@ class TestSavings: assert "total_calls" in d def test_energy_scales_linearly(self) -> None: - """Energy should scale with FLOPs (quadratic in N), not N^3.""" + """Energy should scale linearly with evaluated tokens (KV-cache model).""" from openjarvis.server.savings import compute_savings s1 = compute_savings(1000, 0) s10 = compute_savings(10000, 0) - # FLOPs ~ N^2, so 10x tokens => ~100x FLOPs => ~100x energy + # FLOPs = 2*P*T (linear), so 10x tokens => 10x FLOPs => 10x energy for p1, p10 in zip(s1.per_provider, s10.per_provider): ratio = p10.energy_wh / p1.energy_wh - # Allow some tolerance for the (N+1) factor - assert 90 < ratio < 110, ( - f"{p1.provider}: energy ratio {ratio:.1f}, expected ~100" + assert 9 < ratio < 11, ( + f"{p1.provider}: energy ratio {ratio:.1f}, expected ~10" ) def test_energy_wh_matches_direct_formula(self) -> None: From e2376d29d12900678dfbe0fc14e5127fbcbf0020 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:16:59 -0700 Subject: [PATCH 59/92] feat: add interactive model download and empty-model fallback to init Prompts user to download recommended model during jarvis init. Adds --no-download flag for CI. Shows helpful message when no model fits available memory. Adds exo/nexa next-steps text. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/init_cmd.py | 74 +++++++++++++++++++++++++++- tests/cli/test_init_guidance.py | 85 +++++++++++++++++++++++++++++++-- 2 files changed, 153 insertions(+), 6 deletions(-) diff --git a/src/openjarvis/cli/init_cmd.py b/src/openjarvis/cli/init_cmd.py index 41fb1864..95c946f3 100644 --- a/src/openjarvis/cli/init_cmd.py +++ b/src/openjarvis/cli/init_cmd.py @@ -10,10 +10,12 @@ from rich.console import Console from rich.markup import escape from rich.panel import Panel +from openjarvis.cli.model import find_model_spec, hf_download, ollama_pull from openjarvis.core.config import ( DEFAULT_CONFIG_DIR, DEFAULT_CONFIG_PATH, detect_hardware, + estimated_download_gb, generate_default_toml, generate_minimal_toml, recommend_engine, @@ -132,10 +134,58 @@ def _next_steps_text(engine: str, model: str = "") -> str: "\n" " Run `jarvis doctor` to verify your setup." ), + "exo": ( + "Next steps:\n\n" + " 1. Install and start Exo:\n" + " pip install exo\n" + " exo\n\n" + " 2. Try it out:\n" + " jarvis ask \"Hello\"\n\n" + " Run `jarvis doctor` to verify your setup." + ), + "nexa": ( + "Next steps:\n\n" + " 1. Install and start Nexa:\n" + " pip install nexaai\n" + " nexa server\n\n" + " 2. Try it out:\n" + " jarvis ask \"Hello\"\n\n" + " Run `jarvis doctor` to verify your setup." + ), } return steps.get(engine, steps["ollama"]) +def _do_download(engine: str, model: str, spec, console: Console) -> None: + """Dispatch model download based on engine type.""" + import os + if engine == "ollama": + host = os.environ.get("OLLAMA_HOST", "http://localhost:11434").rstrip("/") + ollama_pull(host, model, console) + elif engine == "llamacpp": + repo = spec.metadata.get("hf_repo", "") + gguf = spec.metadata.get("gguf_file", "") + if repo and gguf: + console.print(f" Downloading [cyan]{gguf}[/cyan] from {repo}...") + hf_download(repo, gguf, console) + else: + console.print(f" [yellow]No GGUF download info for {model}[/yellow]") + elif engine == "mlx": + mlx_repo = spec.metadata.get("mlx_repo", "") + if mlx_repo: + console.print(f" Downloading [cyan]{mlx_repo}[/cyan]...") + hf_download(mlx_repo, None, console) + else: + console.print(f" [yellow]No MLX repo info for {model}[/yellow]") + elif engine in ("vllm", "sglang"): + console.print( + f" [cyan]{model}[/cyan] will download automatically when " + f"{engine} starts serving it." + ) + else: + console.print(f" Download {model} through the {engine} interface.") + + @click.command() @click.option( "--force", is_flag=True, help="Overwrite existing config without prompting." @@ -157,11 +207,15 @@ def _next_steps_text(engine: str, model: str = "") -> str: default=None, help="Inference engine to use (skips interactive selection).", ) +@click.option( + "--no-download", is_flag=True, default=False, help="Skip the model download prompt." +) def init( force: bool, config: Optional[Path], full_config: bool = False, engine: Optional[str] = None, + no_download: bool = False, ) -> None: """Detect hardware and generate ~/.openjarvis/config.toml.""" console = Console() @@ -286,8 +340,24 @@ def init( selected_engine = engine or recommend_engine(hw) model = recommend_model(hw, selected_engine) - if model: - console.print(f"\n [bold]Recommended model:[/bold] {model}") + + if not model: + console.print( + "\n [yellow]! Not enough memory to run any local model.[/yellow]\n" + " Consider a cloud engine or a machine with more RAM." + ) + else: + spec = find_model_spec(model) + size_gb = estimated_download_gb(spec.parameter_count_b) if spec else 0 + console.print( + f"\n [bold]Recommended model:[/bold] {model} (~{size_gb:.1f} GB estimated)" + ) + + if not no_download and spec: + prompt = f" Download {model} (~{size_gb:.1f} GB estimated) now?" + if click.confirm(prompt, default=True): + _do_download(selected_engine, model, spec, console) + console.print() console.print( Panel( diff --git a/tests/cli/test_init_guidance.py b/tests/cli/test_init_guidance.py index 263d786d..6e2cd67f 100644 --- a/tests/cli/test_init_guidance.py +++ b/tests/cli/test_init_guidance.py @@ -20,7 +20,7 @@ class TestInitShowsNextSteps: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): - result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) + result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp", "--no-download"]) assert result.exit_code == 0 assert "Getting Started" in result.output assert "jarvis ask" in result.output @@ -34,7 +34,7 @@ class TestInitShowsNextSteps: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): - result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) + result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp", "--no-download"]) assert result.exit_code == 0 assert "[engine]" in result.output assert "[intelligence]" in result.output @@ -93,7 +93,7 @@ class TestMinimalConfig: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): - result = CliRunner().invoke(cli, ["init", "--engine", "ollama"]) + result = CliRunner().invoke(cli, ["init", "--engine", "ollama", "--no-download"]) assert result.exit_code == 0 content = config_path.read_text() # Minimal config should be short @@ -110,10 +110,87 @@ class TestMinimalConfig: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), ): - result = CliRunner().invoke(cli, ["init", "--full", "--engine", "ollama"]) + result = CliRunner().invoke(cli, ["init", "--full", "--engine", "ollama", "--no-download"]) assert result.exit_code == 0 content = config_path.read_text() # Full config should have many sections assert "[engine.ollama]" in content assert "[server]" in content assert "[security]" in content + + +class TestInitDownloadPrompt: + def test_init_shows_download_prompt(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + ): + result = CliRunner().invoke(cli, ["init", "--engine", "ollama"], input="n\n") + assert result.exit_code == 0 + assert "Download" in result.output and "now?" in result.output + + def test_init_no_download_flag_skips_prompt(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + ): + result = CliRunner().invoke(cli, ["init", "--engine", "ollama", "--no-download"]) + assert result.exit_code == 0 + assert "Download" not in result.output or "now?" not in result.output + + +class TestInitEmptyModelFallback: + def test_init_no_model_shows_warning(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.recommend_model", return_value=""), + ): + result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) + assert result.exit_code == 0 + assert "Not enough memory" in result.output or "not enough memory" in result.output + + +class TestNextStepsExoNexa: + def test_next_steps_exo(self) -> None: + text = _next_steps_text("exo") + assert "exo" in text.lower() + assert "jarvis ask" in text + assert "ollama" not in text.lower() + + def test_next_steps_nexa(self) -> None: + text = _next_steps_text("nexa") + assert "nexa" in text.lower() + assert "jarvis ask" in text + assert "ollama" not in text.lower() + + +class TestInitDownloadDispatch: + def test_init_ollama_download_calls_ollama_pull(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.ollama_pull", return_value=True) as mock_pull, + ): + result = CliRunner().invoke(cli, ["init", "--engine", "ollama"], input="y\n") + assert result.exit_code == 0 + mock_pull.assert_called_once() + + def test_init_vllm_shows_auto_download_message(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + ): + result = CliRunner().invoke(cli, ["init", "--engine", "vllm"], input="y\n") + assert result.exit_code == 0 + assert "automatically" in result.output From 02bc36c1f83bd85c2f1c0434b7c359e4db9160e9 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:22:13 -0700 Subject: [PATCH 60/92] feat: register jarvis scan command and add init privacy hook Registers the new scan command in the CLI. Adds a lightweight privacy check at the end of jarvis init that runs disk encryption and cloud sync checks, with pointer to jarvis scan for full audit. Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/cli/__init__.py | 2 ++ src/openjarvis/cli/init_cmd.py | 19 +++++++++++++++++++ tests/cli/test_init_guidance.py | 30 ++++++++++++++++++++++++++++++ 3 files changed, 51 insertions(+) diff --git a/src/openjarvis/cli/__init__.py b/src/openjarvis/cli/__init__.py index 10a70a94..6ff4ab66 100644 --- a/src/openjarvis/cli/__init__.py +++ b/src/openjarvis/cli/__init__.py @@ -26,6 +26,7 @@ from openjarvis.cli.operators_cmd import operators from openjarvis.cli.optimize_cmd import optimize_group from openjarvis.cli.quickstart_cmd import quickstart from openjarvis.cli.registry_cmd import registry +from openjarvis.cli.scan_cmd import scan from openjarvis.cli.scheduler_cmd import scheduler from openjarvis.cli.serve import serve from openjarvis.cli.skill_cmd import skill @@ -87,6 +88,7 @@ cli.add_command(gateway, "gateway") cli.add_command(tool, "tool") cli.add_command(registry, "registry") cli.add_command(config, "config") +cli.add_command(scan, "scan") def main() -> None: diff --git a/src/openjarvis/cli/init_cmd.py b/src/openjarvis/cli/init_cmd.py index 95c946f3..8d729b7e 100644 --- a/src/openjarvis/cli/init_cmd.py +++ b/src/openjarvis/cli/init_cmd.py @@ -11,6 +11,7 @@ from rich.markup import escape from rich.panel import Panel from openjarvis.cli.model import find_model_spec, hf_download, ollama_pull +from openjarvis.cli.scan_cmd import PrivacyScanner from openjarvis.core.config import ( DEFAULT_CONFIG_DIR, DEFAULT_CONFIG_PATH, @@ -156,6 +157,23 @@ def _next_steps_text(engine: str, model: str = "") -> str: return steps.get(engine, steps["ollama"]) +def _quick_privacy_check(console: Console) -> None: + """Run critical privacy checks and print compact summary.""" + scanner = PrivacyScanner() + results = scanner.run_quick() + if results: + console.print(" [bold]Privacy check:[/bold]") + for r in results: + if r.status == "ok": + console.print(f" [green]\u2713[/green] {r.message}") + elif r.status == "warn": + console.print(f" [yellow]![/yellow] {r.message}") + elif r.status == "fail": + console.print(f" [red]\u2717[/red] {r.message}") + console.print() + console.print(" Run [cyan]jarvis scan[/cyan] for a full environment audit.") + + def _do_download(engine: str, model: str, spec, console: Console) -> None: """Dispatch model download based on engine type.""" import os @@ -358,6 +376,7 @@ def init( if click.confirm(prompt, default=True): _do_download(selected_engine, model, spec, console) + _quick_privacy_check(console) console.print() console.print( Panel( diff --git a/tests/cli/test_init_guidance.py b/tests/cli/test_init_guidance.py index 6e2cd67f..74924c53 100644 --- a/tests/cli/test_init_guidance.py +++ b/tests/cli/test_init_guidance.py @@ -19,6 +19,7 @@ class TestInitShowsNextSteps: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp", "--no-download"]) assert result.exit_code == 0 @@ -33,6 +34,7 @@ class TestInitShowsNextSteps: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp", "--no-download"]) assert result.exit_code == 0 @@ -92,6 +94,7 @@ class TestMinimalConfig: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "ollama", "--no-download"]) assert result.exit_code == 0 @@ -109,6 +112,7 @@ class TestMinimalConfig: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--full", "--engine", "ollama", "--no-download"]) assert result.exit_code == 0 @@ -126,6 +130,7 @@ class TestInitDownloadPrompt: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "ollama"], input="n\n") assert result.exit_code == 0 @@ -137,6 +142,7 @@ class TestInitDownloadPrompt: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "ollama", "--no-download"]) assert result.exit_code == 0 @@ -151,6 +157,7 @@ class TestInitEmptyModelFallback: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.recommend_model", return_value=""), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) assert result.exit_code == 0 @@ -179,6 +186,7 @@ class TestInitDownloadDispatch: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.ollama_pull", return_value=True) as mock_pull, + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "ollama"], input="y\n") assert result.exit_code == 0 @@ -190,7 +198,29 @@ class TestInitDownloadDispatch: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): result = CliRunner().invoke(cli, ["init", "--engine", "vllm"], input="y\n") assert result.exit_code == 0 assert "automatically" in result.output + + +class TestInitPrivacyHook: + def test_init_shows_privacy_summary(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner") as MockScanner, + ): + from openjarvis.cli.scan_cmd import ScanResult + instance = MockScanner.return_value + instance.run_quick.return_value = [ + ScanResult("FileVault", "ok", "FileVault enabled", "darwin"), + ] + result = CliRunner().invoke( + cli, ["init", "--engine", "llamacpp", "--no-download"] + ) + assert result.exit_code == 0 + assert "jarvis scan" in result.output From d9faa4621ebf4d27e170ff3e05b06bbd83048aa5 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:24:07 -0700 Subject: [PATCH 61/92] style: fix E501 line length in model pull --engine option Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/model.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/openjarvis/cli/model.py b/src/openjarvis/cli/model.py index d3243311..453bcca2 100644 --- a/src/openjarvis/cli/model.py +++ b/src/openjarvis/cli/model.py @@ -219,7 +219,9 @@ def hf_download(repo: str, filename: str | None, console: Console) -> bool: @model.command() @click.argument("model_name") -@click.option("--engine", default=None, help="Engine to download for (ollama, llamacpp, mlx).") +@click.option( + "--engine", default=None, help="Engine to download for." +) def pull(model_name: str, engine: str | None) -> None: """Download a model.""" console = Console() From a934977ba8202440a64dca20b4de4f320410c8b4 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 18:26:05 -0700 Subject: [PATCH 62/92] fix: address spec review findings for privacy scanner - Add slots=True to ScanResult dataclass - Fix extra space in IPv6 port f-string - Add missing tests: check_remote_access, check_icloud_sync, run_quick Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/cli/scan_cmd.py | 4 +- tests/cli/test_scan.py | 67 ++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+), 2 deletions(-) diff --git a/src/openjarvis/cli/scan_cmd.py b/src/openjarvis/cli/scan_cmd.py index e8d83e08..6ff2e210 100644 --- a/src/openjarvis/cli/scan_cmd.py +++ b/src/openjarvis/cli/scan_cmd.py @@ -31,7 +31,7 @@ _REMOTE_ACCESS_PROCS = ["xrdp", "x11vnc", "vncserver", "AnyDesk"] # --------------------------------------------------------------------------- -@dataclass +@dataclass(slots=True) class ScanResult: """Result of a single privacy/security check.""" @@ -258,7 +258,7 @@ class PrivacyScanner: exposed: list[int] = [] for port in _ENGINE_PORTS: # Look for patterns like *:PORT or 0.0.0.0:PORT - for token in (f"*:{port}", f"0.0.0.0:{port}", f":::{ port}"): + for token in (f"*:{port}", f"0.0.0.0:{port}", f":::{port}"): if token.replace(" ", "") in output.replace(" ", ""): exposed.append(port) break diff --git a/tests/cli/test_scan.py b/tests/cli/test_scan.py index 9b8eea7e..c89760f3 100644 --- a/tests/cli/test_scan.py +++ b/tests/cli/test_scan.py @@ -254,3 +254,70 @@ class TestPlatformFiltering: result_platforms = {r.platform for r in results} assert other_result_platform not in result_platforms assert "all" in result_platforms or current_plat in result_platforms + + +class TestRemoteAccess: + """Tests for check_remote_access (Linux).""" + + def test_no_remote_access(self) -> None: + scanner = PrivacyScanner() + with patch.object(scanner, "_run") as mock_run: + mock_run.return_value = CompletedProcess([], 1, stdout="", stderr="") + result = scanner.check_remote_access() + assert result.status == "ok" + + def test_xrdp_running(self) -> None: + scanner = PrivacyScanner() + with patch.object(scanner, "_run") as mock_run: + def side_effect(cmd, **kw): + if any("xrdp" in str(c) for c in cmd): + return CompletedProcess(cmd, 0, stdout="12345", stderr="") + return CompletedProcess(cmd, 1, stdout="", stderr="") + mock_run.side_effect = side_effect + result = scanner.check_remote_access() + assert result.status == "warn" + + +class TestICloudSync: + """Tests for check_icloud_sync (macOS).""" + + def test_no_icloud_sync(self) -> None: + scanner = PrivacyScanner() + with patch.object(scanner, "_run") as mock_run: + mock_run.return_value = CompletedProcess( + [], 0, stdout="no relevant output", stderr="" + ) + result = scanner.check_icloud_sync() + assert result.status in ("ok", "skip") + + def test_icloud_defaults_error_falls_through_to_ok(self) -> None: + scanner = PrivacyScanner() + # When _run raises, the nested try/except catches it and falls to ok + with patch.object(scanner, "_run", side_effect=FileNotFoundError): + result = scanner.check_icloud_sync() + assert result.status == "ok" + + +class TestRunQuick: + """Tests for run_quick subset.""" + + def test_run_quick_returns_subset(self) -> None: + scanner = PrivacyScanner() + with patch.object(scanner, "check_filevault") as fv, \ + patch.object(scanner, "check_luks") as luks, \ + patch.object(scanner, "check_icloud_sync") as ic, \ + patch.object(scanner, "check_cloud_sync_agents") as cs: + import sys + plat = sys.platform + fv.return_value = ScanResult("FV", "ok", "ok", "darwin") + luks.return_value = ScanResult("LUKS", "ok", "ok", "linux") + ic.return_value = ScanResult("iCloud", "ok", "ok", "darwin") + cs.return_value = ScanResult("Cloud", "ok", "ok", plat) + results = scanner.run_quick() + # Should only contain current platform results + for r in results: + assert r.platform in (plat, "all") + # Should not contain network or screen recording + names = {r.name for r in results} + assert "Network Exposure" not in names + assert "Screen Recording" not in names From bc2a7e55958677b8625a709221ee02186c8949dc Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 20:31:09 -0700 Subject: [PATCH 63/92] chore: remove spec/plan files from tracked tree These are Claude-generated working documents that should not be checked into the repository. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...t-onboarding-and-privacy-scanner-design.md | 249 --- ...nit-onboarding-and-privacy-scanner-plan.md | 1413 ----------------- 2 files changed, 1662 deletions(-) delete mode 100644 docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md delete mode 100644 docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md diff --git a/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md b/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md deleted file mode 100644 index 3f93ed5f..00000000 --- a/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md +++ /dev/null @@ -1,249 +0,0 @@ -# Design: Init Model Onboarding & Privacy Environment Scanner - -**Date:** 2026-03-24 -**Status:** Draft -**Author:** Jon Saad-Falcon + Claude - -## Problem Statement - -Two issues reported by a user (Ali Shahkar) after setting up OpenJarvis on macOS with llama.cpp: - -1. **`jarvis init` sets a default model with no way to download it.** The config is written with a recommended model, but there's no download prompt, no validation that it's available, and `jarvis doctor` immediately warns it can't be found. On Apple Silicon with MLX, the bug is worse: `recommend_model()` returns an empty string because no Qwen3.5 model lists MLX as a supported engine. - -2. **No environment privacy auditing.** OpenJarvis positions itself as privacy-first local AI, but `jarvis init` doesn't verify the local environment is actually private. Cloud sync agents, MDM profiles, disk encryption status, and network exposure can all undermine the privacy guarantee without the user knowing. - -## Solution Overview - -### Issue 1: Interactive Model Download in `jarvis init` - -**Bug fix:** Add `"mlx"` to `supported_engines` for Qwen3.5 models (3b, 4b, 8b, 14b) in the model catalog. - -**Feature:** After writing the config and displaying the recommended model, `jarvis init` prompts: - -``` - Recommended model: qwen3.5:8b (~4.4 GB) - Download now? [y/n]: -``` - -Download logic is engine-specific: -- **Ollama:** Reuse existing `model pull` code (HTTP stream to `POST /api/pull`). -- **llama.cpp:** Shell out to `huggingface-cli download` with GGUF filename from catalog metadata. -- **MLX:** Shell out to `huggingface-cli download` for pre-quantized MLX repo from catalog metadata. -- **vLLM/SGLang:** Inform the user the model downloads automatically on first serve. -- **LM Studio/Exo/Nexa:** Show manual download instructions (these have their own UIs). - -**Empty model fallback:** If `recommend_model()` returns `""`, display: - -``` - ! Not enough memory to run any local model. - Consider a cloud engine or a machine with more RAM. -``` - -### Issue 2: Privacy Environment Scanner - -**New CLI command:** `jarvis scan` performs a full environment privacy audit. - -**Init hook:** At the end of `jarvis init`, run a lightweight subset (disk encryption + cloud sync overlap with `~/.openjarvis/`) and show a compact summary pointing to `jarvis scan` for the full audit. - -## Detailed Design - -### Model Catalog Changes - -Add `"mlx"` to `supported_engines` and download metadata for Qwen3.5 models in `src/openjarvis/intelligence/model_catalog.py`. - -**Important:** The catalog has TWO Qwen3.5 sections. The first (lines ~42-121) contains the original entries (`qwen3.5:3b`, `qwen3.5:8b`, `qwen3.5:14b`, `qwen3.5:35b`, etc. with `context_length=131072`). The second (lines ~219-274) contains newer entries (`qwen3.5:4b`, `qwen3.5:35b-a3b`, etc. with `context_length=262144`). Both sections' models are candidates for `recommend_model()`. MLX must be added to entries in BOTH sections. - -Models getting MLX support added to `supported_engines`: - -| model_id | Section | Current context_length | Add MLX | -|----------|---------|----------------------|---------| -| `qwen3.5:3b` | First | 131072 | Yes | -| `qwen3.5:4b` | Second | 262144 | Yes | -| `qwen3.5:8b` | First | 131072 | Yes | -| `qwen3.5:14b` | First | 131072 | Yes | - -Larger models (35b+) are excluded because MLX quantized weights aren't widely available and would exceed typical Apple Silicon memory. - -Download metadata to add to each model's `metadata` dict: - -| model_id | `gguf_file` | `mlx_repo` | -|----------|-------------|------------| -| `qwen3.5:3b` | `qwen3.5-3b-q4_k_m.gguf` | `mlx-community/Qwen3.5-3B-4bit` | -| `qwen3.5:4b` | `qwen3.5-4b-q4_k_m.gguf` | `mlx-community/Qwen3.5-4B-4bit` | -| `qwen3.5:8b` | `qwen3.5-8b-q4_k_m.gguf` | `mlx-community/Qwen3.5-8B-4bit` | -| `qwen3.5:14b` | `qwen3.5-14b-q4_k_m.gguf` | `mlx-community/Qwen3.5-14B-4bit` | - -Example updated entry: - -```python -ModelSpec( - model_id="qwen3.5:8b", - name="Qwen3.5 8B", - parameter_count_b=8.0, - active_parameter_count_b=1.0, - context_length=131072, - supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"), # added mlx - provider="alibaba", - metadata={ - "architecture": "moe", - "hf_repo": "Qwen/Qwen3.5-8B", - "gguf_file": "qwen3.5-8b-q4_k_m.gguf", - "mlx_repo": "mlx-community/Qwen3.5-8B-4bit", - }, -) -``` - -### Init Command Changes (`src/openjarvis/cli/init_cmd.py`) - -After the config is written and the "Getting Started" panel is shown, add: - -1. **Size estimate display:** Compute `parameter_count_b * 0.5 * 1.1` and display alongside the model name. Note: this is an estimate based on Q4_K_M quantization. Actual download size may differ slightly by engine/format. The prompt labels it as "~X GB estimated". - -2. **Download prompt:** `click.confirm(f"Download {model} (~{size:.1f} GB estimated) now?", default=True)`. A new `--no-download` flag on the `init` command skips this prompt (for CI/automated environments). - -3. **Engine-specific download functions:** - - `_download_ollama(model, host, console)` — calls a new shared helper `ollama_pull(host, model_name, console) -> bool` extracted from `cli/model.py`. Both the `model pull` CLI command and `init_cmd.py` import this helper, avoiding duplication. - - `_download_llamacpp(model, spec, console)` — `subprocess.run(["huggingface-cli", "download", repo, gguf_file, "--local-dir", cache_dir])`. If `huggingface-cli` is not found (`FileNotFoundError`), prints: `"Install huggingface-cli: pip install huggingface_hub"` and shows the manual download URL as fallback. - - `_download_mlx(model, spec, console)` — `subprocess.run(["huggingface-cli", "download", mlx_repo, "--local-dir", cache_dir])`. Same `FileNotFoundError` handling as llama.cpp. - - `_download_auto(model, engine, console)` — print info message that the model auto-downloads on first serve (vLLM, SGLang). - - `_download_manual(model, engine, console)` — print engine-specific manual instructions (LM Studio, Exo, Nexa). - -4. **Empty model fallback:** Check `if not model:` and print the "not enough memory" warning before the next-steps panel. - -5. **Privacy hook:** After download (or skip), call `_quick_privacy_check()` that runs disk encryption + cloud sync checks and prints a compact summary. - -6. **Next-steps coverage:** Add entries to `_next_steps_text()` for `exo` and `nexa` engines so the Ollama fallback is eliminated. - -### `jarvis model pull` Enhancement (`src/openjarvis/cli/model.py`) - -**Refactor:** Extract the Ollama streaming pull logic (currently lines 165-196) into a reusable function: - -```python -def ollama_pull(host: str, model_name: str, console: Console) -> bool: - """Pull a model via Ollama API. Returns True on success.""" -``` - -The existing `pull` Click command becomes a thin wrapper around this function. - -**Extend** the `pull` command to support engines beyond Ollama: -- Add an `--engine` flag (optional, defaults to configured engine). -- Look up the model in `BUILTIN_MODELS` or `ModelRegistry` to get metadata. -- Dispatch to engine-specific download based on the engine flag. -- Fallback to Ollama pull if no engine specified and model is in Ollama format (name:tag). -- llama.cpp and MLX paths use `huggingface-cli download` with metadata from the catalog, with `FileNotFoundError` handling. - -### Privacy Scanner (`src/openjarvis/cli/scan_cmd.py` — new file) - -#### Data model - -```python -@dataclass -class ScanResult: - name: str # e.g. "FileVault" - status: str # "ok" | "warn" | "fail" | "skip" - message: str # Human-readable explanation - platform: str # "darwin" | "linux" | "all" -``` - -#### `PrivacyScanner` class - -Contains a list of check methods. Each check: -- Is decorated or tagged with the platform it applies to. -- Calls a subprocess and parses the output. -- Returns a `ScanResult`. -- Never raises — catches all exceptions and returns `status="skip"` with the error. - -#### macOS checks - -| Check | Implementation | Status mapping | -|-------|---------------|----------------| -| `check_filevault()` | `subprocess.run(["fdesetup", "status"])`, parse "FileVault is On/Off" | On = ok, Off = fail | -| `check_mdm()` | `subprocess.run(["profiles", "status", "-type", "enrollment"])`, parse for "MDM enrollment" | Not enrolled = ok, enrolled = warn | -| `check_icloud_sync()` | Read `defaults read MobileMeAccounts` for Desktop/Documents sync; check if `~/.openjarvis/` resolves to a path under `~/Library/Mobile Documents/`. Note: `defaults read` output format varies across macOS versions — parsing should be lenient and return `skip` on unexpected output rather than raising. | Not syncing = ok, syncing = warn | -| `check_cloud_sync_agents()` | Check for Dropbox, OneDrive, Google Drive processes via `pgrep`; check if their known sync dirs overlap with `~/.openjarvis/` | No overlap = ok, overlap = warn | -| `check_network_exposure()` | `lsof -iTCP -sTCP:LISTEN -nP`, filter for known engine ports, check bind address | 127.0.0.1 = ok, 0.0.0.0 = warn | -| `check_screen_recording()` | `pgrep` for TeamViewer, AnyDesk, ScreenConnect, vnc | Not found = ok, found = warn | - -#### Linux checks - -| Check | Implementation | Status mapping | -|-------|---------------|----------------| -| `check_luks()` | `lsblk -o NAME,TYPE,FSTYPE -J`, look for `crypto_LUKS` on root device | Found = ok, not found = fail | -| `check_cloud_sync_agents()` | `pgrep` for rclone, dropbox, insync, onedriver | Not found = ok, found = warn | -| `check_network_exposure()` | `ss -tlnp` or `lsof`, same port-check logic | 127.0.0.1 = ok, 0.0.0.0 = warn | -| `check_remote_access()` | `pgrep` for xrdp, x11vnc, vncserver, AnyDesk | Not found = ok, found = warn | - -#### CLI command - -``` -@click.command() -def scan(): - """Audit your environment for privacy and security risks.""" -``` - -Output format: -``` - Privacy & Environment Audit - ──────────────────────────── - ✓ FileVault: enabled - ✓ Network: inference ports bound to localhost only - ! iCloud Drive: Desktop & Documents sync is active - ✗ MDM: enterprise management profile detected - - 1 warning, 1 issue found. -``` - -Symbols: `✓` for ok, `!` for warn, `✗` for fail. Skipped checks are hidden. - -#### Init integration - -A function `_quick_privacy_check()` in `init_cmd.py` that: -1. Instantiates `PrivacyScanner`. -2. Runs only disk encryption and cloud sync overlap checks. -3. Prints a compact 2-3 line summary. -4. Always prints `Run 'jarvis scan' for a full environment audit.` - -### File changes summary - -| File | Type | Description | -|------|------|-------------| -| `src/openjarvis/intelligence/model_catalog.py` | Edit | Add MLX to supported_engines, add gguf_file/mlx_repo metadata | -| `src/openjarvis/core/config.py` | Edit | Add `estimated_download_gb(spec: ModelSpec) -> float` helper (computes `parameter_count_b * 0.5 * 1.1`) | -| `src/openjarvis/cli/init_cmd.py` | Edit | Add download prompt, engine-specific downloaders, empty-model fallback, privacy hook | -| `src/openjarvis/cli/model.py` | Edit | Extend `pull` for llama.cpp and MLX engines | -| `src/openjarvis/cli/scan_cmd.py` | New | `PrivacyScanner` class, check functions, `jarvis scan` command | -| `src/openjarvis/cli/__init__.py` | Edit | Register `scan` command | - -## Testing - -### Issue 1 — Model onboarding - -| Test | File | What it verifies | -|------|------|-----------------| -| MLX model recommendation | `tests/core/test_recommend_model.py` | Apple Silicon 8/16/32/64GB returns valid model with MLX engine (8GB should get `qwen3.5:4b`, 16GB should get `qwen3.5:14b`) | -| Empty model fallback message | `tests/cli/test_init_guidance.py` | `recommend_model` returning `""` shows "not enough memory" | -| Download prompt shown | `tests/cli/test_init_guidance.py` | Init output includes download prompt when model recommended | -| Engine-specific download dispatch | `tests/cli/test_init_guidance.py` | Ollama triggers pull, llamacpp shows HF download, vllm shows auto-download | -| `model pull` multi-engine | `tests/cli/test_model_pull.py` | llama.cpp and MLX pull paths work (mocked subprocess) | - -### Issue 2 — Privacy scanner - -| Test | File | What it verifies | -|------|------|-----------------| -| FileVault check | `tests/cli/test_scan.py` | Parses `fdesetup status` for on/off | -| MDM check | `tests/cli/test_scan.py` | Parses `profiles status` for enrollment | -| iCloud sync detection | `tests/cli/test_scan.py` | Detects overlap with `~/.openjarvis/` | -| LUKS check | `tests/cli/test_scan.py` | Parses `lsblk` for crypto_LUKS | -| Network exposure | `tests/cli/test_scan.py` | Parses `lsof`/`ss` for 0.0.0.0 binds | -| Platform filtering | `tests/cli/test_scan.py` | macOS checks return skip on Linux, vice versa | -| Init privacy hook | `tests/cli/test_init_guidance.py` | Init output includes privacy summary | - -All tests mock subprocess calls via `monkeypatch`. No real system state required. - -## Out of Scope - -- Windows support (future work). -- Post-download inference validation (trying a test query). -- Rust implementation of scanner checks. -- DNS leak detection (complex, unreliable heuristics). -- Automatic remediation (e.g., moving `~/.openjarvis/` out of iCloud). diff --git a/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md b/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md deleted file mode 100644 index acb292da..00000000 --- a/docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-plan.md +++ /dev/null @@ -1,1413 +0,0 @@ -# Init Model Onboarding & Privacy Scanner Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Fix the broken model recommendation for MLX users, add interactive model download to `jarvis init`, and create a `jarvis scan` privacy environment audit command. - -**Architecture:** Two independent features sharing only the init command integration point. Feature 1 modifies the model catalog and init flow. Feature 2 creates a new `PrivacyScanner` class with platform-specific checks exposed via `jarvis scan` CLI command and a lightweight hook in init. - -**Tech Stack:** Python 3.10+, Click (CLI), Rich (terminal UI), httpx (Ollama API), subprocess (system checks) - -**Spec:** `docs/development/specs/2026-03-24-init-onboarding-and-privacy-scanner-design.md` - ---- - -### Task 1: Fix MLX support in model catalog - -**Files:** -- Modify: `src/openjarvis/intelligence/model_catalog.py:42-54` (qwen3.5:3b) -- Modify: `src/openjarvis/intelligence/model_catalog.py:55-67` (qwen3.5:8b) -- Modify: `src/openjarvis/intelligence/model_catalog.py:68-80` (qwen3.5:14b) -- Modify: `src/openjarvis/intelligence/model_catalog.py:219-232` (qwen3.5:4b) -- Test: `tests/core/test_recommend_model.py` - -- [ ] **Step 1: Write failing tests for MLX model recommendation** - -Add a new test class to `tests/core/test_recommend_model.py`: - -```python -class TestRecommendModelMlx: - """Apple Silicon (MLX) model recommendation.""" - - def test_apple_silicon_8gb_mlx(self) -> None: - hw = HardwareInfo( - platform="darwin", - ram_gb=8.0, - gpu=GpuInfo(vendor="apple", name="Apple M1", vram_gb=8.0, count=1), - ) - result = recommend_model(hw, "mlx") - # available = 8 * 0.9 = 7.2 GB - # 8B * 0.5 * 1.1 = 4.4 → fits, but check 14B first: 7.7 → too big - # 4B * 0.5 * 1.1 = 2.2 → fits, but 8B also fits → pick 8B - assert result == "qwen3.5:8b" - - def test_apple_silicon_16gb_mlx(self) -> None: - hw = HardwareInfo( - platform="darwin", - ram_gb=16.0, - gpu=GpuInfo(vendor="apple", name="Apple M2", vram_gb=16.0, count=1), - ) - result = recommend_model(hw, "mlx") - # available = 16 * 0.9 = 14.4 GB - # 14B * 0.5 * 1.1 = 7.7 → fits - assert result == "qwen3.5:14b" - - def test_apple_silicon_32gb_mlx(self) -> None: - hw = HardwareInfo( - platform="darwin", - ram_gb=32.0, - gpu=GpuInfo(vendor="apple", name="Apple M2 Pro", vram_gb=32.0, count=1), - ) - result = recommend_model(hw, "mlx") - # available = 32 * 0.9 = 28.8 GB - # 14B * 0.5 * 1.1 = 7.7 → fits (14b is the largest with mlx support) - assert result == "qwen3.5:14b" - - def test_apple_silicon_64gb_mlx(self) -> None: - hw = HardwareInfo( - platform="darwin", - ram_gb=64.0, - gpu=GpuInfo(vendor="apple", name="Apple M2 Max", vram_gb=64.0, count=1), - ) - result = recommend_model(hw, "mlx") - # available = 64 * 0.9 = 57.6 GB — but 14b is the largest MLX model - assert result == "qwen3.5:14b" -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `uv run pytest tests/core/test_recommend_model.py::TestRecommendModelMlx -v` -Expected: FAIL — all 4 tests fail because no Qwen3.5 model has `"mlx"` in `supported_engines`. - -- [ ] **Step 3: Add MLX to supported_engines and download metadata** - -In `src/openjarvis/intelligence/model_catalog.py`, update the four Qwen3.5 model entries: - -For `qwen3.5:3b` (line 48): change `supported_engines` from `("ollama", "vllm", "llamacpp", "sglang")` to `("ollama", "vllm", "llamacpp", "sglang", "mlx")`. Add `"gguf_file": "qwen3.5-3b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-3B-4bit"` to metadata. - -For `qwen3.5:8b` (line 61): change `supported_engines` from `("ollama", "vllm", "llamacpp", "sglang")` to `("ollama", "vllm", "llamacpp", "sglang", "mlx")`. Add `"gguf_file": "qwen3.5-8b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-8B-4bit"` to metadata. - -For `qwen3.5:14b` (line 74): change `supported_engines` from `("ollama", "vllm", "llamacpp", "sglang")` to `("ollama", "vllm", "llamacpp", "sglang", "mlx")`. Add `"gguf_file": "qwen3.5-14b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-14B-4bit"` to metadata. - -For `qwen3.5:4b` (line 226): change `supported_engines` from `("ollama", "vllm", "sglang", "llamacpp")` to `("ollama", "vllm", "sglang", "llamacpp", "mlx")`. Add `"gguf_file": "qwen3.5-4b-q4_k_m.gguf"` and `"mlx_repo": "mlx-community/Qwen3.5-4B-4bit"` to metadata. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `uv run pytest tests/core/test_recommend_model.py -v` -Expected: ALL 15 tests pass (11 existing + 4 new). - -- [ ] **Step 5: Add `estimated_download_gb` helper to config.py** - -In `src/openjarvis/core/config.py`, after the `recommend_model` function (after line 248), add: - -```python -def estimated_download_gb(parameter_count_b: float) -> float: - """Estimate download size in GB for Q4_K_M quantized model.""" - return parameter_count_b * 0.5 * 1.1 -``` - -- [ ] **Step 6: Commit** - -```bash -git add src/openjarvis/intelligence/model_catalog.py src/openjarvis/core/config.py tests/core/test_recommend_model.py -git commit -m "fix: add MLX engine support to Qwen3.5 model catalog entries - -Fixes recommend_model() returning empty string on Apple Silicon -when MLX is the recommended engine. Also adds gguf_file and -mlx_repo download metadata, and estimated_download_gb helper." -``` - ---- - -### Task 2: Extract Ollama pull helper and extend `jarvis model pull` - -**Files:** -- Modify: `src/openjarvis/cli/model.py:152-197` -- Test: `tests/cli/test_model_pull.py` (new) - -- [ ] **Step 1: Write failing tests for multi-engine pull** - -Create `tests/cli/test_model_pull.py`: - -```python -"""Tests for ``jarvis model pull`` multi-engine support.""" - -from __future__ import annotations - -from unittest import mock - -import pytest -from click.testing import CliRunner -from rich.console import Console - -from openjarvis.cli.model import ollama_pull - - -class TestOllamaPull: - """Test the extracted ollama_pull helper.""" - - def test_ollama_pull_success(self) -> None: - import io - console = Console(file=io.StringIO()) - mock_lines = [ - '{"status": "pulling manifest"}', - '{"status": "downloading", "total": 100, "completed": 100}', - '{"status": "success"}', - ] - mock_resp = mock.MagicMock() - mock_resp.raise_for_status = mock.MagicMock() - mock_resp.iter_lines.return_value = iter(mock_lines) - mock_resp.__enter__ = mock.MagicMock(return_value=mock_resp) - mock_resp.__exit__ = mock.MagicMock(return_value=False) - - with mock.patch("httpx.stream", return_value=mock_resp): - result = ollama_pull("http://localhost:11434", "qwen3.5:3b", console) - assert result is True - - def test_ollama_pull_connect_error(self) -> None: - import io - import httpx - - console = Console(file=io.StringIO()) - with mock.patch("httpx.stream", side_effect=httpx.ConnectError("refused")): - result = ollama_pull("http://localhost:11434", "qwen3.5:3b", console) - assert result is False - - -class TestPullCliMultiEngine: - """Test the pull CLI command dispatches to correct engine.""" - - def test_pull_llamacpp_uses_huggingface_cli(self) -> None: - from openjarvis.cli import cli - - runner = CliRunner() - with ( - mock.patch("openjarvis.cli.model.load_config") as mock_cfg, - mock.patch("subprocess.run") as mock_run, - ): - mock_cfg.return_value.engine.default = "llamacpp" - mock_cfg.return_value.engine.ollama_host = None - mock_run.return_value = mock.MagicMock(returncode=0) - - result = runner.invoke( - cli, ["model", "pull", "qwen3.5:8b", "--engine", "llamacpp"] - ) - - assert result.exit_code == 0 - mock_run.assert_called_once() - call_args = mock_run.call_args[0][0] - assert "huggingface-cli" in call_args - assert "qwen3.5-8b-q4_k_m.gguf" in call_args - - def test_pull_mlx_uses_huggingface_cli(self) -> None: - from openjarvis.cli import cli - - runner = CliRunner() - with ( - mock.patch("openjarvis.cli.model.load_config") as mock_cfg, - mock.patch("subprocess.run") as mock_run, - ): - mock_cfg.return_value.engine.default = "mlx" - mock_cfg.return_value.engine.ollama_host = None - mock_run.return_value = mock.MagicMock(returncode=0) - - result = runner.invoke( - cli, ["model", "pull", "qwen3.5:8b", "--engine", "mlx"] - ) - - assert result.exit_code == 0 - mock_run.assert_called_once() - call_args = mock_run.call_args[0][0] - assert "huggingface-cli" in call_args - assert "mlx-community/Qwen3.5-8B-4bit" in call_args - - def test_pull_llamacpp_huggingface_cli_not_found(self) -> None: - from openjarvis.cli import cli - - runner = CliRunner() - with ( - mock.patch("openjarvis.cli.model.load_config") as mock_cfg, - mock.patch("subprocess.run", side_effect=FileNotFoundError), - ): - mock_cfg.return_value.engine.default = "llamacpp" - mock_cfg.return_value.engine.ollama_host = None - - result = runner.invoke( - cli, ["model", "pull", "qwen3.5:8b", "--engine", "llamacpp"] - ) - - assert result.exit_code != 0 - assert "huggingface_hub" in result.output or "pip install" in result.output -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `uv run pytest tests/cli/test_model_pull.py -v` -Expected: FAIL — `ollama_pull` function doesn't exist yet, `--engine` flag not recognized. - -- [ ] **Step 3: Refactor model.py — extract ollama_pull and add multi-engine support** - -In `src/openjarvis/cli/model.py`: - -1. Add `import subprocess` at the top. -2. Add `from openjarvis.intelligence.model_catalog import BUILTIN_MODELS` at the top. -3. Extract the Ollama pull logic into a standalone function: - -```python -def ollama_pull(host: str, model_name: str, console: Console) -> bool: - """Pull a model via Ollama API. Returns True on success.""" - console.print(f"Pulling [cyan]{model_name}[/cyan] via Ollama...") - try: - with httpx.stream( - "POST", - f"{host}/api/pull", - json={"name": model_name, "stream": True}, - timeout=600.0, - ) as resp: - resp.raise_for_status() - import json - - for line in resp.iter_lines(): - if not line.strip(): - continue - try: - data = json.loads(line) - except Exception: - continue - status = data.get("status", "") - if "total" in data and "completed" in data: - total = data["total"] - done = data["completed"] - pct = int(done / total * 100) if total else 0 - console.print(f" {status}: {pct}%", end="\r") - elif status: - console.print(f" {status}") - console.print(f"\n[green]Successfully pulled {model_name}[/green]") - return True - except httpx.ConnectError: - console.print("[red]Cannot connect to Ollama.[/red] Is it running?") - return False - except httpx.HTTPStatusError as exc: - console.print(f"[red]Ollama error:[/red] {exc.response.status_code}") - return False -``` - -4. Add helper to find model spec: - -```python -def find_model_spec(model_name: str): - """Look up a model in the builtin catalog. Returns None if not found.""" - for spec in BUILTIN_MODELS: - if spec.model_id == model_name: - return spec - return None -``` - -5. Add HuggingFace download helper: - -```python -def hf_download(repo: str, filename: str | None, console: Console) -> bool: - """Download from HuggingFace via huggingface-cli. Returns True on success.""" - cmd = ["huggingface-cli", "download", repo] - if filename: - cmd.append(filename) - try: - result = subprocess.run(cmd, check=True) - console.print(f"[green]Download complete.[/green]") - return True - except FileNotFoundError: - console.print( - "[red]huggingface-cli not found.[/red]\n" - "Install it: [cyan]pip install huggingface_hub[/cyan]\n" - f"Or download manually: https://huggingface.co/{repo}" - ) - return False - except subprocess.CalledProcessError: - console.print(f"[red]Download failed.[/red]") - return False -``` - -6. Replace the existing `pull` command with a multi-engine version: - -```python -@model.command() -@click.argument("model_name") -@click.option("--engine", default=None, help="Engine to download for (ollama, llamacpp, mlx).") -def pull(model_name: str, engine: str | None) -> None: - """Download a model.""" - console = Console() - config = load_config() - - engine = engine or config.engine.default or "ollama" - - if engine == "ollama": - host = ( - config.engine.ollama_host - or os.environ.get("OLLAMA_HOST") - or "http://localhost:11434" - ).rstrip("/") - if not ollama_pull(host, model_name, console): - sys.exit(1) - elif engine in ("llamacpp", "mlx"): - spec = find_model_spec(model_name) - if not spec: - console.print(f"[red]Model not in catalog:[/red] {model_name}") - sys.exit(1) - if engine == "llamacpp": - repo = spec.metadata.get("hf_repo", "") - gguf = spec.metadata.get("gguf_file", "") - if not repo or not gguf: - console.print(f"[red]No GGUF download info for {model_name}[/red]") - sys.exit(1) - console.print(f"Downloading [cyan]{gguf}[/cyan] from {repo}...") - if not hf_download(repo, gguf, console): - sys.exit(1) - else: # mlx - mlx_repo = spec.metadata.get("mlx_repo", "") - if not mlx_repo: - console.print(f"[red]No MLX repo info for {model_name}[/red]") - sys.exit(1) - console.print(f"Downloading [cyan]{mlx_repo}[/cyan]...") - if not hf_download(mlx_repo, None, console): - sys.exit(1) - elif engine in ("vllm", "sglang"): - console.print( - f"[cyan]{model_name}[/cyan] will download automatically when " - f"{engine} starts serving it." - ) - else: - console.print( - f"Manual download required for engine [cyan]{engine}[/cyan].\n" - f"Check the engine documentation for instructions." - ) -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `uv run pytest tests/cli/test_model_pull.py -v` -Expected: ALL tests pass. - -- [ ] **Step 5: Run existing tests to verify no regressions** - -Run: `uv run pytest tests/core/test_recommend_model.py tests/cli/test_init_guidance.py -v` -Expected: ALL pass. - -- [ ] **Step 6: Commit** - -```bash -git add src/openjarvis/cli/model.py tests/cli/test_model_pull.py -git commit -m "feat: extract ollama_pull helper and add multi-engine model pull - -Refactors model pull into reusable ollama_pull() function. Adds ---engine flag to support llamacpp (GGUF) and mlx (HuggingFace) -downloads via huggingface-cli, with FileNotFoundError handling." -``` - ---- - -### Task 3: Add interactive download and empty-model fallback to `jarvis init` - -**Files:** -- Modify: `src/openjarvis/cli/init_cmd.py:139-299` -- Test: `tests/cli/test_init_guidance.py` - -- [ ] **Step 1: Write failing tests for download prompt and empty-model fallback** - -Add to `tests/cli/test_init_guidance.py`: - -```python -class TestInitDownloadPrompt: - """Interactive download prompt during init.""" - - def test_init_shows_download_prompt(self, tmp_path: Path) -> None: - config_dir = tmp_path / ".openjarvis" - config_path = config_dir / "config.toml" - with ( - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), - ): - result = CliRunner().invoke( - cli, ["init", "--engine", "ollama"], input="n\n" - ) - assert result.exit_code == 0 - assert "Download" in result.output and "now?" in result.output - - def test_init_no_download_flag_skips_prompt(self, tmp_path: Path) -> None: - config_dir = tmp_path / ".openjarvis" - config_path = config_dir / "config.toml" - with ( - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), - ): - result = CliRunner().invoke( - cli, ["init", "--engine", "ollama", "--no-download"] - ) - assert result.exit_code == 0 - assert "Download" not in result.output or "now?" not in result.output - - -class TestInitEmptyModelFallback: - """Empty model recommendation shows helpful message.""" - - def test_init_no_model_shows_warning(self, tmp_path: Path) -> None: - config_dir = tmp_path / ".openjarvis" - config_path = config_dir / "config.toml" - with ( - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), - mock.patch( - "openjarvis.cli.init_cmd.recommend_model", return_value="" - ), - ): - result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) - assert result.exit_code == 0 - assert "Not enough memory" in result.output or "not enough memory" in result.output - - -class TestNextStepsExoNexa: - """Exo and Nexa have their own next-steps text.""" - - def test_next_steps_exo(self) -> None: - text = _next_steps_text("exo") - assert "exo" in text.lower() - assert "jarvis ask" in text - # Should NOT fall back to Ollama instructions - assert "ollama" not in text.lower() - - def test_next_steps_nexa(self) -> None: - text = _next_steps_text("nexa") - assert "nexa" in text.lower() - assert "jarvis ask" in text - assert "ollama" not in text.lower() -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `uv run pytest tests/cli/test_init_guidance.py::TestInitDownloadPrompt tests/cli/test_init_guidance.py::TestInitEmptyModelFallback tests/cli/test_init_guidance.py::TestNextStepsExoNexa -v` -Expected: FAIL — no `--no-download` flag, no "Not enough memory" message, no exo/nexa next-steps. - -- [ ] **Step 3: Update init_cmd.py** - -In `src/openjarvis/cli/init_cmd.py`: - -1. Add imports at the top: - -```python -from openjarvis.cli.model import ollama_pull, find_model_spec, hf_download -from openjarvis.core.config import estimated_download_gb -``` - -2. Add `exo` and `nexa` entries to `_next_steps_text()` dict (before the closing `}`): - -```python - "exo": ( - "Next steps:\n" - "\n" - " 1. Install and start Exo:\n" - " pip install exo\n" - " exo\n" - "\n" - " 2. Try it out:\n" - " jarvis ask \"Hello\"\n" - "\n" - " Run `jarvis doctor` to verify your setup." - ), - "nexa": ( - "Next steps:\n" - "\n" - " 1. Install and start Nexa:\n" - " pip install nexaai\n" - " nexa server\n" - "\n" - " 2. Try it out:\n" - " jarvis ask \"Hello\"\n" - "\n" - " Run `jarvis doctor` to verify your setup." - ), -``` - -3. Add `--no-download` flag to the `@click.command()` decorator chain (after `--engine`): - -```python -@click.option( - "--no-download", - is_flag=True, - default=False, - help="Skip the model download prompt.", -) -``` - -4. Update the `init()` function signature to accept `no_download: bool = False`. - -5. Replace the block at lines 287-298 (the model recommendation + next-steps panel) with: - -```python - selected_engine = engine or recommend_engine(hw) - model = recommend_model(hw, selected_engine) - - if not model: - console.print( - "\n [yellow]! Not enough memory to run any local model.[/yellow]\n" - " Consider a cloud engine or a machine with more RAM." - ) - else: - spec = find_model_spec(model) - size_gb = estimated_download_gb(spec.parameter_count_b) if spec else 0 - console.print(f"\n [bold]Recommended model:[/bold] {model} (~{size_gb:.1f} GB estimated)") - - if not no_download and spec: - if click.confirm(f" Download {model} (~{size_gb:.1f} GB estimated) now?", default=True): - _do_download(selected_engine, model, spec, console) - - console.print() - console.print( - Panel( - _next_steps_text(selected_engine, model), - title="Getting Started", - border_style="cyan", - ) - ) -``` - -6. Add the `_do_download` helper function: - -```python -def _do_download(engine: str, model: str, spec, console: Console) -> None: - """Dispatch model download based on engine type.""" - import os - - if engine == "ollama": - host = os.environ.get("OLLAMA_HOST", "http://localhost:11434").rstrip("/") - ollama_pull(host, model, console) - elif engine == "llamacpp": - repo = spec.metadata.get("hf_repo", "") - gguf = spec.metadata.get("gguf_file", "") - if repo and gguf: - console.print(f" Downloading [cyan]{gguf}[/cyan] from {repo}...") - hf_download(repo, gguf, console) - else: - console.print(f" [yellow]No GGUF download info for {model}[/yellow]") - elif engine == "mlx": - mlx_repo = spec.metadata.get("mlx_repo", "") - if mlx_repo: - console.print(f" Downloading [cyan]{mlx_repo}[/cyan]...") - hf_download(mlx_repo, None, console) - else: - console.print(f" [yellow]No MLX repo info for {model}[/yellow]") - elif engine in ("vllm", "sglang"): - console.print( - f" [cyan]{model}[/cyan] will download automatically when " - f"{engine} starts serving it." - ) - else: - console.print( - f" Download {model} through the {engine} interface." - ) -``` - -- [ ] **Step 4: Update existing init tests to add `--no-download`** - -The download prompt breaks existing tests because `click.confirm` will try to download a model. Update all existing init CLI tests in `tests/cli/test_init_guidance.py` to add `"--no-download"` to their invocation args: - -- `test_init_shows_next_steps`: change `["init", "--engine", "llamacpp"]` to `["init", "--engine", "llamacpp", "--no-download"]` -- `test_init_output_shows_toml_sections_literally`: same change -- `test_init_generates_minimal_by_default`: change `["init", "--engine", "ollama"]` to `["init", "--engine", "ollama", "--no-download"]` -- `test_init_full_generates_verbose_config`: change `["init", "--full", "--engine", "ollama"]` to `["init", "--full", "--engine", "ollama", "--no-download"]` - -- [ ] **Step 5: Add download dispatch tests** - -Add to `tests/cli/test_init_guidance.py`: - -```python -class TestInitDownloadDispatch: - """Verify download dispatches correctly for each engine.""" - - def test_init_ollama_download_calls_ollama_pull(self, tmp_path: Path) -> None: - config_dir = tmp_path / ".openjarvis" - config_path = config_dir / "config.toml" - with ( - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), - mock.patch("openjarvis.cli.init_cmd.ollama_pull", return_value=True) as mock_pull, - ): - result = CliRunner().invoke( - cli, ["init", "--engine", "ollama"], input="y\n" - ) - assert result.exit_code == 0 - mock_pull.assert_called_once() - - def test_init_vllm_shows_auto_download_message(self, tmp_path: Path) -> None: - config_dir = tmp_path / ".openjarvis" - config_path = config_dir / "config.toml" - with ( - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), - ): - result = CliRunner().invoke( - cli, ["init", "--engine", "vllm"], input="y\n" - ) - assert result.exit_code == 0 - assert "automatically" in result.output -``` - -- [ ] **Step 6: Run tests to verify they pass** - -Run: `uv run pytest tests/cli/test_init_guidance.py -v` -Expected: ALL tests pass (10 existing + 7 new). - -- [ ] **Step 7: Lint** - -Run: `uv run ruff check src/openjarvis/cli/init_cmd.py --fix` -Expected: Clean or auto-fixed. - -- [ ] **Step 8: Commit** - -```bash -git add src/openjarvis/cli/init_cmd.py tests/cli/test_init_guidance.py -git commit -m "feat: add interactive model download and empty-model fallback to init - -Prompts user to download recommended model during jarvis init. -Adds --no-download flag for CI. Shows helpful message when no -model fits available memory. Adds exo/nexa next-steps text." -``` - ---- - -### Task 4: Create privacy scanner — ScanResult and check functions - -**Files:** -- Create: `src/openjarvis/cli/scan_cmd.py` -- Test: `tests/cli/test_scan.py` (new) - -- [ ] **Step 1: Write failing tests for individual scanner checks** - -Create `tests/cli/test_scan.py`: - -```python -"""Tests for ``jarvis scan`` privacy environment audit.""" - -from __future__ import annotations - -import subprocess -import sys -from unittest import mock - -import pytest - -from openjarvis.cli.scan_cmd import PrivacyScanner, ScanResult - - -class TestScanResultDataclass: - def test_scan_result_fields(self) -> None: - r = ScanResult(name="Test", status="ok", message="All good", platform="all") - assert r.name == "Test" - assert r.status == "ok" - assert r.message == "All good" - assert r.platform == "all" - - -class TestFileVault: - def test_filevault_enabled(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout="FileVault is On.", returncode=0 - ) - result = scanner.check_filevault() - assert result.status == "ok" - assert "enabled" in result.message.lower() - - def test_filevault_disabled(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout="FileVault is Off.", returncode=0 - ) - result = scanner.check_filevault() - assert result.status == "fail" - - def test_filevault_command_not_found(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run", side_effect=FileNotFoundError): - result = scanner.check_filevault() - assert result.status == "skip" - - -class TestMDM: - def test_mdm_not_enrolled(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout="This machine is not enrolled", returncode=0 - ) - result = scanner.check_mdm() - assert result.status == "ok" - - def test_mdm_enrolled(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout="Enrolled via DEP: Yes\nMDM server: example.com", - returncode=0, - ) - result = scanner.check_mdm() - assert result.status == "warn" - - -class TestCloudSync: - def test_no_cloud_sync_agents(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - # pgrep returns exit code 1 when no match - mock_run.return_value = mock.MagicMock(stdout="", returncode=1) - result = scanner.check_cloud_sync_agents() - assert result.status == "ok" - - def test_dropbox_running(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - def side_effect(cmd, **kwargs): - m = mock.MagicMock() - # Match on the process name in the pgrep pattern - if any("Dropbox" in str(c) for c in cmd): - m.stdout = "12345" - m.returncode = 0 - else: - m.stdout = "" - m.returncode = 1 - return m - mock_run.side_effect = side_effect - result = scanner.check_cloud_sync_agents() - assert result.status == "warn" - assert "dropbox" in result.message.lower() - - -class TestNetworkExposure: - def test_no_exposed_ports(self) -> None: - scanner = PrivacyScanner() - lsof_output = ( - "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n" - "ollama 12345 user 5u IPv4 0x1234 0t0 TCP 127.0.0.1:11434 (LISTEN)\n" - ) - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout=lsof_output, returncode=0 - ) - result = scanner.check_network_exposure() - assert result.status == "ok" - - def test_exposed_port(self) -> None: - scanner = PrivacyScanner() - lsof_output = ( - "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n" - "ollama 12345 user 5u IPv4 0x1234 0t0 TCP *:11434 (LISTEN)\n" - ) - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout=lsof_output, returncode=0 - ) - result = scanner.check_network_exposure() - assert result.status == "warn" - assert "11434" in result.message - - -class TestLUKS: - def test_luks_encrypted(self) -> None: - scanner = PrivacyScanner() - lsblk_json = '{"blockdevices": [{"name": "sda", "type": "disk", "fstype": null, "children": [{"name": "sda1", "type": "part", "fstype": "crypto_LUKS"}]}]}' - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout=lsblk_json, returncode=0 - ) - result = scanner.check_luks() - assert result.status == "ok" - - def test_luks_not_encrypted(self) -> None: - scanner = PrivacyScanner() - lsblk_json = '{"blockdevices": [{"name": "sda", "type": "disk", "fstype": null, "children": [{"name": "sda1", "type": "part", "fstype": "ext4"}]}]}' - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock( - stdout=lsblk_json, returncode=0 - ) - result = scanner.check_luks() - assert result.status == "fail" - - -class TestScreenRecording: - def test_no_screen_recording(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - mock_run.return_value = mock.MagicMock(stdout="", returncode=1) - result = scanner.check_screen_recording() - assert result.status == "ok" - - def test_teamviewer_running(self) -> None: - scanner = PrivacyScanner() - with mock.patch("subprocess.run") as mock_run: - def side_effect(cmd, **kwargs): - m = mock.MagicMock() - if any("TeamViewer" in str(c) for c in cmd): - m.stdout = "12345" - m.returncode = 0 - else: - m.stdout = "" - m.returncode = 1 - return m - mock_run.side_effect = side_effect - result = scanner.check_screen_recording() - assert result.status == "warn" - - -class TestPlatformFiltering: - def test_run_all_returns_only_current_platform(self) -> None: - scanner = PrivacyScanner() - with mock.patch.object(scanner, "_get_all_checks") as mock_checks: - mock_checks.return_value = [ - lambda: ScanResult("Test1", "ok", "ok", "darwin"), - lambda: ScanResult("Test2", "ok", "ok", "linux"), - lambda: ScanResult("Test3", "ok", "ok", "all"), - ] - results = scanner.run_all() - current = sys.platform - for r in results: - assert r.platform in (current, "all") -``` - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `uv run pytest tests/cli/test_scan.py -v` -Expected: FAIL — `scan_cmd` module doesn't exist. - -- [ ] **Step 3: Implement PrivacyScanner class** - -Create `src/openjarvis/cli/scan_cmd.py`: - -```python -"""``jarvis scan`` — privacy and environment audit.""" - -from __future__ import annotations - -import json -import subprocess -import sys -from dataclasses import dataclass -from pathlib import Path -from typing import Callable, List - -import click -from rich.console import Console - - -@dataclass(slots=True) -class ScanResult: - """Result of a single privacy check.""" - - name: str - status: str # "ok" | "warn" | "fail" | "skip" - message: str - platform: str # "darwin" | "linux" | "all" - - -# Engine ports to check for network exposure -_ENGINE_PORTS = { - 11434: "Ollama", - 8080: "llama.cpp / MLX", - 8000: "vLLM", - 30000: "SGLang", - 1234: "LM Studio", - 52415: "Exo", - 18181: "Nexa", -} - -# Cloud sync process names to check -_CLOUD_SYNC_PROCESSES = ["Dropbox", "OneDrive", "Google Drive", "iCloudDrive"] - -# Screen recording / remote access process names -_SCREEN_RECORDING_PROCESSES_MACOS = [ - "TeamViewer", "AnyDesk", "ScreenConnect", "VNC", -] -_REMOTE_ACCESS_PROCESSES_LINUX = [ - "xrdp", "x11vnc", "vncserver", "AnyDesk", "TeamViewer", -] - - -class PrivacyScanner: - """Run platform-specific privacy and environment checks.""" - - def _run(self, cmd: list[str], **kwargs) -> subprocess.CompletedProcess: - """Run a subprocess, capturing stdout as text.""" - return subprocess.run( - cmd, capture_output=True, text=True, timeout=10, **kwargs - ) - - # ------------------------------------------------------------------ - # macOS checks - # ------------------------------------------------------------------ - - def check_filevault(self) -> ScanResult: - """Check macOS FileVault disk encryption status.""" - try: - proc = self._run(["fdesetup", "status"]) - if "On" in proc.stdout: - return ScanResult("FileVault", "ok", "FileVault enabled", "darwin") - return ScanResult( - "FileVault", "fail", - "FileVault is disabled — disk is not encrypted", "darwin" - ) - except (FileNotFoundError, subprocess.TimeoutExpired): - return ScanResult("FileVault", "skip", "fdesetup not available", "darwin") - - def check_mdm(self) -> ScanResult: - """Check for MDM / enterprise management enrollment.""" - try: - proc = self._run(["profiles", "status", "-type", "enrollment"]) - output = proc.stdout + proc.stderr - if "MDM" in output or "Enrolled" in output: - return ScanResult( - "MDM", "warn", - "Enterprise management profile detected — this device may be monitored", - "darwin", - ) - return ScanResult("MDM", "ok", "No MDM enrollment detected", "darwin") - except (FileNotFoundError, subprocess.TimeoutExpired): - return ScanResult("MDM", "skip", "profiles command not available", "darwin") - - def check_icloud_sync(self) -> ScanResult: - """Check if ~/.openjarvis/ could be synced by iCloud Drive.""" - try: - oj_path = Path.home() / ".openjarvis" - # Check if the path is under iCloud's mobile documents - mobile_docs = Path.home() / "Library" / "Mobile Documents" - try: - resolved = oj_path.resolve() - if str(resolved).startswith(str(mobile_docs)): - return ScanResult( - "iCloud Drive", "warn", - "~/.openjarvis/ is inside iCloud Drive sync path", - "darwin", - ) - except OSError: - pass - - # Check if Desktop & Documents sync is enabled - proc = self._run( - ["defaults", "read", "com.apple.bird", "optimize-storage"] - ) - # Also check the broader MobileMeAccounts for sync status - proc2 = self._run( - ["defaults", "read", "MobileMeAccounts"] - ) - output = proc.stdout + proc2.stdout - if "MOBILE_DOCUMENTS" in output or "Desktop" in output: - return ScanResult( - "iCloud Drive", "warn", - "iCloud Desktop & Documents sync may be active — " - "~/.openjarvis/ could be synced to Apple servers", - "darwin", - ) - return ScanResult( - "iCloud Drive", "ok", - "No iCloud sync overlap detected", "darwin" - ) - except (FileNotFoundError, subprocess.TimeoutExpired): - return ScanResult( - "iCloud Drive", "skip", - "Could not determine iCloud sync status", "darwin" - ) - except Exception: - return ScanResult( - "iCloud Drive", "skip", - "Could not determine iCloud sync status", "darwin" - ) - - # ------------------------------------------------------------------ - # Linux checks - # ------------------------------------------------------------------ - - def check_luks(self) -> ScanResult: - """Check for LUKS disk encryption on Linux.""" - try: - proc = self._run(["lsblk", "-o", "NAME,TYPE,FSTYPE", "-J"]) - data = json.loads(proc.stdout) - - def _has_luks(devices: list) -> bool: - for dev in devices: - if dev.get("fstype") == "crypto_LUKS": - return True - if _has_luks(dev.get("children", [])): - return True - return False - - if _has_luks(data.get("blockdevices", [])): - return ScanResult( - "Disk Encryption", "ok", - "LUKS encryption detected", "linux" - ) - return ScanResult( - "Disk Encryption", "fail", - "No LUKS encryption detected — disk may not be encrypted", - "linux", - ) - except (FileNotFoundError, subprocess.TimeoutExpired): - return ScanResult( - "Disk Encryption", "skip", - "lsblk not available", "linux" - ) - except (json.JSONDecodeError, KeyError): - return ScanResult( - "Disk Encryption", "skip", - "Could not parse lsblk output", "linux" - ) - - def check_remote_access(self) -> ScanResult: - """Check for remote access tools on Linux.""" - return self._check_processes( - _REMOTE_ACCESS_PROCESSES_LINUX, - "Remote Access", - "Remote access tool detected", - "linux", - ) - - # ------------------------------------------------------------------ - # Cross-platform checks - # ------------------------------------------------------------------ - - def check_cloud_sync_agents(self) -> ScanResult: - """Check for running cloud sync agents.""" - platform = "darwin" if sys.platform == "darwin" else "linux" - return self._check_processes( - _CLOUD_SYNC_PROCESSES, - "Cloud Sync", - "Cloud sync agent detected", - platform, - ) - - def check_network_exposure(self) -> ScanResult: - """Check if inference engine ports are bound to 0.0.0.0.""" - platform = "darwin" if sys.platform == "darwin" else "linux" - try: - if sys.platform == "darwin": - proc = self._run(["lsof", "-iTCP", "-sTCP:LISTEN", "-nP"]) - else: - proc = self._run(["ss", "-tlnp"]) - - exposed = [] - for line in proc.stdout.splitlines(): - for port, engine_name in _ENGINE_PORTS.items(): - port_str = str(port) - if port_str in line and ("*:" + port_str in line or "0.0.0.0:" + port_str in line): - exposed.append(f"{engine_name} (port {port})") - - if exposed: - return ScanResult( - "Network Exposure", "warn", - f"Inference ports exposed to network: {', '.join(exposed)}", - platform, - ) - return ScanResult( - "Network Exposure", "ok", - "Inference ports bound to localhost only", platform - ) - except (FileNotFoundError, subprocess.TimeoutExpired): - return ScanResult( - "Network Exposure", "skip", - "Could not check listening ports", platform - ) - - def check_screen_recording(self) -> ScanResult: - """Check for screen recording / remote desktop tools (macOS).""" - return self._check_processes( - _SCREEN_RECORDING_PROCESSES_MACOS, - "Screen Recording", - "Screen recording or remote access tool detected", - "darwin", - ) - - # ------------------------------------------------------------------ - # Helpers - # ------------------------------------------------------------------ - - def _check_processes( - self, - process_names: list[str], - check_name: str, - warn_message: str, - platform: str, - ) -> ScanResult: - """Check if any of the named processes are running.""" - found = [] - for name in process_names: - try: - proc = self._run(["pgrep", "-i", name]) - if proc.returncode == 0 and proc.stdout.strip(): - found.append(name) - except (FileNotFoundError, subprocess.TimeoutExpired): - continue - if found: - return ScanResult( - check_name, "warn", - f"{warn_message}: {', '.join(found)}", platform - ) - return ScanResult( - check_name, "ok", - f"No {check_name.lower()} detected", platform - ) - - def _get_all_checks(self) -> List[Callable[[], ScanResult]]: - """Return all check methods.""" - return [ - self.check_filevault, - self.check_mdm, - self.check_icloud_sync, - self.check_luks, - self.check_cloud_sync_agents, - self.check_network_exposure, - self.check_screen_recording, - self.check_remote_access, - ] - - def run_all(self) -> list[ScanResult]: - """Run all checks, filtering to current platform.""" - results = [] - for check in self._get_all_checks(): - result = check() - if result.platform in (sys.platform, "all"): - if result.status != "skip": - results.append(result) - return results - - def run_quick(self) -> list[ScanResult]: - """Run only critical checks (for init hook).""" - checks = [self.check_filevault, self.check_luks, self.check_icloud_sync, - self.check_cloud_sync_agents] - results = [] - for check in checks: - result = check() - if result.platform in (sys.platform, "all"): - if result.status != "skip": - results.append(result) - return results - - -# ------------------------------------------------------------------ -# CLI command -# ------------------------------------------------------------------ - -_STATUS_SYMBOLS = {"ok": "\u2713", "warn": "!", "fail": "\u2717"} - - -@click.command() -def scan() -> None: - """Audit your environment for privacy and security risks.""" - console = Console() - scanner = PrivacyScanner() - results = scanner.run_all() - - console.print() - console.print(" [bold]Privacy & Environment Audit[/bold]") - console.print(" " + "\u2500" * 28) - - warns = 0 - fails = 0 - for r in results: - symbol = _STATUS_SYMBOLS.get(r.status, "?") - if r.status == "ok": - console.print(f" [green]{symbol}[/green] {r.name}: {r.message}") - elif r.status == "warn": - console.print(f" [yellow]{symbol}[/yellow] {r.name}: {r.message}") - warns += 1 - elif r.status == "fail": - console.print(f" [red]{symbol}[/red] {r.name}: {r.message}") - fails += 1 - - console.print() - if warns == 0 and fails == 0: - console.print(" [green]No issues found.[/green]") - else: - parts = [] - if warns: - parts.append(f"{warns} warning{'s' if warns != 1 else ''}") - if fails: - parts.append(f"{fails} issue{'s' if fails != 1 else ''}") - console.print(f" {', '.join(parts)} found.") - console.print() -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `uv run pytest tests/cli/test_scan.py -v` -Expected: ALL tests pass. - -- [ ] **Step 5: Lint** - -Run: `uv run ruff check src/openjarvis/cli/scan_cmd.py --fix` -Expected: Clean or auto-fixed. - -- [ ] **Step 6: Commit** - -```bash -git add src/openjarvis/cli/scan_cmd.py tests/cli/test_scan.py -git commit -m "feat: add privacy environment scanner with platform-specific checks - -New PrivacyScanner class with checks for disk encryption (FileVault/LUKS), -MDM profiles, cloud sync agents, network exposure, and screen recording. -Supports macOS and Linux with graceful skip on missing tools." -``` - ---- - -### Task 5: Register `jarvis scan` and integrate privacy hook into init - -**Files:** -- Modify: `src/openjarvis/cli/__init__.py:34` (add import) -- Modify: `src/openjarvis/cli/__init__.py:89` (register command) -- Modify: `src/openjarvis/cli/init_cmd.py` (add privacy hook) -- Test: `tests/cli/test_init_guidance.py` - -- [ ] **Step 1: Write failing test for init privacy hook** - -Add to `tests/cli/test_init_guidance.py`: - -```python -class TestInitPrivacyHook: - """Init shows a lightweight privacy summary.""" - - def test_init_shows_privacy_summary(self, tmp_path: Path) -> None: - config_dir = tmp_path / ".openjarvis" - config_path = config_dir / "config.toml" - with ( - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), - mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), - mock.patch("openjarvis.cli.init_cmd.PrivacyScanner") as MockScanner, - ): - from openjarvis.cli.scan_cmd import ScanResult - instance = MockScanner.return_value - instance.run_quick.return_value = [ - ScanResult("FileVault", "ok", "FileVault enabled", "darwin"), - ] - result = CliRunner().invoke( - cli, ["init", "--engine", "llamacpp", "--no-download"] - ) - assert result.exit_code == 0 - assert "jarvis scan" in result.output -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `uv run pytest tests/cli/test_init_guidance.py::TestInitPrivacyHook -v` -Expected: FAIL — no `PrivacyScanner` import in init_cmd, no privacy output. - -- [ ] **Step 3: Update all existing init tests to mock PrivacyScanner** - -After adding the privacy hook to init, all existing init CLI tests will call real system commands (`fdesetup`, `pgrep`, etc.). Add `mock.patch("openjarvis.cli.init_cmd.PrivacyScanner")` to every existing init CLI test's context manager block in `tests/cli/test_init_guidance.py`. This includes: - -- `test_init_shows_next_steps` -- `test_init_output_shows_toml_sections_literally` -- `test_init_generates_minimal_by_default` -- `test_init_full_generates_verbose_config` -- `test_init_shows_download_prompt` -- `test_init_no_download_flag_skips_prompt` -- `test_init_no_model_shows_warning` -- `test_init_ollama_download_calls_ollama_pull` -- `test_init_vllm_shows_auto_download_message` - -For each, add inside the `with (...)` block: -```python -mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), -``` - -- [ ] **Step 4: Register scan command and add init privacy hook** - -In `src/openjarvis/cli/__init__.py`, add after line 33: - -```python -from openjarvis.cli.scan_cmd import scan -``` - -After line 89 (the last `cli.add_command` call), add: - -```python -cli.add_command(scan, "scan") -``` - -In `src/openjarvis/cli/init_cmd.py`, add import: - -```python -from openjarvis.cli.scan_cmd import PrivacyScanner -``` - -Add `_quick_privacy_check` function: - -```python -def _quick_privacy_check(console: Console) -> None: - """Run critical privacy checks and print compact summary.""" - scanner = PrivacyScanner() - results = scanner.run_quick() - - if results: - console.print(" [bold]Privacy check:[/bold]") - for r in results: - if r.status == "ok": - console.print(f" [green]\u2713[/green] {r.message}") - elif r.status == "warn": - console.print(f" [yellow]![/yellow] {r.message}") - elif r.status == "fail": - console.print(f" [red]\u2717[/red] {r.message}") - - console.print() - console.print(" Run [cyan]jarvis scan[/cyan] for a full environment audit.") -``` - -Call `_quick_privacy_check(console)` at the end of the `init()` function, just before the final "Getting Started" panel. - -- [ ] **Step 4: Run all tests to verify everything passes** - -Run: `uv run pytest tests/cli/test_init_guidance.py tests/cli/test_scan.py tests/core/test_recommend_model.py tests/cli/test_model_pull.py -v` -Expected: ALL tests pass. - -- [ ] **Step 5: Lint all changed files** - -Run: `uv run ruff check src/openjarvis/cli/ --fix` -Expected: Clean. - -- [ ] **Step 6: Commit** - -```bash -git add src/openjarvis/cli/__init__.py src/openjarvis/cli/init_cmd.py tests/cli/test_init_guidance.py -git commit -m "feat: register jarvis scan command and add init privacy hook - -Registers the new scan command in the CLI. Adds a lightweight -privacy check at the end of jarvis init that runs disk encryption -and cloud sync checks, with pointer to jarvis scan for full audit." -``` - ---- - -### Task 6: Final integration test and cleanup - -**Files:** -- All modified files from tasks 1-5 - -- [ ] **Step 1: Run full test suite for all touched modules** - -Run: `uv run pytest tests/core/test_recommend_model.py tests/cli/test_init_guidance.py tests/cli/test_model_pull.py tests/cli/test_scan.py -v` -Expected: ALL pass. - -- [ ] **Step 2: Run linter on all source files** - -Run: `uv run ruff check src/openjarvis/cli/ src/openjarvis/intelligence/model_catalog.py src/openjarvis/core/config.py` -Expected: Clean. - -- [ ] **Step 3: Verify the MLX bug is fixed** - -Run: `uv run python3 -c "from openjarvis.core.config import HardwareInfo, GpuInfo, recommend_model; hw = HardwareInfo(platform='darwin', ram_gb=16.0, gpu=GpuInfo(vendor='apple', name='Apple M1', vram_gb=16.0, count=1)); print(f'MLX model: {recommend_model(hw, \"mlx\")}')"` -Expected output: `MLX model: qwen3.5:14b` - -- [ ] **Step 4: Verify the CLI scan command is registered** - -Run: `uv run jarvis scan --help` -Expected: Shows help text for the scan command. - -- [ ] **Step 5: Commit if any cleanup was needed** - -Only if changes were made during cleanup. Otherwise skip. From cbf6fe3ea21f7fac32aed632a1a209a667a27442 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon Date: Wed, 25 Mar 2026 03:35:53 +0000 Subject: [PATCH 64/92] feat: add take-bot workflow and update contribution docs Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/take-assign.yml | 29 +++++++++++++++++++++++++++++ CONTRIBUTING.md | 5 ++++- docs/development/roadmap.md | 8 ++++---- 3 files changed, 37 insertions(+), 5 deletions(-) create mode 100644 .github/workflows/take-assign.yml diff --git a/.github/workflows/take-assign.yml b/.github/workflows/take-assign.yml new file mode 100644 index 00000000..7fa6d0dc --- /dev/null +++ b/.github/workflows/take-assign.yml @@ -0,0 +1,29 @@ +name: Auto-assign on "take" + +on: + issue_comment: + types: [created] + +permissions: + issues: write + +jobs: + assign: + if: >- + !github.event.issue.pull_request + && contains(fromJSON('["take", "Take", "TAKE"]'), trim(github.event.comment.body)) + runs-on: ubuntu-latest + steps: + - name: Assign commenter + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + COMMENTER="${{ github.event.comment.user.login }}" + ISSUE="${{ github.event.issue.number }}" + CURRENT=$(gh issue view "$ISSUE" --repo "${{ github.repository }}" --json assignees --jq '.assignees[].login' 2>/dev/null) + if echo "$CURRENT" | grep -qx "$COMMENTER"; then + echo "$COMMENTER is already assigned to #$ISSUE" + else + gh issue edit "$ISSUE" --repo "${{ github.repository }}" --add-assignee "$COMMENTER" + echo "Assigned $COMMENTER to #$ISSUE" + fi diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1a953370..816caca1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -104,7 +104,10 @@ For detailed development setup, code conventions, and project structure, see the ## Claiming Issues -Want to work on an issue? Comment **"take"** on the issue to claim it. This lets others know you're working on it and prevents duplicate effort. +1. Browse the [Roadmap](https://open-jarvis.github.io/OpenJarvis/development/roadmap/) for an item that interests you +2. Check if a [GitHub issue](https://github.com/open-jarvis/OpenJarvis/issues) already exists for it — if not, [open one](https://github.com/open-jarvis/OpenJarvis/issues/new/choose) describing what you'd like to work on +3. Comment **"take"** on the issue to get auto-assigned +4. Fork, branch, and start working If you've claimed an issue but can't finish it, please leave a comment so someone else can pick it up. diff --git a/docs/development/roadmap.md b/docs/development/roadmap.md index cb5cbb23..6719d410 100644 --- a/docs/development/roadmap.md +++ b/docs/development/roadmap.md @@ -14,10 +14,10 @@ These are the areas where active development is happening and contributions are ## How to Get Involved -1. Browse the workstreams below for items that interest you -2. Look for **Ready** items and **good first issue** tags -3. Find the matching [GitHub issue](https://github.com/open-jarvis/OpenJarvis/issues) and comment **"take"** to claim it — if no issue exists yet, [open one](https://github.com/open-jarvis/OpenJarvis/issues/new/choose) describing what you'd like to work on -4. Read the [Contributing Guide](https://github.com/open-jarvis/OpenJarvis/blob/main/CONTRIBUTING.md) for the full development setup, PR process, and code conventions +1. Browse the workstreams below for an item that interests you +2. Check if a [GitHub issue](https://github.com/open-jarvis/OpenJarvis/issues) already exists for it — if not, [open one](https://github.com/open-jarvis/OpenJarvis/issues/new/choose) +3. Comment **"take"** on the issue to get auto-assigned +4. Read the [Contributing Guide](https://github.com/open-jarvis/OpenJarvis/blob/main/CONTRIBUTING.md) for development setup and PR process --- From b2152fb40e0209839b53e1cff3260498aa632086 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Tue, 24 Mar 2026 20:38:14 -0700 Subject: [PATCH 65/92] style: fix E501 line-length and unused imports in test files Wraps long CliRunner.invoke() calls, removes unused pytest imports, fixes import ordering, and removes unused variables in test_scan.py. Co-Authored-By: Claude Opus 4.6 (1M context) --- tests/cli/test_init_guidance.py | 60 ++++++++++---- tests/cli/test_model_pull.py | 2 +- tests/cli/test_scan.py | 139 +++++++++++++++++++------------- 3 files changed, 129 insertions(+), 72 deletions(-) diff --git a/tests/cli/test_init_guidance.py b/tests/cli/test_init_guidance.py index 74924c53..acca3c0c 100644 --- a/tests/cli/test_init_guidance.py +++ b/tests/cli/test_init_guidance.py @@ -10,6 +10,8 @@ from click.testing import CliRunner from openjarvis.cli import cli from openjarvis.cli.init_cmd import _next_steps_text +_NO_DL = "--no-download" + class TestInitShowsNextSteps: def test_init_shows_next_steps(self, tmp_path: Path) -> None: @@ -21,7 +23,9 @@ class TestInitShowsNextSteps: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp", "--no-download"]) + result = CliRunner().invoke( + cli, ["init", "--engine", "llamacpp", _NO_DL] + ) assert result.exit_code == 0 assert "Getting Started" in result.output assert "jarvis ask" in result.output @@ -36,7 +40,9 @@ class TestInitShowsNextSteps: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp", "--no-download"]) + result = CliRunner().invoke( + cli, ["init", "--engine", "llamacpp", _NO_DL] + ) assert result.exit_code == 0 assert "[engine]" in result.output assert "[intelligence]" in result.output @@ -96,7 +102,9 @@ class TestMinimalConfig: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "ollama", "--no-download"]) + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama", _NO_DL] + ) assert result.exit_code == 0 content = config_path.read_text() # Minimal config should be short @@ -114,7 +122,10 @@ class TestMinimalConfig: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--full", "--engine", "ollama", "--no-download"]) + result = CliRunner().invoke( + cli, + ["init", "--full", "--engine", "ollama", _NO_DL], + ) assert result.exit_code == 0 content = config_path.read_text() # Full config should have many sections @@ -132,9 +143,12 @@ class TestInitDownloadPrompt: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "ollama"], input="n\n") + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama"], input="n\n" + ) assert result.exit_code == 0 - assert "Download" in result.output and "now?" in result.output + assert "Download" in result.output + assert "now?" in result.output def test_init_no_download_flag_skips_prompt(self, tmp_path: Path) -> None: config_dir = tmp_path / ".openjarvis" @@ -144,9 +158,11 @@ class TestInitDownloadPrompt: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "ollama", "--no-download"]) + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama", _NO_DL] + ) assert result.exit_code == 0 - assert "Download" not in result.output or "now?" not in result.output + assert "Download" not in result.output class TestInitEmptyModelFallback: @@ -159,9 +175,14 @@ class TestInitEmptyModelFallback: mock.patch("openjarvis.cli.init_cmd.recommend_model", return_value=""), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "llamacpp"]) + result = CliRunner().invoke( + cli, ["init", "--engine", "llamacpp"] + ) assert result.exit_code == 0 - assert "Not enough memory" in result.output or "not enough memory" in result.output + assert ( + "Not enough memory" in result.output + or "not enough memory" in result.output + ) class TestNextStepsExoNexa: @@ -179,16 +200,23 @@ class TestNextStepsExoNexa: class TestInitDownloadDispatch: - def test_init_ollama_download_calls_ollama_pull(self, tmp_path: Path) -> None: + def test_init_ollama_download_calls_ollama_pull( + self, tmp_path: Path + ) -> None: config_dir = tmp_path / ".openjarvis" config_path = config_dir / "config.toml" with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), - mock.patch("openjarvis.cli.init_cmd.ollama_pull", return_value=True) as mock_pull, + mock.patch( + "openjarvis.cli.init_cmd.ollama_pull", + return_value=True, + ) as mock_pull, mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "ollama"], input="y\n") + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama"], input="y\n" + ) assert result.exit_code == 0 mock_pull.assert_called_once() @@ -200,7 +228,9 @@ class TestInitDownloadDispatch: mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "vllm"], input="y\n") + result = CliRunner().invoke( + cli, ["init", "--engine", "vllm"], input="y\n" + ) assert result.exit_code == 0 assert "automatically" in result.output @@ -220,7 +250,7 @@ class TestInitPrivacyHook: ScanResult("FileVault", "ok", "FileVault enabled", "darwin"), ] result = CliRunner().invoke( - cli, ["init", "--engine", "llamacpp", "--no-download"] + cli, ["init", "--engine", "llamacpp", _NO_DL] ) assert result.exit_code == 0 assert "jarvis scan" in result.output diff --git a/tests/cli/test_model_pull.py b/tests/cli/test_model_pull.py index a87a55ae..3a724950 100644 --- a/tests/cli/test_model_pull.py +++ b/tests/cli/test_model_pull.py @@ -4,7 +4,6 @@ from __future__ import annotations from unittest import mock -import pytest from click.testing import CliRunner from rich.console import Console @@ -34,6 +33,7 @@ class TestOllamaPull: def test_ollama_pull_connect_error(self) -> None: import io + import httpx console = Console(file=io.StringIO()) diff --git a/tests/cli/test_scan.py b/tests/cli/test_scan.py index c89760f3..536439d8 100644 --- a/tests/cli/test_scan.py +++ b/tests/cli/test_scan.py @@ -3,21 +3,25 @@ from __future__ import annotations import json +import sys from subprocess import CompletedProcess from unittest.mock import MagicMock, patch -import pytest - -from openjarvis.cli.scan_cmd import ScanResult, PrivacyScanner - +from openjarvis.cli.scan_cmd import PrivacyScanner, ScanResult # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- -def _make_proc(stdout: str = "", stderr: str = "", returncode: int = 0) -> CompletedProcess: - return CompletedProcess(args=[], returncode=returncode, stdout=stdout, stderr=stderr) +def _make_proc( + stdout: str = "", + stderr: str = "", + returncode: int = 0, +) -> CompletedProcess: + return CompletedProcess( + args=[], returncode=returncode, stdout=stdout, stderr=stderr + ) # --------------------------------------------------------------------------- @@ -27,7 +31,9 @@ def _make_proc(stdout: str = "", stderr: str = "", returncode: int = 0) -> Compl class TestScanResultDataclass: def test_fields_exist(self) -> None: - r = ScanResult(name="test", status="ok", message="all good", platform="all") + r = ScanResult( + name="test", status="ok", message="all good", platform="all" + ) assert r.name == "test" assert r.status == "ok" assert r.message == "all good" @@ -52,13 +58,15 @@ class TestScanResultDataclass: class TestFileVault: def test_filevault_enabled(self) -> None: scanner = PrivacyScanner() - with patch("subprocess.run", return_value=_make_proc(stdout="FileVault is On.")): + proc = _make_proc(stdout="FileVault is On.") + with patch("subprocess.run", return_value=proc): result = scanner.check_filevault() assert result.status == "ok" def test_filevault_disabled(self) -> None: scanner = PrivacyScanner() - with patch("subprocess.run", return_value=_make_proc(stdout="FileVault is Off.")): + proc = _make_proc(stdout="FileVault is Off.") + with patch("subprocess.run", return_value=proc): result = scanner.check_filevault() assert result.status == "fail" @@ -77,18 +85,20 @@ class TestFileVault: class TestMDM: def test_not_enrolled(self) -> None: scanner = PrivacyScanner() + stdout = "Enrolled via DEP: No\nMDM enrollment: not enrolled" with patch( "subprocess.run", - return_value=_make_proc(stdout="Enrolled via DEP: No\nMDM enrollment: not enrolled"), + return_value=_make_proc(stdout=stdout), ): result = scanner.check_mdm() assert result.status == "ok" def test_enrolled(self) -> None: scanner = PrivacyScanner() + stdout = "MDM enrollment: Yes\nEnrolled via DEP: Yes" with patch( "subprocess.run", - return_value=_make_proc(stdout="MDM enrollment: Yes\nEnrolled via DEP: Yes"), + return_value=_make_proc(stdout=stdout), ): result = scanner.check_mdm() assert result.status == "warn" @@ -101,9 +111,12 @@ class TestMDM: class TestCloudSync: def test_no_agents(self) -> None: - """pgrep returns non-zero (process not found) → no sync agents running.""" + """pgrep returns non-zero → no sync agents running.""" scanner = PrivacyScanner() - with patch("subprocess.run", return_value=_make_proc(returncode=1, stdout="")): + with patch( + "subprocess.run", + return_value=_make_proc(returncode=1, stdout=""), + ): result = scanner.check_cloud_sync_agents() assert result.status == "ok" @@ -128,19 +141,18 @@ class TestCloudSync: class TestNetworkExposure: def test_no_exposed_ports(self) -> None: - """Only localhost bindings → ok.""" + """Only localhost bindings -> ok.""" scanner = PrivacyScanner() - # lsof / ss output showing only 127.0.0.1 - lsof_output = "ollama 1234 user IPv4 TCP 127.0.0.1:11434 (LISTEN)\n" - with patch("subprocess.run", return_value=_make_proc(stdout=lsof_output)): + lsof_out = "ollama 1234 user IPv4 TCP 127.0.0.1:11434 (LISTEN)\n" + with patch("subprocess.run", return_value=_make_proc(stdout=lsof_out)): result = scanner.check_network_exposure() assert result.status == "ok" def test_exposed_port(self) -> None: - """A port bound to 0.0.0.0 or * → warn with port in message.""" + """A port bound to * -> warn with port in message.""" scanner = PrivacyScanner() - lsof_output = "ollama 1234 user IPv4 TCP *:11434 (LISTEN)\n" - with patch("subprocess.run", return_value=_make_proc(stdout=lsof_output)): + lsof_out = "ollama 1234 user IPv4 TCP *:11434 (LISTEN)\n" + with patch("subprocess.run", return_value=_make_proc(stdout=lsof_out)): result = scanner.check_network_exposure() assert result.status == "warn" assert "11434" in result.message @@ -153,7 +165,7 @@ class TestNetworkExposure: class TestLUKS: def test_encrypted(self) -> None: - """lsblk JSON contains crypto_LUKS → ok.""" + """lsblk JSON contains crypto_LUKS -> ok.""" scanner = PrivacyScanner() lsblk_data = { "blockdevices": [ @@ -175,7 +187,7 @@ class TestLUKS: assert result.status == "ok" def test_not_encrypted(self) -> None: - """lsblk JSON has only ext4 → fail.""" + """lsblk JSON has only ext4 -> fail.""" scanner = PrivacyScanner() lsblk_data = { "blockdevices": [ @@ -198,14 +210,17 @@ class TestLUKS: class TestScreenRecording: def test_none_running(self) -> None: - """No screen-recording processes → ok.""" + """No screen-recording processes -> ok.""" scanner = PrivacyScanner() - with patch("subprocess.run", return_value=_make_proc(returncode=1, stdout="")): + with patch( + "subprocess.run", + return_value=_make_proc(returncode=1, stdout=""), + ): result = scanner.check_screen_recording() assert result.status == "ok" def test_teamviewer_running(self) -> None: - """TeamViewer found → warn.""" + """TeamViewer found -> warn.""" scanner = PrivacyScanner() def _pgrep_side_effect(cmd, **kwargs): @@ -225,44 +240,44 @@ class TestScreenRecording: class TestPlatformFiltering: def test_run_all_filters_to_current_platform(self) -> None: - """run_all() should only call checks tagged 'all' or current platform.""" + """run_all() returns only current-platform + 'all' checks.""" scanner = PrivacyScanner() - called_platforms: list[str] = [] - - # Patch every check method to record its platform tag instead of running - checks = scanner._get_all_checks() - for check_fn in checks: - # Run the real method but capture which ones get included - pass - - import sys - current_plat = "darwin" if sys.platform == "darwin" else "linux" other_plat = "linux" if current_plat == "darwin" else "darwin" - # Make every check return immediately with a known platform with patch.object(scanner, "_get_all_checks") as mock_get: - darwin_check = MagicMock(return_value=ScanResult("d", "ok", "msg", "darwin")) - linux_check = MagicMock(return_value=ScanResult("l", "ok", "msg", "linux")) - all_check = MagicMock(return_value=ScanResult("a", "ok", "msg", "all")) + darwin_ck = MagicMock( + return_value=ScanResult("d", "ok", "msg", "darwin") + ) + linux_ck = MagicMock( + return_value=ScanResult("l", "ok", "msg", "linux") + ) + all_ck = MagicMock( + return_value=ScanResult("a", "ok", "msg", "all") + ) - mock_get.return_value = [darwin_check, linux_check, all_check] + mock_get.return_value = [darwin_ck, linux_ck, all_ck] results = scanner.run_all() - # The check for the other platform should not appear in results - other_result_platform = "linux" if current_plat == "darwin" else "darwin" result_platforms = {r.platform for r in results} - assert other_result_platform not in result_platforms + assert other_plat not in result_platforms assert "all" in result_platforms or current_plat in result_platforms +# --------------------------------------------------------------------------- +# TestRemoteAccess +# --------------------------------------------------------------------------- + + class TestRemoteAccess: """Tests for check_remote_access (Linux).""" def test_no_remote_access(self) -> None: scanner = PrivacyScanner() with patch.object(scanner, "_run") as mock_run: - mock_run.return_value = CompletedProcess([], 1, stdout="", stderr="") + mock_run.return_value = CompletedProcess( + [], 1, stdout="", stderr="" + ) result = scanner.check_remote_access() assert result.status == "ok" @@ -271,13 +286,22 @@ class TestRemoteAccess: with patch.object(scanner, "_run") as mock_run: def side_effect(cmd, **kw): if any("xrdp" in str(c) for c in cmd): - return CompletedProcess(cmd, 0, stdout="12345", stderr="") - return CompletedProcess(cmd, 1, stdout="", stderr="") + return CompletedProcess( + cmd, 0, stdout="12345", stderr="" + ) + return CompletedProcess( + cmd, 1, stdout="", stderr="" + ) mock_run.side_effect = side_effect result = scanner.check_remote_access() assert result.status == "warn" +# --------------------------------------------------------------------------- +# TestICloudSync +# --------------------------------------------------------------------------- + + class TestICloudSync: """Tests for check_icloud_sync (macOS).""" @@ -292,32 +316,35 @@ class TestICloudSync: def test_icloud_defaults_error_falls_through_to_ok(self) -> None: scanner = PrivacyScanner() - # When _run raises, the nested try/except catches it and falls to ok with patch.object(scanner, "_run", side_effect=FileNotFoundError): result = scanner.check_icloud_sync() assert result.status == "ok" +# --------------------------------------------------------------------------- +# TestRunQuick +# --------------------------------------------------------------------------- + + class TestRunQuick: """Tests for run_quick subset.""" def test_run_quick_returns_subset(self) -> None: scanner = PrivacyScanner() - with patch.object(scanner, "check_filevault") as fv, \ - patch.object(scanner, "check_luks") as luks, \ - patch.object(scanner, "check_icloud_sync") as ic, \ - patch.object(scanner, "check_cloud_sync_agents") as cs: - import sys - plat = sys.platform + plat = sys.platform + with ( + patch.object(scanner, "check_filevault") as fv, + patch.object(scanner, "check_luks") as luks, + patch.object(scanner, "check_icloud_sync") as ic, + patch.object(scanner, "check_cloud_sync_agents") as cs, + ): fv.return_value = ScanResult("FV", "ok", "ok", "darwin") luks.return_value = ScanResult("LUKS", "ok", "ok", "linux") ic.return_value = ScanResult("iCloud", "ok", "ok", "darwin") cs.return_value = ScanResult("Cloud", "ok", "ok", plat) results = scanner.run_quick() - # Should only contain current platform results for r in results: assert r.platform in (plat, "all") - # Should not contain network or screen recording names = {r.name for r in results} assert "Network Exposure" not in names assert "Screen Recording" not in names From 68a8f5646c7614837ca9ecbce2aff4b62821f1b9 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 08:07:35 -0700 Subject: [PATCH 66/92] fix: add --no-download and PrivacyScanner mock to test_cli init test The test_init_creates_config test in test_cli.py was not updated when the download prompt and privacy hook were added to jarvis init. Co-Authored-By: Claude Opus 4.6 (1M context) --- tests/cli/test_cli.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py index ce9e7f56..c5d1ac16 100644 --- a/tests/cli/test_cli.py +++ b/tests/cli/test_cli.py @@ -80,8 +80,11 @@ class TestCLI: with ( mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), ): - result = CliRunner().invoke(cli, ["init", "--engine", "ollama"]) + result = CliRunner().invoke( + cli, ["init", "--engine", "ollama", "--no-download"] + ) assert result.exit_code == 0 assert config_path.exists() content = config_path.read_text() From 36554ab0d1237b623f794c4b4c9c1a02996379d1 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon Date: Wed, 25 Mar 2026 15:46:12 +0000 Subject: [PATCH 67/92] docs: consolidate README sections and add Contributing - Merge Quick Start, Docker, and Development into single Quick Start - Point to docs site for full documentation (Docker, cloud, dev setup) - Add Contributing section with dev setup and roadmap pointer Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 64 +++++++++++-------------------------------------------- 1 file changed, 13 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index 54e79dcd..84fd6c65 100644 --- a/README.md +++ b/README.md @@ -46,79 +46,41 @@ You also need a local inference backend: [Ollama](https://ollama.com), [vLLM](ht ## Quick Start -The fastest path is Ollama on any machine with Python 3.10+: - ```bash -# 1. Install OpenJarvis +# 1. Install and detect hardware git clone https://github.com/open-jarvis/OpenJarvis.git cd OpenJarvis uv sync - -# 2. Detect hardware and generate config uv run jarvis init -# 3. Install and start Ollama (https://ollama.com) +# 2. Start Ollama and pull a model curl -fsSL https://ollama.com/install.sh | sh -ollama serve # start the Ollama server - -# 4. Pull a model +ollama serve & ollama pull qwen3:8b -# 5. Ask a question +# 3. Ask a question uv run jarvis ask "What is the capital of France?" - -# 6. Verify your setup -uv run jarvis doctor ``` -`jarvis init` auto-detects your hardware and recommends the best engine. After init, it prints engine-specific next steps. Run `uv run jarvis doctor` at any time to diagnose configuration or connectivity issues. +`jarvis init` auto-detects your hardware and recommends the best engine. Run `uv run jarvis doctor` at any time to diagnose issues. -## Docker +Full documentation — including Docker deployment, cloud engines, development setup, and tutorials — at **[open-jarvis.github.io/OpenJarvis](https://open-jarvis.github.io/OpenJarvis/)**. -A Docker Compose configuration bundles Jarvis with Ollama: +## Contributing + +We welcome contributions! See the [Contributing Guide](CONTRIBUTING.md) for incentives, contribution types, and the PR process. + +Quick start for contributors: ```bash -# CPU-only (default) -docker compose -f deploy/docker/docker-compose.yml up -d - -# NVIDIA GPU (requires NVIDIA Container Toolkit) -docker compose -f deploy/docker/docker-compose.yml \ - -f deploy/docker/docker-compose.gpu.nvidia.yml up -d - -# AMD GPU (requires ROCm) -docker compose -f deploy/docker/docker-compose.yml \ - -f deploy/docker/docker-compose.gpu.rocm.yml up -d - -# Pull a model into Ollama -docker compose -f deploy/docker/docker-compose.yml exec ollama ollama pull qwen3:8b -``` - -Services: Jarvis on `:8000`, Ollama on `:11434`. - -## Development - -From source, you need to make sure Rust is installed on System: - -```bash -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -``` - -Then, you need the Rust extension for full functionality (security, tools, agents, etc.): - -```bash -# 1. Clone and install Python deps git clone https://github.com/open-jarvis/OpenJarvis.git cd OpenJarvis uv sync --extra dev - -# 2. Build and install the Rust extension (requires Rust toolchain) -uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml - -# 3. Run tests +uv run pre-commit install uv run pytest tests/ -v ``` -See [Contributing](docs/development/contributing.md) for more. +Browse the [Roadmap](https://open-jarvis.github.io/OpenJarvis/development/roadmap/) for areas where help is needed. Comment **"take"** on any issue to get auto-assigned. ## About From c5cb6a9e491d064447be793f22151ef87ee75f18 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:20:25 -0700 Subject: [PATCH 68/92] feat(evals): add tool_calls field to TurnTrace for rich tool data Adds a `tool_calls` list field to `TurnTrace` capturing name, arguments, and result for each tool invocation, enabling PinchBench transcript integration. Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/evals/core/trace.py | 4 +++ tests/evals/test_pinchbench_transcript.py | 32 +++++++++++++++++++++++ 2 files changed, 36 insertions(+) create mode 100644 tests/evals/test_pinchbench_transcript.py diff --git a/src/openjarvis/evals/core/trace.py b/src/openjarvis/evals/core/trace.py index 92b1f5ea..356f593b 100644 --- a/src/openjarvis/evals/core/trace.py +++ b/src/openjarvis/evals/core/trace.py @@ -28,6 +28,8 @@ class TurnTrace: cost_usd: Optional[float] = None # Per-action energy breakdown (lm_inference vs tool_call granularity) action_energy_breakdown: Optional[List[Dict[str, Any]]] = None + # Detailed tool call data: [{"name": str, "arguments": dict, "result": str}] + tool_calls: List[Dict[str, Any]] = field(default_factory=list) def to_dict(self) -> Dict[str, Any]: return { @@ -45,6 +47,7 @@ class TurnTrace: "cpu_power_avg_watts": self.cpu_power_avg_watts, "cost_usd": self.cost_usd, "action_energy_breakdown": self.action_energy_breakdown, + "tool_calls": [dict(tc) for tc in self.tool_calls], } @classmethod @@ -64,6 +67,7 @@ class TurnTrace: cpu_power_avg_watts=d.get("cpu_power_avg_watts"), cost_usd=d.get("cost_usd"), action_energy_breakdown=d.get("action_energy_breakdown"), + tool_calls=d.get("tool_calls", []), ) diff --git a/tests/evals/test_pinchbench_transcript.py b/tests/evals/test_pinchbench_transcript.py new file mode 100644 index 00000000..7218216b --- /dev/null +++ b/tests/evals/test_pinchbench_transcript.py @@ -0,0 +1,32 @@ +"""Tests for TurnTrace tool_calls field and PinchBench transcript translation.""" + +from openjarvis.evals.core.trace import TurnTrace + + +def test_turn_trace_tool_calls_default_empty(): + """New tool_calls field defaults to empty list.""" + turn = TurnTrace(turn_index=0) + assert turn.tool_calls == [] + + +def test_turn_trace_tool_calls_to_dict(): + """tool_calls round-trips through to_dict/from_dict.""" + calls = [{"name": "file_read", "arguments": {"path": "a.txt"}, "result": "hello"}] + turn = TurnTrace(turn_index=0, tool_calls=calls) + d = turn.to_dict() + assert d["tool_calls"] == calls + + +def test_turn_trace_tool_calls_from_dict(): + """from_dict restores tool_calls.""" + calls = [{"name": "web_search", "arguments": {"q": "test"}, "result": "results"}] + d = {"turn_index": 0, "tool_calls": calls} + turn = TurnTrace.from_dict(d) + assert turn.tool_calls == calls + + +def test_turn_trace_tool_calls_from_dict_missing(): + """from_dict with missing tool_calls defaults to empty list.""" + d = {"turn_index": 0} + turn = TurnTrace.from_dict(d) + assert turn.tool_calls == [] From e0ff20c47356f1e9086230a74340d9e6df4cbaab Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:22:13 -0700 Subject: [PATCH 69/92] feat(tools): include result text in TOOL_CALL_END event metadata Adds a truncated (10KB max) string representation of the tool result's content to the TOOL_CALL_END event payload so grading functions (e.g. PinchBench) can inspect what each tool returned. Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/tools/_stubs.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/openjarvis/tools/_stubs.py b/src/openjarvis/tools/_stubs.py index 7dd72c08..73a15b27 100644 --- a/src/openjarvis/tools/_stubs.py +++ b/src/openjarvis/tools/_stubs.py @@ -255,12 +255,14 @@ class ToolExecutor: # Emit end event if self._bus: + result_text = str(result.content)[:10240] if result.content else "" self._bus.publish( EventType.TOOL_CALL_END, { "tool": tool_call.name, "success": result.success, "latency": latency, + "result": result_text, }, ) From ae0262da82465767664572adcc2a2d31dd9d5430 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:24:26 -0700 Subject: [PATCH 70/92] feat(evals): bridge EventBus to EventRecorder, capture tool_calls in traces Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/evals/core/agentic_runner.py | 43 +++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/src/openjarvis/evals/core/agentic_runner.py b/src/openjarvis/evals/core/agentic_runner.py index 097f74ab..ee272240 100644 --- a/src/openjarvis/evals/core/agentic_runner.py +++ b/src/openjarvis/evals/core/agentic_runner.py @@ -321,6 +321,18 @@ class AgenticRunner: event_recorder.clear() + # Bridge EventBus → EventRecorder so tool events are captured. + # ToolExecutor publishes to JarvisSystem.bus; we relay those into + # the EventRecorder for transcript building and trace enrichment. + _bus_unsubs: list[tuple] = [] + agent_bus = getattr(agent, "bus", None) + if agent_bus is not None: + for etype in (EventType.TOOL_CALL_START, EventType.TOOL_CALL_END): + def _relay(event, _etype=etype): + event_recorder.record(_etype, **(event.data if hasattr(event, "data") else {})) + agent_bus.subscribe(etype, _relay) + _bus_unsubs.append((etype, _relay)) + # Set up per-query workspace if self._run_dir and hasattr(agent, "set_workspace"): instance_id = record.metadata.get("instance_id", record.record_id) @@ -397,6 +409,10 @@ class AgenticRunner: else: response_text = str(agent(record.problem)) + # Wire event recorder to task env if supported + if task_env is not None and hasattr(task_env, "set_event_recorder"): + task_env.set_event_recorder(event_recorder) + # Run tests if task env supports it if task_env is not None and hasattr(task_env, "run_tests"): task_env.run_tests() @@ -425,6 +441,13 @@ class AgenticRunner: except Exception as exc: LOGGER.warning("Agent failed on query %s: %s", query_id, exc) end_time = time.time() + # Unsubscribe EventBus relays + if agent_bus is not None: + for etype, cb in _bus_unsubs: + try: + agent_bus.unsubscribe(etype, cb) + except Exception: + pass return QueryTrace( query_id=query_id, workload_type=str(workload_type), @@ -435,6 +458,14 @@ class AgenticRunner: is_resolved=record.metadata.get("is_resolved"), ) + # Unsubscribe EventBus relays + if agent_bus is not None: + for etype, cb in _bus_unsubs: + try: + agent_bus.unsubscribe(etype, cb) + except Exception: + pass + end_time = time.time() end_ns = time.monotonic_ns() @@ -578,6 +609,8 @@ class AgenticRunner: current_turn_start: Optional[float] = None current_tools: list[str] = [] current_tool_latencies: dict[str, float] = {} + current_tool_calls: list[dict[str, Any]] = [] + current_tool_args: dict[str, Any] = {} tool_start_times: dict[str, float] = {} input_tokens = 0 output_tokens = 0 @@ -624,6 +657,7 @@ class AgenticRunner: output_tokens=output_tokens, tools_called=list(current_tools), tool_latencies_s=dict(current_tool_latencies), + tool_calls=list(current_tool_calls), wall_clock_s=wall_clock, action_energy_breakdown=action_breakdown, ) @@ -633,6 +667,8 @@ class AgenticRunner: current_turn_start = None current_tools = [] current_tool_latencies = {} + current_tool_calls = [] + current_tool_args = {} current_action_spans = [] input_tokens = 0 output_tokens = 0 @@ -640,10 +676,16 @@ class AgenticRunner: elif etype == EventType.TOOL_CALL_START: tool_name = event.metadata.get("tool", "unknown") tool_start_times[tool_name] = event.timestamp + current_tool_args[tool_name] = event.metadata.get("arguments", {}) elif etype == EventType.TOOL_CALL_END: tool_name = event.metadata.get("tool", "unknown") current_tools.append(tool_name) + current_tool_calls.append({ + "name": tool_name, + "arguments": current_tool_args.pop(tool_name, {}), + "result": event.metadata.get("result", ""), + }) start_ts = tool_start_times.pop(tool_name, None) if start_ts is not None: duration = event.timestamp - start_ts @@ -662,6 +704,7 @@ class AgenticRunner: turn_index=0, tools_called=current_tools, tool_latencies_s=current_tool_latencies, + tool_calls=list(current_tool_calls), ) ) From 8e8d4a68e4e6c5691b6803dd8f493596e75b061c Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:25:39 -0700 Subject: [PATCH 71/92] =?UTF-8?q?feat(evals):=20add=20PinchBench=20dataset?= =?UTF-8?q?=20provider=20=E2=80=94=20repo=20clone=20and=20task=20markdown?= =?UTF-8?q?=20parsing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/evals/datasets/pinchbench.py | 202 ++++++++++++++++++++ tests/evals/test_pinchbench_dataset.py | 130 +++++++++++++ 2 files changed, 332 insertions(+) create mode 100644 src/openjarvis/evals/datasets/pinchbench.py create mode 100644 tests/evals/test_pinchbench_dataset.py diff --git a/src/openjarvis/evals/datasets/pinchbench.py b/src/openjarvis/evals/datasets/pinchbench.py new file mode 100644 index 00000000..fdcb8fd0 --- /dev/null +++ b/src/openjarvis/evals/datasets/pinchbench.py @@ -0,0 +1,202 @@ +"""PinchBench dataset provider — real-world agent task benchmark. + +Clones the pinchbench/skill repo at runtime and parses task markdown files +into EvalRecords for use with AgenticRunner. + +Reference: https://github.com/pinchbench/skill +""" + +from __future__ import annotations + +import logging +import random +import re +import shutil +import subprocess +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional + +import yaml + +from openjarvis.evals.core.dataset import DatasetProvider +from openjarvis.evals.core.types import EvalRecord + +LOGGER = logging.getLogger(__name__) + +PINCHBENCH_REPO = "https://github.com/pinchbench/skill.git" +CACHE_DIR = Path.home() / ".cache" / "pinchbench" + + +def _parse_task_markdown(content: str, filename: str = "") -> Dict[str, Any]: + """Parse a PinchBench task markdown file into a dict. + + Extracts YAML frontmatter and markdown sections (## Prompt, + ## Expected Behavior, ## Automated Checks, ## LLM Judge Rubric). + """ + # Split frontmatter + parts = content.split("---", 2) + if len(parts) < 3: + raise ValueError(f"Missing YAML frontmatter in {filename}") + + frontmatter = yaml.safe_load(parts[1]) + body = parts[2] + + # Parse sections by ## headers + sections: Dict[str, str] = {} + current_header: Optional[str] = None + current_lines: List[str] = [] + + for line in body.split("\n"): + header_match = re.match(r"^##\s+(.+)$", line) + if header_match: + if current_header is not None: + sections[current_header] = "\n".join(current_lines).strip() + current_header = header_match.group(1).strip() + current_lines = [] + else: + current_lines.append(line) + + if current_header is not None: + sections[current_header] = "\n".join(current_lines).strip() + + # Extract Python code block from Automated Checks section + automated_checks = None + checks_section = sections.get("Automated Checks", "") + code_match = re.search(r"```python\s*\n(.*?)```", checks_section, re.DOTALL) + if code_match: + automated_checks = code_match.group(1).strip() + + return { + "id": frontmatter.get("id", ""), + "name": frontmatter.get("name", ""), + "category": frontmatter.get("category", ""), + "grading_type": frontmatter.get("grading_type", "automated"), + "timeout_seconds": frontmatter.get("timeout_seconds", 180), + "workspace_files": frontmatter.get("workspace_files", []), + "grading_weights": frontmatter.get("grading_weights"), + "prompt": sections.get("Prompt", ""), + "expected_behavior": sections.get("Expected Behavior", ""), + "grading_criteria": sections.get("Grading Criteria", ""), + "automated_checks": automated_checks, + "llm_judge_rubric": sections.get("LLM Judge Rubric"), + } + + +class PinchBenchDataset(DatasetProvider): + """PinchBench real-world agent benchmark. + + Clones pinchbench/skill from GitHub (or uses a local path) and + parses task markdown files into EvalRecords. + """ + + dataset_id = "pinchbench" + dataset_name = "PinchBench" + + def __init__(self, path: Optional[str] = None) -> None: + self._local_path = Path(path) if path else None + self._repo_dir: Path = self._local_path or CACHE_DIR + self._records: List[EvalRecord] = [] + + def verify_requirements(self) -> List[str]: + issues: List[str] = [] + if self._local_path is None and shutil.which("git") is None: + issues.append("git binary not found. Install git to clone PinchBench tasks.") + if self._repo_dir.exists() and not (self._repo_dir / "tasks").is_dir(): + issues.append( + f"PinchBench cache at {self._repo_dir} is corrupted (missing tasks/). " + "Delete and re-run to re-clone." + ) + return issues + + def _ensure_repo(self) -> Path: + """Clone the repo if not already cached. Returns repo dir.""" + if self._local_path is not None: + if not self._local_path.exists(): + raise FileNotFoundError(f"PinchBench path not found: {self._local_path}") + return self._local_path + + if not self._repo_dir.exists(): + LOGGER.info("Cloning PinchBench from %s ...", PINCHBENCH_REPO) + self._repo_dir.parent.mkdir(parents=True, exist_ok=True) + subprocess.run( + ["git", "clone", "--depth", "1", PINCHBENCH_REPO, str(self._repo_dir)], + check=True, + capture_output=True, + ) + LOGGER.info("PinchBench cloned to %s", self._repo_dir) + + return self._repo_dir + + def load( + self, + *, + max_samples: Optional[int] = None, + split: Optional[str] = None, + seed: Optional[int] = None, + ) -> None: + repo_dir = self._ensure_repo() + tasks_dir = repo_dir / "tasks" + + if not tasks_dir.is_dir(): + raise FileNotFoundError(f"No tasks/ directory in {repo_dir}") + + task_files = sorted(tasks_dir.glob("task_*.md")) + if not task_files: + raise FileNotFoundError(f"No task_*.md files in {tasks_dir}") + + tasks = [] + for tf in task_files: + try: + parsed = _parse_task_markdown(tf.read_text(), filename=tf.name) + tasks.append(parsed) + except Exception as exc: + LOGGER.warning("Skipping %s: %s", tf.name, exc) + + if seed is not None: + random.Random(seed).shuffle(tasks) + if max_samples is not None: + tasks = tasks[:max_samples] + + self._records = [ + EvalRecord( + record_id=t["id"], + problem=t["prompt"], + reference=t["expected_behavior"], + category=t["category"], + subject=t["name"], + metadata={ + "grading_type": t["grading_type"], + "grading_weights": t["grading_weights"], + "automated_checks": t["automated_checks"], + "llm_judge_rubric": t["llm_judge_rubric"], + "timeout_seconds": t["timeout_seconds"], + "workspace_files": t["workspace_files"], + "pinchbench_repo_dir": str(repo_dir), + }, + ) + for t in tasks + ] + + LOGGER.info("PinchBench: loaded %d tasks", len(self._records)) + + def iter_records(self) -> Iterable[EvalRecord]: + return iter(self._records) + + def size(self) -> int: + return len(self._records) + + def set_judge(self, judge_backend: Any, judge_model: str) -> None: + """Set the judge backend/model for LLM-judge and hybrid grading.""" + self._judge_backend = judge_backend + self._judge_model = judge_model + + def create_task_env(self, record: EvalRecord): + from openjarvis.evals.execution.pinchbench_env import PinchBenchTaskEnv + return PinchBenchTaskEnv( + record, + judge_backend=getattr(self, "_judge_backend", None), + judge_model=getattr(self, "_judge_model", "anthropic/claude-opus-4-5"), + ) + + +__all__ = ["PinchBenchDataset"] diff --git a/tests/evals/test_pinchbench_dataset.py b/tests/evals/test_pinchbench_dataset.py new file mode 100644 index 00000000..4abcd9a7 --- /dev/null +++ b/tests/evals/test_pinchbench_dataset.py @@ -0,0 +1,130 @@ +"""Tests for PinchBench dataset provider.""" + +import textwrap + +from openjarvis.evals.datasets.pinchbench import _parse_task_markdown + + +def test_parse_task_markdown_basic(): + """Parse a minimal task markdown file.""" + md = textwrap.dedent("""\ + --- + id: task_00_test + name: Test Task + category: basic + grading_type: automated + timeout_seconds: 60 + workspace_files: [] + --- + + ## Prompt + + Do the thing. + + ## Expected Behavior + + The thing should be done. + + ## Grading Criteria + + - [ ] Thing was done + + ## Automated Checks + + ```python + def grade(transcript, workspace_path): + return {"done": 1.0} + ``` + """) + task = _parse_task_markdown(md, filename="task_00_test.md") + assert task["id"] == "task_00_test" + assert task["name"] == "Test Task" + assert task["category"] == "basic" + assert task["grading_type"] == "automated" + assert "Do the thing." in task["prompt"] + assert "The thing should be done." in task["expected_behavior"] + assert "def grade" in task["automated_checks"] + + +def test_parse_task_markdown_hybrid(): + """Parse a hybrid-graded task with weights.""" + md = textwrap.dedent("""\ + --- + id: task_16_triage + name: Email Triage + category: email + grading_type: hybrid + timeout_seconds: 300 + workspace_files: + - source: emails/email_01.txt + dest: inbox/email_01.txt + grading_weights: + automated: 0.4 + llm_judge: 0.6 + --- + + ## Prompt + + Triage the emails. + + ## Expected Behavior + + Create a report. + + ## Grading Criteria + + - [ ] Report created + + ## Automated Checks + + ```python + def grade(transcript, workspace_path): + return {"report": 1.0} + ``` + + ## LLM Judge Rubric + + ### Criterion 1: Quality (Weight: 100%) + + **Score 1.0**: Excellent + """) + task = _parse_task_markdown(md, filename="task_16_triage.md") + assert task["grading_type"] == "hybrid" + assert task["grading_weights"] == {"automated": 0.4, "llm_judge": 0.6} + assert len(task["workspace_files"]) == 1 + assert "Quality" in task["llm_judge_rubric"] + + +def test_parse_task_markdown_no_automated_checks(): + """LLM-judge-only tasks have no automated checks.""" + md = textwrap.dedent("""\ + --- + id: task_03_blog + name: Blog Post + category: writing + grading_type: llm_judge + timeout_seconds: 180 + workspace_files: [] + --- + + ## Prompt + + Write a blog post. + + ## Expected Behavior + + A good blog post. + + ## Grading Criteria + + - [ ] Well written + + ## LLM Judge Rubric + + ### Writing (Weight: 100%) + + **Score 1.0**: Excellent + """) + task = _parse_task_markdown(md, filename="task_03_blog.md") + assert task["automated_checks"] is None + assert "Writing" in task["llm_judge_rubric"] From 73cb1f1bbdf12ab7c8be4bac53ad28af139ed5bc Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:26:12 -0700 Subject: [PATCH 72/92] =?UTF-8?q?feat(evals):=20add=20PinchBench=20grading?= =?UTF-8?q?=20helpers=20=E2=80=94=20transcript=20translation,=20automated/?= =?UTF-8?q?LLM=20judge/hybrid=20scoring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/evals/scorers/pinchbench.py | 445 +++++++++++++++++++++ tests/evals/test_pinchbench_grading.py | 137 +++++++ tests/evals/test_pinchbench_transcript.py | 56 +++ 3 files changed, 638 insertions(+) create mode 100644 src/openjarvis/evals/scorers/pinchbench.py create mode 100644 tests/evals/test_pinchbench_grading.py diff --git a/src/openjarvis/evals/scorers/pinchbench.py b/src/openjarvis/evals/scorers/pinchbench.py new file mode 100644 index 00000000..68269ec0 --- /dev/null +++ b/src/openjarvis/evals/scorers/pinchbench.py @@ -0,0 +1,445 @@ +"""PinchBench grading helpers and scorer. + +Provides transcript translation (OpenJarvis events → PinchBench format), +automated grading (exec of embedded Python), LLM judge grading, and +hybrid combination. Used by PinchBenchTaskEnv.run_tests() and the +standalone PinchBenchScorer. + +Reference: https://github.com/pinchbench/skill +""" + +from __future__ import annotations + +import json +import logging +import re +from typing import Any, Dict, List, Optional, Tuple + +from openjarvis.evals.core.event_recorder import EventType +from openjarvis.evals.core.scorer import LLMJudgeScorer +from openjarvis.evals.core.types import EvalRecord + +LOGGER = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Tool name mapping: OpenJarvis name → PinchBench-expected name +# --------------------------------------------------------------------------- +_TOOL_NAME_MAP: Dict[str, str] = { + "file_read": "read_file", + "file_write": "write_file", + "image_generate": "generate_image", + # web_search, calculator, shell_exec, etc. use the same names +} + + +# --------------------------------------------------------------------------- +# Transcript translation +# --------------------------------------------------------------------------- + +def events_to_transcript(events: List[Any]) -> List[Dict[str, Any]]: + """Build PinchBench-format transcript from raw EventRecorder events. + + Pairs TOOL_CALL_START/END events to extract tool name, arguments, and + results. Called by run_tests() before QueryTrace exists. + """ + transcript: List[Dict[str, Any]] = [] + + for event in events: + etype = event.event_type + if isinstance(etype, str): + # Normalize string event types to enum comparison + pass + if etype == EventType.TOOL_CALL_START or etype == EventType.TOOL_CALL_START.value: + tool_name = event.metadata.get("tool", "unknown") + mapped = _TOOL_NAME_MAP.get(tool_name, tool_name) + arguments = event.metadata.get("arguments", {}) + transcript.append({ + "type": "message", + "message": { + "role": "assistant", + "content": [{"type": "toolCall", "name": mapped, "arguments": arguments}], + }, + }) + elif etype == EventType.TOOL_CALL_END or etype == EventType.TOOL_CALL_END.value: + result_text = str(event.metadata.get("result", "")) + transcript.append({ + "type": "message", + "message": { + "role": "toolResult", + "content": [{"text": result_text}], + }, + }) + + return transcript + + +def _trace_to_transcript(trace: Any) -> List[Dict[str, Any]]: + """Convert QueryTrace to PinchBench transcript (for EvalRunner path).""" + transcript: List[Dict[str, Any]] = [] + for turn in trace.turns: + for tc in getattr(turn, "tool_calls", []): + mapped = _TOOL_NAME_MAP.get(tc["name"], tc["name"]) + transcript.append({ + "type": "message", + "message": { + "role": "assistant", + "content": [{"type": "toolCall", "name": mapped, "arguments": tc.get("arguments", {})}], + }, + }) + transcript.append({ + "type": "message", + "message": { + "role": "toolResult", + "content": [{"text": tc.get("result", "")}], + }, + }) + return transcript + + +# --------------------------------------------------------------------------- +# Transcript summarization (replicates PinchBench _summarize_transcript) +# --------------------------------------------------------------------------- + +def _summarize_transcript(transcript: List[Dict[str, Any]]) -> str: + """Summarize transcript for LLM judge prompt. + + Produces the same format as PinchBench's lib_grading._summarize_transcript(): + Tool: function_name({"arg1": value1}) + Result: [first 200 chars] + User: [message text] + """ + parts: List[str] = [] + for event in transcript: + if event.get("type") != "message": + continue + msg = event.get("message", {}) + role = msg.get("role") + if role == "assistant": + for item in msg.get("content", []): + if item.get("type") == "toolCall": + parts.append( + f"Tool: {item.get('name')}({json.dumps(item.get('arguments', {}))})" + ) + elif role == "toolResult": + content = msg.get("content", []) + if content: + preview = str(content[0])[:200] + parts.append(f"Result: {preview}") + elif role == "user": + content = msg.get("content", []) + if content: + parts.append(f"User: {content[0]}") + return "\n".join(parts) + + +# --------------------------------------------------------------------------- +# Judge prompt and response parsing (replicates PinchBench exactly) +# --------------------------------------------------------------------------- + +def _build_judge_prompt( + *, + task_prompt: str, + expected_behavior: str, + rubric: str, + transcript_summary: str, +) -> str: + """Build the LLM judge prompt matching PinchBench's format.""" + return f"""You are a strict evaluator assessing an AI agent's performance on a task. +Reserve a score of 1.0 for genuinely excellent performance. + +## Task +{task_prompt} + +## Expected Behavior +{expected_behavior} + +## Agent Transcript +{transcript_summary} + +## Grading Rubric +{rubric} + +## Instructions +Evaluate the agent's performance against each criterion in the rubric. +Return your evaluation as JSON with this exact structure: +```json +{{ + "scores": {{"criterion_name": score, ...}}, + "total": overall_score_0_to_1, + "notes": "brief justification" +}} +``` +Be a strict evaluator. Deduct points for unnecessary steps, verbose output, +or inefficient tool usage.""" + + +def _parse_judge_response(raw: str) -> Dict[str, Any]: + """Parse LLM judge response with fallback chain. + + Tries: JSON code block → balanced braces → regex score extraction. + Matches PinchBench's lib_grading._parse_judge_response() logic. + """ + if not raw or not raw.strip(): + return {"scores": {}, "total": 0.0, "notes": "Empty judge response"} + + # Try JSON code block + code_block = re.search(r"```json\s*(.*?)\s*```", raw, re.DOTALL) + if code_block: + try: + parsed = json.loads(code_block.group(1)) + if isinstance(parsed, dict): + return _normalize_judge_response(parsed) + except json.JSONDecodeError: + pass + + # Try balanced braces extraction + candidates: List[str] = [] + depth = 0 + current: List[str] = [] + for char in raw: + if char == "{": + if depth == 0: + current = [] + depth += 1 + if depth > 0: + current.append(char) + if char == "}": + depth -= 1 + if depth == 0 and current: + candidates.append("".join(current)) + + for candidate in reversed(candidates): + try: + parsed = json.loads(candidate) + if isinstance(parsed, dict) and "scores" in parsed: + return _normalize_judge_response(parsed) + except json.JSONDecodeError: + continue + + for candidate in reversed(candidates): + try: + parsed = json.loads(candidate) + if isinstance(parsed, dict): + return _normalize_judge_response(parsed) + except json.JSONDecodeError: + continue + + # Regex fallback for prose scores + score_match = re.search( + r"(?:total|overall|final)\s*(?:score)?[:\s]*(0\.\d+|1\.0+)", + raw, + re.IGNORECASE, + ) + if score_match: + try: + total = float(score_match.group(1)) + if 0.0 <= total <= 1.0: + LOGGER.warning("Fell back to regex score extraction (total=%.2f)", total) + return {"scores": {}, "total": total, "notes": "Score extracted from prose"} + except ValueError: + pass + + LOGGER.warning("Failed to parse judge response") + return {"scores": {}, "total": 0.0, "notes": "Failed to parse judge response"} + + +def _normalize_judge_response(parsed: Dict[str, Any]) -> Dict[str, Any]: + """Normalize various judge response formats to standard structure. + + Matches PinchBench's lib_grading._normalize_judge_response(). + """ + result: Dict[str, Any] = {"scores": {}, "total": 0.0, "notes": ""} + + # Extract scores + scores_data = parsed.get("scores", parsed.get("criteria_scores", {})) + if isinstance(scores_data, dict): + for key, value in scores_data.items(): + if isinstance(value, dict) and "score" in value: + result["scores"][key] = float(value["score"]) + elif isinstance(value, (int, float)): + result["scores"][key] = float(value) + + # Extract total + for key in ("total", "score", "overall_score"): + if key in parsed and isinstance(parsed[key], (int, float)): + result["total"] = float(parsed[key]) + break + else: + if result["scores"]: + values = [v for v in result["scores"].values() if isinstance(v, (int, float))] + if values: + result["total"] = sum(values) / len(values) + + # Normalize summed totals back to 0..1 + values = [v for v in result["scores"].values() if isinstance(v, (int, float))] + if ( + values + and result["total"] is not None + and result["total"] > 1.0 + and all(0.0 <= float(v) <= 1.0 for v in values) + ): + result["total"] = sum(values) / len(values) + + # Extract notes + for key in ("notes", "justification", "reasoning"): + if key in parsed: + result["notes"] = str(parsed[key]) + break + + return result + + +# --------------------------------------------------------------------------- +# Grading functions +# --------------------------------------------------------------------------- + +def _grade_automated( + record: EvalRecord, + transcript: List[Dict[str, Any]], + workspace_path: str, +) -> Dict[str, Any]: + """Run the embedded Python grade() function from the task definition.""" + code = record.metadata.get("automated_checks") + if not code: + return {"score": 0.0, "breakdown": {}, "notes": "No automated checks defined"} + + namespace: Dict[str, Any] = {} + try: + exec(code, namespace) # noqa: S102 + except Exception as exc: + LOGGER.error("Failed to compile grading code for %s: %s", record.record_id, exc) + return {"score": 0.0, "breakdown": {}, "notes": f"Grading code error: {exc}"} + + grade_fn = namespace.get("grade") + if not callable(grade_fn): + return {"score": 0.0, "breakdown": {}, "notes": "No grade() function found in automated checks"} + + try: + scores = grade_fn(transcript, workspace_path) + except Exception as exc: + LOGGER.error("grade() failed for %s: %s", record.record_id, exc) + return {"score": 0.0, "breakdown": {}, "notes": f"grade() error: {exc}"} + + if not isinstance(scores, dict) or not scores: + return {"score": 0.0, "breakdown": {}, "notes": "grade() returned empty or non-dict"} + + mean_score = sum(scores.values()) / len(scores) + return {"score": mean_score, "breakdown": scores, "notes": ""} + + +def _grade_llm_judge( + record: EvalRecord, + transcript: List[Dict[str, Any]], + workspace_path: str, + judge_backend: Any, + judge_model: str, +) -> Dict[str, Any]: + """Grade using an LLM judge with the task's rubric.""" + rubric = record.metadata.get("llm_judge_rubric") + if not rubric: + return {"score": 0.0, "breakdown": {}, "notes": "No LLM judge rubric defined"} + + if judge_backend is None: + return {"score": 0.0, "breakdown": {}, "notes": "No judge backend configured"} + + summary = _summarize_transcript(transcript) + prompt = _build_judge_prompt( + task_prompt=record.problem, + expected_behavior=record.reference or "", + rubric=rubric, + transcript_summary=summary, + ) + + try: + raw = judge_backend.generate(prompt, model=judge_model, temperature=0.0, max_tokens=2048) + except Exception as exc: + LOGGER.error("LLM judge call failed for %s: %s", record.record_id, exc) + return {"score": 0.0, "breakdown": {}, "notes": f"Judge error: {exc}"} + + parsed = _parse_judge_response(raw) + return { + "score": parsed.get("total", 0.0), + "breakdown": parsed.get("scores", {}), + "notes": parsed.get("notes", ""), + } + + +def _grade_hybrid( + record: EvalRecord, + transcript: List[Dict[str, Any]], + workspace_path: str, + judge_backend: Any, + judge_model: str, +) -> Dict[str, Any]: + """Run both automated and LLM judge grading, combine with weights.""" + weights = record.metadata.get("grading_weights", {"automated": 0.5, "llm_judge": 0.5}) + auto = _grade_automated(record, transcript, workspace_path) + llm = _grade_llm_judge(record, transcript, workspace_path, judge_backend, judge_model) + + auto_w = float(weights.get("automated", 0.5)) + llm_w = float(weights.get("llm_judge", 0.5)) + total_w = auto_w + llm_w + + combined = (auto["score"] * auto_w + llm["score"] * llm_w) / total_w if total_w > 0 else 0.0 + breakdown = { + **{f"automated.{k}": v for k, v in auto["breakdown"].items()}, + **{f"llm_judge.{k}": v for k, v in llm["breakdown"].items()}, + } + notes = " | ".join(filter(None, [auto.get("notes", ""), llm.get("notes", "")])) + return {"score": combined, "breakdown": breakdown, "notes": notes} + + +def grade_pinchbench_task( + *, + record: EvalRecord, + transcript: List[Dict[str, Any]], + workspace_path: str, + judge_backend: Any = None, + judge_model: str = "anthropic/claude-opus-4-5", +) -> Dict[str, Any]: + """Top-level grading entry point. Routes by grading_type. + + Returns {"score": float, "breakdown": dict, "notes": str}. + """ + grading_type = record.metadata.get("grading_type", "automated") + + if grading_type == "automated": + return _grade_automated(record, transcript, workspace_path) + elif grading_type == "llm_judge": + return _grade_llm_judge(record, transcript, workspace_path, judge_backend, judge_model) + elif grading_type == "hybrid": + return _grade_hybrid(record, transcript, workspace_path, judge_backend, judge_model) + else: + return {"score": 0.0, "breakdown": {}, "notes": f"Unknown grading type: {grading_type}"} + + +# --------------------------------------------------------------------------- +# Standalone scorer (for EvalRunner non-agentic path) +# --------------------------------------------------------------------------- + +class PinchBenchScorer(LLMJudgeScorer): + """PinchBench scorer for the non-agentic EvalRunner path.""" + + scorer_id = "pinchbench" + + def score( + self, record: EvalRecord, model_answer: str, + ) -> Tuple[Optional[bool], Dict[str, Any]]: + trace = record.metadata.get("query_trace") + transcript = _trace_to_transcript(trace) if trace else [] + result = grade_pinchbench_task( + record=record, + transcript=transcript, + workspace_path=record.metadata.get("workspace_path", ""), + judge_backend=self._judge_backend, + judge_model=self._judge_model, + ) + is_correct = result["score"] >= 0.5 + return is_correct, {**result} + + +__all__ = [ + "PinchBenchScorer", + "events_to_transcript", + "grade_pinchbench_task", +] diff --git a/tests/evals/test_pinchbench_grading.py b/tests/evals/test_pinchbench_grading.py new file mode 100644 index 00000000..9f37c4d4 --- /dev/null +++ b/tests/evals/test_pinchbench_grading.py @@ -0,0 +1,137 @@ +"""Tests for PinchBench grading functions.""" + +from openjarvis.evals.core.types import EvalRecord +from openjarvis.evals.scorers.pinchbench import ( + _grade_automated, + _parse_judge_response, + _summarize_transcript, + grade_pinchbench_task, +) + + +def _make_record(**meta_overrides) -> EvalRecord: + meta = { + "grading_type": "automated", + "automated_checks": None, + "llm_judge_rubric": None, + "grading_weights": None, + } + meta.update(meta_overrides) + return EvalRecord( + record_id="test_task", + problem="Do the thing", + reference="Expected behavior", + category="test", + subject="Test Task", + metadata=meta, + ) + + +class TestGradeAutomated: + def test_simple_pass(self, tmp_path): + code = ''' +def grade(transcript, workspace_path): + from pathlib import Path + f = Path(workspace_path) / "output.txt" + return {"file_exists": 1.0 if f.exists() else 0.0} +''' + (tmp_path / "output.txt").write_text("hello") + record = _make_record(automated_checks=code) + result = _grade_automated(record, [], str(tmp_path)) + assert result["score"] == 1.0 + assert result["breakdown"]["file_exists"] == 1.0 + + def test_simple_fail(self, tmp_path): + code = ''' +def grade(transcript, workspace_path): + from pathlib import Path + f = Path(workspace_path) / "output.txt" + return {"file_exists": 1.0 if f.exists() else 0.0} +''' + record = _make_record(automated_checks=code) + result = _grade_automated(record, [], str(tmp_path)) + assert result["score"] == 0.0 + + def test_transcript_inspection(self, tmp_path): + code = ''' +def grade(transcript, workspace_path): + used_read = False + for entry in transcript: + if entry.get("type") == "message": + msg = entry.get("message", {}) + if msg.get("role") == "assistant": + for item in msg.get("content", []): + if item.get("type") == "toolCall" and item.get("name") == "read_file": + used_read = True + return {"used_read_file": 1.0 if used_read else 0.0} +''' + transcript = [{ + "type": "message", + "message": { + "role": "assistant", + "content": [{"type": "toolCall", "name": "read_file", "arguments": {"path": "a.txt"}}], + }, + }] + record = _make_record(automated_checks=code) + result = _grade_automated(record, transcript, str(tmp_path)) + assert result["score"] == 1.0 + + def test_no_checks_returns_zero(self, tmp_path): + record = _make_record(automated_checks=None) + result = _grade_automated(record, [], str(tmp_path)) + assert result["score"] == 0.0 + + +class TestParseJudgeResponse: + def test_json_code_block(self): + raw = '```json\n{"scores": {"quality": 0.8}, "total": 0.8, "notes": "good"}\n```' + parsed = _parse_judge_response(raw) + assert parsed["total"] == 0.8 + assert parsed["scores"]["quality"] == 0.8 + + def test_bare_json(self): + raw = 'The agent did well. {"scores": {"a": 0.9}, "total": 0.9, "notes": ""}' + parsed = _parse_judge_response(raw) + assert parsed["total"] == 0.9 + + def test_regex_fallback(self): + raw = "The agent performed reasonably. Overall score: 0.65" + parsed = _parse_judge_response(raw) + assert parsed["total"] == 0.65 + + def test_empty_response(self): + parsed = _parse_judge_response("") + assert parsed["total"] == 0.0 + + +class TestSummarizeTranscript: + def test_tool_call_and_result(self): + transcript = [ + { + "type": "message", + "message": { + "role": "assistant", + "content": [{"type": "toolCall", "name": "read_file", "arguments": {"path": "a.txt"}}], + }, + }, + { + "type": "message", + "message": {"role": "toolResult", "content": [{"text": "file contents"}]}, + }, + ] + summary = _summarize_transcript(transcript) + assert 'Tool: read_file({"path": "a.txt"})' in summary + assert "Result:" in summary + + +class TestGradeRouter: + def test_routes_automated(self, tmp_path): + code = 'def grade(t, w): return {"ok": 1.0}' + record = _make_record(grading_type="automated", automated_checks=code) + result = grade_pinchbench_task(record=record, transcript=[], workspace_path=str(tmp_path)) + assert result["score"] == 1.0 + + def test_unknown_type(self, tmp_path): + record = _make_record(grading_type="unknown") + result = grade_pinchbench_task(record=record, transcript=[], workspace_path=str(tmp_path)) + assert result["score"] == 0.0 diff --git a/tests/evals/test_pinchbench_transcript.py b/tests/evals/test_pinchbench_transcript.py index 7218216b..d9fefc88 100644 --- a/tests/evals/test_pinchbench_transcript.py +++ b/tests/evals/test_pinchbench_transcript.py @@ -30,3 +30,59 @@ def test_turn_trace_tool_calls_from_dict_missing(): d = {"turn_index": 0} turn = TurnTrace.from_dict(d) assert turn.tool_calls == [] + + +from openjarvis.evals.core.event_recorder import AgentEvent, EventType + + +def _make_event(etype, **metadata): + """Helper to create a mock AgentEvent.""" + return AgentEvent(event_type=etype, timestamp=0.0, metadata=metadata) + + +def test_events_to_transcript_tool_call_pair(): + """TOOL_CALL_START + END produces assistant toolCall + toolResult messages.""" + from openjarvis.evals.scorers.pinchbench import events_to_transcript + + events = [ + _make_event(EventType.TOOL_CALL_START, tool="file_read", arguments={"path": "a.txt"}), + _make_event(EventType.TOOL_CALL_END, tool="file_read", result="file contents"), + ] + transcript = events_to_transcript(events) + assert len(transcript) == 2 + assert transcript[0]["type"] == "message" + assert transcript[0]["message"]["role"] == "assistant" + assert transcript[0]["message"]["content"][0]["type"] == "toolCall" + # Tool name mapped: file_read -> read_file + assert transcript[0]["message"]["content"][0]["name"] == "read_file" + assert transcript[1]["message"]["role"] == "toolResult" + assert transcript[1]["message"]["content"][0]["text"] == "file contents" + + +def test_events_to_transcript_tool_name_mapping(): + """OpenJarvis tool names are mapped to PinchBench-expected names.""" + from openjarvis.evals.scorers.pinchbench import events_to_transcript + + events = [ + _make_event(EventType.TOOL_CALL_START, tool="image_generate", arguments={"prompt": "cat"}), + _make_event(EventType.TOOL_CALL_END, tool="image_generate", result="ok"), + ] + transcript = events_to_transcript(events) + assert transcript[0]["message"]["content"][0]["name"] == "generate_image" + + +def test_events_to_transcript_empty(): + """Empty events produce empty transcript.""" + from openjarvis.evals.scorers.pinchbench import events_to_transcript + assert events_to_transcript([]) == [] + + +def test_events_to_transcript_ignores_non_tool_events(): + """Non-tool events are skipped.""" + from openjarvis.evals.scorers.pinchbench import events_to_transcript + + events = [ + _make_event(EventType.LM_INFERENCE_START), + _make_event(EventType.LM_INFERENCE_END, prompt_tokens=10, completion_tokens=5), + ] + assert events_to_transcript(events) == [] From 6ae720baa9df9205530e596de7bb26587dafd59c Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:27:18 -0700 Subject: [PATCH 73/92] =?UTF-8?q?feat(evals):=20add=20PinchBench=20task=20?= =?UTF-8?q?environment=20=E2=80=94=20workspace=20setup=20and=20run=5Ftests?= =?UTF-8?q?()=20grading?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../evals/execution/pinchbench_env.py | 123 ++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 src/openjarvis/evals/execution/pinchbench_env.py diff --git a/src/openjarvis/evals/execution/pinchbench_env.py b/src/openjarvis/evals/execution/pinchbench_env.py new file mode 100644 index 00000000..f492746e --- /dev/null +++ b/src/openjarvis/evals/execution/pinchbench_env.py @@ -0,0 +1,123 @@ +"""PinchBench task environment — per-task workspace setup and grading.""" + +from __future__ import annotations + +import logging +import os +import shutil +import tempfile +from pathlib import Path +from types import TracebackType +from typing import Any, Optional, Type + +from openjarvis.evals.core.types import EvalRecord + +LOGGER = logging.getLogger(__name__) + + +class PinchBenchTaskEnv: + """Per-task workspace environment for PinchBench. + + Context manager that creates an isolated workspace directory, + populates fixture files from the task definition, and runs + grading after agent execution via run_tests(). + """ + + def __init__( + self, + record: EvalRecord, + judge_backend: Any = None, + judge_model: str = "anthropic/claude-opus-4-5", + ) -> None: + self._record = record + self._judge_backend = judge_backend + self._judge_model = judge_model + self._workspace: Optional[Path] = None + self._event_recorder: Any = None + + def set_event_recorder(self, recorder: Any) -> None: + """Receive the EventRecorder from AgenticRunner for transcript building.""" + self._event_recorder = recorder + + def __enter__(self) -> PinchBenchTaskEnv: + # Create isolated workspace + self._workspace = Path(tempfile.mkdtemp(prefix="pinchbench_")) + + # Populate fixture files + repo_dir = Path(self._record.metadata.get("pinchbench_repo_dir", "")) + workspace_files = self._record.metadata.get("workspace_files", []) + + for file_spec in workspace_files: + if "content" in file_spec: + # Inline content + dest = self._workspace / file_spec.get("path", file_spec.get("dest", "file.txt")) + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_text(file_spec["content"]) + elif "source" in file_spec: + # Asset file from repo + source = repo_dir / "assets" / file_spec["source"] + dest_key = file_spec.get("dest", file_spec.get("path", file_spec["source"])) + dest = self._workspace / dest_key + dest.parent.mkdir(parents=True, exist_ok=True) + if source.exists(): + shutil.copy2(str(source), str(dest)) + else: + LOGGER.warning("Asset not found: %s", source) + + self._record.metadata["workspace_path"] = str(self._workspace) + LOGGER.info("PinchBench workspace: %s (task %s)", self._workspace, self._record.record_id) + return self + + def run_tests(self) -> None: + """Grade the agent's work using PinchBench grading logic. + + Called by AgenticRunner after agent execution, before QueryTrace + is constructed. Builds transcript from raw EventRecorder events. + """ + from openjarvis.evals.scorers.pinchbench import ( + events_to_transcript, + grade_pinchbench_task, + ) + + workspace_path = self._record.metadata.get("workspace_path", "") + events = self._event_recorder.get_events() if self._event_recorder else [] + transcript = events_to_transcript(events) + + result = grade_pinchbench_task( + record=self._record, + transcript=transcript, + workspace_path=workspace_path, + judge_backend=self._judge_backend, + judge_model=self._judge_model, + ) + + self._record.metadata["is_resolved"] = result["score"] >= 0.5 + self._record.metadata["reward"] = result["score"] + self._record.metadata["pinchbench_score"] = result["score"] + self._record.metadata["pinchbench_breakdown"] = result["breakdown"] + self._record.metadata["pinchbench_notes"] = result.get("notes", "") + + LOGGER.info( + "PinchBench grading for %s: score=%.2f resolved=%s", + self._record.record_id, + result["score"], + result["score"] >= 0.5, + ) + + def __exit__( + self, + exc_type: Optional[Type[BaseException]], + exc_val: Optional[BaseException], + exc_tb: Optional[TracebackType], + ) -> None: + if self._workspace and self._workspace.exists(): + keep = os.environ.get("PINCHBENCH_KEEP_WORKSPACES", "").strip() + if keep and keep != "0": + LOGGER.info("Keeping workspace: %s", self._workspace) + else: + shutil.rmtree(self._workspace, ignore_errors=True) + self._workspace = None + self._event_recorder = None + + +__all__ = ["PinchBenchTaskEnv"] From b4962482c029aeaa1bd0483d6b5b88a30a736dc0 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:28:29 -0700 Subject: [PATCH 74/92] feat(evals): register PinchBench in CLI and add default TOML config Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/evals/cli.py | 17 +++++++++++++ src/openjarvis/evals/configs/pinchbench.toml | 26 ++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 src/openjarvis/evals/configs/pinchbench.toml diff --git a/src/openjarvis/evals/cli.py b/src/openjarvis/evals/cli.py index f32e13d4..319188a9 100644 --- a/src/openjarvis/evals/cli.py +++ b/src/openjarvis/evals/cli.py @@ -119,6 +119,10 @@ BENCHMARKS = { "category": "use-case", "description": "Web research with fact verification", }, + "pinchbench": { + "category": "agentic", + "description": "PinchBench real-world agent tasks (23 tasks, multi-turn with tool use)", + }, } BACKENDS = { @@ -260,6 +264,9 @@ def _build_dataset(benchmark: str, subset: str | None = None): elif benchmark == "browser_assistant": from openjarvis.evals.datasets.browser_assistant import BrowserAssistantDataset return BrowserAssistantDataset() + elif benchmark == "pinchbench": + from openjarvis.evals.datasets.pinchbench import PinchBenchDataset + return PinchBenchDataset(path=subset) else: raise click.ClickException(f"Unknown benchmark: {benchmark}") @@ -361,6 +368,9 @@ def _build_scorer(benchmark: str, judge_backend, judge_model: str): elif benchmark == "browser_assistant": from openjarvis.evals.scorers.browser_assistant import BrowserAssistantScorer return BrowserAssistantScorer(judge_backend, judge_model) + elif benchmark == "pinchbench": + from openjarvis.evals.scorers.pinchbench import PinchBenchScorer + return PinchBenchScorer(judge_backend, judge_model) else: raise click.ClickException(f"Unknown benchmark: {benchmark}") @@ -519,6 +529,13 @@ def _run_agentic( seed=config.seed, ) + # Wire judge for PinchBench LLM-judge and hybrid grading + if config.benchmark == "pinchbench" and hasattr(dataset, "set_judge"): + judge_engine = getattr(config, "judge_engine", "cloud") or "cloud" + judge_model = getattr(config, "judge_model", None) or "anthropic/claude-opus-4-5" + judge_backend = _build_judge_backend(judge_model, engine_key=judge_engine) + dataset.set_judge(judge_backend, judge_model) + # Verify backend requirements before doing any work if hasattr(dataset, "verify_requirements"): issues = dataset.verify_requirements() diff --git a/src/openjarvis/evals/configs/pinchbench.toml b/src/openjarvis/evals/configs/pinchbench.toml new file mode 100644 index 00000000..fe1a764f --- /dev/null +++ b/src/openjarvis/evals/configs/pinchbench.toml @@ -0,0 +1,26 @@ +[meta] +name = "pinchbench" +description = "PinchBench real-world agent benchmark" + +[defaults] +temperature = 0.0 + +[judge] +model = "anthropic/claude-opus-4-5" +temperature = 0.0 + +[run] +output_dir = "results/pinchbench" + +[[models]] +name = "anthropic/claude-sonnet-4" + +[[benchmarks]] +name = "pinchbench" +backend = "jarvis-agent" +agent = "native_react" +tools = [ + "file_read", "file_write", "web_search", "shell_exec", + "code_interpreter", "browser_navigate", "image_generate", + "calculator", "http_request", "pdf_extract", +] From 784b880130612817b2a04729f87c295b420ce8cd Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 09:31:02 -0700 Subject: [PATCH 75/92] =?UTF-8?q?test(evals):=20add=20PinchBench=20integra?= =?UTF-8?q?tion=20tests=20=E2=80=94=20full=20grading=20pipeline?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- .../evals/execution/pinchbench_env.py | 13 +- tests/evals/test_pinchbench_grading.py | 36 +++++- tests/evals/test_pinchbench_integration.py | 121 ++++++++++++++++++ tests/evals/test_pinchbench_transcript.py | 14 +- 4 files changed, 169 insertions(+), 15 deletions(-) create mode 100644 tests/evals/test_pinchbench_integration.py diff --git a/src/openjarvis/evals/execution/pinchbench_env.py b/src/openjarvis/evals/execution/pinchbench_env.py index f492746e..cd11235e 100644 --- a/src/openjarvis/evals/execution/pinchbench_env.py +++ b/src/openjarvis/evals/execution/pinchbench_env.py @@ -50,13 +50,16 @@ class PinchBenchTaskEnv: for file_spec in workspace_files: if "content" in file_spec: # Inline content - dest = self._workspace / file_spec.get("path", file_spec.get("dest", "file.txt")) + dest_key = file_spec.get("path", file_spec.get("dest", "file.txt")) + dest = self._workspace / dest_key dest.parent.mkdir(parents=True, exist_ok=True) dest.write_text(file_spec["content"]) elif "source" in file_spec: # Asset file from repo source = repo_dir / "assets" / file_spec["source"] - dest_key = file_spec.get("dest", file_spec.get("path", file_spec["source"])) + dest_key = file_spec.get( + "dest", file_spec.get("path", file_spec["source"]) + ) dest = self._workspace / dest_key dest.parent.mkdir(parents=True, exist_ok=True) if source.exists(): @@ -65,7 +68,11 @@ class PinchBenchTaskEnv: LOGGER.warning("Asset not found: %s", source) self._record.metadata["workspace_path"] = str(self._workspace) - LOGGER.info("PinchBench workspace: %s (task %s)", self._workspace, self._record.record_id) + LOGGER.info( + "PinchBench workspace: %s (task %s)", + self._workspace, + self._record.record_id, + ) return self def run_tests(self) -> None: diff --git a/tests/evals/test_pinchbench_grading.py b/tests/evals/test_pinchbench_grading.py index 9f37c4d4..aa107506 100644 --- a/tests/evals/test_pinchbench_grading.py +++ b/tests/evals/test_pinchbench_grading.py @@ -61,7 +61,8 @@ def grade(transcript, workspace_path): msg = entry.get("message", {}) if msg.get("role") == "assistant": for item in msg.get("content", []): - if item.get("type") == "toolCall" and item.get("name") == "read_file": + name = item.get("name") + if item.get("type") == "toolCall" and name == "read_file": used_read = True return {"used_read_file": 1.0 if used_read else 0.0} ''' @@ -69,7 +70,13 @@ def grade(transcript, workspace_path): "type": "message", "message": { "role": "assistant", - "content": [{"type": "toolCall", "name": "read_file", "arguments": {"path": "a.txt"}}], + "content": [ + { + "type": "toolCall", + "name": "read_file", + "arguments": {"path": "a.txt"}, + } + ], }, }] record = _make_record(automated_checks=code) @@ -84,7 +91,9 @@ def grade(transcript, workspace_path): class TestParseJudgeResponse: def test_json_code_block(self): - raw = '```json\n{"scores": {"quality": 0.8}, "total": 0.8, "notes": "good"}\n```' + raw = ( + '```json\n{"scores": {"quality": 0.8}, "total": 0.8, "notes": "good"}\n```' + ) parsed = _parse_judge_response(raw) assert parsed["total"] == 0.8 assert parsed["scores"]["quality"] == 0.8 @@ -111,12 +120,21 @@ class TestSummarizeTranscript: "type": "message", "message": { "role": "assistant", - "content": [{"type": "toolCall", "name": "read_file", "arguments": {"path": "a.txt"}}], + "content": [ + { + "type": "toolCall", + "name": "read_file", + "arguments": {"path": "a.txt"}, + } + ], }, }, { "type": "message", - "message": {"role": "toolResult", "content": [{"text": "file contents"}]}, + "message": { + "role": "toolResult", + "content": [{"text": "file contents"}], + }, }, ] summary = _summarize_transcript(transcript) @@ -128,10 +146,14 @@ class TestGradeRouter: def test_routes_automated(self, tmp_path): code = 'def grade(t, w): return {"ok": 1.0}' record = _make_record(grading_type="automated", automated_checks=code) - result = grade_pinchbench_task(record=record, transcript=[], workspace_path=str(tmp_path)) + result = grade_pinchbench_task( + record=record, transcript=[], workspace_path=str(tmp_path) + ) assert result["score"] == 1.0 def test_unknown_type(self, tmp_path): record = _make_record(grading_type="unknown") - result = grade_pinchbench_task(record=record, transcript=[], workspace_path=str(tmp_path)) + result = grade_pinchbench_task( + record=record, transcript=[], workspace_path=str(tmp_path) + ) assert result["score"] == 0.0 diff --git a/tests/evals/test_pinchbench_integration.py b/tests/evals/test_pinchbench_integration.py new file mode 100644 index 00000000..4edf9334 --- /dev/null +++ b/tests/evals/test_pinchbench_integration.py @@ -0,0 +1,121 @@ +"""Integration tests for PinchBench eval pipeline. + +These tests use synthetic task files to verify the full pipeline +without requiring the actual PinchBench repo or cloud API keys. +""" + +import textwrap +from pathlib import Path + +from openjarvis.evals.core.event_recorder import EventRecorder, EventType +from openjarvis.evals.datasets.pinchbench import PinchBenchDataset + + +def _create_test_repo(tmp_path: Path) -> Path: + """Create a minimal PinchBench repo structure for testing.""" + repo = tmp_path / "skill" + tasks_dir = repo / "tasks" + tasks_dir.mkdir(parents=True) + assets_dir = repo / "assets" + assets_dir.mkdir() + + # Write a simple automated task + (tasks_dir / "task_00_sanity.md").write_text(textwrap.dedent("""\ + --- + id: task_00_sanity + name: Sanity Check + category: basic + grading_type: automated + timeout_seconds: 60 + workspace_files: [] + --- + + ## Prompt + + Write "hello" to a file called output.txt. + + ## Expected Behavior + + A file output.txt exists with the word hello. + + ## Grading Criteria + + - [ ] output.txt exists + - [ ] Contains hello + + ## Automated Checks + + ```python + def grade(transcript, workspace_path): + from pathlib import Path + f = Path(workspace_path) / "output.txt" + exists = f.exists() + has_hello = "hello" in f.read_text().lower() if exists else False + return { + "file_exists": 1.0 if exists else 0.0, + "has_hello": 1.0 if has_hello else 0.0, + } + ``` + """)) + + return repo + + +def test_dataset_loads_from_local_path(tmp_path): + """Dataset loads tasks from a local path.""" + repo = _create_test_repo(tmp_path) + ds = PinchBenchDataset(path=str(repo)) + ds.load() + assert ds.size() == 1 + records = list(ds.iter_records()) + assert records[0].record_id == "task_00_sanity" + assert "hello" in records[0].problem.lower() or "output.txt" in records[0].problem + + +def test_task_env_creates_workspace(tmp_path): + """Task env creates and cleans up workspace.""" + repo = _create_test_repo(tmp_path) + ds = PinchBenchDataset(path=str(repo)) + ds.load() + record = list(ds.iter_records())[0] + + env = ds.create_task_env(record) + with env: + ws = Path(record.metadata["workspace_path"]) + assert ws.exists() + # After exit, workspace is cleaned up + assert not ws.exists() + + +def test_full_grading_pipeline(tmp_path): + """Full pipeline: workspace setup -> simulate agent -> grade.""" + repo = _create_test_repo(tmp_path) + ds = PinchBenchDataset(path=str(repo)) + ds.load() + record = list(ds.iter_records())[0] + + env = ds.create_task_env(record) + with env: + ws = Path(record.metadata["workspace_path"]) + # Simulate agent writing the output file + (ws / "output.txt").write_text("hello world") + + # Simulate EventRecorder with a tool call + recorder = EventRecorder() + recorder.record( + EventType.TOOL_CALL_START, + tool="file_write", + arguments={"path": "output.txt", "content": "hello world"}, + ) + recorder.record( + EventType.TOOL_CALL_END, + tool="file_write", + result="ok", + ) + env.set_event_recorder(recorder) + env.run_tests() + + assert record.metadata["is_resolved"] is True + assert record.metadata["pinchbench_score"] == 1.0 + assert record.metadata["pinchbench_breakdown"]["file_exists"] == 1.0 + assert record.metadata["pinchbench_breakdown"]["has_hello"] == 1.0 diff --git a/tests/evals/test_pinchbench_transcript.py b/tests/evals/test_pinchbench_transcript.py index d9fefc88..d7850c6b 100644 --- a/tests/evals/test_pinchbench_transcript.py +++ b/tests/evals/test_pinchbench_transcript.py @@ -1,5 +1,6 @@ """Tests for TurnTrace tool_calls field and PinchBench transcript translation.""" +from openjarvis.evals.core.event_recorder import AgentEvent, EventType from openjarvis.evals.core.trace import TurnTrace @@ -32,9 +33,6 @@ def test_turn_trace_tool_calls_from_dict_missing(): assert turn.tool_calls == [] -from openjarvis.evals.core.event_recorder import AgentEvent, EventType - - def _make_event(etype, **metadata): """Helper to create a mock AgentEvent.""" return AgentEvent(event_type=etype, timestamp=0.0, metadata=metadata) @@ -45,7 +43,9 @@ def test_events_to_transcript_tool_call_pair(): from openjarvis.evals.scorers.pinchbench import events_to_transcript events = [ - _make_event(EventType.TOOL_CALL_START, tool="file_read", arguments={"path": "a.txt"}), + _make_event( + EventType.TOOL_CALL_START, tool="file_read", arguments={"path": "a.txt"} + ), _make_event(EventType.TOOL_CALL_END, tool="file_read", result="file contents"), ] transcript = events_to_transcript(events) @@ -64,7 +64,11 @@ def test_events_to_transcript_tool_name_mapping(): from openjarvis.evals.scorers.pinchbench import events_to_transcript events = [ - _make_event(EventType.TOOL_CALL_START, tool="image_generate", arguments={"prompt": "cat"}), + _make_event( + EventType.TOOL_CALL_START, + tool="image_generate", + arguments={"prompt": "cat"}, + ), _make_event(EventType.TOOL_CALL_END, tool="image_generate", result="ok"), ] transcript = events_to_transcript(events) From 3ec7e762151221d48625c1c4617afe0b2743c0ef Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:33:46 -0700 Subject: [PATCH 76/92] feat: add inference-gemma optional dependency for pygemma --- pyproject.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/pyproject.toml b/pyproject.toml index f13232dc..67e820eb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -39,6 +39,7 @@ inference-google = [ "google-genai>=1.0", ] inference-litellm = ["litellm>=1.40"] +inference-gemma = ["pygemma>=0.1.3"] tools-search = [ "tavily-python>=0.3", "ddgs>=9.11.4", From 9b511e08f7b986546658babe5b3fd230391823c0 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:33:58 -0700 Subject: [PATCH 77/92] fix(evals): workspace path resolution and grading crash resilience - Store workspace_path in record.metadata from AgenticRunner so task envs can reuse the agent's workspace instead of creating a separate temp dir the agent can't see - PinchBenchTaskEnv now uses the existing workspace when available, copies fixtures there, and sets CWD so file_read/file_write resolve relative paths correctly - Wrap grading in try/except so LLM judge failures don't crash the entire run (graceful degradation to score 0.0) - Restore CWD on exit and only clean up self-owned workspaces Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/evals/core/agentic_runner.py | 1 + .../evals/execution/pinchbench_env.py | 49 +++++++++++++++---- 2 files changed, 40 insertions(+), 10 deletions(-) diff --git a/src/openjarvis/evals/core/agentic_runner.py b/src/openjarvis/evals/core/agentic_runner.py index ee272240..57717ba6 100644 --- a/src/openjarvis/evals/core/agentic_runner.py +++ b/src/openjarvis/evals/core/agentic_runner.py @@ -342,6 +342,7 @@ class AgenticRunner: ) workspace.mkdir(parents=True, exist_ok=True) agent.set_workspace(str(workspace)) + record.metadata["workspace_path"] = str(workspace) # Create per-task execution environment (e.g. Docker for TerminalBench) task_env = None diff --git a/src/openjarvis/evals/execution/pinchbench_env.py b/src/openjarvis/evals/execution/pinchbench_env.py index cd11235e..9df2c698 100644 --- a/src/openjarvis/evals/execution/pinchbench_env.py +++ b/src/openjarvis/evals/execution/pinchbench_env.py @@ -33,15 +33,24 @@ class PinchBenchTaskEnv: self._judge_backend = judge_backend self._judge_model = judge_model self._workspace: Optional[Path] = None + self._owns_workspace: bool = True self._event_recorder: Any = None + self._original_cwd: Optional[str] = None def set_event_recorder(self, recorder: Any) -> None: """Receive the EventRecorder from AgenticRunner for transcript building.""" self._event_recorder = recorder def __enter__(self) -> PinchBenchTaskEnv: - # Create isolated workspace - self._workspace = Path(tempfile.mkdtemp(prefix="pinchbench_")) + # Use the AgenticRunner's workspace if already set (agent.set_workspace + # is called before create_task_env), otherwise create a temp dir. + existing = self._record.metadata.get("workspace_path") + if existing and Path(existing).is_dir(): + self._workspace = Path(existing) + self._owns_workspace = False + else: + self._workspace = Path(tempfile.mkdtemp(prefix="pinchbench_")) + self._owns_workspace = True # Populate fixture files repo_dir = Path(self._record.metadata.get("pinchbench_repo_dir", "")) @@ -68,6 +77,11 @@ class PinchBenchTaskEnv: LOGGER.warning("Asset not found: %s", source) self._record.metadata["workspace_path"] = str(self._workspace) + + # Change CWD so file_read/file_write resolve paths relative to workspace + self._original_cwd = os.getcwd() + os.chdir(str(self._workspace)) + LOGGER.info( "PinchBench workspace: %s (task %s)", self._workspace, @@ -90,13 +104,19 @@ class PinchBenchTaskEnv: events = self._event_recorder.get_events() if self._event_recorder else [] transcript = events_to_transcript(events) - result = grade_pinchbench_task( - record=self._record, - transcript=transcript, - workspace_path=workspace_path, - judge_backend=self._judge_backend, - judge_model=self._judge_model, - ) + try: + result = grade_pinchbench_task( + record=self._record, + transcript=transcript, + workspace_path=workspace_path, + judge_backend=self._judge_backend, + judge_model=self._judge_model, + ) + except Exception as exc: + LOGGER.error( + "Grading failed for %s: %s", self._record.record_id, exc, + ) + result = {"score": 0.0, "breakdown": {}, "notes": f"Grading error: {exc}"} self._record.metadata["is_resolved"] = result["score"] >= 0.5 self._record.metadata["reward"] = result["score"] @@ -117,7 +137,16 @@ class PinchBenchTaskEnv: exc_val: Optional[BaseException], exc_tb: Optional[TracebackType], ) -> None: - if self._workspace and self._workspace.exists(): + # Restore original CWD + if self._original_cwd: + try: + os.chdir(self._original_cwd) + except OSError: + pass + self._original_cwd = None + + # Only clean up workspaces we created ourselves (not AgenticRunner's) + if self._owns_workspace and self._workspace and self._workspace.exists(): keep = os.environ.get("PINCHBENCH_KEEP_WORKSPACES", "").strip() if keep and keep != "0": LOGGER.info("Keeping workspace: %s", self._workspace) From d589bfb55d105909129e9cec1b992c7c4cfaf959 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:34:48 -0700 Subject: [PATCH 78/92] feat: add GemmaCppEngineConfig and wire into EngineConfig Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/core/config.py | 11 +++++++++++ tests/engine/test_gemma_cpp.py | 23 +++++++++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 tests/engine/test_gemma_cpp.py diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 697ce63a..9a829fd4 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -337,6 +337,16 @@ class AppleFmEngineConfig: host: str = "http://localhost:8079" +@dataclass(slots=True) +class GemmaCppEngineConfig: + """Per-engine config for gemma.cpp.""" + + model_path: str = "" + tokenizer_path: str = "" + model_type: str = "" + num_threads: int = 0 + + @dataclass class EngineConfig: """Inference engine settings with nested per-engine configs.""" @@ -352,6 +362,7 @@ class EngineConfig: nexa: NexaEngineConfig = field(default_factory=NexaEngineConfig) uzu: UzuEngineConfig = field(default_factory=UzuEngineConfig) apple_fm: AppleFmEngineConfig = field(default_factory=AppleFmEngineConfig) + gemma_cpp: GemmaCppEngineConfig = field(default_factory=GemmaCppEngineConfig) # Backward-compat properties for old flat attribute names @property diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py new file mode 100644 index 00000000..cad16859 --- /dev/null +++ b/tests/engine/test_gemma_cpp.py @@ -0,0 +1,23 @@ +"""Tests for the gemma.cpp engine backend.""" + +from __future__ import annotations + +import os + +import pytest + +from openjarvis.core.config import EngineConfig, GemmaCppEngineConfig + + +class TestGemmaCppEngineConfig: + def test_default_values(self) -> None: + cfg = GemmaCppEngineConfig() + assert cfg.model_path == "" + assert cfg.tokenizer_path == "" + assert cfg.model_type == "" + assert cfg.num_threads == 0 + + def test_engine_config_has_gemma_cpp_field(self) -> None: + ec = EngineConfig() + assert hasattr(ec, "gemma_cpp") + assert isinstance(ec.gemma_cpp, GemmaCppEngineConfig) From 9cd822611e943d19a8789a81f93e1ee290f90e68 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:36:35 -0700 Subject: [PATCH 79/92] feat: add GemmaCppEngine skeleton with chat template formatting Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/engine/gemma_cpp.py | 103 +++++++++++++++++++++++++++++ tests/engine/test_gemma_cpp.py | 69 +++++++++++++++++++ 2 files changed, 172 insertions(+) create mode 100644 src/openjarvis/engine/gemma_cpp.py diff --git a/src/openjarvis/engine/gemma_cpp.py b/src/openjarvis/engine/gemma_cpp.py new file mode 100644 index 00000000..a235417b --- /dev/null +++ b/src/openjarvis/engine/gemma_cpp.py @@ -0,0 +1,103 @@ +"""gemma.cpp inference engine backend via pygemma pybind11 bindings.""" + +from __future__ import annotations + +import logging +import os +from collections.abc import AsyncIterator, Sequence +from typing import Any, Dict, List + +from openjarvis.core.registry import EngineRegistry +from openjarvis.core.types import Message, Role +from openjarvis.engine._base import InferenceEngine, estimate_prompt_tokens + +logger = logging.getLogger(__name__) + + +@EngineRegistry.register("gemma_cpp") +class GemmaCppEngine(InferenceEngine): + """gemma.cpp backend via pygemma pybind11 bindings (in-process, CPU).""" + + engine_id = "gemma_cpp" + + def __init__( + self, + model_path: str | None = None, + tokenizer_path: str | None = None, + model_type: str | None = None, + num_threads: int = 0, + ) -> None: + self._model_path = ( + model_path + or os.environ.get("GEMMA_CPP_MODEL_PATH", "") + ) + self._tokenizer_path = ( + tokenizer_path + or os.environ.get("GEMMA_CPP_TOKENIZER_PATH", "") + ) + self._model_type = ( + model_type + or os.environ.get("GEMMA_CPP_MODEL_TYPE", "") + ) + self._num_threads = num_threads or int( + os.environ.get("GEMMA_CPP_NUM_THREADS", "0") + ) + self._gemma: Any = None # lazy-loaded pygemma.Gemma instance + + def _messages_to_prompt(self, messages: Sequence[Message]) -> str: + """Format messages into Gemma's chat template.""" + parts: list[str] = [] + system_prefix = "" + for msg in messages: + if msg.role == Role.SYSTEM: + system_prefix += msg.content + "\n\n" + elif msg.role == Role.USER: + content = ( + system_prefix + msg.content if system_prefix else msg.content + ) + system_prefix = "" + parts.append(f"user\n{content}\n") + elif msg.role == Role.ASSISTANT: + parts.append( + f"model\n{msg.content}\n" + ) + parts.append("model\n") + return "".join(parts) + + def generate( + self, + messages: Sequence[Message], + *, + model: str, + temperature: float = 0.7, + max_tokens: int = 1024, + **kwargs: Any, + ) -> Dict[str, Any]: + raise NotImplementedError # implemented in Task 4 + + async def stream( + self, + messages: Sequence[Message], + *, + model: str, + temperature: float = 0.7, + max_tokens: int = 1024, + **kwargs: Any, + ) -> AsyncIterator[str]: + raise NotImplementedError # implemented in Task 4 + yield "" # pragma: no cover + + def list_models(self) -> List[str]: + raise NotImplementedError # implemented in Task 5 + + def health(self) -> bool: + raise NotImplementedError # implemented in Task 5 + + def close(self) -> None: + pass # implemented in Task 4 + + def prepare(self, model: str) -> None: + pass # implemented in Task 4 + + +__all__ = ["GemmaCppEngine"] diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py index cad16859..91f8bcfb 100644 --- a/tests/engine/test_gemma_cpp.py +++ b/tests/engine/test_gemma_cpp.py @@ -21,3 +21,72 @@ class TestGemmaCppEngineConfig: ec = EngineConfig() assert hasattr(ec, "gemma_cpp") assert isinstance(ec.gemma_cpp, GemmaCppEngineConfig) + + +from openjarvis.core.types import Message, Role + + +class TestMessagesToPrompt: + def _make_engine(self): + """Create engine with no paths (won't load model, just test formatting).""" + from openjarvis.engine.gemma_cpp import GemmaCppEngine + return GemmaCppEngine() + + def test_single_user_message(self) -> None: + engine = self._make_engine() + msgs = [Message(role=Role.USER, content="Hello")] + result = engine._messages_to_prompt(msgs) + assert result == ( + "user\nHello\n" + "model\n" + ) + + def test_system_folded_into_user(self) -> None: + engine = self._make_engine() + msgs = [ + Message(role=Role.SYSTEM, content="You are helpful."), + Message(role=Role.USER, content="Hello"), + ] + result = engine._messages_to_prompt(msgs) + assert result == ( + "user\n" + "You are helpful.\n\nHello\n" + "model\n" + ) + + def test_multi_turn_conversation(self) -> None: + engine = self._make_engine() + msgs = [ + Message(role=Role.USER, content="Hi"), + Message(role=Role.ASSISTANT, content="Hello!"), + Message(role=Role.USER, content="How are you?"), + ] + result = engine._messages_to_prompt(msgs) + assert result == ( + "user\nHi\n" + "model\nHello!\n" + "user\nHow are you?\n" + "model\n" + ) + + def test_trailing_system_message_discarded(self) -> None: + engine = self._make_engine() + msgs = [ + Message(role=Role.SYSTEM, content="Ignored system"), + ] + result = engine._messages_to_prompt(msgs) + assert result == "model\n" + + def test_multiple_system_messages_concatenated(self) -> None: + engine = self._make_engine() + msgs = [ + Message(role=Role.SYSTEM, content="Rule 1"), + Message(role=Role.SYSTEM, content="Rule 2"), + Message(role=Role.USER, content="Go"), + ] + result = engine._messages_to_prompt(msgs) + assert result == ( + "user\n" + "Rule 1\n\nRule 2\n\nGo\n" + "model\n" + ) From ed47fece397ea3b216062e2c67c7f3ef8324bdc9 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:38:13 -0700 Subject: [PATCH 80/92] =?UTF-8?q?fix:=20lint=20=E2=80=94=20shorten=20long?= =?UTF-8?q?=20lines=20in=20CLI=20and=20agentic=20runner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/evals/cli.py | 7 +++++-- src/openjarvis/evals/core/agentic_runner.py | 3 ++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/src/openjarvis/evals/cli.py b/src/openjarvis/evals/cli.py index 319188a9..25ed1b5f 100644 --- a/src/openjarvis/evals/cli.py +++ b/src/openjarvis/evals/cli.py @@ -121,7 +121,7 @@ BENCHMARKS = { }, "pinchbench": { "category": "agentic", - "description": "PinchBench real-world agent tasks (23 tasks, multi-turn with tool use)", + "description": "PinchBench real-world agent tasks", }, } @@ -532,7 +532,10 @@ def _run_agentic( # Wire judge for PinchBench LLM-judge and hybrid grading if config.benchmark == "pinchbench" and hasattr(dataset, "set_judge"): judge_engine = getattr(config, "judge_engine", "cloud") or "cloud" - judge_model = getattr(config, "judge_model", None) or "anthropic/claude-opus-4-5" + judge_model = ( + getattr(config, "judge_model", None) + or "anthropic/claude-opus-4-5" + ) judge_backend = _build_judge_backend(judge_model, engine_key=judge_engine) dataset.set_judge(judge_backend, judge_model) diff --git a/src/openjarvis/evals/core/agentic_runner.py b/src/openjarvis/evals/core/agentic_runner.py index 57717ba6..ec98da97 100644 --- a/src/openjarvis/evals/core/agentic_runner.py +++ b/src/openjarvis/evals/core/agentic_runner.py @@ -329,7 +329,8 @@ class AgenticRunner: if agent_bus is not None: for etype in (EventType.TOOL_CALL_START, EventType.TOOL_CALL_END): def _relay(event, _etype=etype): - event_recorder.record(_etype, **(event.data if hasattr(event, "data") else {})) + data = event.data if hasattr(event, "data") else {} + event_recorder.record(_etype, **data) agent_bus.subscribe(etype, _relay) _bus_unsubs.append((etype, _relay)) From 8c2ee6843ad278041f79890237f5624fb679449e Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:38:25 -0700 Subject: [PATCH 81/92] feat: implement gemma_cpp engine lifecycle and inference methods Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/engine/gemma_cpp.py | 76 +++++++++++++++--- tests/engine/test_gemma_cpp.py | 125 +++++++++++++++++++++++++++++ 2 files changed, 192 insertions(+), 9 deletions(-) diff --git a/src/openjarvis/engine/gemma_cpp.py b/src/openjarvis/engine/gemma_cpp.py index a235417b..4e2f8245 100644 --- a/src/openjarvis/engine/gemma_cpp.py +++ b/src/openjarvis/engine/gemma_cpp.py @@ -14,6 +14,12 @@ from openjarvis.engine._base import InferenceEngine, estimate_prompt_tokens logger = logging.getLogger(__name__) +def _import_pygemma(): + """Import and return the pygemma.Gemma class. Raises ImportError if unavailable.""" + from pygemma import Gemma + return Gemma + + @EngineRegistry.register("gemma_cpp") class GemmaCppEngine(InferenceEngine): """gemma.cpp backend via pygemma pybind11 bindings (in-process, CPU).""" @@ -64,6 +70,35 @@ class GemmaCppEngine(InferenceEngine): parts.append("model\n") return "".join(parts) + def _ensure_loaded(self) -> None: + """Lazy model loading — called before inference.""" + if self._gemma is None: + if not self._model_path: + raise FileNotFoundError( + "gemma.cpp model_path not configured. Download weights " + "from Kaggle and set GEMMA_CPP_MODEL_PATH or configure " + "[engine.gemma_cpp] in ~/.openjarvis/config.toml" + ) + if not self._tokenizer_path: + raise FileNotFoundError( + "gemma.cpp tokenizer_path not configured. Set " + "GEMMA_CPP_TOKENIZER_PATH or configure " + "[engine.gemma_cpp] in ~/.openjarvis/config.toml" + ) + Gemma = _import_pygemma() + self._gemma = Gemma() + self._gemma.load_model( + self._tokenizer_path, self._model_path, self._model_type + ) + + def prepare(self, model: str) -> None: + """Load model into memory.""" + self._ensure_loaded() + + def close(self) -> None: + """Unload model and free memory.""" + self._gemma = None + def generate( self, messages: Sequence[Message], @@ -73,7 +108,31 @@ class GemmaCppEngine(InferenceEngine): max_tokens: int = 1024, **kwargs: Any, ) -> Dict[str, Any]: - raise NotImplementedError # implemented in Task 4 + self._ensure_loaded() + if model != self._model_type: + logger.warning( + "gemma_cpp: requested model %r but loaded model is %r; " + "proceeding with loaded model", + model, self._model_type, + ) + prompt = self._messages_to_prompt(messages) + try: + raw = self._gemma.completion(prompt) + except Exception as exc: + raise RuntimeError(f"gemma.cpp inference failed: {exc}") from exc + + prompt_tokens = estimate_prompt_tokens(messages) + completion_tokens = max(1, len(raw) // 4) + return { + "content": raw, + "usage": { + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "total_tokens": prompt_tokens + completion_tokens, + }, + "model": self._model_type, + "finish_reason": "stop", + } async def stream( self, @@ -84,8 +143,13 @@ class GemmaCppEngine(InferenceEngine): max_tokens: int = 1024, **kwargs: Any, ) -> AsyncIterator[str]: - raise NotImplementedError # implemented in Task 4 - yield "" # pragma: no cover + self._ensure_loaded() + prompt = self._messages_to_prompt(messages) + try: + raw = self._gemma.completion(prompt) + except Exception as exc: + raise RuntimeError(f"gemma.cpp inference failed: {exc}") from exc + yield raw def list_models(self) -> List[str]: raise NotImplementedError # implemented in Task 5 @@ -93,11 +157,5 @@ class GemmaCppEngine(InferenceEngine): def health(self) -> bool: raise NotImplementedError # implemented in Task 5 - def close(self) -> None: - pass # implemented in Task 4 - - def prepare(self, model: str) -> None: - pass # implemented in Task 4 - __all__ = ["GemmaCppEngine"] diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py index 91f8bcfb..82d6b146 100644 --- a/tests/engine/test_gemma_cpp.py +++ b/tests/engine/test_gemma_cpp.py @@ -90,3 +90,128 @@ class TestMessagesToPrompt: "Rule 1\n\nRule 2\n\nGo\n" "model\n" ) + + +from unittest.mock import MagicMock, patch + + +class TestGemmaCppLifecycle: + def _make_engine(self, **kwargs): + from openjarvis.engine.gemma_cpp import GemmaCppEngine + defaults = { + "model_path": "/fake/model.sbs", + "tokenizer_path": "/fake/tokenizer.spm", + "model_type": "2b-it", + } + defaults.update(kwargs) + return GemmaCppEngine(**defaults) + + @patch("openjarvis.engine.gemma_cpp._import_pygemma") + def test_prepare_loads_model(self, mock_import) -> None: + mock_gemma_cls = MagicMock() + mock_import.return_value = mock_gemma_cls + engine = self._make_engine() + engine.prepare("2b-it") + mock_gemma_cls.assert_called_once() + mock_gemma_cls.return_value.load_model.assert_called_once_with( + "/fake/tokenizer.spm", "/fake/model.sbs", "2b-it" + ) + + @patch("openjarvis.engine.gemma_cpp._import_pygemma") + def test_prepare_idempotent(self, mock_import) -> None: + mock_gemma_cls = MagicMock() + mock_import.return_value = mock_gemma_cls + engine = self._make_engine() + engine.prepare("2b-it") + engine.prepare("2b-it") + # Only loaded once + assert mock_gemma_cls.call_count == 1 + + @patch("openjarvis.engine.gemma_cpp._import_pygemma") + def test_close_unloads_model(self, mock_import) -> None: + mock_gemma_cls = MagicMock() + mock_import.return_value = mock_gemma_cls + engine = self._make_engine() + engine.prepare("2b-it") + assert engine._gemma is not None + engine.close() + assert engine._gemma is None + + @patch("openjarvis.engine.gemma_cpp._import_pygemma") + def test_generate_returns_dict(self, mock_import) -> None: + mock_gemma_cls = MagicMock() + mock_gemma_instance = MagicMock() + mock_gemma_instance.completion.return_value = "The answer is 4." + mock_gemma_cls.return_value = mock_gemma_instance + mock_import.return_value = mock_gemma_cls + + engine = self._make_engine() + result = engine.generate( + [Message(role=Role.USER, content="What is 2+2?")], + model="2b-it", + ) + assert result["content"] == "The answer is 4." + assert "usage" in result + assert result["usage"]["prompt_tokens"] > 0 + assert result["usage"]["completion_tokens"] > 0 + assert result["usage"]["total_tokens"] > 0 + assert result["model"] == "2b-it" + assert result["finish_reason"] == "stop" + + @patch("openjarvis.engine.gemma_cpp._import_pygemma") + def test_generate_warns_on_model_mismatch(self, mock_import) -> None: + mock_gemma_cls = MagicMock() + mock_gemma_instance = MagicMock() + mock_gemma_instance.completion.return_value = "ok" + mock_gemma_cls.return_value = mock_gemma_instance + mock_import.return_value = mock_gemma_cls + + engine = self._make_engine(model_type="2b-it") + from openjarvis.engine import gemma_cpp as gc_mod + with patch.object(gc_mod.logger, "warning") as mock_warn: + engine.generate( + [Message(role=Role.USER, content="Hi")], + model="9b-it", + ) + mock_warn.assert_called_once() + + @patch("openjarvis.engine.gemma_cpp._import_pygemma") + def test_generate_wraps_runtime_error(self, mock_import) -> None: + mock_gemma_cls = MagicMock() + mock_gemma_instance = MagicMock() + mock_gemma_instance.completion.side_effect = Exception("segfault") + mock_gemma_cls.return_value = mock_gemma_instance + mock_import.return_value = mock_gemma_cls + + engine = self._make_engine() + with pytest.raises(RuntimeError, match="gemma.cpp inference failed"): + engine.generate( + [Message(role=Role.USER, content="Hi")], + model="2b-it", + ) + + +class TestGemmaCppStream: + @patch("openjarvis.engine.gemma_cpp._import_pygemma") + @pytest.mark.asyncio + async def test_stream_yields_content(self, mock_import) -> None: + mock_gemma_cls = MagicMock() + mock_gemma_instance = MagicMock() + mock_gemma_instance.completion.return_value = "streamed output" + mock_gemma_cls.return_value = mock_gemma_instance + mock_import.return_value = mock_gemma_cls + + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( + model_path="/fake/model.sbs", + tokenizer_path="/fake/tokenizer.spm", + model_type="2b-it", + ) + chunks = [] + async for chunk in engine.stream( + [Message(role=Role.USER, content="Hi")], + model="2b-it", + ): + chunks.append(chunk) + assert len(chunks) == 1 + assert chunks[0] == "streamed output" From 8b0fd702c4e246251226b0a8af9c017dba129816 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:39:38 -0700 Subject: [PATCH 82/92] feat: implement gemma_cpp health checks and model listing Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/engine/gemma_cpp.py | 21 ++++++++- tests/engine/test_gemma_cpp.py | 76 ++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 2 deletions(-) diff --git a/src/openjarvis/engine/gemma_cpp.py b/src/openjarvis/engine/gemma_cpp.py index 4e2f8245..7cf71658 100644 --- a/src/openjarvis/engine/gemma_cpp.py +++ b/src/openjarvis/engine/gemma_cpp.py @@ -151,11 +151,28 @@ class GemmaCppEngine(InferenceEngine): raise RuntimeError(f"gemma.cpp inference failed: {exc}") from exc yield raw + def _paths_valid(self) -> bool: + """Check that model and tokenizer paths are configured and exist.""" + return bool( + self._model_path + and self._tokenizer_path + and os.path.isfile(self._model_path) + and os.path.isfile(self._tokenizer_path) + ) + def list_models(self) -> List[str]: - raise NotImplementedError # implemented in Task 5 + if self._model_type and self._paths_valid(): + return [self._model_type] + return [] def health(self) -> bool: - raise NotImplementedError # implemented in Task 5 + if not self._paths_valid(): + return False + try: + _import_pygemma() + return True + except ImportError: + return False __all__ = ["GemmaCppEngine"] diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py index 82d6b146..0cbb5b64 100644 --- a/tests/engine/test_gemma_cpp.py +++ b/tests/engine/test_gemma_cpp.py @@ -215,3 +215,79 @@ class TestGemmaCppStream: chunks.append(chunk) assert len(chunks) == 1 assert chunks[0] == "streamed output" + + +class TestGemmaCppHealth: + def test_health_true_when_configured_and_files_exist(self, tmp_path) -> None: + model_file = tmp_path / "model.sbs" + tokenizer_file = tmp_path / "tokenizer.spm" + model_file.write_text("fake") + tokenizer_file.write_text("fake") + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( + model_path=str(model_file), + tokenizer_path=str(tokenizer_file), + model_type="2b-it", + ) + with patch("openjarvis.engine.gemma_cpp._import_pygemma"): + assert engine.health() is True + + def test_health_false_when_files_missing(self) -> None: + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( + model_path="/nonexistent/model.sbs", + tokenizer_path="/nonexistent/tokenizer.spm", + model_type="2b-it", + ) + assert engine.health() is False + + def test_health_false_when_unconfigured(self) -> None: + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine() + assert engine.health() is False + + def test_health_false_when_pygemma_missing(self, tmp_path) -> None: + model_file = tmp_path / "model.sbs" + tokenizer_file = tmp_path / "tokenizer.spm" + model_file.write_text("fake") + tokenizer_file.write_text("fake") + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( + model_path=str(model_file), + tokenizer_path=str(tokenizer_file), + model_type="2b-it", + ) + with patch( + "openjarvis.engine.gemma_cpp._import_pygemma", + side_effect=ImportError("no pygemma"), + ): + assert engine.health() is False + + +class TestGemmaCppListModels: + def test_list_models_configured(self, tmp_path) -> None: + model_file = tmp_path / "model.sbs" + tokenizer_file = tmp_path / "tokenizer.spm" + model_file.write_text("fake") + tokenizer_file.write_text("fake") + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( + model_path=str(model_file), + tokenizer_path=str(tokenizer_file), + model_type="2b-it", + ) + assert engine.list_models() == ["2b-it"] + + def test_list_models_unconfigured(self) -> None: + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine() + assert engine.list_models() == [] + + def test_list_models_files_missing(self) -> None: + from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( + model_path="/nonexistent/model.sbs", + tokenizer_path="/nonexistent/tokenizer.spm", + model_type="2b-it", + ) + assert engine.list_models() == [] From 411e8229169054150009003a87b7eddbef7dc957 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:40:24 -0700 Subject: [PATCH 83/92] test: add config resolution tests for gemma_cpp engine Co-Authored-By: Claude Sonnet 4.6 --- tests/engine/test_gemma_cpp.py | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py index 0cbb5b64..f7dffe22 100644 --- a/tests/engine/test_gemma_cpp.py +++ b/tests/engine/test_gemma_cpp.py @@ -291,3 +291,35 @@ class TestGemmaCppListModels: model_type="2b-it", ) assert engine.list_models() == [] + + +class TestGemmaCppConfigResolution: + def test_explicit_args_take_priority(self) -> None: + from openjarvis.engine.gemma_cpp import GemmaCppEngine + with patch.dict(os.environ, {"GEMMA_CPP_MODEL_PATH": "/env/model"}): + engine = GemmaCppEngine(model_path="/explicit/model") + assert engine._model_path == "/explicit/model" + + def test_env_vars_fallback(self) -> None: + from openjarvis.engine.gemma_cpp import GemmaCppEngine + env = { + "GEMMA_CPP_MODEL_PATH": "/env/model.sbs", + "GEMMA_CPP_TOKENIZER_PATH": "/env/tokenizer.spm", + "GEMMA_CPP_MODEL_TYPE": "9b-it", + "GEMMA_CPP_NUM_THREADS": "8", + } + with patch.dict(os.environ, env): + engine = GemmaCppEngine() + assert engine._model_path == "/env/model.sbs" + assert engine._tokenizer_path == "/env/tokenizer.spm" + assert engine._model_type == "9b-it" + assert engine._num_threads == 8 + + def test_defaults_when_nothing_set(self) -> None: + from openjarvis.engine.gemma_cpp import GemmaCppEngine + with patch.dict(os.environ, {}, clear=True): + engine = GemmaCppEngine() + assert engine._model_path == "" + assert engine._tokenizer_path == "" + assert engine._model_type == "" + assert engine._num_threads == 0 From ae12d925bb8a949a841c4b2e9ad8aa6b9a634ace Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 12:46:06 -0700 Subject: [PATCH 84/92] fix(traces): allow TraceStore SQLite access from worker threads AgenticRunner dispatches _run_body() to a ThreadPoolExecutor when a task environment is present (for Playwright compatibility). The TraceStore connection was created on the main thread, causing "SQLite objects created in a thread can only be used in that same thread" on the first agentic eval query. Pass check_same_thread=False to sqlite3.connect(), consistent with SchedulerStore, AgentManager, SessionStore, and TelemetryStore which already use this flag. Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/traces/store.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/openjarvis/traces/store.py b/src/openjarvis/traces/store.py index 0efce8a4..9c4ceb66 100644 --- a/src/openjarvis/traces/store.py +++ b/src/openjarvis/traces/store.py @@ -81,7 +81,11 @@ class TraceStore: def __init__(self, db_path: str | Path) -> None: self._db_path = str(db_path) - self._conn = sqlite3.connect(self._db_path) + # check_same_thread=False is safe with WAL mode. The + # AgenticRunner dispatches agent work to a ThreadPoolExecutor + # (for Playwright compat), so trace writes may originate from + # a different thread than the one that opened the connection. + self._conn = sqlite3.connect(self._db_path, check_same_thread=False) self._conn.execute("PRAGMA journal_mode=WAL") self._conn.execute(_CREATE_TRACES) self._conn.execute(_CREATE_STEPS) From d08360ca214cbcf473642900ab2175b49e31ed6a Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 13:15:06 -0700 Subject: [PATCH 85/92] feat: wire gemma_cpp engine into discovery and optional imports Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/engine/__init__.py | 2 +- src/openjarvis/engine/_discovery.py | 12 ++++++++++ tests/engine/test_gemma_cpp.py | 34 +++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+), 1 deletion(-) diff --git a/src/openjarvis/engine/__init__.py b/src/openjarvis/engine/__init__.py index b6f79eae..90486fe6 100644 --- a/src/openjarvis/engine/__init__.py +++ b/src/openjarvis/engine/__init__.py @@ -15,7 +15,7 @@ from openjarvis.engine._base import ( from openjarvis.engine._discovery import discover_engines, discover_models, get_engine # Optional engines — only register if their SDK deps are present -for _optional in ("cloud", "litellm"): +for _optional in ("cloud", "litellm", "gemma_cpp"): try: importlib.import_module(f".{_optional}", __name__) except ImportError: diff --git a/src/openjarvis/engine/_discovery.py b/src/openjarvis/engine/_discovery.py index 4e2fc695..83021858 100644 --- a/src/openjarvis/engine/_discovery.py +++ b/src/openjarvis/engine/_discovery.py @@ -25,12 +25,24 @@ _HOST_MAP: Dict[str, str | None] = { "apple_fm": "apple_fm_host", "cloud": None, "litellm": None, + "gemma_cpp": None, } def _make_engine(key: str, config: JarvisConfig) -> InferenceEngine: """Instantiate a registered engine with the appropriate config host.""" cls = EngineRegistry.get(key) + + # gemma_cpp: pass config fields instead of host + if key == "gemma_cpp": + cfg = config.engine.gemma_cpp + return cls( + model_path=cfg.model_path or None, + tokenizer_path=cfg.tokenizer_path or None, + model_type=cfg.model_type or None, + num_threads=cfg.num_threads, + ) + host_attr = _HOST_MAP.get(key) if host_attr is not None: host = getattr(config.engine, host_attr, None) diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py index f7dffe22..34d7e7d6 100644 --- a/tests/engine/test_gemma_cpp.py +++ b/tests/engine/test_gemma_cpp.py @@ -323,3 +323,37 @@ class TestGemmaCppConfigResolution: assert engine._tokenizer_path == "" assert engine._model_type == "" assert engine._num_threads == 0 + + +class TestGemmaCppDiscovery: + def test_host_map_contains_gemma_cpp(self) -> None: + from openjarvis.engine._discovery import _HOST_MAP + assert "gemma_cpp" in _HOST_MAP + assert _HOST_MAP["gemma_cpp"] is None + + def test_make_engine_passes_config(self) -> None: + from openjarvis.core.config import GemmaCppEngineConfig, JarvisConfig + from openjarvis.core.registry import EngineRegistry + from openjarvis.engine._discovery import _make_engine + from openjarvis.engine.gemma_cpp import GemmaCppEngine + + EngineRegistry.register_value("gemma_cpp", GemmaCppEngine) + config = JarvisConfig() + config.engine.gemma_cpp = GemmaCppEngineConfig( + model_path="/cfg/model.sbs", + tokenizer_path="/cfg/tokenizer.spm", + model_type="9b-it", + num_threads=4, + ) + engine = _make_engine("gemma_cpp", config) + assert engine._model_path == "/cfg/model.sbs" + assert engine._tokenizer_path == "/cfg/tokenizer.spm" + assert engine._model_type == "9b-it" + assert engine._num_threads == 4 + + def test_registry_contains_gemma_cpp(self) -> None: + from openjarvis.core.registry import EngineRegistry + from openjarvis.engine.gemma_cpp import GemmaCppEngine + + EngineRegistry.register_value("gemma_cpp", GemmaCppEngine) + assert EngineRegistry.contains("gemma_cpp") From 16025571fd3bd2c1188226f2a912a5391c1ec436 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 13:17:16 -0700 Subject: [PATCH 86/92] style: fix lint issues in gemma_cpp engine Move mid-file imports to top of test file to resolve E402 violations and apply ruff formatting to both gemma_cpp.py and test_gemma_cpp.py. Co-Authored-By: Claude Sonnet 4.6 --- src/openjarvis/engine/gemma_cpp.py | 27 +++++++++------------------ tests/engine/test_gemma_cpp.py | 27 +++++++++++++++++++-------- 2 files changed, 28 insertions(+), 26 deletions(-) diff --git a/src/openjarvis/engine/gemma_cpp.py b/src/openjarvis/engine/gemma_cpp.py index 7cf71658..1fb09bd9 100644 --- a/src/openjarvis/engine/gemma_cpp.py +++ b/src/openjarvis/engine/gemma_cpp.py @@ -17,6 +17,7 @@ logger = logging.getLogger(__name__) def _import_pygemma(): """Import and return the pygemma.Gemma class. Raises ImportError if unavailable.""" from pygemma import Gemma + return Gemma @@ -33,18 +34,11 @@ class GemmaCppEngine(InferenceEngine): model_type: str | None = None, num_threads: int = 0, ) -> None: - self._model_path = ( - model_path - or os.environ.get("GEMMA_CPP_MODEL_PATH", "") - ) - self._tokenizer_path = ( - tokenizer_path - or os.environ.get("GEMMA_CPP_TOKENIZER_PATH", "") - ) - self._model_type = ( - model_type - or os.environ.get("GEMMA_CPP_MODEL_TYPE", "") + self._model_path = model_path or os.environ.get("GEMMA_CPP_MODEL_PATH", "") + self._tokenizer_path = tokenizer_path or os.environ.get( + "GEMMA_CPP_TOKENIZER_PATH", "" ) + self._model_type = model_type or os.environ.get("GEMMA_CPP_MODEL_TYPE", "") self._num_threads = num_threads or int( os.environ.get("GEMMA_CPP_NUM_THREADS", "0") ) @@ -58,15 +52,11 @@ class GemmaCppEngine(InferenceEngine): if msg.role == Role.SYSTEM: system_prefix += msg.content + "\n\n" elif msg.role == Role.USER: - content = ( - system_prefix + msg.content if system_prefix else msg.content - ) + content = system_prefix + msg.content if system_prefix else msg.content system_prefix = "" parts.append(f"user\n{content}\n") elif msg.role == Role.ASSISTANT: - parts.append( - f"model\n{msg.content}\n" - ) + parts.append(f"model\n{msg.content}\n") parts.append("model\n") return "".join(parts) @@ -113,7 +103,8 @@ class GemmaCppEngine(InferenceEngine): logger.warning( "gemma_cpp: requested model %r but loaded model is %r; " "proceeding with loaded model", - model, self._model_type, + model, + self._model_type, ) prompt = self._messages_to_prompt(messages) try: diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py index 34d7e7d6..ea3dbb22 100644 --- a/tests/engine/test_gemma_cpp.py +++ b/tests/engine/test_gemma_cpp.py @@ -3,10 +3,12 @@ from __future__ import annotations import os +from unittest.mock import MagicMock, patch import pytest from openjarvis.core.config import EngineConfig, GemmaCppEngineConfig +from openjarvis.core.types import Message, Role class TestGemmaCppEngineConfig: @@ -23,13 +25,11 @@ class TestGemmaCppEngineConfig: assert isinstance(ec.gemma_cpp, GemmaCppEngineConfig) -from openjarvis.core.types import Message, Role - - class TestMessagesToPrompt: def _make_engine(self): """Create engine with no paths (won't load model, just test formatting).""" from openjarvis.engine.gemma_cpp import GemmaCppEngine + return GemmaCppEngine() def test_single_user_message(self) -> None: @@ -37,8 +37,7 @@ class TestMessagesToPrompt: msgs = [Message(role=Role.USER, content="Hello")] result = engine._messages_to_prompt(msgs) assert result == ( - "user\nHello\n" - "model\n" + "user\nHello\nmodel\n" ) def test_system_folded_into_user(self) -> None: @@ -78,6 +77,7 @@ class TestMessagesToPrompt: assert result == "model\n" def test_multiple_system_messages_concatenated(self) -> None: + engine = self._make_engine() msgs = [ Message(role=Role.SYSTEM, content="Rule 1"), @@ -92,12 +92,10 @@ class TestMessagesToPrompt: ) -from unittest.mock import MagicMock, patch - - class TestGemmaCppLifecycle: def _make_engine(self, **kwargs): from openjarvis.engine.gemma_cpp import GemmaCppEngine + defaults = { "model_path": "/fake/model.sbs", "tokenizer_path": "/fake/tokenizer.spm", @@ -168,6 +166,7 @@ class TestGemmaCppLifecycle: engine = self._make_engine(model_type="2b-it") from openjarvis.engine import gemma_cpp as gc_mod + with patch.object(gc_mod.logger, "warning") as mock_warn: engine.generate( [Message(role=Role.USER, content="Hi")], @@ -202,6 +201,7 @@ class TestGemmaCppStream: mock_import.return_value = mock_gemma_cls from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( model_path="/fake/model.sbs", tokenizer_path="/fake/tokenizer.spm", @@ -224,6 +224,7 @@ class TestGemmaCppHealth: model_file.write_text("fake") tokenizer_file.write_text("fake") from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( model_path=str(model_file), tokenizer_path=str(tokenizer_file), @@ -234,6 +235,7 @@ class TestGemmaCppHealth: def test_health_false_when_files_missing(self) -> None: from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( model_path="/nonexistent/model.sbs", tokenizer_path="/nonexistent/tokenizer.spm", @@ -243,6 +245,7 @@ class TestGemmaCppHealth: def test_health_false_when_unconfigured(self) -> None: from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine() assert engine.health() is False @@ -252,6 +255,7 @@ class TestGemmaCppHealth: model_file.write_text("fake") tokenizer_file.write_text("fake") from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( model_path=str(model_file), tokenizer_path=str(tokenizer_file), @@ -271,6 +275,7 @@ class TestGemmaCppListModels: model_file.write_text("fake") tokenizer_file.write_text("fake") from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( model_path=str(model_file), tokenizer_path=str(tokenizer_file), @@ -280,11 +285,13 @@ class TestGemmaCppListModels: def test_list_models_unconfigured(self) -> None: from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine() assert engine.list_models() == [] def test_list_models_files_missing(self) -> None: from openjarvis.engine.gemma_cpp import GemmaCppEngine + engine = GemmaCppEngine( model_path="/nonexistent/model.sbs", tokenizer_path="/nonexistent/tokenizer.spm", @@ -296,12 +303,14 @@ class TestGemmaCppListModels: class TestGemmaCppConfigResolution: def test_explicit_args_take_priority(self) -> None: from openjarvis.engine.gemma_cpp import GemmaCppEngine + with patch.dict(os.environ, {"GEMMA_CPP_MODEL_PATH": "/env/model"}): engine = GemmaCppEngine(model_path="/explicit/model") assert engine._model_path == "/explicit/model" def test_env_vars_fallback(self) -> None: from openjarvis.engine.gemma_cpp import GemmaCppEngine + env = { "GEMMA_CPP_MODEL_PATH": "/env/model.sbs", "GEMMA_CPP_TOKENIZER_PATH": "/env/tokenizer.spm", @@ -317,6 +326,7 @@ class TestGemmaCppConfigResolution: def test_defaults_when_nothing_set(self) -> None: from openjarvis.engine.gemma_cpp import GemmaCppEngine + with patch.dict(os.environ, {}, clear=True): engine = GemmaCppEngine() assert engine._model_path == "" @@ -328,6 +338,7 @@ class TestGemmaCppConfigResolution: class TestGemmaCppDiscovery: def test_host_map_contains_gemma_cpp(self) -> None: from openjarvis.engine._discovery import _HOST_MAP + assert "gemma_cpp" in _HOST_MAP assert _HOST_MAP["gemma_cpp"] is None From e4d036dd50a772a9f596646273f9741d65b83acf Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 13:18:13 -0700 Subject: [PATCH 87/92] test: add live integration test stubs for gemma_cpp engine Co-Authored-By: Claude Sonnet 4.6 --- tests/engine/test_gemma_cpp.py | 53 ++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/tests/engine/test_gemma_cpp.py b/tests/engine/test_gemma_cpp.py index ea3dbb22..68a8dc48 100644 --- a/tests/engine/test_gemma_cpp.py +++ b/tests/engine/test_gemma_cpp.py @@ -368,3 +368,56 @@ class TestGemmaCppDiscovery: EngineRegistry.register_value("gemma_cpp", GemmaCppEngine) assert EngineRegistry.contains("gemma_cpp") + + +@pytest.mark.live +class TestGemmaCppLive: + """Integration tests — require pygemma and downloaded Gemma weights. + + Set GEMMA_CPP_MODEL_PATH, GEMMA_CPP_TOKENIZER_PATH, and + GEMMA_CPP_MODEL_TYPE env vars before running. + """ + + def _make_engine(self): + from openjarvis.engine.gemma_cpp import GemmaCppEngine + + return GemmaCppEngine() + + def test_real_inference_produces_output(self) -> None: + engine = self._make_engine() + result = engine.generate( + [Message(role=Role.USER, content="What is 2+2?")], + model=os.environ.get("GEMMA_CPP_MODEL_TYPE", "2b-it"), + ) + assert result["content"] + assert len(result["content"]) > 0 + + def test_prepare_and_close_lifecycle(self) -> None: + engine = self._make_engine() + model_type = os.environ.get("GEMMA_CPP_MODEL_TYPE", "2b-it") + engine.prepare(model_type) + assert engine._gemma is not None + engine.close() + assert engine._gemma is None + + @pytest.mark.asyncio + async def test_stream_yields_content(self) -> None: + engine = self._make_engine() + chunks = [] + async for chunk in engine.stream( + [Message(role=Role.USER, content="Say hello")], + model=os.environ.get("GEMMA_CPP_MODEL_TYPE", "2b-it"), + ): + chunks.append(chunk) + assert len(chunks) > 0 + assert all(len(c) > 0 for c in chunks) + + def test_token_counts_are_positive(self) -> None: + engine = self._make_engine() + result = engine.generate( + [Message(role=Role.USER, content="Tell me a joke")], + model=os.environ.get("GEMMA_CPP_MODEL_TYPE", "2b-it"), + ) + assert result["usage"]["prompt_tokens"] > 0 + assert result["usage"]["completion_tokens"] > 0 + assert result["usage"]["total_tokens"] > 0 From d44076db3f26e087d0cf0e411061ba0c79dd6ce9 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 13:22:13 -0700 Subject: [PATCH 88/92] fix: add pygemma API comment and model-mismatch warning to stream() - Document that pygemma v0.1.3 completion() does not accept temperature/max_tokens (params accepted for ABC compliance) - Add model-mismatch warning to stream() for consistency with generate() Co-Authored-By: Claude Opus 4.6 (1M context) --- src/openjarvis/engine/gemma_cpp.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/openjarvis/engine/gemma_cpp.py b/src/openjarvis/engine/gemma_cpp.py index 1fb09bd9..e19fb32a 100644 --- a/src/openjarvis/engine/gemma_cpp.py +++ b/src/openjarvis/engine/gemma_cpp.py @@ -108,6 +108,9 @@ class GemmaCppEngine(InferenceEngine): ) prompt = self._messages_to_prompt(messages) try: + # pygemma v0.1.3 completion() does not accept temperature/max_tokens; + # these params are accepted in the signature for ABC compliance but + # not forwarded until pygemma or a vendored wrapper supports them. raw = self._gemma.completion(prompt) except Exception as exc: raise RuntimeError(f"gemma.cpp inference failed: {exc}") from exc @@ -135,6 +138,13 @@ class GemmaCppEngine(InferenceEngine): **kwargs: Any, ) -> AsyncIterator[str]: self._ensure_loaded() + if model != self._model_type: + logger.warning( + "gemma_cpp: requested model %r but loaded model is %r; " + "proceeding with loaded model", + model, + self._model_type, + ) prompt = self._messages_to_prompt(messages) try: raw = self._gemma.completion(prompt) From 43b1240a440c6342de85bce1434afd562fcaa3c9 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 20:51:12 -0700 Subject: [PATCH 89/92] feat: remote engine host configuration via CLI (#104) Add jarvis config set command, --host flag to jarvis init, and improved error messages for configuring remote LLM engine endpoints. Closes #104. --- pyproject.toml | 1 + src/openjarvis/cli/ask.py | 124 ++++++++++++------ src/openjarvis/cli/config_cmd.py | 98 +++++++++++++++ src/openjarvis/cli/hints.py | 6 +- src/openjarvis/cli/init_cmd.py | 63 +++++++--- src/openjarvis/core/config.py | 102 ++++++++++++++- tests/cli/test_config_set.py | 154 +++++++++++++++++++++++ tests/cli/test_hints.py | 10 ++ tests/cli/test_init_host.py | 117 +++++++++++++++++ tests/core/test_config_key_validation.py | 54 ++++++++ uv.lock | 111 +++++++++++++++- 11 files changed, 776 insertions(+), 64 deletions(-) create mode 100644 tests/cli/test_config_set.py create mode 100644 tests/cli/test_init_host.py create mode 100644 tests/core/test_config_key_validation.py diff --git a/pyproject.toml b/pyproject.toml index 67e820eb..4af236ee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,6 +17,7 @@ dependencies = [ "python-telegram-bot>=22.6", "rich>=13", "tomli>=2.0; python_version < '3.11'", + "tomlkit>=0.12", ] [project.optional-dependencies] diff --git a/src/openjarvis/cli/ask.py b/src/openjarvis/cli/ask.py index 43014909..594338e4 100644 --- a/src/openjarvis/cli/ask.py +++ b/src/openjarvis/cli/ask.py @@ -44,7 +44,8 @@ def _get_memory_backend(config): if key == "sqlite": backend = MemoryRegistry.create( - key, db_path=config.memory.db_path, + key, + db_path=config.memory.db_path, ) else: backend = MemoryRegistry.create(key) @@ -106,8 +107,7 @@ def _run_agent( if not AgentRegistry.contains(agent_name): raise click.ClickException( - f"Unknown agent: {agent_name}. " - f"Available: {', '.join(AgentRegistry.keys())}" + f"Unknown agent: {agent_name}. Available: {', '.join(AgentRegistry.keys())}" ) agent_cls = AgentRegistry.get(agent_name) @@ -117,6 +117,7 @@ def _run_agent( if tool_names: # Trigger tool registration import openjarvis.tools # noqa: F401 + tools = _build_tools(tool_names, config, engine, model_name) # Build agent with appropriate kwargs @@ -149,7 +150,10 @@ def _run_agent( max_context_tokens=config.memory.context_max_tokens, ) context_messages = inject_context( - query_text, [], backend, config=ctx_cfg, + query_text, + [], + backend, + config=ctx_cfg, ) for msg in context_messages: ctx.conversation.add(msg) @@ -169,9 +173,7 @@ def _print_profile( ) -> None: """Print an inference telemetry profile table from EventBus history.""" # Collect all INFERENCE_END events (agents may fire multiple) - inf_events = [ - e for e in bus.history if e.event_type == EventType.INFERENCE_END - ] + inf_events = [e for e in bus.history if e.event_type == EventType.INFERENCE_END] if not inf_events: console.print("[dim]No inference telemetry recorded.[/dim]") return @@ -186,13 +188,15 @@ def _print_profile( for e in inf_events ) total_prompt = sum( - e.data.get("usage", {}).get("prompt_tokens", 0) - for e in inf_events + e.data.get("usage", {}).get("prompt_tokens", 0) for e in inf_events ) total_energy = sum(e.data.get("energy_joules", 0.0) for e in inf_events) avg_power = 0.0 - power_vals = [e.data.get("power_watts", 0.0) for e in inf_events - if e.data.get("power_watts", 0.0) > 0] + power_vals = [ + e.data.get("power_watts", 0.0) + for e in inf_events + if e.data.get("power_watts", 0.0) > 0 + ] if power_vals: avg_power = sum(power_vals) / len(power_vals) @@ -284,29 +288,42 @@ def _print_profile( @click.option("-m", "--model", "model_name", default=None, help="Model to use.") @click.option("-e", "--engine", "engine_key", default=None, help="Engine backend.") @click.option( - "-t", "--temperature", default=None, type=float, + "-t", + "--temperature", + default=None, + type=float, help="Sampling temperature (default: from config).", ) @click.option( - "--max-tokens", default=None, type=int, + "--max-tokens", + default=None, + type=int, help="Max tokens to generate (default: from config).", ) @click.option("--json", "output_json", is_flag=True, help="Output raw JSON result.") @click.option("--no-stream", is_flag=True, help="Disable streaming (sync mode).") @click.option( - "--no-context", is_flag=True, + "--no-context", + is_flag=True, help="Disable memory context injection.", ) @click.option( - "-a", "--agent", "agent_name", default=None, + "-a", + "--agent", + "agent_name", + default=None, help="Agent to use (simple, orchestrator).", ) @click.option( - "--tools", "tool_names", default=None, + "--tools", + "tool_names", + default=None, help="Comma-separated tool names to enable (e.g. calculator,think).", ) @click.option( - "--profile", "enable_profile", is_flag=True, + "--profile", + "enable_profile", + is_flag=True, help="Print inference telemetry profile (latency, tokens, energy, IPW).", ) def ask( @@ -377,7 +394,10 @@ def ask( " [cyan]ollama serve[/cyan] — start Ollama\n" " [cyan]vllm serve [/cyan] — start vLLM\n" " [cyan]llama-server -m [/cyan] — start llama.cpp\n\n" - "Or set OPENAI_API_KEY / ANTHROPIC_API_KEY for cloud inference." + "Or set OPENAI_API_KEY / ANTHROPIC_API_KEY for cloud inference.\n\n" + "[dim]To use a remote engine:[/dim]\n" + " [cyan]jarvis config set engine.ollama.host http://:11434[/cyan]\n" + " [dim]or[/dim] [cyan]export OLLAMA_HOST=http://:11434[/cyan]" ) sys.exit(1) @@ -385,6 +405,7 @@ def ask( # Apply security guardrails from openjarvis.security import setup_security + sec = setup_security(config, engine, bus) engine = sec.engine @@ -427,12 +448,14 @@ def ask( # the user would have gotten without the analyzer. if not user_set_max_tokens: suggested = adjust_tokens_for_model( - complexity_result.suggested_max_tokens, model_name, + complexity_result.suggested_max_tokens, + model_name, ) max_tokens = max(suggested, config.intelligence.max_tokens) logger.debug( "Using complexity-suggested max_tokens=%d (model=%s)", - max_tokens, model_name, + max_tokens, + model_name, ) # Agent mode @@ -444,8 +467,15 @@ def ask( ) try: result = _run_agent( - agent_name, query_text, engine, model_name, - parsed_tools, config, bus, temperature, max_tokens, + agent_name, + query_text, + engine, + model_name, + parsed_tools, + config, + bus, + temperature, + max_tokens, capability_policy=sec.capability_policy, ) except EngineConnectionError as exc: @@ -454,25 +484,33 @@ def ask( sys.exit(1) if output_json: - click.echo(json_mod.dumps({ - "content": result.content, - "turns": result.turns, - "tool_results": [ + click.echo( + json_mod.dumps( { - "tool_name": tr.tool_name, - "content": tr.content, - "success": tr.success, - } - for tr in result.tool_results - ], - }, indent=2)) + "content": result.content, + "turns": result.turns, + "tool_results": [ + { + "tool_name": tr.tool_name, + "content": tr.content, + "success": tr.success, + } + for tr in result.tool_results + ], + }, + indent=2, + ) + ) else: click.echo(result.content) if enable_profile: _print_profile( - bus, time.monotonic() - wall_start, - engine_name, model_name, console, + bus, + time.monotonic() - wall_start, + engine_name, + model_name, + console, complexity_result=complexity_result, ) @@ -493,17 +531,18 @@ def ask( ContextConfig, inject_context, ) + backend = _get_memory_backend(config) if backend is not None: ctx_cfg = ContextConfig( top_k=config.memory.context_top_k, min_score=config.memory.context_min_score, - max_context_tokens=( - config.memory.context_max_tokens - ), + max_context_tokens=(config.memory.context_max_tokens), ) messages = inject_context( - query_text, messages, backend, + query_text, + messages, + backend, config=ctx_cfg, ) except Exception as exc: @@ -531,8 +570,11 @@ def ask( if enable_profile: _print_profile( - bus, time.monotonic() - wall_start, - engine_name, model_name, console, + bus, + time.monotonic() - wall_start, + engine_name, + model_name, + console, complexity_result=complexity_result, ) diff --git a/src/openjarvis/cli/config_cmd.py b/src/openjarvis/cli/config_cmd.py index 324a348a..43e5be3f 100644 --- a/src/openjarvis/cli/config_cmd.py +++ b/src/openjarvis/cli/config_cmd.py @@ -4,9 +4,11 @@ from __future__ import annotations import json import os +import re from pathlib import Path import click +import httpx from rich.console import Console from rich.panel import Panel from rich.syntax import Syntax @@ -274,4 +276,100 @@ def hardware() -> None: config.add_command(show_group, "show") +def _probe_engine_host(url: str, console: Console) -> None: + """Probe an engine host URL and print reachability status.""" + try: + resp = httpx.get(url.rstrip("/") + "/", timeout=2.0) + if resp.status_code < 500: + console.print(f" [green]Reachable[/green] ({url})") + else: + console.print( + f" [yellow]Warning:[/yellow] Host returned status " + f"{resp.status_code} — config saved anyway." + ) + except Exception: + console.print( + f" [yellow]Warning:[/yellow] Host unreachable ({url}) " + f"— config saved anyway." + ) + + +def _coerce_value(value: str, target_type: type) -> object: + """Coerce a CLI string value to the target Python type.""" + if target_type is bool: + low = value.lower() + if low in ("true", "1", "yes"): + return True + if low in ("false", "0", "no"): + return False + raise ValueError( + f"Invalid boolean value: {value!r} (expected: true/false, yes/no, 1/0)" + ) + if target_type is int: + return int(value) + if target_type is float: + return float(value) + return value + + +@click.command("set") +@click.argument("key") +@click.argument("value") +def set_config(key: str, value: str) -> None: + """Set a configuration value (e.g. jarvis config set engine.ollama.host URL).""" + import tomlkit + + from openjarvis.core.config import DEFAULT_CONFIG_DIR, validate_config_key + + console = Console(stderr=True) + + # Validate key + try: + target_type = validate_config_key(key) + except ValueError as exc: + console.print(f"[red]Error:[/red] {exc}") + raise SystemExit(1) + + # Coerce value + try: + typed_value = _coerce_value(value, target_type) + except (ValueError, TypeError) as exc: + console.print( + f"[red]Error:[/red] Cannot convert {value!r} to " + f"{target_type.__name__}: {exc}" + ) + raise SystemExit(1) + + # Load or create TOML document + config_path = Path( + os.environ.get("OPENJARVIS_CONFIG", DEFAULT_CONFIG_DIR / "config.toml") + ) + if config_path.exists(): + doc = tomlkit.parse(config_path.read_text()) + else: + doc = tomlkit.document() + config_path.parent.mkdir(parents=True, exist_ok=True) + + # Set nested key + parts = key.split(".") + current = doc + for part in parts[:-1]: + if part not in current: + current.add(part, tomlkit.table()) + current = current[part] + current[parts[-1]] = typed_value + + # Write back + config_path.write_text(tomlkit.dumps(doc)) + + console.print(f"[green]Set[/green] {key} = {value!r}") + + # Probe engine host if applicable + if re.match(r"^engine\.\w+\.host$", key): + _probe_engine_host(value, console) + + +config.add_command(set_config, "set") + + __all__ = ["config"] diff --git a/src/openjarvis/cli/hints.py b/src/openjarvis/cli/hints.py index 9c8617cf..c2b1e3cd 100644 --- a/src/openjarvis/cli/hints.py +++ b/src/openjarvis/cli/hints.py @@ -22,7 +22,11 @@ def hint_no_engine(engine_name: Optional[str] = None) -> str: f"[yellow]Hint:[/yellow] Engine '{name}' is not reachable.\n" f" Make sure the {name} server is running.\n" " Run [bold]jarvis doctor[/bold] to check all engines.\n" - " Run [bold]jarvis quickstart[/bold] for guided setup." + " Run [bold]jarvis quickstart[/bold] for guided setup.\n" + "\n" + " [dim]To use a remote engine:[/dim]\n" + f" [cyan]jarvis config set engine.{name}.host http://:[/cyan]\n" + f" [dim]or[/dim] [cyan]export OLLAMA_HOST=http://:11434[/cyan]" ) diff --git a/src/openjarvis/cli/init_cmd.py b/src/openjarvis/cli/init_cmd.py index 8d729b7e..a60f0217 100644 --- a/src/openjarvis/cli/init_cmd.py +++ b/src/openjarvis/cli/init_cmd.py @@ -6,6 +6,7 @@ from pathlib import Path from typing import Optional import click +import httpx from rich.console import Console from rich.markup import escape from rich.panel import Panel @@ -25,8 +26,14 @@ from openjarvis.core.config import ( # Engines supported by ``jarvis init --engine``. _SUPPORTED_ENGINES = [ - "ollama", "vllm", "sglang", "llamacpp", "mlx", "lmstudio", - "exo", "nexa", + "ollama", + "vllm", + "sglang", + "llamacpp", + "mlx", + "lmstudio", + "exo", + "nexa", ] @@ -70,7 +77,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: f" ollama pull {pull_model}\n" "\n" " 3. Try it out:\n" - " jarvis ask \"Hello\"\n" + ' jarvis ask "Hello"\n' "\n" " Run `jarvis doctor` to verify your setup." ), @@ -82,7 +89,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: " vllm serve Qwen/Qwen3-4B\n" "\n" " 2. Try it out:\n" - " jarvis ask \"Hello\"\n" + ' jarvis ask "Hello"\n' "\n" " Run `jarvis doctor` to verify your setup." ), @@ -94,7 +101,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: " llama-server -m path/to/model.gguf\n" "\n" " 2. Try it out:\n" - " jarvis ask \"Hello\"\n" + ' jarvis ask "Hello"\n' "\n" " Run `jarvis doctor` to verify your setup." ), @@ -106,7 +113,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: " python -m sglang.launch_server --model-path Qwen/Qwen3-8B\n" "\n" " 2. Try it out:\n" - " jarvis ask \"Hello\"\n" + ' jarvis ask "Hello"\n' "\n" " Run `jarvis doctor` to verify your setup." ), @@ -118,7 +125,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: " mlx_lm.server --model mlx-community/Qwen2.5-7B-4bit\n" "\n" " 2. Try it out:\n" - " jarvis ask \"Hello\"\n" + ' jarvis ask "Hello"\n' "\n" " Run `jarvis doctor` to verify your setup." ), @@ -131,7 +138,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: " 2. Load a model and start the local server (port 1234)\n" "\n" " 3. Try it out:\n" - " jarvis ask \"Hello\"\n" + ' jarvis ask "Hello"\n' "\n" " Run `jarvis doctor` to verify your setup." ), @@ -141,7 +148,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: " pip install exo\n" " exo\n\n" " 2. Try it out:\n" - " jarvis ask \"Hello\"\n\n" + ' jarvis ask "Hello"\n\n' " Run `jarvis doctor` to verify your setup." ), "nexa": ( @@ -150,7 +157,7 @@ def _next_steps_text(engine: str, model: str = "") -> str: " pip install nexaai\n" " nexa server\n\n" " 2. Try it out:\n" - " jarvis ask \"Hello\"\n\n" + ' jarvis ask "Hello"\n\n' " Run `jarvis doctor` to verify your setup." ), } @@ -177,6 +184,7 @@ def _quick_privacy_check(console: Console) -> None: def _do_download(engine: str, model: str, spec, console: Console) -> None: """Dispatch model download based on engine type.""" import os + if engine == "ollama": host = os.environ.get("OLLAMA_HOST", "http://localhost:11434").rstrip("/") ollama_pull(host, model, console) @@ -228,12 +236,18 @@ def _do_download(engine: str, model: str, spec, console: Console) -> None: @click.option( "--no-download", is_flag=True, default=False, help="Skip the model download prompt." ) +@click.option( + "--host", + default=None, + help="Remote engine host URL (e.g. http://192.168.1.50:11434).", +) def init( force: bool, config: Optional[Path], full_config: bool = False, engine: Optional[str] = None, no_download: bool = False, + host: Optional[str] = None, ) -> None: """Detect hardware and generate ~/.openjarvis/config.toml.""" console = Console() @@ -267,9 +281,7 @@ def init( console.print("[bold]Detecting running inference engines...[/bold]") running = _detect_running_engines() if running: - console.print( - f" Found running: [green]{', '.join(running)}[/green]" - ) + console.print(f" Found running: [green]{', '.join(running)}[/green]") else: console.print(" No running engines detected.") @@ -313,13 +325,31 @@ def init( default=default, ) + # Probe remote host if specified + if host: + console.print("\n[bold]Checking remote host...[/bold]") + try: + resp = httpx.get(host.rstrip("/") + "/", timeout=2.0) + if resp.status_code < 500: + console.print(f" [green]Reachable[/green] ({host})") + else: + console.print( + f" [yellow]Warning:[/yellow] Host returned status " + f"{resp.status_code} — writing config anyway." + ) + except Exception: + console.print( + f" [yellow]Warning:[/yellow] Host unreachable ({host}) " + f"— writing config anyway." + ) + if config: toml_content = config.read_text() else: if full_config: - toml_content = generate_default_toml(hw, engine=engine) + toml_content = generate_default_toml(hw, engine=engine, host=host) else: - toml_content = generate_minimal_toml(hw, engine=engine) + toml_content = generate_minimal_toml(hw, engine=engine, host=host) DEFAULT_CONFIG_DIR.mkdir(parents=True, exist_ok=True) if config: @@ -341,8 +371,7 @@ def init( soul_path = DEFAULT_CONFIG_DIR / "SOUL.md" if not soul_path.exists(): soul_path.write_text( - "# Agent Persona\n\n" - "You are Jarvis, a helpful personal AI assistant.\n" + "# Agent Persona\n\nYou are Jarvis, a helpful personal AI assistant.\n" ) memory_path = DEFAULT_CONFIG_DIR / "MEMORY.md" diff --git a/src/openjarvis/core/config.py b/src/openjarvis/core/config.py index 9a829fd4..d8f1f10f 100644 --- a/src/openjarvis/core/config.py +++ b/src/openjarvis/core/config.py @@ -1157,6 +1157,80 @@ class JarvisConfig: self.tools.storage = value +# --------------------------------------------------------------------------- +# Config key validation +# --------------------------------------------------------------------------- + +# Sections that users may set via ``jarvis config set``. +# ``hardware`` is auto-detected and not user-settable. +_SETTABLE_SECTIONS = frozenset(JarvisConfig.__dataclass_fields__.keys()) - {"hardware"} + + +def validate_config_key(dotted_key: str) -> type: + """Validate a dotted config key and return the leaf field's Python type. + + Raises :class:`ValueError` when the key does not map to a known field. + The function walks the ``JarvisConfig`` dataclass hierarchy using + ``dataclasses.fields()``. + + Examples:: + + validate_config_key("engine.ollama.host") # -> str + validate_config_key("intelligence.temperature") # -> float + """ + from dataclasses import fields as dc_fields + + parts = dotted_key.split(".") + if len(parts) < 2: + raise ValueError( + f"Config key must have at least two segments (e.g. engine.default), " + f"got: {dotted_key!r}" + ) + + if parts[0] not in _SETTABLE_SECTIONS: + raise ValueError( + f"Unknown config key: {dotted_key!r} " + f"(valid top-level sections: {sorted(_SETTABLE_SECTIONS)})" + ) + + # Walk the dataclass tree + current_cls = JarvisConfig + for i, part in enumerate(parts): + field_map = {f.name: f for f in dc_fields(current_cls)} + if part not in field_map: + path_so_far = ".".join(parts[: i + 1]) + raise ValueError( + f"Unknown config key: {dotted_key!r} " + f"(no field {part!r} at {path_so_far}; " + f"valid fields: {sorted(field_map.keys())})" + ) + fld = field_map[part] + # Resolve the type — unwrap Optional, etc. + fld_type = fld.type + if isinstance(fld_type, str): + # Evaluate forward references in the config module namespace + import openjarvis.core.config as _cfg_mod + + fld_type = eval(fld_type, vars(_cfg_mod)) # noqa: S307 + + if i == len(parts) - 1: + # Leaf — return the primitive type + return fld_type + else: + # Must be a nested dataclass + if not hasattr(fld_type, "__dataclass_fields__"): + path_so_far = ".".join(parts[: i + 1]) + raise ValueError( + f"Unknown config key: {dotted_key!r} " + f"({path_so_far} is a leaf of type {fld_type.__name__}, " + f"not a section)" + ) + current_cls = fld_type + + # Should not reach here, but satisfy type checker + raise ValueError(f"Unknown config key: {dotted_key!r}") + + # --------------------------------------------------------------------------- # TOML loading # --------------------------------------------------------------------------- @@ -1283,7 +1357,9 @@ def load_config(path: Optional[Path] = None) -> JarvisConfig: # --------------------------------------------------------------------------- -def generate_minimal_toml(hw: HardwareInfo, engine: str | None = None) -> str: +def generate_minimal_toml( + hw: HardwareInfo, engine: str | None = None, *, host: str | None = None +) -> str: """Render a minimal TOML config with only essential settings.""" engine = engine or recommend_engine(hw) model = recommend_model(hw, engine) @@ -1291,6 +1367,14 @@ def generate_minimal_toml(hw: HardwareInfo, engine: str | None = None) -> str: if hw.gpu: mem_label = "unified memory" if hw.gpu.vendor == "apple" else "VRAM" gpu_comment = f"\n# GPU: {hw.gpu.name} ({hw.gpu.vram_gb} GB {mem_label})" + if host: + engine_host_section = f'\n[engine.{engine}]\nhost = "{host}"\n' + else: + engine_host_section = ( + f"\n[engine.{engine}]\n" + f'# host = "http://localhost:11434" ' + f"# set to remote URL if engine runs elsewhere\n" + ) return f"""\ # OpenJarvis configuration # Hardware: {hw.cpu_brand} ({hw.cpu_count} cores, {hw.ram_gb} GB RAM){gpu_comment} @@ -1298,7 +1382,7 @@ def generate_minimal_toml(hw: HardwareInfo, engine: str | None = None) -> str: [engine] default = "{engine}" - +{engine_host_section} [intelligence] default_model = "{model}" @@ -1310,7 +1394,9 @@ enabled = ["code_interpreter", "web_search", "file_read", "shell_exec"] """ -def generate_default_toml(hw: HardwareInfo, engine: str | None = None) -> str: +def generate_default_toml( + hw: HardwareInfo, engine: str | None = None, *, host: str | None = None +) -> str: """Render a commented TOML string suitable for ``~/.openjarvis/config.toml``.""" engine = engine or recommend_engine(hw) model = recommend_model(hw, engine) @@ -1322,7 +1408,7 @@ def generate_default_toml(hw: HardwareInfo, engine: str | None = None) -> str: if model: model_comment = " # recommended for your hardware" - return f"""\ + result = f"""\ # OpenJarvis configuration # Generated by `jarvis init` # @@ -1517,6 +1603,13 @@ ssrf_protection = true # assistant_name = "Jarvis" # assistant_has_own_number = false """ + if host: + import re as _re + + pattern = _re.escape(f"[engine.{engine}]") + r"\nhost = \"[^\"]*\"" + replacement = f'[engine.{engine}]\\nhost = "{host}"' + result = _re.sub(pattern, replacement, result) + return result __all__ = [ @@ -1581,4 +1674,5 @@ __all__ = [ "load_config", "recommend_engine", "recommend_model", + "validate_config_key", ] diff --git a/tests/cli/test_config_set.py b/tests/cli/test_config_set.py new file mode 100644 index 00000000..54162256 --- /dev/null +++ b/tests/cli/test_config_set.py @@ -0,0 +1,154 @@ +"""Tests for ``jarvis config set`` command.""" + +from __future__ import annotations + +import os +from pathlib import Path +from unittest import mock + +from click.testing import CliRunner + +from openjarvis.cli import cli + + +class TestConfigSet: + def test_set_creates_config_file(self, tmp_path: Path) -> None: + """config set creates config.toml if it doesn't exist.""" + config_file = tmp_path / "config.toml" + env = {"OPENJARVIS_CONFIG": str(config_file)} + with mock.patch.dict(os.environ, env): + result = CliRunner().invoke( + cli, ["config", "set", "engine.default", "vllm"] + ) + assert result.exit_code == 0 + assert config_file.exists() + content = config_file.read_text() + assert "vllm" in content + + def test_set_engine_ollama_host(self, tmp_path: Path) -> None: + """config set writes engine.ollama.host correctly.""" + config_file = tmp_path / "config.toml" + config_file.write_text('[engine]\ndefault = "ollama"\n') + env = {"OPENJARVIS_CONFIG": str(config_file)} + with ( + mock.patch.dict(os.environ, env), + mock.patch("openjarvis.cli.config_cmd.httpx"), + ): + result = CliRunner().invoke( + cli, + ["config", "set", "engine.ollama.host", "http://192.168.1.50:11434"], + ) + assert result.exit_code == 0 + content = config_file.read_text() + assert "http://192.168.1.50:11434" in content + + def test_set_preserves_existing_keys(self, tmp_path: Path) -> None: + """config set preserves other keys in the file.""" + config_file = tmp_path / "config.toml" + config_file.write_text( + '[engine]\ndefault = "ollama"\n\n' + "[intelligence]\n" + 'default_model = "qwen2.5:3b"\n' + ) + env = {"OPENJARVIS_CONFIG": str(config_file)} + with mock.patch.dict(os.environ, env): + result = CliRunner().invoke( + cli, ["config", "set", "engine.default", "vllm"] + ) + assert result.exit_code == 0 + content = config_file.read_text() + assert "vllm" in content + assert "qwen2.5:3b" in content + + def test_set_invalid_key_rejected(self, tmp_path: Path) -> None: + """config set rejects unknown keys.""" + config_file = tmp_path / "config.toml" + config_file.write_text("") + env = {"OPENJARVIS_CONFIG": str(config_file)} + with mock.patch.dict(os.environ, env): + result = CliRunner().invoke( + cli, ["config", "set", "engine.olllama.host", "http://x:1234"] + ) + assert result.exit_code != 0 + assert "Unknown config key" in result.output + + def test_set_engine_host_probes_reachable(self, tmp_path: Path) -> None: + """config set probes engine host and reports reachability.""" + config_file = tmp_path / "config.toml" + config_file.write_text("") + env = {"OPENJARVIS_CONFIG": str(config_file)} + with ( + mock.patch.dict(os.environ, env), + mock.patch("openjarvis.cli.config_cmd.httpx") as mock_httpx, + ): + mock_httpx.get.return_value = mock.Mock(status_code=200) + result = CliRunner().invoke( + cli, + ["config", "set", "engine.ollama.host", "http://myserver:11434"], + ) + assert result.exit_code == 0 + assert "Reachable" in result.output + + def test_set_engine_host_probes_unreachable(self, tmp_path: Path) -> None: + """config set warns when engine host is unreachable, but still writes.""" + config_file = tmp_path / "config.toml" + config_file.write_text("") + env = {"OPENJARVIS_CONFIG": str(config_file)} + with ( + mock.patch.dict(os.environ, env), + mock.patch("openjarvis.cli.config_cmd.httpx") as mock_httpx, + ): + mock_httpx.get.side_effect = Exception("Connection refused") + result = CliRunner().invoke( + cli, + ["config", "set", "engine.ollama.host", "http://myserver:11434"], + ) + assert result.exit_code == 0 + output_lower = result.output.lower() + assert "unreachable" in output_lower or "warning" in output_lower + content = config_file.read_text() + assert "http://myserver:11434" in content + + def test_set_integer_value(self, tmp_path: Path) -> None: + """config set coerces integer values.""" + config_file = tmp_path / "config.toml" + config_file.write_text("") + env = {"OPENJARVIS_CONFIG": str(config_file)} + with mock.patch.dict(os.environ, env): + result = CliRunner().invoke( + cli, ["config", "set", "intelligence.max_tokens", "2048"] + ) + assert result.exit_code == 0 + content = config_file.read_text() + assert "2048" in content + + def test_set_float_value(self, tmp_path: Path) -> None: + """config set coerces float values.""" + config_file = tmp_path / "config.toml" + config_file.write_text("") + env = {"OPENJARVIS_CONFIG": str(config_file)} + with mock.patch.dict(os.environ, env): + result = CliRunner().invoke( + cli, ["config", "set", "intelligence.temperature", "0.9"] + ) + assert result.exit_code == 0 + content = config_file.read_text() + assert "0.9" in content + + def test_set_missing_args(self) -> None: + """config set with missing args shows usage error.""" + result = CliRunner().invoke(cli, ["config", "set"]) + assert result.exit_code != 0 + + def test_set_shows_confirmation(self, tmp_path: Path) -> None: + """config set prints a confirmation message.""" + config_file = tmp_path / "config.toml" + config_file.write_text("") + env = {"OPENJARVIS_CONFIG": str(config_file)} + with mock.patch.dict(os.environ, env): + result = CliRunner().invoke( + cli, ["config", "set", "engine.default", "vllm"] + ) + assert result.exit_code == 0 + assert "Set" in result.output + assert "engine.default" in result.output diff --git a/tests/cli/test_hints.py b/tests/cli/test_hints.py index 0ba4797a..67953d16 100644 --- a/tests/cli/test_hints.py +++ b/tests/cli/test_hints.py @@ -35,3 +35,13 @@ class TestHintFunctions: def test_hint_no_model_with_name(self): msg = hint_no_model("qwen3:8b") assert "qwen3:8b" in msg + + def test_hint_no_engine_includes_remote_tip(self): + msg = hint_no_engine() + assert "config set" in msg + assert "OLLAMA_HOST" in msg + + def test_hint_no_engine_with_name_includes_remote_tip(self): + msg = hint_no_engine("vllm") + assert "config set" in msg + assert "engine.vllm.host" in msg diff --git a/tests/cli/test_init_host.py b/tests/cli/test_init_host.py new file mode 100644 index 00000000..aed663dc --- /dev/null +++ b/tests/cli/test_init_host.py @@ -0,0 +1,117 @@ +"""Tests for ``jarvis init --host`` option.""" + +from __future__ import annotations + +from pathlib import Path +from unittest import mock + +from click.testing import CliRunner + +from openjarvis.cli import cli +from openjarvis.core.config import generate_default_toml, generate_minimal_toml + +_NO_DL = "--no-download" + + +class TestInitHost: + def test_init_host_writes_to_config(self, tmp_path: Path) -> None: + """jarvis init --host writes the host into config.toml.""" + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), + ): + result = CliRunner().invoke( + cli, + [ + "init", + "--engine", + "ollama", + "--host", + "http://192.168.1.50:11434", + _NO_DL, + ], + ) + assert result.exit_code == 0 + content = config_path.read_text() + assert "http://192.168.1.50:11434" in content + + def test_init_host_with_vllm(self, tmp_path: Path) -> None: + """jarvis init --host applies to the selected engine.""" + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), + ): + result = CliRunner().invoke( + cli, + ["init", "--engine", "vllm", "--host", "http://10.0.0.5:8000", _NO_DL], + ) + assert result.exit_code == 0 + content = config_path.read_text() + assert "http://10.0.0.5:8000" in content + + def test_init_host_probes_and_reports(self, tmp_path: Path) -> None: + """jarvis init --host shows reachability status.""" + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), + mock.patch("openjarvis.cli.init_cmd.httpx") as mock_httpx, + ): + mock_httpx.get.side_effect = Exception("Connection refused") + result = CliRunner().invoke( + cli, + ["init", "--engine", "ollama", "--host", "http://bad:11434", _NO_DL], + ) + assert result.exit_code == 0 + output_lower = result.output.lower() + assert "unreachable" in output_lower or "warning" in output_lower + + def test_init_without_host_still_works(self, tmp_path: Path) -> None: + """jarvis init without --host still produces valid config.""" + config_dir = tmp_path / ".openjarvis" + config_path = config_dir / "config.toml" + with ( + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_DIR", config_dir), + mock.patch("openjarvis.cli.init_cmd.DEFAULT_CONFIG_PATH", config_path), + mock.patch("openjarvis.cli.init_cmd.PrivacyScanner"), + ): + result = CliRunner().invoke(cli, ["init", "--engine", "ollama", _NO_DL]) + assert result.exit_code == 0 + content = config_path.read_text() + assert "[engine]" in content + + +class TestGenerateTomlHost: + def test_minimal_toml_with_host(self) -> None: + from openjarvis.core.config import HardwareInfo + + hw = HardwareInfo() + toml_str = generate_minimal_toml( + hw, engine="ollama", host="http://remote:11434" + ) + assert "http://remote:11434" in toml_str + assert "[engine.ollama]" in toml_str + + def test_minimal_toml_without_host_has_comment(self) -> None: + from openjarvis.core.config import HardwareInfo + + hw = HardwareInfo() + toml_str = generate_minimal_toml(hw, engine="ollama") + assert "# host" in toml_str + + def test_default_toml_with_host(self) -> None: + from openjarvis.core.config import HardwareInfo + + hw = HardwareInfo() + toml_str = generate_default_toml( + hw, engine="ollama", host="http://remote:11434" + ) + assert "http://remote:11434" in toml_str diff --git a/tests/core/test_config_key_validation.py b/tests/core/test_config_key_validation.py new file mode 100644 index 00000000..badc706e --- /dev/null +++ b/tests/core/test_config_key_validation.py @@ -0,0 +1,54 @@ +"""Tests for config key validation.""" + +from __future__ import annotations + +import pytest + +from openjarvis.core.config import validate_config_key + + +class TestValidateConfigKey: + def test_valid_engine_default(self): + field_type = validate_config_key("engine.default") + assert field_type is str + + def test_valid_engine_ollama_host(self): + field_type = validate_config_key("engine.ollama.host") + assert field_type is str + + def test_valid_intelligence_temperature(self): + field_type = validate_config_key("intelligence.temperature") + assert field_type is float + + def test_valid_intelligence_max_tokens(self): + field_type = validate_config_key("intelligence.max_tokens") + assert field_type is int + + def test_valid_agent_default_agent(self): + field_type = validate_config_key("agent.default_agent") + assert field_type is str + + def test_invalid_top_level_key(self): + with pytest.raises(ValueError, match="Unknown config key"): + validate_config_key("nonexistent.foo") + + def test_invalid_nested_key(self): + with pytest.raises(ValueError, match="Unknown config key"): + validate_config_key("engine.olllama.host") + + def test_invalid_leaf_key(self): + with pytest.raises(ValueError, match="Unknown config key"): + validate_config_key("engine.ollama.nonexistent") + + def test_empty_key(self): + with pytest.raises(ValueError): + validate_config_key("") + + def test_single_segment(self): + with pytest.raises(ValueError): + validate_config_key("engine") + + def test_hardware_key_rejected(self): + """Hardware is auto-detected, not user-settable.""" + with pytest.raises(ValueError, match="Unknown config key"): + validate_config_key("hardware.cpu_count") diff --git a/uv.lock b/uv.lock index da90bd67..e6821504 100644 --- a/uv.lock +++ b/uv.lock @@ -893,6 +893,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ae/3a/dbeec9d1ee0844c679f6bb5d6ad4e9f198b1224f4e7a32825f47f6192b0c/cffi-2.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9", size = 184195, upload-time = "2025-09-08T23:23:43.004Z" }, ] +[[package]] +name = "cfgv" +version = "3.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4e/b5/721b8799b04bf9afe054a3899c6cf4e880fcf8563cc71c15610242490a0c/cfgv-3.5.0.tar.gz", hash = "sha256:d5b1034354820651caa73ede66a6294d6e95c1b00acc5e9b098e917404669132", size = 7334, upload-time = "2025-11-19T20:55:51.612Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl", hash = "sha256:a8dc6b26ad22ff227d2634a65cb388215ce6cc96bbcc5cfde7641ae87e8dacc0", size = 7445, upload-time = "2025-11-19T20:55:50.744Z" }, +] + [[package]] name = "charset-normalizer" version = "3.4.4" @@ -1632,6 +1641,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" }, ] +[[package]] +name = "distlib" +version = "0.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/96/8e/709914eb2b5749865801041647dc7f4e6d00b549cfe88b65ca192995f07c/distlib-0.4.0.tar.gz", hash = "sha256:feec40075be03a04501a973d81f633735b4b69f98b05450592310c0f401a4e0d", size = 614605, upload-time = "2025-07-17T16:52:00.465Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl", hash = "sha256:9659f7d87e46584a30b5780e43ac7a2143098441670ff0a49d5f9034c54a6c16", size = 469047, upload-time = "2025-07-17T16:51:58.613Z" }, +] + [[package]] name = "distro" version = "1.9.0" @@ -2467,6 +2485,7 @@ wheels = [ name = "griffelib" version = "2.0.0" source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ad/06/eccbd311c9e2b3ca45dbc063b93134c57a1ccc7607c5e545264ad092c4a9/griffelib-2.0.0.tar.gz", hash = "sha256:e504d637a089f5cab9b5daf18f7645970509bf4f53eda8d79ed71cce8bd97934", size = 166312, upload-time = "2026-03-23T21:06:55.954Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/4d/51/c936033e16d12b627ea334aaaaf42229c37620d0f15593456ab69ab48161/griffelib-2.0.0-py3-none-any.whl", hash = "sha256:01284878c966508b6d6f1dbff9b6fa607bc062d8261c5c7253cb285b06422a7f", size = 142004, upload-time = "2026-02-09T19:09:40.561Z" }, ] @@ -2695,6 +2714,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a8/af/48ac8483240de756d2438c380746e7130d1c6f75802ef22f3c6d49982787/huggingface_hub-0.36.2-py3-none-any.whl", hash = "sha256:48f0c8eac16145dfce371e9d2d7772854a4f591bcb56c9cf548accf531d54270", size = 566395, upload-time = "2026-02-06T09:24:11.133Z" }, ] +[[package]] +name = "identify" +version = "2.6.18" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/46/c4/7fb4db12296cdb11893d61c92048fe617ee853f8523b9b296ac03b43757e/identify-2.6.18.tar.gz", hash = "sha256:873ac56a5e3fd63e7438a7ecbc4d91aca692eb3fefa4534db2b7913f3fc352fd", size = 99580, upload-time = "2026-03-15T18:39:50.319Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/46/33/92ef41c6fad0233e41d3d84ba8e8ad18d1780f1e5d99b3c683e6d7f98b63/identify-2.6.18-py2.py3-none-any.whl", hash = "sha256:8db9d3c8ea9079db92cafb0ebf97abdc09d52e97f4dcf773a2e694048b7cd737", size = 99394, upload-time = "2026-03-15T18:39:48.915Z" }, +] + [[package]] name = "idna" version = "3.11" @@ -4237,6 +4265,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/df/93/a7b983643d1253bb223234b5b226e69de6cda02b76cdca7770f684b795f5/ninja-1.13.0-py3-none-win_arm64.whl", hash = "sha256:3c0b40b1f0bba764644385319028650087b4c1b18cdfa6f45cb39a3669b81aa9", size = 290806, upload-time = "2025-08-11T15:10:18.018Z" }, ] +[[package]] +name = "nodeenv" +version = "1.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/24/bf/d1bda4f6168e0b2e9e5958945e01910052158313224ada5ce1fb2e1113b8/nodeenv-1.10.0.tar.gz", hash = "sha256:996c191ad80897d076bdfba80a41994c2b47c68e224c542b48feba42ba00f8bb", size = 55611, upload-time = "2025-12-20T14:08:54.006Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl", hash = "sha256:5bb13e3eed2923615535339b3c620e76779af4cb4c6a90deccc9e36b274d3827", size = 23438, upload-time = "2025-12-20T14:08:52.782Z" }, +] + [[package]] name = "numba" version = "0.61.2" @@ -4682,6 +4719,7 @@ dependencies = [ { name = "python-telegram-bot" }, { name = "rich" }, { name = "tomli", marker = "python_full_version < '3.11'" }, + { name = "tomlkit" }, ] [package.optional-dependencies] @@ -4734,6 +4772,7 @@ dashboard = [ ] dev = [ { name = "maturin" }, + { name = "pre-commit" }, { name = "pytest" }, { name = "pytest-asyncio" }, { name = "pytest-cov" }, @@ -4772,6 +4811,9 @@ inference-cloud = [ { name = "anthropic" }, { name = "openai" }, ] +inference-gemma = [ + { name = "pygemma" }, +] inference-google = [ { name = "google-genai" }, ] @@ -4895,7 +4937,9 @@ requires-dist = [ { name = "pdfplumber", marker = "extra == 'pdf'", specifier = ">=0.10" }, { name = "playwright", marker = "extra == 'browser'", specifier = ">=1.40" }, { name = "praw", marker = "extra == 'channel-reddit'", specifier = ">=7.0" }, + { name = "pre-commit", marker = "extra == 'dev'", specifier = ">=3.0" }, { name = "pydantic", marker = "extra == 'server'", specifier = ">=2.0" }, + { name = "pygemma", marker = "extra == 'inference-gemma'", specifier = ">=0.1.3" }, { name = "pymessenger", marker = "extra == 'channel-messenger'", specifier = ">=0.0.7" }, { name = "pynostr", marker = "extra == 'channel-nostr'", specifier = ">=0.6" }, { name = "pynvml", marker = "extra == 'energy-all'", specifier = ">=12.0" }, @@ -4917,6 +4961,7 @@ requires-dist = [ { name = "tavily-python", marker = "extra == 'tools-search'", specifier = ">=0.3" }, { name = "textual", marker = "extra == 'dashboard'", specifier = ">=0.80" }, { name = "tomli", marker = "python_full_version < '3.11'", specifier = ">=2.0" }, + { name = "tomlkit", specifier = ">=0.12" }, { name = "torch", marker = "extra == 'memory-colbert'", specifier = ">=2.0" }, { name = "torch", marker = "extra == 'orchestrator-training'", specifier = ">=2.0" }, { name = "transformers", marker = "extra == 'orchestrator-training'", specifier = ">=4.40" }, @@ -4930,7 +4975,7 @@ requires-dist = [ { name = "zeus-ml", extras = ["apple"], marker = "extra == 'energy-apple'" }, { name = "zulip", marker = "extra == 'channel-zulip'", specifier = ">=0.9" }, ] -provides-extras = ["dev", "inference-mlx", "inference-vllm", "inference-cloud", "inference-google", "inference-litellm", "tools-search", "memory-faiss", "memory-colbert", "memory-pdf", "memory-bm25", "server", "openhands", "gpu-metrics", "energy-amd", "energy-apple", "energy-all", "orchestrator-training", "learning-dspy", "learning-gepa", "channel-telegram", "channel-discord", "channel-slack", "channel-line", "channel-viber", "channel-messenger", "channel-reddit", "channel-mastodon", "channel-xmpp", "channel-rocketchat", "channel-zulip", "channel-twitch", "channel-nostr", "browser", "media", "pdf", "scheduler", "security-signing", "sandbox-wasm", "sandbox-docker", "dashboard", "speech", "speech-deepgram", "eval-wandb", "eval-sheets", "docs"] +provides-extras = ["dev", "inference-mlx", "inference-vllm", "inference-cloud", "inference-google", "inference-litellm", "inference-gemma", "tools-search", "memory-faiss", "memory-colbert", "memory-pdf", "memory-bm25", "server", "openhands", "gpu-metrics", "energy-amd", "energy-apple", "energy-all", "orchestrator-training", "learning-dspy", "learning-gepa", "channel-telegram", "channel-discord", "channel-slack", "channel-line", "channel-viber", "channel-messenger", "channel-reddit", "channel-mastodon", "channel-xmpp", "channel-rocketchat", "channel-zulip", "channel-twitch", "channel-nostr", "browser", "media", "pdf", "scheduler", "security-signing", "sandbox-wasm", "sandbox-docker", "dashboard", "speech", "speech-deepgram", "eval-wandb", "eval-sheets", "docs"] [package.metadata.requires-dev] dev = [{ name = "maturin", specifier = ">=1.12.6" }] @@ -5601,6 +5646,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/96/5c/8af904314e42d5401afcfaff69940dc448e974f80f7aa39b241a4fbf0cf1/prawcore-2.4.0-py3-none-any.whl", hash = "sha256:29af5da58d85704b439ad3c820873ad541f4535e00bb98c66f0fbcc8c603065a", size = 17203, upload-time = "2023-10-01T23:30:47.651Z" }, ] +[[package]] +name = "pre-commit" +version = "4.5.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cfgv" }, + { name = "identify" }, + { name = "nodeenv" }, + { name = "pyyaml" }, + { name = "virtualenv" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/40/f1/6d86a29246dfd2e9b6237f0b5823717f60cad94d47ddc26afa916d21f525/pre_commit-4.5.1.tar.gz", hash = "sha256:eb545fcff725875197837263e977ea257a402056661f09dae08e4b149b030a61", size = 198232, upload-time = "2025-12-16T21:14:33.552Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5d/19/fd3ef348460c80af7bb4669ea7926651d1f95c23ff2df18b9d24bab4f3fa/pre_commit-4.5.1-py2.py3-none-any.whl", hash = "sha256:3b3afd891e97337708c1674210f8eba659b52a38ea5f822ff142d10786221f77", size = 226437, upload-time = "2025-12-16T21:14:32.409Z" }, +] + [[package]] name = "primp" version = "1.1.3" @@ -6386,6 +6447,16 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a0/c4/b4d4827c93ef43c01f599ef31453ccc1c132b353284fc6c87d535c233129/pyee-13.0.1-py3-none-any.whl", hash = "sha256:af2f8fede4171ef667dfded53f96e2ed0d6e6bd7ee3bb46437f77e3b57689228", size = 15659, upload-time = "2026-02-14T21:12:26.263Z" }, ] +[[package]] +name = "pygemma" +version = "0.1.3.post3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ed/23/ec5446b040c35be02431dffcc04f808b2cb8007af03d811b76a4fd3d08a0/pygemma-0.1.3.post3.tar.gz", hash = "sha256:2456accacd5643514ef28b43931261f86ece5e61e1da4fe51e3e0ce7ec0ac109", size = 3997, upload-time = "2024-03-19T02:09:42.599Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/08/51/426d2e3b3b1253bacb281357d698815c432456ad0827d49cc09e456ac3e1/pygemma-0.1.3.post3-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:d1a61added2ea6765130b55e12e2fc305afbddb77b2cb88e257fcec91c766464", size = 1650822, upload-time = "2024-03-19T02:09:40.274Z" }, + { url = "https://files.pythonhosted.org/packages/72/4a/4bd0aafd5e9073c9a4c3926f56bcc681290cc0811479638290fc7e189b7e/pygemma-0.1.3.post3-cp311-cp311-manylinux1_x86_64.whl", hash = "sha256:5a54175b7ecf76b3fcd6df061f7cc7d3eab90c1f0e00b6884ca9948852e3dd04", size = 1848325, upload-time = "2024-03-19T11:14:02.634Z" }, +] + [[package]] name = "pygments" version = "2.19.2" @@ -6559,6 +6630,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, ] +[[package]] +name = "python-discovery" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "filelock" }, + { name = "platformdirs" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9c/90/bcce6b46823c9bec1757c964dc37ed332579be512e17a30e9698095dcae4/python_discovery-1.2.0.tar.gz", hash = "sha256:7d33e350704818b09e3da2bd419d37e21e7c30db6e0977bb438916e06b41b5b1", size = 58055, upload-time = "2026-03-19T01:43:08.248Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/3c/2005227cb951df502412de2fa781f800663cccbef8d90ec6f1b371ac2c0d/python_discovery-1.2.0-py3-none-any.whl", hash = "sha256:1e108f1bbe2ed0ef089823d28805d5ad32be8e734b86a5f212bf89b71c266e4a", size = 31524, upload-time = "2026-03-19T01:43:07.045Z" }, +] + [[package]] name = "python-dotenv" version = "1.2.1" @@ -8373,6 +8457,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/23/d1/136eb2cb77520a31e1f64cbae9d33ec6df0d78bdf4160398e86eec8a8754/tomli-2.4.0-py3-none-any.whl", hash = "sha256:1f776e7d669ebceb01dee46484485f43a4048746235e683bcdffacdf1fb4785a", size = 14477, upload-time = "2026-01-11T11:22:37.446Z" }, ] +[[package]] +name = "tomlkit" +version = "0.14.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/af/14b24e41977adb296d6bd1fb59402cf7d60ce364f90c890bd2ec65c43b5a/tomlkit-0.14.0.tar.gz", hash = "sha256:cf00efca415dbd57575befb1f6634c4f42d2d87dbba376128adb42c121b87064", size = 187167, upload-time = "2026-01-13T01:14:53.304Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b5/11/87d6d29fb5d237229d67973a6c9e06e048f01cf4994dee194ab0ea841814/tomlkit-0.14.0-py3-none-any.whl", hash = "sha256:592064ed85b40fa213469f81ac584f67a4f2992509a7c3ea2d632208623a3680", size = 39310, upload-time = "2026-01-13T01:14:51.965Z" }, +] + [[package]] name = "torch" version = "2.10.0" @@ -8932,6 +9025,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5d/c1/f01846f10c8383b197a67b4cdfea6eea476aab678ed9f6f5d9ccc8c8dddf/viberbot-1.0.12-py3-none-any.whl", hash = "sha256:ca43fea2945d650c2ef2cbd777f3c546c795bf945278f6620ceda42f4101d801", size = 26164, upload-time = "2021-01-27T13:45:43.726Z" }, ] +[[package]] +name = "virtualenv" +version = "21.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "distlib" }, + { name = "filelock" }, + { name = "platformdirs" }, + { name = "python-discovery" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/aa/92/58199fe10049f9703c2666e809c4f686c54ef0a68b0f6afccf518c0b1eb9/virtualenv-21.2.0.tar.gz", hash = "sha256:1720dc3a62ef5b443092e3f499228599045d7fea4c79199770499df8becf9098", size = 5840618, upload-time = "2026-03-09T17:24:38.013Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c6/59/7d02447a55b2e55755011a647479041bc92a82e143f96a8195cb33bd0a1c/virtualenv-21.2.0-py3-none-any.whl", hash = "sha256:1bd755b504931164a5a496d217c014d098426cddc79363ad66ac78125f9d908f", size = 5825084, upload-time = "2026-03-09T17:24:35.378Z" }, +] + [[package]] name = "vllm" version = "0.17.1" From 1d24c64f117b6fccc6e18ece9a423b7604e2dbfe Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 21:16:57 -0700 Subject: [PATCH 90/92] =?UTF-8?q?fix(evals):=20PinchBench=20harness=20fixe?= =?UTF-8?q?s=20=E2=80=94=20scores=20from=2026%=20to=2084%=20(#124)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(evals): PinchBench harness fixes — scores from 26% to 84% Multiple infrastructure bugs prevented PinchBench from producing accurate scores. This commit fixes the eval harness so model scores match the official leaderboard (Qwen3.5-397B: 26% → 84%, GPT-5.4: 5% → 58%). Key fixes: - EvalRunner: wrap generation in PinchBenchTaskEnv context so workspace files persist through grading (was deleting before scorer ran) - EvalRunner: detect task_env datasets and force sequential processing (CWD changes aren't thread-safe) - EvalRunner: fix episode_mode auto-detection to check for real iter_episodes() override instead of hasattr() (always True) - Scorer: use "params" field in transcripts to match PinchBench grade() functions (was "arguments") - Scorer: add None guard in _trace_to_transcript for tool_calls - Scorer: capture final assistant text response in transcript so text-only tasks (like sanity check) can be graded - Scorer: add _tool_results_to_transcript() helper for EvalRunner path - native_openhands: add native function-calling support (tools= parameter) with text-based fallback, matching monitor_operative - Tools: add Python fallbacks for file_read, file_write, shell_exec, calculator, think, http_request when openjarvis_rust unavailable - Security: add Python fallback for is_sensitive_file() - Config: add "pinchbench" to KNOWN_BENCHMARKS - Add PinchBench eval configs for Qwen3.5-397B and GPT-5.4 Co-Authored-By: Claude Opus 4.6 (1M context) * fix(evals): fix hybrid grading crash when grading_weights is None The 4 tasks with grading_type=hybrid (task_10, task_13, task_16_market, task_22) crashed because grading_weights was explicitly None in task metadata. dict.get("key", default) returns None (not the default) when the key exists with value None. Use `or` to coalesce None to defaults. Co-Authored-By: Claude Opus 4.6 (1M context) * fix(evals): handle MagicMock datasets in runner type checks The iter_episodes and create_task_env type identity checks fail with AttributeError when the dataset is a MagicMock (used in tracker tests). Wrap in try/except to default to False for non-DatasetProvider objects. Co-Authored-By: Claude Opus 4.6 (1M context) --------- Co-authored-by: Claude Opus 4.6 (1M context) --- src/openjarvis/agents/native_openhands.py | 47 +++++++++++- .../evals/configs/pinchbench-gpt54.toml | 34 +++++++++ .../evals/configs/pinchbench-qwen397b.toml | 35 +++++++++ src/openjarvis/evals/core/config.py | 1 + src/openjarvis/evals/core/runner.py | 74 ++++++++++++++++--- src/openjarvis/evals/scorers/pinchbench.py | 70 ++++++++++++++++-- src/openjarvis/security/file_policy.py | 22 +++++- src/openjarvis/tools/calculator.py | 13 +++- src/openjarvis/tools/file_read.py | 9 ++- src/openjarvis/tools/file_write.py | 22 +++--- src/openjarvis/tools/http_request.py | 10 ++- src/openjarvis/tools/shell_exec.py | 53 ++++++------- src/openjarvis/tools/think.py | 9 ++- tests/evals/test_benchmark_datasets.py | 2 +- tests/evals/test_pinchbench_grading.py | 4 +- 15 files changed, 331 insertions(+), 74 deletions(-) create mode 100644 src/openjarvis/evals/configs/pinchbench-gpt54.toml create mode 100644 src/openjarvis/evals/configs/pinchbench-qwen397b.toml diff --git a/src/openjarvis/agents/native_openhands.py b/src/openjarvis/agents/native_openhands.py index cf5579cd..9b518809 100644 --- a/src/openjarvis/agents/native_openhands.py +++ b/src/openjarvis/agents/native_openhands.py @@ -283,14 +283,23 @@ class NativeOpenHandsAgent(ToolUsingAgent): last_content = "" total_usage = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} + # Build OpenAI-format tool schemas for native function calling + openai_tools = ( + self._executor.get_openai_tools() if self._tools else [] + ) + for _turn in range(self._max_turns): turns += 1 # Truncate before every generate call -- tool results may have # expanded the context beyond what the model supports. messages = self._truncate_if_needed(messages) + gen_kwargs: dict[str, Any] = {} + if openai_tools: + gen_kwargs["tools"] = openai_tools + try: - result = self._generate(messages) + result = self._generate(messages, **gen_kwargs) except Exception as exc: error_str = str(exc) if "400" in error_str: @@ -318,6 +327,42 @@ class NativeOpenHandsAgent(ToolUsingAgent): content = self._strip_think_tags(content) last_content = content + # --- Native function-calling path (OpenAI, Anthropic, etc.) --- + raw_tool_calls = result.get("tool_calls", []) + if raw_tool_calls: + native_calls = [ + ToolCall( + id=tc.get("id", f"call_{turns}_{i}"), + name=tc.get("name", ""), + arguments=tc.get("arguments", "{}"), + ) + for i, tc in enumerate(raw_tool_calls) + ] + messages.append( + Message( + role=Role.ASSISTANT, + content=content, + tool_calls=native_calls, + ) + ) + for tc in native_calls: + tool_result = self._executor.execute(tc) + all_tool_results.append(tool_result) + obs_text = tool_result.content + if len(obs_text) > 4000: + obs_text = obs_text[:4000] + "\n\n[Output truncated]" + messages.append( + Message( + role=Role.TOOL, + content=obs_text, + tool_call_id=tc.id, + name=tc.name, + ) + ) + continue + + # --- Text-based fallback (CodeAct / Action-Input format) --- + # Try to extract code code = self._extract_code(content) if code: diff --git a/src/openjarvis/evals/configs/pinchbench-gpt54.toml b/src/openjarvis/evals/configs/pinchbench-gpt54.toml new file mode 100644 index 00000000..e1feee6a --- /dev/null +++ b/src/openjarvis/evals/configs/pinchbench-gpt54.toml @@ -0,0 +1,34 @@ +# PinchBench eval: gpt-5.4 (cloud) +# Agent: native_openhands — all PinchBench-required tools enabled + +[meta] +name = "pinchbench-gpt54" +description = "PinchBench on gpt-5.4-2026-03-05" + +[defaults] +temperature = 0.6 +max_tokens = 8192 + +[judge] +model = "claude-opus-4-5" +temperature = 0.0 +engine = "cloud" + +[run] +max_workers = 1 +output_dir = "results/pinchbench-gpt54/" +seed = 42 + +[[models]] +name = "gpt-5.4-2026-03-05" +engine = "cloud" + +[[benchmarks]] +name = "pinchbench" +backend = "jarvis-agent" +agent = "native_openhands" +tools = [ + "think", "file_read", "file_write", "web_search", "shell_exec", + "code_interpreter", "browser_navigate", "image_generate", + "calculator", "http_request", "pdf_extract", +] diff --git a/src/openjarvis/evals/configs/pinchbench-qwen397b.toml b/src/openjarvis/evals/configs/pinchbench-qwen397b.toml new file mode 100644 index 00000000..b6b1493b --- /dev/null +++ b/src/openjarvis/evals/configs/pinchbench-qwen397b.toml @@ -0,0 +1,35 @@ +# PinchBench eval: Qwen3.5-397B-A17B-FP8 (vLLM, TP=8) +# Agent: native_openhands — all PinchBench-required tools enabled + +[meta] +name = "pinchbench-qwen397b" +description = "PinchBench on Qwen/Qwen3.5-397B-A17B-FP8 (vLLM, TP=8)" + +[defaults] +temperature = 0.6 +max_tokens = 8192 + +[judge] +model = "claude-opus-4-5" +temperature = 0.0 +engine = "cloud" + +[run] +max_workers = 1 +output_dir = "results/pinchbench-qwen397b/" +seed = 42 + +[[models]] +name = "Qwen/Qwen3.5-397B-A17B-FP8" +engine = "vllm" +num_gpus = 8 + +[[benchmarks]] +name = "pinchbench" +backend = "jarvis-agent" +agent = "native_openhands" +tools = [ + "think", "file_read", "file_write", "web_search", "shell_exec", + "code_interpreter", "browser_navigate", "image_generate", + "calculator", "http_request", "pdf_extract", +] diff --git a/src/openjarvis/evals/core/config.py b/src/openjarvis/evals/core/config.py index ac62d191..ab864c71 100644 --- a/src/openjarvis/evals/core/config.py +++ b/src/openjarvis/evals/core/config.py @@ -43,6 +43,7 @@ KNOWN_BENCHMARKS = { "knowledge_base", "coding_task", "coding_assistant", "security_scanner", "daily_digest", "doc_qa", "browser_assistant", + "pinchbench", } diff --git a/src/openjarvis/evals/core/runner.py b/src/openjarvis/evals/core/runner.py index e3d8716e..9effbad2 100644 --- a/src/openjarvis/evals/core/runner.py +++ b/src/openjarvis/evals/core/runner.py @@ -96,11 +96,19 @@ class EvalRunner: seed=cfg.seed, ) - # Auto-enable episode_mode when the dataset has iter_episodes() - # (i.e. it is a lifelong/sequential benchmark like LifelongAgentBench). - # This is enforced at the runner level so it applies regardless of - # how the runner is invoked (CLI, SDK, tests, etc.). - if not cfg.episode_mode and hasattr(self._dataset, "iter_episodes"): + # Auto-enable episode_mode when the dataset *overrides* + # iter_episodes() (i.e. it is a lifelong/sequential benchmark like + # LifelongAgentBench). The base DatasetProvider always defines a + # default iter_episodes() that wraps each record in its own episode, + # so hasattr() is always True — we must check for a real override. + from openjarvis.evals.core.dataset import DatasetProvider as _DP + try: + _overrides_episodes = ( + type(self._dataset).iter_episodes is not _DP.iter_episodes + ) + except AttributeError: + _overrides_episodes = False + if not cfg.episode_mode and _overrides_episodes: LOGGER.info( "%s requires sequential episode processing — " "auto-enabling episode_mode.", @@ -109,6 +117,15 @@ class EvalRunner: cfg = dataclasses.replace(cfg, episode_mode=True) self._config = cfg + # Detect if dataset provides task environments (e.g. PinchBench) + try: + self._has_task_env = ( + type(self._dataset).create_task_env + is not _DP.create_task_env + ) + except AttributeError: + self._has_task_env = False + records = list(self._dataset.iter_records()) LOGGER.info( "Running %s: %d samples, backend=%s, model=%s, workers=%d, " @@ -145,6 +162,15 @@ class EvalRunner: try: if cfg.episode_mode: self._run_episode_mode(records, progress_callback, total) + elif self._has_task_env: + # Task environments (PinchBench etc.) change CWD — + # must process sequentially for thread safety. + for record in records: + result = self._process_one(record) + self._results.append(result) + self._flush_result(result) + if progress_callback is not None: + progress_callback(len(self._results), total) else: with ThreadPoolExecutor(max_workers=cfg.max_workers) as pool: futures = { @@ -224,17 +250,41 @@ class EvalRunner: ) if cfg.system_prompt: gen_kwargs["system"] = cfg.system_prompt - full = self._backend.generate_full( - record.problem, - **gen_kwargs, - ) - content = full.get("content", "") + + if getattr(self, "_has_task_env", False): + from contextlib import nullcontext + task_env = self._dataset.create_task_env(record) + ctx = task_env if task_env is not None else nullcontext() + with ctx: + full = self._backend.generate_full( + record.problem, + **gen_kwargs, + ) + full = full or {} + # Store tool results for the scorer to build transcripts + record.metadata["tool_results"] = full.get( + "tool_results", [], + ) + # Score INSIDE context so workspace files still exist + content = full.get("content", "") + is_correct, scoring_meta = self._scorer.score( + record, content, + ) + else: + full = self._backend.generate_full( + record.problem, + **gen_kwargs, + ) + full = full or {} + content = full.get("content", "") + is_correct, scoring_meta = self._scorer.score( + record, content, + ) + usage = full.get("usage", {}) latency = full.get("latency_seconds", 0.0) cost = full.get("cost_usd", 0.0) - is_correct, scoring_meta = self._scorer.score(record, content) - energy_j = full.get("energy_joules", 0.0) power_w = full.get("power_watts", 0.0) throughput = full.get("throughput_tok_per_sec", 0.0) diff --git a/src/openjarvis/evals/scorers/pinchbench.py b/src/openjarvis/evals/scorers/pinchbench.py index 68269ec0..693f3d12 100644 --- a/src/openjarvis/evals/scorers/pinchbench.py +++ b/src/openjarvis/evals/scorers/pinchbench.py @@ -52,12 +52,12 @@ def events_to_transcript(events: List[Any]) -> List[Dict[str, Any]]: if etype == EventType.TOOL_CALL_START or etype == EventType.TOOL_CALL_START.value: tool_name = event.metadata.get("tool", "unknown") mapped = _TOOL_NAME_MAP.get(tool_name, tool_name) - arguments = event.metadata.get("arguments", {}) + arguments = event.metadata.get("arguments") or {} transcript.append({ "type": "message", "message": { "role": "assistant", - "content": [{"type": "toolCall", "name": mapped, "arguments": arguments}], + "content": [{"type": "toolCall", "name": mapped, "params": arguments}], }, }) elif etype == EventType.TOOL_CALL_END or etype == EventType.TOOL_CALL_END.value: @@ -78,12 +78,14 @@ def _trace_to_transcript(trace: Any) -> List[Dict[str, Any]]: transcript: List[Dict[str, Any]] = [] for turn in trace.turns: for tc in getattr(turn, "tool_calls", []): + if tc is None: + continue mapped = _TOOL_NAME_MAP.get(tc["name"], tc["name"]) transcript.append({ "type": "message", "message": { "role": "assistant", - "content": [{"type": "toolCall", "name": mapped, "arguments": tc.get("arguments", {})}], + "content": [{"type": "toolCall", "name": mapped, "params": tc.get("arguments") or {}}], }, }) transcript.append({ @@ -93,6 +95,42 @@ def _trace_to_transcript(trace: Any) -> List[Dict[str, Any]]: "content": [{"text": tc.get("result", "")}], }, }) + + # Capture final assistant text response (for tasks graded on text output) + response_text = getattr(trace, "response_text", "") + if response_text: + transcript.append({ + "type": "message", + "message": { + "role": "assistant", + "content": [{"type": "text", "text": response_text}], + }, + }) + return transcript + + +def _tool_results_to_transcript( + tool_results: List[Dict[str, Any]], +) -> List[Dict[str, Any]]: + """Build transcript from JarvisAgentBackend tool_results list.""" + transcript: List[Dict[str, Any]] = [] + for tr in tool_results: + tool_name = tr.get("tool_name", "unknown") + mapped = _TOOL_NAME_MAP.get(tool_name, tool_name) + transcript.append({ + "type": "message", + "message": { + "role": "assistant", + "content": [{"type": "toolCall", "name": mapped, "params": {}}], + }, + }) + transcript.append({ + "type": "message", + "message": { + "role": "toolResult", + "content": [{"text": tr.get("content", "")}], + }, + }) return transcript @@ -118,8 +156,10 @@ def _summarize_transcript(transcript: List[Dict[str, Any]]) -> str: for item in msg.get("content", []): if item.get("type") == "toolCall": parts.append( - f"Tool: {item.get('name')}({json.dumps(item.get('arguments', {}))})" + f"Tool: {item.get('name')}({json.dumps(item.get('params', {}))})" ) + elif item.get("type") == "text": + parts.append(f"Assistant: {item.get('text', '')}") elif role == "toolResult": content = msg.get("content", []) if content: @@ -372,7 +412,7 @@ def _grade_hybrid( judge_model: str, ) -> Dict[str, Any]: """Run both automated and LLM judge grading, combine with weights.""" - weights = record.metadata.get("grading_weights", {"automated": 0.5, "llm_judge": 0.5}) + weights = record.metadata.get("grading_weights") or {"automated": 0.5, "llm_judge": 0.5} auto = _grade_automated(record, transcript, workspace_path) llm = _grade_llm_judge(record, transcript, workspace_path, judge_backend, judge_model) @@ -426,7 +466,24 @@ class PinchBenchScorer(LLMJudgeScorer): self, record: EvalRecord, model_answer: str, ) -> Tuple[Optional[bool], Dict[str, Any]]: trace = record.metadata.get("query_trace") - transcript = _trace_to_transcript(trace) if trace else [] + if trace: + transcript = _trace_to_transcript(trace) + else: + # No trace — build transcript from tool_results if available + tool_results = record.metadata.get("tool_results", []) + transcript = _tool_results_to_transcript(tool_results) + + # Always append final model answer as assistant text message + # so grading functions that check for text responses can find it + if model_answer: + transcript.append({ + "type": "message", + "message": { + "role": "assistant", + "content": [{"type": "text", "text": model_answer}], + }, + }) + result = grade_pinchbench_task( record=record, transcript=transcript, @@ -442,4 +499,5 @@ __all__ = [ "PinchBenchScorer", "events_to_transcript", "grade_pinchbench_task", + "_tool_results_to_transcript", ] diff --git a/src/openjarvis/security/file_policy.py b/src/openjarvis/security/file_policy.py index fc4006ae..36482440 100644 --- a/src/openjarvis/security/file_policy.py +++ b/src/openjarvis/security/file_policy.py @@ -30,11 +30,27 @@ def is_sensitive_file(path: Union[str, Path]) -> bool: Checks both the filename and the full name against ``DEFAULT_SENSITIVE_PATTERNS`` using :func:`fnmatch.fnmatch`. + Uses the Rust implementation when available, falls back to Python. """ - from openjarvis._rust_bridge import get_rust_module + try: + from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() - return _rust.is_sensitive_file(str(path)) + _rust = get_rust_module() + return _rust.is_sensitive_file(str(path)) + except ImportError: + return _is_sensitive_file_py(str(path)) + + +def _is_sensitive_file_py(path_str: str) -> bool: + """Pure-Python fallback for sensitive file detection.""" + import fnmatch + + p = Path(path_str) + name = p.name + for pattern in DEFAULT_SENSITIVE_PATTERNS: + if fnmatch.fnmatch(name, pattern) or fnmatch.fnmatch(str(p), pattern): + return True + return False def filter_sensitive_paths(paths: Iterable[Union[str, Path]]) -> List[Path]: diff --git a/src/openjarvis/tools/calculator.py b/src/openjarvis/tools/calculator.py index 14ca4bc4..2ec101eb 100644 --- a/src/openjarvis/tools/calculator.py +++ b/src/openjarvis/tools/calculator.py @@ -89,10 +89,15 @@ def _safe_eval_node(node: ast.AST) -> Any: def safe_eval(expression: str) -> float: - """Evaluate a math expression safely — always via Rust backend.""" - from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() - return float(_rust.CalculatorTool().execute(expression)) + """Evaluate a math expression safely — Rust backend with Python fallback.""" + try: + from openjarvis._rust_bridge import get_rust_module + _rust = get_rust_module() + return float(_rust.CalculatorTool().execute(expression)) + except ImportError: + import ast as _ast + tree = _ast.parse(expression, mode="eval") + return float(_safe_eval_node(tree.body)) @ToolRegistry.register("calculator") diff --git a/src/openjarvis/tools/file_read.py b/src/openjarvis/tools/file_read.py index 36fb258e..6e5b7e38 100644 --- a/src/openjarvis/tools/file_read.py +++ b/src/openjarvis/tools/file_read.py @@ -114,10 +114,15 @@ class FileReadTool(BaseTool): content=f"File too large: {size} bytes (max {_MAX_SIZE_BYTES}).", success=False, ) - from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() try: + from openjarvis._rust_bridge import get_rust_module + _rust = get_rust_module() text = _rust.FileReadTool().execute(str(path)) + except ImportError: + try: + text = path.read_text(encoding="utf-8") + except UnicodeDecodeError: + text = path.read_text(encoding="utf-8", errors="replace") except Exception as exc: return ToolResult( tool_name="file_read", diff --git a/src/openjarvis/tools/file_write.py b/src/openjarvis/tools/file_write.py index d006280c..71be96df 100644 --- a/src/openjarvis/tools/file_write.py +++ b/src/openjarvis/tools/file_write.py @@ -155,26 +155,26 @@ class FileWriteTool(BaseTool): success=False, ) - from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() if mode == "write": try: + from openjarvis._rust_bridge import get_rust_module + _rust = get_rust_module() _rust.FileWriteTool().execute(str(path), content) + except ImportError: + try: + path.write_text(content, encoding="utf-8") + except OSError as exc: + return ToolResult( + tool_name="file_write", + content=f"Write error: {exc}", + success=False, + ) except Exception as exc: return ToolResult( tool_name="file_write", content=f"Write error: {exc}", success=False, ) - elif False: # dead code — all write modes go through Rust - try: - path.write_text(content, encoding="utf-8") - except OSError as exc: - return ToolResult( - tool_name="file_write", - content=f"Write error: {exc}", - success=False, - ) else: # append mode — always Python try: diff --git a/src/openjarvis/tools/http_request.py b/src/openjarvis/tools/http_request.py index 4cde4d60..d519815a 100644 --- a/src/openjarvis/tools/http_request.py +++ b/src/openjarvis/tools/http_request.py @@ -103,9 +103,13 @@ class HttpRequestTool(BaseTool): body = params.get("body") timeout = params.get("timeout", 30) - from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() - if not headers: + _rust = None + try: + from openjarvis._rust_bridge import get_rust_module + _rust = get_rust_module() + except ImportError: + pass + if _rust is not None and not headers: try: content = _rust.HttpRequestTool().execute(url, method, body) return ToolResult( diff --git a/src/openjarvis/tools/shell_exec.py b/src/openjarvis/tools/shell_exec.py index 3952d26d..6611df7a 100644 --- a/src/openjarvis/tools/shell_exec.py +++ b/src/openjarvis/tools/shell_exec.py @@ -125,32 +125,33 @@ class ShellExecTool(BaseTool): if val is not None: env[key] = val - from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() - if True: - try: - output = _rust.ShellExecTool().execute(command, working_dir) - return ToolResult( - tool_name="shell_exec", - content=output or "(no output)", - success=True, - metadata={ - "returncode": 0, - "timeout_used": timeout, - "working_dir": working_dir, - }, - ) - except Exception as exc: - return ToolResult( - tool_name="shell_exec", - content=str(exc), - success=False, - metadata={ - "returncode": -1, - "timeout_used": timeout, - "working_dir": working_dir, - }, - ) + try: + from openjarvis._rust_bridge import get_rust_module + _rust = get_rust_module() + output = _rust.ShellExecTool().execute(command, working_dir) + return ToolResult( + tool_name="shell_exec", + content=output or "(no output)", + success=True, + metadata={ + "returncode": 0, + "timeout_used": timeout, + "working_dir": working_dir, + }, + ) + except ImportError: + pass # Fall through to subprocess below + except Exception as exc: + return ToolResult( + tool_name="shell_exec", + content=str(exc), + success=False, + metadata={ + "returncode": -1, + "timeout_used": timeout, + "working_dir": working_dir, + }, + ) try: result = subprocess.run( command, diff --git a/src/openjarvis/tools/think.py b/src/openjarvis/tools/think.py index 225c888e..699bbe03 100644 --- a/src/openjarvis/tools/think.py +++ b/src/openjarvis/tools/think.py @@ -40,9 +40,12 @@ class ThinkTool(BaseTool): def execute(self, **params: Any) -> ToolResult: thought = params.get("thought", "") - from openjarvis._rust_bridge import get_rust_module - _rust = get_rust_module() - content = _rust.ThinkTool().execute(thought) + try: + from openjarvis._rust_bridge import get_rust_module + _rust = get_rust_module() + content = _rust.ThinkTool().execute(thought) + except ImportError: + content = thought return ToolResult( tool_name="think", content=content, diff --git a/tests/evals/test_benchmark_datasets.py b/tests/evals/test_benchmark_datasets.py index ee5e73a3..e569fa48 100644 --- a/tests/evals/test_benchmark_datasets.py +++ b/tests/evals/test_benchmark_datasets.py @@ -313,7 +313,7 @@ class TestConfigBenchmarks: def test_benchmarks_count(self) -> None: from openjarvis.evals.core.config import KNOWN_BENCHMARKS - assert len(KNOWN_BENCHMARKS) == 25 + assert len(KNOWN_BENCHMARKS) == 26 # --------------------------------------------------------------------------- diff --git a/tests/evals/test_pinchbench_grading.py b/tests/evals/test_pinchbench_grading.py index aa107506..d2396bfc 100644 --- a/tests/evals/test_pinchbench_grading.py +++ b/tests/evals/test_pinchbench_grading.py @@ -74,7 +74,7 @@ def grade(transcript, workspace_path): { "type": "toolCall", "name": "read_file", - "arguments": {"path": "a.txt"}, + "params": {"path": "a.txt"}, } ], }, @@ -124,7 +124,7 @@ class TestSummarizeTranscript: { "type": "toolCall", "name": "read_file", - "arguments": {"path": "a.txt"}, + "params": {"path": "a.txt"}, } ], }, From 2259280f2310767e444746905d144b80b4256c39 Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 21:17:32 -0700 Subject: [PATCH 91/92] feat: auto-clone repo on desktop app first launch (#122) Auto-clone the OpenJarvis repo on first desktop app launch instead of showing an error. Uses git clone --depth 1 to ~/OpenJarvis. Closes #122. --- desktop/src-tauri/src/lib.rs | 91 ++++++++++++++++++++++++++++++++---- 1 file changed, 82 insertions(+), 9 deletions(-) diff --git a/desktop/src-tauri/src/lib.rs b/desktop/src-tauri/src/lib.rs index 14dcf64e..5f6163e3 100644 --- a/desktop/src-tauri/src/lib.rs +++ b/desktop/src-tauri/src/lib.rs @@ -460,17 +460,90 @@ async fn boot_backend(backend: SharedBackend, status: SharedStatus) { return; } - let project_root = find_project_root(); + let mut project_root = find_project_root(); if project_root.is_none() { - let mut s = status.lock().await; - s.error = Some( - "Could not find the OpenJarvis project directory. \ - Clone it with: git clone https://github.com/open-jarvis/OpenJarvis.git ~/OpenJarvis \ - then relaunch." - .into(), - ); - return; + // Auto-clone on first launch + let git_bin = resolve_bin("git"); + + // Check that git is installed + if !std::path::Path::new(&git_bin).exists() && git_bin == "git" { + let mut s = status.lock().await; + s.error = Some( + "Could not find 'git'. \ + Install it from https://git-scm.com then relaunch." + .into(), + ); + return; + } + + let target_path = std::path::PathBuf::from(home_dir()).join("OpenJarvis"); + let clone_target = target_path.display().to_string(); + + // If the directory exists but is not a valid project, don't overwrite + if target_path.exists() && !target_path.join("pyproject.toml").exists() { + let mut s = status.lock().await; + s.error = Some(format!( + "{} exists but is not a valid OpenJarvis project. \ + Remove it and relaunch, or set OPENJARVIS_ROOT to the correct path.", + clone_target, + )); + return; + } + + { + let mut s = status.lock().await; + s.detail = "Downloading OpenJarvis (first launch)...".into(); + } + + let clone_result = tokio::process::Command::new(&git_bin) + .args([ + "clone", + "--depth", + "1", + "https://github.com/open-jarvis/OpenJarvis.git", + &clone_target, + ]) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::piped()) + .spawn(); + + match clone_result { + Ok(child) => match child.wait_with_output().await { + Ok(output) if output.status.success() => { + project_root = Some(target_path); + } + Ok(output) => { + let stderr = String::from_utf8_lossy(&output.stderr); + let mut s = status.lock().await; + s.error = Some(format!( + "Failed to download OpenJarvis: {}. \ + Clone manually: git clone https://github.com/open-jarvis/OpenJarvis.git {}", + stderr.trim(), + clone_target, + )); + return; + } + Err(e) => { + let mut s = status.lock().await; + s.error = Some(format!( + "Failed to download OpenJarvis: {}. \ + Clone manually: git clone https://github.com/open-jarvis/OpenJarvis.git {}", + e, clone_target, + )); + return; + } + }, + Err(e) => { + let mut s = status.lock().await; + s.error = Some(format!( + "Could not run git: {}. \ + Install git from https://git-scm.com then relaunch.", + e, + )); + return; + } + } } // Kill any leftover server on our port from a previous run From 1ff155d4458928dec1a13ae399888a0ae91960ca Mon Sep 17 00:00:00 2001 From: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com> Date: Wed, 25 Mar 2026 21:31:32 -0700 Subject: [PATCH 92/92] fix(ci): skip live and cloud tests in CI The gemma_cpp live tests require local model weights and env vars (GEMMA_CPP_MODEL_PATH, etc.) that are not available in CI, causing 4 failures since the gemma-cpp-engine PR was merged. Add marker filters to the pytest invocation so live and cloud tests are skipped. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 05b90a58..d9b4f9cf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,7 +63,7 @@ jobs: run: uv run maturin develop --manifest-path rust/crates/openjarvis-python/Cargo.toml - name: Run tests - run: uv run pytest tests/ -v --tb=short + run: uv run pytest tests/ -v --tb=short -m "not live and not cloud" rust: runs-on: ubuntu-latest