1d24c64f11
* fix(evals): PinchBench harness fixes — scores from 26% to 84% Multiple infrastructure bugs prevented PinchBench from producing accurate scores. This commit fixes the eval harness so model scores match the official leaderboard (Qwen3.5-397B: 26% → 84%, GPT-5.4: 5% → 58%). Key fixes: - EvalRunner: wrap generation in PinchBenchTaskEnv context so workspace files persist through grading (was deleting before scorer ran) - EvalRunner: detect task_env datasets and force sequential processing (CWD changes aren't thread-safe) - EvalRunner: fix episode_mode auto-detection to check for real iter_episodes() override instead of hasattr() (always True) - Scorer: use "params" field in transcripts to match PinchBench grade() functions (was "arguments") - Scorer: add None guard in _trace_to_transcript for tool_calls - Scorer: capture final assistant text response in transcript so text-only tasks (like sanity check) can be graded - Scorer: add _tool_results_to_transcript() helper for EvalRunner path - native_openhands: add native function-calling support (tools= parameter) with text-based fallback, matching monitor_operative - Tools: add Python fallbacks for file_read, file_write, shell_exec, calculator, think, http_request when openjarvis_rust unavailable - Security: add Python fallback for is_sensitive_file() - Config: add "pinchbench" to KNOWN_BENCHMARKS - Add PinchBench eval configs for Qwen3.5-397B and GPT-5.4 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(evals): fix hybrid grading crash when grading_weights is None The 4 tasks with grading_type=hybrid (task_10, task_13, task_16_market, task_22) crashed because grading_weights was explicitly None in task metadata. dict.get("key", default) returns None (not the default) when the key exists with value None. Use `or` to coalesce None to defaults. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(evals): handle MagicMock datasets in runner type checks The iter_episodes and create_task_env type identity checks fail with AttributeError when the dataset is a MagicMock (used in tracker tests). Wrap in try/except to default to False for non-DatasetProvider objects. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Personal AI, On Personal Devices.
Why OpenJarvis?
Personal AI agents are exploding in popularity, but nearly all of them still route intelligence through cloud APIs. Your "personal" AI continues to depend on someone else's server. At the same time, our Intelligence Per Watt research showed that local language models already handle 88.7% of single-turn chat and reasoning queries, with intelligence efficiency improving 5.3× from 2023 to 2025. The models and hardware are increasingly ready. What has been missing is the software stack to make local-first personal AI practical.
OpenJarvis is that stack. It is an opinionated framework for local-first personal AI, built around three core ideas: shared primitives for building on-device agents; evaluations that treat energy, FLOPs, latency, and dollar cost as first-class constraints alongside accuracy; and a learning loop that improves models using local trace data. The goal is simple: make it possible to build personal AI agents that run locally by default, calling the cloud only when truly necessary. OpenJarvis aims to be both a research platform and a production foundation for local AI, in the spirit of PyTorch.
Installation
git clone https://github.com/open-jarvis/OpenJarvis.git
cd OpenJarvis
uv sync # core framework
uv sync --extra server # + FastAPI server
# Build the Rust extension (requires Rust: https://rustup.rs/)
uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
Python 3.14+: set
PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1before thematurincommand.
You also need a local inference backend: Ollama, vLLM, SGLang, or llama.cpp. Alternatively, use the cloud engine with OpenAI, Anthropic, Google Gemini, OpenRouter, or MiniMax by setting the corresponding API key environment variable.
Quick Start
# 1. Install and detect hardware
git clone https://github.com/open-jarvis/OpenJarvis.git
cd OpenJarvis
uv sync
uv run jarvis init
# 2. Start Ollama and pull a model
curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull qwen3:8b
# 3. Ask a question
uv run jarvis ask "What is the capital of France?"
jarvis init auto-detects your hardware and recommends the best engine. Run uv run jarvis doctor at any time to diagnose issues.
Full documentation — including Docker deployment, cloud engines, development setup, and tutorials — at open-jarvis.github.io/OpenJarvis.
Contributing
We welcome contributions! See the Contributing Guide for incentives, contribution types, and the PR process.
Quick start for contributors:
git clone https://github.com/open-jarvis/OpenJarvis.git
cd OpenJarvis
uv sync --extra dev
uv run pre-commit install
uv run pytest tests/ -v
Browse the Roadmap for areas where help is needed. Comment "take" on any issue to get auto-assigned.
About
OpenJarvis is part of Intelligence Per Watt, a research initiative studying the efficiency of on-device AI systems. The project is developed at Hazy Research and the Scaling Intelligence Lab at Stanford SAIL.
Sponsors
Laude Institute • Stanford Marlowe • Google Cloud Platform • Lambda Labs • Ollama • IBM Research • Stanford HAI
Citation
@misc{saadfalcon2026openjarvis,
title={OpenJarvis: Personal AI, On Personal Devices},
author={Jon Saad-Falcon and Avanika Narayan and Herumb Shandilya and Hakki Orhun Akengin and Robby Manihani and Gabriel Bo and John Hennessy and Christopher R\'{e} and Azalia Mirhoseini},
year={2026},
howpublished={\url{https://scalingintelligence.stanford.edu/blogs/openjarvis/}},
}