docs: add macOS installation guide for llama.cpp

Comprehensive step-by-step guide covering Homebrew, uv, Rust, llama.cpp, model download, Python 3.12 pin (PyO3 compat), and common pitfalls. Cherry-picked from PR #131 by @gridworks — cleaned up to include only the docs content (removed duplicate files, binary artifacts, and unrelated lockfile changes from the original PR). Co-Authored-By: gridworks <5502067+gridworks@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-28 14:07:55 +00:00 · 2026-03-27 16:17:17 +00:00
parent bfdc71d31d
commit 0ce10bd97b
2 changed files with 422 additions and 0 deletions
@@ -0,0 +1,421 @@
+---
+title: macOS Installation Guide
+description: Complete step-by-step guide to installing OpenJarvis on macOS with llama.cpp, including common pitfalls and fixes
+search:
+  boost: 2
+---
+
+# macOS Installation Guide
+
+This guide walks through a complete OpenJarvis installation on macOS using **llama.cpp** as
+the inference engine. It covers every step from scratch — including pitfalls not documented
+elsewhere — and is suitable for both Apple Silicon and Intel Macs.
+
+!!! tip "Prefer Ollama?"
+    If you want the fastest possible setup, use [Ollama](installation.md#ollama-recommended)
+    instead. This guide is for users who want to run GGUF models directly with llama.cpp,
+    or who want a deeper understanding of the full stack.
+
+---
+
+## What You'll Install
+
+| Tool | Purpose |
+|------|---------|
+| Homebrew | macOS package manager — installs everything else |
+| uv | Python version and dependency manager |
+| Git | Clones the OpenJarvis repo |
+| Node.js | Required for the browser UI |
+| Rust | Compiles the OpenJarvis security and memory extension |
+| llama.cpp | Local inference engine that runs GGUF model files |
+| OpenJarvis | The framework itself |
+| A GGUF model | The actual AI model (downloaded separately) |
+
+---
+
+## Step-by-Step Installation
+
+### Step 1 — Install Homebrew
+
+Homebrew is the standard macOS package manager. Everything else in this guide is installed
+through it.
+
+```bash
+/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
+```
+
+If you already have Homebrew, skip this step.
+
+---
+
+### Step 2 — Install uv
+
+`uv` replaces pip, virtualenv, and pyenv in one tool. OpenJarvis uses it to manage Python
+versions, virtual environments, and project dependencies.
+
+```bash
+brew install uv
+```
+
+---
+
+### Step 3 — Install Git
+
+Git is used to clone the OpenJarvis source code. It may already be present if you have
+Xcode Command Line Tools installed.
+
+```bash
+brew install git
+```
+
+---
+
+### Step 4 — Install Node.js
+
+Node.js is required to build and run the browser frontend. Without it you can still use
+the CLI, but not the web UI.
+
+```bash
+brew install node
+```
+
+---
+
+### Step 5 — Install Rust
+
+OpenJarvis includes a Rust extension that provides security scanning, memory indexing,
+rate limiting, and tool execution. It must be compiled from source.
+
+```bash
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+```
+
+After the installer finishes, reload your shell so `rustc` is available:
+
+```bash
+source "$HOME/.cargo/env"
+```
+
+Verify:
+
+```bash
+rustc --version
+```
+
+---
+
+### Step 6 — Install llama.cpp
+
+llama.cpp is the inference engine that loads and runs GGUF model files. It is not a model
+itself — think of it as a media player and the `.gguf` file as the content.
+
+```bash
+brew install llama.cpp
+```
+
+---
+
+### Step 7 — Clone the OpenJarvis repo
+
+Run this from your home directory or any neutral parent folder.
+
+```bash
+cd ~
+git clone https://github.com/open-jarvis/OpenJarvis.git
+cd OpenJarvis
+```
+
+!!! warning "Do not clone from inside an existing OpenJarvis folder"
+    A common mistake is running `git clone` while already inside the repo, creating deeply
+    nested duplicates (`OpenJarvis/OpenJarvis/OpenJarvis`). Always clone from `~` or a
+    neutral parent directory.
+
+---
+
+### Step 8 — Pin Python to 3.12
+
+!!! warning "Critical step — do not skip"
+    OpenJarvis requires Python 3.10–3.13. Its Rust extension uses PyO3, which does not yet
+    support Python 3.14. If `uv` has Python 3.14 available, it will use it by default,
+    causing the Rust extension build to fail silently and resulting in ~250 test failures
+    with `ModuleNotFoundError: No module named 'openjarvis_rust'`.
+
+Pin the project to Python 3.12:
+
+```bash
+echo "3.12" > .python-version
+uv python install 3.12
+rm -rf .venv
+uv venv
+```
+
+**Restart your terminal**, then verify:
+
+```bash
+uv run python --version
+# Must show: Python 3.12.x
+```
+
+!!! tip "Why restart the terminal?"
+    Without restarting, the shell may still reference the old virtual environment. This is
+    the most common reason the version pin appears not to work.
+
+---
+
+### Step 9 — Install Python dependencies
+
+```bash
+uv sync --extra dev --extra server
+```
+
+The `--extra server` flag adds the FastAPI backend required for the browser UI.
+
+---
+
+### Step 10 — Build the Rust extension
+
+This compiles the Rust extension and installs it into the virtual environment. It provides
+security scanning, memory indexing, MCP tool execution, and rate limiting. This step takes
+a few minutes on first run.
+
+```bash
+uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
+```
+
+Verify it built correctly:
+
+```bash
+uv run python -c "import openjarvis_rust; print('Rust extension OK')"
+```
+
+---
+
+### Step 11 — Install frontend dependencies
+
+```bash
+cd frontend && npm install && cd ..
+```
+
+---
+
+### Step 12 — Download a model
+
+OpenJarvis needs a GGUF model file to run inference. First install the Hugging Face CLI,
+then download your chosen model.
+
+```bash
+uv tool install huggingface_hub
+```
+
+!!! note "The CLI command is `hf`, not `huggingface-cli`"
+    When installed via `uv tool`, the Hugging Face CLI is invoked as `hf`.
+
+=== "Qwen3 4B (~2.5 GB)"
+
+    Faster, lower RAM requirement. Good for most everyday tasks.
+
+    ```bash
+    hf download bartowski/Qwen_Qwen3-4B-GGUF \
+      --include "Qwen_Qwen3-4B-Q4_K_M.gguf" \
+      --local-dir ~/models
+    ```
+
+=== "Qwen3 8B (~4.7 GB)"
+
+    Better reasoning and instruction following. Requires more RAM.
+
+    ```bash
+    hf download bartowski/Qwen_Qwen3-8B-GGUF \
+      --include "Qwen_Qwen3-8B-Q4_K_M.gguf" \
+      --local-dir ~/models
+    ```
+
+!!! warning "Use the `Qwen_` prefix"
+    bartowski's Qwen3 repos use the `Qwen_` prefix (e.g. `Qwen_Qwen3-4B-GGUF`). Using
+    the shorter name without the prefix returns a "repository not found" error.
+
+!!! tip "Apple Silicon vs Intel"
+    On Apple Silicon, both models benefit from Metal GPU acceleration when using the MLX
+    engine. On Intel, inference runs on CPU — the 4B model is recommended for speed.
+
+---
+
+### Step 13 — Configure OpenJarvis
+
+Run the init command to detect your hardware and generate a config file:
+
+```bash
+uv run jarvis init
+```
+
+Then open the config and set the default model to match the filename you downloaded:
+
+```bash
+nano ~/.openjarvis/config.toml
+```
+
+Find the `default_model` line and update it, for example:
+
+```toml
+default_model = "Qwen_Qwen3-4B-Q4_K_M.gguf"
+```
+
+---
+
+### Step 14 — Verify the installation
+
+```bash
+uv run jarvis doctor
+```
+
+A healthy setup looks like this:
+
+```
+✓  Python version         3.12.x
+✓  Config file            ~/.openjarvis/config.toml
+✓  Config parsing         Config loaded successfully
+✓  Engine: llamacpp       Reachable
+✓  Models: llamacpp       Qwen_Qwen3-4B-Q4_K_M.gguf
+✓  Default model          Qwen_Qwen3-4B-Q4_K_M.gguf (on llamacpp)
+```
+
+!!! note "Warnings for other engines are normal"
+    The `!` warnings for engines like `ollama`, `vllm`, and `lmstudio` simply mean those
+    backends are not running. You only need `llamacpp` to be reachable.
+
+---
+
+## Running OpenJarvis
+
+### CLI
+
+Start llama-server in one terminal, then run queries in another:
+
+```bash
+# Terminal 1 — start the inference engine
+llama-server -m ~/models/Qwen_Qwen3-4B-Q4_K_M.gguf -c 4096 -t 8
+
+# Terminal 2 — ask a question
+cd ~/OpenJarvis
+uv run jarvis ask "What is the capital of France?"
+```
+
+### Browser UI
+
+```bash
+# Terminal 1 — inference engine
+llama-server -m ~/models/Qwen_Qwen3-4B-Q4_K_M.gguf -c 4096 -t 8
+
+# Terminal 2 — backend
+cd ~/OpenJarvis && uv run jarvis serve --port 8000
+
+# Terminal 3 — frontend
+cd ~/OpenJarvis/frontend && npm run dev
+```
+
+Then open [http://localhost:5173](http://localhost:5173).
+
+### Skip typing `uv run` every time
+
+Activate the virtual environment for your current terminal session:
+
+```bash
+source ~/OpenJarvis/.venv/bin/activate
+```
+
+Your prompt will show `(openjarvis)` when active, and you can type `jarvis ask "..."` directly.
+
+---
+
+## Performance Tips
+
+These tips apply when using llama.cpp for CPU inference.
+
+| Flag | Effect |
+|------|--------|
+| `-c 4096` | Reduces context window from the 32,768 default, freeing RAM for faster inference |
+| `-t 8` | Uses all available CPU threads (default is only 4) — adjust to your machine's thread count |
+| `Q4_K_M` quantization | Best balance of size, speed, and quality for CPU inference |
+
+On Apple Silicon, switching to the [MLX engine](../architecture/engine.md) gives
+significantly better performance than llama.cpp for most models.
+
+---
+
+## Common Errors
+
+### `No such file or directory` when loading model
+
+The path `path/to/model.gguf` in examples is a placeholder. Replace it with your actual
+model path, e.g.:
+
+```bash
+llama-server -m ~/models/Qwen_Qwen3-4B-Q4_K_M.gguf
+```
+
+---
+
+### `No module named 'openjarvis_rust'`
+
+The Rust extension did not build correctly, or was built against the wrong Python version.
+
+1. Confirm Python 3.12 is active: `uv run python --version`
+2. Rebuild: `uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml`
+
+If the version shows 3.14, go back to [Step 8](#step-8--pin-python-to-312).
+
+---
+
+### `PyO3 version error — Python 3.14 too new`
+
+```
+error: the configured Python interpreter version (3.14) is newer than
+PyO3's maximum supported version (3.13)
+```
+
+PyO3 0.23.5 supports Python up to 3.13. Follow [Step 8](#step-8--pin-python-to-312) to
+pin to 3.12, then delete `.venv`, recreate it, and restart your terminal before retrying.
+
+---
+
+### `Repository not found` when downloading model
+
+bartowski's Qwen3 repos use the `Qwen_` prefix. Use:
+
+```
+bartowski/Qwen_Qwen3-4B-GGUF   ✓
+bartowski/Qwen3-4B-GGUF        ✗
+```
+
+---
+
+### `No inference engine available`
+
+llama-server is not running. Start it in a separate terminal before running any `jarvis`
+commands, and wait until you see `model loaded` in the output.
+
+---
+
+### Python version still shows 3.14 after recreating the venv
+
+Close the terminal completely and reopen it. The old venv path is cached in the shell
+environment and persists across commands until the session ends.
+
+---
+
+### `zsh: command not found: huggingface-cli`
+
+When installed via `uv tool`, the CLI is invoked as `hf`, not `huggingface-cli`:
+
+```bash
+hf download ...   # ✓
+huggingface-cli download ...   # ✗
+```
+
+---
+
+## Next Steps
+
+- [Quick Start](quickstart.md) — Run your first query and explore agents and tools
+- [Configuration](configuration.md) — Customize engine hosts, model routing, memory, and more
+- [Architecture](../architecture/overview.md) — Understand how OpenJarvis is structured
@@ -143,6 +143,7 @@ nav:
  - Home: index.md
  - Getting Started:
      - Installation: getting-started/installation.md
+      - macOS Guide: getting-started/macos.md
      - Quick Start: getting-started/quickstart.md
      - Code Snippets: getting-started/snippets.md
      - Configuration: getting-started/configuration.md