commit f1dfeb25676a2c4b3662fb881eee7cd852bbed80 Author: Jon Saad-Falcon Date: Sun Feb 15 00:23:58 2026 +0000 Initial commit: project vision, roadmap, and README Establishes OpenJarvis as a modular AI assistant backend with five composable pillars (Intelligence, Memory, Agents, Inference, Learning). Documents the architecture, supported backends, and phased development plan. Co-Authored-By: Claude Opus 4.6 diff --git a/README.md b/README.md new file mode 100644 index 00000000..de5a2ead --- /dev/null +++ b/README.md @@ -0,0 +1,45 @@ +# OpenJarvis + +**Your AI stack, your rules.** + +A modular, pluggable AI assistant backend. Compose your own stack across five pillars — Intelligence, Learning, Memory, Agents, and Inference — then swap any piece without touching the rest. + +> **Status: Early Development** — Interfaces are being defined. Not yet usable. + +## What is this? + +OpenJarvis lets you build a personal AI assistant from composable parts: + +- **Intelligence** — multi-model management with automatic routing (Qwen3, GPT OSS, Kimi-K2.5, Claude, GPT-5, Gemini) +- **Memory** — persistent, searchable storage with multiple backends (SQLite, FAISS, ColBERTv2, BM25, hybrid) +- **Agents** — pluggable reasoning and tool use (OpenClaw Pi agent, simple, orchestrator, custom) +- **Inference** — hardware-aware engine selection (vLLM, SGLang, Ollama, llama.cpp, MLX) +- **Learning** — router that improves over time (heuristic now, learned later) + +## Documentation + +- **[VISION.md](VISION.md)** — Project vision, architecture, design principles, and detailed pillar descriptions +- **[ROADMAP.md](ROADMAP.md)** — Phased development plan with deliverables and version milestones + +## Quick orientation + +``` +src/openjarvis/ +├── core/ # Registry, types, config, event bus +├── intelligence/ # Model management, routing +├── memory/ # Storage backends (SQLite, FAISS, ColBERT, BM25, hybrid) +├── agents/ # Agent implementations + tool system +├── engine/ # Inference engine wrappers +├── learning/ # Router policy (placeholder) +└── cli/ # CLI entry points (jarvis ask, serve, model, memory) +``` + +## Requirements + +- Python 3.10+ +- An inference backend: [Ollama](https://ollama.com), [vLLM](https://github.com/vllm-project/vllm), or [llama.cpp](https://github.com/ggerganov/llama.cpp) +- Node.js 22+ (only if using OpenClaw agent) + +## License + +TBD diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 00000000..13b2350c --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,345 @@ +# OpenJarvis Roadmap + +Phased development plan for OpenJarvis. Phases are ordered to maximize early usability: foundation first, then intelligence + inference (so you can ask questions), then memory (so it remembers), then agents (so it can act), then learning (so it improves). + +--- + +## Phase 0 — Foundation (~2-3 weeks) + +**Goal:** Repository scaffolding, core abstractions, and CLI skeleton. Nothing runs yet, but all interfaces are defined. + +**Version milestone:** v0.1 + +### Repository structure + +``` +OpenJarvis/ +├── pyproject.toml # uv/hatchling, all deps + extras +├── src/ +│ └── openjarvis/ +│ ├── __init__.py +│ ├── core/ +│ │ ├── registry.py # RegistryBase[T] + typed registries +│ │ ├── types.py # Message, Conversation, ModelSpec, ToolResult, TelemetryRecord +│ │ ├── config.py # JarvisConfig dataclass hierarchy, TOML loader +│ │ └── events.py # Event bus: pub/sub for inter-pillar telemetry +│ ├── intelligence/ # Phase 1 +│ ├── memory/ # Phase 2 +│ ├── agents/ # Phase 3 +│ ├── engine/ # Phase 1 +│ ├── learning/ # Phase 4 +│ └── cli/ # CLI entry points +├── tests/ +├── VISION.md +├── ROADMAP.md +└── README.md +``` + +### Deliverables + +- [ ] **Registry system** — `RegistryBase[T]` adapted from IPW's `registry.py`. Typed subclasses: + - `ModelRegistry` — model specs and metadata + - `EngineRegistry` — inference engine implementations + - `MemoryRegistry` — memory backend implementations + - `AgentRegistry` — agent implementations + - `ToolRegistry` — tools with `ToolSpec` metadata (category, cost, latency, capabilities) + +- [ ] **Core types** (`core/types.py`): + - `Message` — role + content + metadata (tool calls, images, etc.) + - `Conversation` — ordered list of messages with sliding window support + - `ModelSpec` — model ID, parameter count, quantization, context length, hardware compatibility + - `ToolResult` — tool name + output + usage + cost + - `TelemetryRecord` — timestamp, model, tokens, latency, energy (optional), cost + +- [ ] **Config system** (`core/config.py`): + - `JarvisConfig` dataclass hierarchy: `EngineConfig`, `IntelligenceConfig`, `MemoryConfig`, `AgentConfig` + - TOML config file at `~/.openjarvis/config.toml` + - Hardware auto-detection: GPU vendor/model/VRAM/platform → populate defaults + +- [ ] **Event bus** (`core/events.py`): + - Simple pub/sub for inter-pillar communication + - Telemetry events flow without tight coupling between pillars + - Synchronous dispatch (async optional later) + +- [ ] **CLI skeleton** (Click-based): + - `jarvis init` — create `~/.openjarvis/config.toml` with auto-detected defaults + - `jarvis ask` — placeholder (wired in Phase 1) + - `jarvis serve` — placeholder (wired in Phase 3) + - `jarvis model list|pull|info` — placeholder (wired in Phase 1) + - `jarvis memory index|search|stats` — placeholder (wired in Phase 2) + +--- + +## Phase 1 — Intelligence + Inference Engine (~3-4 weeks) + +**Goal:** You can ask OpenJarvis a question and get an answer. Models run locally or via cloud APIs. Basic telemetry records every call. + +**Version milestone:** v0.2 — first usable version + +### Inference Engine + +- [ ] **`InferenceEngine` ABC:** + ```python + class InferenceEngine(ABC): + def generate(self, model: str, messages: list[Message], **params) -> Response: ... + def stream(self, model: str, messages: list[Message], **params) -> Iterator[ResponseChunk]: ... + def list_models(self) -> list[ModelSpec]: ... + def health(self) -> bool: ... + ``` + +- [ ] **Engine implementations:** + - `OllamaEngine` — wraps Ollama HTTP API (`/api/chat`, `/api/tags`). Apple Silicon + NVIDIA. + - `VLLMEngine` — wraps vLLM OpenAI-compatible API. Multi-GPU, tensor parallelism. + - `LlamaCppEngine` — wraps `llama-cpp-python` or llama.cpp server. Maximum compatibility. + - `CloudEngine` — unified wrapper for OpenAI, Anthropic, and Google APIs. Key-based routing. + +- [ ] **Model management:** + - Auto-discovery from running engines (poll `/api/tags`, `/v1/models`) + - `ModelSpec` with hardware compatibility matrix (min VRAM, supported engines, quantization options) + - `jarvis model list` shows all available models across engines + - `jarvis model info ` shows spec, hardware requirements, estimated performance + +### Intelligence + +- [ ] **Hardware profiles:** + - Auto-detect: `nvidia-smi`, `rocm-smi`, `system_profiler` (macOS), `/proc/cpuinfo` + - Map GPU to capabilities: VRAM, compute capability, FP8/FP4 support, unified memory + - Recommend engine: Apple Silicon → Ollama/MLX, NVIDIA datacenter → vLLM, AMD → vLLM+ROCm, CPU → llama.cpp + +- [ ] **Heuristic Router V0:** + - Rule-based routing: short queries (< 50 tokens) → small model, complex (reasoning keywords, multi-step) → large model, code patterns → code specialist + - Fallback chains: if preferred model unavailable, try next in chain + - Configurable via `~/.openjarvis/config.toml` + +- [ ] **Basic telemetry:** + - Wrap every `generate()` / `stream()` call with timing + token counting + - Record to SQLite: model, prompt tokens, completion tokens, latency, cost estimate + - `TelemetryRecord` stored via event bus, accumulated for future learning phase + +### Wire-up + +- [ ] **`jarvis ask "What is X?"` works end-to-end:** + 1. Parse query → detect complexity → route to model + 2. Generate response via selected engine + 3. Record telemetry + 4. Print response (with optional `--json` output) + +--- + +## Phase 2 — Memory / Storage (~3-4 weeks) + +**Goal:** OpenJarvis remembers conversations, can index your documents, and injects relevant context into prompts. + +**Version milestone:** v0.3 + +### Memory backends + +- [ ] **`MemoryBackend` ABC:** + ```python + class MemoryBackend(ABC): + def store(self, content: str, metadata: dict) -> str: ... # Returns doc ID + def retrieve(self, query: str, k: int = 10) -> list[Result]: ... + def delete(self, doc_id: str) -> bool: ... + def clear(self) -> None: ... + ``` + +- [ ] **Memory subtypes:** + - `ConversationMemory` — sliding window (configurable size) + automatic summarization of older turns via LLM call + - `KnowledgeBase` — indexed document collection with multi-backend search + +- [ ] **Backend implementations:** + + - **`SQLiteMemory`** — FTS5 full-text search, zero-config default. Always available, no extra dependencies. + + - **`FAISSMemory`** — Dense neural retrieval. Encodes documents with `sentence-transformers`, builds FAISS index (IVF or flat depending on collection size). GPU-accelerated when available. + + - **`ColBERTMemory`** — ColBERTv2 late interaction retrieval. Best retrieval quality. + - Package: `colbert-ai[torch,faiss-gpu]` + - Indexing: `Indexer(checkpoint="colbertv2.0", config=ColBERTConfig(nbits=2))` → `indexer.index(name, collection)` + - Search: `Searcher(index=name)` → `searcher.search(query, k=10)` returns `(passage_ids, ranks, scores)` + - Token-level MaxSim matching: each query token attends to each document token, max-pooled per query token, summed + - 2-bit residual compression keeps indexes compact (~50x smaller than full embeddings) + - Millisecond query latency, substantially better than single-vector methods on complex queries + + - **`BM25Memory`** — Keyword search baseline using `rank-bm25`. No GPU, no embeddings. Fast and effective for keyword-heavy queries. + + - **`HybridMemory`** — Combines BM25 with a dense backend (FAISS or ColBERT) using Reciprocal Rank Fusion (RRF): + ``` + RRF_score(d) = sum(1 / (k + rank_i(d))) for each retriever i + ``` + Configurable `k` parameter (default 60). Best overall retrieval when you don't know the query type. + +### Document pipeline + +- [ ] **Indexing pipeline:** PDF / Markdown / plain text / code → chunking (configurable size + overlap) → embedding (if using dense/ColBERT backend) → index +- [ ] **Context injection:** auto-retrieve top-k relevant chunks before each LLM call, inject into prompt with source attribution (`[Source: filename:line]`) + +### CLI + +- [ ] `jarvis memory index ` — index a file or directory +- [ ] `jarvis memory search ` — search across all memory backends +- [ ] `jarvis memory stats` — show index sizes, document counts, backend status + +--- + +## Phase 3 — Agentic Logic (~3-4 weeks) + +**Goal:** OpenJarvis can use tools, reason over multiple turns, and serve an OpenAI-compatible API. The default agent is OpenClaw's Pi. + +**Version milestone:** v0.4 + +### Agent framework + +- [ ] **`BaseAgent` ABC:** + ```python + class BaseAgent(ABC): + def run(self, input: str, context: AgentContext) -> AgentResult: ... + ``` + `AgentContext` carries: conversation history, memory handle, tool registry, telemetry recorder, model router. + `AgentResult` contains: response text, tool calls made, tokens used, telemetry data. + +- [ ] **Agent implementations:** + + - **`OpenClawAgent`** (default) — wraps OpenClaw's Pi agent runtime (`@mariozechner/pi-coding-agent` v0.52.12+). Two modes: + 1. **HTTP mode:** OpenClaw gateway running locally on `:18789`. Communicate via WebSocket. Best for persistent sessions. + 2. **Subprocess mode:** invoke `node` with `runEmbeddedPiAgent()` call, JSON over stdin/stdout. No gateway needed. + - Capabilities: multi-turn reasoning, tool calling, streaming, skill composition, context compaction + - Requires Node.js 22+ + + - **`SimpleAgent`** — single-turn: parse query → call model → return response. No tool calling, no multi-turn. Works without Node.js. Good for testing and simple Q&A. + + - **`OrchestratorAgent`** — multi-turn with model selection per step. Adapted from IPW's executor pattern. Each reasoning step can route to a different model (e.g., fast model for planning, large model for synthesis). + + - **`CustomAgent`** — template class for user-defined agents. Subclass `BaseAgent`, implement `run()`, register with `@AgentRegistry.register("my-agent")`. + +### Tool system + +- [ ] **`BaseTool` ABC:** + ```python + class BaseTool(ABC): + name: str + spec: ToolSpec # category, cost_estimate, latency_estimate, capabilities + def execute(self, input: str, **params) -> ToolResult: ... + ``` + +- [ ] **Built-in tools:** + - `Calculator` — evaluate math expressions + - `WebSearch` — search the web (Tavily, SearXNG, or DuckDuckGo) + - `CodeInterpreter` — execute Python in sandboxed environment + - `FileRead` / `FileWrite` — local file operations + - `Think` — internal reasoning scratchpad (zero-cost tool for chain-of-thought) + - `Retrieval` — wired to memory backends, returns relevant documents + - `LLMTool` — call another LLM as a tool (for model composition) + +- [ ] **`ToolRegistry`** with discovery: + - `ToolSpec` metadata: category, estimated latency, estimated cost, required API keys, capabilities list + - Auto-discover available tools based on installed packages and environment + +### API server + +- [ ] **OpenAI-compatible API server:** + - `POST /v1/chat/completions` — standard chat completion with tool use + - `GET /v1/models` — list available models + - Streaming via Server-Sent Events (SSE) + - `jarvis serve --port 8000 --agent openclaw` + +### CLI + +- [ ] `jarvis serve --port 8000 --agent ` — start API server +- [ ] `jarvis ask` now supports `--agent ` flag + +--- + +## Phase 4 — Learning Approach (placeholder) + +**Goal:** Stub interfaces for the learned router. No ML training in this phase — just the contracts and telemetry plumbing so everything is ready when we build it. + +**Version milestone:** v0.5 + +### Stubs + +- [ ] **`RouterPolicy` ABC:** + ```python + class RouterPolicy(ABC): + def select_model(self, query: str, context: RoutingContext) -> ModelSpec: ... + ``` + The heuristic router from Phase 1 implements this as the default. + +- [ ] **`RewardFunction` ABC:** + ```python + class RewardFunction(ABC): + def compute(self, trajectory: Trajectory) -> float: ... + ``` + Placeholder implementations: `QualityReward` (LLM-judge), `LatencyReward` (inverse latency), `EnergyReward` (inverse energy), `CostReward` (inverse cost), `CompositeReward` (weighted combination). + +- [ ] **`TelemetryAggregator`:** + - Reads `TelemetryRecord` entries from SQLite (accumulated since Phase 1) + - Computes per-model statistics: average latency, token throughput, cost, quality (when graded) + - Exports training-ready datasets for the future GRPO pipeline + +- [ ] **Design document:** `docs/learning-pipeline.md` describing the planned GRPO training pipeline: + - Trajectory generation from Phases 1-3 telemetry + - Reward model training + - Policy optimization with GRPO + - Online evaluation and rollout strategy + +--- + +## Phase 5 — Integration & Polish (~3-4 weeks) + +**Goal:** Production-ready packaging, OpenClaw integration, benchmarking, SDK, and documentation. + +**Version milestone:** v1.0 + +### OpenClaw integration + +- [ ] **`openjarvis-openclaw` plugin package:** + - `register()` hook implementing OpenClaw's plugin API + - `registerProvider()` — wraps OpenJarvis as an OpenClaw `ProviderPlugin` (routes through OpenJarvis intelligence + engine) + - `registerTool()` — exposes OpenJarvis tools to OpenClaw + - `MemorySearchManager` — implements OpenClaw's `search()` / `sync()` / `status()` interface, backed by OpenJarvis memory + +### Deployment + +- [ ] **Dockerfile** — multi-stage build with optional GPU support +- [ ] **docker-compose.yml** — OpenJarvis + Ollama/vLLM + optional gateway +- [ ] **Service files** — systemd (Linux) and launchd (macOS) for running as a system service + +### Python SDK + +- [ ] **Programmatic API:** + ```python + from openjarvis import Jarvis + + j = Jarvis() # Auto-loads config + response = await j.ask("Explain transformers") # Uses router + engine + await j.memory.index("~/papers/") # Index documents + results = await j.memory.search("attention mechanism") + ``` + +### Benchmarking + +- [ ] **`jarvis bench` CLI:** + - `BaseBenchmark` / `DatasetBenchmark` ABCs (adapted from IPW's `BenchmarkSuite`) + - Run benchmarks across models, measure accuracy + latency + energy + - Output JSONL results + summary JSON + +### Documentation + +- [ ] Documentation site (MkDocs or similar) +- [ ] Getting started guide +- [ ] Plugin development guide +- [ ] API reference + +--- + +## Version Summary + +| Version | Phase | What you get | +|---------|-------|-------------| +| **v0.1** | Phase 0 | Scaffolding, registries, config, CLI skeleton | +| **v0.2** | Phase 1 | `jarvis ask` works — local & cloud inference with telemetry | +| **v0.3** | Phase 2 | Memory — index docs, conversation history, context injection | +| **v0.4** | Phase 3 | Agents + tools + OpenAI-compatible API server | +| **v0.5** | Phase 4 | Learning stubs — router policy interface, telemetry aggregation | +| **v1.0** | Phase 5 | Production — SDK, OpenClaw plugin, Docker, benchmarks, docs | diff --git a/VISION.md b/VISION.md new file mode 100644 index 00000000..3e28d6d1 --- /dev/null +++ b/VISION.md @@ -0,0 +1,299 @@ +# OpenJarvis + +**Your AI stack, your rules.** + +OpenJarvis is a modular, pluggable AI assistant backend. Instead of locking you into one model, one memory system, or one inference engine, OpenJarvis lets you compose your own stack across five pillars — then swap any piece without touching the rest. + +Built for developers who want full control and researchers who need reproducible, measurable AI systems. + +--- + +## The Five Pillars + +OpenJarvis is organized around five composable pillars. Each pillar defines a clear interface; implementations are discovered at runtime via a decorator-based registry system. + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ OpenJarvis │ +│ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Intelligence │ │ Learning │ │ Memory / │ │ +│ │ (Models) │ │ Approach │ │ Storage │ │ +│ │ │ │ (Router) │ │ │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ Agentic Logic │ │ +│ │ (Orchestration, Tools, Reasoning) │ │ +│ └──────────────────────┬───────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ Inference Engine │ │ +│ │ (vLLM, Ollama, llama.cpp, SGLang, MLX) │ │ +│ └──────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +### 1. Intelligence (Model Layer) + +**What it does:** Manages available language models — local and cloud — and routes queries to the best model for the task. + +**What's pluggable:** Models, model providers, routing heuristics. + +**Supported at launch:** + +| Category | Models | +|----------|--------| +| Open-source (local) | Qwen3 8B, Qwen3 32B, GPT OSS 120B, Kimi-K2.5, MiniMax-M2.5 | +| Cloud APIs | Claude (Anthropic), GPT-4o / GPT-5 (OpenAI), Gemini (Google) | + +**Key components:** + +- **`ModelRegistry`** — decorator-based registry mapping model keys to `ModelSpec` objects (parameter count, quantization, hardware compatibility, context length) +- **Heuristic Router (V0)** — rule-based routing: short queries → small model, complex reasoning → large model, code → code specialist, fallback chains for unavailable models +- **Auto-discovery** — detects models from running inference engines (Ollama, vLLM) and available API keys + +--- + +### 2. Learning Approach (Router Policy) + +**What it does:** Determines *which* model handles a given query. V1 is heuristic; future versions will learn from usage data. + +**What's pluggable:** Routing policy, reward functions, training pipeline. + +**V1 (Placeholder):** +- Heuristic routing based on query characteristics (length, complexity keywords, domain detection) +- Rule-based fallback chains +- All telemetry logged to SQLite for future training + +**Future (Post-V1):** +- Learned router via GRPO (Group Relative Policy Optimization) +- Preference learning from user feedback +- Continual fine-tuning on accumulated trajectories +- Multi-objective optimization: quality vs. latency vs. energy vs. cost + +--- + +### 3. Memory / Storage + +**What it does:** Provides persistent, searchable memory across conversations, documents, and personal notes. Memory is automatically injected into prompts with source attribution. + +**What's pluggable:** Storage backends, retrieval strategies, embedding models, chunking strategies. + +**Memory types:** +- **Conversation Memory** — sliding window with automatic summarization of older turns +- **Knowledge Base** — indexed documents (PDF, Markdown, code, text) with multi-backend search +- **Personal Notes** — user-created persistent notes and preferences +- **Episodic Memory** — records of past interactions, tool uses, and outcomes + +**Backend implementations:** + +| Backend | Type | Description | +|---------|------|-------------| +| **SQLite** (default) | Keyword + FTS | FTS5 full-text search. Zero dependencies, zero config. Always available. | +| **FAISS** | Dense retrieval | Neural semantic search via `sentence-transformers` + FAISS indexes. | +| **ColBERTv2** | Late interaction | Token-level MaxSim matching with 2-bit residual compression. Best retrieval quality. Uses `colbert-ai` package with `Indexer` for offline indexing and `Searcher` for millisecond-latency queries. | +| **BM25** | Sparse retrieval | Classic keyword search baseline. Fast, no GPU needed. | +| **Hybrid** | Fusion | BM25 + dense (or ColBERT) with Reciprocal Rank Fusion (RRF). Best of both worlds. | +| **Vector DB adapters** | Dense retrieval | Qdrant, ChromaDB connectors for users with existing vector infrastructure. | + +**ColBERTv2 details:** +- Late interaction model: queries and documents are encoded independently, then matched at the token level via MaxSim +- 2-bit residual compression keeps indexes compact while preserving quality +- Offline indexing via `Indexer(checkpoint="colbertv2.0", config=ColBERTConfig(nbits=2))` +- Millisecond query latency via `Searcher(index=name).search(query, k=10)` +- Substantially better retrieval quality than single-vector dense methods on complex queries + +--- + +### 4. Agentic Logic + +**What it does:** Orchestrates multi-turn reasoning, tool calling, and task execution. The agent layer sits between the user and the model, managing context, tools, and conversation flow. + +**What's pluggable:** Agent implementations, tools, tool registries, execution strategies. + +**Agent implementations:** + +| Agent | Description | +|-------|-------------| +| **`OpenClawAgent`** (default) | Wraps OpenClaw's Pi agent runtime. Multi-turn reasoning, tool calling, streaming responses, skill composition, context compaction. Two modes: **HTTP** (WebSocket to OpenClaw gateway on `:18789`) or **subprocess** (invoke `node` with `runEmbeddedPiAgent()`, JSON over stdin/stdout). Requires Node.js 22+. | +| **`SimpleAgent`** | Single-turn: query → model → response. No tool calling. Works without Node.js. Good for quick answers and testing. | +| **`OrchestratorAgent`** | Multi-turn with per-step model selection. Adapted from IPW's executor pattern. Routes each reasoning step to the optimal model. | +| **`CustomAgent`** | Template for user-defined agent logic. Subclass `BaseAgent`, implement `run()`, register with `AgentRegistry`. | + +**Tool system:** +- `BaseTool` ABC with `ToolSpec` metadata (category, cost estimate, latency estimate, capabilities) +- `ToolRegistry` — runtime-discoverable tool catalog +- Built-in tools: Calculator, WebSearch, CodeInterpreter, FileRead/Write, Think, Retrieval (wired to memory backends), LLM-as-tool +- MCP (Model Context Protocol) compatible + +**API server:** +- OpenAI-compatible `/v1/chat/completions` and `/v1/models` endpoints +- Streaming via Server-Sent Events (SSE) +- Drop-in replacement for any OpenAI-compatible client + +--- + +### 5. Inference Engine + +**What it does:** Manages the actual LLM inference runtime — loading models, generating tokens, managing GPU memory. + +**What's pluggable:** Engine backends, hardware profiles, quantization strategies. + +**Supported engines:** + +| Engine | Best for | GPU | CPU | +|--------|----------|-----|-----| +| **vLLM** | High-throughput server, multi-GPU, production | NVIDIA, AMD | — | +| **SGLang** | Structured generation, constrained decoding | NVIDIA, AMD | — | +| **Ollama** | Easy setup, Apple Silicon, single-model | NVIDIA, Apple | Yes | +| **llama.cpp** | Maximum hardware compatibility, GGUF models | NVIDIA, AMD, Apple | Yes | +| **MLX** | Apple Silicon native, Metal acceleration | Apple | Apple | + +**Hardware auto-detection:** +- Detects GPU vendor (NVIDIA/AMD/Apple), model, VRAM, compute capability +- Recommends the best engine for detected hardware +- Apple Silicon → Ollama or MLX; NVIDIA datacenter → vLLM; AMD → vLLM with ROCm; CPU-only → llama.cpp + +--- + +## Query Flow + +``` +User query + │ + ▼ +┌──────────┐ ┌──────────┐ ┌──────────┐ +│ Agentic │────▶│ Memory │────▶│ Context │ +│ Logic │ │ Retrieve │ │ Inject │ +└────┬─────┘ └──────────┘ └────┬─────┘ + │ │ + ▼ ▼ +┌──────────┐ ┌──────────┐ ┌──────────┐ +│ Learning │────▶│ Model │────▶│ Inference│ +│ (Router) │ │ Select │ │ Engine │ +└──────────┘ └──────────┘ └────┬─────┘ + │ + ▼ + ┌──────────┐ + │ Response │ + │ + Telem. │ + └──────────┘ +``` + +1. **Agentic Logic** receives the user query, determines if tools or memory are needed +2. **Memory** retrieves relevant context (conversation history, documents, notes) +3. **Context Injection** assembles the full prompt with retrieved content and source attribution +4. **Learning/Router** selects the best model for this query based on routing policy +5. **Inference Engine** runs the selected model and streams the response +6. **Telemetry** records timing, token counts, energy (if available), and cost + +--- + +## User Scenarios + +### Developer on M4 Max MacBook Pro (128 GB unified memory) + +```toml +# ~/.openjarvis/config.toml +[engine] +backend = "ollama" # Native Apple Silicon support + +[intelligence] +default_model = "qwen3-32b" # Fits in 128 GB unified memory +fallback = "qwen3-8b" + +[memory] +backend = "sqlite" # Zero-config, always works +retrieval = "hybrid" # BM25 + FAISS for local docs + +[agent] +type = "openclaw" # Full agent capabilities +mode = "subprocess" # No separate gateway needed +``` + +Day-to-day: codes with `jarvis ask`, indexes project docs with `jarvis memory index`, runs a local OpenAI-compatible server with `jarvis serve` for editor integration. + +### Researcher on DGX Spark (2x B200, 384 GB GPU memory) + +```toml +[engine] +backend = "vllm" +tensor_parallel = 2 + +[intelligence] +default_model = "qwen3-235b-a22b" +router = "heuristic" # Route small queries to 8B, large to 235B + +[memory] +backend = "colbert" # Best retrieval quality for papers +knowledge_base = "~/papers/" + +[agent] +type = "orchestrator" # Multi-model orchestration +``` + +Running benchmarks with `jarvis bench`, profiling energy per query, comparing model efficiency across hardware configurations. + +### Privacy-Focused Offline Setup + +```toml +[engine] +backend = "llamacpp" # No server needed +network = "offline" + +[intelligence] +default_model = "qwen3-8b-q4" # Quantized to fit available RAM + +[memory] +backend = "sqlite" # Everything local +retrieval = "bm25" # No neural models needed + +[agent] +type = "simple" # No external dependencies +``` + +Fully air-gapped. No cloud APIs, no network calls, no telemetry export. All data stays on the machine. + +--- + +## Comparison + +| Feature | OpenJarvis | Ollama | LangChain | OpenClaw | vLLM | +|---------|-----------|--------|-----------|----------|------| +| **Focus** | Composable AI backend | Model runner | LLM app framework | AI coding assistant | Inference server | +| **Model management** | Multi-engine, auto-detect | Single engine | Bring your own | Cloud-first | Single engine | +| **Memory** | Multi-backend retrieval | None | Vector store wrappers | Conversation only | None | +| **Agents** | Pluggable (Pi, custom) | None | Chain-based | Pi agent (built-in) | None | +| **Inference** | vLLM/SGLang/Ollama/llama.cpp/MLX | Ollama only | External | External | vLLM only | +| **Hardware-aware** | Auto-detect + recommend | Manual | No | No | Manual | +| **Telemetry** | Energy, latency, cost | None | Callbacks | Basic | Metrics | +| **Offline** | Full support | Full support | Partial | No | Full support | +| **API** | OpenAI-compatible | OpenAI-compatible | Custom | Custom | OpenAI-compatible | +| **Language** | Python | Go | Python | TypeScript | Python | + +OpenJarvis is **not** a replacement for these tools — it *composes* them. Ollama and vLLM are inference engine options. OpenClaw's Pi agent is the default agentic logic. LangChain-style chains can be implemented as custom agents. + +--- + +## Design Principles + +1. **Pluggable everything** — every component is registered and discoverable at runtime. Swap models, engines, memory backends, and agents without code changes. + +2. **Registry-driven** — `RegistryBase[T]` pattern (adapted from IPW) provides type-safe, decorator-based registration for all extensible components: `ModelRegistry`, `EngineRegistry`, `MemoryRegistry`, `AgentRegistry`, `ToolRegistry`. + +3. **Offline-first** — works without network access. Cloud APIs are optional enhancements, never requirements. + +4. **Telemetry-native** — every inference call records timing, token counts, and (when hardware supports it) energy consumption. Data lands in SQLite for analysis. + +5. **Hardware-aware** — auto-detects GPU vendor, model, VRAM, and platform. Recommends the best engine and model configuration for your hardware. + +6. **Python-first** — core is pure Python (3.10+). Node.js required only for OpenClaw agent integration. No Java, no JVM, no heavy runtimes. + +7. **OpenAI-compatible API** — `jarvis serve` exposes `/v1/chat/completions` and `/v1/models`. Any client that speaks OpenAI protocol works out of the box. + +8. **Standalone** — OpenJarvis is a self-contained backend. OpenClaw is one possible frontend; so is `curl`, a Python SDK call, or any OpenAI-compatible client.