From 70ef42750b9c40d8bb0625c7d39f79b9759497f4 Mon Sep 17 00:00:00 2001 From: Aqil Aziz Date: Fri, 22 May 2026 18:06:51 +0700 Subject: [PATCH] docs(local-ai): document local chat routing (#2383) Co-authored-by: Aqil Aziz --- gitbooks/features/model-routing/local-ai.md | 123 ++++++++++++-------- 1 file changed, 74 insertions(+), 49 deletions(-) diff --git a/gitbooks/features/model-routing/local-ai.md b/gitbooks/features/model-routing/local-ai.md index 08d742523..643a585a9 100644 --- a/gitbooks/features/model-routing/local-ai.md +++ b/gitbooks/features/model-routing/local-ai.md @@ -1,69 +1,94 @@ --- description: >- Optional, opt-in local AI via Ollama or LM Studio. Powers memory embeddings, summary-tree - building, and background loops on-device. Chat / vision / voice are cloud. + building, background loops, and explicitly routed chat/reasoning workloads on-device. icon: microchip --- # Local AI (optional) -OpenHuman can run a local model on your machine for the workloads where keeping data on-device matters most: **memory embeddings, summary-tree building, and background reasoning loops**. It is **opt-in** and ships **off** by default. +OpenHuman can run a local model on your machine for workloads where keeping data on-device matters: **memory embeddings, summary-tree building, background reasoning loops, and explicitly routed chat or reasoning workloads**. It is **opt-in** and ships **off** by default. -This is a deliberate scoping. The previous design tried to put chat, vision, STT and TTS all on-device with Gemma 3, and the result was a heavy, hardware-sensitive footprint that fought with what the rest of the product needed to be. Today, the things that benefit most from being local (recurring, low-latency, privacy-sensitive memory work) run local; the things that benefit most from frontier models (default chat, reasoning, vision) stay cloud. +This is deliberate scoping. The previous design tried to put every modality on-device by default, and the result was a heavy, hardware-sensitive footprint. Today, local AI stays explicit: recurring privacy-sensitive work can run locally, and chat/reasoning can also run locally when you route those workloads to a local provider. ## What runs local when you turn it on -| Workload | Default model | Implementation | -| ------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -| **Memory embeddings** | `all-minilm:latest` | `src/openhuman/embeddings/ollama.rs` - used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search. | -| **Summary-tree building** | `gemma3:1b-it-qat` (configurable) | `src/openhuman/tree_summarizer/ops.rs` - source / topic / global summary builders for the Memory Tree. | -| **Heartbeat loop** | small chat model | `src/openhuman/heartbeat/` - periodic background reflection. | -| **Learning / reflection** | small chat model | `src/openhuman/learning/reflection.rs` - passes that consolidate what was learned. | -| **Subconscious** | small chat model | `src/openhuman/subconscious/executor.rs` - background evaluation loop. | +| Workload | Default model | Implementation | +| ------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| **Memory embeddings** | `all-minilm:latest` | `src/openhuman/embeddings/ollama.rs` - used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search. | +| **Summary-tree building** | `gemma3:1b-it-qat` (configurable) | `src/openhuman/tree_summarizer/ops.rs` - source / topic / global summary builders for the Memory Tree. | +| **Heartbeat loop** | small chat model | `src/openhuman/heartbeat/` - periodic background reflection. | +| **Learning / reflection** | small chat model | `src/openhuman/learning/reflection.rs` - passes that consolidate what was learned. | +| **Subconscious** | small chat model | `src/openhuman/subconscious/executor.rs` - background evaluation loop. | +| **Chat** | configured local chat model | `Config::workload_local_model("chat")` reads `chat_provider`; `src/openhuman/routing/provider.rs` handles hint routing. | +| **Reasoning** | configured local chat model | `Config::workload_local_model("reasoning")` reads `reasoning_provider`; see [Opting in](#opting-in). | -Each of these is a **per-feature opt-in flag**. Turning on local AI does not silently route everything through it, you choose the workloads. +Each of these is an explicit opt-in. Turning on local AI does not silently route everything through it, you choose the workloads. -## What stays in the cloud +## What stays in the cloud by default -| Workload | Why cloud | -| ------------------ | --------------------------------------------------------------------------------------------------- | -| **Chat (default)** | Frontier reasoning quality. Routed via the [model router](README.md) under one subscription. | -| **Vision** | Same. | -| **STT** | Backend-proxied transcription (`src/openhuman/voice/cloud_transcribe.rs`). | -| **TTS** | Hosted [text-to-speech](../native-tools/voice.md) under the hood (`reply_speech.rs`). | -| **Web search** | Backend proxy (no API key on your machine). | +| Workload | Why cloud | +| -------------- | ---------------------------------------------------------------------------------------------- | +| **Chat** | Frontier reasoning quality unless `chat_provider` is explicitly set to a local provider. | +| **Reasoning** | Stronger multi-step quality unless `reasoning_provider` is explicitly set to a local provider. | +| **Vision** | Same. | +| **STT** | Backend-proxied transcription (`src/openhuman/voice/cloud_transcribe.rs`). | +| **TTS** | Hosted [text-to-speech](../native-tools/voice.md) under the hood (`reply_speech.rs`). | +| **Web search** | Backend proxy (no API key on your machine). | -For **lightweight or medium chat hints** (`hint:reaction`, `hint:classify`, `hint:format`, `hint:sentiment`, `hint:summarize`, `hint:medium`, `hint:tool_lite`), the [router](README.md) will prefer the local provider when local AI is enabled and Ollama is reachable. Heavy hints (`hint:reasoning`, `hint:agentic`, `hint:coding`) stay cloud. +For **lightweight or medium chat hints** (`hint:reaction`, `hint:classify`, `hint:format`, `hint:sentiment`, `hint:summarize`, `hint:medium`, `hint:tool_lite`), the [router](README.md) can prefer the local provider only when `local_ai.runtime_enabled = true` and the configured local provider is reachable. + +Heavy hints (`hint:reasoning`, `hint:agentic`, `hint:coding`) stay cloud by default unless the matching workload provider field is explicitly configured locally. ## How it works Under the hood, OpenHuman supports two local provider paths: -* [Ollama](https://ollama.com), used for bundled model lifecycle, embeddings, and the existing model-asset flow. -* [LM Studio](https://lmstudio.ai), used through its local OpenAI-compatible server for chat-style local inference. +- [Ollama](https://ollama.com), used for bundled model lifecycle, embeddings, and the existing model-asset flow. +- [LM Studio](https://lmstudio.ai), used through its local OpenAI-compatible server for chat-style local inference. For Ollama, OpenHuman talks to its OpenAI-compatible `/v1` endpoint where possible. That means: -* The `OpenAiCompatibleProvider` (`src/openhuman/providers/compatible.rs`) wraps Ollama exactly the way it wraps a remote OpenAI-style provider. No special-case code path. -* The provider router creates a _health-gated_ local provider on startup. If Ollama is not reachable, requests transparently fall back to the remote provider, no broken state. -* Models are pulled on demand by Ollama and cached in its own store. OpenHuman doesn't ship the weights itself. +- The `OpenAiCompatibleProvider` (`src/openhuman/providers/compatible.rs`) wraps Ollama exactly the way it wraps a remote OpenAI-style provider. No special-case code path. +- The provider router creates a _health-gated_ local provider on startup. If Ollama is not reachable, requests transparently fall back to the remote provider, no broken state. +- Models are pulled on demand by Ollama and cached in its own store. OpenHuman doesn't ship the weights itself. For LM Studio, set `local_ai.provider = "lm_studio"` and ensure LM Studio's local server is running. OpenHuman defaults to `http://localhost:1234/v1`, probes `GET /v1/models`, and sends chat requests to `POST /v1/chat/completions`. You can override the endpoint with `local_ai.base_url`, `OPENHUMAN_LM_STUDIO_BASE_URL`, or `LM_STUDIO_BASE_URL`. ## Opting in -Local AI is gated by two flags in the core config (`src/openhuman/config/schema/local_ai.rs`): +Local runtime startup is gated in the core config (`src/openhuman/config/schema/local_ai.rs`): -| Flag | Default | Meaning | -| ------------------------------------ | ------- | ------------------------------------------------------------------- | -| `local_ai.runtime_enabled` | `false` | Master switch. `false` ⇒ no local provider is created at all. | -| `local_ai.opt_in_confirmed` | `false` | Explicit opt-in marker. Bootstrap forces `false` unless you re-opt. | -| `local_ai.provider` | `ollama` | Local provider: `ollama` or `lm_studio`. | -| `local_ai.base_url` | unset | Optional provider URL. LM Studio defaults to `http://localhost:1234/v1`. | -| `local_ai.usage.embeddings` | `false` | Use local for memory embeddings. | -| `local_ai.usage.heartbeat` | `false` | Use local for the heartbeat loop. | -| `local_ai.usage.learning_reflection` | `false` | Use local for learning passes. | -| `local_ai.usage.subconscious` | `false` | Use local for the subconscious loop. | +| Flag | Default | Meaning | +| ------------------------------------ | -------- | ------------------------------------------------------------------------ | +| `local_ai.runtime_enabled` | `false` | Master switch. `false` ⇒ no local provider is created at all. | +| `local_ai.opt_in_confirmed` | `false` | Explicit opt-in marker. Bootstrap forces `false` unless you re-opt. | +| `local_ai.provider` | `ollama` | Local provider: `ollama` or `lm_studio`. | +| `local_ai.base_url` | unset | Optional provider URL. LM Studio defaults to `http://localhost:1234/v1`. | +| `local_ai.usage.embeddings` | `false` | Legacy preset/migration flag for memory embeddings. | +| `local_ai.usage.heartbeat` | `false` | Legacy preset/migration flag for the heartbeat loop. | +| `local_ai.usage.learning_reflection` | `false` | Legacy preset/migration flag for learning passes. | +| `local_ai.usage.subconscious` | `false` | Legacy preset/migration flag for the subconscious loop. | + +Unified workload provider fields control chat/reasoning routing. Set them to an Ollama provider string when you want those paths on-device: + +```toml +chat_provider = "ollama:llama3.1:8b" +reasoning_provider = "ollama:qwen2.5:14b" +``` + +On current configs, the `*_provider` fields are the source of truth for workload routing (`Config::workload_local_model(...)` in `src/openhuman/config/schema/types.rs`). Unset, blank, `cloud`, `openhuman`, or any non-`ollama:` value keeps that workload on the cloud/default route. Setting a provider string such as `ollama:all-minilm:latest` or `ollama:qwen2.5:14b` routes that workload on-device when `local_ai.runtime_enabled = true` and the provider health check passes. + +The legacy `local_ai.usage.*` booleans are kept for presets and migration compatibility; they do not override the unified provider fields after migration. For deterministic routing, either set the workload provider field explicitly, or leave it unset / set it to `cloud` to force the default cloud route. The same provider-string pattern is used by `agentic_provider`, `coding_provider`, `memory_provider`, `embeddings_provider`, `heartbeat_provider`, `learning_provider`, and `subconscious_provider`. + +### Legacy flag behavior + +The `local_ai.usage.*` booleans are consulted only during preset application and initial migration. After that, `Config::workload_local_model(...)` treats the matching `*_provider` field as the definitive routing control: + +- `embeddings_provider = "ollama:all-minilm"` routes embeddings on-device even if `local_ai.usage.embeddings = false`. +- An unset, blank, or `cloud` `embeddings_provider` keeps embeddings on the cloud/default route even if `local_ai.usage.embeddings = true`. + +Prefer setting the `*_provider` fields directly when editing configuration by hand. In the desktop app, **Settings → AI & Skills → Local AI** exposes presets, pick one ("embeddings only", "memory + reflection", "everything local") and the right combination of flags is set for you. Status (Ollama reachability, model availability, per-subsystem enablement) is surfaced live via `openhuman.local_ai_status`. @@ -71,29 +96,29 @@ In the desktop app, **Settings → AI & Skills → Local AI** exposes presets, p Local AI is worth turning on if any of these are true: -* You ingest large volumes of email / chat and want **embeddings to never leave the machine**. -* You want **summary-tree building** to work offline. -* You're privacy-sensitive about background reflection ("subconscious") loops. +- Keep embeddings local when ingesting large volumes of email / chat. +- Enable **summary-tree building** to work offline. +- Keep background reflection ("subconscious") loops on-device for privacy-sensitive work. It is **not** worth turning on if you only have a few sources connected, the cloud path is faster and the privacy benefit is small. There is also a hardware cost: Ollama and a small Gemma model want a few GB of RAM and pull a few GB of weights. ## What you'll need -* [**Ollama**](https://ollama.com) installed and running locally, or [**LM Studio**](https://lmstudio.ai) with the local server enabled. -* Enough disk for the models (`gemma3:1b-it-qat` \~700 MB, `all-minilm:latest` \~23 MB). -* Enough RAM to keep the model resident (8 GB+ recommended, 16 GB+ ideal). +- [**Ollama**](https://ollama.com) installed and running locally, or [**LM Studio**](https://lmstudio.ai) with the local server enabled. +- Enough disk for the models (`gemma3:1b-it-qat` \~700 MB, `all-minilm:latest` \~23 MB). +- Enough RAM to keep the model resident (8 GB+ recommended, 16 GB+ ideal). OpenHuman handles the rest: lifecycle (`src/openhuman/local_ai/service/`), API clients (`ollama_api.rs`, `lm_studio_api.rs`), health checks, and graceful fallback to remote when the local provider disappears. ### LM Studio troubleshooting -* Confirm the LM Studio local server is enabled and reachable at `http://localhost:1234/v1`. -* Load the selected model in LM Studio before calling OpenHuman. Diagnostics report `load_lm_studio_model` when the configured `local_ai.chat_model_id` is not present in `/v1/models`. -* If LM Studio uses a different port, set `local_ai.base_url` or `OPENHUMAN_LM_STUDIO_BASE_URL`. -* LM Studio model downloads are managed inside LM Studio. OpenHuman will not pull LM Studio models from the local asset-download controls. +- Confirm the LM Studio local server is enabled and reachable at `http://localhost:1234/v1`. +- Load the selected model in LM Studio before calling OpenHuman. Diagnostics report `load_lm_studio_model` when the configured `local_ai.chat_model_id` is not present in `/v1/models`. +- If LM Studio uses a different port, set `local_ai.base_url` or `OPENHUMAN_LM_STUDIO_BASE_URL`. +- LM Studio model downloads are managed inside LM Studio. OpenHuman will not pull LM Studio models from the local asset-download controls. ## See also -* [Memory Tree](../obsidian-wiki/memory-tree.md). what local embeddings + summarization power. -* [Automatic Model Routing](README.md). how lightweight chat hints prefer the local provider. -* [Privacy & Security](../privacy-and-security.md). what moves on-device when you opt in. +- [Memory Tree](../obsidian-wiki/memory-tree.md). what local embeddings + summarization power. +- [Automatic Model Routing](README.md). how lightweight chat hints prefer the local provider. +- [Privacy & Security](../privacy-and-security.md). what moves on-device when you opt in.