Documentation update and lint

This commit is contained in:
Eddie Richter
2026-03-30 21:48:39 -06:00
parent d9983077f0
commit 13ea4c53db
5 changed files with 36 additions and 15 deletions
+20 -4
View File
@@ -116,6 +116,7 @@ All providers produce the same output format consumed by agents:
| **LM Studio** | `lmstudio` | OpenAI-compatible | 1234 | No (GPU optional) | Desktop GUI, easy model management |
| **Exo** | `exo` | OpenAI-compatible | 52415 | No (distributed) | Distributed inference across heterogeneous devices |
| **Nexa** | `nexa` | OpenAI-compatible | 18181 | No (CPU/GPU) | On-device inference with GGUF models |
| **Lemonade** | `lemonade` | OpenAI-compatible | 8000 | AMD GPU/NPU | AMD consumer GPUs (RDNA), Ryzen AI NPUs |
| **Uzu** | `uzu` | OpenAI-compatible | 8000 | Varies | Uzu inference runtime |
| **Apple FM** | `apple_fm` | OpenAI-compatible | 8079 | Apple Silicon | Apple Foundation Model on-device inference |
| **LiteLLM** | `litellm` | OpenAI-compatible | — | No | Unified proxy to 100+ LLM providers |
@@ -132,7 +133,7 @@ The Ollama backend communicates via Ollama's native HTTP API at `/api/chat` and
### vLLM
The vLLM backend uses the OpenAI-compatible `/v1/chat/completions` API. It is recommended for datacenter GPUs (A100, H100, L40, A10, A30) and AMD GPUs.
The vLLM backend uses the OpenAI-compatible `/v1/chat/completions` API. It is recommended for datacenter GPUs (NVIDIA A100, H100, L40, A10, A30 and AMD MI300, MI325, MI350, MI355).
- **Default host:** `http://localhost:8000`
- **Health check:** `GET /v1/models`
@@ -199,6 +200,15 @@ The Nexa backend connects to the Nexa SDK on-device inference server via a FastA
- **Install:** `pip install nexaai`
- **Best for:** On-device inference with GGUF models on Apple Silicon or CPU
### Lemonade
The Lemonade backend connects to the [Lemonade](https://lemonade-server.ai/) inference server, which is optimized for AMD consumer GPUs (RDNA architecture) and Ryzen AI Neural Processing Units (NPUs). It uses the OpenAI-compatible `/v1/chat/completions` API.
- **Default host:** `http://localhost:8000`
- **Health check:** `GET /v1/models`
- **Install:** Visit [lemonade-server.ai](https://lemonade-server.ai/) for platform-specific installation instructions
- **Best for:** Ryzen AI GPUs and NPUs, and AMD-based desktop and laptop systems
### Uzu
The Uzu backend connects to the Uzu inference runtime. Unlike other OpenAI-compatible engines, Uzu serves its API at the root path (no `/v1` prefix).
@@ -254,7 +264,9 @@ graph TD
D -->|NVIDIA| F{"Datacenter card?<br/>(A100, H100, H200,<br/>L40, A10, A30)"}
F -->|Yes| G["vllm"]
F -->|No| H["ollama"]
D -->|AMD| I["vllm"]
D -->|AMD| I{"Datacenter card?<br/>(MI300, MI325,<br/>MI350, MI355)"}
I -->|Yes| K["vllm"]
I -->|No| L["lemonade"]
D -->|Other| J["llamacpp"]
```
@@ -301,7 +313,7 @@ models = discover_models(engines)
## OpenAI Compatibility Layer
The `_OpenAICompatibleEngine` base class provides a shared implementation for engines that serve the standard `/v1/chat/completions` endpoint. vLLM, SGLang, and llama.cpp all extend this base class with minimal overrides -- typically just setting `engine_id` and `_default_host`.
The `_OpenAICompatibleEngine` base class provides a shared implementation for engines that serve the standard `/v1/chat/completions` endpoint. vLLM, SGLang, llama.cpp, Lemonade, and others all extend this base class with minimal overrides -- typically just setting `engine_id` and `_default_host`.
```python
class _OpenAICompatibleEngine(InferenceEngine):
@@ -343,6 +355,9 @@ host = "http://localhost:30000"
# [engine.llamacpp]
# host = "http://localhost:8080"
# binary_path = ""
# [engine.lemonade]
# host = "http://localhost:8000"
```
The `EngineConfig` dataclass and its per-engine sub-dataclasses map these settings:
@@ -355,9 +370,10 @@ The `EngineConfig` dataclass and its per-engine sub-dataclasses map these settin
| `SGLangEngineConfig` | `host` | `http://localhost:30000` | SGLang server URL |
| `LlamaCppEngineConfig` | `host` | `http://localhost:8080` | llama.cpp server URL |
| `LlamaCppEngineConfig` | `binary_path` | `""` | Path to llama.cpp binary (for managed mode) |
| `LemonadeEngineConfig` | `host` | `http://localhost:8000` | Lemonade server URL |
!!! note "Backward compatibility"
The old flat field names `ollama_host`, `vllm_host`, `llamacpp_host`, `llamacpp_path`, and `sglang_host` under `[engine]` are still accepted as backward-compatible properties on `EngineConfig`. New configurations should use the nested sub-section format.
The old flat field names `ollama_host`, `vllm_host`, `llamacpp_host`, `llamacpp_path`, `sglang_host`, and `lemonade_host` under `[engine]` are still accepted as backward-compatible properties on `EngineConfig`. New configurations should use the nested sub-section format.
---
+1 -4
View File
@@ -330,10 +330,7 @@ class AgentExecutor:
instruction = config.get("instruction", "")
memory = agent.get("summary_memory", "")
if instruction:
input_text = (
f"Current date: {today}\n\n"
f"Standing instruction: {instruction}"
)
input_text = f"Current date: {today}\n\nStanding instruction: {instruction}"
if memory:
input_text += f"\n\nPrevious context: {memory}"
else:
+1
View File
@@ -204,6 +204,7 @@ def compose_run(name: str, query: tuple[str, ...], output_json: bool) -> None:
if output_json:
import json as json_mod
if isinstance(result, dict):
click.echo(json_mod.dumps(result, indent=2, default=str))
elif isinstance(result, str):
+4 -4
View File
@@ -162,10 +162,10 @@ def _next_steps_text(engine: str, model: str = "") -> str:
),
"lemonade": (
"Next steps:\n\n"
" 1. Install and start Lemonade:\n"
" pip install lemonade-server\n"
" lemonade-server\n\n"
" 2. Try it out:\n"
" 1. Install Lemonade for your platform:\n"
" https://lemonade-server.ai/\n\n"
" 2. Start the Lemonade server\n\n"
" 3. Try it out:\n"
' jarvis ask "Hello"\n\n'
" Run `jarvis doctor` to verify your setup."
),
+10 -3
View File
@@ -155,8 +155,10 @@ class TestAMDEngineRecommendation:
cpu_count=96,
ram_gb=768.0,
gpu=GpuInfo(
vendor="amd", name="AMD Instinct MI350X",
vram_gb=288.0, count=1,
vendor="amd",
name="AMD Instinct MI350X",
vram_gb=288.0,
count=1,
),
)
assert recommend_engine(hw) == "vllm"
@@ -167,7 +169,12 @@ class TestAMDEngineRecommendation:
cpu_brand="AMD Ryzen 9 7950X",
cpu_count=32,
ram_gb=64.0,
gpu=GpuInfo(vendor="amd", name="AMD Radeon RX 7900 XTX", vram_gb=24.0, count=1),
gpu=GpuInfo(
vendor="amd",
name="AMD Radeon RX 7900 XTX",
vram_gb=24.0,
count=1,
),
)
assert recommend_engine(hw) == "lemonade"