diff --git a/docs/api/agents.md b/docs/api/agents.md index 05fff4b6..f257a4c6 100644 --- a/docs/api/agents.md +++ b/docs/api/agents.md @@ -6,7 +6,7 @@ coordinating tool calls, memory retrieval, and inference engine interactions. The module also includes the OpenClaw infrastructure for interoperating with external agent frameworks via HTTP or subprocess transport. -## Abstract Base Class and Context +## Abstract Base Classes and Context ### BaseAgent @@ -15,6 +15,13 @@ external agent frameworks via HTTP or subprocess transport. show_source: true members_order: source +### ToolUsingAgent + +::: openjarvis.agents._stubs.ToolUsingAgent + options: + show_source: true + members_order: source + ### AgentContext ::: openjarvis.agents._stubs.AgentContext @@ -47,16 +54,30 @@ external agent frameworks via HTTP or subprocess transport. show_source: true members_order: source -### OpenClawAgent +### NativeReActAgent -::: openjarvis.agents.openclaw.OpenClawAgent +::: openjarvis.agents.native_react.NativeReActAgent options: show_source: true members_order: source -### CustomAgent +### NativeOpenHandsAgent -::: openjarvis.agents.custom.CustomAgent +::: openjarvis.agents.native_openhands.NativeOpenHandsAgent + options: + show_source: true + members_order: source + +### RLMAgent + +::: openjarvis.agents.rlm.RLMAgent + options: + show_source: true + members_order: source + +### OpenHandsAgent + +::: openjarvis.agents.openhands.OpenHandsAgent options: show_source: true members_order: source diff --git a/docs/architecture/agents.md b/docs/architecture/agents.md index 6382fffb..67efb12f 100644 --- a/docs/architecture/agents.md +++ b/docs/architecture/agents.md @@ -1,16 +1,27 @@ # Agentic Logic Pillar -The Agentic Logic pillar provides **pluggable agents** that handle queries with varying levels of sophistication -- from simple single-turn responses to multi-turn tool-calling loops and external agent communication. +The Agentic Logic pillar provides **pluggable agents** that handle queries with varying levels of sophistication -- from simple single-turn responses to multi-turn tool-calling loops, ReAct-style reasoning, CodeAct code execution, recursive decomposition, and external agent communication. --- ## BaseAgent ABC -All agents implement the `BaseAgent` abstract base class: +All agents implement the `BaseAgent` abstract base class, which provides both the `run()` contract and concrete helper methods that eliminate boilerplate in subclasses: ```python class BaseAgent(ABC): agent_id: str + accepts_tools: bool = False # overridden by ToolUsingAgent + + def __init__( + self, + engine: InferenceEngine, + model: str, + *, + bus: Optional[EventBus] = None, + temperature: float = 0.7, + max_tokens: int = 1024, + ) -> None: ... @abstractmethod def run( @@ -22,6 +33,23 @@ class BaseAgent(ABC): """Execute the agent on *input* and return an AgentResult.""" ``` +### Class Attribute: `accepts_tools` + +The `accepts_tools` class attribute (default `False`) enables the CLI and SDK to auto-detect which agents support tool-passing. Agents that set `accepts_tools = True` can receive `--tools` on the CLI and `tools=` in the SDK. + +### Concrete Helper Methods + +`BaseAgent` provides five concrete helpers that subclasses use to avoid duplicating common logic: + +| Helper | Purpose | +|--------|---------| +| `_emit_turn_start(input)` | Publish `AGENT_TURN_START` on the event bus | +| `_emit_turn_end(**data)` | Publish `AGENT_TURN_END` on the event bus | +| `_build_messages(input, context, *, system_prompt)` | Assemble the message list from optional system prompt, conversation context, and user input | +| `_generate(messages, **extra_kwargs)` | Call `engine.generate()` with stored defaults (model, temperature, max_tokens) | +| `_max_turns_result(tool_results, turns, content)` | Build the standard `AgentResult` for when `max_turns` is exceeded | +| `_strip_think_tags(text)` | Remove `...` blocks from model output (static method) | + ### The `run()` Contract The `run()` method is the single entry point for all agent implementations. It receives: @@ -52,13 +80,42 @@ class AgentResult: --- +## ToolUsingAgent + +`ToolUsingAgent` is an intermediate base class for agents that accept and use tools. It extends `BaseAgent` with: + +- **`accepts_tools = True`** -- Enables CLI/SDK tool introspection +- **`ToolExecutor`** -- Initialized from the provided tool list, handles dispatch with JSON argument parsing, latency tracking, and event bus integration +- **`max_turns`** -- Configurable loop iteration limit (default: 10) + +```python +class ToolUsingAgent(BaseAgent): + accepts_tools: bool = True + + def __init__( + self, + engine: InferenceEngine, + model: str, + *, + tools: Optional[List[BaseTool]] = None, + bus: Optional[EventBus] = None, + max_turns: int = 10, + temperature: float = 0.7, + max_tokens: int = 1024, + ) -> None: ... +``` + +All tool-using agents (`OrchestratorAgent`, `NativeReActAgent`, `NativeOpenHandsAgent`, `RLMAgent`) extend this class. + +--- + ## Agent Implementations ### SimpleAgent **Registry key:** `simple` -The simplest agent implementation -- a single-turn, no-tool query-to-response pipeline. +The simplest agent implementation -- a single-turn, no-tool query-to-response pipeline. Extends `BaseAgent` directly (does not accept tools). ```mermaid graph LR @@ -69,10 +126,10 @@ graph LR How it works: -1. Publishes `AGENT_TURN_START` on the event bus -2. Builds a message list from any existing conversation context plus the user's input -3. Calls `instrumented_generate()` (if bus is available) or `engine.generate()` directly -4. Publishes `AGENT_TURN_END` and returns an `AgentResult` with `turns=1` +1. Calls `_emit_turn_start()` to publish `AGENT_TURN_START` on the event bus +2. Calls `_build_messages()` to assemble the message list from conversation context plus user input +3. Calls `_generate()` to invoke the engine with stored defaults +4. Calls `_emit_turn_end()` and returns an `AgentResult` with `turns=1` ```python from openjarvis.agents.simple import SimpleAgent @@ -86,7 +143,12 @@ print(result.content) # "The capital of France is Paris." **Registry key:** `orchestrator` -A multi-turn agent that implements a **tool-calling loop**. The LLM can request tool invocations, and the results are fed back for further processing until the model produces a final text response. +A multi-turn agent that implements a **tool-calling loop**. Extends `ToolUsingAgent`. The LLM can request tool invocations, and the results are fed back for further processing until the model produces a final text response. + +Supports two modes: + +- **`function_calling`** (default) -- Uses OpenAI function-calling format via `ToolExecutor.get_openai_tools()` +- **`structured`** -- Uses structured output format for models that support it ```mermaid graph TD @@ -127,6 +189,161 @@ result = agent.run("What is 2^10 + 3^5?") # The agent may call the calculator tool, get "1267", then respond ``` +### NativeReActAgent + +**Registry key:** `native_react` (alias: `react`) + +A ReAct (Reasoning + Acting) agent that implements a **Thought-Action-Observation** loop. Extends `ToolUsingAgent`. The LLM is prompted to output structured text with `Thought:`, `Action:`, `Action Input:`, and `Final Answer:` fields, which the agent parses to drive tool execution. + +```mermaid +graph TD + Q["User Query"] --> SYS["Build system prompt
with tool names"] + SYS --> GEN["Generate response"] + GEN --> PARSE["Parse ReAct output"] + PARSE --> FINAL{"Final Answer?"} + FINAL -->|Yes| DONE["Return answer"] + FINAL -->|No| ACTION{"Has Action?"} + ACTION -->|No| DONE2["Return content as-is"] + ACTION -->|Yes| EXEC["Execute tool
via ToolExecutor"] + EXEC --> OBS["Append Observation"] + OBS --> MAXCHECK{"Max turns
exceeded?"} + MAXCHECK -->|No| GEN + MAXCHECK -->|Yes| TIMEOUT["Return max_turns_result"] +``` + +How it works: + +1. Builds a system prompt listing available tool names +2. Generates a response and parses the ReAct-structured output +3. If a `Final Answer:` is found, returns it +4. If an `Action:` is found, executes the tool and feeds the result back as an `Observation:` +5. Loops until a final answer is produced or `max_turns` is exceeded + +!!! note "Backward compatibility" + The old `from openjarvis.agents.react import ReActAgent` import path still works via a backward-compat shim. The registry alias `"react"` also maps to `NativeReActAgent`. + +```python +from openjarvis.agents.native_react import NativeReActAgent + +agent = NativeReActAgent( + engine, + model="qwen3:8b", + tools=[CalculatorTool(), ThinkTool()], + max_turns=10, +) +result = agent.run("What is the square root of 256?") +``` + +### NativeOpenHandsAgent + +**Registry key:** `native_openhands` + +A CodeAct-style agent that generates and executes Python code. Extends `ToolUsingAgent`. It can also invoke tools via structured `Action:` / `Action Input:` output. URLs in the input are automatically pre-fetched and inlined for the LLM. + +How it works: + +1. Builds a detailed system prompt with tool descriptions and code execution instructions +2. Pre-fetches any URLs in the user input, inlining the content directly +3. For each turn: + - Generates a response and strips `` tags + - If a `\`\`\`python` code block is found, executes it via `code_interpreter` + - If an `Action:` / `Action Input:` is found, dispatches the tool + - If neither is found, returns the content as the final answer +4. Handles context window overflow with automatic truncation + +```python +from openjarvis.agents.native_openhands import NativeOpenHandsAgent + +agent = NativeOpenHandsAgent( + engine, + model="qwen3:8b", + tools=[CalculatorTool(), WebSearchTool()], + max_turns=3, + max_tokens=2048, +) +result = agent.run("Summarize https://example.com/article") +``` + +### RLMAgent + +**Registry key:** `rlm` + +A Recursive Language Model agent based on the [RLM paper](https://arxiv.org/abs/2512.24601). Instead of passing long context directly in the LLM prompt, RLM stores context as a Python variable in a persistent REPL. A "Root LM" writes Python code to inspect, decompose, and process context using recursive sub-LM calls via `llm_query()` and `llm_batch()`. Extends `ToolUsingAgent`. + +```mermaid +graph TD + Q["User Query +
Context"] --> REPL["Create persistent REPL
(context stored as variable)"] + REPL --> GEN["Generate code"] + GEN --> CODE{"Code block
found?"} + CODE -->|No| DONE["Return content
as final answer"] + CODE -->|Yes| EXEC["Execute in REPL"] + EXEC --> TERM{"FINAL() called?"} + TERM -->|Yes| RESULT["Return final answer"] + TERM -->|No| FEED["Feed output back
as user message"] + FEED --> MAXCHECK{"Max turns
exceeded?"} + MAXCHECK -->|No| GEN + MAXCHECK -->|Yes| TIMEOUT["Return max_turns_result"] +``` + +How it works: + +1. Creates a persistent REPL with `llm_query()` and `llm_batch()` callbacks +2. Injects context from `AgentContext` metadata or memory results into the REPL as a variable +3. Generates code and executes it in the REPL +4. If `FINAL(value)` or `FINAL_VAR("name")` is called, returns the final answer +5. If no code block is found, treats the content as a direct answer + +The agent supports configurable sub-model parameters for recursive calls: + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `sub_model` | same as `model` | Model for sub-LM calls | +| `sub_temperature` | `0.3` | Temperature for sub-LM calls | +| `sub_max_tokens` | `1024` | Max tokens for sub-LM calls | +| `max_output_chars` | `10000` | Max REPL output characters | +| `system_prompt` | `RLM_SYSTEM_PROMPT` | Override the system prompt | + +```python +from openjarvis.agents.rlm import RLMAgent + +agent = RLMAgent( + engine, + model="qwen3:8b", + max_turns=10, + sub_model="qwen3:1.7b", # smaller model for sub-queries + sub_temperature=0.3, +) +result = agent.run("Summarize this document", context=ctx) +``` + +### OpenHandsAgent (SDK) + +**Registry key:** `openhands` + +A thin wrapper around the real `openhands-sdk` package for AI-driven software development tasks. Extends `BaseAgent` directly (does not use `ToolUsingAgent` since tool management is handled by the SDK). + +!!! warning "Optional dependency" + This agent requires the `openhands-sdk` package (`pip install openjarvis[openhands]`). The SDK requires Python 3.12+. + +How it works: + +1. Imports `openhands.sdk` at runtime (lazy import) +2. Creates an LLM, Agent, and Conversation from the SDK +3. Sends the user input as a message and runs the conversation +4. Extracts the final message content from the conversation + +```python +from openjarvis.agents.openhands import OpenHandsAgent + +agent = OpenHandsAgent( + engine, + model="gpt-4", + workspace="/path/to/project", + api_key="sk-...", +) +result = agent.run("Fix the failing test in test_utils.py") +``` + ### OpenClawAgent **Registry key:** `openclaw` @@ -162,30 +379,11 @@ agent = OpenClawAgent(engine, model="qwen3:8b", mode="http") agent = OpenClawAgent(engine, model="qwen3:8b", mode="subprocess") ``` -### CustomAgent - -**Registry key:** `custom` - -A template for user-defined agents. Its `run()` method raises `NotImplementedError` -- users must subclass it and override `run()`: - -```python -from openjarvis.agents.custom import CustomAgent -from openjarvis.core.registry import AgentRegistry - -@AgentRegistry.register("my-agent") -class MyAgent(CustomAgent): - agent_id = "my-agent" - - def run(self, input, context=None, **kwargs): - # Custom logic here - return AgentResult(content="Custom response") -``` - --- ## Tool System Integration -The `OrchestratorAgent` uses the `ToolExecutor` to dispatch tool calls. The tool system is built on the `BaseTool` ABC: +All `ToolUsingAgent` subclasses use the `ToolExecutor` to dispatch tool calls. The tool system is built on the `BaseTool` ABC: ```python class BaseTool(ABC): @@ -312,12 +510,13 @@ All agents integrate with the `EventBus` for telemetry and trace collection: | Event | Published By | When | |-------|-------------|------| -| `AGENT_TURN_START` | All agents | Before starting query processing | -| `AGENT_TURN_END` | All agents | After producing a response | -| `INFERENCE_START` | OrchestratorAgent | Before each `engine.generate()` call | -| `INFERENCE_END` | OrchestratorAgent | After each `engine.generate()` call | -| `TOOL_CALL_START` | ToolExecutor / OpenClawAgent | Before executing a tool | -| `TOOL_CALL_END` | ToolExecutor / OpenClawAgent | After executing a tool | +| `AGENT_TURN_START` | All agents (via `_emit_turn_start` helper) | Before starting query processing | +| `AGENT_TURN_END` | All agents (via `_emit_turn_end` helper) | After producing a response | +| `TOOL_CALL_START` | ToolExecutor (all `ToolUsingAgent` subclasses) | Before executing a tool | +| `TOOL_CALL_END` | ToolExecutor (all `ToolUsingAgent` subclasses) | After executing a tool | + +!!! info "Inference events" + `INFERENCE_START` and `INFERENCE_END` events are published by the `InstrumentedEngine` wrapper (in `telemetry/instrumented_engine.py`), not by agents directly. This keeps telemetry opt-in and transparent to agent code. These events are consumed by the `TelemetryStore` (for metrics) and `TraceCollector` (for interaction traces). @@ -345,7 +544,7 @@ To list all registered agents: from openjarvis.core.registry import AgentRegistry print(AgentRegistry.keys()) -# ("simple", "orchestrator", "openclaw", "custom") +# ("simple", "orchestrator", "native_react", "react", "native_openhands", "rlm", "openhands") ``` To instantiate an agent by key: diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index b838abbd..f63390bf 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -62,7 +62,7 @@ Each engine is configured via its own sub-section in `config.toml` (e.g., `[engi ### Agentic Logic -The Agentic Logic pillar implements **pluggable agents** that handle queries with varying levels of sophistication. `SimpleAgent` provides single-turn query-to-response without tools. `OrchestratorAgent` implements a multi-turn tool-calling loop where the LLM can invoke tools like `calculator`, `think`, `retrieval`, `llm`, and `file_read`, with results fed back for further processing. `OpenClawAgent` communicates with external OpenClaw servers via HTTP or subprocess transport. +The Agentic Logic pillar implements **pluggable agents** that handle queries with varying levels of sophistication. The agent hierarchy is organized around `BaseAgent` (ABC with concrete helpers) and `ToolUsingAgent` (intermediate base for agents that accept tools, with `accepts_tools = True`). Seven agent types are available: `SimpleAgent` (single-turn, no tools), `OrchestratorAgent` (multi-turn tool-calling loop with function_calling and structured modes), `NativeReActAgent` (Thought-Action-Observation loop), `NativeOpenHandsAgent` (CodeAct-style code execution), `RLMAgent` (recursive LM with persistent REPL), `OpenHandsAgent` (wraps real `openhands-sdk`), and `OpenClawAgent` (external agent via HTTP or subprocess transport). Agent behavior is configured through `[agent]` in `config.toml`, including the default agent, turn limits, tool list, optional system prompt, and the `context_from_memory` flag (previously `context_injection`) that controls automatic memory context injection. All agents implement the `BaseAgent` ABC with a `run()` method, and are registered via `@AgentRegistry.register("name")`. @@ -152,11 +152,15 @@ src/openjarvis/ cloud.py Cloud backend (OpenAI, Anthropic, Google SDKs) agents/ Agentic Logic pillar -- pluggable agents - _stubs.py BaseAgent ABC, AgentContext, AgentResult + _stubs.py BaseAgent ABC, ToolUsingAgent, AgentContext, AgentResult simple.py SimpleAgent (single-turn, no tools) - orchestrator.py OrchestratorAgent (multi-turn tool loop) + orchestrator.py OrchestratorAgent (multi-turn tool loop, function_calling + structured) + native_react.py NativeReActAgent (Thought-Action-Observation loop) + native_openhands.py NativeOpenHandsAgent (CodeAct-style code execution) + rlm.py RLMAgent (recursive LM with persistent REPL) + openhands.py OpenHandsAgent (wraps real openhands-sdk) + react.py Backward-compat shim (re-exports NativeReActAgent as ReActAgent) openclaw.py OpenClawAgent (HTTP/subprocess transport) - custom.py CustomAgent (template for user-defined agents) openclaw_protocol.py Wire protocol (MessageType, serialize/deserialize) openclaw_transport.py Transport ABC, HttpTransport, SubprocessTransport openclaw_plugin.py ProviderPlugin, MemorySearchManager diff --git a/docs/development/contributing.md b/docs/development/contributing.md index 0bf3e72f..107d6673 100644 --- a/docs/development/contributing.md +++ b/docs/development/contributing.md @@ -199,12 +199,14 @@ src/openjarvis/ cloud.py # CloudEngine (OpenAI/Anthropic/Google) agents/ # Agent implementations - _stubs.py # BaseAgent ABC, AgentContext, AgentResult + _stubs.py # BaseAgent ABC, ToolUsingAgent, AgentContext, AgentResult simple.py # SimpleAgent — single-turn, no tools - orchestrator.py # OrchestratorAgent — multi-turn tool calling - custom.py # CustomAgent — user template - react.py # ReActAgent - openhands.py # OpenHands agent + orchestrator.py # OrchestratorAgent — multi-turn tool calling (function_calling + structured) + native_react.py # NativeReActAgent — Thought-Action-Observation loop + native_openhands.py # NativeOpenHandsAgent — CodeAct-style code execution + rlm.py # RLMAgent — recursive LM with persistent REPL + openhands.py # OpenHandsAgent — wraps real openhands-sdk + react.py # Backward-compat shim (re-exports NativeReActAgent) openclaw.py # OpenClawAgent — HTTP/subprocess transport openclaw_protocol.py # OpenClaw message protocol openclaw_transport.py # OpenClaw transports (HTTP, subprocess) diff --git a/docs/development/extending.md b/docs/development/extending.md index a613a6f8..6a991ae2 100644 --- a/docs/development/extending.md +++ b/docs/development/extending.md @@ -326,24 +326,25 @@ class RetrievalResult: ## Adding a New Agent Agents implement the logic for handling queries, calling tools, and managing -multi-turn interactions. All agents implement the `BaseAgent` ABC from -`agents/_stubs.py`. +multi-turn interactions. There are two paths depending on whether your agent +uses tools: -### Complete Example +- **Path A: Non-tool agent** -- Extend `BaseAgent` directly +- **Path B: Tool-using agent** -- Extend `ToolUsingAgent` (which sets `accepts_tools = True` and provides a `ToolExecutor`) + +### Path A: Non-tool Agent (extending BaseAgent) Create `src/openjarvis/agents/my_agent.py`: ```python -"""Custom agent implementation.""" +"""Custom agent implementation — single-turn, no tools.""" from __future__ import annotations from typing import Any, Optional from openjarvis.agents._stubs import AgentContext, AgentResult, BaseAgent -from openjarvis.core.events import EventBus, EventType from openjarvis.core.registry import AgentRegistry -from openjarvis.core.types import Message, Role from openjarvis.engine._stubs import InferenceEngine @@ -353,21 +354,6 @@ class MyAgent(BaseAgent): agent_id = "my_agent" - def __init__( - self, - engine: InferenceEngine, - model: str, - *, - bus: Optional[EventBus] = None, - temperature: float = 0.7, - max_tokens: int = 1024, - ) -> None: - self._engine = engine - self._model = model - self._bus = bus - self._temperature = temperature - self._max_tokens = max_tokens - def run( self, input: str, @@ -375,49 +361,127 @@ class MyAgent(BaseAgent): **kwargs: Any, ) -> AgentResult: """Execute the agent on input and return an AgentResult.""" - # Emit turn start event - if self._bus: - self._bus.publish(EventType.AGENT_TURN_START, { - "agent": self.agent_id, - "input": input, - }) + # Use BaseAgent helpers instead of manual event bus code + self._emit_turn_start(input) - # Build messages from context + user input - messages: list[Message] = [] - - # Add a system prompt for your agent's personality - messages.append(Message( - role=Role.SYSTEM, - content="You are a helpful assistant with specialized knowledge.", - )) - - # Include any prior conversation from context - if context and context.conversation.messages: - messages.extend(context.conversation.messages) - - messages.append(Message(role=Role.USER, content=input)) - - # Just call engine.generate() directly -- telemetry is opt-in - # via InstrumentedEngine (see telemetry/instrumented_engine.py) - result = self._engine.generate( - messages, - model=self._model, - temperature=self._temperature, - max_tokens=self._max_tokens, + # Build messages from context + user input (with optional system prompt) + messages = self._build_messages( + input, context, + system_prompt="You are a helpful assistant with specialized knowledge.", ) - content = result.get("content", "") - - # Emit turn end event - if self._bus: - self._bus.publish(EventType.AGENT_TURN_END, { - "agent": self.agent_id, - "content_length": len(content), - }) + # Call engine.generate() with stored defaults (model, temperature, max_tokens) + result = self._generate(messages) + content = self._strip_think_tags(result.get("content", "")) + self._emit_turn_end(turns=1) return AgentResult(content=content, turns=1) ``` +!!! tip "BaseAgent helpers" + `BaseAgent` provides these concrete helpers so you don't need to manually + manage the event bus or engine calls: + + | Helper | Purpose | + |--------|---------| + | `_emit_turn_start(input)` | Publish `AGENT_TURN_START` | + | `_emit_turn_end(**data)` | Publish `AGENT_TURN_END` | + | `_build_messages(input, context, *, system_prompt)` | Assemble message list | + | `_generate(messages, **kwargs)` | Call engine with stored defaults | + | `_strip_think_tags(text)` | Remove `` blocks | + | `_max_turns_result(tool_results, turns, content)` | Standard max-turns result | + +### Path B: Tool-using Agent (extending ToolUsingAgent) + +Create `src/openjarvis/agents/my_tool_agent.py`: + +```python +"""Custom tool-using agent with a multi-turn loop.""" + +from __future__ import annotations + +from typing import Any, List, Optional + +from openjarvis.agents._stubs import AgentContext, AgentResult, ToolUsingAgent +from openjarvis.core.events import EventBus +from openjarvis.core.registry import AgentRegistry +from openjarvis.core.types import ToolCall, ToolResult +from openjarvis.engine._stubs import InferenceEngine +from openjarvis.tools._stubs import BaseTool + + +@AgentRegistry.register("my_tool_agent") +class MyToolAgent(ToolUsingAgent): + """Custom agent with tool-calling loop.""" + + agent_id = "my_tool_agent" + + def __init__( + self, + engine: InferenceEngine, + model: str, + *, + tools: Optional[List[BaseTool]] = None, + bus: Optional[EventBus] = None, + max_turns: int = 10, + temperature: float = 0.7, + max_tokens: int = 1024, + ) -> None: + super().__init__( + engine, model, tools=tools, bus=bus, + max_turns=max_turns, temperature=temperature, + max_tokens=max_tokens, + ) + + def run( + self, + input: str, + context: Optional[AgentContext] = None, + **kwargs: Any, + ) -> AgentResult: + self._emit_turn_start(input) + + messages = self._build_messages(input, context) + tools_spec = self._executor.get_openai_tools() + all_tool_results: list[ToolResult] = [] + turns = 0 + + for _ in range(self._max_turns): + turns += 1 + result = self._generate(messages, tools=tools_spec) + content = result.get("content", "") + tool_calls = result.get("tool_calls", []) + + if not tool_calls: + self._emit_turn_end(turns=turns) + return AgentResult( + content=content, + tool_results=all_tool_results, + turns=turns, + ) + + # Execute each tool call + for tc in tool_calls: + call = ToolCall( + id=tc.get("id", f"call_{turns}"), + name=tc["name"], + arguments=tc["arguments"], + ) + tr = self._executor.execute(call) + all_tool_results.append(tr) + + # Max turns exceeded — use the standard helper + return self._max_turns_result(all_tool_results, turns) +``` + +!!! info "What ToolUsingAgent adds" + `ToolUsingAgent` extends `BaseAgent` with: + + - **`accepts_tools = True`** — enables `--tools` in CLI and `tools=` in SDK + - **`self._executor`** — a `ToolExecutor` initialized from the provided tools + - **`self._tools`** — the raw list of `BaseTool` instances + - **`self._max_turns`** — configurable loop iteration limit (default: 10) + ### Register in `__init__.py` Add to `src/openjarvis/agents/__init__.py`: diff --git a/docs/development/roadmap.md b/docs/development/roadmap.md index d98a7d4b..68275be1 100644 --- a/docs/development/roadmap.md +++ b/docs/development/roadmap.md @@ -12,21 +12,22 @@ a major pillar or cross-cutting capability to the framework. | **v0.1** | Phase 0 -- Scaffolding | :material-check-circle:{ .green } Complete | Project scaffolding, registry system (`RegistryBase[T]`), core types (`Message`, `ModelSpec`, `Conversation`, `ToolResult`), configuration loader with hardware detection, Click CLI skeleton | | **v0.2** | Phase 1 -- Intelligence + Inference | :material-check-circle:{ .green } Complete | Intelligence pillar (model catalog, heuristic router), inference engines (Ollama, vLLM, llama.cpp), engine discovery and health probing, `jarvis ask` command working end-to-end | | **v0.3** | Phase 2 -- Memory | :material-check-circle:{ .green } Complete | Memory backends (SQLite/FTS5, FAISS, ColBERTv2, BM25, Hybrid/RRF), document chunking and ingestion pipeline, context injection with source attribution, `jarvis memory` commands | -| **v0.4** | Phase 3 -- Agents + Tools + Server | :material-check-circle:{ .green } Complete | Agent system (SimpleAgent, OrchestratorAgent, OpenClawAgent, CustomAgent), tool system (Calculator, Think, Retrieval, LLM, FileRead), ToolExecutor dispatch engine, OpenAI-compatible API server (`jarvis serve`) | +| **v0.4** | Phase 3 -- Agents + Tools + Server | :material-check-circle:{ .green } Complete | Agent system (SimpleAgent, OrchestratorAgent, OpenClawAgent), tool system (Calculator, Think, Retrieval, LLM, FileRead), ToolExecutor dispatch engine, OpenAI-compatible API server (`jarvis serve`) | | **v0.5** | Phase 4 -- Learning + Telemetry | :material-check-circle:{ .green } Complete | Learning system (HeuristicRouter policy, TraceDrivenPolicy, GRPO stub), reward functions, telemetry aggregation (per-model/engine stats, export), `--router` CLI flag, `jarvis telemetry` commands | | **v1.0** | Phase 5 -- SDK + Production | :material-check-circle:{ .green } Complete | Python SDK (`Jarvis` class, `MemoryHandle`), OpenClaw agent infrastructure (protocol, transports, plugins), benchmarking framework (latency, throughput), Docker deployment (CPU + GPU), MkDocs documentation site | -| **v1.1** | Phase 6 -- Traces + Learning | :material-progress-clock:{ .amber } In Progress | Trace system (`TraceStore`, `TraceCollector`, `TraceAnalyzer`), trace-driven learning, pluggable agent architectures (ReAct, OpenHands), MCP integration layer | +| **v1.1** | Phase 6 -- Traces + Learning | :material-check-circle:{ .green } Complete | Trace system (`TraceStore`, `TraceCollector`, `TraceAnalyzer`), trace-driven learning, MCP integration layer | +| **v1.5** | Phase 10 -- Agent Restructuring | :material-check-circle:{ .green } Complete | BaseAgent helpers, ToolUsingAgent intermediate base, NativeReActAgent, NativeOpenHandsAgent, RLMAgent, OpenHandsAgent (SDK), `accepts_tools` introspection, backward-compat shims, CustomAgent removed | --- ## Current Status -OpenJarvis v1.0 is complete. The framework provides: +OpenJarvis v1.5 (Phase 10) is complete. The framework provides: - **Four core abstractions** -- Intelligence, Engine, Agentic Logic, Memory -- each with an ABC interface and registry-based discovery - **Five inference engines** -- Ollama, vLLM, llama.cpp, SGLang, Cloud (OpenAI/Anthropic/Google) - **Five memory backends** -- SQLite/FTS5, FAISS, ColBERTv2, BM25, Hybrid (RRF fusion) -- **Multiple agent types** -- Simple, Orchestrator, Custom, OpenClaw, ReAct, OpenHands +- **Seven agent types** -- Simple, Orchestrator, NativeReAct, NativeOpenHands, RLM, OpenHands (SDK), OpenClaw - **Seven built-in tools** -- Calculator, Think, Retrieval, LLM, FileRead, WebSearch, CodeInterpreter - **Python SDK** -- `Jarvis` class for programmatic use - **OpenAI-compatible API server** -- `POST /v1/chat/completions`, `GET /v1/models` @@ -34,43 +35,43 @@ OpenJarvis v1.0 is complete. The framework provides: - **Telemetry and traces** -- SQLite-backed recording and aggregation - **Docker deployment** -- CPU and GPU images with docker-compose -Phase 6 is actively in progress, adding the trace system and trace-driven -learning capabilities. +Phase 10 (Agent Restructuring) is complete. The agent hierarchy has been +refactored with `BaseAgent` helpers, `ToolUsingAgent` intermediate base, and +four new agent types (NativeReActAgent, NativeOpenHandsAgent, RLMAgent, +OpenHandsAgent SDK). --- -## Phase 6 Details +## Phase 10 Details -Phase 6 focuses on closing the loop between execution and learning: +Phase 10 refactored the agent hierarchy for composability and extensibility: -### Trace System +### BaseAgent Helpers -- **TraceStore** -- Persists complete `Trace` objects to SQLite, capturing the - full sequence of steps (route, retrieve, generate, tool_call, respond) with - timing, inputs, outputs, and outcomes -- **TraceCollector** -- Wraps any `BaseAgent` to automatically record traces - during execution via EventBus subscription -- **TraceAnalyzer** -- Read-only query layer providing aggregated statistics - (per-route, per-tool, by query type, time-range filtering) +- **`_emit_turn_start` / `_emit_turn_end`** -- Event bus integration without boilerplate +- **`_build_messages`** -- System prompt + context + input assembly +- **`_generate`** -- Engine call with stored defaults +- **`_max_turns_result`** -- Standard max-turns-exceeded result +- **`_strip_think_tags`** -- Remove `` blocks from model output -### Trace-Driven Learning +### ToolUsingAgent Intermediate Base -- **TraceDrivenPolicy** -- A router policy that learns from historical trace - outcomes to improve model selection over time -- Query classification groups traces by type (code, math, short, long, general) -- Per-model scoring combines success rate and user feedback -- Online updates via `observe()` for incremental learning +- Sets `accepts_tools = True` for CLI/SDK introspection +- Initializes `ToolExecutor` from provided tools +- Configurable `max_turns` loop limit -### Pluggable Agents +### New Agent Types -- **ReActAgent** -- Reasoning + Acting pattern for systematic tool use -- **OpenHands** -- Integration with the OpenHands agent framework +- **NativeReActAgent** (`native_react`, alias `react`) -- Thought-Action-Observation loop +- **NativeOpenHandsAgent** (`native_openhands`) -- CodeAct-style code execution with URL pre-fetching +- **RLMAgent** (`rlm`) -- Recursive LM with persistent REPL and sub-LM calls +- **OpenHandsAgent** (`openhands`) -- Thin wrapper for real `openhands-sdk` --- ## Future Directions -Beyond Phase 6, areas of ongoing exploration include: +Beyond Phase 10, areas of ongoing exploration include: - **GRPO training** -- Reinforcement learning from trace data to train the routing policy, moving beyond heuristics and simple statistics diff --git a/docs/user-guide/agents.md b/docs/user-guide/agents.md index f6f1355d..f11aeb82 100644 --- a/docs/user-guide/agents.md +++ b/docs/user-guide/agents.md @@ -1,15 +1,18 @@ # Agents -Agents are the agentic logic layer of OpenJarvis. They determine how a query is processed -- whether it goes directly to a model, through a tool-calling loop, or via an external agent runtime. All agents implement the `BaseAgent` ABC and are registered via the `AgentRegistry`. +Agents are the agentic logic layer of OpenJarvis. They determine how a query is processed -- whether it goes directly to a model, through a tool-calling loop, via ReAct reasoning, CodeAct code execution, recursive decomposition, or an external agent runtime. All agents implement the `BaseAgent` ABC and are registered via the `AgentRegistry`. ## Overview -| Agent | Registry Key | Tools | Multi-turn | Description | -|------------------|-----------------|-------|------------|----------------------------------------------| -| `SimpleAgent` | `simple` | No | No | Single-turn query-to-response | -| `OrchestratorAgent` | `orchestrator` | Yes | Yes | Multi-turn tool-calling loop | -| `OpenClawAgent` | `openclaw` | Yes | Yes | External agent via HTTP or subprocess | -| `CustomAgent` | `custom` | -- | -- | Template for user-defined agents | +| Agent | Registry Key | `accepts_tools` | Multi-turn | Description | +|---------------------|-------------------|-----------------|------------|----------------------------------------------| +| `SimpleAgent` | `simple` | No | No | Single-turn query-to-response | +| `OrchestratorAgent` | `orchestrator` | Yes | Yes | Multi-turn tool-calling loop (function_calling + structured) | +| `NativeReActAgent` | `native_react` | Yes | Yes | Thought-Action-Observation loop | +| `NativeOpenHandsAgent` | `native_openhands` | Yes | Yes | CodeAct-style code execution + tool calls | +| `RLMAgent` | `rlm` | Yes | Yes | Recursive LM with persistent REPL | +| `OpenHandsAgent` | `openhands` | No | Yes | Wraps real openhands-sdk | +| `OpenClawAgent` | `openclaw` | Yes | Yes | External agent via HTTP or subprocess | --- @@ -23,6 +26,17 @@ from openjarvis.agents._stubs import AgentContext, AgentResult class BaseAgent(ABC): agent_id: str + accepts_tools: bool = False + + def __init__( + self, + engine: InferenceEngine, + model: str, + *, + bus: Optional[EventBus] = None, + temperature: float = 0.7, + max_tokens: int = 1024, + ) -> None: ... @abstractmethod def run( @@ -34,6 +48,12 @@ class BaseAgent(ABC): """Execute the agent on the given input.""" ``` +The `accepts_tools` class attribute controls whether an agent can receive tools via `--tools` on the CLI or `tools=` in the SDK. Agents with `accepts_tools = False` ignore tool arguments. + +`BaseAgent` also provides concrete helper methods (`_emit_turn_start`, `_emit_turn_end`, `_build_messages`, `_generate`, `_max_turns_result`, `_strip_think_tags`) that subclasses use to avoid duplicating common logic. See the [architecture docs](../architecture/agents.md#baseagent-abc) for details. + +**ToolUsingAgent** is an intermediate base class (extends `BaseAgent`) that sets `accepts_tools = True` and adds a `ToolExecutor` and `max_turns` loop limit. All tool-using agents extend this class. + ### AgentContext The runtime context handed to an agent on each invocation. @@ -65,7 +85,7 @@ The `SimpleAgent` is a single-turn agent that sends the query directly to the in **How it works:** 1. Builds a message list from the conversation context (if provided) plus the user query. -2. Calls the inference engine via `instrumented_generate()` for telemetry tracking. +2. Calls the inference engine via `_generate()`. 3. Returns the response as an `AgentResult` with `turns=1`. **Constructor parameters:** @@ -84,7 +104,7 @@ The `SimpleAgent` is a single-turn agent that sends the query directly to the in ## OrchestratorAgent -The `OrchestratorAgent` is a multi-turn agent that implements a tool-calling loop. It is the primary agent for queries that require computation, knowledge retrieval, or structured reasoning. +The `OrchestratorAgent` is a multi-turn agent that implements a tool-calling loop. It is the primary agent for queries that require computation, knowledge retrieval, or structured reasoning. Extends `ToolUsingAgent`. **How it works:** @@ -97,20 +117,144 @@ The `OrchestratorAgent` is a multi-turn agent that implements a tool-calling loo **Constructor parameters:** +| Parameter | Type | Default | Description | +|-----------------|-------------------|---------|--------------------------------------| +| `engine` | `InferenceEngine` | -- | The inference engine to use | +| `model` | `str` | -- | Model identifier | +| `tools` | `list[BaseTool]` | `[]` | Tool instances to make available | +| `bus` | `EventBus` | `None` | Event bus for telemetry | +| `max_turns` | `int` | `10` | Maximum number of tool-calling turns | +| `temperature` | `float` | `0.7` | Sampling temperature | +| `max_tokens` | `int` | `1024` | Maximum tokens to generate | +| `mode` | `str` | `"function_calling"` | Tool-calling mode (`function_calling` or `structured`) | +| `system_prompt` | `str` | `None` | Custom system prompt | + +**When to use:** For queries that need calculation, memory search, sub-model calls, file reading, or multi-step reasoning. + +!!! info "Tool-Calling Loop" + The orchestrator follows the OpenAI function-calling convention. The engine must support returning `tool_calls` in its response for the loop to engage. If tools are provided but the engine does not return any tool calls, the agent behaves like a single-turn agent. + +--- + +## NativeReActAgent + +The `NativeReActAgent` implements a **Thought-Action-Observation** loop following the ReAct pattern. It prompts the LLM to produce structured output (`Thought:`, `Action:`, `Action Input:`, `Final Answer:`) and parses the response to drive tool execution. Extends `ToolUsingAgent`. + +**How it works:** + +1. Builds a system prompt listing available tool names. +2. Generates a response and parses the ReAct-structured output. +3. If a `Final Answer:` is found, returns it. +4. If an `Action:` is found, executes the tool and feeds the result back as an `Observation:`. +5. Loops until a final answer is produced or `max_turns` is exceeded. + +**Constructor parameters:** + | Parameter | Type | Default | Description | |---------------|-------------------|---------|------------------------------------| | `engine` | `InferenceEngine` | -- | The inference engine to use | | `model` | `str` | -- | Model identifier | | `tools` | `list[BaseTool]` | `[]` | Tool instances to make available | | `bus` | `EventBus` | `None` | Event bus for telemetry | -| `max_turns` | `int` | `10` | Maximum number of tool-calling turns | +| `max_turns` | `int` | `10` | Maximum number of reasoning turns | | `temperature` | `float` | `0.7` | Sampling temperature | | `max_tokens` | `int` | `1024` | Maximum tokens to generate | -**When to use:** For queries that need calculation, memory search, sub-model calls, file reading, or multi-step reasoning. +**When to use:** For queries that benefit from explicit step-by-step reasoning with tool use, where you want visibility into the agent's thought process. -!!! info "Tool-Calling Loop" - The orchestrator follows the OpenAI function-calling convention. The engine must support returning `tool_calls` in its response for the loop to engage. If tools are provided but the engine does not return any tool calls, the agent behaves like a single-turn agent. +!!! note "Backward compatibility" + The registry alias `"react"` maps to `NativeReActAgent`. The old import `from openjarvis.agents.react import ReActAgent` also still works. + +--- + +## NativeOpenHandsAgent + +The `NativeOpenHandsAgent` is a CodeAct-style agent that generates and executes Python code alongside structured tool calls. It can also pre-fetch URL content from user input to provide direct context to the LLM. Extends `ToolUsingAgent`. + +**How it works:** + +1. Builds a detailed system prompt with tool descriptions and code execution instructions. +2. Pre-fetches any URLs in the user input, inlining the content directly. +3. For each turn, generates a response and attempts to extract code blocks or tool calls. +4. Code is executed via `code_interpreter`; tool calls are dispatched via `ToolExecutor`. +5. If neither is found, returns the content as the final answer. + +**Constructor parameters:** + +| Parameter | Type | Default | Description | +|---------------|-------------------|---------|------------------------------------| +| `engine` | `InferenceEngine` | -- | The inference engine to use | +| `model` | `str` | -- | Model identifier | +| `tools` | `list[BaseTool]` | `[]` | Tool instances to make available | +| `bus` | `EventBus` | `None` | Event bus for telemetry | +| `max_turns` | `int` | `3` | Maximum number of turns | +| `temperature` | `float` | `0.7` | Sampling temperature | +| `max_tokens` | `int` | `2048` | Maximum tokens to generate | + +**When to use:** For queries involving URL content, code execution, or tasks where the LLM can write and run Python to solve the problem. + +--- + +## RLMAgent + +The `RLMAgent` implements recursive decomposition via a persistent REPL, based on the RLM paper. Context is stored as a Python variable rather than injected into the prompt, enabling processing of arbitrarily long inputs through recursive sub-LM calls. Extends `ToolUsingAgent`. + +**How it works:** + +1. Creates a persistent REPL with `llm_query()` and `llm_batch()` callbacks. +2. Injects context from `AgentContext` into the REPL as a variable. +3. Generates code and executes it in the REPL. +4. If `FINAL(value)` is called, returns the value as the final answer. +5. If no code block is found, treats the content as a direct text answer. + +**Constructor parameters:** + +| Parameter | Type | Default | Description | +|--------------------|-------------------|--------------------|-------------------------------------| +| `engine` | `InferenceEngine` | -- | The inference engine to use | +| `model` | `str` | -- | Model identifier | +| `tools` | `list[BaseTool]` | `[]` | Tool instances (optional) | +| `bus` | `EventBus` | `None` | Event bus for telemetry | +| `max_turns` | `int` | `10` | Maximum number of code-execute turns | +| `temperature` | `float` | `0.7` | Sampling temperature | +| `max_tokens` | `int` | `2048` | Maximum tokens to generate | +| `sub_model` | `str` | same as `model` | Model for sub-LM calls | +| `sub_temperature` | `float` | `0.3` | Temperature for sub-LM calls | +| `sub_max_tokens` | `int` | `1024` | Max tokens for sub-LM calls | +| `max_output_chars` | `int` | `10000` | Max REPL output characters | +| `system_prompt` | `str` | `RLM_SYSTEM_PROMPT` | Override the system prompt | + +**When to use:** For long-context tasks that benefit from recursive decomposition, such as summarizing large documents, processing structured data, or tasks that require programmatic manipulation of context. + +--- + +## OpenHandsAgent (SDK) + +The `OpenHandsAgent` wraps the real `openhands-sdk` package for AI-driven software development. Extends `BaseAgent` directly (tool management is handled by the SDK internally). + +**How it works:** + +1. Imports `openhands.sdk` at runtime. +2. Creates an LLM, Agent, and Conversation from the SDK. +3. Sends the input and runs the conversation. +4. Returns the final message content. + +**Constructor parameters:** + +| Parameter | Type | Default | Description | +|---------------|-------------------|---------------|------------------------------------| +| `engine` | `InferenceEngine` | -- | The inference engine (fallback) | +| `model` | `str` | -- | Model identifier | +| `bus` | `EventBus` | `None` | Event bus for telemetry | +| `temperature` | `float` | `0.7` | Sampling temperature | +| `max_tokens` | `int` | `1024` | Maximum tokens to generate | +| `workspace` | `str` | `os.getcwd()` | Working directory for the agent | +| `api_key` | `str` | `$LLM_API_KEY`| API key for the LLM provider | + +**When to use:** For software development tasks (debugging, code editing, test fixing) where the OpenHands SDK provides a full development agent runtime. + +!!! warning "Optional dependency" + Requires `openhands-sdk` (`pip install openjarvis[openhands]`) and Python 3.12+. --- @@ -146,47 +290,6 @@ The `OpenClawAgent` wraps the OpenClaw Pi agent runtime, communicating via eithe --- -## CustomAgent - -The `CustomAgent` is a template for building user-defined agents. It raises `NotImplementedError` by default -- subclass it and override `run()` to implement your logic. - -```python -from openjarvis.agents._stubs import AgentContext, AgentResult, BaseAgent -from openjarvis.core.registry import AgentRegistry - - -@AgentRegistry.register("my-agent") -class MyAgent(BaseAgent): - agent_id = "my-agent" - - def __init__(self, engine, model, **kwargs): - self._engine = engine - self._model = model - - def run(self, input: str, context: AgentContext | None = None, **kwargs) -> AgentResult: - # Your custom logic here - result = self._engine.generate( - [{"role": "user", "content": input}], - model=self._model, - ) - return AgentResult( - content=result.get("content", ""), - turns=1, - ) -``` - -After registration, you can use your custom agent via the CLI or SDK: - -```bash -jarvis ask --agent my-agent "Hello" -``` - -```python -response = j.ask("Hello", agent="my-agent") -``` - ---- - ## Using Agents ### Via CLI @@ -198,6 +301,21 @@ jarvis ask --agent simple "What is the capital of France?" # Orchestrator with tools jarvis ask --agent orchestrator --tools calculator,think "What is sqrt(256)?" +# NativeReActAgent +jarvis ask --agent native_react --tools calculator "What is 2+2?" + +# ReAct alias (same as native_react) +jarvis ask --agent react --tools calculator,think "Solve step by step: 15% of 340" + +# NativeOpenHandsAgent +jarvis ask --agent native_openhands --tools calculator,web_search "Summarize example.com" + +# RLMAgent +jarvis ask --agent rlm "Summarize this long document" + +# OpenHands SDK agent +jarvis ask --agent openhands "Fix the bug in test_utils.py" + # OpenClaw agent jarvis ask --agent openclaw "Tell me a story" ``` @@ -219,6 +337,13 @@ response = j.ask( tools=["calculator"], ) +# NativeReActAgent with tools +response = j.ask( + "What is sqrt(256)?", + agent="native_react", + tools=["calculator", "think"], +) + # Full result with tool details result = j.ask_full( "What is the square root of 144?", @@ -248,7 +373,8 @@ AgentRegistry.contains("orchestrator") # True agent_cls = AgentRegistry.get("orchestrator") # List all registered agent keys -AgentRegistry.keys() # ["simple", "orchestrator", "openclaw", "custom"] +AgentRegistry.keys() +# ["simple", "orchestrator", "native_react", "react", "native_openhands", "rlm", "openhands"] ``` --- @@ -257,13 +383,14 @@ AgentRegistry.keys() # ["simple", "orchestrator", "openclaw", "custom"] All agents publish events on the `EventBus` when a bus is provided: -| Event | When | -|-------------------------|---------------------------------------------| -| `AGENT_TURN_START` | At the beginning of a run | -| `AGENT_TURN_END` | At the end of a run (includes turn count) | -| `INFERENCE_START` | Before each engine call (orchestrator) | -| `INFERENCE_END` | After each engine call (orchestrator) | -| `TOOL_CALL_START` | Before each tool execution (openclaw) | -| `TOOL_CALL_END` | After each tool execution (openclaw) | +| Event | When | +|-------------------------|-----------------------------------------------------| +| `AGENT_TURN_START` | At the beginning of a run (via `_emit_turn_start`) | +| `AGENT_TURN_END` | At the end of a run (via `_emit_turn_end`) | +| `TOOL_CALL_START` | Before each tool execution (`ToolUsingAgent` subclasses) | +| `TOOL_CALL_END` | After each tool execution (`ToolUsingAgent` subclasses) | + +!!! info "Inference events" + `INFERENCE_START` / `INFERENCE_END` events are published by the `InstrumentedEngine` wrapper, not by agents directly. This keeps telemetry opt-in and transparent to agent code. These events enable the telemetry and trace systems to record detailed interaction data automatically.