mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
docs(plans): phased plan for RLM language-based workflows (tinyagents rhai REPL) (#4510)
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# RLM — Language-Based Workflows (Rhai/`.ragsh`) Integration Plan
|
||||
|
||||
**Goal:** expose TinyAgents' Rhai-backed REPL language (the `.ragsh` / RLM /
|
||||
CodeAct surface, gated behind the `repl` cargo feature in
|
||||
`vendor/tinyagents/Cargo.toml`) as a **first-class `rlm` tool** in the
|
||||
OpenHuman Rust core, so the orchestrator agent can *write its own workflow
|
||||
scripts* — fan-out over subagents, batched tool/model calls, loops,
|
||||
conditionals — and execute them deterministically, similar to Claude Code
|
||||
Workflows and Recursive Language Models (RLMs).
|
||||
|
||||
## Why
|
||||
|
||||
Today the orchestrator composes work through fixed primitives:
|
||||
`spawn_subagent`, `spawn_parallel_agents`, `run_workflow` (WORKFLOW.md
|
||||
bundles), and typed tinyflows graphs. None of these let the model express
|
||||
*ad-hoc control flow* — "spawn N readers, dedupe their findings, verify each
|
||||
survivor with 3 refuters, loop until dry". The `.ragsh` session in TinyAgents
|
||||
is exactly that surface: a sandboxed, policy-bounded scripting engine whose
|
||||
only host access is capability functions (`model_query`, `tool_call`,
|
||||
`agent_query`, batched variants), with fail-closed limits on operations,
|
||||
wall-clock time, output bytes, call counts, and recursion depth.
|
||||
|
||||
## Architecture summary
|
||||
|
||||
```
|
||||
Orchestrator model turn
|
||||
└─ rlm tool call { script, session_id?, timeout_secs?, limits? }
|
||||
└─ src/openhuman/rlm/ (new domain)
|
||||
├─ session manager (persistent ReplSession per rlm session_id)
|
||||
├─ capability bridge (openhuman Tools/Subagents/Provider →
|
||||
│ tinyagents CapabilityRegistry)
|
||||
├─ policy mapping (autonomy tier + SecurityPolicy → ReplPolicy)
|
||||
├─ progress bridge (ReplCallRecord / EventSink → AgentProgress
|
||||
│ + DomainEvent bus)
|
||||
└─ tinyagents::ReplSession::eval_cell (spawn_blocking)
|
||||
└─ rhai engine — model_query / tool_call / agent_query /
|
||||
*_batched / emit / answer (fail-closed ReplPolicy)
|
||||
```
|
||||
|
||||
Two repos change:
|
||||
|
||||
1. **`vendor/tinyagents`** (submodule, separate PR against
|
||||
`tinyhumansai/tinyagents`): host-embedding gaps — external cancellation
|
||||
flag, live capability-call events on the `EventSink`, async-embedding
|
||||
documentation. Branch: `feat/repl-host-embedding`.
|
||||
2. **`openhuman`** (one gigantic PR against `tinyhumansai/openhuman`): the
|
||||
`repl` feature flag, the new `src/openhuman/rlm/` domain, the `rlm` tool,
|
||||
prompt/docs surfacing, and tests. Branch: `feat/rlm-language-workflows`,
|
||||
including the submodule pointer bump once the tinyagents PR lands.
|
||||
|
||||
## Phases
|
||||
|
||||
| Phase | File | Deliverable |
|
||||
| ----- | ---- | ----------- |
|
||||
| 1 | [phase-1-research.md](phase-1-research.md) | Research findings: what tinyagents `repl` provides, what openhuman provides, the gaps |
|
||||
| 2 | [phase-2-tinyagents.md](phase-2-tinyagents.md) | TinyAgents-side changes (cancellation, live events) — separate PR |
|
||||
| 3 | [phase-3-rlm-domain.md](phase-3-rlm-domain.md) | `src/openhuman/rlm/` domain: sessions, capability bridge, policy |
|
||||
| 4 | [phase-4-rlm-tool.md](phase-4-rlm-tool.md) | First-class `rlm` tool: schema, registration, prompt surfacing |
|
||||
| 5 | [phase-5-hardening.md](phase-5-hardening.md) | Error handling, timeouts, cancellation, limits, observability |
|
||||
| 6 | [phase-6-tests.md](phase-6-tests.md) | Tests (written last): unit, timeout/cancel/limit, RPC E2E |
|
||||
| 7 | [phase-7-delivery.md](phase-7-delivery.md) | PR strategy: tinyagents PR + one gigantic openhuman PR |
|
||||
|
||||
Tests are deliberately the **final implementation phase** (per the feature
|
||||
brief): phases 3–5 land the behavior with verbose debug logging; phase 6
|
||||
back-fills unit and E2E coverage to the ≥80% changed-line gate before the PR
|
||||
is opened.
|
||||
@@ -0,0 +1,127 @@
|
||||
# Phase 1 — Research findings
|
||||
|
||||
## 1. What TinyAgents' `repl` feature provides
|
||||
|
||||
Feature gate: `repl = ["dep:rhai"]` with `rhai = { version = "1", features =
|
||||
["sync"] }` (`vendor/tinyagents/Cargo.toml`). The `sync` feature makes engine
|
||||
and values `Send + Sync` so a session can live inside an async task.
|
||||
|
||||
There are **two** `ReplSession` types; we use the scripting one:
|
||||
|
||||
- `tinyagents::repl::ReplSession` — line-oriented command *skeleton*
|
||||
(`load/compile/run/call` return `ReplOutcome::Planned`, never executed).
|
||||
Not our target.
|
||||
- `tinyagents::repl::session::ReplSession` (crate-root re-export
|
||||
`tinyagents::ReplSession` under the feature) — the **Rhai scripting
|
||||
session**. This is the RLM/CodeAct runtime.
|
||||
|
||||
### Session lifecycle
|
||||
|
||||
- `ReplSession::from_parts(capabilities, policy, run_context)`; builders
|
||||
`with_capabilities` / `with_policy` / `with_state` rebuild the engine.
|
||||
- `eval_cell(&mut self, script) -> Result<ReplResult>` — one cell per call;
|
||||
top-level `let` bindings persist across cells (persistent `rhai::Scope`).
|
||||
- `ReplResult { stdout, value, variables_changed, calls, final_answer,
|
||||
elapsed }` — typed, serializable (`ReplValue` = Unit/Bool/Int/Float/
|
||||
String/Array/Map).
|
||||
- Reserved names (`context`, `state`, `messages`, `history`, `run` + 16
|
||||
capability functions) are restored after every cell — scripts can shadow
|
||||
but never permanently replace capabilities.
|
||||
|
||||
### Script-visible built-ins (`src/repl/session/builtins/`)
|
||||
|
||||
| Built-in | Lowers to |
|
||||
| -------- | --------- |
|
||||
| `model_query(#{model, system?, prompt?, structured?})` | `registry.model(name).invoke(...)` |
|
||||
| `tool_call(#{tool, arguments?})` | `registry.tool(name).call(...)` |
|
||||
| `agent_query(#{agent, prompt})` | `registry.agent(name).run(SubAgentInput, events)` |
|
||||
| `model_query_batched / tool_call_batched / agent_query_batched([...])` | bounded concurrency via `stream::iter(..).buffered(max_concurrency)` |
|
||||
| `graph_run / graph_define / graph_validate / graph_compile / graph_diff / graph_register` | `.rag` compiler + resolver + review gate (graph_run returns a *reference*, does not execute) |
|
||||
| `emit(name, #{..})`, `answer(content)`, `show_vars()`, `print/debug` | recorded into `ReplResult` |
|
||||
|
||||
Bridge mechanism: an `Arc<HostContext<State>>` (registry, state, policy,
|
||||
`EventSink`, shared `CellBuffers`) is cloned into every registered closure;
|
||||
results flow back through `Arc<Mutex<..>>` buffers.
|
||||
|
||||
### Fail-closed limits (`ReplPolicy`, defaults)
|
||||
|
||||
`max_operations` 1M · `max_iterations` 16 · `max_script_bytes` 64 KiB ·
|
||||
`max_output_bytes` 256 KiB · `max_model_calls` 64 · `max_agent_calls` 32 ·
|
||||
`max_tool_calls` 128 · `max_graph_calls` 32 · `max_depth` 8 · `timeout`
|
||||
30 s · `max_concurrency` 4 · `generated_graphs_require_review` true.
|
||||
|
||||
Timeout is enforced at **two points**: the engine `on_progress` hook (pure
|
||||
script loops) and `bridge_block_on` (a timer-thread race that drops the
|
||||
in-flight capability future — cancel-safe reqwest). Precise
|
||||
`TinyAgentsError`s (Timeout, LimitExceeded, ModelNotFound, SubAgentDepth, …)
|
||||
are stashed via `CellBuffers::set_host_error` and surfaced verbatim.
|
||||
|
||||
### Async story
|
||||
|
||||
The engine is synchronous; capability calls run through a **blocking
|
||||
bridge** (`futures::executor::block_on`). `eval_cell` must therefore run on
|
||||
`tokio::task::spawn_blocking` (or a dedicated thread) — calling it on an
|
||||
async worker deadlocks a current-thread runtime.
|
||||
|
||||
### Gaps a host must fill (drives Phase 2)
|
||||
|
||||
1. **No external cancellation** — only per-cell wall-clock timeout; no
|
||||
abort handle to stop a running cell on demand.
|
||||
2. **No live progress** — `stdout`/`calls`/`emit` are only readable after
|
||||
the cell returns; nothing streams on the `EventSink` mid-cell.
|
||||
3. **No CodeAct driver loop** — the host owns the "model writes cell →
|
||||
eval → feed result back" loop (in our design, the orchestrator's normal
|
||||
tool-call loop *is* that loop; each `rlm` tool call is one cell).
|
||||
4. `graph_run` doesn't execute compiled graphs (returns a reference map) —
|
||||
out of scope for v1; we expose model/tool/agent capabilities only.
|
||||
5. Sync `eval_cell` + internal `block_on` — handled host-side with
|
||||
`spawn_blocking`, documented in tinyagents as part of Phase 2.
|
||||
|
||||
## 2. What OpenHuman provides (integration points)
|
||||
|
||||
- **Dependency**: `tinyagents = { version = "1.5.0", features = ["sqlite"] }`
|
||||
patched to `path = "vendor/tinyagents"` (git submodule,
|
||||
`tinyhumansai/tinyagents`). We add the `"repl"` feature.
|
||||
- **Tool trait** (`src/openhuman/tools/traits.rs:255`): `name` /
|
||||
`description` / `parameters_schema` / `async execute` (+
|
||||
`execute_with_context`, `permission_level_with_args`, `external_effect`,
|
||||
`timeout_policy`, `display_label/detail`). Registered by adding one
|
||||
`Box::new(...)` line in `all_tools_with_runtime`
|
||||
(`src/openhuman/tools/ops.rs`).
|
||||
- **Tool→tinyagents bridge already exists**: `ToolAdapter`
|
||||
(`src/openhuman/tinyagents/tools.rs:78`) wraps `Arc<dyn openhuman Tool>`
|
||||
and implements `tinyagents::Tool<()>` — we reuse it to project openhuman
|
||||
tools into the REPL's `CapabilityRegistry`.
|
||||
- **Model bridge already exists**: `ProviderModel`
|
||||
(`src/openhuman/tinyagents/model.rs`) implements the tinyagents model
|
||||
trait over openhuman's `Provider`; `assemble_turn_harness`
|
||||
(`src/openhuman/tinyagents/mod.rs:1122`) already builds a
|
||||
`CapabilityRegistry<()>` per turn with models registered.
|
||||
- **Subagents**: `run_subagent(definition, prompt, options)`
|
||||
(`src/openhuman/agent/harness/subagent_runner/`) + parent allowlist
|
||||
(`allowed_subagent_ids`) + `MAX_SPAWN_DEPTH`. We wrap this in a
|
||||
`HarnessAgent` impl so `agent_query("researcher", ...)` spawns real
|
||||
openhuman subagents.
|
||||
- **Timeout/cancel**: `tool_timeout` domain clamps 1–3600 s;
|
||||
`workflows::run_log::register_run_cancel(run_id) -> CancellationToken`.
|
||||
- **Approval/security**: `external_effect_with_args == true` routes through
|
||||
the `ApprovalGate` middleware; per-tool security is enforced inside each
|
||||
tool via `Arc<SecurityPolicy>` — bridged tools keep their own checks, so
|
||||
the REPL inherits them for free.
|
||||
- **Progress**: per-turn `AgentProgress` mpsc sink (streams to UI + run
|
||||
logs) and the global `DomainEvent` bus (`ToolExecutionStarted/Completed`,
|
||||
`Workflow*` events).
|
||||
- **Prompt surfacing**: tool `description()` + `parameters_schema()` ride in
|
||||
the native tool-call API request; the orchestrator's narrative guide is
|
||||
`src/openhuman/agent_registry/agents/orchestrator/prompt.md` + `agent.toml`.
|
||||
- **No existing rhai/RLM surface** in `src/` — this is net-new, but it sits
|
||||
beside `workflows/`, `flows/`, `tinyflows/`, `agent_orchestration/`.
|
||||
|
||||
## 3. Design decision: one cell per tool call
|
||||
|
||||
The orchestrator's existing turn loop already is a CodeAct loop. So the
|
||||
`rlm` tool maps **one tool call → one `eval_cell`**, with an optional
|
||||
persistent `session_id` so a later call continues the same namespace
|
||||
(`let findings = ...` in cell 1, referenced in cell 2). This avoids building
|
||||
a bespoke driver loop, keeps every cell inside the normal approval/permission
|
||||
middleware, and gives the model natural iteration with feedback.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Phase 2 — TinyAgents-side changes (separate PR)
|
||||
|
||||
Branch: `feat/repl-host-embedding` in `vendor/tinyagents` (repo
|
||||
`tinyhumansai/tinyagents`). Raised as its **own PR**; the openhuman PR bumps
|
||||
the submodule pointer to the merged commit.
|
||||
|
||||
Scope is intentionally minimal — only the host-embedding gaps identified in
|
||||
Phase 1 that cannot be worked around from openhuman.
|
||||
|
||||
## 2.1 External cancellation (`ReplCancelFlag`)
|
||||
|
||||
**Problem:** a running cell can only be stopped by its wall-clock timeout.
|
||||
OpenHuman needs to abort an in-flight RLM cell when the user cancels a run
|
||||
(`workflows::run_log::cancel_run`) or the agent turn aborts.
|
||||
|
||||
**Change:** add a shared cancellation flag to the session:
|
||||
|
||||
- `repl/session/types.rs`: `pub struct ReplCancelFlag(Arc<AtomicBool>)` with
|
||||
`new() / cancel(&self) / is_cancelled(&self)`. Cheap `Clone`.
|
||||
- `ReplSession::with_cancel_flag(flag)` stores it; `CellBuffers` carries it
|
||||
alongside `deadline`.
|
||||
- Enforcement mirrors the deadline's two points, fail-closed:
|
||||
- the engine `on_progress` hook terminates the script with a
|
||||
`CANCELLED_TOKEN` sentinel → mapped to a new
|
||||
`TinyAgentsError::Cancelled` in `eval_cell`;
|
||||
- `bridge_block_on_raw` polls the flag in its timer race (tick the select
|
||||
with a short-interval timer when a flag is armed) so a blocked
|
||||
model/tool/agent call is dropped promptly on cancel.
|
||||
- New error variant `TinyAgentsError::Cancelled(String)` in `src/error.rs`.
|
||||
|
||||
## 2.2 Live capability-call events on the `EventSink`
|
||||
|
||||
**Problem:** `ReplResult.calls` is only available after the cell completes;
|
||||
long fan-outs look frozen in the UI.
|
||||
|
||||
**Change:** in `builtins/`, emit a typed event on `HostContext.events`
|
||||
(the existing `EventSink` shared with the run context) at capability-call
|
||||
start and completion — `AgentEvent::Custom`-style records carrying
|
||||
`{session_id, kind: model|tool|agent|emit, name, elapsed?}`. The host
|
||||
(openhuman) subscribes an `EventListener` and forwards to its own progress
|
||||
sink. No new public types beyond what the event enum already supports; if
|
||||
`AgentEvent` lacks a suitable variant, add `AgentEvent::ReplCall
|
||||
{ record: ReplCallRecord, phase: Started|Completed }` behind the `repl`
|
||||
feature.
|
||||
|
||||
## 2.3 Async-embedding documentation
|
||||
|
||||
Document at the `eval_cell` API surface (rustdoc + module README) that the
|
||||
method blocks internally (`futures::executor::block_on`) and must be driven
|
||||
via `spawn_blocking`/dedicated thread from async hosts. This is currently
|
||||
only discoverable by reading `builtins/mod.rs`.
|
||||
|
||||
## 2.4 Tests (in tinyagents, alongside the changes)
|
||||
|
||||
TinyAgents' own convention requires tests with every behavior change, so
|
||||
this crate does **not** defer them:
|
||||
|
||||
- cancel flag set before `eval_cell` → `Cancelled` without starting;
|
||||
- cancel mid-script-loop → terminated promptly (`on_progress` path);
|
||||
- cancel during a hanging capability future → dropped promptly
|
||||
(`bridge_block_on` path);
|
||||
- events observed on the `EventSink` for `model_query`/`tool_call` start +
|
||||
completion with a `ScriptedModel`/`FakeTool`.
|
||||
|
||||
## Acceptance
|
||||
|
||||
`cargo fmt --check && cargo clippy --all-targets -- -D warnings &&
|
||||
cargo test --features repl` green in `vendor/tinyagents`; PR opened against
|
||||
`tinyhumansai/tinyagents` with the summary + commands run.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Phase 3 — The `rlm` domain (`src/openhuman/rlm/`)
|
||||
|
||||
New domain following the canonical module shape. Cargo change: root
|
||||
`Cargo.toml` gains `features = ["sqlite", "repl"]` on the tinyagents dep.
|
||||
|
||||
```
|
||||
src/openhuman/rlm/
|
||||
├── mod.rs # exports only + controller schema pair (none in v1)
|
||||
├── types.rs # RlmSessionId, RlmRunSummary, RlmLimitsOverride, serde types
|
||||
├── policy.rs # autonomy tier + tool_timeout → tinyagents ReplPolicy
|
||||
├── bridge.rs # capability bridge: openhuman tools/model/subagents →
|
||||
│ # tinyagents CapabilityRegistry<()>
|
||||
├── sessions.rs # session manager: persistent ReplSession per session_id
|
||||
├── ops.rs # eval_cell orchestration: spawn_blocking, cancel, events
|
||||
├── tools.rs # RlmTool (Phase 4)
|
||||
└── README.md # module design doc
|
||||
```
|
||||
|
||||
## 3.1 `policy.rs` — mapping openhuman config to `ReplPolicy`
|
||||
|
||||
- Base on `ReplPolicy::default()` (already conservative).
|
||||
- `timeout` = min(caller `timeout_secs` clamped by
|
||||
`tool_timeout::explicit_call_timeout_secs`, global cap). Never unbounded.
|
||||
- `max_agent_calls` / `max_depth` respect `MAX_SPAWN_DEPTH` and the
|
||||
autonomy tier: `readonly` tier ⇒ RLM tool not registered at all (see
|
||||
Phase 4); `supervised` ⇒ defaults; `full` ⇒ caller may raise limits up to
|
||||
a hard ceiling (2× defaults) via the tool's `limits` arg.
|
||||
- `max_concurrency` capped at 8.
|
||||
- Log the resolved policy at `debug` with the `[rlm]` prefix.
|
||||
|
||||
## 3.2 `bridge.rs` — the capability registry
|
||||
|
||||
Builds a `tinyagents::registry::CapabilityRegistry<()>` for a session:
|
||||
|
||||
- **Tools**: take the turn's `Vec<Arc<dyn openhuman Tool>>` (the same list
|
||||
the harness registered, minus exclusions), wrap each in the existing
|
||||
`crate::openhuman::tinyagents::tools::ToolAdapter`, and
|
||||
`registry.replace_tool(name, adapter)`. **Exclusions** (recursion +
|
||||
duplication guards): `rlm` itself, `spawn_subagent`/`spawn_parallel_agents`
|
||||
(use `agent_query` instead), `run_workflow`/`await_workflow`. Because
|
||||
`ToolAdapter` carries each tool's own security/approval behavior, scripts
|
||||
get exactly the same gates as direct tool calls.
|
||||
- **Model**: register the turn's `ProviderModel` (and workload routes via
|
||||
`routes::build_route_models`) under the same names the harness uses, so
|
||||
`model_query(#{model: "chat-v1", ...})` hits the real provider with usage
|
||||
accounting intact.
|
||||
- **Agents**: for each `AgentDefinition` in the parent's
|
||||
`allowed_subagent_ids`, register a `SubagentCapability` — a small adapter
|
||||
implementing `tinyagents::graph::subagent_node::HarnessAgent` whose
|
||||
`run(input, events)` calls
|
||||
`agent::harness::subagent_runner::run_subagent(definition, prompt,
|
||||
options)` with the parent's workspace descriptor, model override, and
|
||||
progress sink threaded through. Depth accounting flows through the
|
||||
session's `RunConfig.depth` so `SubAgentDepth` fails closed.
|
||||
|
||||
## 3.3 `sessions.rs` — session manager
|
||||
|
||||
- `RlmSessionManager` (global `OnceLock`, like `ApprovalGate`): map of
|
||||
`session_id -> Mutex<ReplSessionEntry>` where the entry owns the
|
||||
`ReplSession<(), ()>`, its `ReplCancelFlag`, creation time, and cell count.
|
||||
- `get_or_create(session_id, registry, policy)`; sessions are keyed
|
||||
per-thread/turn-scope (`<thread_id>:<session_id>`) so parallel chats don't
|
||||
share namespaces.
|
||||
- Eviction: idle TTL (30 min) + LRU cap (16 sessions), enforced on access;
|
||||
explicit `close(session_id)`.
|
||||
- The `ReplSession` is `Send` (rhai `sync` feature) but `eval_cell` takes
|
||||
`&mut self` — one cell at a time per session, serialized by the entry
|
||||
mutex; a second concurrent call on the same session returns a typed
|
||||
"session busy" error rather than queueing forever.
|
||||
|
||||
## 3.4 `ops.rs` — evaluating a cell
|
||||
|
||||
`pub async fn eval_rlm_cell(req: RlmEvalRequest) -> anyhow::Result<RlmEvalResponse>`
|
||||
|
||||
1. Resolve/verify session; register cancel token with
|
||||
`workflows::run_log::register_run_cancel`-style bookkeeping tied to the
|
||||
turn's cancellation context (`tinyagents/run_cancellation_context.rs`),
|
||||
linking user cancel → `ReplCancelFlag::cancel()`.
|
||||
2. Subscribe an `EventListener` on the session's `EventSink` that forwards
|
||||
REPL call events to the turn's `AgentProgress` sink
|
||||
(`ToolCallStarted/Completed`-shaped) and publishes coarse
|
||||
`DomainEvent`s (start/finish) on the global bus.
|
||||
3. `tokio::task::spawn_blocking(move || session.eval_cell(&script))` with an
|
||||
outer `tokio::time::timeout` at policy timeout + grace (5 s) as a
|
||||
belt-and-braces bound (the inner deadline should always fire first).
|
||||
4. Map `ReplResult` → `RlmEvalResponse { stdout, value, variables_changed,
|
||||
calls (summarized), final_answer, elapsed_ms, session_id, cells_used,
|
||||
limits_remaining }`, truncating per `max_result_size_chars`.
|
||||
5. Map every `TinyAgentsError` variant to a **model-consumable** error
|
||||
string (Phase 5 details) — errors return as tool results, never panics.
|
||||
|
||||
## 3.5 Wiring
|
||||
|
||||
- `src/openhuman/mod.rs` (domain list): add `pub mod rlm;`.
|
||||
- Debug logging throughout with `[rlm]` prefix, correlation fields
|
||||
`session_id`, `cell_index`, `thread_id`.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Phase 4 — The first-class `rlm` tool
|
||||
|
||||
## 4.1 Tool definition (`src/openhuman/rlm/tools.rs`)
|
||||
|
||||
`RlmTool` implementing `crate::openhuman::tools::Tool`:
|
||||
|
||||
- `name()` → `"rlm"`.
|
||||
- `description()` — teaches the model the surface in one screen: what a
|
||||
cell is, the built-ins (`tool_call`, `agent_query`, `model_query`,
|
||||
`*_batched`, `emit`, `answer`), that `let` bindings persist within a
|
||||
`session_id`, the limits, and 2 worked examples (parallel fan-out over
|
||||
subagents; batched tool calls + reduce loop).
|
||||
- `parameters_schema()`:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"script": { "type": "string", "description": "Rhai workflow cell to evaluate" },
|
||||
"session_id": { "type": "string", "description": "Continue a prior RLM session's namespace; omit for a fresh session" },
|
||||
"timeout_secs": { "type": "integer", "minimum": 1, "maximum": 3600 },
|
||||
"limits": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"max_tool_calls": { "type": "integer" },
|
||||
"max_agent_calls": { "type": "integer" },
|
||||
"max_model_calls": { "type": "integer" },
|
||||
"max_concurrency": { "type": "integer" }
|
||||
}
|
||||
},
|
||||
"close_session": { "type": "boolean", "description": "Close the session after this cell" }
|
||||
},
|
||||
"required": ["script"]
|
||||
}
|
||||
```
|
||||
|
||||
- `permission_level_with_args` → `Execute` (matches `spawn_subagent`).
|
||||
- `external_effect_with_args` → `false` for the tool itself: every effectful
|
||||
operation a script performs goes through the bridged inner tools, each of
|
||||
which carries its **own** `external_effect` and hits the `ApprovalGate`
|
||||
middleware inside `ToolAdapter`'s call path. (Verify in Phase 5 that the
|
||||
approval middleware is actually on the bridged path; if the gate lives
|
||||
only in the harness middleware stack and not in `ToolAdapter`, invoke
|
||||
`ApprovalGate::intercept_audited` inside the bridge — fail closed.)
|
||||
- `timeout_policy(args)` → `ToolTimeout::Secs(clamped timeout_secs)`,
|
||||
default `Secs(300)` — never `Inherit` (a fan-out legitimately outlives the
|
||||
default inherit budget), never `Unbounded`.
|
||||
- `scope()` → `ToolScope::AgentOnly` in v1 (orchestrator surface, not
|
||||
CLI/RPC).
|
||||
- `display_label` → "running RLM workflow"; `display_detail` → first line of
|
||||
the script, elided at 80 chars.
|
||||
|
||||
## 4.2 Registration
|
||||
|
||||
- Re-export from `src/openhuman/rlm/mod.rs`; glob re-export via
|
||||
`src/openhuman/tools/mod.rs` like other domains.
|
||||
- Add to `all_tools_with_runtime` (`src/openhuman/tools/ops.rs`), gated:
|
||||
**not registered** when the autonomy tier is `readonly`, or when
|
||||
`OPENHUMAN_RLM=0`. Env/config default: **on** for `supervised`/`full`.
|
||||
- The tool needs the turn's tool list + provider to build its bridge, so it
|
||||
is constructed with the same runtime handles `all_tools_with_runtime`
|
||||
already passes (security policy, config) plus a late-bound
|
||||
`RlmRuntimeContext` resolved per-call from the fork/turn context (the
|
||||
pattern `SpawnSubagentTool` uses via `current_parent()`).
|
||||
|
||||
## 4.3 Prompt & docs surfacing
|
||||
|
||||
- Native tool-call format carries `description()` + schema automatically —
|
||||
the primary model documentation.
|
||||
- Add a "Language workflows (rlm)" section to
|
||||
`src/openhuman/agent_registry/agents/orchestrator/prompt.md`: when to
|
||||
prefer `rlm` over `spawn_parallel_agents` (ad-hoc control flow, loops,
|
||||
dedup/verify pipelines), the one-cell-per-call model, and session reuse.
|
||||
- Update `src/openhuman/about_app/` feature inventory (new user-facing
|
||||
capability).
|
||||
- `docs`/gitbooks: extend
|
||||
`gitbooks/developing/architecture/agent-harness.md` with the RLM section.
|
||||
@@ -0,0 +1,66 @@
|
||||
# Phase 5 — Error handling, timeouts, cancellation, observability
|
||||
|
||||
The RLM tool executes model-authored code; every failure mode must be
|
||||
fail-closed, bounded, and reported back to the model as a *usable* tool
|
||||
result (so it can fix its script), never as a panic or a hung turn.
|
||||
|
||||
## 5.1 Error taxonomy → model-consumable results
|
||||
|
||||
| Failure | Source | Returned tool result |
|
||||
| ------- | ------ | -------------------- |
|
||||
| Script parse/runtime error | `TinyAgentsError::Validation` | `is_error=true`, rhai diagnostic verbatim + hint ("fix the script and retry in the same session") |
|
||||
| Wall-clock timeout | `TinyAgentsError::Timeout` | which phase timed out (script vs in-flight capability call), elapsed, configured limit |
|
||||
| Op/output/script-size/call-count limit | `LimitExceeded` | the specific limit + counter values + how to split work across cells |
|
||||
| Unknown capability | `ModelNotFound`/`ToolNotFound`/`Capability` | the bad name + the live list of registered tool/agent/model names |
|
||||
| Recursion depth | `SubAgentDepth` | depth + max |
|
||||
| User cancel | `Cancelled` (new, Phase 2) | "cancelled by user", partial `calls` summary |
|
||||
| Session busy / evicted / unknown `session_id` | rlm domain | typed message; unknown id ⇒ fresh session created and noted |
|
||||
| spawn_blocking join error / poisoned session | rlm domain | session is dropped (poisoned namespaces are never reused), error reported |
|
||||
|
||||
Partial results: on timeout/cancel/limit, include whatever `CellBuffers`
|
||||
captured (stdout so far is not available from a failed `eval_cell` in v1 —
|
||||
note this; calls recorded before the failure are, via the live event
|
||||
stream).
|
||||
|
||||
## 5.2 Layered time bounds (belt and braces)
|
||||
|
||||
1. rhai `on_progress` deadline — pure script loops.
|
||||
2. `bridge_block_on` timer race — hung capability futures.
|
||||
3. Outer `tokio::time::timeout(policy.timeout + 5s)` around
|
||||
`spawn_blocking` — defends against bugs in 1–2; logs at `error` if it
|
||||
ever fires (it should not), and drops the session entry (the blocking
|
||||
thread may still be unwinding; never reuse it).
|
||||
4. Harness-level `ToolTimeout::Secs` — the final backstop; the tool's own
|
||||
timeout is always set below it.
|
||||
|
||||
## 5.3 Cancellation flow
|
||||
|
||||
user cancel / turn abort
|
||||
→ existing run-cancellation context (`tinyagents/run_cancellation_context.rs`)
|
||||
→ `ReplCancelFlag::cancel()`
|
||||
→ script terminated at next statement OR in-flight capability future dropped
|
||||
→ `Cancelled` tool result → session left intact (resumable).
|
||||
|
||||
## 5.4 Resource-exhaustion guards beyond ReplPolicy
|
||||
|
||||
- Session manager LRU cap + idle TTL (Phase 3) — no unbounded namespaces.
|
||||
- Result truncation via `max_result_size_chars` before returning to the
|
||||
model.
|
||||
- `calls` summarized (kind, name, elapsed, ok/err) — never raw payloads —
|
||||
in the tool result; full detail only at `debug` log level.
|
||||
- Nested `rlm` is impossible (excluded from the bridge) — no REPL-in-REPL.
|
||||
|
||||
## 5.5 Observability (per debug-logging rules)
|
||||
|
||||
- `[rlm]` tracing on: session create/evict/close, policy resolution, cell
|
||||
start/end (elapsed, counters), every capability call start/finish (name,
|
||||
elapsed, ok/err), every error mapping, cancellation, both timeout layers.
|
||||
- `AgentProgress` events for live UI: cell started, capability calls
|
||||
(forwarded from EventSink), cell completed.
|
||||
- `DomainEvent` bus: reuse the `tool` category events already published by
|
||||
`ToolAdapter` for inner calls; add coarse start/finish workflow events for
|
||||
the cell itself.
|
||||
- Langfuse: inner model/agent calls already traced through the existing
|
||||
provider/observability path; tag spans with `rlm.session_id`.
|
||||
- Never log script-embedded secrets: scripts are logged elided (first line,
|
||||
hash, byte size) at `info`; full script only at `trace`.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Phase 6 — Tests (written last)
|
||||
|
||||
Per the feature brief, tests are back-filled **after** the behavior lands
|
||||
(phases 3–5). Target: ≥80% changed-line coverage (the `pr-ci.yml` diff
|
||||
gate) for the openhuman side. TinyAgents-side tests ship inside Phase 2
|
||||
(that crate's convention does not defer tests).
|
||||
|
||||
## 6.1 Unit tests — `src/openhuman/rlm/`
|
||||
|
||||
Use tinyagents testkit stubs (`ScriptedModel`, `FakeTool`, `StubAgent`) so
|
||||
no network and no real provider is needed.
|
||||
|
||||
`policy.rs` (`#[cfg(test)] mod tests`):
|
||||
- timeout clamped to [1, 3600] and to the global cap; limits override
|
||||
ceiling (≤2× default) enforced; readonly tier → policy builder refuses.
|
||||
|
||||
`bridge.rs`:
|
||||
- excluded tools (`rlm`, `spawn_*`, `run_workflow`) absent from registry;
|
||||
- an openhuman fake tool is callable via the registry and returns its
|
||||
`ToolResult` content;
|
||||
- subagent capability maps `agent_query` prompt → `run_subagent` input
|
||||
(with a stubbed runner seam) and threads depth.
|
||||
|
||||
`sessions.rs`:
|
||||
- namespace persists across two `eval_cell`s in one session;
|
||||
- distinct `session_id`s isolated; thread-scoped keys isolated;
|
||||
- LRU eviction + idle TTL; `close_session` drops the entry;
|
||||
- concurrent second call on a busy session → typed "busy" error (no
|
||||
deadlock);
|
||||
- poisoned/errored session dropped, not reused.
|
||||
|
||||
`ops.rs` / `tools.rs` (the error-proofing matrix, one test per row of the
|
||||
Phase 5 taxonomy):
|
||||
- happy path: script calling `tool_call` + `answer` → RlmEvalResponse with
|
||||
stdout/value/final_answer;
|
||||
- parse error → `is_error` tool result containing the rhai diagnostic;
|
||||
- `while true {}` → Timeout within policy bound (bounded test time: 1–2 s
|
||||
policy);
|
||||
- hanging fake tool future → Timeout via bridge race;
|
||||
- `max_tool_calls = 1` + two calls → LimitExceeded naming the limit;
|
||||
- unknown tool name → error listing registered names;
|
||||
- cancel flag fired mid-cell → Cancelled, session still usable after;
|
||||
- oversized output → LimitExceeded; result truncation applied;
|
||||
- schema validation: missing `script`, bad `timeout_secs` bounds.
|
||||
|
||||
`RlmTool` metadata tests: name/permission/scope/timeout_policy/display
|
||||
label — the same shape other domains' tool tests use.
|
||||
|
||||
## 6.2 Integration — `tests/` (mock backend)
|
||||
|
||||
Extend `tests/json_rpc_e2e.rs` / `scripts/test-rust-with-mock.sh` **only if**
|
||||
the tool is reachable over RPC in v1; since `scope() = AgentOnly`, v1
|
||||
instead adds a Rust integration test that assembles the turn harness with
|
||||
the RLM tool registered and drives a scripted turn where the model calls
|
||||
`rlm` with a fan-out script (mock provider supplies the tool_call). Asserts:
|
||||
progress events observed on the `AgentProgress` sink, final tool result
|
||||
well-formed.
|
||||
|
||||
## 6.3 Frontend
|
||||
|
||||
No new UI surface in v1 (progress rides existing tool-call timeline cards),
|
||||
so no Vitest additions beyond any i18n keys if a display string is added —
|
||||
if one is, add the key to `en.ts` + all locale files and run
|
||||
`pnpm i18n:check`.
|
||||
|
||||
## 6.4 Commands
|
||||
|
||||
```bash
|
||||
GGML_NATIVE=OFF cargo check --manifest-path Cargo.toml
|
||||
pnpm debug rust rlm # targeted domain tests
|
||||
bash scripts/test-rust-with-mock.sh # full rust suite
|
||||
pnpm typecheck && pnpm lint # app unchanged, still gate
|
||||
cd vendor/tinyagents && cargo test --features repl
|
||||
```
|
||||
@@ -0,0 +1,52 @@
|
||||
# Phase 7 — Delivery: two PRs, one gigantic + one focused
|
||||
|
||||
## 7.1 TinyAgents PR (first)
|
||||
|
||||
- Repo: `tinyhumansai/tinyagents`, branch `feat/repl-host-embedding`
|
||||
(created off `357bcc8`, the commit the submodule currently pins).
|
||||
- Content: Phase 2 (cancel flag, live EventSink call events, embedding
|
||||
docs, tests). Small focused commits per tinyagents conventions.
|
||||
- Until it merges, openhuman development proceeds against the local
|
||||
submodule checkout (the `[patch]` already points at
|
||||
`vendor/tinyagents`), so nothing blocks.
|
||||
|
||||
## 7.2 OpenHuman PR (the gigantic one)
|
||||
|
||||
- Repo: `tinyhumansai/openhuman`, branch `feat/rlm-language-workflows` off
|
||||
`upstream/main`; push to `origin` (fork `senamakel/openhuman`), PR
|
||||
`--head senamakel:feat/rlm-language-workflows` against upstream.
|
||||
- One PR containing, as a series of small coherent commits:
|
||||
1. `docs(plans)`: this plan folder.
|
||||
2. `chore(deps)`: enable tinyagents `repl` feature.
|
||||
3. `feat(rlm)`: types + policy mapping.
|
||||
4. `feat(rlm)`: capability bridge.
|
||||
5. `feat(rlm)`: session manager.
|
||||
6. `feat(rlm)`: ops (eval, cancel, events).
|
||||
7. `feat(rlm)`: the `rlm` tool + registration + prompt/about_app docs.
|
||||
8. `fix/feat(rlm)`: hardening pass (error taxonomy, layered timeouts).
|
||||
9. `test(rlm)`: the Phase 6 suite.
|
||||
10. `chore(vendor)`: bump tinyagents submodule pointer to the merged
|
||||
Phase 2 commit (after the tinyagents PR lands; if it hasn't merged
|
||||
yet, the openhuman PR pins the branch head and notes the dependency
|
||||
in the PR body).
|
||||
- PR body: architecture summary (from README), the two-repo dependency,
|
||||
commands run, coverage numbers, and explicit callout that tests were
|
||||
authored in the final phase per the brief.
|
||||
|
||||
## 7.3 Merge order & risk
|
||||
|
||||
1. tinyagents PR merges → retag/pin.
|
||||
2. openhuman PR updates submodule pointer commit, CI full lane re-runs.
|
||||
3. Rollback story: the feature is dark unless the tool registers
|
||||
(`OPENHUMAN_RLM=0` kill switch + not registered on readonly tier);
|
||||
reverting the tool-registration commit disables the surface without
|
||||
touching the domain.
|
||||
|
||||
## 7.4 Follow-ups (explicitly out of v1)
|
||||
|
||||
- Graph execution from scripts (`graph_run` executing compiled graphs) —
|
||||
blocked on tinyagents implementing super-step execution behind the REPL.
|
||||
- RPC/CLI exposure of RLM sessions (controller schemas) + a dedicated UI
|
||||
timeline card for cells.
|
||||
- Durable session persistence across core restarts.
|
||||
- Streaming partial stdout from a running cell.
|
||||
Reference in New Issue
Block a user