docs(plans): phased plan for RLM language-based workflows (tinyagents rhai REPL) (#4510)

This commit is contained in:
Steven Enamakel
2026-07-04 12:29:56 -07:00
committed by GitHub
parent a968f72b2b
commit c0e24d8856
8 changed files with 627 additions and 0 deletions
+66
View File
@@ -0,0 +1,66 @@
# RLM — Language-Based Workflows (Rhai/`.ragsh`) Integration Plan
**Goal:** expose TinyAgents' Rhai-backed REPL language (the `.ragsh` / RLM /
CodeAct surface, gated behind the `repl` cargo feature in
`vendor/tinyagents/Cargo.toml`) as a **first-class `rlm` tool** in the
OpenHuman Rust core, so the orchestrator agent can *write its own workflow
scripts* — fan-out over subagents, batched tool/model calls, loops,
conditionals — and execute them deterministically, similar to Claude Code
Workflows and Recursive Language Models (RLMs).
## Why
Today the orchestrator composes work through fixed primitives:
`spawn_subagent`, `spawn_parallel_agents`, `run_workflow` (WORKFLOW.md
bundles), and typed tinyflows graphs. None of these let the model express
*ad-hoc control flow* — "spawn N readers, dedupe their findings, verify each
survivor with 3 refuters, loop until dry". The `.ragsh` session in TinyAgents
is exactly that surface: a sandboxed, policy-bounded scripting engine whose
only host access is capability functions (`model_query`, `tool_call`,
`agent_query`, batched variants), with fail-closed limits on operations,
wall-clock time, output bytes, call counts, and recursion depth.
## Architecture summary
```
Orchestrator model turn
└─ rlm tool call { script, session_id?, timeout_secs?, limits? }
└─ src/openhuman/rlm/ (new domain)
├─ session manager (persistent ReplSession per rlm session_id)
├─ capability bridge (openhuman Tools/Subagents/Provider →
│ tinyagents CapabilityRegistry)
├─ policy mapping (autonomy tier + SecurityPolicy → ReplPolicy)
├─ progress bridge (ReplCallRecord / EventSink → AgentProgress
│ + DomainEvent bus)
└─ tinyagents::ReplSession::eval_cell (spawn_blocking)
└─ rhai engine — model_query / tool_call / agent_query /
*_batched / emit / answer (fail-closed ReplPolicy)
```
Two repos change:
1. **`vendor/tinyagents`** (submodule, separate PR against
`tinyhumansai/tinyagents`): host-embedding gaps — external cancellation
flag, live capability-call events on the `EventSink`, async-embedding
documentation. Branch: `feat/repl-host-embedding`.
2. **`openhuman`** (one gigantic PR against `tinyhumansai/openhuman`): the
`repl` feature flag, the new `src/openhuman/rlm/` domain, the `rlm` tool,
prompt/docs surfacing, and tests. Branch: `feat/rlm-language-workflows`,
including the submodule pointer bump once the tinyagents PR lands.
## Phases
| Phase | File | Deliverable |
| ----- | ---- | ----------- |
| 1 | [phase-1-research.md](phase-1-research.md) | Research findings: what tinyagents `repl` provides, what openhuman provides, the gaps |
| 2 | [phase-2-tinyagents.md](phase-2-tinyagents.md) | TinyAgents-side changes (cancellation, live events) — separate PR |
| 3 | [phase-3-rlm-domain.md](phase-3-rlm-domain.md) | `src/openhuman/rlm/` domain: sessions, capability bridge, policy |
| 4 | [phase-4-rlm-tool.md](phase-4-rlm-tool.md) | First-class `rlm` tool: schema, registration, prompt surfacing |
| 5 | [phase-5-hardening.md](phase-5-hardening.md) | Error handling, timeouts, cancellation, limits, observability |
| 6 | [phase-6-tests.md](phase-6-tests.md) | Tests (written last): unit, timeout/cancel/limit, RPC E2E |
| 7 | [phase-7-delivery.md](phase-7-delivery.md) | PR strategy: tinyagents PR + one gigantic openhuman PR |
Tests are deliberately the **final implementation phase** (per the feature
brief): phases 35 land the behavior with verbose debug logging; phase 6
back-fills unit and E2E coverage to the ≥80% changed-line gate before the PR
is opened.
@@ -0,0 +1,127 @@
# Phase 1 — Research findings
## 1. What TinyAgents' `repl` feature provides
Feature gate: `repl = ["dep:rhai"]` with `rhai = { version = "1", features =
["sync"] }` (`vendor/tinyagents/Cargo.toml`). The `sync` feature makes engine
and values `Send + Sync` so a session can live inside an async task.
There are **two** `ReplSession` types; we use the scripting one:
- `tinyagents::repl::ReplSession` — line-oriented command *skeleton*
(`load/compile/run/call` return `ReplOutcome::Planned`, never executed).
Not our target.
- `tinyagents::repl::session::ReplSession` (crate-root re-export
`tinyagents::ReplSession` under the feature) — the **Rhai scripting
session**. This is the RLM/CodeAct runtime.
### Session lifecycle
- `ReplSession::from_parts(capabilities, policy, run_context)`; builders
`with_capabilities` / `with_policy` / `with_state` rebuild the engine.
- `eval_cell(&mut self, script) -> Result<ReplResult>` — one cell per call;
top-level `let` bindings persist across cells (persistent `rhai::Scope`).
- `ReplResult { stdout, value, variables_changed, calls, final_answer,
elapsed }` — typed, serializable (`ReplValue` = Unit/Bool/Int/Float/
String/Array/Map).
- Reserved names (`context`, `state`, `messages`, `history`, `run` + 16
capability functions) are restored after every cell — scripts can shadow
but never permanently replace capabilities.
### Script-visible built-ins (`src/repl/session/builtins/`)
| Built-in | Lowers to |
| -------- | --------- |
| `model_query(#{model, system?, prompt?, structured?})` | `registry.model(name).invoke(...)` |
| `tool_call(#{tool, arguments?})` | `registry.tool(name).call(...)` |
| `agent_query(#{agent, prompt})` | `registry.agent(name).run(SubAgentInput, events)` |
| `model_query_batched / tool_call_batched / agent_query_batched([...])` | bounded concurrency via `stream::iter(..).buffered(max_concurrency)` |
| `graph_run / graph_define / graph_validate / graph_compile / graph_diff / graph_register` | `.rag` compiler + resolver + review gate (graph_run returns a *reference*, does not execute) |
| `emit(name, #{..})`, `answer(content)`, `show_vars()`, `print/debug` | recorded into `ReplResult` |
Bridge mechanism: an `Arc<HostContext<State>>` (registry, state, policy,
`EventSink`, shared `CellBuffers`) is cloned into every registered closure;
results flow back through `Arc<Mutex<..>>` buffers.
### Fail-closed limits (`ReplPolicy`, defaults)
`max_operations` 1M · `max_iterations` 16 · `max_script_bytes` 64 KiB ·
`max_output_bytes` 256 KiB · `max_model_calls` 64 · `max_agent_calls` 32 ·
`max_tool_calls` 128 · `max_graph_calls` 32 · `max_depth` 8 · `timeout`
30 s · `max_concurrency` 4 · `generated_graphs_require_review` true.
Timeout is enforced at **two points**: the engine `on_progress` hook (pure
script loops) and `bridge_block_on` (a timer-thread race that drops the
in-flight capability future — cancel-safe reqwest). Precise
`TinyAgentsError`s (Timeout, LimitExceeded, ModelNotFound, SubAgentDepth, …)
are stashed via `CellBuffers::set_host_error` and surfaced verbatim.
### Async story
The engine is synchronous; capability calls run through a **blocking
bridge** (`futures::executor::block_on`). `eval_cell` must therefore run on
`tokio::task::spawn_blocking` (or a dedicated thread) — calling it on an
async worker deadlocks a current-thread runtime.
### Gaps a host must fill (drives Phase 2)
1. **No external cancellation** — only per-cell wall-clock timeout; no
abort handle to stop a running cell on demand.
2. **No live progress** — `stdout`/`calls`/`emit` are only readable after
the cell returns; nothing streams on the `EventSink` mid-cell.
3. **No CodeAct driver loop** — the host owns the "model writes cell →
eval → feed result back" loop (in our design, the orchestrator's normal
tool-call loop *is* that loop; each `rlm` tool call is one cell).
4. `graph_run` doesn't execute compiled graphs (returns a reference map) —
out of scope for v1; we expose model/tool/agent capabilities only.
5. Sync `eval_cell` + internal `block_on` — handled host-side with
`spawn_blocking`, documented in tinyagents as part of Phase 2.
## 2. What OpenHuman provides (integration points)
- **Dependency**: `tinyagents = { version = "1.5.0", features = ["sqlite"] }`
patched to `path = "vendor/tinyagents"` (git submodule,
`tinyhumansai/tinyagents`). We add the `"repl"` feature.
- **Tool trait** (`src/openhuman/tools/traits.rs:255`): `name` /
`description` / `parameters_schema` / `async execute` (+
`execute_with_context`, `permission_level_with_args`, `external_effect`,
`timeout_policy`, `display_label/detail`). Registered by adding one
`Box::new(...)` line in `all_tools_with_runtime`
(`src/openhuman/tools/ops.rs`).
- **Tool→tinyagents bridge already exists**: `ToolAdapter`
(`src/openhuman/tinyagents/tools.rs:78`) wraps `Arc<dyn openhuman Tool>`
and implements `tinyagents::Tool<()>` — we reuse it to project openhuman
tools into the REPL's `CapabilityRegistry`.
- **Model bridge already exists**: `ProviderModel`
(`src/openhuman/tinyagents/model.rs`) implements the tinyagents model
trait over openhuman's `Provider`; `assemble_turn_harness`
(`src/openhuman/tinyagents/mod.rs:1122`) already builds a
`CapabilityRegistry<()>` per turn with models registered.
- **Subagents**: `run_subagent(definition, prompt, options)`
(`src/openhuman/agent/harness/subagent_runner/`) + parent allowlist
(`allowed_subagent_ids`) + `MAX_SPAWN_DEPTH`. We wrap this in a
`HarnessAgent` impl so `agent_query("researcher", ...)` spawns real
openhuman subagents.
- **Timeout/cancel**: `tool_timeout` domain clamps 13600 s;
`workflows::run_log::register_run_cancel(run_id) -> CancellationToken`.
- **Approval/security**: `external_effect_with_args == true` routes through
the `ApprovalGate` middleware; per-tool security is enforced inside each
tool via `Arc<SecurityPolicy>` — bridged tools keep their own checks, so
the REPL inherits them for free.
- **Progress**: per-turn `AgentProgress` mpsc sink (streams to UI + run
logs) and the global `DomainEvent` bus (`ToolExecutionStarted/Completed`,
`Workflow*` events).
- **Prompt surfacing**: tool `description()` + `parameters_schema()` ride in
the native tool-call API request; the orchestrator's narrative guide is
`src/openhuman/agent_registry/agents/orchestrator/prompt.md` + `agent.toml`.
- **No existing rhai/RLM surface** in `src/` — this is net-new, but it sits
beside `workflows/`, `flows/`, `tinyflows/`, `agent_orchestration/`.
## 3. Design decision: one cell per tool call
The orchestrator's existing turn loop already is a CodeAct loop. So the
`rlm` tool maps **one tool call → one `eval_cell`**, with an optional
persistent `session_id` so a later call continues the same namespace
(`let findings = ...` in cell 1, referenced in cell 2). This avoids building
a bespoke driver loop, keeps every cell inside the normal approval/permission
middleware, and gives the model natural iteration with feedback.
@@ -0,0 +1,69 @@
# Phase 2 — TinyAgents-side changes (separate PR)
Branch: `feat/repl-host-embedding` in `vendor/tinyagents` (repo
`tinyhumansai/tinyagents`). Raised as its **own PR**; the openhuman PR bumps
the submodule pointer to the merged commit.
Scope is intentionally minimal — only the host-embedding gaps identified in
Phase 1 that cannot be worked around from openhuman.
## 2.1 External cancellation (`ReplCancelFlag`)
**Problem:** a running cell can only be stopped by its wall-clock timeout.
OpenHuman needs to abort an in-flight RLM cell when the user cancels a run
(`workflows::run_log::cancel_run`) or the agent turn aborts.
**Change:** add a shared cancellation flag to the session:
- `repl/session/types.rs`: `pub struct ReplCancelFlag(Arc<AtomicBool>)` with
`new() / cancel(&self) / is_cancelled(&self)`. Cheap `Clone`.
- `ReplSession::with_cancel_flag(flag)` stores it; `CellBuffers` carries it
alongside `deadline`.
- Enforcement mirrors the deadline's two points, fail-closed:
- the engine `on_progress` hook terminates the script with a
`CANCELLED_TOKEN` sentinel → mapped to a new
`TinyAgentsError::Cancelled` in `eval_cell`;
- `bridge_block_on_raw` polls the flag in its timer race (tick the select
with a short-interval timer when a flag is armed) so a blocked
model/tool/agent call is dropped promptly on cancel.
- New error variant `TinyAgentsError::Cancelled(String)` in `src/error.rs`.
## 2.2 Live capability-call events on the `EventSink`
**Problem:** `ReplResult.calls` is only available after the cell completes;
long fan-outs look frozen in the UI.
**Change:** in `builtins/`, emit a typed event on `HostContext.events`
(the existing `EventSink` shared with the run context) at capability-call
start and completion — `AgentEvent::Custom`-style records carrying
`{session_id, kind: model|tool|agent|emit, name, elapsed?}`. The host
(openhuman) subscribes an `EventListener` and forwards to its own progress
sink. No new public types beyond what the event enum already supports; if
`AgentEvent` lacks a suitable variant, add `AgentEvent::ReplCall
{ record: ReplCallRecord, phase: Started|Completed }` behind the `repl`
feature.
## 2.3 Async-embedding documentation
Document at the `eval_cell` API surface (rustdoc + module README) that the
method blocks internally (`futures::executor::block_on`) and must be driven
via `spawn_blocking`/dedicated thread from async hosts. This is currently
only discoverable by reading `builtins/mod.rs`.
## 2.4 Tests (in tinyagents, alongside the changes)
TinyAgents' own convention requires tests with every behavior change, so
this crate does **not** defer them:
- cancel flag set before `eval_cell``Cancelled` without starting;
- cancel mid-script-loop → terminated promptly (`on_progress` path);
- cancel during a hanging capability future → dropped promptly
(`bridge_block_on` path);
- events observed on the `EventSink` for `model_query`/`tool_call` start +
completion with a `ScriptedModel`/`FakeTool`.
## Acceptance
`cargo fmt --check && cargo clippy --all-targets -- -D warnings &&
cargo test --features repl` green in `vendor/tinyagents`; PR opened against
`tinyhumansai/tinyagents` with the summary + commands run.
@@ -0,0 +1,96 @@
# Phase 3 — The `rlm` domain (`src/openhuman/rlm/`)
New domain following the canonical module shape. Cargo change: root
`Cargo.toml` gains `features = ["sqlite", "repl"]` on the tinyagents dep.
```
src/openhuman/rlm/
├── mod.rs # exports only + controller schema pair (none in v1)
├── types.rs # RlmSessionId, RlmRunSummary, RlmLimitsOverride, serde types
├── policy.rs # autonomy tier + tool_timeout → tinyagents ReplPolicy
├── bridge.rs # capability bridge: openhuman tools/model/subagents →
│ # tinyagents CapabilityRegistry<()>
├── sessions.rs # session manager: persistent ReplSession per session_id
├── ops.rs # eval_cell orchestration: spawn_blocking, cancel, events
├── tools.rs # RlmTool (Phase 4)
└── README.md # module design doc
```
## 3.1 `policy.rs` — mapping openhuman config to `ReplPolicy`
- Base on `ReplPolicy::default()` (already conservative).
- `timeout` = min(caller `timeout_secs` clamped by
`tool_timeout::explicit_call_timeout_secs`, global cap). Never unbounded.
- `max_agent_calls` / `max_depth` respect `MAX_SPAWN_DEPTH` and the
autonomy tier: `readonly` tier ⇒ RLM tool not registered at all (see
Phase 4); `supervised` ⇒ defaults; `full` ⇒ caller may raise limits up to
a hard ceiling (2× defaults) via the tool's `limits` arg.
- `max_concurrency` capped at 8.
- Log the resolved policy at `debug` with the `[rlm]` prefix.
## 3.2 `bridge.rs` — the capability registry
Builds a `tinyagents::registry::CapabilityRegistry<()>` for a session:
- **Tools**: take the turn's `Vec<Arc<dyn openhuman Tool>>` (the same list
the harness registered, minus exclusions), wrap each in the existing
`crate::openhuman::tinyagents::tools::ToolAdapter`, and
`registry.replace_tool(name, adapter)`. **Exclusions** (recursion +
duplication guards): `rlm` itself, `spawn_subagent`/`spawn_parallel_agents`
(use `agent_query` instead), `run_workflow`/`await_workflow`. Because
`ToolAdapter` carries each tool's own security/approval behavior, scripts
get exactly the same gates as direct tool calls.
- **Model**: register the turn's `ProviderModel` (and workload routes via
`routes::build_route_models`) under the same names the harness uses, so
`model_query(#{model: "chat-v1", ...})` hits the real provider with usage
accounting intact.
- **Agents**: for each `AgentDefinition` in the parent's
`allowed_subagent_ids`, register a `SubagentCapability` — a small adapter
implementing `tinyagents::graph::subagent_node::HarnessAgent` whose
`run(input, events)` calls
`agent::harness::subagent_runner::run_subagent(definition, prompt,
options)` with the parent's workspace descriptor, model override, and
progress sink threaded through. Depth accounting flows through the
session's `RunConfig.depth` so `SubAgentDepth` fails closed.
## 3.3 `sessions.rs` — session manager
- `RlmSessionManager` (global `OnceLock`, like `ApprovalGate`): map of
`session_id -> Mutex<ReplSessionEntry>` where the entry owns the
`ReplSession<(), ()>`, its `ReplCancelFlag`, creation time, and cell count.
- `get_or_create(session_id, registry, policy)`; sessions are keyed
per-thread/turn-scope (`<thread_id>:<session_id>`) so parallel chats don't
share namespaces.
- Eviction: idle TTL (30 min) + LRU cap (16 sessions), enforced on access;
explicit `close(session_id)`.
- The `ReplSession` is `Send` (rhai `sync` feature) but `eval_cell` takes
`&mut self` — one cell at a time per session, serialized by the entry
mutex; a second concurrent call on the same session returns a typed
"session busy" error rather than queueing forever.
## 3.4 `ops.rs` — evaluating a cell
`pub async fn eval_rlm_cell(req: RlmEvalRequest) -> anyhow::Result<RlmEvalResponse>`
1. Resolve/verify session; register cancel token with
`workflows::run_log::register_run_cancel`-style bookkeeping tied to the
turn's cancellation context (`tinyagents/run_cancellation_context.rs`),
linking user cancel → `ReplCancelFlag::cancel()`.
2. Subscribe an `EventListener` on the session's `EventSink` that forwards
REPL call events to the turn's `AgentProgress` sink
(`ToolCallStarted/Completed`-shaped) and publishes coarse
`DomainEvent`s (start/finish) on the global bus.
3. `tokio::task::spawn_blocking(move || session.eval_cell(&script))` with an
outer `tokio::time::timeout` at policy timeout + grace (5 s) as a
belt-and-braces bound (the inner deadline should always fire first).
4. Map `ReplResult` → `RlmEvalResponse { stdout, value, variables_changed,
calls (summarized), final_answer, elapsed_ms, session_id, cells_used,
limits_remaining }`, truncating per `max_result_size_chars`.
5. Map every `TinyAgentsError` variant to a **model-consumable** error
string (Phase 5 details) — errors return as tool results, never panics.
## 3.5 Wiring
- `src/openhuman/mod.rs` (domain list): add `pub mod rlm;`.
- Debug logging throughout with `[rlm]` prefix, correlation fields
`session_id`, `cell_index`, `thread_id`.
@@ -0,0 +1,77 @@
# Phase 4 — The first-class `rlm` tool
## 4.1 Tool definition (`src/openhuman/rlm/tools.rs`)
`RlmTool` implementing `crate::openhuman::tools::Tool`:
- `name()``"rlm"`.
- `description()` — teaches the model the surface in one screen: what a
cell is, the built-ins (`tool_call`, `agent_query`, `model_query`,
`*_batched`, `emit`, `answer`), that `let` bindings persist within a
`session_id`, the limits, and 2 worked examples (parallel fan-out over
subagents; batched tool calls + reduce loop).
- `parameters_schema()`:
```json
{
"type": "object",
"properties": {
"script": { "type": "string", "description": "Rhai workflow cell to evaluate" },
"session_id": { "type": "string", "description": "Continue a prior RLM session's namespace; omit for a fresh session" },
"timeout_secs": { "type": "integer", "minimum": 1, "maximum": 3600 },
"limits": {
"type": "object",
"properties": {
"max_tool_calls": { "type": "integer" },
"max_agent_calls": { "type": "integer" },
"max_model_calls": { "type": "integer" },
"max_concurrency": { "type": "integer" }
}
},
"close_session": { "type": "boolean", "description": "Close the session after this cell" }
},
"required": ["script"]
}
```
- `permission_level_with_args``Execute` (matches `spawn_subagent`).
- `external_effect_with_args``false` for the tool itself: every effectful
operation a script performs goes through the bridged inner tools, each of
which carries its **own** `external_effect` and hits the `ApprovalGate`
middleware inside `ToolAdapter`'s call path. (Verify in Phase 5 that the
approval middleware is actually on the bridged path; if the gate lives
only in the harness middleware stack and not in `ToolAdapter`, invoke
`ApprovalGate::intercept_audited` inside the bridge — fail closed.)
- `timeout_policy(args)``ToolTimeout::Secs(clamped timeout_secs)`,
default `Secs(300)` — never `Inherit` (a fan-out legitimately outlives the
default inherit budget), never `Unbounded`.
- `scope()``ToolScope::AgentOnly` in v1 (orchestrator surface, not
CLI/RPC).
- `display_label` → "running RLM workflow"; `display_detail` → first line of
the script, elided at 80 chars.
## 4.2 Registration
- Re-export from `src/openhuman/rlm/mod.rs`; glob re-export via
`src/openhuman/tools/mod.rs` like other domains.
- Add to `all_tools_with_runtime` (`src/openhuman/tools/ops.rs`), gated:
**not registered** when the autonomy tier is `readonly`, or when
`OPENHUMAN_RLM=0`. Env/config default: **on** for `supervised`/`full`.
- The tool needs the turn's tool list + provider to build its bridge, so it
is constructed with the same runtime handles `all_tools_with_runtime`
already passes (security policy, config) plus a late-bound
`RlmRuntimeContext` resolved per-call from the fork/turn context (the
pattern `SpawnSubagentTool` uses via `current_parent()`).
## 4.3 Prompt & docs surfacing
- Native tool-call format carries `description()` + schema automatically —
the primary model documentation.
- Add a "Language workflows (rlm)" section to
`src/openhuman/agent_registry/agents/orchestrator/prompt.md`: when to
prefer `rlm` over `spawn_parallel_agents` (ad-hoc control flow, loops,
dedup/verify pipelines), the one-cell-per-call model, and session reuse.
- Update `src/openhuman/about_app/` feature inventory (new user-facing
capability).
- `docs`/gitbooks: extend
`gitbooks/developing/architecture/agent-harness.md` with the RLM section.
@@ -0,0 +1,66 @@
# Phase 5 — Error handling, timeouts, cancellation, observability
The RLM tool executes model-authored code; every failure mode must be
fail-closed, bounded, and reported back to the model as a *usable* tool
result (so it can fix its script), never as a panic or a hung turn.
## 5.1 Error taxonomy → model-consumable results
| Failure | Source | Returned tool result |
| ------- | ------ | -------------------- |
| Script parse/runtime error | `TinyAgentsError::Validation` | `is_error=true`, rhai diagnostic verbatim + hint ("fix the script and retry in the same session") |
| Wall-clock timeout | `TinyAgentsError::Timeout` | which phase timed out (script vs in-flight capability call), elapsed, configured limit |
| Op/output/script-size/call-count limit | `LimitExceeded` | the specific limit + counter values + how to split work across cells |
| Unknown capability | `ModelNotFound`/`ToolNotFound`/`Capability` | the bad name + the live list of registered tool/agent/model names |
| Recursion depth | `SubAgentDepth` | depth + max |
| User cancel | `Cancelled` (new, Phase 2) | "cancelled by user", partial `calls` summary |
| Session busy / evicted / unknown `session_id` | rlm domain | typed message; unknown id ⇒ fresh session created and noted |
| spawn_blocking join error / poisoned session | rlm domain | session is dropped (poisoned namespaces are never reused), error reported |
Partial results: on timeout/cancel/limit, include whatever `CellBuffers`
captured (stdout so far is not available from a failed `eval_cell` in v1 —
note this; calls recorded before the failure are, via the live event
stream).
## 5.2 Layered time bounds (belt and braces)
1. rhai `on_progress` deadline — pure script loops.
2. `bridge_block_on` timer race — hung capability futures.
3. Outer `tokio::time::timeout(policy.timeout + 5s)` around
`spawn_blocking` — defends against bugs in 12; logs at `error` if it
ever fires (it should not), and drops the session entry (the blocking
thread may still be unwinding; never reuse it).
4. Harness-level `ToolTimeout::Secs` — the final backstop; the tool's own
timeout is always set below it.
## 5.3 Cancellation flow
user cancel / turn abort
→ existing run-cancellation context (`tinyagents/run_cancellation_context.rs`)
`ReplCancelFlag::cancel()`
→ script terminated at next statement OR in-flight capability future dropped
`Cancelled` tool result → session left intact (resumable).
## 5.4 Resource-exhaustion guards beyond ReplPolicy
- Session manager LRU cap + idle TTL (Phase 3) — no unbounded namespaces.
- Result truncation via `max_result_size_chars` before returning to the
model.
- `calls` summarized (kind, name, elapsed, ok/err) — never raw payloads —
in the tool result; full detail only at `debug` log level.
- Nested `rlm` is impossible (excluded from the bridge) — no REPL-in-REPL.
## 5.5 Observability (per debug-logging rules)
- `[rlm]` tracing on: session create/evict/close, policy resolution, cell
start/end (elapsed, counters), every capability call start/finish (name,
elapsed, ok/err), every error mapping, cancellation, both timeout layers.
- `AgentProgress` events for live UI: cell started, capability calls
(forwarded from EventSink), cell completed.
- `DomainEvent` bus: reuse the `tool` category events already published by
`ToolAdapter` for inner calls; add coarse start/finish workflow events for
the cell itself.
- Langfuse: inner model/agent calls already traced through the existing
provider/observability path; tag spans with `rlm.session_id`.
- Never log script-embedded secrets: scripts are logged elided (first line,
hash, byte size) at `info`; full script only at `trace`.
+74
View File
@@ -0,0 +1,74 @@
# Phase 6 — Tests (written last)
Per the feature brief, tests are back-filled **after** the behavior lands
(phases 35). Target: ≥80% changed-line coverage (the `pr-ci.yml` diff
gate) for the openhuman side. TinyAgents-side tests ship inside Phase 2
(that crate's convention does not defer tests).
## 6.1 Unit tests — `src/openhuman/rlm/`
Use tinyagents testkit stubs (`ScriptedModel`, `FakeTool`, `StubAgent`) so
no network and no real provider is needed.
`policy.rs` (`#[cfg(test)] mod tests`):
- timeout clamped to [1, 3600] and to the global cap; limits override
ceiling (≤2× default) enforced; readonly tier → policy builder refuses.
`bridge.rs`:
- excluded tools (`rlm`, `spawn_*`, `run_workflow`) absent from registry;
- an openhuman fake tool is callable via the registry and returns its
`ToolResult` content;
- subagent capability maps `agent_query` prompt → `run_subagent` input
(with a stubbed runner seam) and threads depth.
`sessions.rs`:
- namespace persists across two `eval_cell`s in one session;
- distinct `session_id`s isolated; thread-scoped keys isolated;
- LRU eviction + idle TTL; `close_session` drops the entry;
- concurrent second call on a busy session → typed "busy" error (no
deadlock);
- poisoned/errored session dropped, not reused.
`ops.rs` / `tools.rs` (the error-proofing matrix, one test per row of the
Phase 5 taxonomy):
- happy path: script calling `tool_call` + `answer` → RlmEvalResponse with
stdout/value/final_answer;
- parse error → `is_error` tool result containing the rhai diagnostic;
- `while true {}` → Timeout within policy bound (bounded test time: 12 s
policy);
- hanging fake tool future → Timeout via bridge race;
- `max_tool_calls = 1` + two calls → LimitExceeded naming the limit;
- unknown tool name → error listing registered names;
- cancel flag fired mid-cell → Cancelled, session still usable after;
- oversized output → LimitExceeded; result truncation applied;
- schema validation: missing `script`, bad `timeout_secs` bounds.
`RlmTool` metadata tests: name/permission/scope/timeout_policy/display
label — the same shape other domains' tool tests use.
## 6.2 Integration — `tests/` (mock backend)
Extend `tests/json_rpc_e2e.rs` / `scripts/test-rust-with-mock.sh` **only if**
the tool is reachable over RPC in v1; since `scope() = AgentOnly`, v1
instead adds a Rust integration test that assembles the turn harness with
the RLM tool registered and drives a scripted turn where the model calls
`rlm` with a fan-out script (mock provider supplies the tool_call). Asserts:
progress events observed on the `AgentProgress` sink, final tool result
well-formed.
## 6.3 Frontend
No new UI surface in v1 (progress rides existing tool-call timeline cards),
so no Vitest additions beyond any i18n keys if a display string is added —
if one is, add the key to `en.ts` + all locale files and run
`pnpm i18n:check`.
## 6.4 Commands
```bash
GGML_NATIVE=OFF cargo check --manifest-path Cargo.toml
pnpm debug rust rlm # targeted domain tests
bash scripts/test-rust-with-mock.sh # full rust suite
pnpm typecheck && pnpm lint # app unchanged, still gate
cd vendor/tinyagents && cargo test --features repl
```
@@ -0,0 +1,52 @@
# Phase 7 — Delivery: two PRs, one gigantic + one focused
## 7.1 TinyAgents PR (first)
- Repo: `tinyhumansai/tinyagents`, branch `feat/repl-host-embedding`
(created off `357bcc8`, the commit the submodule currently pins).
- Content: Phase 2 (cancel flag, live EventSink call events, embedding
docs, tests). Small focused commits per tinyagents conventions.
- Until it merges, openhuman development proceeds against the local
submodule checkout (the `[patch]` already points at
`vendor/tinyagents`), so nothing blocks.
## 7.2 OpenHuman PR (the gigantic one)
- Repo: `tinyhumansai/openhuman`, branch `feat/rlm-language-workflows` off
`upstream/main`; push to `origin` (fork `senamakel/openhuman`), PR
`--head senamakel:feat/rlm-language-workflows` against upstream.
- One PR containing, as a series of small coherent commits:
1. `docs(plans)`: this plan folder.
2. `chore(deps)`: enable tinyagents `repl` feature.
3. `feat(rlm)`: types + policy mapping.
4. `feat(rlm)`: capability bridge.
5. `feat(rlm)`: session manager.
6. `feat(rlm)`: ops (eval, cancel, events).
7. `feat(rlm)`: the `rlm` tool + registration + prompt/about_app docs.
8. `fix/feat(rlm)`: hardening pass (error taxonomy, layered timeouts).
9. `test(rlm)`: the Phase 6 suite.
10. `chore(vendor)`: bump tinyagents submodule pointer to the merged
Phase 2 commit (after the tinyagents PR lands; if it hasn't merged
yet, the openhuman PR pins the branch head and notes the dependency
in the PR body).
- PR body: architecture summary (from README), the two-repo dependency,
commands run, coverage numbers, and explicit callout that tests were
authored in the final phase per the brief.
## 7.3 Merge order & risk
1. tinyagents PR merges → retag/pin.
2. openhuman PR updates submodule pointer commit, CI full lane re-runs.
3. Rollback story: the feature is dark unless the tool registers
(`OPENHUMAN_RLM=0` kill switch + not registered on readonly tier);
reverting the tool-registration commit disables the surface without
touching the domain.
## 7.4 Follow-ups (explicitly out of v1)
- Graph execution from scripts (`graph_run` executing compiled graphs) —
blocked on tinyagents implementing super-step execution behind the REPL.
- RPC/CLI exposure of RLM sessions (controller schemas) + a dedicated UI
timeline card for cells.
- Durable session persistence across core restarts.
- Streaming partial stdout from a running cell.