[merge 5/5] docs: agent harness E2E coverage matrix + plan (#3471) (#3668)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
YellowSnnowmann
2026-06-15 17:01:37 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent a54854af03
commit 7d71b06711
2 changed files with 1704 additions and 1 deletions
+20 -1
View File
@@ -180,7 +180,7 @@ Canonical mapping of every product feature to its test source(s). Drives gap-fil
| ----- | -------------------------------------------------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 4.2.1 | User Message Handling | WD+RI | `conversations-web-channel-flow.spec.ts`, `tests/json_rpc_e2e.rs` | ✅ | |
| 4.2.2 | AI Response Generation | WD | `agent-review.spec.ts` | ✅ | Mock LLM |
| 4.2.3 | Streaming Responses | RI | `tests/json_rpc_e2e.rs` | 🟡 | UI streaming assertion thin |
| 4.2.3 | Streaming Responses | RI | `tests/json_rpc_e2e.rs`, `tests/agent_harness_e2e.rs` | | `tests/agent_harness_e2e.rs` adds provider-level SSE tool-arg accumulation (chunked args reassembled + parsed) and engine-level delta forwarding (#3471) |
| 4.2.4 | Parallel inference (cross-thread + within-thread forked turns) | RU+VU | `src/openhuman/channels/providers/web_tests.rs`, `app/src/store/__tests__/chatRuntimeSlice.test.ts`, `app/src/providers/__tests__/ChatRuntimeProvider.test.tsx` | 🟡 | Concurrent same-/cross-thread dispatch, cooperative `CancellationToken` teardown, and parallel-lane stream routing covered; dedicated WD E2E is a follow-up |
### 4.3 Tool Invocation
@@ -195,6 +195,25 @@ Canonical mapping of every product feature to its test source(s). Drives gap-fil
| 4.3.6 | Background Monitor Tools | RU+RI | `src/openhuman/monitor/`, `src/openhuman/tools/ops_tests.rs`, `tests/json_rpc_e2e.rs` | ✅ | First-class monitor domain covers command denial, line streaming, timeout, stop, bounded output, registry exposure, and JSON-RPC list/read surface for #3371 |
| 4.3.7 | Mascot Avatar Animation | VU | `app/src/features/human/Mascot/RiveMascot.test.tsx`, `app/src/features/human/Mascot/riveMaps.test.ts` | ✅ | Rive `MascotSM` state machine: face→pose mapping, Oculus→`visme_codes` viseme normalization, and idle random pose rotation for the `tiny_mascot.riv` upgrade |
### 4.4 Agent Harness Behaviors
End-to-end coverage of the agent harness via the web-chat RPC surface against an in-test scripted upstream (`tests/agent_harness_e2e.rs`) plus browser-level Redux/DOM assertions (`app/test/e2e/specs/agent-harness-behaviors.spec.ts`). Added for #3471.
| ID | Feature | Layer | Test path(s) | Status | Notes |
| ------ | --------------------------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------- |
| 4.4.1 | Subagent Delegation | RI+WD | `tests/agent_harness_e2e.rs`, `app/test/e2e/specs/agent-harness-behaviors.spec.ts` | ✅ | Orchestrator→researcher delegation asserts ≥3 upstream calls, distinct researcher persona, no "Unknown tool:" |
| 4.4.2 | Subagent Clarification (input-required) | RI+WD | `tests/agent_harness_e2e.rs`, `app/test/e2e/specs/agent-harness-behaviors.spec.ts`, `chat-harness-subagent-continue.spec.ts` | ✅ | Subagent question surfaced + reply completes turn; full `continue_subagent` resume covered by the continue spec |
| 4.4.3 | Approval Gate — Approve | RI+WD | `tests/agent_harness_e2e.rs`, `app/test/e2e/specs/agent-harness-behaviors.spec.ts` | ✅ | `approval_request` event → approve → tool executes (timeline `success`, file written) |
| 4.4.4 | Approval Gate — Deny | RI+WD | `tests/agent_harness_e2e.rs`, `app/test/e2e/specs/agent-harness-behaviors.spec.ts` | ✅ | Deny → tool not executed (timeline `error`, file unchanged), agent explains gracefully |
| 4.4.5 | Approval Gate — TTL Timeout | RI | `tests/agent_harness_e2e.rs` | ✅ | `OPENHUMAN_APPROVAL_TTL_SECS` (debug-only) → no decision → auto-deny |
| 4.4.6 | Subagent + Approval Gate | RI | `tests/agent_harness_e2e.rs` | ✅ | Approval fires inside subagent context; decision propagates; artifact content asserted |
| 4.4.7 | Multi-turn State Persistence | RI | `tests/agent_harness_e2e.rs` | ✅ | Turn-2 upstream request carries turn-1 user + assistant messages |
| 4.4.8 | Parallel Subagent Fan-out | RI | `tests/agent_harness_e2e.rs` | ✅ | `spawn_parallel_agents` 2-task fan-out; both child results reach synthesis call |
| 4.4.9 | Max Iterations / Empty Response | RI | `tests/agent_harness_e2e.rs` | ✅ | Tool-call limit surface + empty-completion graceful terminal (both skip Sentry) |
| 4.4.10 | Provider Error Retry | RI | `tests/agent_harness_e2e.rs` | ✅ | First upstream 500 retried by ReliableProvider; second succeeds |
| 4.4.11 | Inference Phase Transitions | WD | `app/test/e2e/specs/agent-harness-behaviors.spec.ts` | ✅ | Redux `inferenceStatusByThread` observes `subagent` phase then clears to idle |
| 4.4.12 | Tool Timeline Completeness | WD | `app/test/e2e/specs/agent-harness-behaviors.spec.ts` | ✅ | Timeline entries carry id/name/status/round; subagent row reaches `success`; rounds non-decreasing |
---
## 5. Built-in Intelligence Skills
File diff suppressed because it is too large Load Diff