openhuman/docs/voice-automate-plan.md

# Phase 1.5 Implementation Plan — `automate(app, goal)`

**Parent tracker:** [`voice-system-actions.md`](voice-system-actions.md) (Change 1.14 / Phase 1.5)
**Decided approach:** Rust inner loop + fast model (chat LLM out of the click loop)
**First proof target:** Music — "play `<song>`" end-to-end
**Status:** Plan — awaiting approval before code

---

## 1. Goal

Turn a single high-level intent ("play Numb by Linkin Park") into a multi-step UI
automation that completes in **one tool call from the orchestrator**, runs fast,
and self-corrects — instead of N separate chat-LLM turns over the raw
`ax_interact` primitives (today's flow; see tracker §1.10–1.13 for why that's
slow and fragile).

## 2. Architecture

```text
 orchestrator (chat LLM)
        │  one call: automate{ app, goal }
        ▼
 AutomateTool (tools/impl/computer/automate.rs)
        │  delegates to
        ▼
 accessibility::automate::run(app, goal)         ← the inner loop (Rust)
        │
        ├─ fast-path dispatch ── app_fastpaths/{music,spotify,slack}.rs
        │      (deterministic; skip the loop entirely when available)
        │
        └─ general loop ──► perceive → decide → act → settle → verify ──┐
               ▲                                                         │
               └────────────── repeat until done / fail / budget ───────┘
                 perceive: ax_list_elements_filtered (existing)
                 decide:   create_chat_provider("automation", cfg) → JSON action
                 act:      ax_press_element / ax_set_field_value / launch_app (existing)
                 settle:   helper "ax_wait_settled" (new) — AXObserver, not sleep
                 verify:   re-read state; confirm the action took effect
```

The **chat model is invoked once** (to pick `automate` and its `goal`). The
**fast model** runs the inner loop with a tiny context (goal + current filtered
snapshot + last result), so each step is ~0.5–1s and cheap.

## 3. Inner-loop algorithm

State carried across iterations: `goal`, `app`, `history: Vec<Step>`, `budget`.

Each iteration:
1. **Perceive** — `ax_list_elements_filtered(app, last_filter_or_"")`, capped/filtered
   exactly as the `ax_interact` tool does today (≤60 elements, never a raw dump).
2. **Decide** — call the fast model with a strict system prompt + the JSON action
   schema (below). Parse one action.
3. **Act** — execute via existing helpers. `launch` → `launch_app`; `press` →
   `ax_press_element`; `set_value` → `ax_set_field_value`; `list` → just re-perceive
   with a new filter.
4. **Settle** — `ax_wait_settled(app, timeout)` (new helper): block until the AX
   tree stops changing (debounced AXObserver notifications) or timeout. Removes the
   timing-race class deterministically.
5. **Verify** — re-read; confirm the expected post-condition (e.g. a new control
   appeared, focus changed, a value was set). Record success/failure in `history`.
6. **Loop** until the model emits `done`/`fail`, or the step budget (e.g. 12) is hit.

### Action schema (fast model output — strict JSON)
```jsonc
{
  "thought": "short reasoning",
  "action": "launch | list | press | set_value | done | fail",
  "app": "Music",            // optional override; defaults to the task app
  "filter": "Highway",       // for list
  "label": "Play",           // for press / set_value
  "value": "Highway to Hell", // for set_value
  "summary": "what happened / why done"  // for done|fail
}
```
Invalid JSON or unknown action → one repair retry, then `fail` with the raw text
logged (never act on a guess — this is the §1.13 hallucination lesson).

## 4. New files & changes (grounded in current layout)

**New**
- `src/openhuman/accessibility/automate.rs` — `run(app, goal, opts) -> Result<AutomateOutcome, String>`; the loop, action schema (serde), fast-model call, step budget, structured `history`.
- `src/openhuman/accessibility/app_fastpaths/mod.rs` + `music.rs` (Spotify/Slack land later) — `try_fastpath(app, goal) -> Option<Result<…>>`.
- `src/openhuman/tools/impl/computer/automate.rs` — `AutomateTool { allow_mutations }`; reuses the `ax_interact` gating posture (mutations opt-in, `SENSITIVE_APPS` denylist, `permission_level_with_args` = Dangerous, `external_effect_with_args` = true).
- `src/openhuman/accessibility/automate_tests.rs` — unit tests for the loop (mock perceive/act/decide), schema parse/repair, budget, fast-path dispatch.

**Changed**
- `accessibility/helper.rs` (macOS Swift) — add `ax_wait_settled` (AXObserver on `kAXValueChanged`/`kAXFocusedUIElementChanged`/`kAXCreated`, debounce ~150ms, bounded ~3s) and return richer element fields (enabled / on-screen / supported actions) from `ax_list`.
- `accessibility/ax_interact.rs` — surface a `ax_wait_settled` Rust wrapper; extend `AXElement` with the new optional fields (back-compat: `#[serde(default)]`).
- `accessibility/mod.rs` — declare `automate`, `app_fastpaths`.
- `inference/provider/factory.rs` — add an `"automation"` role (falls back to the fast/summarization tier) so the loop's model is independently configurable.
- `tools/ops.rs` (`all_tools_with_runtime`), `tools/user_filter.rs` (new `"automate"` family), `agent_registry/agents/orchestrator/agent.toml` (`named` list), `app/src/utils/toolDefinitions.ts` (Settings → Agent Access toggle).
- Tracker: flip Change 1.14 / Phase 1.5 rows from ⏳ Planned → in progress as milestones land.

## 5. Fast-model call

`create_chat_provider("automation", &cfg)` → `(provider, model)`; build a
`ChatRequest { messages, tools: None, stream: None }` with a system prompt that
pins the JSON schema and a user message carrying `{goal, snapshot, history_tail}`.
No tools array — we want a single JSON object back, parsed by us, executed by us.
Temperature low. Token budget small (snapshot is already ≤60 elements).

## 6. Music proof (first target)

`app_fastpaths/music.rs` encodes the §1.11 proven sequence behind one entry:
1. `launch_app("Music")`
2. open `music://music.apple.com/search?term=<query>` (URL scheme)
3. `ax_wait_settled`
4. `ax_list_elements_filtered("Music", <query>)` → find the song row
5. `ax_press_element` the row (navigate into detail)
6. `ax_wait_settled` → `ax_list` the detail page → `ax_press_element("Play")`
7. verify `osascript … get player state == playing` (best-effort, logged)

If the fast-path can't find the row (timing/locale), fall through to the **general
loop**, which is what proves the architecture is app-agnostic.

## 7. Progress streaming

Emit a `DomainEvent` per step (`AutomateProgress { app, step, action, ok }`) on the
event bus; a subscriber bridges to the existing notch/voice status surface
(PR #3166) so the user sees "Opening Music → searching → playing" live. Reuses the
`ApprovalSurfaceSubscriber` bridging pattern.

## 8. Testing

- **Unit** (`automate_tests.rs`, CI-safe): action JSON parse + repair; budget exhaustion → `fail`; fast-path dispatch chosen over loop; verify-failure triggers retry/alternate. Perceive/act/decide are trait-injected so tests need no mic/AX/LLM.
- **Integration** (`#[ignore]`, run on a real Mac): the Music flow end-to-end (mirrors `ax_interact_tests::test_full_flow_search_and_play_acdc`); tool-level success hard-asserted, playback best-effort.
- **Agent-in-the-loop**: ask the running app "play `<song>`", confirm it picks `automate` and the song plays; watch `[automate]` logs.

## 9. Milestones (sequenced)

1. **M1** — `automate.rs` loop skeleton + action schema + fast-model call + `AutomateTool` (gated, registered). Loop runs against existing (non-settled) `ax_interact` helpers. Unit tests. *Compiles + agent can call it.*
2. **M2** — `ax_wait_settled` (helper + wrapper) + verify step wired into the loop. Kills the timing-race class.
3. **M3** — Music fast-path; prove the flow end-to-end on a Mac.
4. **M4** — progress streaming to the notch surface.
5. **M5** — richer element model (enabled/onscreen/actions) for better matching.
6. *(later)* Spotify + Slack fast-paths; vision fallback for Electron; Windows UIA settle parity.

## 10. Risks / open questions

- **Fast model availability** — if no fast tier is configured, fall back to the
  chat model for the loop (still one tool call; just slower). The `"automation"`
  role makes this a config decision, not a hard dependency.
- **AXObserver from the Swift helper** — needs a short run-loop pump; if flaky,
  fall back to a polling settle (count-stable-for-150ms) behind the same wrapper.
- **macOS-only first** — Windows UIA settle/verify parity is M6, gated like the
  existing cfg-dispatch; non-mac/non-win returns the existing clean runtime error.
- **Safety** — `automate` is a mutating tool: same opt-in + `SENSITIVE_APPS`
  denylist + ApprovalGate routing as `ax_interact`; the inner loop may not target a
  denylisted app even if the model asks.