admin/openhuman
1
0
Fork 0
mirror of https://github.com/tinyhumansai/openhuman.git synced 2026-07-27 21:08:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a629627afa07e8536cc7b9c76e3d6d810461f2f7
openhuman/docs/voice-system-actions.md
T

80 KiB
Raw Blame History

Voice → System Action Feature Tracker

GitHub Issue: #3148
Branch: feat/voice-always-on-all (cumulative) — all merged to main. Phase 1 landed via #3168; the full feature was split from the mega-PR #3307 (now closed in favour of the stack) into an 8-PR stack — all 8 merged 2026-06-04: #3340 (main-thread input + CEF fix), #3341 (AX/UIA perception + automate engine), #3342 (wire automate/ax_interact tools), #3343 (Phase 2 always-on engine + RPC), #3344 (always-on Settings toggle + i18n), #3345 (notch status pill — supersedes the closed #3166), #3346 (Phase 3 fast command router), #3362 (Phase 1.5 vision-click fallback)
Started: 2026-06-02
Last updated: 2026-06-09 — added Change 1.17 (PR #3558): browser & app (Spotify/Apple Music/Slack) keyboard-shortcut fast-paths, cross-platform shortcut tables, the hotkey verb, artist-aware Music verification, and actionable automate failure responses — all from live-transcript analysis. Earlier this day: verified all 8 always-on stack PRs merged to main (code paths confirmed post-refactor #3424); global push-to-talk (#3090 via #3349) landed separately. Only open item across all phases: the on-device audio wake-word model (Phase 3) — not started; text-based "Hey Tiny" match remains the interim.

Goal

Enable the app to continuously listen to the user, understand spoken commands, and perform system actions on the laptop — e.g., saying "open my Music player" causes the app to open it, without any hotkey press or manual send.

Companion Feature (Separate PR)

Notch Live Activity Indicator — PR #3166
A transparent NSPanel pill at the top of the primary screen (the macOS notch area) that shows live voice/agent status. Built alongside this feature; will light up automatically once always-on listening is implemented.

Phase 1 — Quick Wins ✅ Complete

Low-effort changes that make the existing hotkey-triggered dictation flow work end-to-end without manual sends or approval prompts.

Change 1.1 — Auto-send after transcription

Status: ✅ Done
Commit: 7269f4373

Problem: After speaking via the dictation hotkey, the transcript appeared in the chat composer but the user had to press Enter manually to send it.

Fix:

  • app/src/hooks/useDictationHotkey.ts — added autoSend: true to the dictation://insert-text event detail
  • app/src/pages/Conversations.tsx — onDictationInsert now checks the flag; when set, calls handleSendMessage(text) directly instead of inserting into the textarea. Added handleSendMessageRef (updated every render) so the mount-time effect can access the latest send function

Result: Press hotkey → speak → message auto-sends to agent. No Enter key needed.

Change 1.2 — Shell allowlist for app-launching — ⚠️ REVERTED / SUPERSEDED

Status: ❌ Reverted — superseded by Change 1.4 (launch_app) and the security review.

What was tried (commit 7269f4373): added "open" / "xdg-open" to READ_ONLY_BASES so open -a Music would run without an approval prompt.

Why reverted: base-command classification can't see args, and open/xdg-open/start can open arbitrary https:// URLs and custom URI handlers — too broad for the Read (no-approval) path (maintainer security review). They were removed from READ_ONLY_BASES; the current code (policy_command.rs:514-520) deliberately keeps them out, with a comment. App launching now goes through the dedicated, gated launch_app tool (Change 1.4), which is scoped to named applications only.

Change 1.3 — Shell tool description fix

Status: ✅ Done
Commit: ec8f5be2e

Problem: Shell tool description said "Execute a shell command in the workspace directory" — the LLM was reasoning that it could only run workspace commands, not launch apps.

Fix:

  • src/openhuman/tools/impl/system/shell.rs — updated description to explicitly mention system actions and app launching examples

Result: Agent now understands the shell tool can perform system actions, not just workspace file operations.

Change 1.4 — Dedicated launch_app tool

Status: ✅ Done
Commit: 802fbca76

Problem: Using the shell tool for app launching requires loosening workspace_only and expanding allowed_commands — a security regression. The shell tool also couldn't be used because the orchestrator's strict named tool list excluded it.

Fix (production approach):

  • src/openhuman/tools/impl/system/launch_app.rs — new tool with PermissionLevel::ReadOnly (never triggers approval gate)
    • macOS: open -a "<app_name>" via tokio::process::Command
    • Linux: gtk-launch, fallback xdg-open
    • Windows: Start-Process via PowerShell
    • Input validation: rejects paths, metacharacters, empty names
    • Unit tests: name, permission, schema, validation, error cases
  • src/openhuman/tools/impl/system/mod.rs — registered module + pub use
  • src/openhuman/tools/ops.rs — added LaunchAppTool to all_tools_with_runtime
  • src/openhuman/tools/user_filter.rs — added "launch_app" family, default_enabled = true
  • app/src/utils/toolDefinitions.ts — added to frontend tool catalog (Settings → Agent Access toggle)

Result: Agent has a purpose-built, always-allow tool for launching apps. No shell exposure, no path security concerns.

Change 1.5 — Orchestrator agent tool scope

Status: ✅ Done
Commit: 7d04fc4bc

Problem: Even though launch_app was registered, it was invisible to the agent. The orchestrator (src/openhuman/agent_registry/agents/orchestrator/agent.toml) has a strict named = [...] allowlist. launch_app was not in it, so it was filtered out. Confirmed via logs: visible=24, names=[...no launch_app...].

Fix:

  • src/openhuman/agent_registry/agents/orchestrator/agent.toml — added "launch_app" to the [tools] named list, alongside "current_time" (same pattern: direct answer without delegation)

Confirmed working via logs:

visible=25, names=[..., launch_app, ...]
[launch_app] ▶ execute called  app_name="Music"
[launch_app] macOS: running `open -a "Music"`
[launch_app] macOS: `open -a` exit=exit status: 0  stderr=
[launch_app] ✓ launch succeeded  msg="Opened 'Music'."

Result: Saying "open my Music app" now opens Music directly. No approval prompt, no delegation, no refusal.

Change 1.6 — SOUL.md capability hint

Status: ✅ Done
Commit: cdd3bb4a4

Problem: Even with the tool available, the agent was refusing ("I can't open apps on your device") because its training overrides the function-calling schema.

Fix:

  • src/openhuman/agent/prompts/SOUL.md — added explicit "What you can do on the user's machine" section listing launch_app, shell, file_read/file_write with the instruction: "Never say 'I can't open apps' when you have a tool to do it. Use the tool."

Result: Agent now knows it has these capabilities and is instructed to use them.

Change 1.7 — Diagnostic logging

Status: ✅ Done
Commit: cdd3bb4a4

Added logging to:

  • src/openhuman/tools/impl/system/launch_app.rs — logs every step: ▶ execute, validation pass/fail, platform dispatch, open -a exit code + stderr, fallback result
  • src/openhuman/agent/harness/session/builder.rs — logs the full list of visible tool names at session build time (previously only logged count)

Result: Can now confirm at a glance whether launch_app is in the tool list and trace every step of its execution.

Change 1.8 — Computer control (mouse + keyboard) — ⚠️ REVERTED

Status: ❌ Reverted (commits 50ca434b7 add, bi0rd96sa revert)

Problem: Agent could open apps but couldn't interact with their UI.

What was tried: Enabled the existing mouse + keyboard tools (enigo / CGEventPost), wired into the orchestrator, user filter, and frontend catalog.

Why reverted: CGEventPost injects synthetic events to the currently focused window. When the focused window was OpenHuman's own CEF renderer (the chat UI), a Space keypress crashed the app — EXC_BREAKPOINT / SIGTRAP in CFRelease → NSKeyValueWillChangeWithPerThreadPendingNotifications → -[NSApplication stop:]. CEF can't handle arbitrary key injection. Confirmed via crash report OpenHuman-2026-06-02-035139.ips.

Replaced by: Change 1.9 (ax_interact) — AXUIElement targets elements directly by label with no synthetic events and no CEF crash risk.

Change 1.9 — AXUIElement app UI interaction (ax_interact)

Status: ✅ Done
Commits: 4f9ca1cad (feature), 2c32b59c9 (exact-match fix), betuerj11/test commits

Problem: Need to interact with desktop app UIs reliably, without the CEF crash from synthetic events.

Fix — uses the macOS Accessibility API (AXUIElement) instead of CGEventPost:

  • src/openhuman/accessibility/helper.rs — extended the unified Swift helper with three commands:
    • ax_list → walk the AX tree, return interactive elements (buttons, fields, cells)
    • ax_press → AXUIElementPerformAction(kAXPressAction) by label, exact match preferred over contains (so "Play" beats "Playlist")
    • ax_set_value → AXUIElementSetAttributeValue(kAXValueAttribute) by label
  • src/openhuman/accessibility/ax_interact.rs (new) — Rust wrappers ax_list_elements, ax_press_element, ax_set_field_value
  • src/openhuman/tools/impl/computer/ax_interact.rs (new) — AxInteractTool with actions list / press / set_value, PermissionLevel::ReadOnly
  • src/openhuman/accessibility/ax_interact_tests.rs (new) — integration tests (open Music → search AC/DC → find row → press)
  • Wired into tools/ops.rs, tools/user_filter.rs, toolDefinitions.ts (App UI Control), orchestrator/agent.toml, SOUL.md

Why it's better than mouse/keyboard:

mouse/keyboard (reverted) ax_interact
Mechanism CGEventPost synthetic events AXUIElementPerformAction direct API
CEF crash risk Yes None
Coordinates Required (needs screenshot) None — finds by label
Works when app unfocused No Yes

Verified working: Direct AX test against Music listed 256 elements including Bollywood Hits, Play, etc.; pressing Bollywood Hits then Play both returned exact=true and acted correctly.

Change 1.10 — Multi-step UI workflow guidance

Status: ✅ Done

Problem: When asked to "play Highway to Hell by AC/DC", the agent ran: launch → list → press Library → press Songs → press "Show Filter Field" → set_value "Highway to Hell" → press "Play". The final press hit the global playback bar Play button (plays last queue item), not the specific song row. Result: app navigated correctly but the wrong/no track played.

Fix:

  • src/openhuman/agent/prompts/SOUL.md — added explicit multi-step workflow:
    1. list → discover elements
    2. set_value → type in filter/search
    3. list again → see filtered results
    4. press the specific item (song row), not the generic Play button
  • Added Apple Music guidance: use shell to open music://music.apple.com/search?term=..., then ax_interact list to see song rows as AXCells, then press the specific row. More reliable than the Library filter field.

Result: Agent is directed to select the specific item before pressing playback, instead of pressing the global Play button after filtering.

Change 1.11 — Apple Music two-step play (navigate then play)

Status: ✅ Done

Problem: When asked to "play Highway to Hell by AC/DC", the agent navigated to the right screen but nothing played. Pressing a search-result row in Apple Music only selects/navigates — it does not start playback. The agent then pressed the global transport Play button, but nothing was queued.

Investigation (empirical AX probing against live Music):

  • Every "Highway to Hell" element (AXCell, AXGroup, AXButton) exposes only the AXPress action — which selects/navigates, never plays.
  • Double AXPress, a real CGEvent double-click on the Top-Results card, and AX-select + Return key all left player state stopped.
  • Working sequence found: AXPress the search-result card to navigate into the song's detail page, then AXPress the Play button on that detail page → player state: playing ✅

Fix:

  • src/openhuman/agent/prompts/SOUL.md — replaced the Apple Music guidance with the exact 5-step sequence: URL-scheme search → list → press song row (navigates in) → list detail page → press detail-page Play. Explicitly warns that pressing a search result only navigates, and the second Play press is mandatory.
  • src/openhuman/accessibility/ax_interact_tests.rs — test_full_flow_search_and_play_acdc exercises the full navigate-then-play flow and logs the osascript ... get player state outcome. Playback is best-effort, not hard-asserted (Apple Music's UI is nondeterministic — see change 1.13); the test hard-asserts only the tool-level press/list successes.

Verified:

[step 4] navigate into song: Ok("Pressed 'Highway to Hell' in 'Music'.")
[step 5] press detail Play: Ok("Pressed 'Play' in 'Music'.")
[step 6] player state: playing
test ... ok

Change 1.12 — One-shot play_music tool (root-cause fix)

Status: ✅ Done

Problem: Even after change 1.11, the agent still used the broken filter-field approach and didn't play. Transcript analysis (~/.openhuman/users/<id>/workspace/session_raw/*.jsonl) revealed two real root causes:

  1. The orchestrator has no shell tool. Change 1.11 put the play guidance in SOUL.md — but the orchestrator runs with omit_identity = true and never sees SOUL.md. Change 1.11b moved it to the ax_interact description, which told the agent to "use the shell tool to open music://..." — but the orchestrator can't run shell (it delegates). The agent wrapped the command in a prompt arg to a delegation tool; it never executed, and it fell back to the filter approach.
  2. Cross-chat memory contamination. The user message was prefixed with [Cross-chat context — historical] containing prior filter-approach "Progress Checkpoint" steps, biasing the agent back to the wrong method.

Fix — stop relying on the LLM to orchestrate a fragile multi-step flow with a tool it lacks. Encapsulate the whole proven sequence in native Rust:

  • src/openhuman/accessibility/ax_interact.rs — play_apple_music(query): open search URL → AX-find + press song cell (navigate) → press detail-page Play → verify player state == playing
  • src/openhuman/tools/impl/computer/play_music.rs (new) — PlayMusicTool, single call play_music{query}, PermissionLevel::ReadOnly, runs the blocking flow via spawn_blocking
  • Registered in ops.rs, user_filter.rs, orchestrator/agent.toml, toolDefinitions.ts

Result: Agent calls play_music{query:'Highway to Hell AC/DC'} once; Rust does search→navigate→play. No shell dependency, no multi-step LLM orchestration, no filter-field fallback. Unit tests pass; the underlying flow is exercised by test_full_flow_search_and_play_acdc (tool-level success hard-asserted, playback best-effort). Note: play_music was later removed in change 1.13 in favour of the generic ax_interact tool — this entry documents the investigation that led there.

Key learning: The orchestrator (chat agent) only reads tool descriptions + agent.toml — NOT SOUL.md (omit_identity=true). Behavior guidance for the chat agent must live in tool descriptions or be encapsulated in the tool itself.

Change 1.13 — Generic any-app tool + filtered list (remove play_music)

Status: ✅ Done

Problem: "Play Numb by Linkin Park" still failed, and the agent hallucinated. Transcript (session_raw/*.jsonl) showed:

  1. play_music hit a 4s timing race — results hadn't rendered, so it returned "No matching song found. Top result cells: [empty]".
  2. The agent fell back to ax_interact list, which dumped 273 elements. The tool result was truncated mid-list, so the model reasoned over a partial view and hallucinated a wrong result ("Numb - Single by Marshmello").

Feedback: A music-specific tool is the wrong abstraction. Build a generic tool that interacts with any app.

Fix:

  • Removed play_music tool + play_apple_music helper and all registrations.
  • ax_interact is now a robust generic any-app tool:
    • ax_list_elements_filtered(app, filter) — Rust-side label filter so list returns only relevant elements (fixes the truncation→hallucination root cause).
    • list action takes a new filter param; output capped at 60 elements with a "narrow your filter" hint; empty-match returns a "UI may still be loading" hint instead of failing hard.
    • Description rewritten to be app-agnostic and document the general navigate-then-activate pattern (pressing a list row/search result selects/opens it; press the action button afterward) — no hardcoded Apple Music steps.

Key learning: Dumping a full AX tree (hundreds of elements) overflows the tool-result budget; the truncated view makes the model hallucinate. Always filter list results to keep them small and accurate.

Change 1.14 — automate(app, goal): Rust-driven multi-step automation ✅ M1–M5 done

Status: ✅ M1 + M2 + M3 + M4 + M5 shipped; M3 proven live on macOS. The only remaining automate work is the vision fallback for Electron/partial-AX apps — tracked under Phase 1.5 below and voice-automate-plan.md. (Per-app native fast-paths beyond the shipped Music one are descoped.)

Agent-in-the-loop fixes (2026-06-03, from two live chat sessions):

  • Mutations were off — the agent correctly called automate but it (and ax_interact) refused because computer_control.ax_interact_mutations=false. Enabled it; also rewrote both refusal messages to point at Settings → Agent Access instead of a config key (the agent had relayed "controls are locked down").
  • Query mis-parse — orchestrator goal …search for "Highway to Hell" by AC/DC, and play it made the after-"play" parser extract "it". extract_play_query now prefers a quoted title + by <artist> and rejects bare pronouns. (Unit-tested with the exact failing goal.)
  • General loop spun — pressed "Search" 11× to budget exhaustion. Added a no-progress guard: 3 identical actions in a row → abort.
  • Search-results timing — the fast-path's retry burned out before catalog results rendered (settle reports count-stable while the network fetch is pending). Added a real, mockable wait between attempts (6 × ~800ms).

M5 finding — AXEnabled is unreliable: plumbed an enabled field end-to-end (Swift axEnabled → AXElement.enabled → Windows stub), but Apple Music reports its pressable search-result rows as enabled=Some(false). Gating pick_row on it broke playback. So enabled is kept informational only (documented on the struct); matchers never skip on it. The better future actionability signal is AXPress-action support, not AXEnabled.

M4 — live progress in the notch (2026-06-03): the notch indicator (originally PR #3166) was cherry-picked onto this branch (feat(notch) + fmt commits → notch_window.rs NSPanel + notch/NotchApp.tsx, auto-shown on startup, transparent when idle). The automate loop and Music fast-path now call overlay::publish_attention(...) at each step (Opening …, Searching Music for …, Pressing …, Typing …, Playing …, plus done/fail), which the existing Socket.IO bridge emits as overlay:attention and the notch renders as a pill — so the user sees the automation happening live. Verified: app boots with [notch-window] panel shown at top-center; Tauri shell + frontend compile; 31 automate unit tests green.

M3 live proof (2026-06-03): music_fastpath_live drives real Apple Music end-to-end and hard-asserts player state == playing — confirmed: pre-state paused → post-state playing. Three bugs the live runs surfaced, all fixed + tested:

  1. Perceive filter was the whole multi-word query — a substring filter can't match a full title → now filters by the first strong token and pick_row does the token match.
  2. Search results render late (§1.13 race) — retry perceive across up to 4 settles; AC/DC now percent-encodes correctly.
  3. False success: pressed the toolbar Play, not the song's — the first run reported success but nothing played. AX probing showed the search screen has only the toolbar transport Play (empty queue → silence); pressing the song row navigates to a detail page where a second Play appears (23→24 controls). Fix: capture the baseline Play count, wait for the detail Play to render, press it, then verify real playback via osascript … player state and retry (≤3×). Added verify_playing() to AutomateBackend (macOS osascript; None elsewhere = best-effort). automate now only reports a play success when audio is actually playing — the false-success class (§1.11) is closed.

M3 — shipped (Music fast-path):

  • src/openhuman/accessibility/app_fastpaths/{mod.rs,music.rs} (new) — deterministic accelerators consulted by run() before the general loop. Music encodes the §1.11 proven sequence: launch → open music://…/search?term=… → settle → press the song row (navigate) → settle → press the detail-page Play. Pure helpers matches / extract_play_query (handles "play X by Y", "launch Music and play …", "play X in Apple Music").
  • Structurally different from the removed play_music tool (§1.13): this is internal to automate, not a tool the LLM selects, and on any failure/None the loop falls through to the general model-driven path — so it can only help. Added open_url to the AutomateBackend trait (cross-platform opener; fast-path only).
  • Tests: 9 unit (parser cases, full scripted sequence, no-row fallthrough, dispatch) + 1 #[ignore] macOS live test. Live proof on a Mac: cargo test --lib music_fastpath_live -- --ignored --nocapture (needs Music + Accessibility permission).

M2 — shipped (real settle):

  • src/openhuman/accessibility/ax_interact.rs — new ax_wait_settled(app, stable_ms, timeout_ms): polls the app's interactive-element count and returns once it holds steady for stable_ms (or timeout_ms elapses). Portable — rides on ax_list_elements, which already cfg-dispatches (macOS AX / Windows UIA). Pure decision core counts_settled(history, n) extracted and unit-tested (5 non-OS-gated tests).
  • automate.rs — RealBackend::settle now calls ax_wait_settled (240ms stable / 2s cap) via spawn_blocking instead of the M1 blind 450ms wait. This is the piece that removes the timing-race failure class (§1.11/§1.13): the next perceive always sees a settled tree. An AXObserver-driven settle can later sit behind the same signature.

M1 — shipped:

  • src/openhuman/accessibility/automate.rs (new) — the perceive→decide→act→settle loop, generic over an injectable AutomateBackend (so the model + AX + launcher are all mockable). Strict JSON action schema (launch/list/press/set_value/done/fail) with a one-shot repair retry on unparseable output (never acts on a hallucinated guess), a step budget (default 12), and a snapshot cap (40 elements) mirroring ax_interact's anti-truncation guard. RealBackend calls the existing AX primitives + launch_platform, and routes decisions through the fast tier (create_chat_provider("memory", …) for now; a dedicated automation_provider knob is a follow-up). Settle is a short fixed wait in M1 (M2 makes it AXObserver-driven).
  • src/openhuman/tools/impl/computer/automate.rs (new) — AutomateTool { app, goal }. Always Dangerous + external_effect (routes through the ApprovalGate); reuses ax_interact's mutations opt-in (computer_control.ax_interact_mutations) and the shared is_sensitive_app denylist.
  • Registered everywhere: tools/ops.rs, tools/user_filter.rs (automate family), orchestrator/agent.toml (named), app/src/utils/toolDefinitions.ts (Settings → "App Automation").
  • Tests: 18 passing — loop happy path, navigate-then-activate, app override, budget exhaustion, repair retry (1 ok / 2 fail), explicit fail, non-fatal press failure, JSON parse (plain/fenced/garbage), snapshot cap/empty-hint; tool gating (missing args, mutations-off, sensitive-app refusal, schema).

Problem (the real-time bar): The user's target is "whatever I say happens, live, in front of me" — e.g. "Launch Music and play Numb by Linkin Park" or "open Slack and message Steven 'hi'". Today every UI step (list → set_value → list → press …) is a separate chat-LLM turn. A Slack message is ~7 turns; at 1–3 s each that's 15–25 s, and each turn is a fresh chance to hit a timing race (1.13) or hallucinate. The heavy chat model is sitting inside the click loop — the wrong place for it.

Root causes (all four documented earlier in Phase 1):

  1. Timing races — list/press do a single AX walk with no settle/wait; the UI hasn't rendered yet (1.11/1.13).
  2. Navigate-then-activate is re-reasoned every call — pressing a row selects; you must then press the action control. That logic lives in prose, so it's re-derived (often wrongly) each turn (1.10/1.11).
  3. Round-trip explosion — N full chat turns per task = latency + cost + N chances to fail.
  4. Weak element model + no verification — list returns flat [role, label]; press reports success on AXAction == .success even when nothing changed.

Design — take the chat model out of the click loop:

  • New tool automate { app, goal } — one call from the orchestrator. Rust then runs a tight perceive → act → verify loop internally: read a filtered AX snapshot → pick the next action → act → wait for the UI to settle (AXObserver, not fixed sleeps) → verify it took effect → repeat until the goal is met or a step budget is hit.
  • A fast model drives the inner loop (Haiku-class) with a tiny context: just the goal, the current small AX snapshot, and the last result — not the whole conversation. Each inner step is ~0.5–1 s and self-corrects, instead of one 3 s chat turn that falsely reports success.
  • Settle + verify in Rust between steps — deterministic, kills the timing-race class in one place.
  • Native fast-paths for high-value apps (skip the UI entirely where possible):
    • Music — music:// search URL → AX play (already explored in 1.11), or AppleScript for library. (Shipped — M3.)
    • Spotify — Web API search → spotify:track:… URI + AppleScript play. Descoped — the general loop covers it; not worth the per-app maintenance now.
    • Slack — deep link slack://channel?… to open the DM, then AX to type + send. Descoped — handled by the general loop / vision fallback. The general AX loop is the fallback for everything else.
  • Vision fallback for Electron/Chromium apps (Slack, Discord, VS Code, Spotify-desktop) whose AX/UIA tree is partial (documented limitation). Slack needs accessibility enabled (defaults write com.tinyspeck.slackmacgap AccessibilityEnabled -bool true, relaunch). Where AX returns empty, fall back to screenshot → vision-locate → guarded click. This is the reverted CGEventPost path (1.8) — but it crashed only when events hit OpenHuman's own focused CEF window; a guarded click into a different, foregrounded app does not have that failure mode.
  • Stream progress events to the UI / notch pill (PR #3166) so the user sees each step happen live.

Why a generic automate, not per-app tools: Change 1.13 already established that app-specific tools (play_music) are the wrong abstraction. The abstraction that is generic is the navigate-then-activate sequence itself — automate(app, goal) encapsulates it once, in Rust, for every app, instead of asking the chat model to re-orchestrate fragile primitives every time.

Phase 1.5 — Reliable, real-time multi-step automation ✅ Complete

The bridge between today's ax_interact primitives and the always-on voice work. Prerequisite for Phase 3 — fast voice routing into a slow/fragile action loop still feels slow. This is where "whatever I say happens, live" actually gets delivered.

Status: the Rust inner loop (M1), poll-until-stable settle (M2), Music fast-path (M3, proven live), notch progress streaming (M4), the richer element model (M5), the vision fallback for Electron/partial-AX apps (Change 1.16), and Change 1.17 (browser & app fast-paths + keyboard-shortcut driving + artist-aware Music + actionable responses) are all shipped. Per-app fast-paths now cover Music, browsers, Spotify, Apple Music, and Slack (Spotify/Slack — previously descoped — are now handled by the app_shortcuts.rs keyboard-shortcut path).

Detailed implementation plan: voice-automate-plan.md — decided approach: Rust inner loop + fast model, first proof target Music.

Planned files:

  • src/openhuman/accessibility/automate.rs (new) — the perceive→act→verify loop + settle/verify primitives, reusing ax_interact helpers.
  • src/openhuman/accessibility/app_fastpaths/ (new) — per-app deterministic paths behind a generic dispatch. Shipped: music.rs (M3). Further per-app fast-paths (Spotify/Slack) are descoped — the general loop handles them.
  • src/openhuman/tools/impl/computer/automate.rs (new) — AutomateTool { app, goal }, gated like ax_interact (mutations opt-in, sensitive-app denylist reused).
  • macOS helper (accessibility/helper.rs) — AXObserver-based settle (ax_wait_settled) + post-action verify; richer element model (enabled/onscreen/actions).
  • Vision fallback — screenshot via accessibility/capture.rs → locate → guarded click (only when AX tree is empty, target app foregrounded, never OpenHuman's own window).

Acceptance criteria:

  • One automate{app, goal} call performs a multi-step flow end-to-end (no per-step chat turns)
  • Settle/verify removes the timing-race + false-success failure classes (1.11/1.13 do not recur)
  • Music flow ("play ") works end-to-end via the inner loop (proven live, hard-asserts player state == playing)
  • Electron/partial-AX apps fall back to vision+guarded-click without the CEF crash (Change 1.16)
  • Step-by-step progress streamed to the UI / notch indicator (M4)

Change 1.15 — Full computer control (mouse/keyboard/screenshot) ✅ Crash fixed (main-thread dispatch)

Status: ✅ Keyboard/mouse now run on the app main thread → no CEF crash. Screenshot downscales for inline view. Live: [computer] registered main-thread synthetic-input executor on boot.

The fix: the crash was enigo's TSMGetInputSourceProperty running on a tokio worker (_dispatch_assert_queue_fail/SIGTRAP). macOS TSM must run on the main thread. New tools/impl/computer/main_thread.rs (MainThreadInputOp + run_input_on_main) dispatches each enigo op over the native registry to a handler the Tauri shell registers at startup, which runs it via AppHandle::run_on_main_thread. Keyboard + mouse tools no longer spawn_blocking enigo on a worker. Headless/CLI (no executor) returns a clear error instead of crashing. 66 keyboard/mouse tests green.

Goal: make the agent fully autonomous — when the accessibility tree is empty (Electron apps: Slack/Discord/VS Code), fall back to vision + synthetic input. Enabled computer_control.enabled, added mouse/keyboard/screenshot to the orchestrator named list + autonomy.auto_approve, and taught prompt.md a keyboard-first ladder (foreground via launch_app → keyboard type + Enter; Slack Cmd+K recipe).

Foreground-first: automate::run now open -as the target app at the very start, always, so AX/input hit the right window.

Screenshot fix: oversized Retina captures were returned as "too large to base64-encode inline" (the model was blind). Now downscaled to a viewable JPEG (downscale_to_jpeg) with reported dimensions.

Root cause (now fixed) — OpenHuman-2026-06-03-170058.ips: EXC_BREAKPOINT/SIGTRAP on a tokio-rt-worker thread:

enigo::macos::get_layoutdependent_keycode → TSMGetInputSourceProperty
→ dispatch_assert_queue → _dispatch_assert_queue_fail → SIGTRAP

enigo's keyboard-layout lookup (TSMGetInputSourceProperty) must run on the app's main thread; the keyboard tool ran on a tokio worker → macOS trapped. Not a focus issue (same §1.8 root cause); a frontmost-app guard would not have fixed it.

Fix applied: all enigo operations now run on the Tauri main thread via run_input_on_main (new main_thread.rs module + a native-registry handler the shell registers, dispatched through AppHandle::run_on_main_thread) and wrapped in catch_unwind. Keyboard/mouse tools and vision_click now execute without TSM traps.

Tests: voice-actions + autonomy suite is exhaustive — 220 feature unit tests + a JSON-RPC E2E (json_rpc_voice_server_settings_roundtrip_always_on_and_wake_word). The E2E caught + fixed real gaps (wake_word missing from the get output and the update RPC path). Screenshot downscale unit-tested.

Change 1.16 — vision_click: vision fallback for Electron/partial-AX apps ✅ Done

Status: ✅ Shipped — closes the last open Phase 1.5 item. Electron/Chromium apps (Slack, Discord, VS Code) expose little or no AX/UIA tree, so the automate perceive→press loop had nothing to act on. The loop can now see the screen and click a described element.

Problem: when perceive returns an empty (or target-missing) element list, the loop was stuck — there was no label to press. The planned answer (tracker §1.5) was screenshot → vision-locate → guarded click, but two things blocked it: (1) the fast inner-loop model is text-only, and (2) the deferred F2 coordinate-mapping gap — the screenshot is windowed + downscaled and Retina is 2×, while mouse expects absolute screen points, so a vision-returned coordinate would click the wrong spot.

Fix — a new model-chosen vision_click { description } action in the automate loop:

  • Reuses the existing multimodal path — the OpenAI-compatible provider promotes an embedded [IMAGE:<data-uri>] marker into a real image_url part (inference/provider/compatible_types.rs, #3205), so vision-locate rides the existing chat_with_system call with no new inference API. The locate call uses the main chat/vision provider (create_chat_provider("chat", …)) — reliable UI grounding, and the fallback only fires when AX is empty (rare).
  • Folds in the F2 coordinate transform — src/openhuman/accessibility/vision_click.rs (new): CaptureGeometry (window screen-rect in points + image pixel size) + a pure, unit-tested image_to_screen(geom, px, py). The px→pt ratio absorbs both the downscale and the Retina backing scale, so no explicit scale factor is needed; the result is clamped strictly inside the target window.
  • §1.8 safety guard — a vision_click only fires when the target app is frontmost (focus::foreground_context); on positive evidence a different app is focused it re-foregrounds once and otherwise refuses, so synthetic input never lands on OpenHuman's own CEF window. None (can't tell) = best-effort, since the loop already foregrounded the app at start.
  • Main-thread click — the guarded left-click runs via run_input_on_main (Change 1.15), so off-thread enigo never traps TSM.
  • Wired into the automate loop: new Action.description, a vision_click system-prompt verb ("use when the element list is EMPTY — Electron apps"), the no-progress signature extended with description, and RealBackend::{screenshot,locate,frontmost_app,click}. Inherits automate's existing gating (Dangerous + mutations opt-in + sensitive-app denylist) — no new tool, no new approval surface.

Tests: pure image_to_screen (downscale / Retina 2× / origin offset / out-of-range + negative clamp / zero-dim safety), parse_locate_response (found / not-found / fenced / prose / garbage), build_locate_user (marker), image_dims_from_data_uri (real PNG round-trip); loop integration via the scripted backend (locate-and-click when frontmost, proceed when frontmost unknown, refuse when another app is frontmost, no-click on not-found, empty-description skip). A macOS live run against a real Electron app is the remaining manual validation (vision is nondeterministic — assert tool-level success, treat the visual outcome as best-effort, same caveat as Music).

Change 1.17 — browser & app fast-paths, keyboard-shortcut driving, artist-aware Music, actionable responses ✅ Done

Status: ✅ Shipped (PR #3558). One body of work driven by live-transcript analysis of the automate agent across browsers, Apple Music, Spotify, and Slack. The transcripts surfaced four recurring failures the agent kept hitting:

  1. Browsers — for "open my Brave browser, go to youtube.com and play a music video" the desktop_control_agent was re-delegated 6× (each sub re-launched Brave + re-navigated), ran ax_interact on the wrong app names (Google Chrome/Safari, never the real Brave Browser, because Chromium exposes no AX tree), then "played" via a blind hardcoded mouse click at (400,350) with no verification.
  2. Wrong track — "play Numb by Linkin Park" played a same-titled song by the wrong artist ("Numb" by Marshmello & Khalid) and the agent claimed success, because the AX row label is title-only and verification only checked that something played.
  3. Inert responses — the automate tool's replies were honest but gave no next move, so the agent re-ran the same failing search 6× instead of changing tactics.
  4. No keyboard-shortcut lever — the loop hunted AX labels even where an app's own shortcut would be instant and reliable.

The six parts below address all four. All fast-paths are consulted by automate::run before the model loop (try_fastpath order: music → browser → app_shortcuts); on any failure they fall through, so they can only help.

(a) Browser fast-path — app_fastpaths/browser.rs (new). Pure parsers + run(app, goal, backend):

  • resolve_browser(app, goal) — aliases → macOS display names (brave→Brave Browser, chrome→Google Chrome, edge→Microsoft Edge, safari/firefox/arc). Checks the app arg first, then the goal; a generic "browser" with no named product → None so it never guesses the wrong app (the original bug).
  • extract_destination(goal) — YouTube → youtube.com/results?search_query=… (percent-encoded); Google/"search for X" → google.com/search?q=…; a bare domain/URL → normalized to https://….
  • Navigation is one deterministic step: backend.open_url_in_app(browser, url) → open -a "<browser>" "<url>" launches/foregrounds + navigates — no address-bar typing, no AX. Pure-navigation goals complete here; play goals navigate to the results URL then return non-success so the loop does the single "click first result" via vision_click (no native shortcut selects a result — we don't fake it). In-page media control ("pause/next the video") sends the YouTube shortcut directly.

(b) Cross-platform browser shortcut table — app_fastpaths/browser_shortcuts.rs (new). shortcut(intent, browser, os) resolves browser-level intents (address bar, new/close/reopen tab, tab N, next/prev tab, find, reload/hard-reload, back/forward, history, downloads, zoom, private window, fullscreen) per Chrome/Firefox/Safari/Edge × macOS/Windows/Linux. Nearly uniform (only ⌘↔Ctrl differs); encodes the real exceptions (Firefox-Linux Alt+1‑8, Firefox private window ⌘/Ctrl+Shift+P, per-browser History/Downloads, Mac ⌘[/] back-forward, F11 vs ⌃⌘F fullscreen). Sourced from the official Chrome/Firefox/Edge/Safari docs. browser.rs parses commands ("open a new tab", "go back", "reload", "switch to tab 3") and dispatches the resolved chord. (Brave/Arc/Chromium → Chrome family.)

(c) App-shortcut fast-path — app_fastpaths/app_shortcuts.rs (new). Same model for Spotify, Apple Music, Slack: shortcut(intent, app, os) + goal parser. Drives transport/navigation by each app's own global shortcut (faster + more reliable than AX). Media intents (play/pause, next, prev, volume ±, mute, shuffle, repeat, search) for Spotify + Apple Music; Slack nav (quick-switcher, search, new message/compose DM, next/prev unread, next/prev channel, threads, all-unread, mark read). Encodes the quirks: Spotify uses ↓/↑ for next/prev (not ←/→); Apple Music prefixes everything with Ctrl on Windows (Ctrl+Space vs bare Space on Mac); Slack channel-nav uses Option/Alt+arrows. Intents with no simple chord (Spotify volume, Apple Music mute / Windows search access-key) return None → fall through. Complementary to music.rs, which still owns Apple Music song search/play.

(d) Keyboard-shortcut plumbing —

  • Extracted shared pub(crate) helpers run_hotkey/run_key/run_type_text from tools/impl/computer/keyboard.rs (one place for validation + main-thread enigo dispatch, Change 1.15) — no behavior change to the keyboard tool.
  • New AutomateBackend methods (non-breaking defaults): open_url_in_app, key, type_text, now_playing. RealBackend impls open -a per-app, delegates keystrokes to the keyboard helpers, and reads Music state via osascript.
  • New hotkey verb in the general loop + system prompt ({"action":"hotkey","keys":["Cmd","L"]}) so the model can use any app's shortcuts — folded into the no-progress signature.

(e) Artist-aware Music verification — music.rs + now_playing. Apple Music's AX row label is title-only ("Numb - Single"), so a title match lands on the wrong artist. music.rs now parses the requested artist (extract_artist, "…by Linkin Park") and after Play reads the now-playing name + artist via AppleScript (now_playing → osascript … get {name, artist} of current track). On a match the summary names the verified track + artist; on a mismatch it says so honestly ("Now playing 'Numb' by 'Tom Odell'…") instead of claiming the requested track. Closes the wrong-artist false-success. Follow-up: correcting to the right artist is limited by AX press-by-label hitting only the first same-titled row — a library AppleScript play (track whose name … and artist …) fallback is the next step.

(f) Actionable automate responses — the tool's failures are now agent-actionable, not inert. Four fixes: (1) terminal failures (music.rs no-match/mismatch, automate.rs budget-exhausted/no-progress) carry next-step guidance + a "don't repeat this" steer; (2) they list the actual candidates / on-screen labels (candidate_labels, screen_hint) instead of a bare negative; (3) verified-only success — never "Playing 'X'" unless now_playing confirms it, and an artist/album press is reported as "navigated, no track started"; (4) pick_row now prefers real song rows (AXCell/AXRow) over artist/album buttons (the live bug pressed the "LINKIN PARK" artist row) and the step log names the element type pressed.

Also: the always-on toggle in VoicePanel is now shown in dev/debug builds (IS_DEV_LIKE) and hidden in production.

Follow-up fix — Full OS access now enables app control. ax_interact (press/set_value) and automate gated only on computer_control.ax_interact_mutations, which is independent of the autonomy level. A user who granted Full access in Settings → Agent Access still got "App control isn't enabled yet…" because that flag was untouched (confirmed live: SecurityPolicy autonomy=Full while the tool refused). Fixed with a shared app_control_enabled(explicit_opt_in) gate (tools/impl/computer/ax_interact.rs) that returns true when the explicit flag is set or the live policy (security::live_policy::current()) is AutonomyLevel::Full — so granting Full access takes effect immediately, no session restart or separate flag flip. Both refusal messages now point at Settings → Agent Access ("Grant Full access (or turn on App UI Control / App Automation)"). Covered by app_control_enabled_combines_optin_and_full_access + the two pinned-policy refusal tests.

Follow-up fix — Windows browser nav + the re-delegation loop. Live Windows transcript of "open my browser, go to youtube.com and play a video": the orchestrator delegated to desktop_control_agent 7×, every sub re-launch_app-ed Chrome (→ ~10 windows), and the AX path dead-ended — Chromium exposes no page content to UIA, screenshot/vision_click is unimplemented on Windows, and ax_interact set_value on the address bar can't be submitted (UIA Invoke on an Edit fails, no Enter). Two fixes:

  • open_url_in_app now works on Windows (accessibility/automate.rs) — was macOS-only (fell back to the default browser). Maps the resolved browser display name → the shell start App-Paths token (windows_browser_launch_token: Chrome→chrome, Edge→msedge, Brave→brave, Firefox→firefox; Safari/Arc→None→default handler) and runs cmd /C start "" <token> "<url>". Launches/foregrounds the named browser and navigates in one step; when the browser is already open this lands in a new tab of the existing window, so the fast-path no longer piles up windows. The browser fast-path's deterministic nav is now cross-platform (mac open -a, win start). Tested by browser_display_names_map_to_start_tokens (#[cfg(target_os="windows")]).
  • desktop_control_agent prompt steered to automate for browsers (agent_registry/agents/desktop_control_agent/prompt.md) — it had automate but its only examples were Music/Slack, so it AX-fumbled the browser. Now: web browsers → use automate{app, goal} (deterministic URL nav), never type a URL into the address bar via ax_interact; foreground each app at most once per task (repeated launch_app piles up windows); and stop after two failed attempts at a step instead of re-launching and looping. (Still a Windows gap: vision_click for the final "click first result" — screenshot is unimplemented on Windows, so play-from-search-results can't complete deterministically yet; navigation/search now do.)

Net for the browser example: open -a "Brave Browser" "https://www.youtube.com/results?search_query=music%20video" + one vision_click — instead of 6 delegations and 5 failed AX calls.

Tests (all green): app_fastpaths 58, accessibility::automate 21, computer::keyboard 26. Coverage: browser/app resolution + destination/intent parsers (incl. Spotify ↓/↑, Apple Music Win-Ctrl, Slack Alt-nav, generic-browser→None, percent-encoding); scripted-backend sequences (model-free nav, play→fall-through, media-control hotkey, dispatch order with music-wins-play); artist match→names-track / mismatch→honest; response shapes (candidate listing, wrong-artist steer, non-song honesty, song-row preference, budget/no-progress on-screen hints); a hotkey-verb loop test; #[ignore] macOS live tests. Follow-ups: wire the shortcut tables into the Phase 3 voice command_router (voice "pause"/"next" → frontmost-app shortcut); AppleScript library play-by-name+artist correction; expose vision_click as a standalone tool.

Windows port — app interaction 🪟 ✅ Implemented

Phase 1's app-interaction layer is now ported to Windows. The macOS path uses the Accessibility API via a Swift helper; the Windows path uses Microsoft UI Automation (UIA) called directly from Rust (no helper process). The agent-facing tool is a single ax_interact tool on both platforms — only the backend differs, via cfg-dispatch. The sections below were the design plan; see "Windows port — implementation status" at the end of this part for what shipped and the test evidence.

What already works cross-platform

Capability macOS (done) Windows status
Auto-send dictation transcript TS (useDictationHotkey/Conversations) ✅ Already cross-platform (frontend)
App launching launch_app / policy_command.rs ✅ Launchers (open/xdg-open/start) stay OUT of READ_ONLY_BASES (can open arbitrary URLs/handlers); app launching goes through the gated launch_app tool.
launch_app open -a ⚠️ Already has a Windows branch (Start-Process) — verify it resolves app display names
ax_interact (list/press/set_value) AXUIElement Swift helper ❌ Needs a UI Automation (UIA) backend

1. Launching apps on Windows (launch_app)

launch_app.rs already has a #[cfg(target_os = "windows")] branch using PowerShell Start-Process "<app_name>". Caveats to verify on the Windows machine:

  • Start-Process "Spotify" works for apps on PATH or registered App Paths, but Store/UWP apps (e.g. the Windows "Media Player", "Spotify" from the Store) need their AUMID: Start-Process "shell:AppsFolder\<AUMID>". Consider enumerating Store apps via Get-StartApps (returns Name + AppID) and matching by display name.
  • For URIs (e.g. spotify:, mailto:), Start-Process "<uri>" works the same as macOS open.

2. App UI interaction (ax_interact → UI Automation)

The Windows analog of macOS AXUIElement is Microsoft UI Automation (UIA) — the OS-level accessibility tree. It exposes the same concepts:

macOS AX concept Windows UIA equivalent
AXUIElement IUIAutomationElement
kAXRoleAttribute (AXButton, AXCell…) ControlType (Button, ListItem, Edit, Text…)
kAXTitleAttribute / kAXDescriptionAttribute Name property (+ AutomationId, HelpText)
AXUIElementPerformAction(kAXPressAction) InvokePattern.Invoke() (buttons) / SelectionItemPattern.Select() (list rows)
AXUIElementSetAttributeValue(kAXValueAttribute) ValuePattern.SetValue() (text fields)
AXUIElementCopyAttributeValue(kAXChildrenAttribute) TreeWalker / FindAll(TreeScope.Descendants, …)
Walk tree from app PID IUIAutomation.ElementFromHandle(hwnd) or root + ProcessId condition

Recommended implementation path (Rust-native, no helper process needed):

  • Use the uiautomation crate (safe Rust bindings over the UIA COM API). This is cleaner than macOS, where we had to shell out to a Swift helper — on Windows the COM API is callable directly from Rust.
  • Mirror the existing accessibility::ax_interact surface so the tool stays identical:
    • list(app, filter) → CreateTreeWalker over the app's window, collect elements whose Name matches filter, return [{control_type, name}].
    • press(app, label) → find element by Name (exact-first), then call InvokePattern if supported, else SelectionItemPattern.Select(), else LegacyIAccessiblePattern.DoDefaultAction().
    • set_value(app, label, value) → find Edit/ComboBox, call ValuePattern.SetValue().
  • Key win over macOS: UIA Invoke is generally a real "activate" (it triggers the control's default action), so the navigate-then-activate two-step that plagued Apple Music is less likely. A list-item Invoke on most Windows media apps plays directly. Still expect per-app quirks.

Suggested module layout (parallel to macOS):

src/openhuman/accessibility/
  ax_interact.rs          # macOS (existing)
  uia_interact.rs         # NEW — Windows UIA backend, same fn signatures
  mod.rs                  # cfg-dispatch: pub use the right backend per-OS

Then tools/impl/computer/ax_interact.rs calls a thin cfg-gated facade so the agent-facing tool is one tool on both platforms (same name ax_interact, same actions). Update its description to be OS-neutral ("uses the platform accessibility API").

3. Permissions

  • macOS needs the Accessibility permission. Windows UIA needs no special permission for same-session, same-integrity-level apps — a big simplification. Caveat: a non-elevated process can't drive an elevated app's UI (UIPI). If the agent must control an elevated app, OpenHuman would need to run elevated too (avoid unless necessary).

4. Diagnostics

Keep the same [ax_interact]/[uia_interact] log prefixes and the verbose step logging (▶ action, found-count, press result) — they were essential for diagnosing the macOS flow and will be just as useful on Windows.

5. Testing

Port the integration tests using a built-in Windows app that's always present and UIA-friendly:

  • Calculator (calc.exe) — press digit/operator buttons by Name, read the result Text element. Deterministic, no network, ideal smoke test.
  • Notepad — set_value into the Edit control, verify via ValuePattern.Value.
  • Media: Windows Media Player or the Store Media Player for a play test, but expect the same nondeterminism caveat as Apple Music — assert tool-level success, log playback as best-effort.

6. Known-different behaviors to watch for

  • Element naming: Windows apps often populate AutomationId (stable) where macOS only had a visible title. Prefer matching Name, fall back to AutomationId.
  • Chromium/Electron apps (Slack, Discord, VS Code, Spotify desktop): on Windows these expose a partial UIA tree by default; some require the app to have accessibility enabled. Same class of limitation as the macOS chromiumAppPatterns special-casing already in helper.rs.
  • Focus/foreground: UIA generally doesn't require foregrounding to read/invoke, like macOS AX. No CGEventPost-style CEF crash risk because UIA Invoke is not synthetic input injection.

Quick start for the Windows machine

  1. launch_app should already work — test "open notepad" / "open calculator" first.
  2. Do NOT add launchers to READ_ONLY_BASES — launch_app (gated, URI-rejecting) is the Windows app-launch path. Start-Process lives inside that tool, not the shell allowlist.
  3. Build uia_interact.rs against the uiautomation crate, mirroring the three ax_interact fns.
  4. cfg-dispatch in accessibility/mod.rs so ax_interact the tool resolves to UIA on Windows.
  5. Smoke-test with Calculator (deterministic), then a media app (best-effort).

Cross-platform compatibility audit (current state)

Every Phase 1 change was written to compile on all platforms — nothing here breaks the Windows build. macOS-specific native code is #[cfg(target_os = "macos")]-gated and the non-macOS branches return a clean "…macOS-only" error at runtime rather than failing to build.

Change Cross-platform status Notes for Windows
Auto-send dictation transcript (TS) ✅ Fully portable Pure frontend; no OS code. Works as-is.
launch_app ✅ macOS / Linux / Windows branches Windows branch now uses Start-Process with a Store/UWP (Get-StartApps AUMID) fallback and injection-safe env passing (§1).
ax_interact tool (tools/impl/computer/ax_interact.rs) ✅ Functional on Windows Delegates to accessibility::ax_interact fns, which now cfg-dispatch to the UIA backend on Windows. Description made OS-neutral.
accessibility::ax_interact helpers ✅ cfg-dispatched macOS → Swift helper; Windows → uia_interact.rs; other → clean runtime error.
accessibility::uia_interact (NEW) ✅ Windows backend UIA list/press/set_value via the uiautomation crate; same fn signatures as the macOS path.
Swift unified helper (accessibility/helper.rs) ⚠️ macOS-only by design Windows needs no helper process — UIA COM API is called directly from Rust.
App launching ✅ Done Launchers stay out of READ_ONLY_BASES; launch_app (gated) handles Windows Start-Process.
Notch indicator (separate PR #3166) ⚠️ macOS NSPanel A Windows equivalent would be a borderless always-on-top WebView2 window or a tray flyout — out of scope for this branch.

Before merging the Windows port, confirm the whole branch still builds and runs on macOS too (cargo check on both Cargo.toml and app/src-tauri/Cargo.toml) so the cfg-dispatch doesn't regress the working macOS path.

⚠️ Mandatory: extensive E2E testing on Windows

The macOS path was hardened only through repeated real-app runs (each bug — CEF crash, select-vs-play, list truncation/hallucination — surfaced only by actually driving live apps, not by unit tests). Do the same on Windows before considering it done. Treat the following as the required E2E matrix:

  1. App launch — launch_app for: a Win32 app (Notepad), a Store/UWP app (Media Player / Spotify from Store), and a URI (spotify:). Confirm each actually opens.
  2. Deterministic UI control — Calculator: list filter='5' → press '5', press '+', press '=', then read the result element. Assert the computed value. This is the Windows analog of the AC/DC test and should be a hard-asserted automated test (Calculator is deterministic).
  3. Text entry — Notepad: set_value into the Edit control, verify via ValuePattern.Value.
  4. Filtered list correctness — confirm list with a filter returns a small, accurate set (the truncation→hallucination bug must not recur; verify the 60-element cap + filter behaves on a busy app like Settings or a browser).
  5. Real-world app — drive a media app end-to-end (open → search → play). Assert tool-level success; treat playback state as best-effort (same nondeterminism caveat as Apple Music).
  6. Chromium/Electron app — Slack/Discord/VS Code: confirm whether their UIA tree is exposed; document any app that needs accessibility explicitly enabled.
  7. Permissions/elevation — verify behavior against a normal app vs an elevated one (UIPI boundary); document what fails and why.
  8. Agent-in-the-loop run — the real test: ask the running agent (chat) to perform each action and confirm it picks launch_app / ax_interact and the action lands. Watch [ax_interact]/[launch_app] logs exactly as we did on macOS.
  9. Regression — re-run the macOS E2E suite after the Windows changes land to prove cfg-dispatch didn't break the Mac path.

Add the deterministic ones (Calculator, Notepad, launch) as #[cfg(target_os = "windows")] #[ignore] integration tests mirroring ax_interact_tests.rs, runnable with cargo test ... -- --include-ignored on the Windows machine.

Windows port — implementation status ✅

Shipped on the Windows machine (2026-06-02):

Code

  • Cargo.toml — uiautomation = "0.25" under [target.'cfg(windows)'.dependencies]; Win32_System_Com feature added to windows-sys for COM init.
  • src/openhuman/accessibility/uia_interact.rs (new) — UIA backend. list / press / set_value over the UIA COM tree via the uiautomation crate, mirroring the macOS ax_interact fn signatures. press activates via UIA patterns in order — Invoke → SelectionItem.Select → LegacyIAccessible.DoDefaultAction — never injecting synthetic input. set_value finds an editable field, preferring Edit, then ComboBox, then Document (so the Win11 RichEdit Notepad, whose editor is a Document, works). Exact-label match preferred over substring. Per-thread COM init via CoInitializeEx(MTA).
  • src/openhuman/accessibility/ax_interact.rs — the three public helpers now cfg-dispatch: macOS → Swift helper, Windows → uia_interact, else → clean runtime error. Module + tool docs made OS-neutral.
  • src/openhuman/accessibility/mod.rs — declares uia_interact (cfg-gated to Windows).
  • src/openhuman/tools/impl/computer/ax_interact.rs — description rewritten to be platform-neutral ("platform accessibility API (macOS AXUIElement / Windows UI Automation)").
  • src/openhuman/tools/impl/system/launch_app.rs — Windows launcher hardened: app name passed via env var (no string interpolation → no injection), Start-Process first, then Store/UWP fallback by display name via Get-StartApps → AUMID (shell:AppsFolder\<AppID>); stderr surfaced on failure.
  • src/openhuman/security/policy_command.rs — launchers (open/xdg-open/start) deliberately kept OUT of READ_ONLY_BASES; launch_app is the gated launch path.
  • src/openhuman/accessibility/uia_interact_tests.rs (new) — #[cfg(all(test, target_os = "windows"))] integration tests, wired into ax_interact.rs.

Test evidence (real apps on Windows 11)

  • test_uia_calculator_five_plus_five ✅ — drove the live Calculator entirely by element label: list → 41 interactive elements; pressed Five/Plus/Five/Equals; hard-asserted the readout [Text] "Display is 10" (5 + 5 = 10). Deterministic — the Windows analogue of the macOS AC/DC test.
  • test_uia_notepad_set_value ✅ — set_value wrote into the live Win11 Notepad's Document "Text editor" (Ok("Set 'Text editor' in 'Notepad' to the provided value.")). The Document fallback is what makes the redesigned Notepad work.
  • test_uia_list_nonexistent_app ✅ (non-ignored) — exercises COM init + window walk + error path deterministically.
  • launch_app (×8) and ax_interact tool (×4) unit tests ✅.
  • Full cargo test --lib --no-run compiles clean on Windows (warnings only, all pre-existing).

Environment gotcha (this machine): Norton real-time protection blocks link.exe from writing the freshly-linked ~150 MB test .exe (LNK1104, "Access denied" creating the file). Fix: exclude the repo's target dir under Norton's Auto-Protect / SONAR / Download Intelligence exclusion list (not the separate "Scans" list), and restore the file from Quarantine if already flagged.

Follow-ups / not done here

  • macOS regression check — the cfg-dispatch is additive (the #[cfg(target_os="macos")] arms are untouched; only the non-macOS catch-all message changed), but per the branch note, re-run cargo check + the macOS E2E suite on a Mac before merge to prove the Mac path didn't regress (can't be done from the Windows machine).
  • Agent-in-the-loop run (E2E item 8) — the full Tauri desktop app was built and run on Windows (pnpm dev:app:win) and the in-process core booted fine (verbose [launch_app]/[ax_interact]/[uia_interact] logging wired via RUST_LOG). The first chat attempt couldn't complete because the configured local AI model was still downloading (kind="empty_provider_response" — the agent returned an empty response, so it never reached a tool call). Still pending: a working model (finish the local download or select a configured cloud model), then ask the agent "open Calculator" / "press 5 in Calculator" and confirm it picks launch_app/ax_interact and the action lands.
  • Chromium/Electron UIA coverage, elevation/UIPI behavior, and a busy-app filtered-list check (E2E items 4/6/7) remain to be spot-checked manually.

Phase 2 — Always-On Listening ✅ Implemented

Continuous microphone listening without requiring a hotkey press.

Shipped:

  • src/openhuman/voice/always_on.rs — pure VadSegmenter (onset / silence-hangover / min-speech / max-utterance, 7 unit tests) plus the continuous capture loop: a dedicated cpal thread streams 16 kHz mono frames → segmenter → each utterance is encoded (encode_wav_16k) → voice_transcribe_bytes → publish_transcription (so it reaches the agent's auto-send and the notch, exactly like hotkey dictation). Started at boot in credentials::ops.
  • src/openhuman/config/schema/voice_server.rs — always_on_enabled flag + VAD tuning (vad_onset_threshold, vad_hangover_ms, vad_min_speech_ms, vad_max_utterance_secs), opt-in/off by default.
  • Settings toggle — "Always-on listening" in the Voice debug panel, wired through get/update_voice_server_settings (RPC patch → apply → snapshot); i18n in en + all 13 locales.
  • Privacy hook — spawn_lock_watcher pauses capture + resets the segmenter while the screen is locked (macOS via CGSessionCopyCurrentDictionary, null/type-safe FFI; other platforms never pause yet).
  • Reused audio_capture helpers (to_mono/resample/chunk_rms made pub(crate) + new encode_wav_16k).

Acceptance criteria:

  • User can speak without pressing any hotkey
  • VAD detects end of utterance and sends to agent
  • Toggle in Settings → Voice

Wake word "Hey Tiny" (live-fix, 2026-06-03): always-on now only delivers an utterance to the agent when its transcript contains the wake word (config.voice_server.wake_word, default "Hey Tiny"); the phrase is stripped and the remainder is sent. Tolerant match (case/punctuation/leading-filler), empty wake word = deliver everything. This is a text-based wake word (transcribe-then-gate) — a first cut of Phase 3's trigger phrase; it fixes the "sends every utterance" spam but still runs STT on all speech (an on-device audio wake-word model for efficiency is the Phase 3 follow-up).

Live-fixes found by running it:

  • Toggle did nothing — always_on_enabled wasn't in the update_voice_server_settings RPC param schema, so validation rejected it before the handler. Added it; the config RPC now also calls always_on::start_if_enabled so the toggle starts/idles capture live (runtime ENABLED gate, no restart).
  • transcription failed: local ai is disabled — always-on used voice_transcribe_bytes (local whisper only). Now routes through effective_stt_provider + create_stt_provider (same factory dispatch as voice.stt_dispatch), honoring cloud STT.
  • Toggle surfaced in the reachable VoicePanel (Settings → Advanced → AI → Voice), not the hidden debug panel.

Pending live validation (mic-dependent, can't be CI-tested): say "Hey Tiny, " and confirm the command reaches the agent; tune vad_onset_threshold/vad_hangover_ms to the user's mic + room. Windows/Linux screen-lock pause is a follow-up (no signal wired).

Phase 3 — Wake-Word + Fast Routing 🔨 Fast routing shipped; audio wake-word model pending

Fast command routing is merged (#3346). The single remaining Phase 3 item is the on-device audio wake-word model — not started; the text-based "Hey Tiny" transcribe-then-gate match from Phase 2 is the interim.

Fast routing — DONE & WIRED: src/openhuman/voice/command_router.rs (new) — pure route(transcript) -> VoiceIntent classifier (Play/Pause/Resume/Next/Previous/OpenApp/SetVolume/VolumeUp·Down/Mute/Unmute, else Unknown). High-confidence intents execute directly (launch_app / Music fast-path / osascript volume) without a full chat-LLM turn — the ≤500 ms path; Unknown defers to the agent so routing only ever shortcuts. Filler-tolerant ("please open up slack"). 5 unit tests.

Router wired into delivery: always_on::deliver_command now routes the extracted command through command_router::route first. Recognized intents run locally via execute_intent (Play → automate Music fast-path; OpenApp → launch_platform; Pause/Resume/Next/Previous/volume/mute → osa osascript); VoiceIntent::Unknown (and any local-exec failure) falls back to the full agent turn. Verified live: clean transport/open/volume commands execute on the fast path; complex/garbled play queries (e.g. mangled STT like "bts s latest song wind blowing") fall through to the agent as designed. Committed on feat/voice-phase3-fast-routing.

Remaining Phase 3: on-device audio wake-word model (Porcupine / ONNX) in inference/voice/wake_word.rs to gate STT before transcription — the text-based "Hey Tiny" wake match from Phase 2 (fuzzy anchor on the "tiny" token, Levenshtein ≤1) is the interim. This is the only open Phase 3 item; not yet started.

Phase 3 — Wake-Word + Fast Routing (original plan)

Activate only on a trigger phrase; route simple commands locally without a full LLM turn.

Planned files:

  • src/openhuman/inference/voice/wake_word.rs (new) — lightweight always-on model (Porcupine or custom ONNX) — ⏳ not started (interim: text-based "Hey Tiny" match in Phase 2)
  • src/openhuman/voice/command_router.rs (new) — intent→tool mapping for high-confidence commands, LLM fallback for ambiguous input — ✅ shipped & wired (see Phase 3 status above)

Acceptance criteria:

  • Wake-word detection runs fully on-device (still pending — audio wake-word model not yet built)
  • Latency from end-of-utterance to action start ≤ 500ms for local-routed commands (command router executes recognized intents on the local fast path)

Phase 4 — Polish ⏳ Not Started

Voice confirmation loop, UI indicator, computer control onboarding.

Planned:

  • TTS confirmation before executing sensitive actions ("Opening Music — confirm?")
  • Always-on status indicator (notch pill from PR #3166 will handle this automatically)
  • Computer control (mouse/keyboard tools) toggle in Settings onboarding

Fine-tuning backlog ⏳ (deferred until all phases complete)

From live agent-in-the-loop testing on 2026-06-03 (grounded in ~/.openhuman/logs/openhuman.2026-06-03.log, session_raw/*.jsonl, and the dev run: keyboard=69 / mouse=0 / screenshot=10 tool calls; 26 wake matches vs 93 misses; emit=true utterances ranged 0.7s–28s). The feature works but needs tuning. Do not implement until Phases 3–4 land.

F1 — Listening window too short for long commands

  • Observed: vad_hangover_ms = 800 closes an utterance on any pause > 0.8s, so multi-clause commands ("Hey Tiny, open Slack and message the team channel saying …") split across utterances — the tail lacks the wake word and is dropped. Compounded by the notch "Listening" pill TTL (2500ms) expiring mid-speech, so it looks like it stopped listening.
  • Resolve: (a) raise vad_hangover_ms to ~1500ms; (b) two-stage capture — once the wake word is detected, open a dedicated longer command window (until a longer silence / N-second cap) instead of relying on a single VAD utterance; (c) keep the "Listening" pill alive for the whole utterance (extend/re-emit on each voiced frame, clear on SpeechEnd) so the notch reflects real mic state.

F2 — Agent uses keyboard only, never the mouse

  • Observed: keyboard=69, mouse=0. Two causes: the orchestrator prompt is deliberately keyboard-first, and the downscaled screenshot's coordinates don't map to screen pixels — the capture is shrunk to ≤1568px while mouse expects absolute screen pixels (and Retina is 2× points), so any coordinate read from the image clicks the wrong spot. Vision-driven clicking is therefore currently unsafe and the agent (correctly) avoids it.
  • Resolve: (a) make screenshot emit a coordinate transform (shown WxH + real screen WxH + backing scale) or have mouse accept image-relative coordinates and convert internally; (b) once coordinates are trustworthy, soften the prompt so the agent uses screenshot→mouse to click specific elements, not just keyboard.

F3 — No periodic screenshot/verify + foreground re-check

  • Observed: the agent screenshots ad-hoc (0 in the last session); automate only foregrounds at the start.
  • Resolve: in the automate loop and the orchestrator prompt — screenshot + verify at start, after every ~3 actions, and at the end; before each action confirm the frontmost app is the target and re-launch_app (foreground) it if not, then proceed. Fold the actual-vs-expected check into the loop's verify step.

Permissions matrix — what the desktop agent needs to run

Grounded in src/openhuman/accessibility/permissions.rs (detection), src/openhuman/accessibility/types.rs (PermissionKind), app/src-tauri/Info.plist (usage strings), and app/src-tauri/entitlements.sidecar.plist (Hardened-Runtime entitlements). Use this as the pre-test checklist when building for macOS (Apple Silicon + Intel) and Windows.

The four OS permissions the app tracks

detect_permissions() returns a PermissionStatus over these four PermissionKinds:

Permission Detected via (macOS) Agent capability that needs it
Microphone CPAL input-device probe (detect_microphone_permission) All voice capture — dictation hotkey and always-on listening (voice/always_on.rs mic stream). Required for "voice on" to capture anything.
Accessibility AXIsProcessTrusted() The whole app-control surface: ax_interact (AX tree read/press), automate, autocomplete focus query, and synthetic keyboard/mouse injection (enigo / CGEvent).
Screen Recording CGPreflightScreenCaptureAccess() screenshot → vision_click (vision fallback for Electron/partial-AX apps) and screen-intelligence capture.
Input Monitoring IOHIDCheckAccess(LISTEN_EVENT) Global hotkey listening (dictation hotkey / rdev / Globe-key listener — accessibility/keys.rs, globe.rs). Not needed for always-on (no hotkey).

Supporting macOS entitlements + usage strings (Hardened Runtime)

These are baked into the signed build, not runtime toggles. If missing, the OS blocks the call before any consent dialog renders — so they must be correct in the signed .app/DMG, and cannot be validated with tauri dev alone.

  • Info.plist usage strings: NSMicrophoneUsageDescription, NSCameraUsageDescription, NSAppleEventsUsageDescription (+ Bluetooth / Location / Contacts / Calendar / folder strings — those are mostly for the embedded provider webviews, not the agent itself).
  • entitlements.sidecar.plist: device.audio-input, device.camera, automation.apple-events (drives osascript / System Events — Music transport, volume, foreground detection), network.client / network.server.

Capability → permission map (agent surfaces)

Capability Microphone Accessibility Screen Recording Apple Events / Automation Input Monitoring
Always-on capture (mic → VAD → STT) ✅ — — — —
Dictation hotkey ✅ — — — ✅ (listen)
launch_app / OpenApp intent — — — — —
Pause/Next/volume intents (osascript) — — — ✅ —
Play (Music fast-path) — ✅ — ✅ (verify player state) —
ax_interact (list/press/set_value) — ✅ — — —
automate general loop — ✅ ✅ (only for vision_click) ✅ (where it uses osascript) —
vision_click (Electron fallback) — ✅ ✅ — —
keyboard / mouse (synthetic input) — ✅ — — —

Minimal "voice on" path: Microphone to capture; then per intent — OpenApp needs nothing extra, transport/volume need Automation, Play needs Accessibility + Automation.

Per-OS differences (the three test machines)

macOS (M-chip & Intel — identical) Windows Linux
Microphone Real TCC prompt; does prompt Real privacy gate — Settings → Privacy → Microphone Ungated on standard desktops; Flatpak needs an XDG portal
Accessibility Manual grant in System Settings → Privacy & Security; does not reliably auto-prompt; usually needs an app restart after granting No analog — UIA needs no permission for same-integrity apps; cannot drive elevated apps (UIPI) App-interaction effectively unsupported (backend is macOS Swift helper / Windows UIA only; Linux returns a clean runtime error)
Screen Recording Manual grant; no reliable auto-prompt screenshot/vision_click is currently unimplemented on Windows — the vision fallback can't run there yet Unsupported
Input Monitoring Manual grant for hotkey listening Not required Not required
Entitlements Must be present in the signed build n/a n/a

Gotchas to flag before testing

  1. Detection ≠ entitlement. On macOS the runtime checks report Denied/Unknown if the entitlement is missing from the signed build, independent of what the user clicked — so test on a properly signed build, not tauri dev.
  2. Microphone detection is a proxy. detect_microphone_permission infers from CPAL device enumeration, so "no device connected" and "permission denied" both surface as Unknown on macOS/Windows (permissions.rs). The always-on capture logging ([voice::always_on] microphone permission: … + device name + first-chunk confirmation) disambiguates this live.
  3. Restart after granting Accessibility / Screen Recording / Input Monitoring on macOS — TCC changes are not always picked up by a running process.

Summary

Phase Item Status
1 Auto-send after transcription ✅ Done
1 Shell allowlist for open/xdg-open ✅ Done
1 Shell tool description clarification ✅ Done
1 Dedicated launch_app tool ✅ Done
1 Orchestrator tool scope ✅ Done
1 SOUL.md capability hint ✅ Done
1 Diagnostic logging ✅ Done
1 Computer control (mouse/keyboard) ❌ Reverted (CEF crash)
1 AXUIElement app UI interaction (ax_interact) ✅ Done
1 Multi-step UI workflow guidance ✅ Done
1 Apple Music two-step play (navigate→play) ✅ Done (playback best-effort)
1 Full computer control (mouse/keyboard/screenshot) — CEF crash (Change 1.15) ✅ Fixed (main-thread synthetic-input dispatch; screenshot downscale)
1 Windows port (launch_app + UIA ax_interact) ✅ Done (Calculator + Notepad hard-asserted on Win11)
1 automate(app, goal) Rust-driven loop (Change 1.14) ✅ M1–M5 done (proven live on macOS)
1.5 M1: automate loop skeleton + tool ✅ Done
1.5 M2: poll-until-stable settle ✅ Done
1.5 M3: Music fast-path ✅ Done (proven live on macOS)
1.5 Robustness: quoted-query parse + no-progress guard ✅ Done (from live agent failures)
1.5 M4: progress streaming to notch ✅ Done — notch cherry-picked in; automate streams live steps
1.5 M5: richer element model (enabled) ✅ Plumbed; AXEnabled found unreliable → informational only
1.5 Native fast-paths beyond Music (Spotify/Slack) ➖ Descoped (general loop covers them; Music shipped in M3)
1.5 Vision fallback for Electron apps (Change 1.16) ✅ Done (vision_click: screenshot → vision-locate → guarded click; frontmost guard; F2 coord-transform folded in)
1.5 Browser fast-path (Change 1.17a) ✅ Done (browser.rs: deterministic open -a "<browser>" url nav; play→fall-through to vision_click)
1.5 Cross-platform browser shortcut table (Change 1.17b) ✅ Done (browser_shortcuts.rs: Chrome/Firefox/Safari/Edge × macOS/Win/Linux)
1.5 App-shortcut fast-path: Spotify/Apple Music/Slack (Change 1.17c) ✅ Done (app_shortcuts.rs: transport + nav via each app's shortcuts)
1.5 Keyboard-shortcut plumbing: hotkey verb + backend methods (Change 1.17d) ✅ Done (key/type_text/open_url_in_app/now_playing; shared keyboard helpers)
1.5 Artist-aware Music verification (Change 1.17e) ✅ Done (now_playing osascript; honest wrong-artist report, no false success)
1.5 Actionable automate responses (Change 1.17f) ✅ Done (next-step guidance, candidate/label lists, verified-only success, song-row preference)
2 Always-on microphone loop ✅ Done (cpal → VAD → STT → agent)
2 always_on_enabled config flag + Settings toggle ✅ Done (RPC + UI + i18n)
2 Privacy hook (screen lock pause) ✅ Done (macOS; other OSes follow-up)
2 Text-based "Hey Tiny" wake word ✅ Done (interim; gates delivery, strips phrase)
3 Local command router (intent classifier) ✅ Done & wired (recognized intents run on the ≤500ms local path; Unknown defers to agent)
3 On-device audio wake-word model ⏳ Not started (text-based match is the interim)
4 Voice confirmation loop ⏳ Not started
4 Computer-control onboarding toggle ⏳ Not started
4 Always-on UI indicator ✅ Done (notch PR #3166)
Reference in New Issue View Git Blame Copy Permalink