diff --git a/docs/voice-system-actions.md b/docs/voice-system-actions.md index f5ffa593e..9c098d671 100644 --- a/docs/voice-system-actions.md +++ b/docs/voice-system-actions.md @@ -1,9 +1,9 @@ # Voice β†’ System Action Feature Tracker **GitHub Issue:** [#3148](https://github.com/tinyhumansai/openhuman/issues/3148) -**Branch:** `feat/voice-always-on` -**PR:** [#3168](https://github.com/tinyhumansai/openhuman/pull/3168) +**Branch:** `feat/voice-always-on-all` (cumulative) β€” Phase 1 landed via [#3168](https://github.com/tinyhumansai/openhuman/pull/3168); the full feature was split from the mega-PR [#3307](https://github.com/tinyhumansai/openhuman/pull/3307) into an 8-PR stack [#3340–#3346](https://github.com/tinyhumansai/openhuman/pull/3340) + [#3362](https://github.com/tinyhumansai/openhuman/pull/3362) (Phase 1.5 vision fallback) **Started:** 2026-06-02 +**Last updated:** 2026-06-04 --- @@ -268,9 +268,9 @@ test ... ok --- -### Change 1.14 β€” `automate(app, goal)`: Rust-driven multi-step automation πŸ”¨ In progress (M1 done) +### Change 1.14 β€” `automate(app, goal)`: Rust-driven multi-step automation βœ… M1–M5 done -**Status:** πŸ”¨ In progress β€” **M1 + M2 + M3 shipped and M3 proven live on macOS**; M4–M6 pending. See **Phase 1.5** below and [`voice-automate-plan.md`](voice-automate-plan.md). +**Status:** βœ… **M1 + M2 + M3 + M4 + M5 shipped; M3 proven live on macOS.** The only remaining `automate` work is the **vision fallback** for Electron/partial-AX apps β€” tracked under **Phase 1.5** below and [`voice-automate-plan.md`](voice-automate-plan.md). (Per-app native fast-paths beyond the shipped Music one are **descoped**.) **Agent-in-the-loop fixes (2026-06-03, from two live chat sessions):** - **Mutations were off** β€” the agent correctly called `automate` but it (and `ax_interact`) refused because `computer_control.ax_interact_mutations=false`. Enabled it; also rewrote both refusal messages to point at **Settings β†’ Agent Access** instead of a config key (the agent had relayed "controls are locked down"). @@ -315,9 +315,9 @@ test ... ok - **A fast model drives the inner loop** (Haiku-class) with a *tiny* context: just the goal, the current small AX snapshot, and the last result β€” not the whole conversation. Each inner step is ~0.5–1 s and self-corrects, instead of one 3 s chat turn that falsely reports success. - **Settle + verify in Rust** between steps β€” deterministic, kills the timing-race class in one place. - **Native fast-paths for high-value apps** (skip the UI entirely where possible): - - **Music** β€” `music://` search URL β†’ AX play (already explored in 1.11), or AppleScript for library. - - **Spotify** β€” Web API search β†’ `spotify:track:…` URI + AppleScript `play`. Fully deterministic, no UI poking. - - **Slack** β€” deep link `slack://channel?…` to open the DM, then AX to type + send. + - **Music** β€” `music://` search URL β†’ AX play (already explored in 1.11), or AppleScript for library. **(Shipped β€” M3.)** + - ~~**Spotify** β€” Web API search β†’ `spotify:track:…` URI + AppleScript `play`.~~ **Descoped** β€” the general loop covers it; not worth the per-app maintenance now. + - ~~**Slack** β€” deep link `slack://channel?…` to open the DM, then AX to type + send.~~ **Descoped** β€” handled by the general loop / vision fallback. The general AX loop is the fallback for everything else. - **Vision fallback for Electron/Chromium apps** (Slack, Discord, VS Code, Spotify-desktop) whose AX/UIA tree is partial (documented limitation). Slack needs accessibility enabled (`defaults write com.tinyspeck.slackmacgap AccessibilityEnabled -bool true`, relaunch). Where AX returns empty, fall back to **screenshot β†’ vision-locate β†’ guarded click**. This is the reverted CGEventPost path (1.8) β€” but it crashed only when events hit *OpenHuman's own focused CEF window*; a guarded click into a *different, foregrounded* app does not have that failure mode. - **Stream progress events** to the UI / notch pill (PR #3166) so the user sees each step happen live. @@ -326,26 +326,27 @@ test ... ok --- -## Phase 1.5 β€” Reliable, real-time multi-step automation ⏳ Not Started +## Phase 1.5 β€” Reliable, real-time multi-step automation βœ… Complete > The bridge between today's `ax_interact` primitives and the always-on voice work. **Prerequisite for Phase 3** β€” fast voice routing into a slow/fragile action loop still feels slow. This is where "whatever I say happens, live" actually gets delivered. +> +> **Status:** the Rust inner loop (M1), poll-until-stable settle (M2), Music fast-path (M3, proven live), notch progress streaming (M4), the richer element model (M5), and the **vision fallback for Electron/partial-AX apps** (Change 1.16) are all shipped. (Additional per-app native fast-paths beyond Music β€” e.g. Spotify/Slack β€” are **descoped**: the general model-driven loop covers them, so a deterministic accelerator isn't worth the per-app maintenance right now.) **Detailed implementation plan:** [`voice-automate-plan.md`](voice-automate-plan.md) β€” decided approach: **Rust inner loop + fast model**, first proof target **Music**. **Planned files:** - `src/openhuman/accessibility/automate.rs` (new) β€” the perceiveβ†’actβ†’verify loop + settle/verify primitives, reusing `ax_interact` helpers. -- `src/openhuman/accessibility/app_fastpaths/` (new) β€” per-app deterministic paths (`music.rs`, `spotify.rs`, `slack.rs`), behind a generic dispatch. +- `src/openhuman/accessibility/app_fastpaths/` (new) β€” per-app deterministic paths behind a generic dispatch. **Shipped:** `music.rs` (M3). Further per-app fast-paths (Spotify/Slack) are **descoped** β€” the general loop handles them. - `src/openhuman/tools/impl/computer/automate.rs` (new) β€” `AutomateTool { app, goal }`, gated like `ax_interact` (mutations opt-in, sensitive-app denylist reused). - macOS helper (`accessibility/helper.rs`) β€” AXObserver-based settle (`ax_wait_settled`) + post-action verify; richer element model (enabled/onscreen/actions). - Vision fallback β€” screenshot via `accessibility/capture.rs` β†’ locate β†’ guarded click (only when AX tree is empty, target app foregrounded, never OpenHuman's own window). **Acceptance criteria:** -- [ ] One `automate{app, goal}` call performs a multi-step flow end-to-end (no per-step chat turns) -- [ ] Settle/verify removes the timing-race + false-success failure classes (1.11/1.13 do not recur) -- [ ] Music flow ("play ") works end-to-end via the inner loop -- [ ] Spotify + Slack fast-paths land their action deterministically -- [ ] Electron/partial-AX apps fall back to vision+guarded-click without the CEF crash -- [ ] Step-by-step progress streamed to the UI / notch indicator +- [x] One `automate{app, goal}` call performs a multi-step flow end-to-end (no per-step chat turns) +- [x] Settle/verify removes the timing-race + false-success failure classes (1.11/1.13 do not recur) +- [x] Music flow ("play ") works end-to-end via the inner loop (proven live, hard-asserts `player state == playing`) +- [x] Electron/partial-AX apps fall back to vision+guarded-click without the CEF crash (Change 1.16) +- [x] Step-by-step progress streamed to the UI / notch indicator (M4) --- @@ -368,12 +369,29 @@ enigo::macos::get_layoutdependent_keycode β†’ TSMGetInputSourceProperty ``` enigo's keyboard-layout lookup (`TSMGetInputSourceProperty`) **must run on the app's main thread**; the keyboard tool ran on a tokio worker β†’ macOS trapped. **Not** a focus issue (same Β§1.8 root cause); a frontmost-app guard would not have fixed it. -**Fix applied:** enigo now runs on the Tauri **main thread** via `AppHandle::run_on_main_thread`, bridged to the core through a native-registry handler (see *The fix* above) and wrapped in `catch_unwind` so an FFI panic can't unwind across the main thread. (Alternative considered but not needed: TSM-free primitives β€” `CGEventKeyboardSetUnicodeString` for text + raw virtual keycodes for keys/hotkeys.) +**Fix applied:** all enigo operations now run on the Tauri **main thread** via `run_input_on_main` (new `main_thread.rs` module + a native-registry handler the shell registers, dispatched through `AppHandle::run_on_main_thread`) and wrapped in `catch_unwind`. Keyboard/mouse tools **and** `vision_click` now execute without TSM traps. **Tests:** voice-actions + autonomy suite is exhaustive β€” 220 feature unit tests + a JSON-RPC E2E (`json_rpc_voice_server_settings_roundtrip_always_on_and_wake_word`). The E2E caught + fixed real gaps (`wake_word` missing from the get output and the update RPC path). Screenshot downscale unit-tested. --- +### Change 1.16 β€” `vision_click`: vision fallback for Electron/partial-AX apps βœ… Done + +**Status:** βœ… Shipped β€” closes the last open Phase 1.5 item. Electron/Chromium apps (Slack, Discord, VS Code) expose little or no AX/UIA tree, so the `automate` perceiveβ†’press loop had nothing to act on. The loop can now *see* the screen and click a described element. + +**Problem:** when `perceive` returns an empty (or target-missing) element list, the loop was stuck β€” there was no label to press. The planned answer (tracker Β§1.5) was *screenshot β†’ vision-locate β†’ guarded click*, but two things blocked it: (1) the fast inner-loop model is text-only, and (2) the deferred **F2** coordinate-mapping gap β€” the screenshot is windowed + downscaled and Retina is 2Γ—, while `mouse` expects absolute screen points, so a vision-returned coordinate would click the wrong spot. + +**Fix β€” a new model-chosen `vision_click { description }` action in the `automate` loop:** +- **Reuses the existing multimodal path** β€” the OpenAI-compatible provider promotes an embedded `[IMAGE:]` marker into a real `image_url` part (`inference/provider/compatible_types.rs`, #3205), so vision-locate rides the existing `chat_with_system` call with **no new inference API**. The locate call uses the **main `chat`/vision provider** (`create_chat_provider("chat", …)`) β€” reliable UI grounding, and the fallback only fires when AX is empty (rare). +- **Folds in the F2 coordinate transform** β€” `src/openhuman/accessibility/vision_click.rs` (new): `CaptureGeometry` (window screen-rect in points + image pixel size) + a **pure, unit-tested** `image_to_screen(geom, px, py)`. The pxβ†’pt ratio absorbs both the downscale and the Retina backing scale, so no explicit scale factor is needed; the result is clamped strictly inside the target window. +- **Β§1.8 safety guard** β€” a `vision_click` only fires when the target app is **frontmost** (`focus::foreground_context`); on positive evidence a different app is focused it re-foregrounds once and otherwise **refuses**, so synthetic input never lands on OpenHuman's own CEF window. `None` (can't tell) = best-effort, since the loop already foregrounded the app at start. +- **Main-thread click** β€” the guarded left-click runs via `run_input_on_main` (Change 1.15), so off-thread enigo never traps TSM. +- Wired into the `automate` loop: new `Action.description`, a `vision_click` system-prompt verb ("use when the element list is EMPTY β€” Electron apps"), the no-progress signature extended with `description`, and `RealBackend::{screenshot,locate,frontmost_app,click}`. Inherits `automate`'s existing gating (Dangerous + mutations opt-in + sensitive-app denylist) β€” no new tool, no new approval surface. + +**Tests:** pure `image_to_screen` (downscale / Retina 2Γ— / origin offset / out-of-range + negative clamp / zero-dim safety), `parse_locate_response` (found / not-found / fenced / prose / garbage), `build_locate_user` (marker), `image_dims_from_data_uri` (real PNG round-trip); loop integration via the scripted backend (locate-and-click when frontmost, proceed when frontmost unknown, **refuse when another app is frontmost**, no-click on not-found, empty-description skip). A macOS live run against a real Electron app is the remaining manual validation (vision is nondeterministic β€” assert tool-level success, treat the visual outcome as best-effort, same caveat as Music). + +--- + ## Windows port β€” app interaction πŸͺŸ βœ… Implemented Phase 1's app-interaction layer is now ported to Windows. The macOS path uses the @@ -549,17 +567,27 @@ Shipped on the Windows machine (2026-06-02): --- -## Phase 3 β€” Wake-Word + Fast Routing ⏳ Not Started +## Phase 3 β€” Wake-Word + Fast Routing πŸ”¨ In progress + +**Fast routing β€” DONE & WIRED:** `src/openhuman/voice/command_router.rs` (new) β€” pure `route(transcript) -> VoiceIntent` classifier (Play/Pause/Resume/Next/Previous/OpenApp/SetVolume/VolumeUpΒ·Down/Mute/Unmute, else `Unknown`). High-confidence intents execute directly (launch_app / Music fast-path / osascript volume) without a full chat-LLM turn β€” the ≀500 ms path; `Unknown` defers to the agent so routing only ever shortcuts. Filler-tolerant ("please open up slack"). 5 unit tests. + +**Router wired into delivery:** `always_on::deliver_command` now routes the extracted command through `command_router::route` first. Recognized intents run locally via `execute_intent` (Play β†’ `automate` Music fast-path; OpenApp β†’ `launch_platform`; Pause/Resume/Next/Previous/volume/mute β†’ `osa` osascript); `VoiceIntent::Unknown` (and any local-exec failure) falls back to the full agent turn. Verified live: clean transport/open/volume commands execute on the fast path; complex/garbled `play` queries (e.g. mangled STT like "bts s latest song wind blowing") fall through to the agent as designed. Committed on `feat/voice-phase3-fast-routing`. + +**Remaining Phase 3:** on-device audio wake-word model (Porcupine / ONNX) in `inference/voice/wake_word.rs` to gate STT *before* transcription β€” the text-based "Hey Tiny" wake match from Phase 2 (fuzzy anchor on the "tiny" token, Levenshtein ≀1) is the interim. This is the only open Phase 3 item; not yet started. + +--- + +## Phase 3 β€” Wake-Word + Fast Routing (original plan) > Activate only on a trigger phrase; route simple commands locally without a full LLM turn. **Planned files:** -- `src/openhuman/inference/voice/wake_word.rs` (new) β€” lightweight always-on model (Porcupine or custom ONNX) -- `src/openhuman/voice/command_router.rs` (new) β€” intentβ†’tool mapping for high-confidence commands, LLM fallback for ambiguous input +- `src/openhuman/inference/voice/wake_word.rs` (new) β€” lightweight always-on model (Porcupine or custom ONNX) β€” ⏳ **not started** (interim: text-based "Hey Tiny" match in Phase 2) +- `src/openhuman/voice/command_router.rs` (new) β€” intentβ†’tool mapping for high-confidence commands, LLM fallback for ambiguous input β€” βœ… **shipped & wired** (see Phase 3 status above) **Acceptance criteria:** -- [ ] Wake-word detection runs fully on-device -- [ ] Latency from end-of-utterance to action start ≀ 500ms for local-routed commands +- [ ] Wake-word detection runs fully on-device (still pending β€” audio wake-word model not yet built) +- [x] Latency from end-of-utterance to action start ≀ 500ms for local-routed commands (command router executes recognized intents on the local fast path) --- @@ -607,19 +635,23 @@ From live agent-in-the-loop testing on 2026-06-03 (grounded in `~/.openhuman/log | 1 | AXUIElement app UI interaction (`ax_interact`) | βœ… Done | | 1 | Multi-step UI workflow guidance | βœ… Done | | 1 | Apple Music two-step play (navigateβ†’play) | βœ… Done (playback best-effort) | -| 1 | `automate(app, goal)` Rust-driven loop (Change 1.14) | πŸ”¨ M1+M2+M3 done (37 tests; live proof pending) | +| 1 | Full computer control (mouse/keyboard/screenshot) β€” CEF crash (Change 1.15) | βœ… Fixed (main-thread synthetic-input dispatch; screenshot downscale) | +| 1 | Windows port (`launch_app` + UIA `ax_interact`) | βœ… Done (Calculator + Notepad hard-asserted on Win11) | +| 1 | `automate(app, goal)` Rust-driven loop (Change 1.14) | βœ… M1–M5 done (proven live on macOS) | | 1.5 | M1: automate loop skeleton + tool | βœ… Done | | 1.5 | M2: poll-until-stable settle | βœ… Done | | 1.5 | M3: Music fast-path | βœ… Done (proven live on macOS) | | 1.5 | Robustness: quoted-query parse + no-progress guard | βœ… Done (from live agent failures) | | 1.5 | M4: progress streaming to notch | βœ… Done β€” notch cherry-picked in; automate streams live steps | | 1.5 | M5: richer element model (`enabled`) | βœ… Plumbed; AXEnabled found unreliable β†’ informational only | -| 1.5 | Native fast-paths (Music/Spotify/Slack) | ⏳ Not started | -| 1.5 | Vision fallback for Electron apps | ⏳ Not started | +| 1.5 | Native fast-paths beyond Music (Spotify/Slack) | βž– Descoped (general loop covers them; Music shipped in M3) | +| 1.5 | Vision fallback for Electron apps (Change 1.16) | βœ… Done (`vision_click`: screenshot β†’ vision-locate β†’ guarded click; frontmost guard; F2 coord-transform folded in) | | 2 | Always-on microphone loop | βœ… Done (cpal β†’ VAD β†’ STT β†’ agent) | | 2 | `always_on_enabled` config flag + Settings toggle | βœ… Done (RPC + UI + i18n) | | 2 | Privacy hook (screen lock pause) | βœ… Done (macOS; other OSes follow-up) | -| 3 | Wake-word detection | ⏳ Not started | -| 3 | Local command router | ⏳ Not started | +| 2 | Text-based "Hey Tiny" wake word | βœ… Done (interim; gates delivery, strips phrase) | +| 3 | Local command router (intent classifier) | βœ… Done & wired (recognized intents run on the ≀500ms local path; Unknown defers to agent) | +| 3 | On-device audio wake-word model | ⏳ Not started (text-based match is the interim) | | 4 | Voice confirmation loop | ⏳ Not started | +| 4 | Computer-control onboarding toggle | ⏳ Not started | | 4 | Always-on UI indicator | βœ… Done (notch PR #3166) | diff --git a/src/openhuman/accessibility/automate.rs b/src/openhuman/accessibility/automate.rs index 910d2f8e5..3ec3dbf10 100644 --- a/src/openhuman/accessibility/automate.rs +++ b/src/openhuman/accessibility/automate.rs @@ -68,6 +68,9 @@ pub struct Action { /// Text to enter for `set_value`. #[serde(default)] pub value: String, + /// Natural-language target for `vision_click` (e.g. "the green Call button"). + #[serde(default)] + pub description: String, /// Final message for `done` / `fail`. #[serde(default)] pub summary: String, @@ -116,6 +119,38 @@ pub trait AutomateBackend: Send + Sync { async fn verify_playing(&self) -> Option { None } + /// Capture the target app's window + the geometry needed to map a click + /// from image pixels to screen points. Used by the `vision_click` fallback + /// for apps with no usable accessibility tree (Electron/Chromium). Default + /// errors so backends without screen access (tests, headless) opt out. + async fn screenshot( + &self, + _app: &str, + ) -> Result<(String, super::vision_click::CaptureGeometry), String> { + Err("screenshot unsupported by this backend".to_string()) + } + /// Ask the vision model for the absolute *screen* coordinates of the + /// described element in `screenshot`. `Ok(None)` = not visible. Default + /// `Ok(None)` so non-vision backends never click. + async fn locate( + &self, + _screenshot: &str, + _geom: &super::vision_click::CaptureGeometry, + _description: &str, + ) -> Result, String> { + Ok(None) + } + /// Name of the frontmost application, if known. Used as the Β§1.8 safety + /// guard: a `vision_click` only fires when the target app is frontmost, so + /// synthetic input never lands on OpenHuman's own window. `None` = unknown. + async fn frontmost_app(&self) -> Option { + None + } + /// Issue a single guarded left-click at absolute screen coordinates. Default + /// errors so backends without input access can't click. + async fn click(&self, _x: i32, _y: i32) -> Result { + Err("click unsupported by this backend".to_string()) + } /// Block until the UI settles after an action. async fn settle(&self, app: &str); /// Wait ~`ms` of real time. Used by fast-paths to let asynchronous content @@ -155,6 +190,10 @@ fn system_prompt() -> String { β€’ list β€” re-read elements; set `filter` to a substring to narrow them\n\ β€’ press β€” activate the element whose label matches `label`\n\ β€’ set_value β€” type `value` into the field matching `label` (omit label = first field)\n\ + β€’ vision_click β€” click an element by sight; put a short `description` of the \ + target (e.g. 'the green Call button'). Use this when the element list is \ + EMPTY or missing your target β€” common for Electron apps (Slack, Discord, \ + VS Code) that expose no accessibility tree.\n\ β€’ done β€” goal achieved; put a short result in `summary`\n\ β€’ fail β€” goal cannot be achieved; explain in `summary`\n\ \n\ @@ -164,6 +203,8 @@ fn system_prompt() -> String { (e.g. open a song, THEN press its 'Play'). After such a press, `list` again \ to see the new screen.\n\ - Prefer an exact label match. Keep `filter` specific so the snapshot stays small.\n\ + - If the app shows NO elements, prefer `vision_click` with a clear \ + `description` over guessing labels.\n\ - Output JSON only β€” no prose, no code fences." .to_string() } @@ -319,7 +360,10 @@ pub async fn run( // ── no-progress guard ── if !matches!(action.action.as_str(), "done" | "fail") { - let sig = format!("{}|{}|{}", action.action, action.label, action.filter); + let sig = format!( + "{}|{}|{}|{}", + action.action, action.label, action.filter, action.description + ); if sig == last_sig { repeat_count += 1; } else { @@ -411,6 +455,69 @@ pub async fn run( } backend.settle(target_app).await; } + "vision_click" => { + let description = action.description.trim(); + if description.is_empty() { + steps.push("vision_click skipped: empty description".to_string()); + continue; + } + // ── Β§1.8 safety guard ── + // Only click when the target app is frontmost, so synthetic + // input never lands on OpenHuman's own window (the CEF crash). + // `None` = can't tell β†’ proceed best-effort (the loop already + // foregrounded the app at start). We only REFUSE on positive + // evidence that a different app is focused. + if let Some(front) = backend.frontmost_app().await { + if !front.eq_ignore_ascii_case(target_app) { + log::warn!( + "{LOG_PREFIX} vision_click: {target_app:?} not frontmost ({front:?}); re-foregrounding" + ); + let _ = backend.act_launch(target_app).await; + backend.settle(target_app).await; + let still_wrong = backend + .frontmost_app() + .await + .map(|f| !f.eq_ignore_ascii_case(target_app)) + .unwrap_or(false); + if still_wrong { + steps.push(format!( + "vision_click refused: {target_app} is not frontmost" + )); + continue; + } + } + } + progress( + format!("Looking for {description}…"), + OverlayAttentionTone::Accent, + ); + let (shot, geom) = match backend.screenshot(target_app).await { + Ok(pair) => pair, + Err(e) => { + steps.push(format!("vision_click FAILED: screenshot: {e}")); + continue; + } + }; + match backend.locate(&shot, &geom, description).await { + Ok(Some((x, y))) => { + progress( + format!("Clicking {description}…"), + OverlayAttentionTone::Accent, + ); + match backend.click(x, y).await { + Ok(msg) => steps.push(format!("vision_click: {msg}")), + Err(e) => steps.push(format!("vision_click FAILED: click: {e}")), + } + backend.settle(target_app).await; + } + Ok(None) => { + steps.push(format!("vision_click: '{description}' not found on screen")); + } + Err(e) => { + steps.push(format!("vision_click FAILED: locate: {e}")); + } + } + } other => { steps.push(format!("unknown action {other:?} ignored")); } @@ -470,6 +577,60 @@ impl AutomateBackend for RealBackend { ax::ax_set_field_value(app, label, value) } + async fn screenshot( + &self, + app: &str, + ) -> Result<(String, super::vision_click::CaptureGeometry), String> { + // Capture whatever window is frontmost β€” the loop guarantees the target + // is frontmost before a vision_click, so this resolves to its window. + let ctx = super::foreground_context() + .ok_or_else(|| "could not resolve the foreground window for capture".to_string())?; + if let Some(name) = ctx.app_name.as_deref() { + if !name.eq_ignore_ascii_case(app) { + log::warn!( + "{LOG_PREFIX} screenshot: frontmost {name:?} != target {app:?}; capturing frontmost" + ); + } + } + // `capture_window_geometry` shells out to `screencapture` (blocking). + match tokio::task::spawn_blocking(move || { + super::vision_click::capture_window_geometry(&ctx) + }) + .await + { + Ok(inner) => inner, + Err(e) => Err(format!("capture task join failed: {e}")), + } + } + + async fn locate( + &self, + screenshot: &str, + geom: &super::vision_click::CaptureGeometry, + description: &str, + ) -> Result, String> { + // Use the main `chat` provider's vision model (per plan): reliable UI + // grounding, and the fallback only fires when AX is empty (rare). + let (provider, model) = + crate::openhuman::inference::provider::create_chat_provider("chat", &self.config) + .map_err(|e| format!("vision provider unavailable: {e}"))?; + let coords = + super::vision_click::locate_via_vision(&*provider, &model, screenshot, description) + .await?; + Ok(coords.map(|(px, py)| super::vision_click::image_to_screen(geom, px, py))) + } + + async fn frontmost_app(&self) -> Option { + tokio::task::spawn_blocking(|| super::foreground_context().and_then(|c| c.app_name)) + .await + .ok() + .flatten() + } + + async fn click(&self, x: i32, y: i32) -> Result { + super::vision_click::guarded_click(x, y).await + } + async fn open_url(&self, url: &str) -> Result { // Cross-platform URI opener. macOS `open`, Linux `xdg-open`, Windows // `cmd /C start`. Only invoked by fast-paths with app-controlled URLs diff --git a/src/openhuman/accessibility/automate_tests.rs b/src/openhuman/accessibility/automate_tests.rs index 6b169e98b..1cf15a831 100644 --- a/src/openhuman/accessibility/automate_tests.rs +++ b/src/openhuman/accessibility/automate_tests.rs @@ -3,6 +3,7 @@ //! with no mic, no AX tree, and no LLM. use super::*; +use crate::openhuman::accessibility::vision_click::CaptureGeometry; use std::sync::Mutex; /// Scripted backend: `decide` returns the next queued response each call; @@ -16,6 +17,10 @@ struct ScriptedBackend { acts: Mutex>, /// Force act_press to error (to exercise the failure-recording path). press_errors: bool, + /// What `frontmost_app` returns (the Β§1.8 guard input). `None` = unknown. + frontmost: Option, + /// What `locate` returns: `Some` screen coords = found; `None` = not found. + locate_coord: Option<(i32, i32)>, } impl ScriptedBackend { @@ -28,11 +33,36 @@ impl ScriptedBackend { ], acts: Mutex::new(Vec::new()), press_errors: false, + frontmost: None, + locate_coord: None, } } fn acts(&self) -> Vec { self.acts.lock().unwrap().clone() } + /// Set the frontmost app name reported to the `vision_click` guard. + fn with_frontmost(mut self, app: &str) -> Self { + self.frontmost = Some(app.to_string()); + self + } + /// Make `locate` report the element found at the given screen coords. + fn with_located(mut self, x: i32, y: i32) -> Self { + self.locate_coord = Some((x, y)); + self + } +} + +/// A throwaway geometry for the scripted `screenshot` β€” tests override `locate` +/// directly, so the transform isn't exercised here (it has its own unit tests). +fn dummy_geom() -> CaptureGeometry { + CaptureGeometry { + rect_x: 0, + rect_y: 0, + rect_w_pts: 1, + rect_h_pts: 1, + img_w_px: 1, + img_h_px: 1, + } } #[async_trait] @@ -71,6 +101,29 @@ impl AutomateBackend for ScriptedBackend { .push(format!("set_value:{app}:{label}={value}")); Ok(format!("Set '{label}' in '{app}'.")) } + async fn screenshot(&self, app: &str) -> Result<(String, CaptureGeometry), String> { + self.acts.lock().unwrap().push(format!("screenshot:{app}")); + Ok(("data:image/png;base64,TEST".to_string(), dummy_geom())) + } + async fn locate( + &self, + _shot: &str, + _geom: &CaptureGeometry, + description: &str, + ) -> Result, String> { + self.acts + .lock() + .unwrap() + .push(format!("locate:{description}")); + Ok(self.locate_coord) + } + async fn frontmost_app(&self) -> Option { + self.frontmost.clone() + } + async fn click(&self, x: i32, y: i32) -> Result { + self.acts.lock().unwrap().push(format!("click:{x},{y}")); + Ok(format!("Clicked at ({x}, {y})")) + } async fn open_url(&self, url: &str) -> Result { self.acts.lock().unwrap().push(format!("open_url:{url}")); Ok(format!("Opened {url}")) @@ -264,3 +317,96 @@ fn render_snapshot_empty_hint() { let s = render_snapshot("Music", "zzz", &[]); assert!(s.contains("no elements")); } + +// ── vision_click fallback ──────────────────────────────────────────────────── + +#[tokio::test] +async fn vision_click_locates_and_clicks_when_frontmost() { + let backend = ScriptedBackend::new(&[ + r#"{"action":"vision_click","description":"the Call button"}"#, + r#"{"action":"done","summary":"clicked"}"#, + ]) + .with_frontmost("Slack") + .with_located(640, 360); + let out = run("Slack", "click the call button", &backend, opts(5)).await; + assert!(out.success, "{out:?}"); + let acts = backend.acts(); + assert!(acts.contains(&"screenshot:Slack".to_string()), "{acts:?}"); + assert!( + acts.contains(&"locate:the Call button".to_string()), + "{acts:?}" + ); + assert!(acts.contains(&"click:640,360".to_string()), "{acts:?}"); +} + +#[tokio::test] +async fn vision_click_proceeds_when_frontmost_unknown() { + // `None` frontmost (e.g. can't determine) is best-effort: the loop already + // foregrounded the app, so we still click. + let backend = ScriptedBackend::new(&[ + r#"{"action":"vision_click","description":"X"}"#, + r#"{"action":"done"}"#, + ]) + .with_located(10, 20); + let out = run("Slack", "x", &backend, opts(5)).await; + assert!(out.success); + assert!(backend.acts().contains(&"click:10,20".to_string())); +} + +#[tokio::test] +async fn vision_click_refused_when_other_app_frontmost() { + // Positive evidence a different app is focused β†’ refuse (the Β§1.8 guard). + let backend = ScriptedBackend::new(&[ + r#"{"action":"vision_click","description":"the Call button"}"#, + r#"{"action":"done","summary":"done"}"#, + ]) + .with_frontmost("Finder") + .with_located(640, 360); + let out = run("Slack", "click call", &backend, opts(5)).await; + let acts = backend.acts(); + assert!( + !acts.iter().any(|a| a.starts_with("click:")), + "must not click into a non-target app: {acts:?}" + ); + assert!( + !acts.iter().any(|a| a.starts_with("screenshot:")), + "must not even screenshot when refused: {acts:?}" + ); + let _ = out; +} + +#[tokio::test] +async fn vision_click_not_found_does_not_click() { + let backend = ScriptedBackend::new(&[ + r#"{"action":"vision_click","description":"the Call button"}"#, + r#"{"action":"done","summary":"gave up"}"#, + ]) + .with_frontmost("Slack"); // locate_coord stays None β†’ not found + let out = run("Slack", "click call", &backend, opts(5)).await; + assert!(out.success); + let acts = backend.acts(); + assert!(acts.contains(&"screenshot:Slack".to_string()), "{acts:?}"); + assert!( + acts.contains(&"locate:the Call button".to_string()), + "{acts:?}" + ); + assert!( + !acts.iter().any(|a| a.starts_with("click:")), + "no click when the element isn't found: {acts:?}" + ); +} + +#[tokio::test] +async fn vision_click_empty_description_skipped() { + let backend = ScriptedBackend::new(&[ + r#"{"action":"vision_click","description":" "}"#, + r#"{"action":"done"}"#, + ]) + .with_frontmost("Slack"); + let out = run("Slack", "x", &backend, opts(5)).await; + assert!(out.success); + assert!( + !backend.acts().iter().any(|a| a.starts_with("screenshot:")), + "empty description must be skipped before any capture" + ); +} diff --git a/src/openhuman/accessibility/mod.rs b/src/openhuman/accessibility/mod.rs index d90a7cb40..a7b1de0ab 100644 --- a/src/openhuman/accessibility/mod.rs +++ b/src/openhuman/accessibility/mod.rs @@ -20,6 +20,9 @@ mod permissions; mod terminal; mod text_util; mod types; +// Vision fallback for `automate`: screenshot β†’ vision-locate β†’ guarded click, +// for Electron/partial-AX apps. Consumed by `automate.rs`'s `RealBackend`. +mod vision_click; // Windows accessibility backend for `ax_interact` (UI Automation). Sibling of // the macOS Swift-helper path; selected via cfg-dispatch in `ax_interact.rs`. #[cfg(target_os = "windows")] diff --git a/src/openhuman/accessibility/vision_click.rs b/src/openhuman/accessibility/vision_click.rs new file mode 100644 index 000000000..7c28664de --- /dev/null +++ b/src/openhuman/accessibility/vision_click.rs @@ -0,0 +1,213 @@ +//! Vision fallback for `automate`: click a described on-screen element when an +//! app exposes no usable accessibility tree. +//! +//! Electron/Chromium apps (Slack, Discord, VS Code) expose little or no AX/UIA, +//! so the perceiveβ†’press loop has nothing to act on (tracker Β§1.5). The answer +//! is *screenshot β†’ vision-locate β†’ guarded click*: +//! +//! 1. **screenshot** the target app's window (`super::capture`), recording the +//! window's screen rect so pixels can be mapped back to screen points. +//! 2. **locate** β€” ask the main vision model for the PIXEL coordinates of the +//! described element, passing the image via the provider's `[IMAGE:]` +//! marker (promoted to a real image part in +//! `inference/provider/compatible_types.rs`, #3205). +//! 3. **map** image pixels β†’ absolute screen points ([`image_to_screen`]). +//! 4. **guarded click** β€” only ever issued by the caller once the target app +//! is frontmost, never into OpenHuman's own window (the Β§1.8 CEF-crash +//! guard). Synthetic input runs on the app main thread via +//! `run_input_on_main` (Change 1.15 β€” off-thread enigo traps TSM). +//! +//! The coordinate transform and the model-response parser are pure and unit +//! tested; the capture / model / click side-effects sit behind them. + +use super::types::{AppContext, ElementBounds}; + +const LOG_PREFIX: &str = "[vision_click]"; + +/// Geometry needed to map a point in the (possibly downscaled) screenshot back +/// to absolute screen coordinates. +/// +/// `rect_*` is the captured window's screen rect in **points** (from +/// [`AppContext::bounds`]); `img_*` is the screenshot's **pixel** size. The +/// pixelβ†’point ratio absorbs both the capture downscale *and* the Retina backing +/// scale, so the mapping needs no explicit scale factor (this is what closes the +/// deferred F2 coordinate-mapping gap). +// `pub` (not `pub(crate)`) so it doesn't read as "more private" than the +// `pub AutomateBackend` trait methods that name it; the enclosing `mod +// vision_click` is private, so it stays crate-internal in practice. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct CaptureGeometry { + pub rect_x: i32, + pub rect_y: i32, + pub rect_w_pts: i32, + pub rect_h_pts: i32, + pub img_w_px: u32, + pub img_h_px: u32, +} + +impl CaptureGeometry { + fn from_bounds(b: &ElementBounds, img_w_px: u32, img_h_px: u32) -> Self { + Self { + rect_x: b.x, + rect_y: b.y, + rect_w_pts: b.width, + rect_h_pts: b.height, + img_w_px, + img_h_px, + } + } +} + +/// Map an image-pixel coordinate to an absolute screen point. +/// +/// Clamps the input to the image and the output to the window rect, so a model +/// that returns an out-of-range guess still lands inside the *target* window β€” +/// never elsewhere on screen (and never on OpenHuman's own window). +pub(crate) fn image_to_screen(geom: &CaptureGeometry, px: i32, py: i32) -> (i32, i32) { + let img_w = geom.img_w_px.max(1) as f64; + let img_h = geom.img_h_px.max(1) as f64; + // Clamp the sampled pixel into the image, then express it as a 0..1 fraction. + let fx = (px.max(0) as f64).min(img_w - 1.0) / img_w; + let fy = (py.max(0) as f64).min(img_h - 1.0) / img_h; + let sx = geom.rect_x as f64 + fx * geom.rect_w_pts as f64; + let sy = geom.rect_y as f64 + fy * geom.rect_h_pts as f64; + // Keep the result strictly inside the window rect. + let max_x = geom.rect_x + (geom.rect_w_pts - 1).max(0); + let max_y = geom.rect_y + (geom.rect_h_pts - 1).max(0); + ( + (sx.round() as i32).clamp(geom.rect_x, max_x), + (sy.round() as i32).clamp(geom.rect_y, max_y), + ) +} + +/// The JSON the vision model must return: whether the element was found and its +/// center pixel coordinates within the screenshot. +#[derive(Debug, serde::Deserialize)] +struct LocateResponse { + #[serde(default)] + found: bool, + #[serde(default)] + x: i32, + #[serde(default)] + y: i32, +} + +/// System prompt pinning the locate contract for the vision model. +fn locate_system_prompt() -> &'static str { + "You locate a single UI element in a screenshot of one application window. \ + You are given the image and a description of the element to click. Reply \ + with EXACTLY ONE JSON object and nothing else: \ + {\"found\":true,\"x\":,\"y\":} where x,y are the PIXEL \ + coordinates (origin top-left) of the CENTER of that element within the \ + image. If the element is not visible, reply {\"found\":false,\"x\":0,\"y\":0}. \ + Output JSON only β€” no prose, no code fences." +} + +/// Build the user turn: the target description plus the screenshot as a +/// `[IMAGE:]` marker the compatible provider promotes to an image part. +fn build_locate_user(description: &str, screenshot_data_uri: &str) -> String { + format!("Element to click: {description}\n[IMAGE:{screenshot_data_uri}]") +} + +/// Parse the model's locate reply, tolerating code fences / surrounding prose by +/// extracting the first balanced `{...}` (mirrors `automate::parse_action`). +/// `found:false` β†’ `Ok(None)`; unparseable β†’ `Err` so the caller can report it +/// rather than act on a hallucinated guess (tracker Β§1.13 lesson). +fn parse_locate_response(raw: &str) -> Result, String> { + let trimmed = raw.trim(); + let parsed = serde_json::from_str::(trimmed).or_else(|_| { + match (trimmed.find('{'), trimmed.rfind('}')) { + (Some(s), Some(e)) if e > s => serde_json::from_str::(&trimmed[s..=e]), + // Re-run on the trimmed text so the error type matches the arm. + _ => serde_json::from_str::(trimmed), + } + }); + match parsed { + Ok(r) if r.found => Ok(Some((r.x, r.y))), + Ok(_) => Ok(None), + Err(_) => Err(format!("could not parse locate response: {trimmed:?}")), + } +} + +/// Decode the pixel dimensions of a `data:image/...;base64,...` URI. +fn image_dims_from_data_uri(data_uri: &str) -> Result<(u32, u32), String> { + use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _}; + use image::GenericImageView as _; + let b64 = data_uri + .split_once(',') + .map(|(_, payload)| payload) + .ok_or_else(|| "malformed image data URI (no base64 payload)".to_string())?; + let bytes = BASE64_STANDARD + .decode(b64) + .map_err(|e| format!("screenshot base64 decode failed: {e}"))?; + let img = + image::load_from_memory(&bytes).map_err(|e| format!("screenshot decode failed: {e}"))?; + Ok(img.dimensions()) +} + +/// Capture the app window and pair it with the geometry needed to map a click. +/// +/// Requires `ctx.bounds` (the window's screen rect) β€” without it the image +/// pixels can't be mapped to screen points, so we refuse rather than click blind. +pub(crate) fn capture_window_geometry( + ctx: &AppContext, +) -> Result<(String, CaptureGeometry), String> { + let bounds = ctx + .bounds + .ok_or_else(|| "window bounds unavailable β€” cannot map click coordinates".to_string())?; + let data_uri = super::capture::capture_screen_image_ref_for_context(Some(ctx))?; + let (w, h) = image_dims_from_data_uri(&data_uri)?; + log::debug!( + "{LOG_PREFIX} captured window rect=({},{},{},{}) image={}x{}px", + bounds.x, + bounds.y, + bounds.width, + bounds.height, + w, + h + ); + Ok((data_uri, CaptureGeometry::from_bounds(&bounds, w, h))) +} + +/// Ask the vision model for the target's pixel coordinates within `screenshot`. +/// Returns `Ok(None)` when the model reports the element isn't visible. +pub(crate) async fn locate_via_vision( + provider: &dyn crate::openhuman::inference::provider::Provider, + model: &str, + screenshot_data_uri: &str, + description: &str, +) -> Result, String> { + let user = build_locate_user(description, screenshot_data_uri); + let raw = provider + .chat_with_system(Some(locate_system_prompt()), &user, model, 0.0) + .await + .map_err(|e| format!("vision model call failed: {e}"))?; + log::debug!("{LOG_PREFIX} locate raw response: {raw:?}"); + parse_locate_response(&raw) +} + +/// Single guarded left-click at absolute screen coordinates, run on the app +/// main thread. Off-thread enigo traps macOS TSM and crashes the CEF host +/// (Change 1.15 / Β§1.8) β€” so the click closure is dispatched via +/// `run_input_on_main`, exactly like the `mouse` tool. +pub(crate) async fn guarded_click(x: i32, y: i32) -> Result { + use crate::openhuman::tools::implementations::run_input_on_main; + log::info!("{LOG_PREFIX} β–Ά click at screen ({x}, {y})"); + run_input_on_main(move || { + use enigo::{Button, Coordinate, Direction, Enigo, Mouse, Settings}; + let mut enigo = + Enigo::new(&Settings::default()).map_err(|e| format!("enigo init failed: {e}"))?; + enigo + .move_mouse(x, y, Coordinate::Abs) + .map_err(|e| format!("move_mouse failed: {e}"))?; + enigo + .button(Button::Left, Direction::Click) + .map_err(|e| format!("click failed: {e}"))?; + Ok(format!("Clicked at ({x}, {y})")) + }) + .await +} + +#[cfg(test)] +#[path = "vision_click_tests.rs"] +mod tests; diff --git a/src/openhuman/accessibility/vision_click_tests.rs b/src/openhuman/accessibility/vision_click_tests.rs new file mode 100644 index 000000000..6cec3a510 --- /dev/null +++ b/src/openhuman/accessibility/vision_click_tests.rs @@ -0,0 +1,148 @@ +//! Unit tests for the vision-click primitives. The coordinate transform and the +//! locate-response parser are pure, so they're exercised with no screen, no +//! model, and no synthetic input. + +use super::*; + +fn geom( + rect_x: i32, + rect_y: i32, + rect_w_pts: i32, + rect_h_pts: i32, + img_w_px: u32, + img_h_px: u32, +) -> CaptureGeometry { + CaptureGeometry { + rect_x, + rect_y, + rect_w_pts, + rect_h_pts, + img_w_px, + img_h_px, + } +} + +// ── image_to_screen ───────────────────────────────────────────────────────── + +#[test] +fn maps_center_when_image_matches_rect() { + // Image pixels == window points (no scaling): center maps to center. + let g = geom(0, 0, 1000, 800, 1000, 800); + assert_eq!(image_to_screen(&g, 500, 400), (500, 400)); +} + +#[test] +fn maps_through_downscaled_screenshot() { + // Capture was downscaled to half size; a center pixel still maps to the + // window center in screen points. + let g = geom(0, 0, 1000, 800, 500, 400); + assert_eq!(image_to_screen(&g, 250, 200), (500, 400)); +} + +#[test] +fn maps_through_retina_2x_backing_scale() { + // Retina capture is 2Γ— the window's point size; the pxβ†’pt ratio absorbs it. + let g = geom(0, 0, 1000, 800, 2000, 1600); + assert_eq!(image_to_screen(&g, 1000, 800), (500, 400)); +} + +#[test] +fn applies_window_origin_offset() { + // A window not at the screen origin: image coords are window-relative, the + // result is absolute screen coords. + let g = geom(100, 50, 400, 300, 400, 300); + assert_eq!(image_to_screen(&g, 200, 150), (300, 200)); +} + +#[test] +fn clamps_out_of_range_pixel_into_window() { + // A wild guess past the image bounds still lands strictly inside the window. + let g = geom(100, 50, 400, 300, 400, 300); + let (x, y) = image_to_screen(&g, 99_999, 99_999); + assert_eq!((x, y), (499, 349)); // rect_x + w - 1, rect_y + h - 1 +} + +#[test] +fn clamps_negative_pixel_to_window_origin() { + let g = geom(100, 50, 400, 300, 400, 300); + assert_eq!(image_to_screen(&g, -10, -10), (100, 50)); +} + +#[test] +fn handles_zero_image_dims_without_panicking() { + // Defensive: a degenerate (0px) image must not divide by zero. + let g = geom(0, 0, 100, 100, 0, 0); + let (x, y) = image_to_screen(&g, 10, 10); + assert!((0..100).contains(&x) && (0..100).contains(&y)); +} + +// ── parse_locate_response ──────────────────────────────────────────────────── + +#[test] +fn parses_found_coordinates() { + let r = parse_locate_response(r#"{"found":true,"x":120,"y":340}"#).unwrap(); + assert_eq!(r, Some((120, 340))); +} + +#[test] +fn not_found_is_none() { + let r = parse_locate_response(r#"{"found":false,"x":0,"y":0}"#).unwrap(); + assert_eq!(r, None); +} + +#[test] +fn tolerates_code_fences() { + let raw = "```json\n{\"found\":true,\"x\":5,\"y\":6}\n```"; + assert_eq!(parse_locate_response(raw).unwrap(), Some((5, 6))); +} + +#[test] +fn tolerates_surrounding_prose() { + let raw = "Sure! Here it is: {\"found\":true,\"x\":7,\"y\":8} β€” hope that helps"; + assert_eq!(parse_locate_response(raw).unwrap(), Some((7, 8))); +} + +#[test] +fn garbage_is_error() { + assert!(parse_locate_response("no json here").is_err()); +} + +#[test] +fn missing_found_defaults_to_not_found() { + // `found` defaults to false when absent, so an answer without it is None. + let r = parse_locate_response(r#"{"x":1,"y":2}"#).unwrap(); + assert_eq!(r, None); +} + +// ── build_locate_user ──────────────────────────────────────────────────────── + +#[test] +fn user_turn_embeds_description_and_image_marker() { + let user = build_locate_user("the green Call button", "data:image/png;base64,AAA"); + assert!(user.contains("the green Call button")); + assert!(user.contains("[IMAGE:data:image/png;base64,AAA]")); +} + +// ── image_dims_from_data_uri ───────────────────────────────────────────────── + +fn png_data_uri(w: u32, h: u32) -> String { + use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _}; + let img = image::DynamicImage::ImageRgb8(image::RgbImage::new(w, h)); + let mut buf = std::io::Cursor::new(Vec::new()); + img.write_to(&mut buf, image::ImageFormat::Png).unwrap(); + format!( + "data:image/png;base64,{}", + BASE64_STANDARD.encode(buf.get_ref()) + ) +} + +#[test] +fn reads_png_dimensions() { + let uri = png_data_uri(640, 480); + assert_eq!(image_dims_from_data_uri(&uri).unwrap(), (640, 480)); +} + +#[test] +fn rejects_malformed_data_uri() { + assert!(image_dims_from_data_uri("not-a-data-uri").is_err()); +}