mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
This commit is contained in:
Mega Mind
@@ -1,9 +1,9 @@
|
||||
# Voice → System Action Feature Tracker
|
||||
|
||||
**GitHub Issue:** [#3148](https://github.com/tinyhumansai/openhuman/issues/3148)
|
||||
**Branch:** `feat/voice-always-on`
|
||||
**PR:** [#3168](https://github.com/tinyhumansai/openhuman/pull/3168)
|
||||
**Branch:** `feat/voice-always-on-all` (cumulative) — Phase 1 landed via [#3168](https://github.com/tinyhumansai/openhuman/pull/3168); the full feature was split from the mega-PR [#3307](https://github.com/tinyhumansai/openhuman/pull/3307) into an 8-PR stack [#3340–#3346](https://github.com/tinyhumansai/openhuman/pull/3340) + [#3362](https://github.com/tinyhumansai/openhuman/pull/3362) (Phase 1.5 vision fallback)
|
||||
**Started:** 2026-06-02
|
||||
**Last updated:** 2026-06-04
|
||||
|
||||
---
|
||||
|
||||
@@ -268,9 +268,9 @@ test ... ok
|
||||
|
||||
---
|
||||
|
||||
### Change 1.14 — `automate(app, goal)`: Rust-driven multi-step automation 🔨 In progress (M1 done)
|
||||
### Change 1.14 — `automate(app, goal)`: Rust-driven multi-step automation ✅ M1–M5 done
|
||||
|
||||
**Status:** 🔨 In progress — **M1 + M2 + M3 shipped and M3 proven live on macOS**; M4–M6 pending. See **Phase 1.5** below and [`voice-automate-plan.md`](voice-automate-plan.md).
|
||||
**Status:** ✅ **M1 + M2 + M3 + M4 + M5 shipped; M3 proven live on macOS.** The only remaining `automate` work is the **vision fallback** for Electron/partial-AX apps — tracked under **Phase 1.5** below and [`voice-automate-plan.md`](voice-automate-plan.md). (Per-app native fast-paths beyond the shipped Music one are **descoped**.)
|
||||
|
||||
**Agent-in-the-loop fixes (2026-06-03, from two live chat sessions):**
|
||||
- **Mutations were off** — the agent correctly called `automate` but it (and `ax_interact`) refused because `computer_control.ax_interact_mutations=false`. Enabled it; also rewrote both refusal messages to point at **Settings → Agent Access** instead of a config key (the agent had relayed "controls are locked down").
|
||||
@@ -315,9 +315,9 @@ test ... ok
|
||||
- **A fast model drives the inner loop** (Haiku-class) with a *tiny* context: just the goal, the current small AX snapshot, and the last result — not the whole conversation. Each inner step is ~0.5–1 s and self-corrects, instead of one 3 s chat turn that falsely reports success.
|
||||
- **Settle + verify in Rust** between steps — deterministic, kills the timing-race class in one place.
|
||||
- **Native fast-paths for high-value apps** (skip the UI entirely where possible):
|
||||
- **Music** — `music://` search URL → AX play (already explored in 1.11), or AppleScript for library.
|
||||
- **Spotify** — Web API search → `spotify:track:…` URI + AppleScript `play`. Fully deterministic, no UI poking.
|
||||
- **Slack** — deep link `slack://channel?…` to open the DM, then AX to type + send.
|
||||
- **Music** — `music://` search URL → AX play (already explored in 1.11), or AppleScript for library. **(Shipped — M3.)**
|
||||
- ~~**Spotify** — Web API search → `spotify:track:…` URI + AppleScript `play`.~~ **Descoped** — the general loop covers it; not worth the per-app maintenance now.
|
||||
- ~~**Slack** — deep link `slack://channel?…` to open the DM, then AX to type + send.~~ **Descoped** — handled by the general loop / vision fallback.
|
||||
The general AX loop is the fallback for everything else.
|
||||
- **Vision fallback for Electron/Chromium apps** (Slack, Discord, VS Code, Spotify-desktop) whose AX/UIA tree is partial (documented limitation). Slack needs accessibility enabled (`defaults write com.tinyspeck.slackmacgap AccessibilityEnabled -bool true`, relaunch). Where AX returns empty, fall back to **screenshot → vision-locate → guarded click**. This is the reverted CGEventPost path (1.8) — but it crashed only when events hit *OpenHuman's own focused CEF window*; a guarded click into a *different, foregrounded* app does not have that failure mode.
|
||||
- **Stream progress events** to the UI / notch pill (PR #3166) so the user sees each step happen live.
|
||||
@@ -326,26 +326,27 @@ test ... ok
|
||||
|
||||
---
|
||||
|
||||
## Phase 1.5 — Reliable, real-time multi-step automation ⏳ Not Started
|
||||
## Phase 1.5 — Reliable, real-time multi-step automation ✅ Complete
|
||||
|
||||
> The bridge between today's `ax_interact` primitives and the always-on voice work. **Prerequisite for Phase 3** — fast voice routing into a slow/fragile action loop still feels slow. This is where "whatever I say happens, live" actually gets delivered.
|
||||
>
|
||||
> **Status:** the Rust inner loop (M1), poll-until-stable settle (M2), Music fast-path (M3, proven live), notch progress streaming (M4), the richer element model (M5), and the **vision fallback for Electron/partial-AX apps** (Change 1.16) are all shipped. (Additional per-app native fast-paths beyond Music — e.g. Spotify/Slack — are **descoped**: the general model-driven loop covers them, so a deterministic accelerator isn't worth the per-app maintenance right now.)
|
||||
|
||||
**Detailed implementation plan:** [`voice-automate-plan.md`](voice-automate-plan.md) — decided approach: **Rust inner loop + fast model**, first proof target **Music**.
|
||||
|
||||
**Planned files:**
|
||||
- `src/openhuman/accessibility/automate.rs` (new) — the perceive→act→verify loop + settle/verify primitives, reusing `ax_interact` helpers.
|
||||
- `src/openhuman/accessibility/app_fastpaths/` (new) — per-app deterministic paths (`music.rs`, `spotify.rs`, `slack.rs`), behind a generic dispatch.
|
||||
- `src/openhuman/accessibility/app_fastpaths/` (new) — per-app deterministic paths behind a generic dispatch. **Shipped:** `music.rs` (M3). Further per-app fast-paths (Spotify/Slack) are **descoped** — the general loop handles them.
|
||||
- `src/openhuman/tools/impl/computer/automate.rs` (new) — `AutomateTool { app, goal }`, gated like `ax_interact` (mutations opt-in, sensitive-app denylist reused).
|
||||
- macOS helper (`accessibility/helper.rs`) — AXObserver-based settle (`ax_wait_settled`) + post-action verify; richer element model (enabled/onscreen/actions).
|
||||
- Vision fallback — screenshot via `accessibility/capture.rs` → locate → guarded click (only when AX tree is empty, target app foregrounded, never OpenHuman's own window).
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] One `automate{app, goal}` call performs a multi-step flow end-to-end (no per-step chat turns)
|
||||
- [ ] Settle/verify removes the timing-race + false-success failure classes (1.11/1.13 do not recur)
|
||||
- [ ] Music flow ("play <song>") works end-to-end via the inner loop
|
||||
- [ ] Spotify + Slack fast-paths land their action deterministically
|
||||
- [ ] Electron/partial-AX apps fall back to vision+guarded-click without the CEF crash
|
||||
- [ ] Step-by-step progress streamed to the UI / notch indicator
|
||||
- [x] One `automate{app, goal}` call performs a multi-step flow end-to-end (no per-step chat turns)
|
||||
- [x] Settle/verify removes the timing-race + false-success failure classes (1.11/1.13 do not recur)
|
||||
- [x] Music flow ("play <song>") works end-to-end via the inner loop (proven live, hard-asserts `player state == playing`)
|
||||
- [x] Electron/partial-AX apps fall back to vision+guarded-click without the CEF crash (Change 1.16)
|
||||
- [x] Step-by-step progress streamed to the UI / notch indicator (M4)
|
||||
|
||||
---
|
||||
|
||||
@@ -368,12 +369,29 @@ enigo::macos::get_layoutdependent_keycode → TSMGetInputSourceProperty
|
||||
```
|
||||
enigo's keyboard-layout lookup (`TSMGetInputSourceProperty`) **must run on the app's main thread**; the keyboard tool ran on a tokio worker → macOS trapped. **Not** a focus issue (same §1.8 root cause); a frontmost-app guard would not have fixed it.
|
||||
|
||||
**Fix applied:** enigo now runs on the Tauri **main thread** via `AppHandle::run_on_main_thread`, bridged to the core through a native-registry handler (see *The fix* above) and wrapped in `catch_unwind` so an FFI panic can't unwind across the main thread. (Alternative considered but not needed: TSM-free primitives — `CGEventKeyboardSetUnicodeString` for text + raw virtual keycodes for keys/hotkeys.)
|
||||
**Fix applied:** all enigo operations now run on the Tauri **main thread** via `run_input_on_main` (new `main_thread.rs` module + a native-registry handler the shell registers, dispatched through `AppHandle::run_on_main_thread`) and wrapped in `catch_unwind`. Keyboard/mouse tools **and** `vision_click` now execute without TSM traps.
|
||||
|
||||
**Tests:** voice-actions + autonomy suite is exhaustive — 220 feature unit tests + a JSON-RPC E2E (`json_rpc_voice_server_settings_roundtrip_always_on_and_wake_word`). The E2E caught + fixed real gaps (`wake_word` missing from the get output and the update RPC path). Screenshot downscale unit-tested.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.16 — `vision_click`: vision fallback for Electron/partial-AX apps ✅ Done
|
||||
|
||||
**Status:** ✅ Shipped — closes the last open Phase 1.5 item. Electron/Chromium apps (Slack, Discord, VS Code) expose little or no AX/UIA tree, so the `automate` perceive→press loop had nothing to act on. The loop can now *see* the screen and click a described element.
|
||||
|
||||
**Problem:** when `perceive` returns an empty (or target-missing) element list, the loop was stuck — there was no label to press. The planned answer (tracker §1.5) was *screenshot → vision-locate → guarded click*, but two things blocked it: (1) the fast inner-loop model is text-only, and (2) the deferred **F2** coordinate-mapping gap — the screenshot is windowed + downscaled and Retina is 2×, while `mouse` expects absolute screen points, so a vision-returned coordinate would click the wrong spot.
|
||||
|
||||
**Fix — a new model-chosen `vision_click { description }` action in the `automate` loop:**
|
||||
- **Reuses the existing multimodal path** — the OpenAI-compatible provider promotes an embedded `[IMAGE:<data-uri>]` marker into a real `image_url` part (`inference/provider/compatible_types.rs`, #3205), so vision-locate rides the existing `chat_with_system` call with **no new inference API**. The locate call uses the **main `chat`/vision provider** (`create_chat_provider("chat", …)`) — reliable UI grounding, and the fallback only fires when AX is empty (rare).
|
||||
- **Folds in the F2 coordinate transform** — `src/openhuman/accessibility/vision_click.rs` (new): `CaptureGeometry` (window screen-rect in points + image pixel size) + a **pure, unit-tested** `image_to_screen(geom, px, py)`. The px→pt ratio absorbs both the downscale and the Retina backing scale, so no explicit scale factor is needed; the result is clamped strictly inside the target window.
|
||||
- **§1.8 safety guard** — a `vision_click` only fires when the target app is **frontmost** (`focus::foreground_context`); on positive evidence a different app is focused it re-foregrounds once and otherwise **refuses**, so synthetic input never lands on OpenHuman's own CEF window. `None` (can't tell) = best-effort, since the loop already foregrounded the app at start.
|
||||
- **Main-thread click** — the guarded left-click runs via `run_input_on_main` (Change 1.15), so off-thread enigo never traps TSM.
|
||||
- Wired into the `automate` loop: new `Action.description`, a `vision_click` system-prompt verb ("use when the element list is EMPTY — Electron apps"), the no-progress signature extended with `description`, and `RealBackend::{screenshot,locate,frontmost_app,click}`. Inherits `automate`'s existing gating (Dangerous + mutations opt-in + sensitive-app denylist) — no new tool, no new approval surface.
|
||||
|
||||
**Tests:** pure `image_to_screen` (downscale / Retina 2× / origin offset / out-of-range + negative clamp / zero-dim safety), `parse_locate_response` (found / not-found / fenced / prose / garbage), `build_locate_user` (marker), `image_dims_from_data_uri` (real PNG round-trip); loop integration via the scripted backend (locate-and-click when frontmost, proceed when frontmost unknown, **refuse when another app is frontmost**, no-click on not-found, empty-description skip). A macOS live run against a real Electron app is the remaining manual validation (vision is nondeterministic — assert tool-level success, treat the visual outcome as best-effort, same caveat as Music).
|
||||
|
||||
---
|
||||
|
||||
## Windows port — app interaction 🪟 ✅ Implemented
|
||||
|
||||
Phase 1's app-interaction layer is now ported to Windows. The macOS path uses the
|
||||
@@ -549,17 +567,27 @@ Shipped on the Windows machine (2026-06-02):
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Wake-Word + Fast Routing ⏳ Not Started
|
||||
## Phase 3 — Wake-Word + Fast Routing 🔨 In progress
|
||||
|
||||
**Fast routing — DONE & WIRED:** `src/openhuman/voice/command_router.rs` (new) — pure `route(transcript) -> VoiceIntent` classifier (Play/Pause/Resume/Next/Previous/OpenApp/SetVolume/VolumeUp·Down/Mute/Unmute, else `Unknown`). High-confidence intents execute directly (launch_app / Music fast-path / osascript volume) without a full chat-LLM turn — the ≤500 ms path; `Unknown` defers to the agent so routing only ever shortcuts. Filler-tolerant ("please open up slack"). 5 unit tests.
|
||||
|
||||
**Router wired into delivery:** `always_on::deliver_command` now routes the extracted command through `command_router::route` first. Recognized intents run locally via `execute_intent` (Play → `automate` Music fast-path; OpenApp → `launch_platform`; Pause/Resume/Next/Previous/volume/mute → `osa` osascript); `VoiceIntent::Unknown` (and any local-exec failure) falls back to the full agent turn. Verified live: clean transport/open/volume commands execute on the fast path; complex/garbled `play` queries (e.g. mangled STT like "bts s latest song wind blowing") fall through to the agent as designed. Committed on `feat/voice-phase3-fast-routing`.
|
||||
|
||||
**Remaining Phase 3:** on-device audio wake-word model (Porcupine / ONNX) in `inference/voice/wake_word.rs` to gate STT *before* transcription — the text-based "Hey Tiny" wake match from Phase 2 (fuzzy anchor on the "tiny" token, Levenshtein ≤1) is the interim. This is the only open Phase 3 item; not yet started.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Wake-Word + Fast Routing (original plan)
|
||||
|
||||
> Activate only on a trigger phrase; route simple commands locally without a full LLM turn.
|
||||
|
||||
**Planned files:**
|
||||
- `src/openhuman/inference/voice/wake_word.rs` (new) — lightweight always-on model (Porcupine or custom ONNX)
|
||||
- `src/openhuman/voice/command_router.rs` (new) — intent→tool mapping for high-confidence commands, LLM fallback for ambiguous input
|
||||
- `src/openhuman/inference/voice/wake_word.rs` (new) — lightweight always-on model (Porcupine or custom ONNX) — ⏳ **not started** (interim: text-based "Hey Tiny" match in Phase 2)
|
||||
- `src/openhuman/voice/command_router.rs` (new) — intent→tool mapping for high-confidence commands, LLM fallback for ambiguous input — ✅ **shipped & wired** (see Phase 3 status above)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Wake-word detection runs fully on-device
|
||||
- [ ] Latency from end-of-utterance to action start ≤ 500ms for local-routed commands
|
||||
- [ ] Wake-word detection runs fully on-device (still pending — audio wake-word model not yet built)
|
||||
- [x] Latency from end-of-utterance to action start ≤ 500ms for local-routed commands (command router executes recognized intents on the local fast path)
|
||||
|
||||
---
|
||||
|
||||
@@ -607,19 +635,23 @@ From live agent-in-the-loop testing on 2026-06-03 (grounded in `~/.openhuman/log
|
||||
| 1 | AXUIElement app UI interaction (`ax_interact`) | ✅ Done |
|
||||
| 1 | Multi-step UI workflow guidance | ✅ Done |
|
||||
| 1 | Apple Music two-step play (navigate→play) | ✅ Done (playback best-effort) |
|
||||
| 1 | `automate(app, goal)` Rust-driven loop (Change 1.14) | 🔨 M1+M2+M3 done (37 tests; live proof pending) |
|
||||
| 1 | Full computer control (mouse/keyboard/screenshot) — CEF crash (Change 1.15) | ✅ Fixed (main-thread synthetic-input dispatch; screenshot downscale) |
|
||||
| 1 | Windows port (`launch_app` + UIA `ax_interact`) | ✅ Done (Calculator + Notepad hard-asserted on Win11) |
|
||||
| 1 | `automate(app, goal)` Rust-driven loop (Change 1.14) | ✅ M1–M5 done (proven live on macOS) |
|
||||
| 1.5 | M1: automate loop skeleton + tool | ✅ Done |
|
||||
| 1.5 | M2: poll-until-stable settle | ✅ Done |
|
||||
| 1.5 | M3: Music fast-path | ✅ Done (proven live on macOS) |
|
||||
| 1.5 | Robustness: quoted-query parse + no-progress guard | ✅ Done (from live agent failures) |
|
||||
| 1.5 | M4: progress streaming to notch | ✅ Done — notch cherry-picked in; automate streams live steps |
|
||||
| 1.5 | M5: richer element model (`enabled`) | ✅ Plumbed; AXEnabled found unreliable → informational only |
|
||||
| 1.5 | Native fast-paths (Music/Spotify/Slack) | ⏳ Not started |
|
||||
| 1.5 | Vision fallback for Electron apps | ⏳ Not started |
|
||||
| 1.5 | Native fast-paths beyond Music (Spotify/Slack) | ➖ Descoped (general loop covers them; Music shipped in M3) |
|
||||
| 1.5 | Vision fallback for Electron apps (Change 1.16) | ✅ Done (`vision_click`: screenshot → vision-locate → guarded click; frontmost guard; F2 coord-transform folded in) |
|
||||
| 2 | Always-on microphone loop | ✅ Done (cpal → VAD → STT → agent) |
|
||||
| 2 | `always_on_enabled` config flag + Settings toggle | ✅ Done (RPC + UI + i18n) |
|
||||
| 2 | Privacy hook (screen lock pause) | ✅ Done (macOS; other OSes follow-up) |
|
||||
| 3 | Wake-word detection | ⏳ Not started |
|
||||
| 3 | Local command router | ⏳ Not started |
|
||||
| 2 | Text-based "Hey Tiny" wake word | ✅ Done (interim; gates delivery, strips phrase) |
|
||||
| 3 | Local command router (intent classifier) | ✅ Done & wired (recognized intents run on the ≤500ms local path; Unknown defers to agent) |
|
||||
| 3 | On-device audio wake-word model | ⏳ Not started (text-based match is the interim) |
|
||||
| 4 | Voice confirmation loop | ⏳ Not started |
|
||||
| 4 | Computer-control onboarding toggle | ⏳ Not started |
|
||||
| 4 | Always-on UI indicator | ✅ Done (notch PR #3166) |
|
||||
|
||||
@@ -68,6 +68,9 @@ pub struct Action {
|
||||
/// Text to enter for `set_value`.
|
||||
#[serde(default)]
|
||||
pub value: String,
|
||||
/// Natural-language target for `vision_click` (e.g. "the green Call button").
|
||||
#[serde(default)]
|
||||
pub description: String,
|
||||
/// Final message for `done` / `fail`.
|
||||
#[serde(default)]
|
||||
pub summary: String,
|
||||
@@ -116,6 +119,38 @@ pub trait AutomateBackend: Send + Sync {
|
||||
async fn verify_playing(&self) -> Option<bool> {
|
||||
None
|
||||
}
|
||||
/// Capture the target app's window + the geometry needed to map a click
|
||||
/// from image pixels to screen points. Used by the `vision_click` fallback
|
||||
/// for apps with no usable accessibility tree (Electron/Chromium). Default
|
||||
/// errors so backends without screen access (tests, headless) opt out.
|
||||
async fn screenshot(
|
||||
&self,
|
||||
_app: &str,
|
||||
) -> Result<(String, super::vision_click::CaptureGeometry), String> {
|
||||
Err("screenshot unsupported by this backend".to_string())
|
||||
}
|
||||
/// Ask the vision model for the absolute *screen* coordinates of the
|
||||
/// described element in `screenshot`. `Ok(None)` = not visible. Default
|
||||
/// `Ok(None)` so non-vision backends never click.
|
||||
async fn locate(
|
||||
&self,
|
||||
_screenshot: &str,
|
||||
_geom: &super::vision_click::CaptureGeometry,
|
||||
_description: &str,
|
||||
) -> Result<Option<(i32, i32)>, String> {
|
||||
Ok(None)
|
||||
}
|
||||
/// Name of the frontmost application, if known. Used as the §1.8 safety
|
||||
/// guard: a `vision_click` only fires when the target app is frontmost, so
|
||||
/// synthetic input never lands on OpenHuman's own window. `None` = unknown.
|
||||
async fn frontmost_app(&self) -> Option<String> {
|
||||
None
|
||||
}
|
||||
/// Issue a single guarded left-click at absolute screen coordinates. Default
|
||||
/// errors so backends without input access can't click.
|
||||
async fn click(&self, _x: i32, _y: i32) -> Result<String, String> {
|
||||
Err("click unsupported by this backend".to_string())
|
||||
}
|
||||
/// Block until the UI settles after an action.
|
||||
async fn settle(&self, app: &str);
|
||||
/// Wait ~`ms` of real time. Used by fast-paths to let asynchronous content
|
||||
@@ -155,6 +190,10 @@ fn system_prompt() -> String {
|
||||
• list — re-read elements; set `filter` to a substring to narrow them\n\
|
||||
• press — activate the element whose label matches `label`\n\
|
||||
• set_value — type `value` into the field matching `label` (omit label = first field)\n\
|
||||
• vision_click — click an element by sight; put a short `description` of the \
|
||||
target (e.g. 'the green Call button'). Use this when the element list is \
|
||||
EMPTY or missing your target — common for Electron apps (Slack, Discord, \
|
||||
VS Code) that expose no accessibility tree.\n\
|
||||
• done — goal achieved; put a short result in `summary`\n\
|
||||
• fail — goal cannot be achieved; explain in `summary`\n\
|
||||
\n\
|
||||
@@ -164,6 +203,8 @@ fn system_prompt() -> String {
|
||||
(e.g. open a song, THEN press its 'Play'). After such a press, `list` again \
|
||||
to see the new screen.\n\
|
||||
- Prefer an exact label match. Keep `filter` specific so the snapshot stays small.\n\
|
||||
- If the app shows NO elements, prefer `vision_click` with a clear \
|
||||
`description` over guessing labels.\n\
|
||||
- Output JSON only — no prose, no code fences."
|
||||
.to_string()
|
||||
}
|
||||
@@ -319,7 +360,10 @@ pub async fn run(
|
||||
|
||||
// ── no-progress guard ──
|
||||
if !matches!(action.action.as_str(), "done" | "fail") {
|
||||
let sig = format!("{}|{}|{}", action.action, action.label, action.filter);
|
||||
let sig = format!(
|
||||
"{}|{}|{}|{}",
|
||||
action.action, action.label, action.filter, action.description
|
||||
);
|
||||
if sig == last_sig {
|
||||
repeat_count += 1;
|
||||
} else {
|
||||
@@ -411,6 +455,69 @@ pub async fn run(
|
||||
}
|
||||
backend.settle(target_app).await;
|
||||
}
|
||||
"vision_click" => {
|
||||
let description = action.description.trim();
|
||||
if description.is_empty() {
|
||||
steps.push("vision_click skipped: empty description".to_string());
|
||||
continue;
|
||||
}
|
||||
// ── §1.8 safety guard ──
|
||||
// Only click when the target app is frontmost, so synthetic
|
||||
// input never lands on OpenHuman's own window (the CEF crash).
|
||||
// `None` = can't tell → proceed best-effort (the loop already
|
||||
// foregrounded the app at start). We only REFUSE on positive
|
||||
// evidence that a different app is focused.
|
||||
if let Some(front) = backend.frontmost_app().await {
|
||||
if !front.eq_ignore_ascii_case(target_app) {
|
||||
log::warn!(
|
||||
"{LOG_PREFIX} vision_click: {target_app:?} not frontmost ({front:?}); re-foregrounding"
|
||||
);
|
||||
let _ = backend.act_launch(target_app).await;
|
||||
backend.settle(target_app).await;
|
||||
let still_wrong = backend
|
||||
.frontmost_app()
|
||||
.await
|
||||
.map(|f| !f.eq_ignore_ascii_case(target_app))
|
||||
.unwrap_or(false);
|
||||
if still_wrong {
|
||||
steps.push(format!(
|
||||
"vision_click refused: {target_app} is not frontmost"
|
||||
));
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
progress(
|
||||
format!("Looking for {description}…"),
|
||||
OverlayAttentionTone::Accent,
|
||||
);
|
||||
let (shot, geom) = match backend.screenshot(target_app).await {
|
||||
Ok(pair) => pair,
|
||||
Err(e) => {
|
||||
steps.push(format!("vision_click FAILED: screenshot: {e}"));
|
||||
continue;
|
||||
}
|
||||
};
|
||||
match backend.locate(&shot, &geom, description).await {
|
||||
Ok(Some((x, y))) => {
|
||||
progress(
|
||||
format!("Clicking {description}…"),
|
||||
OverlayAttentionTone::Accent,
|
||||
);
|
||||
match backend.click(x, y).await {
|
||||
Ok(msg) => steps.push(format!("vision_click: {msg}")),
|
||||
Err(e) => steps.push(format!("vision_click FAILED: click: {e}")),
|
||||
}
|
||||
backend.settle(target_app).await;
|
||||
}
|
||||
Ok(None) => {
|
||||
steps.push(format!("vision_click: '{description}' not found on screen"));
|
||||
}
|
||||
Err(e) => {
|
||||
steps.push(format!("vision_click FAILED: locate: {e}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
other => {
|
||||
steps.push(format!("unknown action {other:?} ignored"));
|
||||
}
|
||||
@@ -470,6 +577,60 @@ impl AutomateBackend for RealBackend {
|
||||
ax::ax_set_field_value(app, label, value)
|
||||
}
|
||||
|
||||
async fn screenshot(
|
||||
&self,
|
||||
app: &str,
|
||||
) -> Result<(String, super::vision_click::CaptureGeometry), String> {
|
||||
// Capture whatever window is frontmost — the loop guarantees the target
|
||||
// is frontmost before a vision_click, so this resolves to its window.
|
||||
let ctx = super::foreground_context()
|
||||
.ok_or_else(|| "could not resolve the foreground window for capture".to_string())?;
|
||||
if let Some(name) = ctx.app_name.as_deref() {
|
||||
if !name.eq_ignore_ascii_case(app) {
|
||||
log::warn!(
|
||||
"{LOG_PREFIX} screenshot: frontmost {name:?} != target {app:?}; capturing frontmost"
|
||||
);
|
||||
}
|
||||
}
|
||||
// `capture_window_geometry` shells out to `screencapture` (blocking).
|
||||
match tokio::task::spawn_blocking(move || {
|
||||
super::vision_click::capture_window_geometry(&ctx)
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(inner) => inner,
|
||||
Err(e) => Err(format!("capture task join failed: {e}")),
|
||||
}
|
||||
}
|
||||
|
||||
async fn locate(
|
||||
&self,
|
||||
screenshot: &str,
|
||||
geom: &super::vision_click::CaptureGeometry,
|
||||
description: &str,
|
||||
) -> Result<Option<(i32, i32)>, String> {
|
||||
// Use the main `chat` provider's vision model (per plan): reliable UI
|
||||
// grounding, and the fallback only fires when AX is empty (rare).
|
||||
let (provider, model) =
|
||||
crate::openhuman::inference::provider::create_chat_provider("chat", &self.config)
|
||||
.map_err(|e| format!("vision provider unavailable: {e}"))?;
|
||||
let coords =
|
||||
super::vision_click::locate_via_vision(&*provider, &model, screenshot, description)
|
||||
.await?;
|
||||
Ok(coords.map(|(px, py)| super::vision_click::image_to_screen(geom, px, py)))
|
||||
}
|
||||
|
||||
async fn frontmost_app(&self) -> Option<String> {
|
||||
tokio::task::spawn_blocking(|| super::foreground_context().and_then(|c| c.app_name))
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
}
|
||||
|
||||
async fn click(&self, x: i32, y: i32) -> Result<String, String> {
|
||||
super::vision_click::guarded_click(x, y).await
|
||||
}
|
||||
|
||||
async fn open_url(&self, url: &str) -> Result<String, String> {
|
||||
// Cross-platform URI opener. macOS `open`, Linux `xdg-open`, Windows
|
||||
// `cmd /C start`. Only invoked by fast-paths with app-controlled URLs
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
//! with no mic, no AX tree, and no LLM.
|
||||
|
||||
use super::*;
|
||||
use crate::openhuman::accessibility::vision_click::CaptureGeometry;
|
||||
use std::sync::Mutex;
|
||||
|
||||
/// Scripted backend: `decide` returns the next queued response each call;
|
||||
@@ -16,6 +17,10 @@ struct ScriptedBackend {
|
||||
acts: Mutex<Vec<String>>,
|
||||
/// Force act_press to error (to exercise the failure-recording path).
|
||||
press_errors: bool,
|
||||
/// What `frontmost_app` returns (the §1.8 guard input). `None` = unknown.
|
||||
frontmost: Option<String>,
|
||||
/// What `locate` returns: `Some` screen coords = found; `None` = not found.
|
||||
locate_coord: Option<(i32, i32)>,
|
||||
}
|
||||
|
||||
impl ScriptedBackend {
|
||||
@@ -28,11 +33,36 @@ impl ScriptedBackend {
|
||||
],
|
||||
acts: Mutex::new(Vec::new()),
|
||||
press_errors: false,
|
||||
frontmost: None,
|
||||
locate_coord: None,
|
||||
}
|
||||
}
|
||||
fn acts(&self) -> Vec<String> {
|
||||
self.acts.lock().unwrap().clone()
|
||||
}
|
||||
/// Set the frontmost app name reported to the `vision_click` guard.
|
||||
fn with_frontmost(mut self, app: &str) -> Self {
|
||||
self.frontmost = Some(app.to_string());
|
||||
self
|
||||
}
|
||||
/// Make `locate` report the element found at the given screen coords.
|
||||
fn with_located(mut self, x: i32, y: i32) -> Self {
|
||||
self.locate_coord = Some((x, y));
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
/// A throwaway geometry for the scripted `screenshot` — tests override `locate`
|
||||
/// directly, so the transform isn't exercised here (it has its own unit tests).
|
||||
fn dummy_geom() -> CaptureGeometry {
|
||||
CaptureGeometry {
|
||||
rect_x: 0,
|
||||
rect_y: 0,
|
||||
rect_w_pts: 1,
|
||||
rect_h_pts: 1,
|
||||
img_w_px: 1,
|
||||
img_h_px: 1,
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
@@ -71,6 +101,29 @@ impl AutomateBackend for ScriptedBackend {
|
||||
.push(format!("set_value:{app}:{label}={value}"));
|
||||
Ok(format!("Set '{label}' in '{app}'."))
|
||||
}
|
||||
async fn screenshot(&self, app: &str) -> Result<(String, CaptureGeometry), String> {
|
||||
self.acts.lock().unwrap().push(format!("screenshot:{app}"));
|
||||
Ok(("data:image/png;base64,TEST".to_string(), dummy_geom()))
|
||||
}
|
||||
async fn locate(
|
||||
&self,
|
||||
_shot: &str,
|
||||
_geom: &CaptureGeometry,
|
||||
description: &str,
|
||||
) -> Result<Option<(i32, i32)>, String> {
|
||||
self.acts
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(format!("locate:{description}"));
|
||||
Ok(self.locate_coord)
|
||||
}
|
||||
async fn frontmost_app(&self) -> Option<String> {
|
||||
self.frontmost.clone()
|
||||
}
|
||||
async fn click(&self, x: i32, y: i32) -> Result<String, String> {
|
||||
self.acts.lock().unwrap().push(format!("click:{x},{y}"));
|
||||
Ok(format!("Clicked at ({x}, {y})"))
|
||||
}
|
||||
async fn open_url(&self, url: &str) -> Result<String, String> {
|
||||
self.acts.lock().unwrap().push(format!("open_url:{url}"));
|
||||
Ok(format!("Opened {url}"))
|
||||
@@ -264,3 +317,96 @@ fn render_snapshot_empty_hint() {
|
||||
let s = render_snapshot("Music", "zzz", &[]);
|
||||
assert!(s.contains("no elements"));
|
||||
}
|
||||
|
||||
// ── vision_click fallback ────────────────────────────────────────────────────
|
||||
|
||||
#[tokio::test]
|
||||
async fn vision_click_locates_and_clicks_when_frontmost() {
|
||||
let backend = ScriptedBackend::new(&[
|
||||
r#"{"action":"vision_click","description":"the Call button"}"#,
|
||||
r#"{"action":"done","summary":"clicked"}"#,
|
||||
])
|
||||
.with_frontmost("Slack")
|
||||
.with_located(640, 360);
|
||||
let out = run("Slack", "click the call button", &backend, opts(5)).await;
|
||||
assert!(out.success, "{out:?}");
|
||||
let acts = backend.acts();
|
||||
assert!(acts.contains(&"screenshot:Slack".to_string()), "{acts:?}");
|
||||
assert!(
|
||||
acts.contains(&"locate:the Call button".to_string()),
|
||||
"{acts:?}"
|
||||
);
|
||||
assert!(acts.contains(&"click:640,360".to_string()), "{acts:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn vision_click_proceeds_when_frontmost_unknown() {
|
||||
// `None` frontmost (e.g. can't determine) is best-effort: the loop already
|
||||
// foregrounded the app, so we still click.
|
||||
let backend = ScriptedBackend::new(&[
|
||||
r#"{"action":"vision_click","description":"X"}"#,
|
||||
r#"{"action":"done"}"#,
|
||||
])
|
||||
.with_located(10, 20);
|
||||
let out = run("Slack", "x", &backend, opts(5)).await;
|
||||
assert!(out.success);
|
||||
assert!(backend.acts().contains(&"click:10,20".to_string()));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn vision_click_refused_when_other_app_frontmost() {
|
||||
// Positive evidence a different app is focused → refuse (the §1.8 guard).
|
||||
let backend = ScriptedBackend::new(&[
|
||||
r#"{"action":"vision_click","description":"the Call button"}"#,
|
||||
r#"{"action":"done","summary":"done"}"#,
|
||||
])
|
||||
.with_frontmost("Finder")
|
||||
.with_located(640, 360);
|
||||
let out = run("Slack", "click call", &backend, opts(5)).await;
|
||||
let acts = backend.acts();
|
||||
assert!(
|
||||
!acts.iter().any(|a| a.starts_with("click:")),
|
||||
"must not click into a non-target app: {acts:?}"
|
||||
);
|
||||
assert!(
|
||||
!acts.iter().any(|a| a.starts_with("screenshot:")),
|
||||
"must not even screenshot when refused: {acts:?}"
|
||||
);
|
||||
let _ = out;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn vision_click_not_found_does_not_click() {
|
||||
let backend = ScriptedBackend::new(&[
|
||||
r#"{"action":"vision_click","description":"the Call button"}"#,
|
||||
r#"{"action":"done","summary":"gave up"}"#,
|
||||
])
|
||||
.with_frontmost("Slack"); // locate_coord stays None → not found
|
||||
let out = run("Slack", "click call", &backend, opts(5)).await;
|
||||
assert!(out.success);
|
||||
let acts = backend.acts();
|
||||
assert!(acts.contains(&"screenshot:Slack".to_string()), "{acts:?}");
|
||||
assert!(
|
||||
acts.contains(&"locate:the Call button".to_string()),
|
||||
"{acts:?}"
|
||||
);
|
||||
assert!(
|
||||
!acts.iter().any(|a| a.starts_with("click:")),
|
||||
"no click when the element isn't found: {acts:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn vision_click_empty_description_skipped() {
|
||||
let backend = ScriptedBackend::new(&[
|
||||
r#"{"action":"vision_click","description":" "}"#,
|
||||
r#"{"action":"done"}"#,
|
||||
])
|
||||
.with_frontmost("Slack");
|
||||
let out = run("Slack", "x", &backend, opts(5)).await;
|
||||
assert!(out.success);
|
||||
assert!(
|
||||
!backend.acts().iter().any(|a| a.starts_with("screenshot:")),
|
||||
"empty description must be skipped before any capture"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -20,6 +20,9 @@ mod permissions;
|
||||
mod terminal;
|
||||
mod text_util;
|
||||
mod types;
|
||||
// Vision fallback for `automate`: screenshot → vision-locate → guarded click,
|
||||
// for Electron/partial-AX apps. Consumed by `automate.rs`'s `RealBackend`.
|
||||
mod vision_click;
|
||||
// Windows accessibility backend for `ax_interact` (UI Automation). Sibling of
|
||||
// the macOS Swift-helper path; selected via cfg-dispatch in `ax_interact.rs`.
|
||||
#[cfg(target_os = "windows")]
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
//! Vision fallback for `automate`: click a described on-screen element when an
|
||||
//! app exposes no usable accessibility tree.
|
||||
//!
|
||||
//! Electron/Chromium apps (Slack, Discord, VS Code) expose little or no AX/UIA,
|
||||
//! so the perceive→press loop has nothing to act on (tracker §1.5). The answer
|
||||
//! is *screenshot → vision-locate → guarded click*:
|
||||
//!
|
||||
//! 1. **screenshot** the target app's window (`super::capture`), recording the
|
||||
//! window's screen rect so pixels can be mapped back to screen points.
|
||||
//! 2. **locate** — ask the main vision model for the PIXEL coordinates of the
|
||||
//! described element, passing the image via the provider's `[IMAGE:<uri>]`
|
||||
//! marker (promoted to a real image part in
|
||||
//! `inference/provider/compatible_types.rs`, #3205).
|
||||
//! 3. **map** image pixels → absolute screen points ([`image_to_screen`]).
|
||||
//! 4. **guarded click** — only ever issued by the caller once the target app
|
||||
//! is frontmost, never into OpenHuman's own window (the §1.8 CEF-crash
|
||||
//! guard). Synthetic input runs on the app main thread via
|
||||
//! `run_input_on_main` (Change 1.15 — off-thread enigo traps TSM).
|
||||
//!
|
||||
//! The coordinate transform and the model-response parser are pure and unit
|
||||
//! tested; the capture / model / click side-effects sit behind them.
|
||||
|
||||
use super::types::{AppContext, ElementBounds};
|
||||
|
||||
const LOG_PREFIX: &str = "[vision_click]";
|
||||
|
||||
/// Geometry needed to map a point in the (possibly downscaled) screenshot back
|
||||
/// to absolute screen coordinates.
|
||||
///
|
||||
/// `rect_*` is the captured window's screen rect in **points** (from
|
||||
/// [`AppContext::bounds`]); `img_*` is the screenshot's **pixel** size. The
|
||||
/// pixel→point ratio absorbs both the capture downscale *and* the Retina backing
|
||||
/// scale, so the mapping needs no explicit scale factor (this is what closes the
|
||||
/// deferred F2 coordinate-mapping gap).
|
||||
// `pub` (not `pub(crate)`) so it doesn't read as "more private" than the
|
||||
// `pub AutomateBackend` trait methods that name it; the enclosing `mod
|
||||
// vision_click` is private, so it stays crate-internal in practice.
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
pub struct CaptureGeometry {
|
||||
pub rect_x: i32,
|
||||
pub rect_y: i32,
|
||||
pub rect_w_pts: i32,
|
||||
pub rect_h_pts: i32,
|
||||
pub img_w_px: u32,
|
||||
pub img_h_px: u32,
|
||||
}
|
||||
|
||||
impl CaptureGeometry {
|
||||
fn from_bounds(b: &ElementBounds, img_w_px: u32, img_h_px: u32) -> Self {
|
||||
Self {
|
||||
rect_x: b.x,
|
||||
rect_y: b.y,
|
||||
rect_w_pts: b.width,
|
||||
rect_h_pts: b.height,
|
||||
img_w_px,
|
||||
img_h_px,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Map an image-pixel coordinate to an absolute screen point.
|
||||
///
|
||||
/// Clamps the input to the image and the output to the window rect, so a model
|
||||
/// that returns an out-of-range guess still lands inside the *target* window —
|
||||
/// never elsewhere on screen (and never on OpenHuman's own window).
|
||||
pub(crate) fn image_to_screen(geom: &CaptureGeometry, px: i32, py: i32) -> (i32, i32) {
|
||||
let img_w = geom.img_w_px.max(1) as f64;
|
||||
let img_h = geom.img_h_px.max(1) as f64;
|
||||
// Clamp the sampled pixel into the image, then express it as a 0..1 fraction.
|
||||
let fx = (px.max(0) as f64).min(img_w - 1.0) / img_w;
|
||||
let fy = (py.max(0) as f64).min(img_h - 1.0) / img_h;
|
||||
let sx = geom.rect_x as f64 + fx * geom.rect_w_pts as f64;
|
||||
let sy = geom.rect_y as f64 + fy * geom.rect_h_pts as f64;
|
||||
// Keep the result strictly inside the window rect.
|
||||
let max_x = geom.rect_x + (geom.rect_w_pts - 1).max(0);
|
||||
let max_y = geom.rect_y + (geom.rect_h_pts - 1).max(0);
|
||||
(
|
||||
(sx.round() as i32).clamp(geom.rect_x, max_x),
|
||||
(sy.round() as i32).clamp(geom.rect_y, max_y),
|
||||
)
|
||||
}
|
||||
|
||||
/// The JSON the vision model must return: whether the element was found and its
|
||||
/// center pixel coordinates within the screenshot.
|
||||
#[derive(Debug, serde::Deserialize)]
|
||||
struct LocateResponse {
|
||||
#[serde(default)]
|
||||
found: bool,
|
||||
#[serde(default)]
|
||||
x: i32,
|
||||
#[serde(default)]
|
||||
y: i32,
|
||||
}
|
||||
|
||||
/// System prompt pinning the locate contract for the vision model.
|
||||
fn locate_system_prompt() -> &'static str {
|
||||
"You locate a single UI element in a screenshot of one application window. \
|
||||
You are given the image and a description of the element to click. Reply \
|
||||
with EXACTLY ONE JSON object and nothing else: \
|
||||
{\"found\":true,\"x\":<int>,\"y\":<int>} where x,y are the PIXEL \
|
||||
coordinates (origin top-left) of the CENTER of that element within the \
|
||||
image. If the element is not visible, reply {\"found\":false,\"x\":0,\"y\":0}. \
|
||||
Output JSON only — no prose, no code fences."
|
||||
}
|
||||
|
||||
/// Build the user turn: the target description plus the screenshot as a
|
||||
/// `[IMAGE:<data-uri>]` marker the compatible provider promotes to an image part.
|
||||
fn build_locate_user(description: &str, screenshot_data_uri: &str) -> String {
|
||||
format!("Element to click: {description}\n[IMAGE:{screenshot_data_uri}]")
|
||||
}
|
||||
|
||||
/// Parse the model's locate reply, tolerating code fences / surrounding prose by
|
||||
/// extracting the first balanced `{...}` (mirrors `automate::parse_action`).
|
||||
/// `found:false` → `Ok(None)`; unparseable → `Err` so the caller can report it
|
||||
/// rather than act on a hallucinated guess (tracker §1.13 lesson).
|
||||
fn parse_locate_response(raw: &str) -> Result<Option<(i32, i32)>, String> {
|
||||
let trimmed = raw.trim();
|
||||
let parsed = serde_json::from_str::<LocateResponse>(trimmed).or_else(|_| {
|
||||
match (trimmed.find('{'), trimmed.rfind('}')) {
|
||||
(Some(s), Some(e)) if e > s => serde_json::from_str::<LocateResponse>(&trimmed[s..=e]),
|
||||
// Re-run on the trimmed text so the error type matches the arm.
|
||||
_ => serde_json::from_str::<LocateResponse>(trimmed),
|
||||
}
|
||||
});
|
||||
match parsed {
|
||||
Ok(r) if r.found => Ok(Some((r.x, r.y))),
|
||||
Ok(_) => Ok(None),
|
||||
Err(_) => Err(format!("could not parse locate response: {trimmed:?}")),
|
||||
}
|
||||
}
|
||||
|
||||
/// Decode the pixel dimensions of a `data:image/...;base64,...` URI.
|
||||
fn image_dims_from_data_uri(data_uri: &str) -> Result<(u32, u32), String> {
|
||||
use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _};
|
||||
use image::GenericImageView as _;
|
||||
let b64 = data_uri
|
||||
.split_once(',')
|
||||
.map(|(_, payload)| payload)
|
||||
.ok_or_else(|| "malformed image data URI (no base64 payload)".to_string())?;
|
||||
let bytes = BASE64_STANDARD
|
||||
.decode(b64)
|
||||
.map_err(|e| format!("screenshot base64 decode failed: {e}"))?;
|
||||
let img =
|
||||
image::load_from_memory(&bytes).map_err(|e| format!("screenshot decode failed: {e}"))?;
|
||||
Ok(img.dimensions())
|
||||
}
|
||||
|
||||
/// Capture the app window and pair it with the geometry needed to map a click.
|
||||
///
|
||||
/// Requires `ctx.bounds` (the window's screen rect) — without it the image
|
||||
/// pixels can't be mapped to screen points, so we refuse rather than click blind.
|
||||
pub(crate) fn capture_window_geometry(
|
||||
ctx: &AppContext,
|
||||
) -> Result<(String, CaptureGeometry), String> {
|
||||
let bounds = ctx
|
||||
.bounds
|
||||
.ok_or_else(|| "window bounds unavailable — cannot map click coordinates".to_string())?;
|
||||
let data_uri = super::capture::capture_screen_image_ref_for_context(Some(ctx))?;
|
||||
let (w, h) = image_dims_from_data_uri(&data_uri)?;
|
||||
log::debug!(
|
||||
"{LOG_PREFIX} captured window rect=({},{},{},{}) image={}x{}px",
|
||||
bounds.x,
|
||||
bounds.y,
|
||||
bounds.width,
|
||||
bounds.height,
|
||||
w,
|
||||
h
|
||||
);
|
||||
Ok((data_uri, CaptureGeometry::from_bounds(&bounds, w, h)))
|
||||
}
|
||||
|
||||
/// Ask the vision model for the target's pixel coordinates within `screenshot`.
|
||||
/// Returns `Ok(None)` when the model reports the element isn't visible.
|
||||
pub(crate) async fn locate_via_vision(
|
||||
provider: &dyn crate::openhuman::inference::provider::Provider,
|
||||
model: &str,
|
||||
screenshot_data_uri: &str,
|
||||
description: &str,
|
||||
) -> Result<Option<(i32, i32)>, String> {
|
||||
let user = build_locate_user(description, screenshot_data_uri);
|
||||
let raw = provider
|
||||
.chat_with_system(Some(locate_system_prompt()), &user, model, 0.0)
|
||||
.await
|
||||
.map_err(|e| format!("vision model call failed: {e}"))?;
|
||||
log::debug!("{LOG_PREFIX} locate raw response: {raw:?}");
|
||||
parse_locate_response(&raw)
|
||||
}
|
||||
|
||||
/// Single guarded left-click at absolute screen coordinates, run on the app
|
||||
/// main thread. Off-thread enigo traps macOS TSM and crashes the CEF host
|
||||
/// (Change 1.15 / §1.8) — so the click closure is dispatched via
|
||||
/// `run_input_on_main`, exactly like the `mouse` tool.
|
||||
pub(crate) async fn guarded_click(x: i32, y: i32) -> Result<String, String> {
|
||||
use crate::openhuman::tools::implementations::run_input_on_main;
|
||||
log::info!("{LOG_PREFIX} ▶ click at screen ({x}, {y})");
|
||||
run_input_on_main(move || {
|
||||
use enigo::{Button, Coordinate, Direction, Enigo, Mouse, Settings};
|
||||
let mut enigo =
|
||||
Enigo::new(&Settings::default()).map_err(|e| format!("enigo init failed: {e}"))?;
|
||||
enigo
|
||||
.move_mouse(x, y, Coordinate::Abs)
|
||||
.map_err(|e| format!("move_mouse failed: {e}"))?;
|
||||
enigo
|
||||
.button(Button::Left, Direction::Click)
|
||||
.map_err(|e| format!("click failed: {e}"))?;
|
||||
Ok(format!("Clicked at ({x}, {y})"))
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "vision_click_tests.rs"]
|
||||
mod tests;
|
||||
@@ -0,0 +1,148 @@
|
||||
//! Unit tests for the vision-click primitives. The coordinate transform and the
|
||||
//! locate-response parser are pure, so they're exercised with no screen, no
|
||||
//! model, and no synthetic input.
|
||||
|
||||
use super::*;
|
||||
|
||||
fn geom(
|
||||
rect_x: i32,
|
||||
rect_y: i32,
|
||||
rect_w_pts: i32,
|
||||
rect_h_pts: i32,
|
||||
img_w_px: u32,
|
||||
img_h_px: u32,
|
||||
) -> CaptureGeometry {
|
||||
CaptureGeometry {
|
||||
rect_x,
|
||||
rect_y,
|
||||
rect_w_pts,
|
||||
rect_h_pts,
|
||||
img_w_px,
|
||||
img_h_px,
|
||||
}
|
||||
}
|
||||
|
||||
// ── image_to_screen ─────────────────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn maps_center_when_image_matches_rect() {
|
||||
// Image pixels == window points (no scaling): center maps to center.
|
||||
let g = geom(0, 0, 1000, 800, 1000, 800);
|
||||
assert_eq!(image_to_screen(&g, 500, 400), (500, 400));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn maps_through_downscaled_screenshot() {
|
||||
// Capture was downscaled to half size; a center pixel still maps to the
|
||||
// window center in screen points.
|
||||
let g = geom(0, 0, 1000, 800, 500, 400);
|
||||
assert_eq!(image_to_screen(&g, 250, 200), (500, 400));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn maps_through_retina_2x_backing_scale() {
|
||||
// Retina capture is 2× the window's point size; the px→pt ratio absorbs it.
|
||||
let g = geom(0, 0, 1000, 800, 2000, 1600);
|
||||
assert_eq!(image_to_screen(&g, 1000, 800), (500, 400));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn applies_window_origin_offset() {
|
||||
// A window not at the screen origin: image coords are window-relative, the
|
||||
// result is absolute screen coords.
|
||||
let g = geom(100, 50, 400, 300, 400, 300);
|
||||
assert_eq!(image_to_screen(&g, 200, 150), (300, 200));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clamps_out_of_range_pixel_into_window() {
|
||||
// A wild guess past the image bounds still lands strictly inside the window.
|
||||
let g = geom(100, 50, 400, 300, 400, 300);
|
||||
let (x, y) = image_to_screen(&g, 99_999, 99_999);
|
||||
assert_eq!((x, y), (499, 349)); // rect_x + w - 1, rect_y + h - 1
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clamps_negative_pixel_to_window_origin() {
|
||||
let g = geom(100, 50, 400, 300, 400, 300);
|
||||
assert_eq!(image_to_screen(&g, -10, -10), (100, 50));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn handles_zero_image_dims_without_panicking() {
|
||||
// Defensive: a degenerate (0px) image must not divide by zero.
|
||||
let g = geom(0, 0, 100, 100, 0, 0);
|
||||
let (x, y) = image_to_screen(&g, 10, 10);
|
||||
assert!((0..100).contains(&x) && (0..100).contains(&y));
|
||||
}
|
||||
|
||||
// ── parse_locate_response ────────────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn parses_found_coordinates() {
|
||||
let r = parse_locate_response(r#"{"found":true,"x":120,"y":340}"#).unwrap();
|
||||
assert_eq!(r, Some((120, 340)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn not_found_is_none() {
|
||||
let r = parse_locate_response(r#"{"found":false,"x":0,"y":0}"#).unwrap();
|
||||
assert_eq!(r, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tolerates_code_fences() {
|
||||
let raw = "```json\n{\"found\":true,\"x\":5,\"y\":6}\n```";
|
||||
assert_eq!(parse_locate_response(raw).unwrap(), Some((5, 6)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tolerates_surrounding_prose() {
|
||||
let raw = "Sure! Here it is: {\"found\":true,\"x\":7,\"y\":8} — hope that helps";
|
||||
assert_eq!(parse_locate_response(raw).unwrap(), Some((7, 8)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn garbage_is_error() {
|
||||
assert!(parse_locate_response("no json here").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_found_defaults_to_not_found() {
|
||||
// `found` defaults to false when absent, so an answer without it is None.
|
||||
let r = parse_locate_response(r#"{"x":1,"y":2}"#).unwrap();
|
||||
assert_eq!(r, None);
|
||||
}
|
||||
|
||||
// ── build_locate_user ────────────────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn user_turn_embeds_description_and_image_marker() {
|
||||
let user = build_locate_user("the green Call button", "data:image/png;base64,AAA");
|
||||
assert!(user.contains("the green Call button"));
|
||||
assert!(user.contains("[IMAGE:data:image/png;base64,AAA]"));
|
||||
}
|
||||
|
||||
// ── image_dims_from_data_uri ─────────────────────────────────────────────────
|
||||
|
||||
fn png_data_uri(w: u32, h: u32) -> String {
|
||||
use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _};
|
||||
let img = image::DynamicImage::ImageRgb8(image::RgbImage::new(w, h));
|
||||
let mut buf = std::io::Cursor::new(Vec::new());
|
||||
img.write_to(&mut buf, image::ImageFormat::Png).unwrap();
|
||||
format!(
|
||||
"data:image/png;base64,{}",
|
||||
BASE64_STANDARD.encode(buf.get_ref())
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reads_png_dimensions() {
|
||||
let uri = png_data_uri(640, 480);
|
||||
assert_eq!(image_dims_from_data_uri(&uri).unwrap(), (640, 480));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_malformed_data_uri() {
|
||||
assert!(image_dims_from_data_uri("not-a-data-uri").is_err());
|
||||
}
|
||||
Reference in New Issue
Block a user