mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
Feat/overlay (#378)
* feat(autocomplete): add overlay TTL configuration to AutocompletePanel - Introduced `overlay_ttl_ms` parameter to the Autocomplete configuration, allowing users to set the overlay display duration. - Updated AutocompletePanel to include a new input field for adjusting the overlay TTL in milliseconds. - Enhanced parsing and saving logic to handle the new configuration parameter. - Added corresponding tests to ensure functionality and validate the new overlay TTL feature. This update improves user control over the autocomplete overlay behavior, enhancing the overall user experience. * Refactor accessibility code for improved readability and consistency - Simplified log statements in `precompile_helper_background` for better clarity. - Reformatted `detect_input_monitoring_permission` check in `keys.rs` for enhanced readability. - Rearranged imports in `mod.rs` to maintain consistent structure. - Improved formatting of `ElementBounds` initialization across multiple test cases in `overlay.rs` and `types.rs` for better visual alignment. - Enhanced test context creation in `types.rs` for improved clarity. These changes enhance code maintainability and readability across the accessibility module. * fix(overlay): parent core RPC, voice toggle, and debug for #342 - Pass OPENHUMAN_OVERLAY_PARENT_RPC_URL from sidecar spawn and strip inherited OPENHUMAN_CORE_PORT so the overlay no longer fights for the parent listen port. - Overlay UI uses HTTP JSON-RPC to the parent sidecar for globe, debug, and voice STT so state matches the main app; add parentCoreRpc helper mirroring legacy method aliases. - Skip embedded JSON-RPC server when parent URL is set; use OPENHUMAN_OVERLAY_EMBEDDED_CORE_PORT (default 7799) for standalone dev. - Fix screen intelligence status method name; add voice_status polling, STT section, collapsible debug summary, and connection banner when core is unreachable. - Voice capture switch gates the mic with real STT availability from voice_status. Closes #342 Made-with: Cursor * fix: address CodeRabbit review on overlay/autocomplete (PR #378) - helper: correlate JSON-RPC replies with monotonic request ids; discard mismatched lines until deadline (fixes stale response after timeout). - helper: serialize Swift compile via HELPER_COMPILE_LOCK (precompile vs first use). - Overlay Tab hint: pass tab_hint from Rust from accept_with_tab; Swift hides hint when empty. - keys: re-check Input Monitoring on an interval when denied so grant without restart works. - engine: use non-zero confidence placeholder (0.75) until inline_complete returns scores. - overlay dedupe: suppress identical badge only within 400ms, not for process lifetime. Co-authored-by: Code review feedback <noreply@github.com> Made-with: Cursor * fix(ci): resolve clippy and warning issues for Rust gates - Use match on anchor_bounds in autocomplete overlay (avoid unnecessary unwrap) - Drop unused test imports in registry_ops and rpc dispatch - Prefix unused notion_doc_id in subconscious integration test Made-with: Cursor * fix(overlay): improve parent RPC URL handling in App component - Updated the useEffect hook to manage the parent RPC URL more robustly by introducing a mounted flag to prevent state updates on unmounted components. - Added error handling to set the parent RPC URL to null in case of invocation failure, enhancing the reliability of the component's behavior. * feat(overlay): implement timeout handling for parent core RPC requests - Introduced a default timeout for parent core RPC requests, enhancing reliability by preventing indefinite waiting for responses. - Added an AbortController to manage request timeouts, throwing a specific error message when a timeout occurs. - Updated the `callParentCoreRpc` function to accept a customizable timeout parameter, improving flexibility for RPC calls. * fix(overlay): allow stopping active recording regardless of config state - Updated the main button handler in the App component to always permit stopping an active recording when the status is "listening", improving user experience and control over the recording process. - Removed redundant code that previously checked the status before stopping the recording, streamlining the logic. * feat(overlay): update Cargo.lock with new dependencies and versions - Added new packages including `alsa`, `alsa-sys`, `arboard`, `block`, `cocoa`, `core-foundation`, `core-graphics`, `coreaudio-rs`, `coreaudio-sys`, `cpal`, `crunchy`, and `dasp_sample` to enhance functionality and support for audio processing and system interactions. - Updated existing dependencies to their latest versions for improved performance and compatibility. - Modified the `show_overlay` function in `ops.rs` to include an additional parameter, enhancing the overlay display functionality. * feat(autocomplete): add overlay_ttl_ms parameter to Autocomplete interfaces - Introduced a new optional parameter `overlay_ttl_ms` to both `AutocompleteSetStyleParams` and `AutocompleteConfig` interfaces, allowing for customizable overlay timeout settings. - This enhancement improves the flexibility of the autocomplete feature by enabling developers to specify how long the overlay should remain visible. --------- Co-authored-by: Code review feedback <noreply@github.com> Co-authored-by: Steven Enamakel <enamakel@tinyhumans.ai>
This commit is contained in:
co-authored by
Code review feedback
Steven Enamakel
parent
e8fd08c800
commit
8627cee960
@@ -28,6 +28,7 @@ const DEFAULT_CONFIG: AutocompleteConfig = {
|
||||
style_examples: [],
|
||||
disabled_apps: [],
|
||||
accept_with_tab: true,
|
||||
overlay_ttl_ms: 1100,
|
||||
};
|
||||
|
||||
const MAX_LOG_ENTRIES = 200;
|
||||
@@ -56,6 +57,10 @@ const parseAutocompleteConfig = (raw: unknown): AutocompleteConfig => {
|
||||
typeof value.accept_with_tab === 'boolean'
|
||||
? value.accept_with_tab
|
||||
: DEFAULT_CONFIG.accept_with_tab,
|
||||
overlay_ttl_ms:
|
||||
typeof value.overlay_ttl_ms === 'number'
|
||||
? value.overlay_ttl_ms
|
||||
: DEFAULT_CONFIG.overlay_ttl_ms,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -77,6 +82,7 @@ const AutocompletePanel = () => {
|
||||
DEFAULT_CONFIG.disabled_apps.join('\n')
|
||||
);
|
||||
const [acceptWithTab, setAcceptWithTab] = useState<boolean>(DEFAULT_CONFIG.accept_with_tab);
|
||||
const [overlayTtlMs, setOverlayTtlMs] = useState<string>(String(DEFAULT_CONFIG.overlay_ttl_ms));
|
||||
const [contextOverride, setContextOverride] = useState<string>('');
|
||||
const [focusDebug, setFocusDebug] = useState<string>('');
|
||||
const [logs, setLogs] = useState<string[]>([]);
|
||||
@@ -153,6 +159,7 @@ const AutocompletePanel = () => {
|
||||
setStyleExamplesText(config.style_examples.join('\n'));
|
||||
setDisabledAppsText(config.disabled_apps.join('\n'));
|
||||
setAcceptWithTab(config.accept_with_tab);
|
||||
setOverlayTtlMs(String(config.overlay_ttl_ms));
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to load autocomplete settings');
|
||||
} finally {
|
||||
@@ -253,6 +260,7 @@ const AutocompletePanel = () => {
|
||||
appendUiLog('saving autocomplete settings');
|
||||
const debounce = Number(debounceMs);
|
||||
const max = Number(maxChars);
|
||||
const ttl = Number(overlayTtlMs);
|
||||
const response = await openhumanAutocompleteSetStyle({
|
||||
enabled,
|
||||
debounce_ms: Number.isFinite(debounce) ? Math.min(Math.max(debounce, 50), 2000) : 120,
|
||||
@@ -268,6 +276,7 @@ const AutocompletePanel = () => {
|
||||
.map(entry => entry.trim())
|
||||
.filter(Boolean),
|
||||
accept_with_tab: acceptWithTab,
|
||||
overlay_ttl_ms: Number.isFinite(ttl) ? Math.min(Math.max(ttl, 300), 10000) : 1100,
|
||||
});
|
||||
|
||||
setEnabled(response.result.config.enabled);
|
||||
@@ -278,6 +287,7 @@ const AutocompletePanel = () => {
|
||||
setStyleExamplesText(response.result.config.style_examples.join('\n'));
|
||||
setDisabledAppsText(response.result.config.disabled_apps.join('\n'));
|
||||
setAcceptWithTab(response.result.config.accept_with_tab);
|
||||
setOverlayTtlMs(String(response.result.config.overlay_ttl_ms));
|
||||
appendLogs(response.logs);
|
||||
setMessage('Autocomplete settings saved.');
|
||||
await refreshStatus();
|
||||
@@ -507,6 +517,18 @@ const AutocompletePanel = () => {
|
||||
className="w-28 rounded border border-stone-300 bg-white px-2 py-1 text-xs text-stone-700"
|
||||
/>
|
||||
</label>
|
||||
<label className="flex items-center justify-between rounded-xl border border-stone-200 bg-stone-50 px-3 py-2">
|
||||
<span className="text-sm text-stone-700">Overlay TTL (ms)</span>
|
||||
<input
|
||||
type="number"
|
||||
min={300}
|
||||
max={10000}
|
||||
step={100}
|
||||
value={overlayTtlMs}
|
||||
onChange={event => setOverlayTtlMs(event.target.value)}
|
||||
className="w-28 rounded border border-stone-300 bg-white px-2 py-1 text-xs text-stone-700"
|
||||
/>
|
||||
</label>
|
||||
<label className="flex items-center justify-between rounded-xl border border-stone-200 bg-stone-50 px-3 py-2">
|
||||
<span className="text-sm text-stone-700">Style Preset</span>
|
||||
<select
|
||||
|
||||
@@ -81,6 +81,7 @@ describe('AutocompletePanel', () => {
|
||||
style_examples: [],
|
||||
disabled_apps: [],
|
||||
accept_with_tab: true,
|
||||
overlay_ttl_ms: 1100,
|
||||
},
|
||||
};
|
||||
|
||||
|
||||
@@ -78,6 +78,7 @@ export interface AutocompleteSetStyleParams {
|
||||
style_examples?: string[];
|
||||
disabled_apps?: string[];
|
||||
accept_with_tab?: boolean;
|
||||
overlay_ttl_ms?: number;
|
||||
}
|
||||
|
||||
export interface AutocompleteConfig {
|
||||
@@ -89,6 +90,7 @@ export interface AutocompleteConfig {
|
||||
style_examples: string[];
|
||||
disabled_apps: string[];
|
||||
accept_with_tab: boolean;
|
||||
overlay_ttl_ms: number;
|
||||
}
|
||||
|
||||
export interface AutocompleteSetStyleResult {
|
||||
|
||||
Generated
+677
-78
File diff suppressed because it is too large
Load Diff
@@ -3,9 +3,9 @@ mod log_bridge;
|
||||
use log_bridge::{LogBuffer, LogEntry, TauriLogLayer};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
use tauri::Manager;
|
||||
#[cfg(target_os = "macos")]
|
||||
use tauri::ActivationPolicy;
|
||||
use tauri::Manager;
|
||||
use tracing_subscriber::layer::SubscriberExt;
|
||||
use tracing_subscriber::util::SubscriberInitExt;
|
||||
use tracing_subscriber::EnvFilter;
|
||||
@@ -40,6 +40,19 @@ fn set_click_through(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// JSON-RPC URL of the desktop core sidecar, when the overlay was spawned by it.
|
||||
/// When set, the web UI should prefer HTTP `fetch` to this URL so autocomplete,
|
||||
/// screen intelligence, and voice state match the main app (see `overlay/src/parentCoreRpc.ts`).
|
||||
#[tauri::command]
|
||||
fn overlay_parent_rpc_url() -> Option<String> {
|
||||
let url = std::env::var("OPENHUMAN_OVERLAY_PARENT_RPC_URL").ok()?;
|
||||
let trimmed = url.trim();
|
||||
if trimmed.is_empty() {
|
||||
return None;
|
||||
}
|
||||
Some(trimmed.to_string())
|
||||
}
|
||||
|
||||
/// Forward an RPC call to openhuman_core's dispatch in-process.
|
||||
/// Uses the same invoke_method path as the HTTP JSON-RPC server.
|
||||
#[tauri::command]
|
||||
@@ -76,6 +89,7 @@ pub fn run() {
|
||||
.invoke_handler(tauri::generate_handler![
|
||||
get_log_history,
|
||||
set_click_through,
|
||||
overlay_parent_rpc_url,
|
||||
core_rpc,
|
||||
insert_text_into_focused_field,
|
||||
])
|
||||
@@ -112,20 +126,33 @@ pub fn run() {
|
||||
|
||||
log::info!("[overlay] overlay process started, tracing bridge active");
|
||||
|
||||
// ── Start openhuman_core JSON-RPC server in-process ─────────
|
||||
// Use port 7799 to avoid conflicts with a standalone core on 7788.
|
||||
// Override with OPENHUMAN_CORE_PORT env var.
|
||||
tauri::async_runtime::spawn(async move {
|
||||
let port = std::env::var("OPENHUMAN_CORE_PORT")
|
||||
.ok()
|
||||
.and_then(|p| p.parse::<u16>().ok())
|
||||
.unwrap_or(7799);
|
||||
log::info!("[overlay] starting openhuman_core server on 127.0.0.1:{}...", port);
|
||||
match openhuman_core::core::jsonrpc::run_server(None, Some(port), true).await {
|
||||
Ok(()) => log::info!("[overlay] core server shut down cleanly"),
|
||||
Err(e) => log::error!("[overlay] core server error: {}", e),
|
||||
}
|
||||
});
|
||||
// ── Optional in-process JSON-RPC (standalone / dev without a parent core) ──
|
||||
// When spawned by the desktop sidecar, OPENHUMAN_OVERLAY_PARENT_RPC_URL is set and
|
||||
// the web UI talks to the parent over HTTP — do not bind a second server on 7788.
|
||||
// Use OPENHUMAN_OVERLAY_EMBEDDED_CORE_PORT (default 7799), not OPENHUMAN_CORE_PORT.
|
||||
let parent_rpc = std::env::var("OPENHUMAN_OVERLAY_PARENT_RPC_URL")
|
||||
.ok()
|
||||
.filter(|s| !s.trim().is_empty());
|
||||
if parent_rpc.is_some() {
|
||||
log::info!(
|
||||
"[overlay] parent core RPC URL set — skipping embedded JSON-RPC server"
|
||||
);
|
||||
} else {
|
||||
tauri::async_runtime::spawn(async move {
|
||||
let port = std::env::var("OPENHUMAN_OVERLAY_EMBEDDED_CORE_PORT")
|
||||
.ok()
|
||||
.and_then(|p| p.parse::<u16>().ok())
|
||||
.unwrap_or(7799);
|
||||
log::info!(
|
||||
"[overlay] starting embedded openhuman_core server on 127.0.0.1:{} (standalone)",
|
||||
port
|
||||
);
|
||||
match openhuman_core::core::jsonrpc::run_server_embedded(None, Some(port), true).await {
|
||||
Ok(()) => log::info!("[overlay] embedded core server shut down cleanly"),
|
||||
Err(e) => log::error!("[overlay] embedded core server error: {}", e),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// ── macOS: floating panel + visible on all workspaces ───────
|
||||
#[cfg(target_os = "macos")]
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
"label": "overlay",
|
||||
"title": "",
|
||||
"width": 376,
|
||||
"height": 432,
|
||||
"height": 520,
|
||||
"minWidth": 376,
|
||||
"minHeight": 432,
|
||||
"transparent": true,
|
||||
|
||||
+211
-41
@@ -2,6 +2,8 @@ import { invoke } from "@tauri-apps/api/core";
|
||||
import { getCurrentWindow } from "@tauri-apps/api/window";
|
||||
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
|
||||
import { callParentCoreRpc } from "./parentCoreRpc";
|
||||
|
||||
const TARGET_SAMPLE_RATE = 16000;
|
||||
|
||||
type OverlayStatus = "idle" | "listening" | "transcribing" | "ready" | "error";
|
||||
@@ -64,13 +66,30 @@ interface AutocompleteStatus {
|
||||
suggestion: AutocompleteSuggestion | null;
|
||||
}
|
||||
|
||||
/** Matches `VoiceStatus` in src/openhuman/voice/types.rs */
|
||||
interface VoiceStatus {
|
||||
stt_available: boolean;
|
||||
tts_available: boolean;
|
||||
stt_model_id: string;
|
||||
tts_voice_id: string;
|
||||
whisper_binary: string | null;
|
||||
piper_binary: string | null;
|
||||
stt_model_path: string | null;
|
||||
tts_voice_path: string | null;
|
||||
whisper_in_process: boolean;
|
||||
llm_cleanup_enabled: boolean;
|
||||
}
|
||||
|
||||
interface OverlayDebugSnapshot {
|
||||
screen: AccessibilityStatus | null;
|
||||
autocomplete: AutocompleteStatus | null;
|
||||
voice: VoiceStatus | null;
|
||||
updatedAt: number | null;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
const DEBUG_EXPANDED_KEY = "openhuman_overlay_debug_expanded";
|
||||
|
||||
function logOverlay(message: string, details?: unknown) {
|
||||
if (details) {
|
||||
console.debug(`[overlay] ${message}`, details);
|
||||
@@ -205,16 +224,67 @@ export function App() {
|
||||
const sessionIdRef = useRef(0);
|
||||
const globePollInFlightRef = useRef(false);
|
||||
|
||||
/** `undefined` until Tauri reports env; then URL string or `null` (embedded core only). */
|
||||
const [parentRpcUrl, setParentRpcUrl] = useState<string | null | undefined>(undefined);
|
||||
const [coreReachable, setCoreReachable] = useState(true);
|
||||
const [voiceCaptureEnabled, setVoiceCaptureEnabled] = useState(true);
|
||||
const [debugExpanded, setDebugExpanded] = useState(() => {
|
||||
try {
|
||||
return typeof localStorage !== "undefined" && localStorage.getItem(DEBUG_EXPANDED_KEY) === "1";
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
});
|
||||
|
||||
const [status, setStatus] = useState<OverlayStatus>("idle");
|
||||
const [message, setMessage] = useState("Click to start listening");
|
||||
const [transcript, setTranscript] = useState("");
|
||||
const [debugSnapshot, setDebugSnapshot] = useState<OverlayDebugSnapshot>({
|
||||
screen: null,
|
||||
autocomplete: null,
|
||||
voice: null,
|
||||
updatedAt: null,
|
||||
error: null,
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
let mounted = true;
|
||||
void invoke<string | null>("overlay_parent_rpc_url")
|
||||
.then((url) => {
|
||||
if (!mounted) return;
|
||||
const trimmed = url?.trim();
|
||||
setParentRpcUrl(trimmed && trimmed.length > 0 ? trimmed : null);
|
||||
})
|
||||
.catch(() => {
|
||||
if (mounted) setParentRpcUrl(null);
|
||||
});
|
||||
return () => {
|
||||
mounted = false;
|
||||
};
|
||||
}, []);
|
||||
|
||||
const rpc = useCallback(
|
||||
async <T,>(method: string, params: Record<string, unknown> = {}): Promise<T> => {
|
||||
if (parentRpcUrl === undefined) {
|
||||
throw new Error("[overlay] RPC not initialized");
|
||||
}
|
||||
if (parentRpcUrl) {
|
||||
return callParentCoreRpc<T>(parentRpcUrl, method, params);
|
||||
}
|
||||
return invoke<T>("core_rpc", { method, params });
|
||||
},
|
||||
[parentRpcUrl],
|
||||
);
|
||||
|
||||
const persistDebugExpanded = useCallback((expanded: boolean) => {
|
||||
setDebugExpanded(expanded);
|
||||
try {
|
||||
localStorage.setItem(DEBUG_EXPANDED_KEY, expanded ? "1" : "0");
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
let disposed = false;
|
||||
|
||||
@@ -228,11 +298,11 @@ export function App() {
|
||||
};
|
||||
|
||||
const startGlobeListener = async () => {
|
||||
if (parentRpcUrl === undefined) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const result = await invoke<GlobeHotkeyStatus>("core_rpc", {
|
||||
method: "openhuman.screen_intelligence_globe_listener_start",
|
||||
params: {},
|
||||
});
|
||||
const result = await rpc<GlobeHotkeyStatus>("openhuman.screen_intelligence_globe_listener_start", {});
|
||||
logOverlay("globe listener start result", result);
|
||||
|
||||
if (!result.supported) {
|
||||
@@ -252,16 +322,13 @@ export function App() {
|
||||
};
|
||||
|
||||
const pollGlobeListener = async () => {
|
||||
if (disposed || globePollInFlightRef.current) {
|
||||
if (disposed || parentRpcUrl === undefined || globePollInFlightRef.current) {
|
||||
return;
|
||||
}
|
||||
globePollInFlightRef.current = true;
|
||||
|
||||
try {
|
||||
const result = await invoke<GlobeHotkeyPollResult>("core_rpc", {
|
||||
method: "openhuman.screen_intelligence_globe_listener_poll",
|
||||
params: {},
|
||||
});
|
||||
const result = await rpc<GlobeHotkeyPollResult>("openhuman.screen_intelligence_globe_listener_poll", {});
|
||||
|
||||
if (disposed) {
|
||||
return;
|
||||
@@ -297,33 +364,43 @@ export function App() {
|
||||
return () => {
|
||||
disposed = true;
|
||||
window.clearInterval(intervalId);
|
||||
void invoke("core_rpc", {
|
||||
method: "openhuman.screen_intelligence_globe_listener_stop",
|
||||
params: {},
|
||||
}).catch(() => {});
|
||||
if (parentRpcUrl === undefined) {
|
||||
return;
|
||||
}
|
||||
void rpc("openhuman.screen_intelligence_globe_listener_stop", {}).catch(() => {});
|
||||
};
|
||||
}, [appWindow]);
|
||||
}, [appWindow, parentRpcUrl, rpc]);
|
||||
|
||||
useEffect(() => {
|
||||
let disposed = false;
|
||||
let pollInFlight = false;
|
||||
|
||||
const pollDebugState = async () => {
|
||||
if (disposed || pollInFlight) {
|
||||
if (disposed || parentRpcUrl === undefined || pollInFlight) {
|
||||
return;
|
||||
}
|
||||
pollInFlight = true;
|
||||
|
||||
try {
|
||||
const [screen, autocomplete] = await Promise.all([
|
||||
invoke<AccessibilityStatus>("core_rpc", {
|
||||
method: "openhuman.accessibility_status",
|
||||
params: {},
|
||||
}),
|
||||
invoke<AutocompleteStatus>("core_rpc", {
|
||||
method: "openhuman.autocomplete_status",
|
||||
params: {},
|
||||
}),
|
||||
if (parentRpcUrl) {
|
||||
try {
|
||||
await rpc<{ ok?: boolean }>("core.ping", {});
|
||||
if (!disposed) {
|
||||
setCoreReachable(true);
|
||||
}
|
||||
} catch {
|
||||
if (!disposed) {
|
||||
setCoreReachable(false);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
setCoreReachable(true);
|
||||
}
|
||||
|
||||
const [screen, autocomplete, voice] = await Promise.all([
|
||||
rpc<AccessibilityStatus>("openhuman.screen_intelligence_status", {}),
|
||||
rpc<AutocompleteStatus>("openhuman.autocomplete_status", {}),
|
||||
rpc<VoiceStatus>("openhuman.voice_status", {}),
|
||||
]);
|
||||
|
||||
if (disposed) {
|
||||
@@ -335,11 +412,13 @@ export function App() {
|
||||
captureCount: screen.session.capture_count,
|
||||
autocompletePhase: autocomplete.phase,
|
||||
hasSuggestion: Boolean(autocomplete.suggestion?.value),
|
||||
sttAvailable: voice.stt_available,
|
||||
});
|
||||
|
||||
setDebugSnapshot({
|
||||
screen,
|
||||
autocomplete,
|
||||
voice,
|
||||
updatedAt: Date.now(),
|
||||
error: null,
|
||||
});
|
||||
@@ -370,7 +449,7 @@ export function App() {
|
||||
disposed = true;
|
||||
window.clearInterval(intervalId);
|
||||
};
|
||||
}, []);
|
||||
}, [parentRpcUrl, rpc]);
|
||||
|
||||
const insertTranscriptIntoFocusedField = useCallback(
|
||||
async (text: string) => {
|
||||
@@ -404,13 +483,10 @@ export function App() {
|
||||
async (blob: Blob, sessionId: number) => {
|
||||
try {
|
||||
const audioBytes = await convertBlobToWavBytes(blob);
|
||||
const result = await invoke<TranscribeResult>("core_rpc", {
|
||||
method: "openhuman.voice_transcribe_bytes",
|
||||
params: {
|
||||
audio_bytes: audioBytes,
|
||||
extension: "wav",
|
||||
skip_cleanup: false,
|
||||
},
|
||||
const result = await rpc<TranscribeResult>("openhuman.voice_transcribe_bytes", {
|
||||
audio_bytes: audioBytes,
|
||||
extension: "wav",
|
||||
skip_cleanup: false,
|
||||
});
|
||||
|
||||
if (sessionIdRef.current !== sessionId) {
|
||||
@@ -444,7 +520,7 @@ export function App() {
|
||||
setMessage(error instanceof Error ? error.message : "Transcription failed");
|
||||
}
|
||||
},
|
||||
[insertTranscriptIntoFocusedField],
|
||||
[insertTranscriptIntoFocusedField, rpc],
|
||||
);
|
||||
|
||||
const stopRecording = useCallback(() => {
|
||||
@@ -521,15 +597,33 @@ export function App() {
|
||||
}, [cleanupStream, transcribeBlob]);
|
||||
|
||||
const handleMainButton = useCallback(() => {
|
||||
// Always allow stopping an active recording, regardless of config state.
|
||||
if (status === "listening") {
|
||||
logOverlay("main button toggled to stop listening");
|
||||
stopRecording();
|
||||
return;
|
||||
}
|
||||
|
||||
if (!voiceCaptureEnabled) {
|
||||
setMessage("Turn on voice capture to use the microphone");
|
||||
return;
|
||||
}
|
||||
if (debugSnapshot.voice && !debugSnapshot.voice.stt_available) {
|
||||
setMessage(
|
||||
"Speech-to-text is not available. Configure Local AI / voice in the main OpenHuman app.",
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
logOverlay("main button toggled to start listening", { priorStatus: status });
|
||||
void startRecording();
|
||||
}, [startRecording, status, stopRecording]);
|
||||
}, [
|
||||
debugSnapshot.voice,
|
||||
startRecording,
|
||||
status,
|
||||
stopRecording,
|
||||
voiceCaptureEnabled,
|
||||
]);
|
||||
|
||||
const shellClassName = useMemo(() => {
|
||||
if (status === "listening") {
|
||||
@@ -560,6 +654,11 @@ export function App() {
|
||||
const autocompleteRunning =
|
||||
debugSnapshot.autocomplete?.running && debugSnapshot.autocomplete?.enabled;
|
||||
|
||||
const sttAvailable = debugSnapshot.voice?.stt_available ?? true;
|
||||
const voiceBlocked =
|
||||
!voiceCaptureEnabled || (debugSnapshot.voice !== null && !debugSnapshot.voice.stt_available);
|
||||
const waitingForCoreConfig = parentRpcUrl === undefined;
|
||||
|
||||
return (
|
||||
<div className="flex h-screen w-screen items-start justify-start bg-transparent p-3">
|
||||
<div className="relative select-none">
|
||||
@@ -579,6 +678,13 @@ export function App() {
|
||||
void appWindow.startDragging();
|
||||
}}
|
||||
>
|
||||
{parentRpcUrl && !coreReachable ? (
|
||||
<div className="mb-2 rounded-2xl border border-amber-400/35 bg-amber-950/35 px-3 py-2 text-[11px] leading-4 text-amber-50">
|
||||
Cannot reach the OpenHuman core at the sidecar URL. Autocomplete and screen debug may
|
||||
be stale. Check that the main app is running.
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="mb-3 flex items-center justify-between gap-2">
|
||||
<span className="rounded-full bg-black/15 px-2 py-1 text-[10px] font-semibold uppercase tracking-[0.24em]">
|
||||
Voice
|
||||
@@ -593,11 +699,45 @@ export function App() {
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="mb-3 flex items-center justify-between gap-2 rounded-2xl border border-white/10 bg-black/15 px-3 py-2">
|
||||
<div className="min-w-0">
|
||||
<p className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-80">
|
||||
Voice capture
|
||||
</p>
|
||||
<p className="mt-0.5 text-[11px] leading-4 opacity-85">
|
||||
{waitingForCoreConfig
|
||||
? "Checking core…"
|
||||
: sttAvailable
|
||||
? debugSnapshot.voice?.whisper_in_process
|
||||
? "STT ready (in-process)"
|
||||
: "STT ready"
|
||||
: "STT unavailable — configure voice in the main app"}
|
||||
</p>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={voiceCaptureEnabled}
|
||||
aria-label={voiceCaptureEnabled ? "Turn voice capture off" : "Turn voice capture on"}
|
||||
className={`relative h-8 w-[52px] shrink-0 rounded-full border border-white/15 transition ${
|
||||
voiceCaptureEnabled ? "bg-emerald-500/50" : "bg-black/35"
|
||||
}`}
|
||||
onClick={() => setVoiceCaptureEnabled((previous) => !previous)}
|
||||
>
|
||||
<span
|
||||
className={`absolute top-1 h-6 w-6 rounded-full bg-white shadow transition ${
|
||||
voiceCaptureEnabled ? "left-7" : "left-1"
|
||||
}`}
|
||||
/>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="flex items-start gap-3">
|
||||
<button
|
||||
type="button"
|
||||
onClick={handleMainButton}
|
||||
className={`group relative flex h-[108px] w-[108px] shrink-0 items-center justify-center rounded-full border border-white/20 bg-black/20 transition duration-200 hover:bg-black/28 ${
|
||||
disabled={waitingForCoreConfig || voiceBlocked}
|
||||
className={`group relative flex h-[108px] w-[108px] shrink-0 items-center justify-center rounded-full border border-white/20 bg-black/20 transition duration-200 hover:bg-black/28 disabled:cursor-not-allowed disabled:opacity-40 ${
|
||||
status === "listening" ? "scale-[1.02]" : ""
|
||||
}`}
|
||||
aria-label={status === "listening" ? "Stop listening" : "Start listening"}
|
||||
@@ -632,21 +772,50 @@ export function App() {
|
||||
|
||||
<div className="mt-3 rounded-[24px] border border-white/10 bg-black/15 p-3">
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<span className="text-[10px] font-semibold uppercase tracking-[0.22em] opacity-75">
|
||||
Debug
|
||||
</span>
|
||||
<span className="text-[10px] opacity-65">
|
||||
{debugSnapshot.updatedAt ? formatTimestamp(debugSnapshot.updatedAt) : "waiting"}
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
className="flex flex-1 items-center justify-between gap-2 text-left"
|
||||
onClick={() => persistDebugExpanded(!debugExpanded)}
|
||||
aria-expanded={debugExpanded}
|
||||
>
|
||||
<span className="text-[10px] font-semibold uppercase tracking-[0.22em] opacity-75">
|
||||
Debug {debugExpanded ? "▼" : "▶"}
|
||||
</span>
|
||||
<span className="text-[10px] opacity-65">
|
||||
{debugSnapshot.updatedAt ? formatTimestamp(debugSnapshot.updatedAt) : "waiting"}
|
||||
</span>
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{!debugExpanded ? (
|
||||
<p className="mt-2 text-[11px] leading-4 opacity-80">
|
||||
Screen: {debugSnapshot.screen?.session.active ? "session on" : "idle"} ·
|
||||
Autocomplete: {autocompletePhase}
|
||||
{debugSnapshot.autocomplete?.last_error ? " · error" : ""}
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
{debugSnapshot.error ? (
|
||||
<div className="mt-3 rounded-2xl border border-red-300/20 bg-red-950/20 px-3 py-2 text-[11px] leading-4 text-red-100">
|
||||
{debugSnapshot.error}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{debugExpanded ? (
|
||||
<div className="mt-3 grid gap-3">
|
||||
<section className="rounded-2xl border border-white/8 bg-black/10 px-3 py-2">
|
||||
<div className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-70">
|
||||
Voice / STT
|
||||
</div>
|
||||
<div className="mt-2 space-y-1 text-[11px] leading-4 opacity-90">
|
||||
<p>STT: {sttAvailable ? "available" : "unavailable"}</p>
|
||||
<p className="truncate">Model: {debugSnapshot.voice?.stt_model_id ?? "—"}</p>
|
||||
<p className="truncate">
|
||||
Whisper: {debugSnapshot.voice?.whisper_binary ?? "not found"}
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section className="rounded-2xl border border-white/8 bg-black/10 px-3 py-2">
|
||||
<div className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-70">
|
||||
Screen Intelligence
|
||||
@@ -704,6 +873,7 @@ export function App() {
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
/**
|
||||
* HTTP JSON-RPC to the desktop core sidecar (same process as the main app).
|
||||
* Mirrors the naming normalization in app/src/services/coreRpcClient.ts (subset).
|
||||
*/
|
||||
|
||||
let nextJsonRpcId = 1;
|
||||
|
||||
export function normalizeLegacyMethod(method: string): string {
|
||||
if (method.startsWith("openhuman.accessibility_")) {
|
||||
return method.replace("openhuman.accessibility_", "openhuman.screen_intelligence_");
|
||||
}
|
||||
return method;
|
||||
}
|
||||
|
||||
/** RpcOutcome with non-empty logs serializes as `{ result, logs }` in the core. */
|
||||
function unwrapCliCompatibleJson<T>(raw: unknown): T {
|
||||
if (
|
||||
raw !== null &&
|
||||
typeof raw === "object" &&
|
||||
"result" in raw &&
|
||||
"logs" in raw &&
|
||||
Array.isArray((raw as { logs: unknown }).logs)
|
||||
) {
|
||||
return (raw as { result: T }).result;
|
||||
}
|
||||
return raw as T;
|
||||
}
|
||||
|
||||
interface JsonRpcError {
|
||||
code: number;
|
||||
message: string;
|
||||
data?: unknown;
|
||||
}
|
||||
|
||||
interface JsonRpcResponse<T> {
|
||||
jsonrpc?: string;
|
||||
id?: number | string | null;
|
||||
result?: T;
|
||||
error?: JsonRpcError;
|
||||
}
|
||||
|
||||
/** Default timeout for parent core RPC requests (ms). */
|
||||
const DEFAULT_RPC_TIMEOUT_MS = 10_000;
|
||||
|
||||
export async function callParentCoreRpc<T>(
|
||||
rpcUrl: string,
|
||||
method: string,
|
||||
params: Record<string, unknown> = {},
|
||||
timeoutMs: number = DEFAULT_RPC_TIMEOUT_MS,
|
||||
): Promise<T> {
|
||||
const normalizedMethod = normalizeLegacyMethod(method);
|
||||
const payload = {
|
||||
jsonrpc: "2.0" as const,
|
||||
id: nextJsonRpcId++,
|
||||
method: normalizedMethod,
|
||||
params,
|
||||
};
|
||||
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
response = await fetch(rpcUrl, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(payload),
|
||||
signal: controller.signal,
|
||||
});
|
||||
} catch (err) {
|
||||
clearTimeout(timer);
|
||||
if (err instanceof DOMException && err.name === "AbortError") {
|
||||
throw new Error(`Core RPC request timed out after ${timeoutMs}ms (method: ${normalizedMethod})`);
|
||||
}
|
||||
throw err;
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
throw new Error(`Core RPC HTTP ${response.status}: ${text || response.statusText}`);
|
||||
}
|
||||
|
||||
const json = (await response.json()) as JsonRpcResponse<unknown>;
|
||||
|
||||
if (json.error) {
|
||||
throw new Error(json.error.message || "Core RPC returned an error");
|
||||
}
|
||||
if (!Object.prototype.hasOwnProperty.call(json, "result")) {
|
||||
throw new Error("Core RPC response missing result");
|
||||
}
|
||||
|
||||
return unwrapCliCompatibleJson<T>(json.result);
|
||||
}
|
||||
+52
-3
@@ -570,6 +570,29 @@ pub async fn run_server(
|
||||
host: Option<&str>,
|
||||
port: Option<u16>,
|
||||
socketio_enabled: bool,
|
||||
) -> anyhow::Result<()> {
|
||||
run_server_inner(host, port, socketio_enabled, false).await
|
||||
}
|
||||
|
||||
/// Like [`run_server`] but marks the instance as embedded — the overlay will
|
||||
/// **not** be auto-spawned, preventing recursive overlay launches.
|
||||
pub async fn run_server_embedded(
|
||||
host: Option<&str>,
|
||||
port: Option<u16>,
|
||||
socketio_enabled: bool,
|
||||
) -> anyhow::Result<()> {
|
||||
run_server_inner(host, port, socketio_enabled, true).await
|
||||
}
|
||||
|
||||
/// Internal server entrypoint.
|
||||
///
|
||||
/// When `embedded_core` is `true` the server skips auto-spawning the overlay
|
||||
/// (useful when the overlay itself hosts an embedded core to avoid recursion).
|
||||
async fn run_server_inner(
|
||||
host: Option<&str>,
|
||||
port: Option<u16>,
|
||||
socketio_enabled: bool,
|
||||
embedded_core: bool,
|
||||
) -> anyhow::Result<()> {
|
||||
// Ensure all controllers are registered before starting.
|
||||
let _ = all::all_registered_controllers();
|
||||
@@ -631,8 +654,31 @@ pub async fn run_server(
|
||||
log::info!("[rpc:socketio] disabled (--jsonrpc-only)");
|
||||
}
|
||||
|
||||
// Derive the real bound address from the listener so ephemeral port 0,
|
||||
// wildcard binds, and IPv6 are handled correctly.
|
||||
let bound_addr = listener.local_addr()?;
|
||||
let parent_core_rpc_url = {
|
||||
let (h, p) = (bound_addr.ip(), bound_addr.port());
|
||||
let effective_ip = if h.is_unspecified() {
|
||||
if h.is_ipv6() {
|
||||
std::net::IpAddr::V6(std::net::Ipv6Addr::LOCALHOST)
|
||||
} else {
|
||||
std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST)
|
||||
}
|
||||
} else {
|
||||
h
|
||||
};
|
||||
if effective_ip.is_ipv6() {
|
||||
format!("http://[{effective_ip}]:{p}/rpc")
|
||||
} else {
|
||||
format!("http://{effective_ip}:{p}/rpc")
|
||||
}
|
||||
};
|
||||
log::debug!("[core] parent_core_rpc_url = {parent_core_rpc_url}");
|
||||
|
||||
// Optional background bootstrap for local AI services.
|
||||
tokio::spawn(async {
|
||||
let is_embedded = embedded_core;
|
||||
tokio::spawn(async move {
|
||||
match crate::openhuman::config::Config::load_or_init().await {
|
||||
Ok(config) => {
|
||||
if config.local_ai.enabled {
|
||||
@@ -641,8 +687,11 @@ pub async fn run_server(
|
||||
}
|
||||
|
||||
// Launch the overlay Tauri app (transparent debug/voice panel) as a child process.
|
||||
if config.overlay_enabled {
|
||||
crate::openhuman::overlay::spawn_overlay();
|
||||
// Skip when running as an embedded core inside the overlay itself.
|
||||
if is_embedded {
|
||||
log::info!("[overlay] embedded core — skipping overlay auto-spawn");
|
||||
} else if config.overlay_enabled {
|
||||
crate::openhuman::overlay::spawn_overlay(parent_core_rpc_url.as_str());
|
||||
} else {
|
||||
log::info!("[overlay] overlay disabled by config (overlay_enabled = false)");
|
||||
}
|
||||
|
||||
@@ -2,71 +2,177 @@
|
||||
//!
|
||||
//! Replaces the separate osascript subprocess spawns and standalone overlay binary
|
||||
//! with a single persistent Swift process communicating via stdin/stdout JSON.
|
||||
//!
|
||||
//! ## Mutex architecture
|
||||
//!
|
||||
//! Three globals prevent deadlock between fire-and-forget (show/hide) and
|
||||
//! request-response (focus/paste) callers:
|
||||
//!
|
||||
//! - `UNIFIED_HELPER`: guards the process handle + stdin writer.
|
||||
//! Held only for the brief duration of a stdin write (~μs).
|
||||
//! - `RESPONSE_RX`: guards the mpsc receiver that the background reader
|
||||
//! thread populates. Held only for the duration of `recv_timeout`.
|
||||
//! - `RECV_SERIALISER`: held for the entire send+receive round-trip so that
|
||||
//! two callers cannot interleave their reads.
|
||||
//!
|
||||
//! Fire-and-forget callers never touch `RESPONSE_RX` or `RECV_SERIALISER`,
|
||||
//! so `show`/`hide` can proceed while a `focus` query is in-flight.
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
use once_cell::sync::Lazy;
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::io::{BufRead, BufReader, Write};
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::sync::Mutex as StdMutex;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::sync::{mpsc, Mutex as StdMutex};
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::time::{Duration, Instant};
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::{
|
||||
fs,
|
||||
path::PathBuf,
|
||||
process::{Child, ChildStdin, ChildStdout, Command, Stdio},
|
||||
process::{Child, ChildStdin, Command, Stdio},
|
||||
};
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
use serde_json::Value;
|
||||
|
||||
/// Process handle + stdin writer. Held only briefly for writes.
|
||||
#[cfg(target_os = "macos")]
|
||||
struct UnifiedHelperProcess {
|
||||
child: Child,
|
||||
stdin: ChildStdin,
|
||||
stdout: BufReader<ChildStdout>,
|
||||
}
|
||||
|
||||
/// Guards the process handle and stdin.
|
||||
#[cfg(target_os = "macos")]
|
||||
static UNIFIED_HELPER: Lazy<StdMutex<Option<UnifiedHelperProcess>>> =
|
||||
Lazy::new(|| StdMutex::new(None));
|
||||
|
||||
/// Channel receiver fed by the background stdout-reader thread.
|
||||
/// Separate from UNIFIED_HELPER so fire-and-forget callers never contend here.
|
||||
#[cfg(target_os = "macos")]
|
||||
static RESPONSE_RX: Lazy<StdMutex<Option<mpsc::Receiver<String>>>> =
|
||||
Lazy::new(|| StdMutex::new(None));
|
||||
|
||||
/// Serialises request/response pairs so two callers cannot interleave reads.
|
||||
/// Fire-and-forget callers never acquire this lock.
|
||||
#[cfg(target_os = "macos")]
|
||||
static RECV_SERIALISER: Lazy<StdMutex<()>> = Lazy::new(|| StdMutex::new(()));
|
||||
|
||||
/// Prevents concurrent Swift compiles from `ensure_helper_binary` vs background precompile.
|
||||
#[cfg(target_os = "macos")]
|
||||
static HELPER_COMPILE_LOCK: Lazy<StdMutex<()>> = Lazy::new(|| StdMutex::new(()));
|
||||
|
||||
/// Monotonic ids for `helper_send_receive` so a late line cannot be consumed as the wrong reply.
|
||||
#[cfg(target_os = "macos")]
|
||||
static HELPER_REQ_ID: AtomicU64 = AtomicU64::new(1);
|
||||
|
||||
/// Timeout for a single request/response round-trip with the Swift helper.
|
||||
#[cfg(target_os = "macos")]
|
||||
const HELPER_RECV_TIMEOUT: Duration = Duration::from_secs(8);
|
||||
|
||||
/// Send a JSON request and read a JSON response (one line each).
|
||||
/// Used for `focus` and `paste` commands that produce a response.
|
||||
///
|
||||
/// Holds `RECV_SERIALISER` for the full round-trip, but releases
|
||||
/// `UNIFIED_HELPER` before blocking on the channel recv, so fire-and-forget
|
||||
/// callers (`show`/`hide`) are never blocked by an in-flight focus query.
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) fn helper_send_receive(
|
||||
request: &serde_json::Value,
|
||||
) -> Result<serde_json::Value, String> {
|
||||
ensure_helper_running()?;
|
||||
let mut guard = UNIFIED_HELPER
|
||||
// Serialise request/response pairs — prevents interleaved reads.
|
||||
let _rr_guard = RECV_SERIALISER
|
||||
.lock()
|
||||
.map_err(|_| "unified helper lock poisoned".to_string())?;
|
||||
let helper = guard
|
||||
.as_mut()
|
||||
.ok_or_else(|| "unified helper unavailable".to_string())?;
|
||||
.map_err(|_| "recv serialiser lock poisoned".to_string())?;
|
||||
|
||||
// Write request
|
||||
let line = request.to_string();
|
||||
helper
|
||||
.stdin
|
||||
.write_all(line.as_bytes())
|
||||
.and_then(|_| helper.stdin.write_all(b"\n"))
|
||||
.and_then(|_| helper.stdin.flush())
|
||||
.map_err(|e| format!("failed to write to helper stdin: {e}"))?;
|
||||
ensure_helper_running()?;
|
||||
|
||||
// Read response (one line)
|
||||
let mut response_line = String::new();
|
||||
helper
|
||||
.stdout
|
||||
.read_line(&mut response_line)
|
||||
.map_err(|e| format!("failed to read helper stdout: {e}"))?;
|
||||
let id_num = HELPER_REQ_ID.fetch_add(1, Ordering::Relaxed);
|
||||
let id_str = id_num.to_string();
|
||||
|
||||
if response_line.trim().is_empty() {
|
||||
return Err("helper returned empty response".to_string());
|
||||
let mut req = request.clone();
|
||||
let req_obj = req
|
||||
.as_object_mut()
|
||||
.ok_or_else(|| "helper request must be a JSON object".to_string())?;
|
||||
req_obj.insert("id".to_string(), Value::String(id_str.clone()));
|
||||
|
||||
// Write the request, holding UNIFIED_HELPER only for this brief write.
|
||||
{
|
||||
let mut guard = UNIFIED_HELPER
|
||||
.lock()
|
||||
.map_err(|_| "unified helper lock poisoned".to_string())?;
|
||||
let helper = guard
|
||||
.as_mut()
|
||||
.ok_or_else(|| "unified helper unavailable".to_string())?;
|
||||
let line = req.to_string();
|
||||
helper
|
||||
.stdin
|
||||
.write_all(line.as_bytes())
|
||||
.and_then(|_| helper.stdin.write_all(b"\n"))
|
||||
.and_then(|_| helper.stdin.flush())
|
||||
.map_err(|e| format!("failed to write to helper stdin: {e}"))?;
|
||||
} // UNIFIED_HELPER released here — fire-and-forget callers can proceed
|
||||
|
||||
// Read until the line matches `id` (discards stale lines after a timeout or reordering).
|
||||
let deadline = Instant::now() + HELPER_RECV_TIMEOUT;
|
||||
loop {
|
||||
let remaining = deadline.saturating_duration_since(Instant::now());
|
||||
if remaining.is_zero() {
|
||||
return Err(format!(
|
||||
"helper response timed out waiting for id {id_str} (stale lines may follow)"
|
||||
));
|
||||
}
|
||||
let chunk = remaining.min(Duration::from_millis(500));
|
||||
let response_line = {
|
||||
let rx_guard = RESPONSE_RX
|
||||
.lock()
|
||||
.map_err(|_| "response rx lock poisoned".to_string())?;
|
||||
let rx = rx_guard
|
||||
.as_ref()
|
||||
.ok_or_else(|| "response channel unavailable".to_string())?;
|
||||
match rx.recv_timeout(chunk) {
|
||||
Ok(line) => line,
|
||||
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
|
||||
// Non-fatal: the outer loop will check `remaining` and
|
||||
// either retry or surface the top-level timeout.
|
||||
continue;
|
||||
}
|
||||
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
|
||||
return Err("helper response channel disconnected".to_string());
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
if response_line.trim().is_empty() {
|
||||
log::debug!(
|
||||
"[accessibility] helper skipped empty stdout line while waiting for id {id_str}"
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
let value: Value = serde_json::from_str(response_line.trim())
|
||||
.map_err(|e| format!("failed to parse helper response: {e}"))?;
|
||||
|
||||
let matches = value
|
||||
.get("id")
|
||||
.and_then(|v| v.as_str())
|
||||
.is_some_and(|rid| rid == id_str.as_str());
|
||||
if matches {
|
||||
return Ok(value);
|
||||
}
|
||||
log::debug!(
|
||||
"[accessibility] discarding helper response with mismatched or missing id (want id={})",
|
||||
id_str
|
||||
);
|
||||
}
|
||||
|
||||
serde_json::from_str(response_line.trim())
|
||||
.map_err(|e| format!("failed to parse helper response: {e}"))
|
||||
}
|
||||
|
||||
/// Send a JSON request without waiting for a response.
|
||||
/// Used for `show`, `hide`, and `quit` commands.
|
||||
/// Only acquires UNIFIED_HELPER (for the stdin write) — never blocks on I/O.
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) fn helper_send_fire_and_forget(request: &serde_json::Value) -> Result<(), String> {
|
||||
ensure_helper_running()?;
|
||||
@@ -90,6 +196,13 @@ pub(super) fn helper_send_fire_and_forget(request: &serde_json::Value) -> Result
|
||||
/// Quit and clean up the helper process.
|
||||
#[cfg(target_os = "macos")]
|
||||
pub(super) fn helper_quit() -> Result<(), String> {
|
||||
// Drop the response channel first so the reader thread exits cleanly.
|
||||
{
|
||||
let mut rx_guard = RESPONSE_RX
|
||||
.lock()
|
||||
.map_err(|_| "response rx lock poisoned".to_string())?;
|
||||
rx_guard.take();
|
||||
}
|
||||
let mut guard = UNIFIED_HELPER
|
||||
.lock()
|
||||
.map_err(|_| "unified helper lock poisoned".to_string())?;
|
||||
@@ -103,6 +216,8 @@ pub(super) fn helper_quit() -> Result<(), String> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Ensure the helper process is running. Spawns it (and the stdout reader
|
||||
/// thread) if not yet started or if it has exited unexpectedly.
|
||||
#[cfg(target_os = "macos")]
|
||||
fn ensure_helper_running() -> Result<(), String> {
|
||||
let mut guard = UNIFIED_HELPER
|
||||
@@ -120,6 +235,10 @@ fn ensure_helper_running() -> Result<(), String> {
|
||||
}
|
||||
log::debug!("[accessibility] unified helper exited, restarting");
|
||||
*guard = None;
|
||||
// Also drop the stale receiver so a new one will be created below.
|
||||
if let Ok(mut rx_guard) = RESPONSE_RX.lock() {
|
||||
rx_guard.take();
|
||||
}
|
||||
}
|
||||
|
||||
let binary_path = ensure_helper_binary()?;
|
||||
@@ -139,17 +258,69 @@ fn ensure_helper_running() -> Result<(), String> {
|
||||
.take()
|
||||
.ok_or_else(|| "failed to capture helper stdout".to_string())?;
|
||||
|
||||
*guard = Some(UnifiedHelperProcess {
|
||||
child,
|
||||
stdin,
|
||||
stdout: BufReader::new(stdout),
|
||||
// Spawn a background thread that continuously reads lines from the helper's
|
||||
// stdout and forwards them into the channel. The thread exits when the
|
||||
// sender is dropped (i.e. when helper_quit drops RESPONSE_RX) or when the
|
||||
// process closes its stdout.
|
||||
let (tx, rx) = mpsc::channel::<String>();
|
||||
std::thread::spawn(move || {
|
||||
let mut reader = BufReader::new(stdout);
|
||||
let mut line = String::new();
|
||||
loop {
|
||||
line.clear();
|
||||
match reader.read_line(&mut line) {
|
||||
Ok(0) => break, // EOF — helper exited
|
||||
Ok(_) => {
|
||||
let trimmed = line.trim().to_string();
|
||||
if !trimmed.is_empty() && tx.send(trimmed).is_err() {
|
||||
break; // Receiver dropped — time to exit
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
log::debug!("[accessibility] helper stdout reader error: {e}");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
log::debug!("[accessibility] helper stdout reader thread exiting");
|
||||
});
|
||||
|
||||
// Store the new receiver.
|
||||
if let Ok(mut rx_guard) = RESPONSE_RX.lock() {
|
||||
*rx_guard = Some(rx);
|
||||
}
|
||||
|
||||
*guard = Some(UnifiedHelperProcess { child, stdin });
|
||||
log::debug!("[accessibility] unified helper started");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Compile the Swift helper binary in the background so the first overlay
|
||||
/// request does not incur the compile latency. Safe to call multiple times;
|
||||
/// subsequent calls are no-ops (the binary is cached by `ensure_helper_binary`).
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn precompile_helper_background() {
|
||||
std::thread::spawn(|| {
|
||||
log::debug!("[accessibility] precompile_helper_background: starting");
|
||||
match ensure_helper_binary() {
|
||||
Ok(path) => log::debug!("[accessibility] helper binary ready: {}", path.display()),
|
||||
Err(e) => log::warn!(
|
||||
"[accessibility] helper precompile failed (will retry on first use): {e}"
|
||||
),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// No-op on non-macOS platforms.
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
pub fn precompile_helper_background() {}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn ensure_helper_binary() -> Result<PathBuf, String> {
|
||||
let _compile_guard = HELPER_COMPILE_LOCK
|
||||
.lock()
|
||||
.map_err(|_| "helper compile lock poisoned".to_string())?;
|
||||
|
||||
let cache_dir = std::env::temp_dir().join("openhuman-accessibility-helper");
|
||||
fs::create_dir_all(&cache_dir).map_err(|e| format!("failed to create cache dir: {e}"))?;
|
||||
let source_path = cache_dir.join("unified_helper.swift");
|
||||
@@ -479,11 +650,33 @@ func pasteText(id: String?, text: String) -> [String: Any] {
|
||||
final class OverlayController {
|
||||
private var panel: NSPanel?
|
||||
private var textField: NSTextField?
|
||||
private var hintField: NSTextField?
|
||||
private var hideWorkItem: DispatchWorkItem?
|
||||
|
||||
func show(x: CGFloat, yTop: CGFloat, width: CGFloat, height: CGFloat, text: String, ttlMs: Int) {
|
||||
let panelWidth = min(420, max(140, CGFloat(text.count) * 7 + 26))
|
||||
let panelHeight: CGFloat = 26
|
||||
func show(x: CGFloat, yTop: CGFloat, width: CGFloat, height: CGFloat, text: String, ttlMs: Int, tabHint: String) {
|
||||
let showTabHint = !tabHint.isEmpty
|
||||
// Detect current system appearance for contrast-appropriate colors.
|
||||
let isDark: Bool = {
|
||||
if #available(macOS 10.14, *) {
|
||||
return NSApp.effectiveAppearance
|
||||
.bestMatch(from: [.darkAqua, .aqua]) == .darkAqua
|
||||
}
|
||||
return false
|
||||
}()
|
||||
let bgColor = isDark
|
||||
? NSColor(white: 0.92, alpha: 0.82) // light badge on dark background
|
||||
: NSColor(white: 0.10, alpha: 0.82) // dark badge on light background
|
||||
let textColor = isDark
|
||||
? NSColor(white: 0.08, alpha: 0.95)
|
||||
: NSColor(white: 1.0, alpha: 0.95)
|
||||
|
||||
// Measure badge width from actual text metrics instead of char-count estimate.
|
||||
let font = NSFont.systemFont(ofSize: 13)
|
||||
let attrs: [NSAttributedString.Key: Any] = [.font: font]
|
||||
let measured = (text as NSString).size(withAttributes: attrs)
|
||||
let hintPad: CGFloat = showTabHint ? 80 : 16
|
||||
let panelWidth = min(480, max(140, ceil(measured.width) + hintPad))
|
||||
let panelHeight: CGFloat = 28
|
||||
|
||||
// Multi-monitor: find the screen containing the target or mouse cursor.
|
||||
let screen: NSScreen? = {
|
||||
@@ -537,23 +730,42 @@ final class OverlayController {
|
||||
let content = NSView(frame: NSRect(x: 0, y: 0, width: panelWidth, height: panelHeight))
|
||||
content.wantsLayer = true
|
||||
content.layer?.cornerRadius = 6
|
||||
content.layer?.backgroundColor = NSColor(white: 0.08, alpha: 0.35).cgColor
|
||||
content.layer?.backgroundColor = bgColor.cgColor
|
||||
p.contentView = content
|
||||
|
||||
let label = NSTextField(labelWithString: text)
|
||||
label.frame = NSRect(x: 8, y: 4, width: panelWidth - 12, height: 18)
|
||||
label.textColor = NSColor(white: 1.0, alpha: 0.46)
|
||||
label.font = NSFont.systemFont(ofSize: 13)
|
||||
label.frame = NSRect(x: 8, y: 5, width: panelWidth - (showTabHint ? 62 : 16), height: 18)
|
||||
label.textColor = textColor
|
||||
label.font = font
|
||||
label.lineBreakMode = .byTruncatingTail
|
||||
content.addSubview(label)
|
||||
|
||||
let hint = NSTextField(labelWithString: tabHint.isEmpty ? "Tab ↵" : tabHint)
|
||||
hint.frame = NSRect(x: panelWidth - 54, y: 5, width: 48, height: 18)
|
||||
hint.textColor = NSColor(white: isDark ? 0.35 : 0.65, alpha: 1.0)
|
||||
hint.font = NSFont.monospacedSystemFont(ofSize: 10, weight: .regular)
|
||||
hint.alignment = .right
|
||||
hint.isHidden = !showTabHint
|
||||
content.addSubview(hint)
|
||||
|
||||
panel = p
|
||||
textField = label
|
||||
hintField = hint
|
||||
}
|
||||
|
||||
// Re-apply colors on every show so runtime appearance changes are reflected.
|
||||
panel?.contentView?.layer?.backgroundColor = bgColor.cgColor
|
||||
textField?.textColor = textColor
|
||||
hintField?.textColor = NSColor(white: isDark ? 0.35 : 0.65, alpha: 1.0)
|
||||
hintField?.isHidden = !showTabHint
|
||||
if showTabHint {
|
||||
hintField?.stringValue = tabHint
|
||||
}
|
||||
|
||||
panel?.setFrame(NSRect(x: originX, y: originYCocoa, width: panelWidth, height: panelHeight), display: true)
|
||||
panel?.contentView?.frame = NSRect(x: 0, y: 0, width: panelWidth, height: panelHeight)
|
||||
textField?.frame = NSRect(x: 8, y: 4, width: panelWidth - 12, height: 18)
|
||||
textField?.frame = NSRect(x: 8, y: 5, width: panelWidth - (showTabHint ? 62 : 16), height: 18)
|
||||
hintField?.frame = NSRect(x: panelWidth - 54, y: 5, width: 48, height: 18)
|
||||
textField?.stringValue = text
|
||||
panel?.orderFrontRegardless()
|
||||
|
||||
@@ -602,8 +814,12 @@ DispatchQueue.global(qos: .userInitiated).async {
|
||||
let h = CGFloat((payload["h"] as? NSNumber)?.doubleValue ?? 0)
|
||||
let text = (payload["text"] as? String) ?? ""
|
||||
let ttl = (payload["ttl_ms"] as? NSNumber)?.intValue ?? 900
|
||||
let tabHint: String = {
|
||||
if let s = payload["tab_hint"] as? String { return s }
|
||||
return "Tab ↵"
|
||||
}()
|
||||
DispatchQueue.main.async {
|
||||
controller.show(x: x, yTop: y, width: w, height: h, text: text, ttlMs: ttl)
|
||||
controller.show(x: x, yTop: y, width: w, height: h, text: text, ttlMs: ttl, tabHint: tabHint)
|
||||
}
|
||||
|
||||
case "hide":
|
||||
|
||||
@@ -1,7 +1,70 @@
|
||||
//! Key state probes via direct FFI (lightweight, no helper needed).
|
||||
//!
|
||||
//! Tab and Escape detection is gated on the Input Monitoring permission.
|
||||
//! The permission is cached; if initially denied, we re-check occasionally so
|
||||
//! granting permission without restarting the app still enables Tab/Escape.
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::sync::atomic::{AtomicBool, AtomicI64, Ordering};
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
static INPUT_MONITORING_GRANTED: AtomicBool = AtomicBool::new(false);
|
||||
/// Last time we called `detect_input_monitoring_permission` (ms since UNIX epoch).
|
||||
#[cfg(target_os = "macos")]
|
||||
static INPUT_MONITORING_LAST_CHECK_MS: AtomicI64 = AtomicI64::new(0);
|
||||
|
||||
/// Re-check interval when permission is still denied (avoid IOHID every tick).
|
||||
#[cfg(target_os = "macos")]
|
||||
const INPUT_MONITORING_RECHECK_MS: i64 = 2500;
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn refresh_input_monitoring_cache() {
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
let now_ms = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_millis() as i64)
|
||||
.unwrap_or(0);
|
||||
|
||||
if INPUT_MONITORING_GRANTED.load(Ordering::Relaxed) {
|
||||
return;
|
||||
}
|
||||
|
||||
let last = INPUT_MONITORING_LAST_CHECK_MS.load(Ordering::Relaxed);
|
||||
if now_ms - last < INPUT_MONITORING_RECHECK_MS && last != 0 {
|
||||
return;
|
||||
}
|
||||
INPUT_MONITORING_LAST_CHECK_MS.store(now_ms, Ordering::Relaxed);
|
||||
|
||||
use super::permissions::detect_input_monitoring_permission;
|
||||
use super::types::PermissionState;
|
||||
let granted = matches!(
|
||||
detect_input_monitoring_permission(),
|
||||
PermissionState::Granted
|
||||
);
|
||||
if granted {
|
||||
INPUT_MONITORING_GRANTED.store(true, Ordering::Relaxed);
|
||||
log::info!("[accessibility] Input Monitoring granted — Tab/Escape key detection enabled.");
|
||||
return;
|
||||
}
|
||||
|
||||
// First denial: warn once (avoid spam on every recheck interval).
|
||||
static WARNED: AtomicBool = AtomicBool::new(false);
|
||||
if !WARNED.swap(true, Ordering::Relaxed) {
|
||||
log::warn!(
|
||||
"[accessibility] Input Monitoring permission not granted; \
|
||||
Tab/Escape key detection disabled until granted. \
|
||||
Grant in System Settings → Privacy & Security → Input Monitoring."
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn is_tab_key_down() -> bool {
|
||||
refresh_input_monitoring_cache();
|
||||
if !INPUT_MONITORING_GRANTED.load(Ordering::Relaxed) {
|
||||
return false;
|
||||
}
|
||||
unsafe { CGEventSourceKeyState(KCG_EVENT_SOURCE_STATE_COMBINED_SESSION_STATE, KVK_TAB) }
|
||||
}
|
||||
|
||||
@@ -12,6 +75,10 @@ pub fn is_tab_key_down() -> bool {
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn is_escape_key_down() -> bool {
|
||||
refresh_input_monitoring_cache();
|
||||
if !INPUT_MONITORING_GRANTED.load(Ordering::Relaxed) {
|
||||
return false;
|
||||
}
|
||||
unsafe { CGEventSourceKeyState(KCG_EVENT_SOURCE_STATE_COMBINED_SESSION_STATE, KVK_ESCAPE) }
|
||||
}
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ pub use globe::{
|
||||
globe_listener_poll, globe_listener_start, globe_listener_stop, GlobeHotkeyPollResult,
|
||||
GlobeHotkeyStatus,
|
||||
};
|
||||
pub use helper::precompile_helper_background;
|
||||
pub use keys::{is_escape_key_down, is_tab_key_down};
|
||||
pub use overlay::{hide_overlay, quit_overlay, show_overlay};
|
||||
pub use paste::apply_text_to_focused_field;
|
||||
|
||||
@@ -4,8 +4,16 @@ use super::text_util::truncate_tail;
|
||||
use super::types::ElementBounds;
|
||||
|
||||
/// Show an overlay badge near the given element bounds.
|
||||
///
|
||||
/// When `tab_hint` is empty, the Swift helper hides the Tab keyboard hint (used when
|
||||
/// `accept_with_tab` is disabled in config).
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn show_overlay(bounds: &ElementBounds, text: &str, ttl_ms: u32) -> Result<(), String> {
|
||||
pub fn show_overlay(
|
||||
bounds: &ElementBounds,
|
||||
text: &str,
|
||||
ttl_ms: u32,
|
||||
tab_hint: &str,
|
||||
) -> Result<(), String> {
|
||||
let message = serde_json::json!({
|
||||
"type": "show",
|
||||
"x": bounds.x,
|
||||
@@ -13,7 +21,8 @@ pub fn show_overlay(bounds: &ElementBounds, text: &str, ttl_ms: u32) -> Result<(
|
||||
"w": bounds.width,
|
||||
"h": bounds.height,
|
||||
"text": truncate_tail(text, 96),
|
||||
"ttl_ms": ttl_ms
|
||||
"ttl_ms": ttl_ms,
|
||||
"tab_hint": tab_hint,
|
||||
});
|
||||
super::helper::helper_send_fire_and_forget(&message)
|
||||
}
|
||||
@@ -32,7 +41,12 @@ pub fn quit_overlay() -> Result<(), String> {
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
pub fn show_overlay(_bounds: &ElementBounds, _text: &str, _ttl_ms: u32) -> Result<(), String> {
|
||||
pub fn show_overlay(
|
||||
_bounds: &ElementBounds,
|
||||
_text: &str,
|
||||
_ttl_ms: u32,
|
||||
_tab_hint: &str,
|
||||
) -> Result<(), String> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -45,3 +59,86 @@ pub fn hide_overlay() -> Result<(), String> {
|
||||
pub fn quit_overlay() -> Result<(), String> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
use super::*;
|
||||
|
||||
// --- Non-macOS stubs always succeed ---
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overlay_non_macos_returns_ok() {
|
||||
let bounds = ElementBounds {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 100,
|
||||
height: 50,
|
||||
};
|
||||
assert!(show_overlay(&bounds, "suggestion text", 900, "Tab ↵").is_ok());
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overlay_non_macos_empty_text_returns_ok() {
|
||||
let bounds = ElementBounds {
|
||||
x: 10,
|
||||
y: 20,
|
||||
width: 0,
|
||||
height: 0,
|
||||
};
|
||||
assert!(show_overlay(&bounds, "", 500, "").is_ok());
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overlay_non_macos_zero_ttl_returns_ok() {
|
||||
let bounds = ElementBounds {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 200,
|
||||
height: 30,
|
||||
};
|
||||
assert!(show_overlay(&bounds, "hello", 0, "Tab ↵").is_ok());
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overlay_non_macos_max_ttl_returns_ok() {
|
||||
let bounds = ElementBounds {
|
||||
x: -10,
|
||||
y: -5,
|
||||
width: 300,
|
||||
height: 60,
|
||||
};
|
||||
assert!(show_overlay(&bounds, "test", u32::MAX, "Tab ↵").is_ok());
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn hide_overlay_non_macos_returns_ok() {
|
||||
assert!(hide_overlay().is_ok());
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn quit_overlay_non_macos_returns_ok() {
|
||||
assert!(quit_overlay().is_ok());
|
||||
}
|
||||
|
||||
// Verify overlay functions can be called multiple times without error
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn hide_overlay_idempotent() {
|
||||
assert!(hide_overlay().is_ok());
|
||||
assert!(hide_overlay().is_ok());
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn quit_overlay_idempotent() {
|
||||
assert!(quit_overlay().is_ok());
|
||||
assert!(quit_overlay().is_ok());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -152,12 +152,16 @@ fn clipboard_save_osascript() -> Option<String> {
|
||||
}
|
||||
|
||||
/// Fallback insertion: direct AXValue write via AppleScript.
|
||||
/// Reads `AXSelectedTextRange` to insert at the cursor position rather than
|
||||
/// always appending to the end of the field.
|
||||
#[cfg(target_os = "macos")]
|
||||
fn apply_text_via_axvalue(text: &str) -> Result<(), String> {
|
||||
let escaped = text
|
||||
.replace('\\', "\\\\")
|
||||
.replace('\"', "\\\"")
|
||||
.replace('\n', " ");
|
||||
// AXSelectedTextRange.location is 0-based; AppleScript string indices are 1-based.
|
||||
// "text 1 thru 0" evaluates to "" in AppleScript — correct for cursor-at-start.
|
||||
let script = format!(
|
||||
r##"
|
||||
tell application "System Events"
|
||||
@@ -168,16 +172,29 @@ tell application "System Events"
|
||||
set currentValue to value of attribute "AXValue" of focusedElement as text
|
||||
end try
|
||||
if currentValue is "missing value" then set currentValue to ""
|
||||
if currentValue is "" then
|
||||
try
|
||||
set currentValue to value of attribute "AXSelectedText" of focusedElement as text
|
||||
end try
|
||||
if currentValue is "missing value" then set currentValue to ""
|
||||
-- Read cursor position from AXSelectedTextRange (0-based location).
|
||||
set insertionOffset to -1
|
||||
try
|
||||
set selRange to value of attribute "AXSelectedTextRange" of focusedElement
|
||||
set insertionOffset to location of selRange
|
||||
end try
|
||||
-- Insert at cursor when available, otherwise append.
|
||||
if insertionOffset >= 0 and insertionOffset <= (length of currentValue) then
|
||||
if insertionOffset = 0 then
|
||||
set value of attribute "AXValue" of focusedElement to ("{}" & currentValue)
|
||||
else if insertionOffset >= (length of currentValue) then
|
||||
set value of attribute "AXValue" of focusedElement to (currentValue & "{}")
|
||||
else
|
||||
set beforeCursor to text 1 thru insertionOffset of currentValue
|
||||
set afterCursor to text (insertionOffset + 1) thru -1 of currentValue
|
||||
set value of attribute "AXValue" of focusedElement to (beforeCursor & "{}" & afterCursor)
|
||||
end if
|
||||
else
|
||||
set value of attribute "AXValue" of focusedElement to (currentValue & "{}")
|
||||
end if
|
||||
set value of attribute "AXValue" of focusedElement to (currentValue & "{}")
|
||||
end tell
|
||||
"##,
|
||||
escaped
|
||||
escaped, escaped, escaped, escaped
|
||||
);
|
||||
let output = std::process::Command::new("osascript")
|
||||
.arg("-e")
|
||||
|
||||
@@ -34,3 +34,149 @@ pub fn parse_ax_number(raw: &str) -> Option<i32> {
|
||||
Some(rounded as i32)
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
// --- truncate_tail ---
|
||||
|
||||
#[test]
|
||||
fn truncate_tail_shorter_than_max_returns_original() {
|
||||
assert_eq!(truncate_tail("hello", 10), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_tail_exactly_max_returns_original() {
|
||||
assert_eq!(truncate_tail("hello", 5), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_tail_longer_than_max_returns_tail() {
|
||||
assert_eq!(truncate_tail("hello", 3), "llo");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_tail_empty_string() {
|
||||
assert_eq!(truncate_tail("", 5), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_tail_zero_max_returns_empty() {
|
||||
assert_eq!(truncate_tail("hello", 0), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_tail_multibyte_chars_counts_chars_not_bytes() {
|
||||
// "héllo" is 5 chars; last 3 = "llo"
|
||||
assert_eq!(truncate_tail("héllo", 3), "llo");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_tail_unicode_emoji_counts_codepoints() {
|
||||
// "ab🎉cd" — 5 codepoints; last 3 = "🎉cd"
|
||||
assert_eq!(truncate_tail("ab🎉cd", 3), "🎉cd");
|
||||
}
|
||||
|
||||
// --- normalize_ax_value ---
|
||||
|
||||
#[test]
|
||||
fn normalize_ax_value_trims_whitespace() {
|
||||
assert_eq!(normalize_ax_value(" hello "), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_ax_value_missing_value_lowercase_returns_empty() {
|
||||
assert_eq!(normalize_ax_value("missing value"), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_ax_value_missing_value_uppercase_returns_empty() {
|
||||
assert_eq!(normalize_ax_value("MISSING VALUE"), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_ax_value_mixed_case_missing_value_returns_empty() {
|
||||
assert_eq!(normalize_ax_value("Missing Value"), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_ax_value_empty_string_returns_empty() {
|
||||
assert_eq!(normalize_ax_value(""), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_ax_value_only_whitespace_returns_empty() {
|
||||
assert_eq!(normalize_ax_value(" "), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_ax_value_regular_text_unchanged() {
|
||||
assert_eq!(normalize_ax_value("some value"), "some value");
|
||||
}
|
||||
|
||||
// --- parse_ax_number ---
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_integer_string() {
|
||||
assert_eq!(parse_ax_number("42"), Some(42));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_negative_integer() {
|
||||
assert_eq!(parse_ax_number("-7"), Some(-7));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_float_rounds_to_nearest() {
|
||||
assert_eq!(parse_ax_number("42.4"), Some(42));
|
||||
assert_eq!(parse_ax_number("42.6"), Some(43));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_comma_treated_as_decimal_separator() {
|
||||
// Locale-style: "1,5" → 1.5 → rounds to 2
|
||||
assert_eq!(parse_ax_number("1,5"), Some(2));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_missing_value_returns_none() {
|
||||
assert_eq!(parse_ax_number("missing value"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_empty_returns_none() {
|
||||
assert_eq!(parse_ax_number(""), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_whitespace_only_returns_none() {
|
||||
assert_eq!(parse_ax_number(" "), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_non_numeric_returns_none() {
|
||||
assert_eq!(parse_ax_number("abc"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_nan_returns_none() {
|
||||
assert_eq!(parse_ax_number("NaN"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_infinity_returns_none() {
|
||||
assert_eq!(parse_ax_number("inf"), None);
|
||||
assert_eq!(parse_ax_number("infinity"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_zero() {
|
||||
assert_eq!(parse_ax_number("0"), Some(0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ax_number_trims_surrounding_whitespace() {
|
||||
assert_eq!(parse_ax_number(" 10 "), Some(10));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -71,3 +71,146 @@ pub enum PermissionKind {
|
||||
Accessibility,
|
||||
InputMonitoring,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn make_ctx(
|
||||
app: Option<&str>,
|
||||
title: Option<&str>,
|
||||
bounds: Option<ElementBounds>,
|
||||
) -> AppContext {
|
||||
AppContext {
|
||||
app_name: app.map(str::to_string),
|
||||
window_title: title.map(str::to_string),
|
||||
bounds,
|
||||
}
|
||||
}
|
||||
|
||||
fn make_bounds(x: i32, y: i32, w: i32, h: i32) -> ElementBounds {
|
||||
ElementBounds {
|
||||
x,
|
||||
y,
|
||||
width: w,
|
||||
height: h,
|
||||
}
|
||||
}
|
||||
|
||||
// --- AppContext::same_as ---
|
||||
|
||||
#[test]
|
||||
fn same_as_identical_contexts_true() {
|
||||
let a = make_ctx(
|
||||
Some("App"),
|
||||
Some("Window"),
|
||||
Some(make_bounds(0, 0, 800, 600)),
|
||||
);
|
||||
let b = make_ctx(
|
||||
Some("App"),
|
||||
Some("Window"),
|
||||
Some(make_bounds(0, 0, 800, 600)),
|
||||
);
|
||||
assert!(a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_both_none_fields_true() {
|
||||
let a = make_ctx(None, None, None);
|
||||
let b = make_ctx(None, None, None);
|
||||
assert!(a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_different_app_name_false() {
|
||||
let a = make_ctx(Some("AppA"), Some("Window"), None);
|
||||
let b = make_ctx(Some("AppB"), Some("Window"), None);
|
||||
assert!(!a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_different_window_title_false() {
|
||||
let a = make_ctx(Some("App"), Some("Win1"), None);
|
||||
let b = make_ctx(Some("App"), Some("Win2"), None);
|
||||
assert!(!a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_one_has_bounds_other_none_false() {
|
||||
let a = make_ctx(Some("App"), None, Some(make_bounds(0, 0, 100, 100)));
|
||||
let b = make_ctx(Some("App"), None, None);
|
||||
assert!(!a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_different_bounds_x_false() {
|
||||
let a = make_ctx(None, None, Some(make_bounds(10, 0, 100, 100)));
|
||||
let b = make_ctx(None, None, Some(make_bounds(20, 0, 100, 100)));
|
||||
assert!(!a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_different_bounds_y_false() {
|
||||
let a = make_ctx(None, None, Some(make_bounds(0, 10, 100, 100)));
|
||||
let b = make_ctx(None, None, Some(make_bounds(0, 20, 100, 100)));
|
||||
assert!(!a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_different_bounds_width_false() {
|
||||
let a = make_ctx(None, None, Some(make_bounds(0, 0, 100, 100)));
|
||||
let b = make_ctx(None, None, Some(make_bounds(0, 0, 200, 100)));
|
||||
assert!(!a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_different_bounds_height_false() {
|
||||
let a = make_ctx(None, None, Some(make_bounds(0, 0, 100, 100)));
|
||||
let b = make_ctx(None, None, Some(make_bounds(0, 0, 100, 200)));
|
||||
assert!(!a.same_as(&b));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_as_reflexive() {
|
||||
let a = make_ctx(Some("App"), Some("Win"), Some(make_bounds(1, 2, 3, 4)));
|
||||
assert!(a.same_as(&a));
|
||||
}
|
||||
|
||||
// --- AppContext::as_compound_text ---
|
||||
|
||||
#[test]
|
||||
fn as_compound_text_both_some_lowercase() {
|
||||
let ctx = make_ctx(Some("MyApp"), Some("My Window"), None);
|
||||
assert_eq!(ctx.as_compound_text(), "myapp my window");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn as_compound_text_app_none_title_some() {
|
||||
let ctx = make_ctx(None, Some("Window Title"), None);
|
||||
assert_eq!(ctx.as_compound_text(), " window title");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn as_compound_text_app_some_title_none() {
|
||||
let ctx = make_ctx(Some("AppName"), None, None);
|
||||
assert_eq!(ctx.as_compound_text(), "appname ");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn as_compound_text_both_none_returns_space() {
|
||||
let ctx = make_ctx(None, None, None);
|
||||
assert_eq!(ctx.as_compound_text(), " ");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn as_compound_text_already_lowercase_unchanged() {
|
||||
let ctx = make_ctx(Some("slack"), Some("general"), None);
|
||||
assert_eq!(ctx.as_compound_text(), "slack general");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn as_compound_text_mixed_case_lowercased() {
|
||||
let ctx = make_ctx(Some("VS Code"), Some("README.md"), None);
|
||||
assert_eq!(ctx.as_compound_text(), "vs code readme.md");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@ use crate::openhuman::config::Config;
|
||||
use crate::openhuman::local_ai;
|
||||
use chrono::Utc;
|
||||
use once_cell::sync::Lazy;
|
||||
use std::sync::Arc;
|
||||
use std::sync::{Arc, Once};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio::task::JoinHandle;
|
||||
use tokio::time::{self, Duration, Instant};
|
||||
@@ -114,6 +114,14 @@ impl AutocompleteEngine {
|
||||
return Ok(AutocompleteStartResult { started: false });
|
||||
}
|
||||
|
||||
// Kick off Swift helper compilation in the background so the first
|
||||
// suggestion request does not stall waiting for `swiftc`.
|
||||
// Only after we know config loaded and autocomplete is enabled.
|
||||
static PRECOMPILE_ONCE: Once = Once::new();
|
||||
PRECOMPILE_ONCE.call_once(|| {
|
||||
crate::openhuman::accessibility::precompile_helper_background();
|
||||
});
|
||||
|
||||
let debounce_ms = params
|
||||
.debounce_ms
|
||||
.unwrap_or(config.autocomplete.debounce_ms)
|
||||
@@ -180,6 +188,8 @@ impl AutocompleteEngine {
|
||||
Some(&error_message),
|
||||
None,
|
||||
None,
|
||||
700,
|
||||
false,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -346,7 +356,7 @@ impl AutocompleteEngine {
|
||||
state.last_overlay_signature = None;
|
||||
}
|
||||
if should_apply {
|
||||
show_overflow_badge("accepted", Some(&cleaned), None, None, None);
|
||||
show_overflow_badge("accepted", Some(&cleaned), None, None, None, 700, false);
|
||||
}
|
||||
|
||||
// Persist acceptance for personalisation (fire-and-forget).
|
||||
@@ -425,6 +435,9 @@ impl AutocompleteEngine {
|
||||
if let Some(accept_with_tab) = params.accept_with_tab {
|
||||
config.autocomplete.accept_with_tab = accept_with_tab;
|
||||
}
|
||||
if let Some(overlay_ttl_ms) = params.overlay_ttl_ms {
|
||||
config.autocomplete.overlay_ttl_ms = overlay_ttl_ms.clamp(300, 10_000);
|
||||
}
|
||||
config.save().await.map_err(|e| e.to_string())?;
|
||||
|
||||
let mut state = self.inner.lock().await;
|
||||
@@ -620,7 +633,8 @@ impl AutocompleteEngine {
|
||||
}
|
||||
state.suggestion = Some(AutocompleteSuggestion {
|
||||
value: suggestion.clone(),
|
||||
confidence: 0.72,
|
||||
// Placeholder until `local_ai::inline_complete` surfaces a real score (avoid 0.0 so UI/thresholds keep signal).
|
||||
confidence: 0.75,
|
||||
});
|
||||
state.phase = "ready".to_string();
|
||||
state.last_error = None;
|
||||
@@ -631,6 +645,7 @@ impl AutocompleteEngine {
|
||||
);
|
||||
if !is_in_app && state.last_overlay_signature.as_deref() != Some(ready_signature.as_str()) {
|
||||
state.last_overlay_signature = Some(ready_signature);
|
||||
let overlay_ttl_ms = config.autocomplete.overlay_ttl_ms;
|
||||
drop(state);
|
||||
show_overflow_badge(
|
||||
"ready",
|
||||
@@ -638,6 +653,8 @@ impl AutocompleteEngine {
|
||||
None,
|
||||
app_name.as_deref(),
|
||||
focused.bounds.as_ref(),
|
||||
overlay_ttl_ms,
|
||||
config.autocomplete.accept_with_tab,
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
@@ -723,7 +740,15 @@ impl AutocompleteEngine {
|
||||
.unwrap_or_default()
|
||||
.to_lowercase();
|
||||
if !app_lower.contains("openhuman") {
|
||||
show_overflow_badge("accepted", Some(&cleaned), None, None, None);
|
||||
show_overflow_badge(
|
||||
"accepted",
|
||||
Some(&cleaned),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
700,
|
||||
false,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -783,7 +808,7 @@ impl AutocompleteEngine {
|
||||
.unwrap_or_default()
|
||||
.to_lowercase();
|
||||
if !app_lower.contains("openhuman") {
|
||||
show_overflow_badge("rejected", Some(&value), None, None, None);
|
||||
show_overflow_badge("rejected", Some(&value), None, None, None, 700, false);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
|
||||
@@ -23,10 +23,12 @@ pub(super) fn show_overflow_badge(
|
||||
error: Option<&str>,
|
||||
app_name: Option<&str>,
|
||||
anchor_bounds: Option<&ElementBounds>,
|
||||
ttl_ms: u32,
|
||||
// When `kind == "ready"`, show the Tab hint in the overlay only if true.
|
||||
show_tab_hint: bool,
|
||||
) {
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
const READY_THROTTLE_MS: i64 = 1_200;
|
||||
let now_ms = Utc::now().timestamp_millis();
|
||||
let signature = format!(
|
||||
"{}:{}:{}:{}",
|
||||
@@ -36,12 +38,11 @@ pub(super) fn show_overflow_badge(
|
||||
error.unwrap_or_default()
|
||||
);
|
||||
|
||||
// Deduplicate rapid duplicate events only (same payload within a short window).
|
||||
const DEDUP_MS: i64 = 400;
|
||||
if let Ok(mut guard) = LAST_OVERFLOW_BADGE.lock() {
|
||||
if let Some((last_signature, last_ms)) = guard.as_ref() {
|
||||
if *last_signature == signature {
|
||||
return;
|
||||
}
|
||||
if kind == "ready" && (now_ms - *last_ms) < READY_THROTTLE_MS {
|
||||
if *last_signature == signature && now_ms - *last_ms < DEDUP_MS {
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -58,16 +59,18 @@ pub(super) fn show_overflow_badge(
|
||||
width: 0,
|
||||
height: 0,
|
||||
};
|
||||
let bounds = if anchor_bounds.is_some() {
|
||||
anchor_bounds.unwrap()
|
||||
} else {
|
||||
log::debug!(
|
||||
"[autocomplete] overlay: no anchor bounds, falling back to zero bounds (mouse cursor); suggestion={:?}",
|
||||
truncate_tail(suggestion_text, 40)
|
||||
);
|
||||
&fallback_bounds
|
||||
let bounds: &ElementBounds = match anchor_bounds {
|
||||
Some(b) => b,
|
||||
None => {
|
||||
log::debug!(
|
||||
"[autocomplete] overlay: no anchor bounds, falling back to zero bounds (mouse cursor); suggestion={:?}",
|
||||
truncate_tail(suggestion_text, 40)
|
||||
);
|
||||
&fallback_bounds
|
||||
}
|
||||
};
|
||||
if accessibility::show_overlay(bounds, suggestion_text, 1100).is_ok() {
|
||||
let tab_hint = if show_tab_hint { "Tab ↵" } else { "" };
|
||||
if accessibility::show_overlay(bounds, suggestion_text, ttl_ms, tab_hint).is_ok() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -133,3 +136,141 @@ fn escape_osascript_text(raw: &str) -> String {
|
||||
pub(super) fn overlay_helper_quit() -> Result<(), String> {
|
||||
accessibility::quit_overlay()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
// --- overlay_helper_quit (cross-platform) ---
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn overlay_helper_quit_non_macos_returns_ok() {
|
||||
assert!(overlay_helper_quit().is_ok());
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn overlay_helper_quit_non_macos_idempotent() {
|
||||
assert!(overlay_helper_quit().is_ok());
|
||||
assert!(overlay_helper_quit().is_ok());
|
||||
}
|
||||
|
||||
// --- escape_osascript_text (macOS-only) ---
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_plain_string_unchanged() {
|
||||
assert_eq!(escape_osascript_text("hello world"), "hello world");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_escapes_double_quotes() {
|
||||
assert_eq!(escape_osascript_text(r#"say "hello""#), r#"say \"hello\""#);
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_escapes_backslash() {
|
||||
assert_eq!(escape_osascript_text(r"back\slash"), r"back\\slash");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_replaces_newline_with_space() {
|
||||
assert_eq!(escape_osascript_text("line1\nline2"), "line1 line2");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_replaces_carriage_return_with_space() {
|
||||
assert_eq!(escape_osascript_text("line1\rline2"), "line1 line2");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_crlf_both_replaced() {
|
||||
// \r and \n are each replaced individually → two spaces
|
||||
assert_eq!(escape_osascript_text("a\r\nb"), "a b");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_empty_string_unchanged() {
|
||||
assert_eq!(escape_osascript_text(""), "");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_backslash_before_quote_double_escapes() {
|
||||
// r#"\"# + `"` = `\"` — backslash first becomes `\\`, then `"` becomes `\"`
|
||||
assert_eq!(escape_osascript_text("\\\""), "\\\\\\\"");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
#[test]
|
||||
fn escape_osascript_text_multiple_quotes() {
|
||||
assert_eq!(
|
||||
escape_osascript_text(r#""a" and "b""#),
|
||||
r#"\"a\" and \"b\""#
|
||||
);
|
||||
}
|
||||
|
||||
// --- show_overflow_badge signature (non-macOS no-op smoke test) ---
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overflow_badge_non_macos_does_not_panic_ready() {
|
||||
let bounds = crate::openhuman::accessibility::ElementBounds {
|
||||
x: 0,
|
||||
y: 0,
|
||||
width: 100,
|
||||
height: 50,
|
||||
};
|
||||
// Should be a no-op and not panic.
|
||||
show_overflow_badge(
|
||||
"ready",
|
||||
Some("suggestion"),
|
||||
None,
|
||||
Some("TestApp"),
|
||||
Some(&bounds),
|
||||
900,
|
||||
true,
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overflow_badge_non_macos_does_not_panic_error() {
|
||||
show_overflow_badge(
|
||||
"error",
|
||||
None,
|
||||
Some("something failed"),
|
||||
None,
|
||||
None,
|
||||
500,
|
||||
false,
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overflow_badge_non_macos_does_not_panic_accepted() {
|
||||
show_overflow_badge(
|
||||
"accepted",
|
||||
Some("accepted text"),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
false,
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
#[test]
|
||||
fn show_overflow_badge_non_macos_does_not_panic_rejected() {
|
||||
show_overflow_badge("rejected", None, None, None, None, 200, false);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,3 +26,135 @@ pub(super) fn sanitize_suggestion(text: &str) -> String {
|
||||
pub(super) fn is_no_text_candidate_error(err: &str) -> bool {
|
||||
err.contains("ERROR:no_text_candidate_found")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
// --- truncate_head ---
|
||||
|
||||
#[test]
|
||||
fn truncate_head_shorter_than_max_returns_original() {
|
||||
assert_eq!(truncate_head("hello", 10), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_head_exactly_max_returns_original() {
|
||||
assert_eq!(truncate_head("hello", 5), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_head_longer_than_max_returns_head() {
|
||||
assert_eq!(truncate_head("hello world", 5), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_head_empty_string() {
|
||||
assert_eq!(truncate_head("", 5), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_head_zero_max_returns_empty() {
|
||||
assert_eq!(truncate_head("hello", 0), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_head_multibyte_chars_counts_codepoints() {
|
||||
// "héllo" is 5 chars; first 3 = "hél"
|
||||
assert_eq!(truncate_head("héllo", 3), "hél");
|
||||
}
|
||||
|
||||
// --- sanitize_suggestion ---
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_plain_text() {
|
||||
assert_eq!(sanitize_suggestion("hello world"), "hello world");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_trims_leading_and_trailing_whitespace() {
|
||||
assert_eq!(sanitize_suggestion(" hello "), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_strips_surrounding_double_quotes() {
|
||||
assert_eq!(sanitize_suggestion("\"quoted\""), "quoted");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_takes_first_line_only() {
|
||||
assert_eq!(sanitize_suggestion("line one\nline two"), "line one");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_crlf_newline_takes_first_line() {
|
||||
assert_eq!(sanitize_suggestion("line one\r\nline two"), "line one");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_replaces_embedded_tabs_with_spaces() {
|
||||
// Leading/trailing tabs are stripped by trim(); only interior tabs become spaces.
|
||||
assert_eq!(sanitize_suggestion("he\tllo"), "he llo");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_empty_input_returns_empty() {
|
||||
assert_eq!(sanitize_suggestion(""), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_whitespace_only_returns_empty() {
|
||||
assert_eq!(sanitize_suggestion(" \n "), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_truncates_to_max_chars() {
|
||||
// MAX_SUGGESTION_CHARS is 64 — a 70-char string should be cut to 64.
|
||||
let long = "a".repeat(70);
|
||||
let result = sanitize_suggestion(&long);
|
||||
assert_eq!(result.len(), 64);
|
||||
assert!(result.chars().all(|c| c == 'a'));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_exactly_max_chars_unchanged() {
|
||||
let exact = "b".repeat(64);
|
||||
assert_eq!(sanitize_suggestion(&exact), exact);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_suggestion_removes_bare_carriage_return() {
|
||||
// Bare \r is NOT treated as a line ending by lines(), so it stays in the
|
||||
// first-line content and is then removed by replace('\r', "").
|
||||
assert_eq!(sanitize_suggestion("hello\rworld"), "helloworld");
|
||||
}
|
||||
|
||||
// --- is_no_text_candidate_error ---
|
||||
|
||||
#[test]
|
||||
fn is_no_text_candidate_error_exact_match() {
|
||||
assert!(is_no_text_candidate_error("ERROR:no_text_candidate_found"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_no_text_candidate_error_substring_match() {
|
||||
assert!(is_no_text_candidate_error(
|
||||
"AX query failed: ERROR:no_text_candidate_found"
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_no_text_candidate_error_unrelated_error() {
|
||||
assert!(!is_no_text_candidate_error("some other error"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_no_text_candidate_error_empty_string() {
|
||||
assert!(!is_no_text_candidate_error(""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn is_no_text_candidate_error_partial_prefix_no_match() {
|
||||
assert!(!is_no_text_candidate_error("ERROR:no_text"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -92,6 +92,7 @@ pub struct AutocompleteSetStyleParams {
|
||||
pub style_examples: Option<Vec<String>>,
|
||||
pub disabled_apps: Option<Vec<String>>,
|
||||
pub accept_with_tab: Option<bool>,
|
||||
pub overlay_ttl_ms: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
|
||||
@@ -19,6 +19,8 @@ pub struct AutocompleteConfig {
|
||||
pub disabled_apps: Vec<String>,
|
||||
#[serde(default = "default_accept_with_tab")]
|
||||
pub accept_with_tab: bool,
|
||||
#[serde(default = "default_overlay_ttl_ms")]
|
||||
pub overlay_ttl_ms: u32,
|
||||
}
|
||||
|
||||
fn default_enabled() -> bool {
|
||||
@@ -41,6 +43,10 @@ fn default_accept_with_tab() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn default_overlay_ttl_ms() -> u32 {
|
||||
1100
|
||||
}
|
||||
|
||||
impl Default for AutocompleteConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
@@ -52,6 +58,7 @@ impl Default for AutocompleteConfig {
|
||||
style_examples: Vec::new(),
|
||||
disabled_apps: vec![],
|
||||
accept_with_tab: default_accept_with_tab(),
|
||||
overlay_ttl_ms: default_overlay_ttl_ms(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -10,22 +10,40 @@ use std::process::{Command, Stdio};
|
||||
|
||||
/// Attempt to find and spawn the overlay binary.
|
||||
///
|
||||
/// `parent_core_rpc_url` must be the JSON-RPC URL of the **same** core process
|
||||
/// that spawned the overlay (e.g. `http://127.0.0.1:7788/rpc`) so the overlay
|
||||
/// UI can query autocomplete, screen intelligence, and voice state from the
|
||||
/// desktop sidecar instead of a separate in-process core instance.
|
||||
///
|
||||
/// This is best-effort: if the binary is not found or fails to launch, a
|
||||
/// warning is logged and the core continues normally.
|
||||
pub fn spawn_overlay() {
|
||||
pub fn spawn_overlay(parent_core_rpc_url: &str) {
|
||||
let Some(overlay_bin) = find_overlay_binary() else {
|
||||
log::debug!("[overlay] openhuman-overlay binary not found — skipping overlay launch");
|
||||
log::warn!(
|
||||
"[overlay] openhuman-overlay binary not found — skipping overlay launch (set OPENHUMAN_OVERLAY_BIN or build overlay/src-tauri)"
|
||||
);
|
||||
return;
|
||||
};
|
||||
|
||||
log::info!("[overlay] launching overlay: {}", overlay_bin.display());
|
||||
log::info!(
|
||||
"[overlay] launching overlay: {} (parent RPC {})",
|
||||
overlay_bin.display(),
|
||||
parent_core_rpc_url
|
||||
);
|
||||
|
||||
match Command::new(&overlay_bin)
|
||||
.stdin(Stdio::null())
|
||||
let mut cmd = Command::new(&overlay_bin);
|
||||
cmd.stdin(Stdio::null())
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null())
|
||||
.spawn()
|
||||
{
|
||||
.env(
|
||||
"OPENHUMAN_OVERLAY_PARENT_RPC_URL",
|
||||
parent_core_rpc_url.trim(),
|
||||
)
|
||||
// The desktop core sets OPENHUMAN_CORE_PORT; the overlay must not inherit it or its
|
||||
// optional embedded JSON-RPC server will try to bind the same port as the parent.
|
||||
.env_remove("OPENHUMAN_CORE_PORT");
|
||||
|
||||
match cmd.spawn() {
|
||||
Ok(child) => {
|
||||
log::info!("[overlay] overlay process spawned (pid={})", child.id());
|
||||
}
|
||||
|
||||
@@ -409,7 +409,7 @@ pub async fn skills_list_available(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::registry_cache::{cache_path, is_cache_fresh, write_cache};
|
||||
use super::super::registry_cache::{is_cache_fresh, write_cache};
|
||||
use super::super::registry_types::{CachedRegistry, RegistrySkillCategories, SkillCategory};
|
||||
use super::*;
|
||||
|
||||
|
||||
@@ -103,7 +103,7 @@ mod tests {
|
||||
)
|
||||
.await;
|
||||
|
||||
let notion_doc_id = ingest(
|
||||
let _notion_doc_id = ingest(
|
||||
&memory,
|
||||
"skill-notion",
|
||||
"tick1-tracker",
|
||||
|
||||
@@ -114,7 +114,7 @@ pub async fn show_ghost(
|
||||
element_bounds.height,
|
||||
);
|
||||
|
||||
match accessibility::show_overlay(&element_bounds, ¶ms.text, ttl_ms) {
|
||||
match accessibility::show_overlay(&element_bounds, ¶ms.text, ttl_ms, "") {
|
||||
Ok(()) => Ok(RpcOutcome::single_log(
|
||||
ShowGhostTextResult {
|
||||
shown: true,
|
||||
|
||||
@@ -21,9 +21,7 @@ pub async fn try_dispatch(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use futures_util::FutureExt;
|
||||
use serde_json::json;
|
||||
use std::panic::AssertUnwindSafe;
|
||||
|
||||
use super::try_dispatch;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user