Feat/overlay (#378)

* feat(autocomplete): add overlay TTL configuration to AutocompletePanel

- Introduced `overlay_ttl_ms` parameter to the Autocomplete configuration, allowing users to set the overlay display duration.
- Updated AutocompletePanel to include a new input field for adjusting the overlay TTL in milliseconds.
- Enhanced parsing and saving logic to handle the new configuration parameter.
- Added corresponding tests to ensure functionality and validate the new overlay TTL feature.

This update improves user control over the autocomplete overlay behavior, enhancing the overall user experience.

* Refactor accessibility code for improved readability and consistency

- Simplified log statements in `precompile_helper_background` for better clarity.
- Reformatted `detect_input_monitoring_permission` check in `keys.rs` for enhanced readability.
- Rearranged imports in `mod.rs` to maintain consistent structure.
- Improved formatting of `ElementBounds` initialization across multiple test cases in `overlay.rs` and `types.rs` for better visual alignment.
- Enhanced test context creation in `types.rs` for improved clarity.

These changes enhance code maintainability and readability across the accessibility module.

* fix(overlay): parent core RPC, voice toggle, and debug for #342

- Pass OPENHUMAN_OVERLAY_PARENT_RPC_URL from sidecar spawn and strip inherited
  OPENHUMAN_CORE_PORT so the overlay no longer fights for the parent listen port.
- Overlay UI uses HTTP JSON-RPC to the parent sidecar for globe, debug, and voice
  STT so state matches the main app; add parentCoreRpc helper mirroring legacy
  method aliases.
- Skip embedded JSON-RPC server when parent URL is set; use
  OPENHUMAN_OVERLAY_EMBEDDED_CORE_PORT (default 7799) for standalone dev.
- Fix screen intelligence status method name; add voice_status polling, STT
  section, collapsible debug summary, and connection banner when core is unreachable.
- Voice capture switch gates the mic with real STT availability from voice_status.

Closes #342

Made-with: Cursor

* fix: address CodeRabbit review on overlay/autocomplete (PR #378)

- helper: correlate JSON-RPC replies with monotonic request ids; discard mismatched
  lines until deadline (fixes stale response after timeout).
- helper: serialize Swift compile via HELPER_COMPILE_LOCK (precompile vs first use).
- Overlay Tab hint: pass tab_hint from Rust from accept_with_tab; Swift hides hint when empty.
- keys: re-check Input Monitoring on an interval when denied so grant without restart works.
- engine: use non-zero confidence placeholder (0.75) until inline_complete returns scores.
- overlay dedupe: suppress identical badge only within 400ms, not for process lifetime.

Co-authored-by: Code review feedback <noreply@github.com>
Made-with: Cursor

* fix(ci): resolve clippy and warning issues for Rust gates

- Use match on anchor_bounds in autocomplete overlay (avoid unnecessary unwrap)
- Drop unused test imports in registry_ops and rpc dispatch
- Prefix unused notion_doc_id in subconscious integration test

Made-with: Cursor

* fix(overlay): improve parent RPC URL handling in App component

- Updated the useEffect hook to manage the parent RPC URL more robustly by introducing a mounted flag to prevent state updates on unmounted components.
- Added error handling to set the parent RPC URL to null in case of invocation failure, enhancing the reliability of the component's behavior.

* feat(overlay): implement timeout handling for parent core RPC requests

- Introduced a default timeout for parent core RPC requests, enhancing reliability by preventing indefinite waiting for responses.
- Added an AbortController to manage request timeouts, throwing a specific error message when a timeout occurs.
- Updated the `callParentCoreRpc` function to accept a customizable timeout parameter, improving flexibility for RPC calls.

* fix(overlay): allow stopping active recording regardless of config state

- Updated the main button handler in the App component to always permit stopping an active recording when the status is "listening", improving user experience and control over the recording process.
- Removed redundant code that previously checked the status before stopping the recording, streamlining the logic.

* feat(overlay): update Cargo.lock with new dependencies and versions

- Added new packages including `alsa`, `alsa-sys`, `arboard`, `block`, `cocoa`, `core-foundation`, `core-graphics`, `coreaudio-rs`, `coreaudio-sys`, `cpal`, `crunchy`, and `dasp_sample` to enhance functionality and support for audio processing and system interactions.
- Updated existing dependencies to their latest versions for improved performance and compatibility.
- Modified the `show_overlay` function in `ops.rs` to include an additional parameter, enhancing the overlay display functionality.

* feat(autocomplete): add overlay_ttl_ms parameter to Autocomplete interfaces

- Introduced a new optional parameter `overlay_ttl_ms` to both `AutocompleteSetStyleParams` and `AutocompleteConfig` interfaces, allowing for customizable overlay timeout settings.
- This enhancement improves the flexibility of the autocomplete feature by enabling developers to specify how long the overlay should remain visible.

---------

Co-authored-by: Code review feedback <noreply@github.com>
Co-authored-by: Steven Enamakel <enamakel@tinyhumans.ai>
This commit is contained in:
Mega Mind
2026-04-06 21:11:27 -07:00
committed by GitHub
co-authored by Code review feedback Steven Enamakel
parent e8fd08c800
commit 8627cee960
26 changed files with 2194 additions and 220 deletions
@@ -28,6 +28,7 @@ const DEFAULT_CONFIG: AutocompleteConfig = {
style_examples: [],
disabled_apps: [],
accept_with_tab: true,
overlay_ttl_ms: 1100,
};
const MAX_LOG_ENTRIES = 200;
@@ -56,6 +57,10 @@ const parseAutocompleteConfig = (raw: unknown): AutocompleteConfig => {
typeof value.accept_with_tab === 'boolean'
? value.accept_with_tab
: DEFAULT_CONFIG.accept_with_tab,
overlay_ttl_ms:
typeof value.overlay_ttl_ms === 'number'
? value.overlay_ttl_ms
: DEFAULT_CONFIG.overlay_ttl_ms,
};
};
@@ -77,6 +82,7 @@ const AutocompletePanel = () => {
DEFAULT_CONFIG.disabled_apps.join('\n')
);
const [acceptWithTab, setAcceptWithTab] = useState<boolean>(DEFAULT_CONFIG.accept_with_tab);
const [overlayTtlMs, setOverlayTtlMs] = useState<string>(String(DEFAULT_CONFIG.overlay_ttl_ms));
const [contextOverride, setContextOverride] = useState<string>('');
const [focusDebug, setFocusDebug] = useState<string>('');
const [logs, setLogs] = useState<string[]>([]);
@@ -153,6 +159,7 @@ const AutocompletePanel = () => {
setStyleExamplesText(config.style_examples.join('\n'));
setDisabledAppsText(config.disabled_apps.join('\n'));
setAcceptWithTab(config.accept_with_tab);
setOverlayTtlMs(String(config.overlay_ttl_ms));
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to load autocomplete settings');
} finally {
@@ -253,6 +260,7 @@ const AutocompletePanel = () => {
appendUiLog('saving autocomplete settings');
const debounce = Number(debounceMs);
const max = Number(maxChars);
const ttl = Number(overlayTtlMs);
const response = await openhumanAutocompleteSetStyle({
enabled,
debounce_ms: Number.isFinite(debounce) ? Math.min(Math.max(debounce, 50), 2000) : 120,
@@ -268,6 +276,7 @@ const AutocompletePanel = () => {
.map(entry => entry.trim())
.filter(Boolean),
accept_with_tab: acceptWithTab,
overlay_ttl_ms: Number.isFinite(ttl) ? Math.min(Math.max(ttl, 300), 10000) : 1100,
});
setEnabled(response.result.config.enabled);
@@ -278,6 +287,7 @@ const AutocompletePanel = () => {
setStyleExamplesText(response.result.config.style_examples.join('\n'));
setDisabledAppsText(response.result.config.disabled_apps.join('\n'));
setAcceptWithTab(response.result.config.accept_with_tab);
setOverlayTtlMs(String(response.result.config.overlay_ttl_ms));
appendLogs(response.logs);
setMessage('Autocomplete settings saved.');
await refreshStatus();
@@ -507,6 +517,18 @@ const AutocompletePanel = () => {
className="w-28 rounded border border-stone-300 bg-white px-2 py-1 text-xs text-stone-700"
/>
</label>
<label className="flex items-center justify-between rounded-xl border border-stone-200 bg-stone-50 px-3 py-2">
<span className="text-sm text-stone-700">Overlay TTL (ms)</span>
<input
type="number"
min={300}
max={10000}
step={100}
value={overlayTtlMs}
onChange={event => setOverlayTtlMs(event.target.value)}
className="w-28 rounded border border-stone-300 bg-white px-2 py-1 text-xs text-stone-700"
/>
</label>
<label className="flex items-center justify-between rounded-xl border border-stone-200 bg-stone-50 px-3 py-2">
<span className="text-sm text-stone-700">Style Preset</span>
<select
@@ -81,6 +81,7 @@ describe('AutocompletePanel', () => {
style_examples: [],
disabled_apps: [],
accept_with_tab: true,
overlay_ttl_ms: 1100,
},
};
@@ -78,6 +78,7 @@ export interface AutocompleteSetStyleParams {
style_examples?: string[];
disabled_apps?: string[];
accept_with_tab?: boolean;
overlay_ttl_ms?: number;
}
export interface AutocompleteConfig {
@@ -89,6 +90,7 @@ export interface AutocompleteConfig {
style_examples: string[];
disabled_apps: string[];
accept_with_tab: boolean;
overlay_ttl_ms: number;
}
export interface AutocompleteSetStyleResult {
+677 -78
View File
File diff suppressed because it is too large Load Diff
+42 -15
View File
@@ -3,9 +3,9 @@ mod log_bridge;
use log_bridge::{LogBuffer, LogEntry, TauriLogLayer};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use tauri::Manager;
#[cfg(target_os = "macos")]
use tauri::ActivationPolicy;
use tauri::Manager;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::EnvFilter;
@@ -40,6 +40,19 @@ fn set_click_through(
Ok(())
}
/// JSON-RPC URL of the desktop core sidecar, when the overlay was spawned by it.
/// When set, the web UI should prefer HTTP `fetch` to this URL so autocomplete,
/// screen intelligence, and voice state match the main app (see `overlay/src/parentCoreRpc.ts`).
#[tauri::command]
fn overlay_parent_rpc_url() -> Option<String> {
let url = std::env::var("OPENHUMAN_OVERLAY_PARENT_RPC_URL").ok()?;
let trimmed = url.trim();
if trimmed.is_empty() {
return None;
}
Some(trimmed.to_string())
}
/// Forward an RPC call to openhuman_core's dispatch in-process.
/// Uses the same invoke_method path as the HTTP JSON-RPC server.
#[tauri::command]
@@ -76,6 +89,7 @@ pub fn run() {
.invoke_handler(tauri::generate_handler![
get_log_history,
set_click_through,
overlay_parent_rpc_url,
core_rpc,
insert_text_into_focused_field,
])
@@ -112,20 +126,33 @@ pub fn run() {
log::info!("[overlay] overlay process started, tracing bridge active");
// ── Start openhuman_core JSON-RPC server in-process ─────────
// Use port 7799 to avoid conflicts with a standalone core on 7788.
// Override with OPENHUMAN_CORE_PORT env var.
tauri::async_runtime::spawn(async move {
let port = std::env::var("OPENHUMAN_CORE_PORT")
.ok()
.and_then(|p| p.parse::<u16>().ok())
.unwrap_or(7799);
log::info!("[overlay] starting openhuman_core server on 127.0.0.1:{}...", port);
match openhuman_core::core::jsonrpc::run_server(None, Some(port), true).await {
Ok(()) => log::info!("[overlay] core server shut down cleanly"),
Err(e) => log::error!("[overlay] core server error: {}", e),
}
});
// ── Optional in-process JSON-RPC (standalone / dev without a parent core) ──
// When spawned by the desktop sidecar, OPENHUMAN_OVERLAY_PARENT_RPC_URL is set and
// the web UI talks to the parent over HTTP — do not bind a second server on 7788.
// Use OPENHUMAN_OVERLAY_EMBEDDED_CORE_PORT (default 7799), not OPENHUMAN_CORE_PORT.
let parent_rpc = std::env::var("OPENHUMAN_OVERLAY_PARENT_RPC_URL")
.ok()
.filter(|s| !s.trim().is_empty());
if parent_rpc.is_some() {
log::info!(
"[overlay] parent core RPC URL set — skipping embedded JSON-RPC server"
);
} else {
tauri::async_runtime::spawn(async move {
let port = std::env::var("OPENHUMAN_OVERLAY_EMBEDDED_CORE_PORT")
.ok()
.and_then(|p| p.parse::<u16>().ok())
.unwrap_or(7799);
log::info!(
"[overlay] starting embedded openhuman_core server on 127.0.0.1:{} (standalone)",
port
);
match openhuman_core::core::jsonrpc::run_server_embedded(None, Some(port), true).await {
Ok(()) => log::info!("[overlay] embedded core server shut down cleanly"),
Err(e) => log::error!("[overlay] embedded core server error: {}", e),
}
});
}
// ── macOS: floating panel + visible on all workspaces ───────
#[cfg(target_os = "macos")]
+1 -1
View File
@@ -16,7 +16,7 @@
"label": "overlay",
"title": "",
"width": 376,
"height": 432,
"height": 520,
"minWidth": 376,
"minHeight": 432,
"transparent": true,
+211 -41
View File
@@ -2,6 +2,8 @@ import { invoke } from "@tauri-apps/api/core";
import { getCurrentWindow } from "@tauri-apps/api/window";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { callParentCoreRpc } from "./parentCoreRpc";
const TARGET_SAMPLE_RATE = 16000;
type OverlayStatus = "idle" | "listening" | "transcribing" | "ready" | "error";
@@ -64,13 +66,30 @@ interface AutocompleteStatus {
suggestion: AutocompleteSuggestion | null;
}
/** Matches `VoiceStatus` in src/openhuman/voice/types.rs */
interface VoiceStatus {
stt_available: boolean;
tts_available: boolean;
stt_model_id: string;
tts_voice_id: string;
whisper_binary: string | null;
piper_binary: string | null;
stt_model_path: string | null;
tts_voice_path: string | null;
whisper_in_process: boolean;
llm_cleanup_enabled: boolean;
}
interface OverlayDebugSnapshot {
screen: AccessibilityStatus | null;
autocomplete: AutocompleteStatus | null;
voice: VoiceStatus | null;
updatedAt: number | null;
error: string | null;
}
const DEBUG_EXPANDED_KEY = "openhuman_overlay_debug_expanded";
function logOverlay(message: string, details?: unknown) {
if (details) {
console.debug(`[overlay] ${message}`, details);
@@ -205,16 +224,67 @@ export function App() {
const sessionIdRef = useRef(0);
const globePollInFlightRef = useRef(false);
/** `undefined` until Tauri reports env; then URL string or `null` (embedded core only). */
const [parentRpcUrl, setParentRpcUrl] = useState<string | null | undefined>(undefined);
const [coreReachable, setCoreReachable] = useState(true);
const [voiceCaptureEnabled, setVoiceCaptureEnabled] = useState(true);
const [debugExpanded, setDebugExpanded] = useState(() => {
try {
return typeof localStorage !== "undefined" && localStorage.getItem(DEBUG_EXPANDED_KEY) === "1";
} catch {
return false;
}
});
const [status, setStatus] = useState<OverlayStatus>("idle");
const [message, setMessage] = useState("Click to start listening");
const [transcript, setTranscript] = useState("");
const [debugSnapshot, setDebugSnapshot] = useState<OverlayDebugSnapshot>({
screen: null,
autocomplete: null,
voice: null,
updatedAt: null,
error: null,
});
useEffect(() => {
let mounted = true;
void invoke<string | null>("overlay_parent_rpc_url")
.then((url) => {
if (!mounted) return;
const trimmed = url?.trim();
setParentRpcUrl(trimmed && trimmed.length > 0 ? trimmed : null);
})
.catch(() => {
if (mounted) setParentRpcUrl(null);
});
return () => {
mounted = false;
};
}, []);
const rpc = useCallback(
async <T,>(method: string, params: Record<string, unknown> = {}): Promise<T> => {
if (parentRpcUrl === undefined) {
throw new Error("[overlay] RPC not initialized");
}
if (parentRpcUrl) {
return callParentCoreRpc<T>(parentRpcUrl, method, params);
}
return invoke<T>("core_rpc", { method, params });
},
[parentRpcUrl],
);
const persistDebugExpanded = useCallback((expanded: boolean) => {
setDebugExpanded(expanded);
try {
localStorage.setItem(DEBUG_EXPANDED_KEY, expanded ? "1" : "0");
} catch {
/* ignore */
}
}, []);
useEffect(() => {
let disposed = false;
@@ -228,11 +298,11 @@ export function App() {
};
const startGlobeListener = async () => {
if (parentRpcUrl === undefined) {
return;
}
try {
const result = await invoke<GlobeHotkeyStatus>("core_rpc", {
method: "openhuman.screen_intelligence_globe_listener_start",
params: {},
});
const result = await rpc<GlobeHotkeyStatus>("openhuman.screen_intelligence_globe_listener_start", {});
logOverlay("globe listener start result", result);
if (!result.supported) {
@@ -252,16 +322,13 @@ export function App() {
};
const pollGlobeListener = async () => {
if (disposed || globePollInFlightRef.current) {
if (disposed || parentRpcUrl === undefined || globePollInFlightRef.current) {
return;
}
globePollInFlightRef.current = true;
try {
const result = await invoke<GlobeHotkeyPollResult>("core_rpc", {
method: "openhuman.screen_intelligence_globe_listener_poll",
params: {},
});
const result = await rpc<GlobeHotkeyPollResult>("openhuman.screen_intelligence_globe_listener_poll", {});
if (disposed) {
return;
@@ -297,33 +364,43 @@ export function App() {
return () => {
disposed = true;
window.clearInterval(intervalId);
void invoke("core_rpc", {
method: "openhuman.screen_intelligence_globe_listener_stop",
params: {},
}).catch(() => {});
if (parentRpcUrl === undefined) {
return;
}
void rpc("openhuman.screen_intelligence_globe_listener_stop", {}).catch(() => {});
};
}, [appWindow]);
}, [appWindow, parentRpcUrl, rpc]);
useEffect(() => {
let disposed = false;
let pollInFlight = false;
const pollDebugState = async () => {
if (disposed || pollInFlight) {
if (disposed || parentRpcUrl === undefined || pollInFlight) {
return;
}
pollInFlight = true;
try {
const [screen, autocomplete] = await Promise.all([
invoke<AccessibilityStatus>("core_rpc", {
method: "openhuman.accessibility_status",
params: {},
}),
invoke<AutocompleteStatus>("core_rpc", {
method: "openhuman.autocomplete_status",
params: {},
}),
if (parentRpcUrl) {
try {
await rpc<{ ok?: boolean }>("core.ping", {});
if (!disposed) {
setCoreReachable(true);
}
} catch {
if (!disposed) {
setCoreReachable(false);
}
}
} else {
setCoreReachable(true);
}
const [screen, autocomplete, voice] = await Promise.all([
rpc<AccessibilityStatus>("openhuman.screen_intelligence_status", {}),
rpc<AutocompleteStatus>("openhuman.autocomplete_status", {}),
rpc<VoiceStatus>("openhuman.voice_status", {}),
]);
if (disposed) {
@@ -335,11 +412,13 @@ export function App() {
captureCount: screen.session.capture_count,
autocompletePhase: autocomplete.phase,
hasSuggestion: Boolean(autocomplete.suggestion?.value),
sttAvailable: voice.stt_available,
});
setDebugSnapshot({
screen,
autocomplete,
voice,
updatedAt: Date.now(),
error: null,
});
@@ -370,7 +449,7 @@ export function App() {
disposed = true;
window.clearInterval(intervalId);
};
}, []);
}, [parentRpcUrl, rpc]);
const insertTranscriptIntoFocusedField = useCallback(
async (text: string) => {
@@ -404,13 +483,10 @@ export function App() {
async (blob: Blob, sessionId: number) => {
try {
const audioBytes = await convertBlobToWavBytes(blob);
const result = await invoke<TranscribeResult>("core_rpc", {
method: "openhuman.voice_transcribe_bytes",
params: {
audio_bytes: audioBytes,
extension: "wav",
skip_cleanup: false,
},
const result = await rpc<TranscribeResult>("openhuman.voice_transcribe_bytes", {
audio_bytes: audioBytes,
extension: "wav",
skip_cleanup: false,
});
if (sessionIdRef.current !== sessionId) {
@@ -444,7 +520,7 @@ export function App() {
setMessage(error instanceof Error ? error.message : "Transcription failed");
}
},
[insertTranscriptIntoFocusedField],
[insertTranscriptIntoFocusedField, rpc],
);
const stopRecording = useCallback(() => {
@@ -521,15 +597,33 @@ export function App() {
}, [cleanupStream, transcribeBlob]);
const handleMainButton = useCallback(() => {
// Always allow stopping an active recording, regardless of config state.
if (status === "listening") {
logOverlay("main button toggled to stop listening");
stopRecording();
return;
}
if (!voiceCaptureEnabled) {
setMessage("Turn on voice capture to use the microphone");
return;
}
if (debugSnapshot.voice && !debugSnapshot.voice.stt_available) {
setMessage(
"Speech-to-text is not available. Configure Local AI / voice in the main OpenHuman app.",
);
return;
}
logOverlay("main button toggled to start listening", { priorStatus: status });
void startRecording();
}, [startRecording, status, stopRecording]);
}, [
debugSnapshot.voice,
startRecording,
status,
stopRecording,
voiceCaptureEnabled,
]);
const shellClassName = useMemo(() => {
if (status === "listening") {
@@ -560,6 +654,11 @@ export function App() {
const autocompleteRunning =
debugSnapshot.autocomplete?.running && debugSnapshot.autocomplete?.enabled;
const sttAvailable = debugSnapshot.voice?.stt_available ?? true;
const voiceBlocked =
!voiceCaptureEnabled || (debugSnapshot.voice !== null && !debugSnapshot.voice.stt_available);
const waitingForCoreConfig = parentRpcUrl === undefined;
return (
<div className="flex h-screen w-screen items-start justify-start bg-transparent p-3">
<div className="relative select-none">
@@ -579,6 +678,13 @@ export function App() {
void appWindow.startDragging();
}}
>
{parentRpcUrl && !coreReachable ? (
<div className="mb-2 rounded-2xl border border-amber-400/35 bg-amber-950/35 px-3 py-2 text-[11px] leading-4 text-amber-50">
Cannot reach the OpenHuman core at the sidecar URL. Autocomplete and screen debug may
be stale. Check that the main app is running.
</div>
) : null}
<div className="mb-3 flex items-center justify-between gap-2">
<span className="rounded-full bg-black/15 px-2 py-1 text-[10px] font-semibold uppercase tracking-[0.24em]">
Voice
@@ -593,11 +699,45 @@ export function App() {
</button>
</div>
<div className="mb-3 flex items-center justify-between gap-2 rounded-2xl border border-white/10 bg-black/15 px-3 py-2">
<div className="min-w-0">
<p className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-80">
Voice capture
</p>
<p className="mt-0.5 text-[11px] leading-4 opacity-85">
{waitingForCoreConfig
? "Checking core…"
: sttAvailable
? debugSnapshot.voice?.whisper_in_process
? "STT ready (in-process)"
: "STT ready"
: "STT unavailable — configure voice in the main app"}
</p>
</div>
<button
type="button"
role="switch"
aria-checked={voiceCaptureEnabled}
aria-label={voiceCaptureEnabled ? "Turn voice capture off" : "Turn voice capture on"}
className={`relative h-8 w-[52px] shrink-0 rounded-full border border-white/15 transition ${
voiceCaptureEnabled ? "bg-emerald-500/50" : "bg-black/35"
}`}
onClick={() => setVoiceCaptureEnabled((previous) => !previous)}
>
<span
className={`absolute top-1 h-6 w-6 rounded-full bg-white shadow transition ${
voiceCaptureEnabled ? "left-7" : "left-1"
}`}
/>
</button>
</div>
<div className="flex items-start gap-3">
<button
type="button"
onClick={handleMainButton}
className={`group relative flex h-[108px] w-[108px] shrink-0 items-center justify-center rounded-full border border-white/20 bg-black/20 transition duration-200 hover:bg-black/28 ${
disabled={waitingForCoreConfig || voiceBlocked}
className={`group relative flex h-[108px] w-[108px] shrink-0 items-center justify-center rounded-full border border-white/20 bg-black/20 transition duration-200 hover:bg-black/28 disabled:cursor-not-allowed disabled:opacity-40 ${
status === "listening" ? "scale-[1.02]" : ""
}`}
aria-label={status === "listening" ? "Stop listening" : "Start listening"}
@@ -632,21 +772,50 @@ export function App() {
<div className="mt-3 rounded-[24px] border border-white/10 bg-black/15 p-3">
<div className="flex items-center justify-between gap-2">
<span className="text-[10px] font-semibold uppercase tracking-[0.22em] opacity-75">
Debug
</span>
<span className="text-[10px] opacity-65">
{debugSnapshot.updatedAt ? formatTimestamp(debugSnapshot.updatedAt) : "waiting"}
</span>
<button
type="button"
className="flex flex-1 items-center justify-between gap-2 text-left"
onClick={() => persistDebugExpanded(!debugExpanded)}
aria-expanded={debugExpanded}
>
<span className="text-[10px] font-semibold uppercase tracking-[0.22em] opacity-75">
Debug {debugExpanded ? "▼" : "▶"}
</span>
<span className="text-[10px] opacity-65">
{debugSnapshot.updatedAt ? formatTimestamp(debugSnapshot.updatedAt) : "waiting"}
</span>
</button>
</div>
{!debugExpanded ? (
<p className="mt-2 text-[11px] leading-4 opacity-80">
Screen: {debugSnapshot.screen?.session.active ? "session on" : "idle"} ·
Autocomplete: {autocompletePhase}
{debugSnapshot.autocomplete?.last_error ? " · error" : ""}
</p>
) : null}
{debugSnapshot.error ? (
<div className="mt-3 rounded-2xl border border-red-300/20 bg-red-950/20 px-3 py-2 text-[11px] leading-4 text-red-100">
{debugSnapshot.error}
</div>
) : null}
{debugExpanded ? (
<div className="mt-3 grid gap-3">
<section className="rounded-2xl border border-white/8 bg-black/10 px-3 py-2">
<div className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-70">
Voice / STT
</div>
<div className="mt-2 space-y-1 text-[11px] leading-4 opacity-90">
<p>STT: {sttAvailable ? "available" : "unavailable"}</p>
<p className="truncate">Model: {debugSnapshot.voice?.stt_model_id ?? "—"}</p>
<p className="truncate">
Whisper: {debugSnapshot.voice?.whisper_binary ?? "not found"}
</p>
</div>
</section>
<section className="rounded-2xl border border-white/8 bg-black/10 px-3 py-2">
<div className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-70">
Screen Intelligence
@@ -704,6 +873,7 @@ export function App() {
</div>
</section>
</div>
) : null}
</div>
</div>
</div>
+95
View File
@@ -0,0 +1,95 @@
/**
* HTTP JSON-RPC to the desktop core sidecar (same process as the main app).
* Mirrors the naming normalization in app/src/services/coreRpcClient.ts (subset).
*/
let nextJsonRpcId = 1;
export function normalizeLegacyMethod(method: string): string {
if (method.startsWith("openhuman.accessibility_")) {
return method.replace("openhuman.accessibility_", "openhuman.screen_intelligence_");
}
return method;
}
/** RpcOutcome with non-empty logs serializes as `{ result, logs }` in the core. */
function unwrapCliCompatibleJson<T>(raw: unknown): T {
if (
raw !== null &&
typeof raw === "object" &&
"result" in raw &&
"logs" in raw &&
Array.isArray((raw as { logs: unknown }).logs)
) {
return (raw as { result: T }).result;
}
return raw as T;
}
interface JsonRpcError {
code: number;
message: string;
data?: unknown;
}
interface JsonRpcResponse<T> {
jsonrpc?: string;
id?: number | string | null;
result?: T;
error?: JsonRpcError;
}
/** Default timeout for parent core RPC requests (ms). */
const DEFAULT_RPC_TIMEOUT_MS = 10_000;
export async function callParentCoreRpc<T>(
rpcUrl: string,
method: string,
params: Record<string, unknown> = {},
timeoutMs: number = DEFAULT_RPC_TIMEOUT_MS,
): Promise<T> {
const normalizedMethod = normalizeLegacyMethod(method);
const payload = {
jsonrpc: "2.0" as const,
id: nextJsonRpcId++,
method: normalizedMethod,
params,
};
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
let response: Response;
try {
response = await fetch(rpcUrl, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(payload),
signal: controller.signal,
});
} catch (err) {
clearTimeout(timer);
if (err instanceof DOMException && err.name === "AbortError") {
throw new Error(`Core RPC request timed out after ${timeoutMs}ms (method: ${normalizedMethod})`);
}
throw err;
} finally {
clearTimeout(timer);
}
if (!response.ok) {
const text = await response.text();
throw new Error(`Core RPC HTTP ${response.status}: ${text || response.statusText}`);
}
const json = (await response.json()) as JsonRpcResponse<unknown>;
if (json.error) {
throw new Error(json.error.message || "Core RPC returned an error");
}
if (!Object.prototype.hasOwnProperty.call(json, "result")) {
throw new Error("Core RPC response missing result");
}
return unwrapCliCompatibleJson<T>(json.result);
}
+52 -3
View File
@@ -570,6 +570,29 @@ pub async fn run_server(
host: Option<&str>,
port: Option<u16>,
socketio_enabled: bool,
) -> anyhow::Result<()> {
run_server_inner(host, port, socketio_enabled, false).await
}
/// Like [`run_server`] but marks the instance as embedded — the overlay will
/// **not** be auto-spawned, preventing recursive overlay launches.
pub async fn run_server_embedded(
host: Option<&str>,
port: Option<u16>,
socketio_enabled: bool,
) -> anyhow::Result<()> {
run_server_inner(host, port, socketio_enabled, true).await
}
/// Internal server entrypoint.
///
/// When `embedded_core` is `true` the server skips auto-spawning the overlay
/// (useful when the overlay itself hosts an embedded core to avoid recursion).
async fn run_server_inner(
host: Option<&str>,
port: Option<u16>,
socketio_enabled: bool,
embedded_core: bool,
) -> anyhow::Result<()> {
// Ensure all controllers are registered before starting.
let _ = all::all_registered_controllers();
@@ -631,8 +654,31 @@ pub async fn run_server(
log::info!("[rpc:socketio] disabled (--jsonrpc-only)");
}
// Derive the real bound address from the listener so ephemeral port 0,
// wildcard binds, and IPv6 are handled correctly.
let bound_addr = listener.local_addr()?;
let parent_core_rpc_url = {
let (h, p) = (bound_addr.ip(), bound_addr.port());
let effective_ip = if h.is_unspecified() {
if h.is_ipv6() {
std::net::IpAddr::V6(std::net::Ipv6Addr::LOCALHOST)
} else {
std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST)
}
} else {
h
};
if effective_ip.is_ipv6() {
format!("http://[{effective_ip}]:{p}/rpc")
} else {
format!("http://{effective_ip}:{p}/rpc")
}
};
log::debug!("[core] parent_core_rpc_url = {parent_core_rpc_url}");
// Optional background bootstrap for local AI services.
tokio::spawn(async {
let is_embedded = embedded_core;
tokio::spawn(async move {
match crate::openhuman::config::Config::load_or_init().await {
Ok(config) => {
if config.local_ai.enabled {
@@ -641,8 +687,11 @@ pub async fn run_server(
}
// Launch the overlay Tauri app (transparent debug/voice panel) as a child process.
if config.overlay_enabled {
crate::openhuman::overlay::spawn_overlay();
// Skip when running as an embedded core inside the overlay itself.
if is_embedded {
log::info!("[overlay] embedded core — skipping overlay auto-spawn");
} else if config.overlay_enabled {
crate::openhuman::overlay::spawn_overlay(parent_core_rpc_url.as_str());
} else {
log::info!("[overlay] overlay disabled by config (overlay_enabled = false)");
}
+257 -41
View File
@@ -2,71 +2,177 @@
//!
//! Replaces the separate osascript subprocess spawns and standalone overlay binary
//! with a single persistent Swift process communicating via stdin/stdout JSON.
//!
//! ## Mutex architecture
//!
//! Three globals prevent deadlock between fire-and-forget (show/hide) and
//! request-response (focus/paste) callers:
//!
//! - `UNIFIED_HELPER`: guards the process handle + stdin writer.
//! Held only for the brief duration of a stdin write (~μs).
//! - `RESPONSE_RX`: guards the mpsc receiver that the background reader
//! thread populates. Held only for the duration of `recv_timeout`.
//! - `RECV_SERIALISER`: held for the entire send+receive round-trip so that
//! two callers cannot interleave their reads.
//!
//! Fire-and-forget callers never touch `RESPONSE_RX` or `RECV_SERIALISER`,
//! so `show`/`hide` can proceed while a `focus` query is in-flight.
#[cfg(target_os = "macos")]
use once_cell::sync::Lazy;
#[cfg(target_os = "macos")]
use std::io::{BufRead, BufReader, Write};
#[cfg(target_os = "macos")]
use std::sync::Mutex as StdMutex;
use std::sync::atomic::{AtomicU64, Ordering};
#[cfg(target_os = "macos")]
use std::sync::{mpsc, Mutex as StdMutex};
#[cfg(target_os = "macos")]
use std::time::{Duration, Instant};
#[cfg(target_os = "macos")]
use std::{
fs,
path::PathBuf,
process::{Child, ChildStdin, ChildStdout, Command, Stdio},
process::{Child, ChildStdin, Command, Stdio},
};
#[cfg(target_os = "macos")]
use serde_json::Value;
/// Process handle + stdin writer. Held only briefly for writes.
#[cfg(target_os = "macos")]
struct UnifiedHelperProcess {
child: Child,
stdin: ChildStdin,
stdout: BufReader<ChildStdout>,
}
/// Guards the process handle and stdin.
#[cfg(target_os = "macos")]
static UNIFIED_HELPER: Lazy<StdMutex<Option<UnifiedHelperProcess>>> =
Lazy::new(|| StdMutex::new(None));
/// Channel receiver fed by the background stdout-reader thread.
/// Separate from UNIFIED_HELPER so fire-and-forget callers never contend here.
#[cfg(target_os = "macos")]
static RESPONSE_RX: Lazy<StdMutex<Option<mpsc::Receiver<String>>>> =
Lazy::new(|| StdMutex::new(None));
/// Serialises request/response pairs so two callers cannot interleave reads.
/// Fire-and-forget callers never acquire this lock.
#[cfg(target_os = "macos")]
static RECV_SERIALISER: Lazy<StdMutex<()>> = Lazy::new(|| StdMutex::new(()));
/// Prevents concurrent Swift compiles from `ensure_helper_binary` vs background precompile.
#[cfg(target_os = "macos")]
static HELPER_COMPILE_LOCK: Lazy<StdMutex<()>> = Lazy::new(|| StdMutex::new(()));
/// Monotonic ids for `helper_send_receive` so a late line cannot be consumed as the wrong reply.
#[cfg(target_os = "macos")]
static HELPER_REQ_ID: AtomicU64 = AtomicU64::new(1);
/// Timeout for a single request/response round-trip with the Swift helper.
#[cfg(target_os = "macos")]
const HELPER_RECV_TIMEOUT: Duration = Duration::from_secs(8);
/// Send a JSON request and read a JSON response (one line each).
/// Used for `focus` and `paste` commands that produce a response.
///
/// Holds `RECV_SERIALISER` for the full round-trip, but releases
/// `UNIFIED_HELPER` before blocking on the channel recv, so fire-and-forget
/// callers (`show`/`hide`) are never blocked by an in-flight focus query.
#[cfg(target_os = "macos")]
pub(super) fn helper_send_receive(
request: &serde_json::Value,
) -> Result<serde_json::Value, String> {
ensure_helper_running()?;
let mut guard = UNIFIED_HELPER
// Serialise request/response pairs — prevents interleaved reads.
let _rr_guard = RECV_SERIALISER
.lock()
.map_err(|_| "unified helper lock poisoned".to_string())?;
let helper = guard
.as_mut()
.ok_or_else(|| "unified helper unavailable".to_string())?;
.map_err(|_| "recv serialiser lock poisoned".to_string())?;
// Write request
let line = request.to_string();
helper
.stdin
.write_all(line.as_bytes())
.and_then(|_| helper.stdin.write_all(b"\n"))
.and_then(|_| helper.stdin.flush())
.map_err(|e| format!("failed to write to helper stdin: {e}"))?;
ensure_helper_running()?;
// Read response (one line)
let mut response_line = String::new();
helper
.stdout
.read_line(&mut response_line)
.map_err(|e| format!("failed to read helper stdout: {e}"))?;
let id_num = HELPER_REQ_ID.fetch_add(1, Ordering::Relaxed);
let id_str = id_num.to_string();
if response_line.trim().is_empty() {
return Err("helper returned empty response".to_string());
let mut req = request.clone();
let req_obj = req
.as_object_mut()
.ok_or_else(|| "helper request must be a JSON object".to_string())?;
req_obj.insert("id".to_string(), Value::String(id_str.clone()));
// Write the request, holding UNIFIED_HELPER only for this brief write.
{
let mut guard = UNIFIED_HELPER
.lock()
.map_err(|_| "unified helper lock poisoned".to_string())?;
let helper = guard
.as_mut()
.ok_or_else(|| "unified helper unavailable".to_string())?;
let line = req.to_string();
helper
.stdin
.write_all(line.as_bytes())
.and_then(|_| helper.stdin.write_all(b"\n"))
.and_then(|_| helper.stdin.flush())
.map_err(|e| format!("failed to write to helper stdin: {e}"))?;
} // UNIFIED_HELPER released here — fire-and-forget callers can proceed
// Read until the line matches `id` (discards stale lines after a timeout or reordering).
let deadline = Instant::now() + HELPER_RECV_TIMEOUT;
loop {
let remaining = deadline.saturating_duration_since(Instant::now());
if remaining.is_zero() {
return Err(format!(
"helper response timed out waiting for id {id_str} (stale lines may follow)"
));
}
let chunk = remaining.min(Duration::from_millis(500));
let response_line = {
let rx_guard = RESPONSE_RX
.lock()
.map_err(|_| "response rx lock poisoned".to_string())?;
let rx = rx_guard
.as_ref()
.ok_or_else(|| "response channel unavailable".to_string())?;
match rx.recv_timeout(chunk) {
Ok(line) => line,
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
// Non-fatal: the outer loop will check `remaining` and
// either retry or surface the top-level timeout.
continue;
}
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
return Err("helper response channel disconnected".to_string());
}
}
};
if response_line.trim().is_empty() {
log::debug!(
"[accessibility] helper skipped empty stdout line while waiting for id {id_str}"
);
continue;
}
let value: Value = serde_json::from_str(response_line.trim())
.map_err(|e| format!("failed to parse helper response: {e}"))?;
let matches = value
.get("id")
.and_then(|v| v.as_str())
.is_some_and(|rid| rid == id_str.as_str());
if matches {
return Ok(value);
}
log::debug!(
"[accessibility] discarding helper response with mismatched or missing id (want id={})",
id_str
);
}
serde_json::from_str(response_line.trim())
.map_err(|e| format!("failed to parse helper response: {e}"))
}
/// Send a JSON request without waiting for a response.
/// Used for `show`, `hide`, and `quit` commands.
/// Only acquires UNIFIED_HELPER (for the stdin write) — never blocks on I/O.
#[cfg(target_os = "macos")]
pub(super) fn helper_send_fire_and_forget(request: &serde_json::Value) -> Result<(), String> {
ensure_helper_running()?;
@@ -90,6 +196,13 @@ pub(super) fn helper_send_fire_and_forget(request: &serde_json::Value) -> Result
/// Quit and clean up the helper process.
#[cfg(target_os = "macos")]
pub(super) fn helper_quit() -> Result<(), String> {
// Drop the response channel first so the reader thread exits cleanly.
{
let mut rx_guard = RESPONSE_RX
.lock()
.map_err(|_| "response rx lock poisoned".to_string())?;
rx_guard.take();
}
let mut guard = UNIFIED_HELPER
.lock()
.map_err(|_| "unified helper lock poisoned".to_string())?;
@@ -103,6 +216,8 @@ pub(super) fn helper_quit() -> Result<(), String> {
Ok(())
}
/// Ensure the helper process is running. Spawns it (and the stdout reader
/// thread) if not yet started or if it has exited unexpectedly.
#[cfg(target_os = "macos")]
fn ensure_helper_running() -> Result<(), String> {
let mut guard = UNIFIED_HELPER
@@ -120,6 +235,10 @@ fn ensure_helper_running() -> Result<(), String> {
}
log::debug!("[accessibility] unified helper exited, restarting");
*guard = None;
// Also drop the stale receiver so a new one will be created below.
if let Ok(mut rx_guard) = RESPONSE_RX.lock() {
rx_guard.take();
}
}
let binary_path = ensure_helper_binary()?;
@@ -139,17 +258,69 @@ fn ensure_helper_running() -> Result<(), String> {
.take()
.ok_or_else(|| "failed to capture helper stdout".to_string())?;
*guard = Some(UnifiedHelperProcess {
child,
stdin,
stdout: BufReader::new(stdout),
// Spawn a background thread that continuously reads lines from the helper's
// stdout and forwards them into the channel. The thread exits when the
// sender is dropped (i.e. when helper_quit drops RESPONSE_RX) or when the
// process closes its stdout.
let (tx, rx) = mpsc::channel::<String>();
std::thread::spawn(move || {
let mut reader = BufReader::new(stdout);
let mut line = String::new();
loop {
line.clear();
match reader.read_line(&mut line) {
Ok(0) => break, // EOF — helper exited
Ok(_) => {
let trimmed = line.trim().to_string();
if !trimmed.is_empty() && tx.send(trimmed).is_err() {
break; // Receiver dropped — time to exit
}
}
Err(e) => {
log::debug!("[accessibility] helper stdout reader error: {e}");
break;
}
}
}
log::debug!("[accessibility] helper stdout reader thread exiting");
});
// Store the new receiver.
if let Ok(mut rx_guard) = RESPONSE_RX.lock() {
*rx_guard = Some(rx);
}
*guard = Some(UnifiedHelperProcess { child, stdin });
log::debug!("[accessibility] unified helper started");
Ok(())
}
/// Compile the Swift helper binary in the background so the first overlay
/// request does not incur the compile latency. Safe to call multiple times;
/// subsequent calls are no-ops (the binary is cached by `ensure_helper_binary`).
#[cfg(target_os = "macos")]
pub fn precompile_helper_background() {
std::thread::spawn(|| {
log::debug!("[accessibility] precompile_helper_background: starting");
match ensure_helper_binary() {
Ok(path) => log::debug!("[accessibility] helper binary ready: {}", path.display()),
Err(e) => log::warn!(
"[accessibility] helper precompile failed (will retry on first use): {e}"
),
}
});
}
/// No-op on non-macOS platforms.
#[cfg(not(target_os = "macos"))]
pub fn precompile_helper_background() {}
#[cfg(target_os = "macos")]
fn ensure_helper_binary() -> Result<PathBuf, String> {
let _compile_guard = HELPER_COMPILE_LOCK
.lock()
.map_err(|_| "helper compile lock poisoned".to_string())?;
let cache_dir = std::env::temp_dir().join("openhuman-accessibility-helper");
fs::create_dir_all(&cache_dir).map_err(|e| format!("failed to create cache dir: {e}"))?;
let source_path = cache_dir.join("unified_helper.swift");
@@ -479,11 +650,33 @@ func pasteText(id: String?, text: String) -> [String: Any] {
final class OverlayController {
private var panel: NSPanel?
private var textField: NSTextField?
private var hintField: NSTextField?
private var hideWorkItem: DispatchWorkItem?
func show(x: CGFloat, yTop: CGFloat, width: CGFloat, height: CGFloat, text: String, ttlMs: Int) {
let panelWidth = min(420, max(140, CGFloat(text.count) * 7 + 26))
let panelHeight: CGFloat = 26
func show(x: CGFloat, yTop: CGFloat, width: CGFloat, height: CGFloat, text: String, ttlMs: Int, tabHint: String) {
let showTabHint = !tabHint.isEmpty
// Detect current system appearance for contrast-appropriate colors.
let isDark: Bool = {
if #available(macOS 10.14, *) {
return NSApp.effectiveAppearance
.bestMatch(from: [.darkAqua, .aqua]) == .darkAqua
}
return false
}()
let bgColor = isDark
? NSColor(white: 0.92, alpha: 0.82) // light badge on dark background
: NSColor(white: 0.10, alpha: 0.82) // dark badge on light background
let textColor = isDark
? NSColor(white: 0.08, alpha: 0.95)
: NSColor(white: 1.0, alpha: 0.95)
// Measure badge width from actual text metrics instead of char-count estimate.
let font = NSFont.systemFont(ofSize: 13)
let attrs: [NSAttributedString.Key: Any] = [.font: font]
let measured = (text as NSString).size(withAttributes: attrs)
let hintPad: CGFloat = showTabHint ? 80 : 16
let panelWidth = min(480, max(140, ceil(measured.width) + hintPad))
let panelHeight: CGFloat = 28
// Multi-monitor: find the screen containing the target or mouse cursor.
let screen: NSScreen? = {
@@ -537,23 +730,42 @@ final class OverlayController {
let content = NSView(frame: NSRect(x: 0, y: 0, width: panelWidth, height: panelHeight))
content.wantsLayer = true
content.layer?.cornerRadius = 6
content.layer?.backgroundColor = NSColor(white: 0.08, alpha: 0.35).cgColor
content.layer?.backgroundColor = bgColor.cgColor
p.contentView = content
let label = NSTextField(labelWithString: text)
label.frame = NSRect(x: 8, y: 4, width: panelWidth - 12, height: 18)
label.textColor = NSColor(white: 1.0, alpha: 0.46)
label.font = NSFont.systemFont(ofSize: 13)
label.frame = NSRect(x: 8, y: 5, width: panelWidth - (showTabHint ? 62 : 16), height: 18)
label.textColor = textColor
label.font = font
label.lineBreakMode = .byTruncatingTail
content.addSubview(label)
let hint = NSTextField(labelWithString: tabHint.isEmpty ? "Tab ↵" : tabHint)
hint.frame = NSRect(x: panelWidth - 54, y: 5, width: 48, height: 18)
hint.textColor = NSColor(white: isDark ? 0.35 : 0.65, alpha: 1.0)
hint.font = NSFont.monospacedSystemFont(ofSize: 10, weight: .regular)
hint.alignment = .right
hint.isHidden = !showTabHint
content.addSubview(hint)
panel = p
textField = label
hintField = hint
}
// Re-apply colors on every show so runtime appearance changes are reflected.
panel?.contentView?.layer?.backgroundColor = bgColor.cgColor
textField?.textColor = textColor
hintField?.textColor = NSColor(white: isDark ? 0.35 : 0.65, alpha: 1.0)
hintField?.isHidden = !showTabHint
if showTabHint {
hintField?.stringValue = tabHint
}
panel?.setFrame(NSRect(x: originX, y: originYCocoa, width: panelWidth, height: panelHeight), display: true)
panel?.contentView?.frame = NSRect(x: 0, y: 0, width: panelWidth, height: panelHeight)
textField?.frame = NSRect(x: 8, y: 4, width: panelWidth - 12, height: 18)
textField?.frame = NSRect(x: 8, y: 5, width: panelWidth - (showTabHint ? 62 : 16), height: 18)
hintField?.frame = NSRect(x: panelWidth - 54, y: 5, width: 48, height: 18)
textField?.stringValue = text
panel?.orderFrontRegardless()
@@ -602,8 +814,12 @@ DispatchQueue.global(qos: .userInitiated).async {
let h = CGFloat((payload["h"] as? NSNumber)?.doubleValue ?? 0)
let text = (payload["text"] as? String) ?? ""
let ttl = (payload["ttl_ms"] as? NSNumber)?.intValue ?? 900
let tabHint: String = {
if let s = payload["tab_hint"] as? String { return s }
return "Tab ↵"
}()
DispatchQueue.main.async {
controller.show(x: x, yTop: y, width: w, height: h, text: text, ttlMs: ttl)
controller.show(x: x, yTop: y, width: w, height: h, text: text, ttlMs: ttl, tabHint: tabHint)
}
case "hide":
+67
View File
@@ -1,7 +1,70 @@
//! Key state probes via direct FFI (lightweight, no helper needed).
//!
//! Tab and Escape detection is gated on the Input Monitoring permission.
//! The permission is cached; if initially denied, we re-check occasionally so
//! granting permission without restarting the app still enables Tab/Escape.
#[cfg(target_os = "macos")]
use std::sync::atomic::{AtomicBool, AtomicI64, Ordering};
#[cfg(target_os = "macos")]
static INPUT_MONITORING_GRANTED: AtomicBool = AtomicBool::new(false);
/// Last time we called `detect_input_monitoring_permission` (ms since UNIX epoch).
#[cfg(target_os = "macos")]
static INPUT_MONITORING_LAST_CHECK_MS: AtomicI64 = AtomicI64::new(0);
/// Re-check interval when permission is still denied (avoid IOHID every tick).
#[cfg(target_os = "macos")]
const INPUT_MONITORING_RECHECK_MS: i64 = 2500;
#[cfg(target_os = "macos")]
fn refresh_input_monitoring_cache() {
use std::time::{SystemTime, UNIX_EPOCH};
let now_ms = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_millis() as i64)
.unwrap_or(0);
if INPUT_MONITORING_GRANTED.load(Ordering::Relaxed) {
return;
}
let last = INPUT_MONITORING_LAST_CHECK_MS.load(Ordering::Relaxed);
if now_ms - last < INPUT_MONITORING_RECHECK_MS && last != 0 {
return;
}
INPUT_MONITORING_LAST_CHECK_MS.store(now_ms, Ordering::Relaxed);
use super::permissions::detect_input_monitoring_permission;
use super::types::PermissionState;
let granted = matches!(
detect_input_monitoring_permission(),
PermissionState::Granted
);
if granted {
INPUT_MONITORING_GRANTED.store(true, Ordering::Relaxed);
log::info!("[accessibility] Input Monitoring granted — Tab/Escape key detection enabled.");
return;
}
// First denial: warn once (avoid spam on every recheck interval).
static WARNED: AtomicBool = AtomicBool::new(false);
if !WARNED.swap(true, Ordering::Relaxed) {
log::warn!(
"[accessibility] Input Monitoring permission not granted; \
Tab/Escape key detection disabled until granted. \
Grant in System Settings → Privacy & Security → Input Monitoring."
);
}
}
#[cfg(target_os = "macos")]
pub fn is_tab_key_down() -> bool {
refresh_input_monitoring_cache();
if !INPUT_MONITORING_GRANTED.load(Ordering::Relaxed) {
return false;
}
unsafe { CGEventSourceKeyState(KCG_EVENT_SOURCE_STATE_COMBINED_SESSION_STATE, KVK_TAB) }
}
@@ -12,6 +75,10 @@ pub fn is_tab_key_down() -> bool {
#[cfg(target_os = "macos")]
pub fn is_escape_key_down() -> bool {
refresh_input_monitoring_cache();
if !INPUT_MONITORING_GRANTED.load(Ordering::Relaxed) {
return false;
}
unsafe { CGEventSourceKeyState(KCG_EVENT_SOURCE_STATE_COMBINED_SESSION_STATE, KVK_ESCAPE) }
}
+1
View File
@@ -26,6 +26,7 @@ pub use globe::{
globe_listener_poll, globe_listener_start, globe_listener_stop, GlobeHotkeyPollResult,
GlobeHotkeyStatus,
};
pub use helper::precompile_helper_background;
pub use keys::{is_escape_key_down, is_tab_key_down};
pub use overlay::{hide_overlay, quit_overlay, show_overlay};
pub use paste::apply_text_to_focused_field;
+100 -3
View File
@@ -4,8 +4,16 @@ use super::text_util::truncate_tail;
use super::types::ElementBounds;
/// Show an overlay badge near the given element bounds.
///
/// When `tab_hint` is empty, the Swift helper hides the Tab keyboard hint (used when
/// `accept_with_tab` is disabled in config).
#[cfg(target_os = "macos")]
pub fn show_overlay(bounds: &ElementBounds, text: &str, ttl_ms: u32) -> Result<(), String> {
pub fn show_overlay(
bounds: &ElementBounds,
text: &str,
ttl_ms: u32,
tab_hint: &str,
) -> Result<(), String> {
let message = serde_json::json!({
"type": "show",
"x": bounds.x,
@@ -13,7 +21,8 @@ pub fn show_overlay(bounds: &ElementBounds, text: &str, ttl_ms: u32) -> Result<(
"w": bounds.width,
"h": bounds.height,
"text": truncate_tail(text, 96),
"ttl_ms": ttl_ms
"ttl_ms": ttl_ms,
"tab_hint": tab_hint,
});
super::helper::helper_send_fire_and_forget(&message)
}
@@ -32,7 +41,12 @@ pub fn quit_overlay() -> Result<(), String> {
}
#[cfg(not(target_os = "macos"))]
pub fn show_overlay(_bounds: &ElementBounds, _text: &str, _ttl_ms: u32) -> Result<(), String> {
pub fn show_overlay(
_bounds: &ElementBounds,
_text: &str,
_ttl_ms: u32,
_tab_hint: &str,
) -> Result<(), String> {
Ok(())
}
@@ -45,3 +59,86 @@ pub fn hide_overlay() -> Result<(), String> {
pub fn quit_overlay() -> Result<(), String> {
Ok(())
}
#[cfg(test)]
mod tests {
#[cfg(not(target_os = "macos"))]
use super::*;
// --- Non-macOS stubs always succeed ---
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overlay_non_macos_returns_ok() {
let bounds = ElementBounds {
x: 0,
y: 0,
width: 100,
height: 50,
};
assert!(show_overlay(&bounds, "suggestion text", 900, "Tab ↵").is_ok());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overlay_non_macos_empty_text_returns_ok() {
let bounds = ElementBounds {
x: 10,
y: 20,
width: 0,
height: 0,
};
assert!(show_overlay(&bounds, "", 500, "").is_ok());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overlay_non_macos_zero_ttl_returns_ok() {
let bounds = ElementBounds {
x: 0,
y: 0,
width: 200,
height: 30,
};
assert!(show_overlay(&bounds, "hello", 0, "Tab ↵").is_ok());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overlay_non_macos_max_ttl_returns_ok() {
let bounds = ElementBounds {
x: -10,
y: -5,
width: 300,
height: 60,
};
assert!(show_overlay(&bounds, "test", u32::MAX, "Tab ↵").is_ok());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn hide_overlay_non_macos_returns_ok() {
assert!(hide_overlay().is_ok());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn quit_overlay_non_macos_returns_ok() {
assert!(quit_overlay().is_ok());
}
// Verify overlay functions can be called multiple times without error
#[cfg(not(target_os = "macos"))]
#[test]
fn hide_overlay_idempotent() {
assert!(hide_overlay().is_ok());
assert!(hide_overlay().is_ok());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn quit_overlay_idempotent() {
assert!(quit_overlay().is_ok());
assert!(quit_overlay().is_ok());
}
}
+24 -7
View File
@@ -152,12 +152,16 @@ fn clipboard_save_osascript() -> Option<String> {
}
/// Fallback insertion: direct AXValue write via AppleScript.
/// Reads `AXSelectedTextRange` to insert at the cursor position rather than
/// always appending to the end of the field.
#[cfg(target_os = "macos")]
fn apply_text_via_axvalue(text: &str) -> Result<(), String> {
let escaped = text
.replace('\\', "\\\\")
.replace('\"', "\\\"")
.replace('\n', " ");
// AXSelectedTextRange.location is 0-based; AppleScript string indices are 1-based.
// "text 1 thru 0" evaluates to "" in AppleScript — correct for cursor-at-start.
let script = format!(
r##"
tell application "System Events"
@@ -168,16 +172,29 @@ tell application "System Events"
set currentValue to value of attribute "AXValue" of focusedElement as text
end try
if currentValue is "missing value" then set currentValue to ""
if currentValue is "" then
try
set currentValue to value of attribute "AXSelectedText" of focusedElement as text
end try
if currentValue is "missing value" then set currentValue to ""
-- Read cursor position from AXSelectedTextRange (0-based location).
set insertionOffset to -1
try
set selRange to value of attribute "AXSelectedTextRange" of focusedElement
set insertionOffset to location of selRange
end try
-- Insert at cursor when available, otherwise append.
if insertionOffset >= 0 and insertionOffset <= (length of currentValue) then
if insertionOffset = 0 then
set value of attribute "AXValue" of focusedElement to ("{}" & currentValue)
else if insertionOffset >= (length of currentValue) then
set value of attribute "AXValue" of focusedElement to (currentValue & "{}")
else
set beforeCursor to text 1 thru insertionOffset of currentValue
set afterCursor to text (insertionOffset + 1) thru -1 of currentValue
set value of attribute "AXValue" of focusedElement to (beforeCursor & "{}" & afterCursor)
end if
else
set value of attribute "AXValue" of focusedElement to (currentValue & "{}")
end if
set value of attribute "AXValue" of focusedElement to (currentValue & "{}")
end tell
"##,
escaped
escaped, escaped, escaped, escaped
);
let output = std::process::Command::new("osascript")
.arg("-e")
+146
View File
@@ -34,3 +34,149 @@ pub fn parse_ax_number(raw: &str) -> Option<i32> {
Some(rounded as i32)
})
}
#[cfg(test)]
mod tests {
use super::*;
// --- truncate_tail ---
#[test]
fn truncate_tail_shorter_than_max_returns_original() {
assert_eq!(truncate_tail("hello", 10), "hello");
}
#[test]
fn truncate_tail_exactly_max_returns_original() {
assert_eq!(truncate_tail("hello", 5), "hello");
}
#[test]
fn truncate_tail_longer_than_max_returns_tail() {
assert_eq!(truncate_tail("hello", 3), "llo");
}
#[test]
fn truncate_tail_empty_string() {
assert_eq!(truncate_tail("", 5), "");
}
#[test]
fn truncate_tail_zero_max_returns_empty() {
assert_eq!(truncate_tail("hello", 0), "");
}
#[test]
fn truncate_tail_multibyte_chars_counts_chars_not_bytes() {
// "héllo" is 5 chars; last 3 = "llo"
assert_eq!(truncate_tail("héllo", 3), "llo");
}
#[test]
fn truncate_tail_unicode_emoji_counts_codepoints() {
// "ab🎉cd" — 5 codepoints; last 3 = "🎉cd"
assert_eq!(truncate_tail("ab🎉cd", 3), "🎉cd");
}
// --- normalize_ax_value ---
#[test]
fn normalize_ax_value_trims_whitespace() {
assert_eq!(normalize_ax_value(" hello "), "hello");
}
#[test]
fn normalize_ax_value_missing_value_lowercase_returns_empty() {
assert_eq!(normalize_ax_value("missing value"), "");
}
#[test]
fn normalize_ax_value_missing_value_uppercase_returns_empty() {
assert_eq!(normalize_ax_value("MISSING VALUE"), "");
}
#[test]
fn normalize_ax_value_mixed_case_missing_value_returns_empty() {
assert_eq!(normalize_ax_value("Missing Value"), "");
}
#[test]
fn normalize_ax_value_empty_string_returns_empty() {
assert_eq!(normalize_ax_value(""), "");
}
#[test]
fn normalize_ax_value_only_whitespace_returns_empty() {
assert_eq!(normalize_ax_value(" "), "");
}
#[test]
fn normalize_ax_value_regular_text_unchanged() {
assert_eq!(normalize_ax_value("some value"), "some value");
}
// --- parse_ax_number ---
#[test]
fn parse_ax_number_integer_string() {
assert_eq!(parse_ax_number("42"), Some(42));
}
#[test]
fn parse_ax_number_negative_integer() {
assert_eq!(parse_ax_number("-7"), Some(-7));
}
#[test]
fn parse_ax_number_float_rounds_to_nearest() {
assert_eq!(parse_ax_number("42.4"), Some(42));
assert_eq!(parse_ax_number("42.6"), Some(43));
}
#[test]
fn parse_ax_number_comma_treated_as_decimal_separator() {
// Locale-style: "1,5" → 1.5 → rounds to 2
assert_eq!(parse_ax_number("1,5"), Some(2));
}
#[test]
fn parse_ax_number_missing_value_returns_none() {
assert_eq!(parse_ax_number("missing value"), None);
}
#[test]
fn parse_ax_number_empty_returns_none() {
assert_eq!(parse_ax_number(""), None);
}
#[test]
fn parse_ax_number_whitespace_only_returns_none() {
assert_eq!(parse_ax_number(" "), None);
}
#[test]
fn parse_ax_number_non_numeric_returns_none() {
assert_eq!(parse_ax_number("abc"), None);
}
#[test]
fn parse_ax_number_nan_returns_none() {
assert_eq!(parse_ax_number("NaN"), None);
}
#[test]
fn parse_ax_number_infinity_returns_none() {
assert_eq!(parse_ax_number("inf"), None);
assert_eq!(parse_ax_number("infinity"), None);
}
#[test]
fn parse_ax_number_zero() {
assert_eq!(parse_ax_number("0"), Some(0));
}
#[test]
fn parse_ax_number_trims_surrounding_whitespace() {
assert_eq!(parse_ax_number(" 10 "), Some(10));
}
}
+143
View File
@@ -71,3 +71,146 @@ pub enum PermissionKind {
Accessibility,
InputMonitoring,
}
#[cfg(test)]
mod tests {
use super::*;
fn make_ctx(
app: Option<&str>,
title: Option<&str>,
bounds: Option<ElementBounds>,
) -> AppContext {
AppContext {
app_name: app.map(str::to_string),
window_title: title.map(str::to_string),
bounds,
}
}
fn make_bounds(x: i32, y: i32, w: i32, h: i32) -> ElementBounds {
ElementBounds {
x,
y,
width: w,
height: h,
}
}
// --- AppContext::same_as ---
#[test]
fn same_as_identical_contexts_true() {
let a = make_ctx(
Some("App"),
Some("Window"),
Some(make_bounds(0, 0, 800, 600)),
);
let b = make_ctx(
Some("App"),
Some("Window"),
Some(make_bounds(0, 0, 800, 600)),
);
assert!(a.same_as(&b));
}
#[test]
fn same_as_both_none_fields_true() {
let a = make_ctx(None, None, None);
let b = make_ctx(None, None, None);
assert!(a.same_as(&b));
}
#[test]
fn same_as_different_app_name_false() {
let a = make_ctx(Some("AppA"), Some("Window"), None);
let b = make_ctx(Some("AppB"), Some("Window"), None);
assert!(!a.same_as(&b));
}
#[test]
fn same_as_different_window_title_false() {
let a = make_ctx(Some("App"), Some("Win1"), None);
let b = make_ctx(Some("App"), Some("Win2"), None);
assert!(!a.same_as(&b));
}
#[test]
fn same_as_one_has_bounds_other_none_false() {
let a = make_ctx(Some("App"), None, Some(make_bounds(0, 0, 100, 100)));
let b = make_ctx(Some("App"), None, None);
assert!(!a.same_as(&b));
}
#[test]
fn same_as_different_bounds_x_false() {
let a = make_ctx(None, None, Some(make_bounds(10, 0, 100, 100)));
let b = make_ctx(None, None, Some(make_bounds(20, 0, 100, 100)));
assert!(!a.same_as(&b));
}
#[test]
fn same_as_different_bounds_y_false() {
let a = make_ctx(None, None, Some(make_bounds(0, 10, 100, 100)));
let b = make_ctx(None, None, Some(make_bounds(0, 20, 100, 100)));
assert!(!a.same_as(&b));
}
#[test]
fn same_as_different_bounds_width_false() {
let a = make_ctx(None, None, Some(make_bounds(0, 0, 100, 100)));
let b = make_ctx(None, None, Some(make_bounds(0, 0, 200, 100)));
assert!(!a.same_as(&b));
}
#[test]
fn same_as_different_bounds_height_false() {
let a = make_ctx(None, None, Some(make_bounds(0, 0, 100, 100)));
let b = make_ctx(None, None, Some(make_bounds(0, 0, 100, 200)));
assert!(!a.same_as(&b));
}
#[test]
fn same_as_reflexive() {
let a = make_ctx(Some("App"), Some("Win"), Some(make_bounds(1, 2, 3, 4)));
assert!(a.same_as(&a));
}
// --- AppContext::as_compound_text ---
#[test]
fn as_compound_text_both_some_lowercase() {
let ctx = make_ctx(Some("MyApp"), Some("My Window"), None);
assert_eq!(ctx.as_compound_text(), "myapp my window");
}
#[test]
fn as_compound_text_app_none_title_some() {
let ctx = make_ctx(None, Some("Window Title"), None);
assert_eq!(ctx.as_compound_text(), " window title");
}
#[test]
fn as_compound_text_app_some_title_none() {
let ctx = make_ctx(Some("AppName"), None, None);
assert_eq!(ctx.as_compound_text(), "appname ");
}
#[test]
fn as_compound_text_both_none_returns_space() {
let ctx = make_ctx(None, None, None);
assert_eq!(ctx.as_compound_text(), " ");
}
#[test]
fn as_compound_text_already_lowercase_unchanged() {
let ctx = make_ctx(Some("slack"), Some("general"), None);
assert_eq!(ctx.as_compound_text(), "slack general");
}
#[test]
fn as_compound_text_mixed_case_lowercased() {
let ctx = make_ctx(Some("VS Code"), Some("README.md"), None);
assert_eq!(ctx.as_compound_text(), "vs code readme.md");
}
}
+30 -5
View File
@@ -2,7 +2,7 @@ use crate::openhuman::config::Config;
use crate::openhuman::local_ai;
use chrono::Utc;
use once_cell::sync::Lazy;
use std::sync::Arc;
use std::sync::{Arc, Once};
use tokio::sync::Mutex;
use tokio::task::JoinHandle;
use tokio::time::{self, Duration, Instant};
@@ -114,6 +114,14 @@ impl AutocompleteEngine {
return Ok(AutocompleteStartResult { started: false });
}
// Kick off Swift helper compilation in the background so the first
// suggestion request does not stall waiting for `swiftc`.
// Only after we know config loaded and autocomplete is enabled.
static PRECOMPILE_ONCE: Once = Once::new();
PRECOMPILE_ONCE.call_once(|| {
crate::openhuman::accessibility::precompile_helper_background();
});
let debounce_ms = params
.debounce_ms
.unwrap_or(config.autocomplete.debounce_ms)
@@ -180,6 +188,8 @@ impl AutocompleteEngine {
Some(&error_message),
None,
None,
700,
false,
);
}
}
@@ -346,7 +356,7 @@ impl AutocompleteEngine {
state.last_overlay_signature = None;
}
if should_apply {
show_overflow_badge("accepted", Some(&cleaned), None, None, None);
show_overflow_badge("accepted", Some(&cleaned), None, None, None, 700, false);
}
// Persist acceptance for personalisation (fire-and-forget).
@@ -425,6 +435,9 @@ impl AutocompleteEngine {
if let Some(accept_with_tab) = params.accept_with_tab {
config.autocomplete.accept_with_tab = accept_with_tab;
}
if let Some(overlay_ttl_ms) = params.overlay_ttl_ms {
config.autocomplete.overlay_ttl_ms = overlay_ttl_ms.clamp(300, 10_000);
}
config.save().await.map_err(|e| e.to_string())?;
let mut state = self.inner.lock().await;
@@ -620,7 +633,8 @@ impl AutocompleteEngine {
}
state.suggestion = Some(AutocompleteSuggestion {
value: suggestion.clone(),
confidence: 0.72,
// Placeholder until `local_ai::inline_complete` surfaces a real score (avoid 0.0 so UI/thresholds keep signal).
confidence: 0.75,
});
state.phase = "ready".to_string();
state.last_error = None;
@@ -631,6 +645,7 @@ impl AutocompleteEngine {
);
if !is_in_app && state.last_overlay_signature.as_deref() != Some(ready_signature.as_str()) {
state.last_overlay_signature = Some(ready_signature);
let overlay_ttl_ms = config.autocomplete.overlay_ttl_ms;
drop(state);
show_overflow_badge(
"ready",
@@ -638,6 +653,8 @@ impl AutocompleteEngine {
None,
app_name.as_deref(),
focused.bounds.as_ref(),
overlay_ttl_ms,
config.autocomplete.accept_with_tab,
);
return Ok(());
}
@@ -723,7 +740,15 @@ impl AutocompleteEngine {
.unwrap_or_default()
.to_lowercase();
if !app_lower.contains("openhuman") {
show_overflow_badge("accepted", Some(&cleaned), None, None, None);
show_overflow_badge(
"accepted",
Some(&cleaned),
None,
None,
None,
700,
false,
);
}
}
@@ -783,7 +808,7 @@ impl AutocompleteEngine {
.unwrap_or_default()
.to_lowercase();
if !app_lower.contains("openhuman") {
show_overflow_badge("rejected", Some(&value), None, None, None);
show_overflow_badge("rejected", Some(&value), None, None, None, 700, false);
}
}
Ok(())
+155 -14
View File
@@ -23,10 +23,12 @@ pub(super) fn show_overflow_badge(
error: Option<&str>,
app_name: Option<&str>,
anchor_bounds: Option<&ElementBounds>,
ttl_ms: u32,
// When `kind == "ready"`, show the Tab hint in the overlay only if true.
show_tab_hint: bool,
) {
#[cfg(target_os = "macos")]
{
const READY_THROTTLE_MS: i64 = 1_200;
let now_ms = Utc::now().timestamp_millis();
let signature = format!(
"{}:{}:{}:{}",
@@ -36,12 +38,11 @@ pub(super) fn show_overflow_badge(
error.unwrap_or_default()
);
// Deduplicate rapid duplicate events only (same payload within a short window).
const DEDUP_MS: i64 = 400;
if let Ok(mut guard) = LAST_OVERFLOW_BADGE.lock() {
if let Some((last_signature, last_ms)) = guard.as_ref() {
if *last_signature == signature {
return;
}
if kind == "ready" && (now_ms - *last_ms) < READY_THROTTLE_MS {
if *last_signature == signature && now_ms - *last_ms < DEDUP_MS {
return;
}
}
@@ -58,16 +59,18 @@ pub(super) fn show_overflow_badge(
width: 0,
height: 0,
};
let bounds = if anchor_bounds.is_some() {
anchor_bounds.unwrap()
} else {
log::debug!(
"[autocomplete] overlay: no anchor bounds, falling back to zero bounds (mouse cursor); suggestion={:?}",
truncate_tail(suggestion_text, 40)
);
&fallback_bounds
let bounds: &ElementBounds = match anchor_bounds {
Some(b) => b,
None => {
log::debug!(
"[autocomplete] overlay: no anchor bounds, falling back to zero bounds (mouse cursor); suggestion={:?}",
truncate_tail(suggestion_text, 40)
);
&fallback_bounds
}
};
if accessibility::show_overlay(bounds, suggestion_text, 1100).is_ok() {
let tab_hint = if show_tab_hint { "Tab ↵" } else { "" };
if accessibility::show_overlay(bounds, suggestion_text, ttl_ms, tab_hint).is_ok() {
return;
}
}
@@ -133,3 +136,141 @@ fn escape_osascript_text(raw: &str) -> String {
pub(super) fn overlay_helper_quit() -> Result<(), String> {
accessibility::quit_overlay()
}
#[cfg(test)]
mod tests {
use super::*;
// --- overlay_helper_quit (cross-platform) ---
#[cfg(not(target_os = "macos"))]
#[test]
fn overlay_helper_quit_non_macos_returns_ok() {
assert!(overlay_helper_quit().is_ok());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn overlay_helper_quit_non_macos_idempotent() {
assert!(overlay_helper_quit().is_ok());
assert!(overlay_helper_quit().is_ok());
}
// --- escape_osascript_text (macOS-only) ---
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_plain_string_unchanged() {
assert_eq!(escape_osascript_text("hello world"), "hello world");
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_escapes_double_quotes() {
assert_eq!(escape_osascript_text(r#"say "hello""#), r#"say \"hello\""#);
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_escapes_backslash() {
assert_eq!(escape_osascript_text(r"back\slash"), r"back\\slash");
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_replaces_newline_with_space() {
assert_eq!(escape_osascript_text("line1\nline2"), "line1 line2");
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_replaces_carriage_return_with_space() {
assert_eq!(escape_osascript_text("line1\rline2"), "line1 line2");
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_crlf_both_replaced() {
// \r and \n are each replaced individually → two spaces
assert_eq!(escape_osascript_text("a\r\nb"), "a b");
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_empty_string_unchanged() {
assert_eq!(escape_osascript_text(""), "");
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_backslash_before_quote_double_escapes() {
// r#"\"# + `"` = `\"` — backslash first becomes `\\`, then `"` becomes `\"`
assert_eq!(escape_osascript_text("\\\""), "\\\\\\\"");
}
#[cfg(target_os = "macos")]
#[test]
fn escape_osascript_text_multiple_quotes() {
assert_eq!(
escape_osascript_text(r#""a" and "b""#),
r#"\"a\" and \"b\""#
);
}
// --- show_overflow_badge signature (non-macOS no-op smoke test) ---
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overflow_badge_non_macos_does_not_panic_ready() {
let bounds = crate::openhuman::accessibility::ElementBounds {
x: 0,
y: 0,
width: 100,
height: 50,
};
// Should be a no-op and not panic.
show_overflow_badge(
"ready",
Some("suggestion"),
None,
Some("TestApp"),
Some(&bounds),
900,
true,
);
}
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overflow_badge_non_macos_does_not_panic_error() {
show_overflow_badge(
"error",
None,
Some("something failed"),
None,
None,
500,
false,
);
}
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overflow_badge_non_macos_does_not_panic_accepted() {
show_overflow_badge(
"accepted",
Some("accepted text"),
None,
None,
None,
0,
false,
);
}
#[cfg(not(target_os = "macos"))]
#[test]
fn show_overflow_badge_non_macos_does_not_panic_rejected() {
show_overflow_badge("rejected", None, None, None, None, 200, false);
}
}
+132
View File
@@ -26,3 +26,135 @@ pub(super) fn sanitize_suggestion(text: &str) -> String {
pub(super) fn is_no_text_candidate_error(err: &str) -> bool {
err.contains("ERROR:no_text_candidate_found")
}
#[cfg(test)]
mod tests {
use super::*;
// --- truncate_head ---
#[test]
fn truncate_head_shorter_than_max_returns_original() {
assert_eq!(truncate_head("hello", 10), "hello");
}
#[test]
fn truncate_head_exactly_max_returns_original() {
assert_eq!(truncate_head("hello", 5), "hello");
}
#[test]
fn truncate_head_longer_than_max_returns_head() {
assert_eq!(truncate_head("hello world", 5), "hello");
}
#[test]
fn truncate_head_empty_string() {
assert_eq!(truncate_head("", 5), "");
}
#[test]
fn truncate_head_zero_max_returns_empty() {
assert_eq!(truncate_head("hello", 0), "");
}
#[test]
fn truncate_head_multibyte_chars_counts_codepoints() {
// "héllo" is 5 chars; first 3 = "hél"
assert_eq!(truncate_head("héllo", 3), "hél");
}
// --- sanitize_suggestion ---
#[test]
fn sanitize_suggestion_plain_text() {
assert_eq!(sanitize_suggestion("hello world"), "hello world");
}
#[test]
fn sanitize_suggestion_trims_leading_and_trailing_whitespace() {
assert_eq!(sanitize_suggestion(" hello "), "hello");
}
#[test]
fn sanitize_suggestion_strips_surrounding_double_quotes() {
assert_eq!(sanitize_suggestion("\"quoted\""), "quoted");
}
#[test]
fn sanitize_suggestion_takes_first_line_only() {
assert_eq!(sanitize_suggestion("line one\nline two"), "line one");
}
#[test]
fn sanitize_suggestion_crlf_newline_takes_first_line() {
assert_eq!(sanitize_suggestion("line one\r\nline two"), "line one");
}
#[test]
fn sanitize_suggestion_replaces_embedded_tabs_with_spaces() {
// Leading/trailing tabs are stripped by trim(); only interior tabs become spaces.
assert_eq!(sanitize_suggestion("he\tllo"), "he llo");
}
#[test]
fn sanitize_suggestion_empty_input_returns_empty() {
assert_eq!(sanitize_suggestion(""), "");
}
#[test]
fn sanitize_suggestion_whitespace_only_returns_empty() {
assert_eq!(sanitize_suggestion(" \n "), "");
}
#[test]
fn sanitize_suggestion_truncates_to_max_chars() {
// MAX_SUGGESTION_CHARS is 64 — a 70-char string should be cut to 64.
let long = "a".repeat(70);
let result = sanitize_suggestion(&long);
assert_eq!(result.len(), 64);
assert!(result.chars().all(|c| c == 'a'));
}
#[test]
fn sanitize_suggestion_exactly_max_chars_unchanged() {
let exact = "b".repeat(64);
assert_eq!(sanitize_suggestion(&exact), exact);
}
#[test]
fn sanitize_suggestion_removes_bare_carriage_return() {
// Bare \r is NOT treated as a line ending by lines(), so it stays in the
// first-line content and is then removed by replace('\r', "").
assert_eq!(sanitize_suggestion("hello\rworld"), "helloworld");
}
// --- is_no_text_candidate_error ---
#[test]
fn is_no_text_candidate_error_exact_match() {
assert!(is_no_text_candidate_error("ERROR:no_text_candidate_found"));
}
#[test]
fn is_no_text_candidate_error_substring_match() {
assert!(is_no_text_candidate_error(
"AX query failed: ERROR:no_text_candidate_found"
));
}
#[test]
fn is_no_text_candidate_error_unrelated_error() {
assert!(!is_no_text_candidate_error("some other error"));
}
#[test]
fn is_no_text_candidate_error_empty_string() {
assert!(!is_no_text_candidate_error(""));
}
#[test]
fn is_no_text_candidate_error_partial_prefix_no_match() {
assert!(!is_no_text_candidate_error("ERROR:no_text"));
}
}
+1
View File
@@ -92,6 +92,7 @@ pub struct AutocompleteSetStyleParams {
pub style_examples: Option<Vec<String>>,
pub disabled_apps: Option<Vec<String>>,
pub accept_with_tab: Option<bool>,
pub overlay_ttl_ms: Option<u32>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -19,6 +19,8 @@ pub struct AutocompleteConfig {
pub disabled_apps: Vec<String>,
#[serde(default = "default_accept_with_tab")]
pub accept_with_tab: bool,
#[serde(default = "default_overlay_ttl_ms")]
pub overlay_ttl_ms: u32,
}
fn default_enabled() -> bool {
@@ -41,6 +43,10 @@ fn default_accept_with_tab() -> bool {
true
}
fn default_overlay_ttl_ms() -> u32 {
1100
}
impl Default for AutocompleteConfig {
fn default() -> Self {
Self {
@@ -52,6 +58,7 @@ impl Default for AutocompleteConfig {
style_examples: Vec::new(),
disabled_apps: vec![],
accept_with_tab: default_accept_with_tab(),
overlay_ttl_ms: default_overlay_ttl_ms(),
}
}
}
+25 -7
View File
@@ -10,22 +10,40 @@ use std::process::{Command, Stdio};
/// Attempt to find and spawn the overlay binary.
///
/// `parent_core_rpc_url` must be the JSON-RPC URL of the **same** core process
/// that spawned the overlay (e.g. `http://127.0.0.1:7788/rpc`) so the overlay
/// UI can query autocomplete, screen intelligence, and voice state from the
/// desktop sidecar instead of a separate in-process core instance.
///
/// This is best-effort: if the binary is not found or fails to launch, a
/// warning is logged and the core continues normally.
pub fn spawn_overlay() {
pub fn spawn_overlay(parent_core_rpc_url: &str) {
let Some(overlay_bin) = find_overlay_binary() else {
log::debug!("[overlay] openhuman-overlay binary not found — skipping overlay launch");
log::warn!(
"[overlay] openhuman-overlay binary not found — skipping overlay launch (set OPENHUMAN_OVERLAY_BIN or build overlay/src-tauri)"
);
return;
};
log::info!("[overlay] launching overlay: {}", overlay_bin.display());
log::info!(
"[overlay] launching overlay: {} (parent RPC {})",
overlay_bin.display(),
parent_core_rpc_url
);
match Command::new(&overlay_bin)
.stdin(Stdio::null())
let mut cmd = Command::new(&overlay_bin);
cmd.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
{
.env(
"OPENHUMAN_OVERLAY_PARENT_RPC_URL",
parent_core_rpc_url.trim(),
)
// The desktop core sets OPENHUMAN_CORE_PORT; the overlay must not inherit it or its
// optional embedded JSON-RPC server will try to bind the same port as the parent.
.env_remove("OPENHUMAN_CORE_PORT");
match cmd.spawn() {
Ok(child) => {
log::info!("[overlay] overlay process spawned (pid={})", child.id());
}
+1 -1
View File
@@ -409,7 +409,7 @@ pub async fn skills_list_available(
#[cfg(test)]
mod tests {
use super::super::registry_cache::{cache_path, is_cache_fresh, write_cache};
use super::super::registry_cache::{is_cache_fresh, write_cache};
use super::super::registry_types::{CachedRegistry, RegistrySkillCategories, SkillCategory};
use super::*;
@@ -103,7 +103,7 @@ mod tests {
)
.await;
let notion_doc_id = ingest(
let _notion_doc_id = ingest(
&memory,
"skill-notion",
"tick1-tracker",
+1 -1
View File
@@ -114,7 +114,7 @@ pub async fn show_ghost(
element_bounds.height,
);
match accessibility::show_overlay(&element_bounds, &params.text, ttl_ms) {
match accessibility::show_overlay(&element_bounds, &params.text, ttl_ms, "") {
Ok(()) => Ok(RpcOutcome::single_log(
ShowGhostTextResult {
shown: true,
-2
View File
@@ -21,9 +21,7 @@ pub async fn try_dispatch(
#[cfg(test)]
mod tests {
use futures_util::FutureExt;
use serde_json::json;
use std::panic::AssertUnwindSafe;
use super::try_dispatch;