mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
feat: improve overlay debug visibility and triggers (#326)
* feat: initialize OpenHuman overlay with log viewer and Tauri integration - Added a new overlay project for OpenHuman, featuring a transparent window with a log viewer. - Implemented core components including TitleBar, ModuleFilter, LogViewer, and StatusBar for enhanced user interaction. - Integrated Tauri for desktop application capabilities, allowing for real-time log monitoring and management. - Configured TypeScript, Tailwind CSS, and Vite for a modern development experience. - Included necessary files such as package.json, tsconfig.json, and README.md to guide setup and usage. * refactor: update async task spawning in Tauri integration - Replaced `tokio::spawn` with `tauri::async_runtime::spawn` for improved compatibility with Tauri's async runtime. - This change enhances the integration of the OpenHuman core server within the Tauri application, ensuring better performance and stability. * feat: implement click-through toggle in overlay title bar - Added a click-through toggle feature to the TitleBar component, allowing mouse events to pass through the overlay. - Updated the App component to manage the click-through state and handle its toggling. - Enhanced the MODULE_LABELS in types.ts to include additional known modules for filtering. - Modified Tauri backend to support the click-through functionality, ensuring proper state management and event handling. * feat: implement audio transcription functionality in overlay - Added audio recording and transcription capabilities to the overlay, allowing users to capture and transcribe speech. - Introduced functions for audio processing, including converting audio blobs to WAV format and handling audio streams. - Updated the App component to manage recording states and display transcription results. - Enhanced the user interface with a microphone icon indicating recording status. - Adjusted window dimensions and properties in the Tauri configuration for improved user experience. * feat: enhance transcript insertion and logging in overlay - Implemented a new function to insert transcribed text into the currently focused field of the active application, improving user experience. - Added detailed logging for overlay actions, including recording start/stop events and transcript insertion attempts, to aid in debugging and monitoring. - Updated the App component to utilize the new insertion functionality and log relevant information during transcription processes. * feat: implement macOS Globe/Fn key listener integration - Added a new Globe/Fn key listener feature for macOS, enabling the application to respond to Globe key events. - Implemented functions to start, poll, and stop the Globe listener, enhancing user interaction with the overlay. - Updated the App component to manage the listener's lifecycle and display relevant status messages. - Modified Tauri configuration to adjust overlay visibility settings for improved user experience. - Introduced a new module for Globe listener management, encapsulating related functionality and improving code organization. * feat: add macOS support for accessory activation policy and create Info.plist - Introduced a new Info.plist file to configure macOS application settings. - Implemented accessory activation policy for the overlay on macOS, enhancing user experience by allowing the app to run in the background without a dock icon. - Updated the application setup to include macOS-specific configurations for improved functionality. * feat: enhance overlay debug state and UI components - Introduced new interfaces for managing accessibility and autocomplete statuses, improving the structure of debug information. - Implemented a polling mechanism to refresh the overlay debug state, capturing accessibility and autocomplete data in real-time. - Updated the App component to display active application and window titles, along with autocomplete suggestions and phases. - Adjusted the overlay dimensions in Tauri configuration for better user experience and visibility. - Enhanced logging for debug snapshot updates, aiding in monitoring and troubleshooting. * refactor: improve code formatting and readability in accessibility and screen intelligence modules - Reformatted code in `globe.rs`, `ops.rs`, `schemas.rs`, and `types.rs` for better clarity and consistency. - Enhanced logging statements and function definitions to follow a more uniform style. - Updated imports in `types.rs` to maintain organization and improve accessibility module integration.
This commit is contained in:
Steven Enamakel
@@ -0,0 +1,24 @@
|
||||
# Logs
|
||||
logs
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
pnpm-debug.log*
|
||||
lerna-debug.log*
|
||||
|
||||
node_modules
|
||||
dist
|
||||
dist-ssr
|
||||
*.local
|
||||
|
||||
# Editor directories and files
|
||||
.vscode/*
|
||||
!.vscode/extensions.json
|
||||
.idea
|
||||
.DS_Store
|
||||
*.suo
|
||||
*.ntvs*
|
||||
*.njsproj
|
||||
*.sln
|
||||
*.sw?
|
||||
Vendored
+3
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"recommendations": ["tauri-apps.tauri-vscode", "rust-lang.rust-analyzer"]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
# Tauri + Vanilla TS
|
||||
|
||||
This template should help get you started developing with Tauri in vanilla HTML, CSS and Typescript.
|
||||
|
||||
## Recommended IDE Setup
|
||||
|
||||
- [VS Code](https://code.visualstudio.com/) + [Tauri](https://marketplace.visualstudio.com/items?itemName=tauri-apps.tauri-vscode) + [rust-analyzer](https://marketplace.visualstudio.com/items?itemName=rust-lang.rust-analyzer)
|
||||
@@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>OpenHuman Overlay</title>
|
||||
</head>
|
||||
<body class="bg-transparent">
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/main.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "openhuman-overlay",
|
||||
"private": true,
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
"build": "tsc && vite build",
|
||||
"preview": "vite preview",
|
||||
"tauri": "tauri"
|
||||
},
|
||||
"dependencies": {
|
||||
"@tauri-apps/api": "^2",
|
||||
"@tauri-apps/plugin-opener": "^2",
|
||||
"react": "^19.1.0",
|
||||
"react-dom": "^19.1.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@tauri-apps/cli": "^2",
|
||||
"@types/react": "^19.1.0",
|
||||
"@types/react-dom": "^19.1.0",
|
||||
"@vitejs/plugin-react": "^4.4.1",
|
||||
"autoprefixer": "^10.4.20",
|
||||
"postcss": "^8.5.3",
|
||||
"tailwindcss": "^3.4.17",
|
||||
"typescript": "~5.6.2",
|
||||
"vite": "^6.0.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
export default {
|
||||
plugins: {
|
||||
tailwindcss: {},
|
||||
autoprefixer: {},
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,7 @@
|
||||
# Generated by Cargo
|
||||
# will have compiled files and executables
|
||||
/target/
|
||||
|
||||
# Generated by Tauri
|
||||
# will have schema files for capabilities auto-completion
|
||||
/gen/schemas
|
||||
Generated
+9100
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,29 @@
|
||||
[package]
|
||||
name = "openhuman-overlay"
|
||||
version = "0.1.0"
|
||||
description = "OpenHuman transparent overlay with debug log viewer"
|
||||
authors = ["you"]
|
||||
edition = "2021"
|
||||
|
||||
[lib]
|
||||
name = "overlay_lib"
|
||||
crate-type = ["staticlib", "cdylib", "rlib"]
|
||||
|
||||
[build-dependencies]
|
||||
tauri-build = { version = "2", features = [] }
|
||||
|
||||
[dependencies]
|
||||
tauri = { version = "2", features = ["macos-private-api"] }
|
||||
tauri-plugin-opener = "2"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
log = "0.4"
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
|
||||
tracing-log = "0.2"
|
||||
chrono = "0.4"
|
||||
parking_lot = "0.12"
|
||||
|
||||
# Core library — runs in-process (package name is "openhuman", lib name is "openhuman_core")
|
||||
openhuman = { path = "../../" }
|
||||
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>LSUIElement</key>
|
||||
<true/>
|
||||
</dict>
|
||||
</plist>
|
||||
@@ -0,0 +1,3 @@
|
||||
fn main() {
|
||||
tauri_build::build()
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"$schema": "../gen/schemas/desktop-schema.json",
|
||||
"identifier": "default",
|
||||
"description": "Capability for the overlay window",
|
||||
"windows": ["overlay"],
|
||||
"permissions": [
|
||||
"core:default",
|
||||
"core:window:default",
|
||||
"core:window:allow-start-dragging",
|
||||
"core:window:allow-set-size",
|
||||
"core:window:allow-set-position",
|
||||
"core:window:allow-set-always-on-top",
|
||||
"core:window:allow-set-ignore-cursor-events",
|
||||
"core:window:allow-hide",
|
||||
"core:window:allow-show",
|
||||
"core:window:allow-close",
|
||||
"core:window:allow-minimize",
|
||||
"core:event:default",
|
||||
"core:event:allow-listen",
|
||||
"core:event:allow-emit",
|
||||
"opener:default"
|
||||
]
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 3.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 974 B |
Binary file not shown.
|
After Width: | Height: | Size: 2.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.6 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 903 B |
Binary file not shown.
|
After Width: | Height: | Size: 8.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.3 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.5 KiB |
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
@@ -0,0 +1,144 @@
|
||||
mod log_bridge;
|
||||
|
||||
use log_bridge::{LogBuffer, LogEntry, TauriLogLayer};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
use tauri::Manager;
|
||||
#[cfg(target_os = "macos")]
|
||||
use tauri::ActivationPolicy;
|
||||
use tracing_subscriber::layer::SubscriberExt;
|
||||
use tracing_subscriber::util::SubscriberInitExt;
|
||||
use tracing_subscriber::EnvFilter;
|
||||
|
||||
/// Tauri state holding the log ring buffer and click-through toggle.
|
||||
struct OverlayState {
|
||||
log_buffer: Arc<LogBuffer>,
|
||||
click_through: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
// ── Tauri commands ──────────────────────────────────────────────────────────
|
||||
|
||||
/// Return all buffered log entries (for initial load / reconnect).
|
||||
#[tauri::command]
|
||||
fn get_log_history(state: tauri::State<'_, OverlayState>) -> Vec<LogEntry> {
|
||||
state.log_buffer.snapshot()
|
||||
}
|
||||
|
||||
/// Toggle click-through mode. When enabled, mouse events pass through
|
||||
/// the overlay to the window underneath.
|
||||
#[tauri::command]
|
||||
fn set_click_through(
|
||||
window: tauri::WebviewWindow,
|
||||
state: tauri::State<'_, OverlayState>,
|
||||
enabled: bool,
|
||||
) -> Result<(), String> {
|
||||
state.click_through.store(enabled, Ordering::Relaxed);
|
||||
window
|
||||
.set_ignore_cursor_events(enabled)
|
||||
.map_err(|e| e.to_string())?;
|
||||
log::debug!("[overlay] click-through set to {}", enabled);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Forward an RPC call to openhuman_core's dispatch in-process.
|
||||
/// Uses the same invoke_method path as the HTTP JSON-RPC server.
|
||||
#[tauri::command]
|
||||
async fn core_rpc(method: String, params: serde_json::Value) -> Result<serde_json::Value, String> {
|
||||
log::debug!("[overlay] core_rpc: method={}", method);
|
||||
let state = openhuman_core::core::jsonrpc::default_state();
|
||||
openhuman_core::core::jsonrpc::invoke_method(state, &method, params).await
|
||||
}
|
||||
|
||||
/// Insert text into the currently focused field in the previously active app.
|
||||
#[tauri::command]
|
||||
fn insert_text_into_focused_field(text: String) -> Result<(), String> {
|
||||
log::debug!(
|
||||
"[overlay] insert_text_into_focused_field len={}",
|
||||
text.chars().count()
|
||||
);
|
||||
openhuman_core::openhuman::accessibility::apply_text_to_focused_field(&text)
|
||||
}
|
||||
|
||||
// ── App entry ───────────────────────────────────────────────────────────────
|
||||
|
||||
#[cfg_attr(mobile, tauri::mobile_entry_point)]
|
||||
pub fn run() {
|
||||
// Shared state
|
||||
let log_buffer = Arc::new(LogBuffer::new(5000));
|
||||
let click_through = Arc::new(AtomicBool::new(false));
|
||||
|
||||
tauri::Builder::default()
|
||||
.plugin(tauri_plugin_opener::init())
|
||||
.manage(OverlayState {
|
||||
log_buffer: log_buffer.clone(),
|
||||
click_through: click_through.clone(),
|
||||
})
|
||||
.invoke_handler(tauri::generate_handler![
|
||||
get_log_history,
|
||||
set_click_through,
|
||||
core_rpc,
|
||||
insert_text_into_focused_field,
|
||||
])
|
||||
.setup(move |app| {
|
||||
let app_handle = app.handle().clone();
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
app.set_activation_policy(ActivationPolicy::Accessory);
|
||||
log::debug!("[overlay] macOS: activation policy set to accessory");
|
||||
}
|
||||
|
||||
// ── Tracing subscriber with Tauri bridge layer ──────────────
|
||||
let env_filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| {
|
||||
EnvFilter::new(
|
||||
"debug,hyper=info,reqwest=info,tungstenite=info,tokio_tungstenite=info",
|
||||
)
|
||||
});
|
||||
|
||||
let fmt_layer = tracing_subscriber::fmt::layer()
|
||||
.with_target(true)
|
||||
.with_ansi(true);
|
||||
|
||||
let tauri_layer = TauriLogLayer::new(app_handle.clone(), log_buffer.clone());
|
||||
|
||||
tracing_subscriber::registry()
|
||||
.with(env_filter)
|
||||
.with(fmt_layer)
|
||||
.with(tauri_layer)
|
||||
.init();
|
||||
|
||||
// Bridge `log` crate macros into tracing
|
||||
tracing_log::LogTracer::init().ok();
|
||||
|
||||
log::info!("[overlay] overlay process started, tracing bridge active");
|
||||
|
||||
// ── Start openhuman_core JSON-RPC server in-process ─────────
|
||||
// Use port 7799 to avoid conflicts with a standalone core on 7788.
|
||||
// Override with OPENHUMAN_CORE_PORT env var.
|
||||
tauri::async_runtime::spawn(async move {
|
||||
let port = std::env::var("OPENHUMAN_CORE_PORT")
|
||||
.ok()
|
||||
.and_then(|p| p.parse::<u16>().ok())
|
||||
.unwrap_or(7799);
|
||||
log::info!("[overlay] starting openhuman_core server on 127.0.0.1:{}...", port);
|
||||
match openhuman_core::core::jsonrpc::run_server(None, Some(port), true).await {
|
||||
Ok(()) => log::info!("[overlay] core server shut down cleanly"),
|
||||
Err(e) => log::error!("[overlay] core server error: {}", e),
|
||||
}
|
||||
});
|
||||
|
||||
// ── macOS: floating panel + visible on all workspaces ───────
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
if let Some(window) = app.get_webview_window("overlay") {
|
||||
window.set_always_on_top(true).ok();
|
||||
window.set_visible_on_all_workspaces(true).ok();
|
||||
log::debug!("[overlay] macOS: set always-on-top + visible-on-all-workspaces");
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
})
|
||||
.run(tauri::generate_context!())
|
||||
.expect("error while running overlay");
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
//! Captures `tracing` logs from openhuman_core and forwards them as Tauri events.
|
||||
//!
|
||||
//! Each log entry is emitted as a `core:log` event with a JSON payload:
|
||||
//! ```json
|
||||
//! { "ts": "2026-04-04T12:00:00Z", "level": "DEBUG", "module": "skills", "message": "..." }
|
||||
//! ```
|
||||
//! The frontend can filter by module to show logs from specific subsystems
|
||||
//! (skills, screen_recorder, autocomplete, rpc, etc.).
|
||||
|
||||
use chrono::Utc;
|
||||
use parking_lot::Mutex;
|
||||
use serde::Serialize;
|
||||
use std::sync::Arc;
|
||||
use tauri::{AppHandle, Emitter};
|
||||
use tracing::field::{Field, Visit};
|
||||
use tracing::span;
|
||||
use tracing_subscriber::layer::Context;
|
||||
use tracing_subscriber::Layer;
|
||||
|
||||
/// A single log entry forwarded to the overlay frontend.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct LogEntry {
|
||||
pub ts: String,
|
||||
pub level: String,
|
||||
pub module: String,
|
||||
pub target: String,
|
||||
pub message: String,
|
||||
}
|
||||
|
||||
/// Ring buffer that keeps the last N log entries so the frontend can fetch
|
||||
/// history on connect without missing early startup logs.
|
||||
pub struct LogBuffer {
|
||||
entries: Mutex<Vec<LogEntry>>,
|
||||
capacity: usize,
|
||||
}
|
||||
|
||||
impl LogBuffer {
|
||||
pub fn new(capacity: usize) -> Self {
|
||||
Self {
|
||||
entries: Mutex::new(Vec::with_capacity(capacity)),
|
||||
capacity,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn push(&self, entry: LogEntry) {
|
||||
let mut entries = self.entries.lock();
|
||||
if entries.len() >= self.capacity {
|
||||
entries.remove(0);
|
||||
}
|
||||
entries.push(entry);
|
||||
}
|
||||
|
||||
pub fn snapshot(&self) -> Vec<LogEntry> {
|
||||
self.entries.lock().clone()
|
||||
}
|
||||
}
|
||||
|
||||
/// tracing Layer that captures events and sends them to the Tauri frontend.
|
||||
pub struct TauriLogLayer {
|
||||
app: AppHandle,
|
||||
buffer: Arc<LogBuffer>,
|
||||
}
|
||||
|
||||
impl TauriLogLayer {
|
||||
pub fn new(app: AppHandle, buffer: Arc<LogBuffer>) -> Self {
|
||||
Self { app, buffer }
|
||||
}
|
||||
}
|
||||
|
||||
/// Visitor that extracts the `message` field from tracing events.
|
||||
struct MessageVisitor {
|
||||
message: String,
|
||||
}
|
||||
|
||||
impl MessageVisitor {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
message: String::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Visit for MessageVisitor {
|
||||
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
||||
if field.name() == "message" {
|
||||
self.message = format!("{:?}", value);
|
||||
} else if self.message.is_empty() {
|
||||
// Fall back to first field if no explicit "message"
|
||||
self.message = format!("{}: {:?}", field.name(), value);
|
||||
}
|
||||
}
|
||||
|
||||
fn record_str(&mut self, field: &Field, value: &str) {
|
||||
if field.name() == "message" {
|
||||
self.message = value.to_string();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Derive a human-friendly module name from the tracing target.
|
||||
/// e.g. "openhuman::skills::qjs_engine" -> "skills"
|
||||
/// "openhuman::rpc" -> "rpc"
|
||||
/// "core_server::dispatch" -> "core_server"
|
||||
fn module_from_target(target: &str) -> String {
|
||||
let parts: Vec<&str> = target.split("::").collect();
|
||||
// Try to find the second segment under "openhuman::"
|
||||
if parts.len() >= 2 && parts[0] == "openhuman" {
|
||||
return parts[1].to_string();
|
||||
}
|
||||
if parts.len() >= 2 && parts[0] == "openhuman_core" {
|
||||
return parts[1].to_string();
|
||||
}
|
||||
// For other crates, use the first segment
|
||||
parts.first().unwrap_or(&"unknown").to_string()
|
||||
}
|
||||
|
||||
impl<S> Layer<S> for TauriLogLayer
|
||||
where
|
||||
S: tracing::Subscriber + for<'lookup> tracing_subscriber::registry::LookupSpan<'lookup>,
|
||||
{
|
||||
fn on_event(&self, event: &tracing::Event<'_>, _ctx: Context<'_, S>) {
|
||||
let metadata = event.metadata();
|
||||
let level = metadata.level().to_string();
|
||||
let target = metadata.target().to_string();
|
||||
let module = module_from_target(&target);
|
||||
|
||||
let mut visitor = MessageVisitor::new();
|
||||
event.record(&mut visitor);
|
||||
|
||||
let entry = LogEntry {
|
||||
ts: Utc::now().to_rfc3339(),
|
||||
level,
|
||||
module,
|
||||
target,
|
||||
message: visitor.message,
|
||||
};
|
||||
|
||||
// Buffer for late-joining frontends
|
||||
self.buffer.push(entry.clone());
|
||||
|
||||
// Emit to all listening webviews — fire-and-forget
|
||||
let _ = self.app.emit("core:log", &entry);
|
||||
}
|
||||
|
||||
fn on_new_span(&self, _attrs: &span::Attributes<'_>, _id: &span::Id, _ctx: Context<'_, S>) {}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
// Prevents additional console window on Windows in release, DO NOT REMOVE!!
|
||||
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
|
||||
|
||||
fn main() {
|
||||
overlay_lib::run()
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"$schema": "https://schema.tauri.app/config/2",
|
||||
"productName": "openhuman-overlay",
|
||||
"version": "0.1.0",
|
||||
"identifier": "com.openhuman.overlay",
|
||||
"build": {
|
||||
"beforeDevCommand": "yarn dev",
|
||||
"devUrl": "http://localhost:1430",
|
||||
"beforeBuildCommand": "yarn build",
|
||||
"frontendDist": "../dist"
|
||||
},
|
||||
"app": {
|
||||
"withGlobalTauri": true,
|
||||
"windows": [
|
||||
{
|
||||
"label": "overlay",
|
||||
"title": "",
|
||||
"width": 376,
|
||||
"height": 432,
|
||||
"minWidth": 376,
|
||||
"minHeight": 432,
|
||||
"transparent": true,
|
||||
"decorations": false,
|
||||
"alwaysOnTop": true,
|
||||
"skipTaskbar": true,
|
||||
"resizable": false,
|
||||
"visible": false,
|
||||
"center": false,
|
||||
"x": 16,
|
||||
"y": 16
|
||||
}
|
||||
],
|
||||
"macOSPrivateApi": true,
|
||||
"security": {
|
||||
"csp": null
|
||||
}
|
||||
},
|
||||
"bundle": {
|
||||
"active": true,
|
||||
"targets": "all",
|
||||
"icon": [
|
||||
"icons/32x32.png",
|
||||
"icons/128x128.png",
|
||||
"icons/128x128@2x.png",
|
||||
"icons/icon.icns",
|
||||
"icons/icon.ico"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,712 @@
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
import { getCurrentWindow } from "@tauri-apps/api/window";
|
||||
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
|
||||
const TARGET_SAMPLE_RATE = 16000;
|
||||
|
||||
type OverlayStatus = "idle" | "listening" | "transcribing" | "ready" | "error";
|
||||
|
||||
interface TranscribeResult {
|
||||
text: string;
|
||||
raw_text: string;
|
||||
model_id: string;
|
||||
}
|
||||
|
||||
interface GlobeHotkeyStatus {
|
||||
supported: boolean;
|
||||
running: boolean;
|
||||
input_monitoring_permission: string;
|
||||
last_error: string | null;
|
||||
events_pending: number;
|
||||
}
|
||||
|
||||
interface GlobeHotkeyPollResult {
|
||||
status: GlobeHotkeyStatus;
|
||||
events: string[];
|
||||
}
|
||||
|
||||
interface AppContextInfo {
|
||||
app_name: string | null;
|
||||
window_title: string | null;
|
||||
}
|
||||
|
||||
interface AccessibilitySessionStatus {
|
||||
active: boolean;
|
||||
capture_count: number;
|
||||
frames_in_memory: number;
|
||||
last_capture_at_ms: number | null;
|
||||
last_context: string | null;
|
||||
last_window_title: string | null;
|
||||
vision_enabled: boolean;
|
||||
vision_state: string;
|
||||
vision_queue_depth: number;
|
||||
}
|
||||
|
||||
interface AccessibilityStatus {
|
||||
is_context_blocked: boolean;
|
||||
foreground_context: AppContextInfo | null;
|
||||
session: AccessibilitySessionStatus;
|
||||
}
|
||||
|
||||
interface AutocompleteSuggestion {
|
||||
value: string;
|
||||
confidence: number;
|
||||
}
|
||||
|
||||
interface AutocompleteStatus {
|
||||
platform_supported: boolean;
|
||||
enabled: boolean;
|
||||
running: boolean;
|
||||
phase: string;
|
||||
app_name: string | null;
|
||||
last_error: string | null;
|
||||
updated_at_ms: number | null;
|
||||
suggestion: AutocompleteSuggestion | null;
|
||||
}
|
||||
|
||||
interface OverlayDebugSnapshot {
|
||||
screen: AccessibilityStatus | null;
|
||||
autocomplete: AutocompleteStatus | null;
|
||||
updatedAt: number | null;
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
function logOverlay(message: string, details?: unknown) {
|
||||
if (details) {
|
||||
console.debug(`[overlay] ${message}`, details);
|
||||
return;
|
||||
}
|
||||
console.debug(`[overlay] ${message}`);
|
||||
}
|
||||
|
||||
function formatTimestamp(timestampMs: number | null): string {
|
||||
if (!timestampMs) {
|
||||
return "none";
|
||||
}
|
||||
|
||||
try {
|
||||
return new Date(timestampMs).toLocaleTimeString([], {
|
||||
hour: "2-digit",
|
||||
minute: "2-digit",
|
||||
second: "2-digit",
|
||||
});
|
||||
} catch {
|
||||
return String(timestampMs);
|
||||
}
|
||||
}
|
||||
|
||||
function floatTo16BitPCM(output: DataView, offset: number, input: Float32Array) {
|
||||
for (let i = 0; i < input.length; i += 1, offset += 2) {
|
||||
const sample = Math.max(-1, Math.min(1, input[i]));
|
||||
output.setInt16(offset, sample < 0 ? sample * 0x8000 : sample * 0x7fff, true);
|
||||
}
|
||||
}
|
||||
|
||||
function encodeWavMono16k(samples: Float32Array, sampleRate: number): Uint8Array {
|
||||
const bytesPerSample = 2;
|
||||
const blockAlign = bytesPerSample;
|
||||
const byteRate = sampleRate * blockAlign;
|
||||
const dataSize = samples.length * bytesPerSample;
|
||||
const buffer = new ArrayBuffer(44 + dataSize);
|
||||
const view = new DataView(buffer);
|
||||
|
||||
const writeString = (offset: number, value: string) => {
|
||||
for (let i = 0; i < value.length; i += 1) {
|
||||
view.setUint8(offset + i, value.charCodeAt(i));
|
||||
}
|
||||
};
|
||||
|
||||
writeString(0, "RIFF");
|
||||
view.setUint32(4, 36 + dataSize, true);
|
||||
writeString(8, "WAVE");
|
||||
writeString(12, "fmt ");
|
||||
view.setUint32(16, 16, true);
|
||||
view.setUint16(20, 1, true);
|
||||
view.setUint16(22, 1, true);
|
||||
view.setUint32(24, sampleRate, true);
|
||||
view.setUint32(28, byteRate, true);
|
||||
view.setUint16(32, blockAlign, true);
|
||||
view.setUint16(34, 16, true);
|
||||
writeString(36, "data");
|
||||
view.setUint32(40, dataSize, true);
|
||||
floatTo16BitPCM(view, 44, samples);
|
||||
|
||||
return new Uint8Array(buffer);
|
||||
}
|
||||
|
||||
async function toMono16k(audioBuffer: AudioBuffer): Promise<Float32Array> {
|
||||
const channels = audioBuffer.numberOfChannels;
|
||||
const mono = new Float32Array(audioBuffer.length);
|
||||
|
||||
for (let c = 0; c < channels; c += 1) {
|
||||
const channelData = audioBuffer.getChannelData(c);
|
||||
for (let i = 0; i < audioBuffer.length; i += 1) {
|
||||
mono[i] += channelData[i] / channels;
|
||||
}
|
||||
}
|
||||
|
||||
if (audioBuffer.sampleRate === TARGET_SAMPLE_RATE) {
|
||||
return mono;
|
||||
}
|
||||
|
||||
const targetLength = Math.max(
|
||||
1,
|
||||
Math.round((mono.length * TARGET_SAMPLE_RATE) / audioBuffer.sampleRate),
|
||||
);
|
||||
const offline = new OfflineAudioContext(1, targetLength, TARGET_SAMPLE_RATE);
|
||||
const sourceBuffer = offline.createBuffer(1, mono.length, audioBuffer.sampleRate);
|
||||
sourceBuffer.copyToChannel(mono, 0);
|
||||
const source = offline.createBufferSource();
|
||||
source.buffer = sourceBuffer;
|
||||
source.connect(offline.destination);
|
||||
source.start();
|
||||
const rendered = await offline.startRendering();
|
||||
return rendered.getChannelData(0).slice();
|
||||
}
|
||||
|
||||
async function convertBlobToWavBytes(blob: Blob): Promise<number[]> {
|
||||
const arrayBuffer = await blob.arrayBuffer();
|
||||
const audioContext = new AudioContext();
|
||||
|
||||
try {
|
||||
const decoded = await audioContext.decodeAudioData(arrayBuffer.slice(0));
|
||||
const mono16k = await toMono16k(decoded);
|
||||
return Array.from(encodeWavMono16k(mono16k, TARGET_SAMPLE_RATE));
|
||||
} finally {
|
||||
await audioContext.close();
|
||||
}
|
||||
}
|
||||
|
||||
function MicrophoneIcon({ active }: { active: boolean }) {
|
||||
return (
|
||||
<svg
|
||||
aria-hidden="true"
|
||||
className={`h-9 w-9 transition-transform duration-200 ${active ? "scale-105" : ""}`}
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
strokeWidth="1.8"
|
||||
strokeLinecap="round"
|
||||
strokeLinejoin="round"
|
||||
>
|
||||
<rect x="9" y="3" width="6" height="11" rx="3" />
|
||||
<path d="M6 11a6 6 0 0 0 12 0" />
|
||||
<path d="M12 17v4" />
|
||||
<path d="M8.5 21h7" />
|
||||
</svg>
|
||||
);
|
||||
}
|
||||
|
||||
export function App() {
|
||||
const appWindow = getCurrentWindow();
|
||||
const mediaRecorderRef = useRef<MediaRecorder | null>(null);
|
||||
const streamRef = useRef<MediaStream | null>(null);
|
||||
const chunksRef = useRef<Blob[]>([]);
|
||||
const sessionIdRef = useRef(0);
|
||||
const globePollInFlightRef = useRef(false);
|
||||
|
||||
const [status, setStatus] = useState<OverlayStatus>("idle");
|
||||
const [message, setMessage] = useState("Click to start listening");
|
||||
const [transcript, setTranscript] = useState("");
|
||||
const [debugSnapshot, setDebugSnapshot] = useState<OverlayDebugSnapshot>({
|
||||
screen: null,
|
||||
autocomplete: null,
|
||||
updatedAt: null,
|
||||
error: null,
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
let disposed = false;
|
||||
|
||||
const showOverlayFallback = async (message: string) => {
|
||||
if (disposed) {
|
||||
return;
|
||||
}
|
||||
logOverlay("globe listener unavailable", { message });
|
||||
setMessage(message);
|
||||
await appWindow.show().catch(() => {});
|
||||
};
|
||||
|
||||
const startGlobeListener = async () => {
|
||||
try {
|
||||
const result = await invoke<GlobeHotkeyStatus>("core_rpc", {
|
||||
method: "openhuman.screen_intelligence_globe_listener_start",
|
||||
params: {},
|
||||
});
|
||||
logOverlay("globe listener start result", result);
|
||||
|
||||
if (!result.supported) {
|
||||
await showOverlayFallback("Globe/Fn hotkey is only supported on macOS");
|
||||
return;
|
||||
}
|
||||
|
||||
if (!result.running) {
|
||||
await showOverlayFallback(
|
||||
result.last_error ?? "Globe/Fn listener could not start. Check Input Monitoring.",
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("[overlay] failed to start globe listener", error);
|
||||
await showOverlayFallback("Failed to start Globe/Fn listener");
|
||||
}
|
||||
};
|
||||
|
||||
const pollGlobeListener = async () => {
|
||||
if (disposed || globePollInFlightRef.current) {
|
||||
return;
|
||||
}
|
||||
globePollInFlightRef.current = true;
|
||||
|
||||
try {
|
||||
const result = await invoke<GlobeHotkeyPollResult>("core_rpc", {
|
||||
method: "openhuman.screen_intelligence_globe_listener_poll",
|
||||
params: {},
|
||||
});
|
||||
|
||||
if (disposed) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (!result.status.running && result.status.last_error) {
|
||||
setMessage(result.status.last_error);
|
||||
}
|
||||
|
||||
if (result.events.includes("FN_UP")) {
|
||||
const visible = await appWindow.isVisible();
|
||||
logOverlay("received FN_UP", { visible });
|
||||
if (visible) {
|
||||
await appWindow.hide();
|
||||
} else {
|
||||
await appWindow.show();
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
if (!disposed) {
|
||||
console.warn("[overlay] globe listener poll failed", error);
|
||||
}
|
||||
} finally {
|
||||
globePollInFlightRef.current = false;
|
||||
}
|
||||
};
|
||||
|
||||
void startGlobeListener();
|
||||
const intervalId = window.setInterval(() => {
|
||||
void pollGlobeListener();
|
||||
}, 175);
|
||||
|
||||
return () => {
|
||||
disposed = true;
|
||||
window.clearInterval(intervalId);
|
||||
void invoke("core_rpc", {
|
||||
method: "openhuman.screen_intelligence_globe_listener_stop",
|
||||
params: {},
|
||||
}).catch(() => {});
|
||||
};
|
||||
}, [appWindow]);
|
||||
|
||||
useEffect(() => {
|
||||
let disposed = false;
|
||||
let pollInFlight = false;
|
||||
|
||||
const pollDebugState = async () => {
|
||||
if (disposed || pollInFlight) {
|
||||
return;
|
||||
}
|
||||
pollInFlight = true;
|
||||
|
||||
try {
|
||||
const [screen, autocomplete] = await Promise.all([
|
||||
invoke<AccessibilityStatus>("core_rpc", {
|
||||
method: "openhuman.accessibility_status",
|
||||
params: {},
|
||||
}),
|
||||
invoke<AutocompleteStatus>("core_rpc", {
|
||||
method: "openhuman.autocomplete_status",
|
||||
params: {},
|
||||
}),
|
||||
]);
|
||||
|
||||
if (disposed) {
|
||||
return;
|
||||
}
|
||||
|
||||
logOverlay("debug snapshot refreshed", {
|
||||
screenActive: screen.session.active,
|
||||
captureCount: screen.session.capture_count,
|
||||
autocompletePhase: autocomplete.phase,
|
||||
hasSuggestion: Boolean(autocomplete.suggestion?.value),
|
||||
});
|
||||
|
||||
setDebugSnapshot({
|
||||
screen,
|
||||
autocomplete,
|
||||
updatedAt: Date.now(),
|
||||
error: null,
|
||||
});
|
||||
} catch (error) {
|
||||
if (disposed) {
|
||||
return;
|
||||
}
|
||||
|
||||
const nextError =
|
||||
error instanceof Error ? error.message : "Failed to refresh overlay debug state";
|
||||
console.warn("[overlay] debug snapshot poll failed", error);
|
||||
setDebugSnapshot((previous) => ({
|
||||
...previous,
|
||||
updatedAt: Date.now(),
|
||||
error: nextError,
|
||||
}));
|
||||
} finally {
|
||||
pollInFlight = false;
|
||||
}
|
||||
};
|
||||
|
||||
void pollDebugState();
|
||||
const intervalId = window.setInterval(() => {
|
||||
void pollDebugState();
|
||||
}, 900);
|
||||
|
||||
return () => {
|
||||
disposed = true;
|
||||
window.clearInterval(intervalId);
|
||||
};
|
||||
}, []);
|
||||
|
||||
const insertTranscriptIntoFocusedField = useCallback(
|
||||
async (text: string) => {
|
||||
logOverlay("inserting transcript into focused field", { length: text.length });
|
||||
await appWindow.hide();
|
||||
await new Promise((resolve) => window.setTimeout(resolve, 120));
|
||||
|
||||
try {
|
||||
await invoke("insert_text_into_focused_field", { text });
|
||||
logOverlay("transcript inserted via accessibility helper");
|
||||
} catch (error) {
|
||||
console.warn("[overlay] accessibility insert failed, falling back to clipboard", error);
|
||||
await navigator.clipboard.writeText(text);
|
||||
}
|
||||
},
|
||||
[appWindow],
|
||||
);
|
||||
|
||||
const resetForNextCapture = useCallback(() => {
|
||||
setTranscript("");
|
||||
setMessage("Click to start listening");
|
||||
setStatus("idle");
|
||||
}, []);
|
||||
|
||||
const cleanupStream = useCallback(() => {
|
||||
streamRef.current?.getTracks().forEach((track) => track.stop());
|
||||
streamRef.current = null;
|
||||
}, []);
|
||||
|
||||
const transcribeBlob = useCallback(
|
||||
async (blob: Blob, sessionId: number) => {
|
||||
try {
|
||||
const audioBytes = await convertBlobToWavBytes(blob);
|
||||
const result = await invoke<TranscribeResult>("core_rpc", {
|
||||
method: "openhuman.voice_transcribe_bytes",
|
||||
params: {
|
||||
audio_bytes: audioBytes,
|
||||
extension: "wav",
|
||||
skip_cleanup: false,
|
||||
},
|
||||
});
|
||||
|
||||
if (sessionIdRef.current !== sessionId) {
|
||||
return;
|
||||
}
|
||||
|
||||
const nextTranscript = result.text.trim();
|
||||
if (!nextTranscript) {
|
||||
setTranscript("");
|
||||
setStatus("error");
|
||||
setMessage("No speech detected");
|
||||
return;
|
||||
}
|
||||
|
||||
setTranscript(nextTranscript);
|
||||
setStatus("ready");
|
||||
setMessage("Inserting text...");
|
||||
await insertTranscriptIntoFocusedField(nextTranscript);
|
||||
if (sessionIdRef.current !== sessionId) {
|
||||
return;
|
||||
}
|
||||
setMessage("Inserted into active field");
|
||||
} catch (error) {
|
||||
if (sessionIdRef.current !== sessionId) {
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("[overlay] transcription failed", error);
|
||||
setTranscript("");
|
||||
setStatus("error");
|
||||
setMessage(error instanceof Error ? error.message : "Transcription failed");
|
||||
}
|
||||
},
|
||||
[insertTranscriptIntoFocusedField],
|
||||
);
|
||||
|
||||
const stopRecording = useCallback(() => {
|
||||
if (!mediaRecorderRef.current || mediaRecorderRef.current.state === "inactive") {
|
||||
return;
|
||||
}
|
||||
|
||||
setStatus("transcribing");
|
||||
setMessage("Transcribing...");
|
||||
mediaRecorderRef.current.stop();
|
||||
mediaRecorderRef.current = null;
|
||||
}, []);
|
||||
|
||||
const startRecording = useCallback(async () => {
|
||||
const nextSessionId = sessionIdRef.current + 1;
|
||||
sessionIdRef.current = nextSessionId;
|
||||
setTranscript("");
|
||||
setStatus("listening");
|
||||
setMessage("Listening...");
|
||||
chunksRef.current = [];
|
||||
|
||||
try {
|
||||
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||
streamRef.current = stream;
|
||||
|
||||
const mimeType = MediaRecorder.isTypeSupported("audio/webm;codecs=opus")
|
||||
? "audio/webm;codecs=opus"
|
||||
: MediaRecorder.isTypeSupported("audio/webm")
|
||||
? "audio/webm"
|
||||
: "audio/ogg";
|
||||
|
||||
const recorder = new MediaRecorder(stream, { mimeType });
|
||||
mediaRecorderRef.current = recorder;
|
||||
|
||||
recorder.ondataavailable = (event) => {
|
||||
if (event.data.size > 0) {
|
||||
chunksRef.current.push(event.data);
|
||||
}
|
||||
};
|
||||
|
||||
recorder.onerror = (event) => {
|
||||
console.error("[overlay] media recorder error", event);
|
||||
cleanupStream();
|
||||
setStatus("error");
|
||||
setMessage("Microphone recording failed");
|
||||
};
|
||||
|
||||
recorder.onstop = () => {
|
||||
cleanupStream();
|
||||
const blob = new Blob(chunksRef.current, { type: mimeType });
|
||||
chunksRef.current = [];
|
||||
|
||||
if (blob.size === 0) {
|
||||
setStatus("error");
|
||||
setMessage("No audio recorded");
|
||||
return;
|
||||
}
|
||||
|
||||
logOverlay("recording stopped, starting transcription", {
|
||||
blobSize: blob.size,
|
||||
mimeType,
|
||||
});
|
||||
void transcribeBlob(blob, nextSessionId);
|
||||
};
|
||||
|
||||
logOverlay("recording started", { mimeType });
|
||||
recorder.start(100);
|
||||
} catch (error) {
|
||||
console.error("[overlay] getUserMedia failed", error);
|
||||
cleanupStream();
|
||||
setStatus("error");
|
||||
setMessage(error instanceof Error ? error.message : "Microphone access failed");
|
||||
}
|
||||
}, [cleanupStream, transcribeBlob]);
|
||||
|
||||
const handleMainButton = useCallback(() => {
|
||||
if (status === "listening") {
|
||||
logOverlay("main button toggled to stop listening");
|
||||
stopRecording();
|
||||
return;
|
||||
}
|
||||
|
||||
logOverlay("main button toggled to start listening", { priorStatus: status });
|
||||
void startRecording();
|
||||
}, [startRecording, status, stopRecording]);
|
||||
|
||||
const shellClassName = useMemo(() => {
|
||||
if (status === "listening") {
|
||||
return "from-red-500/90 via-rose-500/80 to-orange-400/85 text-white shadow-[0_0_64px_rgba(248,113,113,0.38)]";
|
||||
}
|
||||
if (status === "transcribing") {
|
||||
return "from-amber-400/90 via-orange-400/80 to-yellow-300/80 text-stone-950 shadow-[0_0_56px_rgba(251,191,36,0.34)]";
|
||||
}
|
||||
if (status === "error") {
|
||||
return "from-red-600/90 via-rose-700/80 to-stone-900/90 text-white shadow-[0_0_56px_rgba(190,24,93,0.35)]";
|
||||
}
|
||||
if (status === "ready") {
|
||||
return "from-emerald-400/90 via-teal-400/80 to-cyan-300/80 text-stone-950 shadow-[0_0_56px_rgba(45,212,191,0.34)]";
|
||||
}
|
||||
return "from-slate-900/92 via-slate-800/92 to-slate-700/92 text-white shadow-[0_0_48px_rgba(15,23,42,0.42)]";
|
||||
}, [status]);
|
||||
|
||||
const activeScreenApp =
|
||||
debugSnapshot.screen?.foreground_context?.app_name ??
|
||||
debugSnapshot.screen?.session.last_context ??
|
||||
"Unknown app";
|
||||
const activeScreenWindow =
|
||||
debugSnapshot.screen?.foreground_context?.window_title ??
|
||||
debugSnapshot.screen?.session.last_window_title ??
|
||||
"No active window title";
|
||||
const autocompleteSuggestion = debugSnapshot.autocomplete?.suggestion?.value?.trim() ?? "";
|
||||
const autocompletePhase = debugSnapshot.autocomplete?.phase ?? "unknown";
|
||||
const autocompleteRunning =
|
||||
debugSnapshot.autocomplete?.running && debugSnapshot.autocomplete?.enabled;
|
||||
|
||||
return (
|
||||
<div className="flex h-screen w-screen items-start justify-start bg-transparent p-3">
|
||||
<div className="relative select-none">
|
||||
{status === "listening" ? (
|
||||
<>
|
||||
<span className="pointer-events-none absolute inset-0 rounded-full border border-white/15 animate-ping" />
|
||||
<span className="pointer-events-none absolute -inset-3 rounded-full border border-red-300/30 blur-[2px]" />
|
||||
</>
|
||||
) : null}
|
||||
|
||||
<div
|
||||
className={`relative w-[348px] rounded-[32px] border border-white/15 bg-gradient-to-br p-3 backdrop-blur-xl transition-all duration-200 ${shellClassName}`}
|
||||
onMouseDown={(event) => {
|
||||
if (event.target instanceof HTMLElement && event.target.closest("button")) {
|
||||
return;
|
||||
}
|
||||
void appWindow.startDragging();
|
||||
}}
|
||||
>
|
||||
<div className="mb-3 flex items-center justify-between gap-2">
|
||||
<span className="rounded-full bg-black/15 px-2 py-1 text-[10px] font-semibold uppercase tracking-[0.24em]">
|
||||
Voice
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
className="h-7 w-7 rounded-full border border-white/15 bg-black/20 text-sm transition hover:bg-black/30"
|
||||
onClick={() => appWindow.hide()}
|
||||
aria-label="Hide overlay"
|
||||
>
|
||||
×
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div className="flex items-start gap-3">
|
||||
<button
|
||||
type="button"
|
||||
onClick={handleMainButton}
|
||||
className={`group relative flex h-[108px] w-[108px] shrink-0 items-center justify-center rounded-full border border-white/20 bg-black/20 transition duration-200 hover:bg-black/28 ${
|
||||
status === "listening" ? "scale-[1.02]" : ""
|
||||
}`}
|
||||
aria-label={status === "listening" ? "Stop listening" : "Start listening"}
|
||||
>
|
||||
<span className="absolute inset-3 rounded-full border border-white/12" />
|
||||
<MicrophoneIcon active={status === "listening"} />
|
||||
</button>
|
||||
|
||||
<div className="min-w-0 flex-1 pt-1">
|
||||
<p className="text-[11px] font-semibold uppercase tracking-[0.18em] opacity-80">
|
||||
{status}
|
||||
</p>
|
||||
<p className="mt-1 text-xs leading-4 opacity-90">{message}</p>
|
||||
|
||||
{transcript ? (
|
||||
<div className="mt-3 rounded-2xl border border-white/10 bg-black/15 px-3 py-2 text-[11px] leading-4 opacity-95">
|
||||
{transcript}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{(status === "ready" || status === "error") && !transcript ? (
|
||||
<button
|
||||
type="button"
|
||||
className="mt-3 w-full rounded-full border border-white/12 bg-black/15 px-3 py-2 text-[11px] font-medium transition hover:bg-black/25"
|
||||
onClick={resetForNextCapture}
|
||||
>
|
||||
Reset
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="mt-3 rounded-[24px] border border-white/10 bg-black/15 p-3">
|
||||
<div className="flex items-center justify-between gap-2">
|
||||
<span className="text-[10px] font-semibold uppercase tracking-[0.22em] opacity-75">
|
||||
Debug
|
||||
</span>
|
||||
<span className="text-[10px] opacity-65">
|
||||
{debugSnapshot.updatedAt ? formatTimestamp(debugSnapshot.updatedAt) : "waiting"}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{debugSnapshot.error ? (
|
||||
<div className="mt-3 rounded-2xl border border-red-300/20 bg-red-950/20 px-3 py-2 text-[11px] leading-4 text-red-100">
|
||||
{debugSnapshot.error}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="mt-3 grid gap-3">
|
||||
<section className="rounded-2xl border border-white/8 bg-black/10 px-3 py-2">
|
||||
<div className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-70">
|
||||
Screen Intelligence
|
||||
</div>
|
||||
<div className="mt-2 space-y-1 text-[11px] leading-4 opacity-90">
|
||||
<p>Active screen: {activeScreenApp}</p>
|
||||
<p className="truncate">Window: {activeScreenWindow}</p>
|
||||
<p>
|
||||
Screenshots: {debugSnapshot.screen?.session.capture_count ?? 0} total,{" "}
|
||||
{debugSnapshot.screen?.session.frames_in_memory ?? 0} in memory
|
||||
</p>
|
||||
<p>
|
||||
Session: {debugSnapshot.screen?.session.active ? "active" : "idle"} | Vision:{" "}
|
||||
{debugSnapshot.screen?.session.vision_enabled ? "on" : "off"} /{" "}
|
||||
{debugSnapshot.screen?.session.vision_state ?? "idle"}
|
||||
</p>
|
||||
<p>
|
||||
Queue: {debugSnapshot.screen?.session.vision_queue_depth ?? 0} | Blocked:{" "}
|
||||
{debugSnapshot.screen?.is_context_blocked ? "yes" : "no"}
|
||||
</p>
|
||||
<p>
|
||||
Last capture:{" "}
|
||||
{formatTimestamp(debugSnapshot.screen?.session.last_capture_at_ms ?? null)}
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section className="rounded-2xl border border-white/8 bg-black/10 px-3 py-2">
|
||||
<div className="text-[10px] font-semibold uppercase tracking-[0.18em] opacity-70">
|
||||
Autocomplete
|
||||
</div>
|
||||
<div className="mt-2 space-y-1 text-[11px] leading-4 opacity-90">
|
||||
<p>
|
||||
Status: {autocompleteRunning ? "active" : "idle"} | Phase: {autocompletePhase}
|
||||
</p>
|
||||
<p>App: {debugSnapshot.autocomplete?.app_name ?? activeScreenApp}</p>
|
||||
<p>
|
||||
Processing:{" "}
|
||||
{autocompletePhase === "refreshing" || autocompletePhase === "processing"
|
||||
? "yes"
|
||||
: "no"}
|
||||
</p>
|
||||
<p>
|
||||
Suggestions:{" "}
|
||||
{autocompleteSuggestion ? "1 ready" : "none"}
|
||||
</p>
|
||||
<div className="rounded-xl border border-white/8 bg-black/10 px-2 py-2 text-[11px] leading-4">
|
||||
{autocompleteSuggestion || "No autocomplete suggestion available."}
|
||||
</div>
|
||||
{debugSnapshot.autocomplete?.last_error ? (
|
||||
<p className="text-red-100">
|
||||
Error: {debugSnapshot.autocomplete.last_error}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
import { useEffect, useMemo, useRef } from "react";
|
||||
import type { LogEntry } from "../types";
|
||||
import { LEVEL_COLORS } from "../types";
|
||||
|
||||
interface LogViewerProps {
|
||||
entries: LogEntry[];
|
||||
activeModule: string;
|
||||
levelFilter: string;
|
||||
}
|
||||
|
||||
/** Format ISO timestamp to HH:MM:SS.mmm */
|
||||
function formatTime(ts: string): string {
|
||||
try {
|
||||
const d = new Date(ts);
|
||||
const h = String(d.getHours()).padStart(2, "0");
|
||||
const m = String(d.getMinutes()).padStart(2, "0");
|
||||
const s = String(d.getSeconds()).padStart(2, "0");
|
||||
const ms = String(d.getMilliseconds()).padStart(3, "0");
|
||||
return `${h}:${m}:${s}.${ms}`;
|
||||
} catch {
|
||||
return ts.slice(11, 23);
|
||||
}
|
||||
}
|
||||
|
||||
const LEVEL_ORDER: Record<string, number> = {
|
||||
TRACE: 0,
|
||||
DEBUG: 1,
|
||||
INFO: 2,
|
||||
WARN: 3,
|
||||
ERROR: 4,
|
||||
FATAL: 5,
|
||||
};
|
||||
|
||||
/**
|
||||
* Virtualized-ish log viewer. Auto-scrolls to bottom as new entries arrive.
|
||||
* Filters by module and minimum log level.
|
||||
*/
|
||||
export function LogViewer({ entries, activeModule, levelFilter }: LogViewerProps) {
|
||||
const bottomRef = useRef<HTMLDivElement>(null);
|
||||
const containerRef = useRef<HTMLDivElement>(null);
|
||||
const isAtBottom = useRef(true);
|
||||
|
||||
const filtered = useMemo(() => {
|
||||
const minLevel = LEVEL_ORDER[levelFilter] ?? 0;
|
||||
return entries.filter((e) => {
|
||||
if (activeModule !== "all" && e.module !== activeModule) return false;
|
||||
const entryLevel = LEVEL_ORDER[e.level] ?? 0;
|
||||
return entryLevel >= minLevel;
|
||||
});
|
||||
}, [entries, activeModule, levelFilter]);
|
||||
|
||||
// Track scroll position to decide auto-scroll
|
||||
const handleScroll = () => {
|
||||
const el = containerRef.current;
|
||||
if (!el) return;
|
||||
const threshold = 40;
|
||||
isAtBottom.current = el.scrollHeight - el.scrollTop - el.clientHeight < threshold;
|
||||
};
|
||||
|
||||
// Auto-scroll when new entries arrive (only if already at bottom)
|
||||
useEffect(() => {
|
||||
if (isAtBottom.current) {
|
||||
bottomRef.current?.scrollIntoView({ behavior: "instant" });
|
||||
}
|
||||
}, [filtered.length]);
|
||||
|
||||
return (
|
||||
<div
|
||||
ref={containerRef}
|
||||
onScroll={handleScroll}
|
||||
className="flex-1 overflow-y-auto log-scroll font-mono text-[11px] leading-[18px] px-2 py-1 bg-gray-950/90"
|
||||
>
|
||||
{filtered.length === 0 && (
|
||||
<div className="text-white/20 text-center py-8 text-xs">
|
||||
No logs yet...
|
||||
</div>
|
||||
)}
|
||||
|
||||
{filtered.map((entry, i) => (
|
||||
<div key={i} className="flex gap-2 hover:bg-white/[0.03] px-1 rounded">
|
||||
<span className="text-white/25 shrink-0">{formatTime(entry.ts)}</span>
|
||||
<span className={`shrink-0 w-[42px] text-right ${LEVEL_COLORS[entry.level] ?? "text-white/40"}`}>
|
||||
{entry.level}
|
||||
</span>
|
||||
<span className="text-purple-400/60 shrink-0 w-[80px] truncate" title={entry.target}>
|
||||
{entry.module}
|
||||
</span>
|
||||
<span className="text-white/80 break-all">{entry.message}</span>
|
||||
</div>
|
||||
))}
|
||||
|
||||
<div ref={bottomRef} />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
import { MODULE_LABELS } from "../types";
|
||||
|
||||
interface ModuleFilterProps {
|
||||
modules: Set<string>;
|
||||
activeModule: string;
|
||||
onSelect: (module: string) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Horizontal tab bar for filtering logs by module.
|
||||
* Shows "All" plus every module that has emitted at least one log.
|
||||
*/
|
||||
export function ModuleFilter({ modules, activeModule, onSelect }: ModuleFilterProps) {
|
||||
const tabs = ["all", ...Array.from(modules).sort()];
|
||||
|
||||
return (
|
||||
<div className="flex items-center gap-1 px-2 py-1.5 bg-gray-900/60 border-b border-white/5 overflow-x-auto shrink-0">
|
||||
{tabs.map((mod) => {
|
||||
const isActive = mod === activeModule;
|
||||
const label = MODULE_LABELS[mod] ?? mod;
|
||||
return (
|
||||
<button
|
||||
key={mod}
|
||||
onClick={() => onSelect(mod)}
|
||||
className={`px-2 py-0.5 rounded text-[10px] font-mono whitespace-nowrap transition-colors ${
|
||||
isActive
|
||||
? "bg-primary-500/30 text-primary-500 border border-primary-500/40"
|
||||
: "text-white/40 hover:text-white/60 hover:bg-white/5"
|
||||
}`}
|
||||
>
|
||||
{label}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
interface StatusBarProps {
|
||||
filteredInfo: string;
|
||||
levelFilter: string;
|
||||
onLevelChange: (level: string) => void;
|
||||
onClear: () => void;
|
||||
}
|
||||
|
||||
const LEVELS = ["TRACE", "DEBUG", "INFO", "WARN", "ERROR"];
|
||||
|
||||
/**
|
||||
* Bottom status bar showing entry count, level filter, and clear button.
|
||||
*/
|
||||
export function StatusBar({
|
||||
filteredInfo,
|
||||
levelFilter,
|
||||
onLevelChange,
|
||||
onClear,
|
||||
}: StatusBarProps) {
|
||||
return (
|
||||
<div className="flex items-center justify-between px-3 py-1 bg-gray-900/80 border-t border-white/5 shrink-0">
|
||||
<div className="flex items-center gap-3">
|
||||
<span className="text-[10px] text-white/30 font-mono">
|
||||
{filteredInfo}
|
||||
</span>
|
||||
|
||||
<select
|
||||
value={levelFilter}
|
||||
onChange={(e) => onLevelChange(e.target.value)}
|
||||
className="text-[10px] bg-transparent text-white/50 border border-white/10 rounded px-1 py-0.5 cursor-pointer hover:border-white/20"
|
||||
>
|
||||
{LEVELS.map((l) => (
|
||||
<option key={l} value={l} className="bg-gray-900 text-white">
|
||||
{l}+
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<button
|
||||
onClick={onClear}
|
||||
className="text-[10px] text-white/30 hover:text-white/60 transition-colors"
|
||||
>
|
||||
Clear
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
import { getCurrentWindow } from "@tauri-apps/api/window";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
|
||||
interface TitleBarProps {
|
||||
clickThrough: boolean;
|
||||
onClickThroughToggle: () => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Custom title bar for the frameless overlay window.
|
||||
* Supports dragging, window controls, and click-through toggle.
|
||||
*/
|
||||
export function TitleBar({ clickThrough, onClickThroughToggle }: TitleBarProps) {
|
||||
const appWindow = getCurrentWindow();
|
||||
|
||||
const handleClickThrough = async () => {
|
||||
const next = !clickThrough;
|
||||
try {
|
||||
await invoke("set_click_through", { enabled: next });
|
||||
onClickThroughToggle();
|
||||
} catch (e) {
|
||||
console.error("Failed to set click-through:", e);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div
|
||||
data-tauri-drag-region
|
||||
className="flex items-center justify-between h-8 px-3 bg-gray-900/80 backdrop-blur-md border-b border-white/5 cursor-move select-none shrink-0"
|
||||
>
|
||||
<span className="text-[11px] font-medium text-white/60 tracking-wide uppercase">
|
||||
OpenHuman
|
||||
</span>
|
||||
|
||||
<div className="flex items-center gap-2">
|
||||
{/* Click-through toggle */}
|
||||
<button
|
||||
onClick={handleClickThrough}
|
||||
className={`text-[10px] px-1.5 py-0.5 rounded transition-colors ${
|
||||
clickThrough
|
||||
? "bg-blue-500/30 text-blue-400 border border-blue-500/40"
|
||||
: "text-white/30 hover:text-white/50"
|
||||
}`}
|
||||
title={clickThrough ? "Click-through ON (clicks pass through)" : "Click-through OFF"}
|
||||
>
|
||||
{clickThrough ? "CT" : "CT"}
|
||||
</button>
|
||||
|
||||
{/* Minimize */}
|
||||
<button
|
||||
onClick={() => appWindow.minimize()}
|
||||
className="w-3 h-3 rounded-full bg-amber-500/80 hover:bg-amber-400 transition-colors"
|
||||
title="Minimize"
|
||||
/>
|
||||
{/* Close */}
|
||||
<button
|
||||
onClick={() => appWindow.hide()}
|
||||
className="w-3 h-3 rounded-full bg-red-500/80 hover:bg-red-400 transition-colors"
|
||||
title="Hide"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
import { useCallback, useEffect, useRef, useState } from "react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
import { listen } from "@tauri-apps/api/event";
|
||||
import type { LogEntry } from "../types";
|
||||
|
||||
const MAX_ENTRIES = 5000;
|
||||
|
||||
/**
|
||||
* Subscribes to `core:log` Tauri events and manages the log buffer.
|
||||
* On mount, fetches buffered history so we don't miss startup logs.
|
||||
*/
|
||||
export function useLogs() {
|
||||
const [entries, setEntries] = useState<LogEntry[]>([]);
|
||||
const [modules, setModules] = useState<Set<string>>(new Set());
|
||||
const entriesRef = useRef<LogEntry[]>([]);
|
||||
|
||||
// Track seen modules for the filter UI
|
||||
const modulesRef = useRef<Set<string>>(new Set());
|
||||
|
||||
const addEntries = useCallback((newEntries: LogEntry[]) => {
|
||||
const current = entriesRef.current;
|
||||
const updated = [...current, ...newEntries];
|
||||
// Trim to max
|
||||
if (updated.length > MAX_ENTRIES) {
|
||||
updated.splice(0, updated.length - MAX_ENTRIES);
|
||||
}
|
||||
entriesRef.current = updated;
|
||||
setEntries(updated);
|
||||
|
||||
// Track modules
|
||||
let modulesChanged = false;
|
||||
for (const e of newEntries) {
|
||||
if (!modulesRef.current.has(e.module)) {
|
||||
modulesRef.current.add(e.module);
|
||||
modulesChanged = true;
|
||||
}
|
||||
}
|
||||
if (modulesChanged) {
|
||||
setModules(new Set(modulesRef.current));
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
// Fetch buffered history from Rust
|
||||
invoke<LogEntry[]>("get_log_history")
|
||||
.then((history) => {
|
||||
if (history.length > 0) {
|
||||
addEntries(history);
|
||||
}
|
||||
})
|
||||
.catch(console.error);
|
||||
|
||||
// Subscribe to live log events
|
||||
const unlisten = listen<LogEntry>("core:log", (event) => {
|
||||
addEntries([event.payload]);
|
||||
});
|
||||
|
||||
return () => {
|
||||
unlisten.then((fn) => fn());
|
||||
};
|
||||
}, [addEntries]);
|
||||
|
||||
const clear = useCallback(() => {
|
||||
entriesRef.current = [];
|
||||
setEntries([]);
|
||||
}, []);
|
||||
|
||||
return { entries, modules, clear };
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import React from "react";
|
||||
import ReactDOM from "react-dom/client";
|
||||
import { App } from "./App";
|
||||
import "./styles.css";
|
||||
|
||||
ReactDOM.createRoot(document.getElementById("root")!).render(
|
||||
<React.StrictMode>
|
||||
<App />
|
||||
</React.StrictMode>
|
||||
);
|
||||
@@ -0,0 +1,33 @@
|
||||
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600&family=JetBrains+Mono:wght@300;400;500&display=swap');
|
||||
|
||||
@tailwind base;
|
||||
@tailwind components;
|
||||
@tailwind utilities;
|
||||
|
||||
html, body, #root {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
height: 100%;
|
||||
background: transparent;
|
||||
overflow: hidden;
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: "Inter", system-ui, sans-serif;
|
||||
}
|
||||
|
||||
/* Scrollbar styling for log viewer */
|
||||
.log-scroll::-webkit-scrollbar {
|
||||
width: 6px;
|
||||
}
|
||||
.log-scroll::-webkit-scrollbar-track {
|
||||
background: transparent;
|
||||
}
|
||||
.log-scroll::-webkit-scrollbar-thumb {
|
||||
background: rgba(255, 255, 255, 0.15);
|
||||
border-radius: 3px;
|
||||
}
|
||||
.log-scroll::-webkit-scrollbar-thumb:hover {
|
||||
background: rgba(255, 255, 255, 0.25);
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
/** A single log entry from the Rust core, received via `core:log` event. */
|
||||
export interface LogEntry {
|
||||
ts: string;
|
||||
level: string;
|
||||
module: string;
|
||||
target: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
/** Known module names for filtering.
|
||||
* As new apps/domains emit logs, they auto-appear in the filter bar.
|
||||
* This map provides friendly labels for known modules. */
|
||||
export const MODULE_LABELS: Record<string, string> = {
|
||||
all: "All",
|
||||
// ── Core domains ──
|
||||
skills: "Skills",
|
||||
rpc: "RPC",
|
||||
core: "Core",
|
||||
core_server: "Server",
|
||||
config: "Config",
|
||||
cron: "Cron",
|
||||
memory: "Memory",
|
||||
channels: "Channels",
|
||||
overlay: "Overlay",
|
||||
about_app: "About",
|
||||
subconscious: "Subconscious",
|
||||
// ── Apps / subsystems ──
|
||||
screen_recorder: "Screen Rec",
|
||||
autocomplete: "Autocomplete",
|
||||
agent: "Agent",
|
||||
search: "Search",
|
||||
// ── Infra ──
|
||||
axum: "HTTP",
|
||||
tower_http: "HTTP",
|
||||
socketioxide: "Socket.IO",
|
||||
hyper: "Hyper",
|
||||
reqwest: "Reqwest",
|
||||
rusqlite: "SQLite",
|
||||
};
|
||||
|
||||
/** Level colors for the log viewer. */
|
||||
export const LEVEL_COLORS: Record<string, string> = {
|
||||
TRACE: "text-gray-500",
|
||||
DEBUG: "text-blue-400",
|
||||
INFO: "text-green-400",
|
||||
WARN: "text-amber-400",
|
||||
ERROR: "text-red-400",
|
||||
FATAL: "text-red-600",
|
||||
};
|
||||
@@ -0,0 +1,24 @@
|
||||
/** @type {import('tailwindcss').Config} */
|
||||
export default {
|
||||
content: ["./index.html", "./src/**/*.{ts,tsx}"],
|
||||
theme: {
|
||||
extend: {
|
||||
fontFamily: {
|
||||
mono: ["JetBrains Mono", "Menlo", "Monaco", "monospace"],
|
||||
sans: ["Inter", "system-ui", "sans-serif"],
|
||||
},
|
||||
colors: {
|
||||
canvas: {
|
||||
50: "#FAFAF9",
|
||||
100: "#F5F5F4",
|
||||
200: "#E5E5E3",
|
||||
},
|
||||
primary: {
|
||||
500: "#4A83DD",
|
||||
600: "#3D6DC4",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
plugins: [],
|
||||
};
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2020",
|
||||
"useDefineForClassFields": true,
|
||||
"lib": ["ES2020", "DOM", "DOM.Iterable"],
|
||||
"module": "ESNext",
|
||||
"skipLibCheck": true,
|
||||
"moduleResolution": "bundler",
|
||||
"allowImportingTsExtensions": true,
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"noEmit": true,
|
||||
"jsx": "react-jsx",
|
||||
"strict": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true,
|
||||
"noFallthroughCasesInSwitch": true
|
||||
},
|
||||
"include": ["src"]
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { defineConfig } from "vite";
|
||||
import react from "@vitejs/plugin-react";
|
||||
|
||||
// @ts-expect-error process is a nodejs global
|
||||
const host = process.env.TAURI_DEV_HOST;
|
||||
|
||||
export default defineConfig(async () => ({
|
||||
plugins: [react()],
|
||||
clearScreen: false,
|
||||
server: {
|
||||
port: 1430,
|
||||
strictPort: true,
|
||||
host: host || false,
|
||||
hmr: host
|
||||
? {
|
||||
protocol: "ws",
|
||||
host,
|
||||
port: 1431,
|
||||
}
|
||||
: undefined,
|
||||
watch: {
|
||||
ignored: ["**/src-tauri/**"],
|
||||
},
|
||||
},
|
||||
}));
|
||||
+1284
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,487 @@
|
||||
//! macOS Globe/Fn key listener helper management.
|
||||
//!
|
||||
//! The listener runs as a tiny Swift process that monitors `flagsChanged`
|
||||
//! events globally and reports `FN_DOWN` / `FN_UP` lines over stdout.
|
||||
|
||||
use super::{detect_permissions, PermissionState};
|
||||
use std::collections::VecDeque;
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
use once_cell::sync::Lazy;
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::fs;
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::io::{BufRead, BufReader};
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::path::PathBuf;
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::process::{Child, Command, Stdio};
|
||||
#[cfg(target_os = "macos")]
|
||||
use std::sync::{Arc, Mutex as StdMutex};
|
||||
|
||||
const LOG_PREFIX: &str = "[globe_hotkey]";
|
||||
const MAX_PENDING_EVENTS: usize = 64;
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct GlobeHotkeyStatus {
|
||||
pub supported: bool,
|
||||
pub running: bool,
|
||||
pub input_monitoring_permission: PermissionState,
|
||||
pub last_error: Option<String>,
|
||||
pub events_pending: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct GlobeHotkeyPollResult {
|
||||
pub status: GlobeHotkeyStatus,
|
||||
pub events: Vec<String>,
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
struct GlobeListenerProcess {
|
||||
child: Child,
|
||||
event_queue: Arc<StdMutex<VecDeque<String>>>,
|
||||
last_error: Arc<StdMutex<Option<String>>>,
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
static GLOBE_LISTENER: Lazy<StdMutex<Option<GlobeListenerProcess>>> =
|
||||
Lazy::new(|| StdMutex::new(None));
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn push_event(queue: &Arc<StdMutex<VecDeque<String>>>, event: String) {
|
||||
let Ok(mut guard) = queue.lock() else {
|
||||
log::warn!("{LOG_PREFIX} failed to lock queue for event");
|
||||
return;
|
||||
};
|
||||
guard.push_back(event);
|
||||
while guard.len() > MAX_PENDING_EVENTS {
|
||||
let _ = guard.pop_front();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn set_last_error(error_store: &Arc<StdMutex<Option<String>>>, message: Option<String>) {
|
||||
let Ok(mut guard) = error_store.lock() else {
|
||||
log::warn!("{LOG_PREFIX} failed to lock last_error store");
|
||||
return;
|
||||
};
|
||||
*guard = message;
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn drain_events(queue: &Arc<StdMutex<VecDeque<String>>>) -> Vec<String> {
|
||||
let Ok(mut guard) = queue.lock() else {
|
||||
log::warn!("{LOG_PREFIX} failed to lock queue for drain");
|
||||
return Vec::new();
|
||||
};
|
||||
guard.drain(..).collect()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn queue_len(queue: &Arc<StdMutex<VecDeque<String>>>) -> usize {
|
||||
let Ok(guard) = queue.lock() else {
|
||||
return 0;
|
||||
};
|
||||
guard.len()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn current_error(error_store: &Arc<StdMutex<Option<String>>>) -> Option<String> {
|
||||
let Ok(guard) = error_store.lock() else {
|
||||
return Some("failed to read globe listener error state".to_string());
|
||||
};
|
||||
guard.clone()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn ensure_running_locked(
|
||||
state: &mut Option<GlobeListenerProcess>,
|
||||
) -> Result<GlobeHotkeyStatus, String> {
|
||||
let input_monitoring_permission = detect_permissions().input_monitoring;
|
||||
if input_monitoring_permission != PermissionState::Granted {
|
||||
let message =
|
||||
"input monitoring permission is required for the macOS Globe/Fn listener".to_string();
|
||||
log::warn!(
|
||||
"{LOG_PREFIX} start skipped: input_monitoring_permission={:?}",
|
||||
input_monitoring_permission
|
||||
);
|
||||
if let Some(process) = state.as_ref() {
|
||||
set_last_error(&process.last_error, Some(message.clone()));
|
||||
}
|
||||
return Ok(GlobeHotkeyStatus {
|
||||
supported: true,
|
||||
running: false,
|
||||
input_monitoring_permission,
|
||||
last_error: Some(message),
|
||||
events_pending: 0,
|
||||
});
|
||||
}
|
||||
|
||||
if let Some(process) = state.as_mut() {
|
||||
match process.child.try_wait() {
|
||||
Ok(None) => {
|
||||
return Ok(GlobeHotkeyStatus {
|
||||
supported: true,
|
||||
running: true,
|
||||
input_monitoring_permission,
|
||||
last_error: current_error(&process.last_error),
|
||||
events_pending: queue_len(&process.event_queue),
|
||||
});
|
||||
}
|
||||
Ok(Some(status)) => {
|
||||
let message = format!("globe listener exited unexpectedly: {status}");
|
||||
log::warn!("{LOG_PREFIX} {message}");
|
||||
set_last_error(&process.last_error, Some(message));
|
||||
*state = None;
|
||||
}
|
||||
Err(err) => {
|
||||
let message = format!("failed to inspect globe listener state: {err}");
|
||||
log::warn!("{LOG_PREFIX} {message}");
|
||||
set_last_error(&process.last_error, Some(message));
|
||||
*state = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let binary_path = ensure_globe_helper_binary()?;
|
||||
log::info!("{LOG_PREFIX} starting helper {}", binary_path.display());
|
||||
let mut child = Command::new(&binary_path)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.spawn()
|
||||
.map_err(|e| format!("failed to spawn globe listener helper: {e}"))?;
|
||||
|
||||
let stdout = child
|
||||
.stdout
|
||||
.take()
|
||||
.ok_or_else(|| "failed to capture globe listener stdout".to_string())?;
|
||||
let stderr = child
|
||||
.stderr
|
||||
.take()
|
||||
.ok_or_else(|| "failed to capture globe listener stderr".to_string())?;
|
||||
|
||||
let event_queue = Arc::new(StdMutex::new(VecDeque::with_capacity(MAX_PENDING_EVENTS)));
|
||||
let last_error = Arc::new(StdMutex::new(None));
|
||||
|
||||
{
|
||||
let queue = event_queue.clone();
|
||||
let error_store = last_error.clone();
|
||||
std::thread::spawn(move || {
|
||||
let reader = BufReader::new(stdout);
|
||||
for line in reader.lines() {
|
||||
match line {
|
||||
Ok(line) => {
|
||||
let trimmed = line.trim();
|
||||
if trimmed.is_empty() {
|
||||
continue;
|
||||
}
|
||||
log::debug!("{LOG_PREFIX} helper event={trimmed}");
|
||||
push_event(&queue, trimmed.to_string());
|
||||
set_last_error(&error_store, None);
|
||||
}
|
||||
Err(err) => {
|
||||
let message = format!("failed reading globe listener stdout: {err}");
|
||||
log::warn!("{LOG_PREFIX} {message}");
|
||||
set_last_error(&error_store, Some(message));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
log::debug!("{LOG_PREFIX} stdout reader exited");
|
||||
});
|
||||
}
|
||||
|
||||
{
|
||||
let error_store = last_error.clone();
|
||||
std::thread::spawn(move || {
|
||||
let reader = BufReader::new(stderr);
|
||||
for line in reader.lines() {
|
||||
match line {
|
||||
Ok(line) => {
|
||||
let trimmed = line.trim();
|
||||
if trimmed.is_empty() {
|
||||
continue;
|
||||
}
|
||||
log::warn!("{LOG_PREFIX} helper stderr={trimmed}");
|
||||
set_last_error(&error_store, Some(trimmed.to_string()));
|
||||
}
|
||||
Err(err) => {
|
||||
let message = format!("failed reading globe listener stderr: {err}");
|
||||
log::warn!("{LOG_PREFIX} {message}");
|
||||
set_last_error(&error_store, Some(message));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
log::debug!("{LOG_PREFIX} stderr reader exited");
|
||||
});
|
||||
}
|
||||
|
||||
*state = Some(GlobeListenerProcess {
|
||||
child,
|
||||
event_queue,
|
||||
last_error,
|
||||
});
|
||||
|
||||
let process = state
|
||||
.as_ref()
|
||||
.ok_or_else(|| "globe listener process missing after spawn".to_string())?;
|
||||
Ok(GlobeHotkeyStatus {
|
||||
supported: true,
|
||||
running: true,
|
||||
input_monitoring_permission,
|
||||
last_error: current_error(&process.last_error),
|
||||
events_pending: queue_len(&process.event_queue),
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn ensure_globe_helper_binary() -> Result<PathBuf, String> {
|
||||
let cache_dir = std::env::temp_dir().join("openhuman-globe-listener");
|
||||
fs::create_dir_all(&cache_dir).map_err(|e| format!("failed to create globe cache dir: {e}"))?;
|
||||
|
||||
let source_path = cache_dir.join("globe_listener.swift");
|
||||
let binary_path = cache_dir.join("globe_listener_bin");
|
||||
let source = globe_swift_source();
|
||||
|
||||
let needs_write = match fs::read_to_string(&source_path) {
|
||||
Ok(existing) => existing != source,
|
||||
Err(_) => true,
|
||||
};
|
||||
if needs_write {
|
||||
fs::write(&source_path, &source)
|
||||
.map_err(|e| format!("failed to write globe helper source: {e}"))?;
|
||||
}
|
||||
|
||||
let needs_compile = needs_write || !binary_path.exists();
|
||||
if needs_compile {
|
||||
log::debug!("{LOG_PREFIX} compiling Swift helper");
|
||||
let output = Command::new("xcrun")
|
||||
.args(["swiftc", "-O", "-framework", "Cocoa"])
|
||||
.arg(&source_path)
|
||||
.arg("-o")
|
||||
.arg(&binary_path)
|
||||
.output()
|
||||
.or_else(|_| {
|
||||
Command::new("swiftc")
|
||||
.args(["-O", "-framework", "Cocoa"])
|
||||
.arg(&source_path)
|
||||
.arg("-o")
|
||||
.arg(&binary_path)
|
||||
.output()
|
||||
})
|
||||
.map_err(|e| format!("failed to invoke swiftc for globe listener: {e}"))?;
|
||||
if !output.status.success() {
|
||||
let stderr = String::from_utf8_lossy(&output.stderr).trim().to_string();
|
||||
return Err(format!(
|
||||
"failed to compile globe listener helper: {}",
|
||||
if stderr.is_empty() {
|
||||
"swiftc returned non-zero exit status".to_string()
|
||||
} else {
|
||||
stderr
|
||||
}
|
||||
));
|
||||
}
|
||||
log::debug!("{LOG_PREFIX} Swift helper compiled successfully");
|
||||
}
|
||||
|
||||
Ok(binary_path)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
fn globe_swift_source() -> String {
|
||||
r##"import Cocoa
|
||||
import Darwin
|
||||
|
||||
var fnIsDown = false
|
||||
var lastModifierFlags: NSEvent.ModifierFlags = []
|
||||
|
||||
let rightModifiers: [(UInt16, NSEvent.ModifierFlags, String)] = [
|
||||
(61, .option, "RightOption"),
|
||||
(54, .command, "RightCommand"),
|
||||
(62, .control, "RightControl"),
|
||||
(60, .shift, "RightShift"),
|
||||
]
|
||||
|
||||
let modifierMask: NSEvent.ModifierFlags = [.control, .command, .option, .shift]
|
||||
|
||||
let releases: [(NSEvent.ModifierFlags, String)] = [
|
||||
(.control, "control"),
|
||||
(.command, "command"),
|
||||
(.option, "option"),
|
||||
(.shift, "shift"),
|
||||
]
|
||||
|
||||
func emit(_ message: String) {
|
||||
FileHandle.standardOutput.write((message + "\n").data(using: .utf8)!)
|
||||
fflush(stdout)
|
||||
}
|
||||
|
||||
guard let monitor = NSEvent.addGlobalMonitorForEvents(matching: .flagsChanged, handler: { event in
|
||||
let flags = event.modifierFlags
|
||||
let containsFn = flags.contains(.function)
|
||||
|
||||
if containsFn && !fnIsDown {
|
||||
fnIsDown = true
|
||||
emit("FN_DOWN")
|
||||
} else if !containsFn && fnIsDown {
|
||||
fnIsDown = false
|
||||
emit("FN_UP")
|
||||
}
|
||||
|
||||
let keyCode = event.keyCode
|
||||
for (code, flag, name) in rightModifiers {
|
||||
if keyCode == code {
|
||||
emit(flags.contains(flag) ? "RIGHT_MOD_DOWN:\(name)" : "RIGHT_MOD_UP:\(name)")
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
let currentModifiers = flags.intersection(modifierMask)
|
||||
if currentModifiers != lastModifierFlags {
|
||||
let released = lastModifierFlags.subtracting(currentModifiers)
|
||||
for (flag, name) in releases {
|
||||
if released.contains(flag) {
|
||||
emit("MODIFIER_UP:\(name)")
|
||||
}
|
||||
}
|
||||
lastModifierFlags = currentModifiers
|
||||
}
|
||||
}) else {
|
||||
FileHandle.standardError.write("Failed to create event monitor\n".data(using: .utf8)!)
|
||||
exit(1)
|
||||
}
|
||||
|
||||
let signalSource = DispatchSource.makeSignalSource(signal: SIGTERM, queue: .main)
|
||||
signal(SIGTERM, SIG_IGN)
|
||||
signalSource.setEventHandler {
|
||||
NSEvent.removeMonitor(monitor)
|
||||
exit(0)
|
||||
}
|
||||
signalSource.resume()
|
||||
|
||||
let app = NSApplication.shared
|
||||
app.setActivationPolicy(.accessory)
|
||||
app.run()
|
||||
"##
|
||||
.to_string()
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn globe_listener_start() -> Result<GlobeHotkeyStatus, String> {
|
||||
let mut guard = GLOBE_LISTENER
|
||||
.lock()
|
||||
.map_err(|_| "globe listener lock poisoned".to_string())?;
|
||||
ensure_running_locked(&mut guard)
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn globe_listener_poll() -> Result<GlobeHotkeyPollResult, String> {
|
||||
let mut guard = GLOBE_LISTENER
|
||||
.lock()
|
||||
.map_err(|_| "globe listener lock poisoned".to_string())?;
|
||||
let status = ensure_running_locked(&mut guard)?;
|
||||
let events = guard
|
||||
.as_ref()
|
||||
.map(|process| drain_events(&process.event_queue))
|
||||
.unwrap_or_default();
|
||||
Ok(GlobeHotkeyPollResult {
|
||||
status: GlobeHotkeyStatus {
|
||||
events_pending: 0,
|
||||
..status
|
||||
},
|
||||
events,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(target_os = "macos")]
|
||||
pub fn globe_listener_stop() -> Result<GlobeHotkeyStatus, String> {
|
||||
let mut guard = GLOBE_LISTENER
|
||||
.lock()
|
||||
.map_err(|_| "globe listener lock poisoned".to_string())?;
|
||||
if let Some(mut process) = guard.take() {
|
||||
log::info!("{LOG_PREFIX} stopping helper pid={}", process.child.id());
|
||||
let _ = process.child.kill();
|
||||
let _ = process.child.wait();
|
||||
let events = drain_events(&process.event_queue);
|
||||
log::debug!(
|
||||
"{LOG_PREFIX} drained {} queued events on stop",
|
||||
events.len()
|
||||
);
|
||||
}
|
||||
|
||||
Ok(GlobeHotkeyStatus {
|
||||
supported: true,
|
||||
running: false,
|
||||
input_monitoring_permission: detect_permissions().input_monitoring,
|
||||
last_error: None,
|
||||
events_pending: 0,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
pub fn globe_listener_start() -> Result<GlobeHotkeyStatus, String> {
|
||||
Ok(GlobeHotkeyStatus {
|
||||
supported: false,
|
||||
running: false,
|
||||
input_monitoring_permission: detect_permissions().input_monitoring,
|
||||
last_error: Some("Globe/Fn hotkey listener is only supported on macOS".to_string()),
|
||||
events_pending: 0,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
pub fn globe_listener_poll() -> Result<GlobeHotkeyPollResult, String> {
|
||||
Ok(GlobeHotkeyPollResult {
|
||||
status: GlobeHotkeyStatus {
|
||||
supported: false,
|
||||
running: false,
|
||||
input_monitoring_permission: detect_permissions().input_monitoring,
|
||||
last_error: Some("Globe/Fn hotkey listener is only supported on macOS".to_string()),
|
||||
events_pending: 0,
|
||||
},
|
||||
events: Vec::new(),
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
pub fn globe_listener_stop() -> Result<GlobeHotkeyStatus, String> {
|
||||
Ok(GlobeHotkeyStatus {
|
||||
supported: false,
|
||||
running: false,
|
||||
input_monitoring_permission: detect_permissions().input_monitoring,
|
||||
last_error: Some("Globe/Fn hotkey listener is only supported on macOS".to_string()),
|
||||
events_pending: 0,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::MAX_PENDING_EVENTS;
|
||||
use std::collections::VecDeque;
|
||||
|
||||
fn push_event_local(queue: &mut VecDeque<String>, event: String) {
|
||||
queue.push_back(event);
|
||||
while queue.len() > MAX_PENDING_EVENTS {
|
||||
let _ = queue.pop_front();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn event_queue_keeps_latest_events() {
|
||||
let mut queue = VecDeque::new();
|
||||
for index in 0..(MAX_PENDING_EVENTS + 5) {
|
||||
push_event_local(&mut queue, format!("event-{index}"));
|
||||
}
|
||||
|
||||
assert_eq!(queue.len(), MAX_PENDING_EVENTS);
|
||||
assert_eq!(queue.front().map(String::as_str), Some("event-5"));
|
||||
let expected_last = format!("event-{}", MAX_PENDING_EVENTS + 4);
|
||||
assert_eq!(
|
||||
queue.back().map(String::as_str),
|
||||
Some(expected_last.as_str())
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@
|
||||
|
||||
mod capture;
|
||||
mod focus;
|
||||
mod globe;
|
||||
mod helper;
|
||||
mod keys;
|
||||
mod overlay;
|
||||
@@ -21,6 +22,10 @@ pub use focus::{
|
||||
focused_text_context, focused_text_context_verbose, foreground_context,
|
||||
parse_foreground_output, validate_focused_target,
|
||||
};
|
||||
pub use globe::{
|
||||
globe_listener_poll, globe_listener_start, globe_listener_stop, GlobeHotkeyPollResult,
|
||||
GlobeHotkeyStatus,
|
||||
};
|
||||
pub use keys::{is_escape_key_down, is_tab_key_down};
|
||||
pub use overlay::{hide_overlay, quit_overlay, show_overlay};
|
||||
pub use paste::apply_text_to_focused_field;
|
||||
|
||||
@@ -36,6 +36,7 @@ struct SessionRuntime {
|
||||
panic_hotkey: String,
|
||||
stop_reason: Option<String>,
|
||||
last_capture_at_ms: Option<i64>,
|
||||
capture_count: u64,
|
||||
frames: VecDeque<CaptureFrame>,
|
||||
last_context: Option<AppContext>,
|
||||
task: Option<JoinHandle<()>>,
|
||||
@@ -143,6 +144,7 @@ impl AccessibilityEngine {
|
||||
panic_hotkey: state.config.panic_stop_hotkey.clone(),
|
||||
stop_reason: None,
|
||||
last_capture_at_ms: None,
|
||||
capture_count: 0,
|
||||
frames: VecDeque::new(),
|
||||
last_context: None,
|
||||
task: None,
|
||||
@@ -198,6 +200,14 @@ impl AccessibilityEngine {
|
||||
state.permissions = detect_permissions();
|
||||
|
||||
let context = foreground_context();
|
||||
let foreground_context = context.as_ref().map(|ctx| AppContextInfo {
|
||||
app_name: ctx.app_name.clone(),
|
||||
window_title: ctx.window_title.clone(),
|
||||
bounds_x: ctx.bounds.map(|b| b.x),
|
||||
bounds_y: ctx.bounds.map(|b| b.y),
|
||||
bounds_width: ctx.bounds.map(|b| b.width),
|
||||
bounds_height: ctx.bounds.map(|b| b.height),
|
||||
});
|
||||
let blocked = context
|
||||
.as_ref()
|
||||
.map(|ctx| !self.should_capture_context(ctx, &state.config))
|
||||
@@ -214,12 +224,17 @@ impl AccessibilityEngine {
|
||||
ttl_secs: session.ttl_secs,
|
||||
panic_hotkey: session.panic_hotkey.clone(),
|
||||
stop_reason: session.stop_reason.clone(),
|
||||
capture_count: session.capture_count,
|
||||
frames_in_memory: session.frames.len(),
|
||||
last_capture_at_ms: session.last_capture_at_ms,
|
||||
last_context: session
|
||||
.last_context
|
||||
.as_ref()
|
||||
.and_then(|c| c.app_name.clone()),
|
||||
last_window_title: session
|
||||
.last_context
|
||||
.as_ref()
|
||||
.and_then(|c| c.window_title.clone()),
|
||||
vision_enabled: session.vision_enabled,
|
||||
vision_state: session.vision_state.clone(),
|
||||
vision_queue_depth: session.vision_queue_depth,
|
||||
@@ -234,9 +249,11 @@ impl AccessibilityEngine {
|
||||
ttl_secs: state.config.session_ttl_secs,
|
||||
panic_hotkey: state.config.panic_stop_hotkey.clone(),
|
||||
stop_reason: None,
|
||||
capture_count: 0,
|
||||
frames_in_memory: 0,
|
||||
last_capture_at_ms: None,
|
||||
last_context: None,
|
||||
last_window_title: None,
|
||||
vision_enabled: state.config.vision_enabled,
|
||||
vision_state: "idle".to_string(),
|
||||
vision_queue_depth: 0,
|
||||
@@ -259,6 +276,7 @@ impl AccessibilityEngine {
|
||||
permissions,
|
||||
features,
|
||||
session,
|
||||
foreground_context,
|
||||
config,
|
||||
denylist,
|
||||
is_context_blocked: blocked,
|
||||
@@ -369,6 +387,7 @@ impl AccessibilityEngine {
|
||||
panic_hotkey: state.config.panic_stop_hotkey.clone(),
|
||||
stop_reason: None,
|
||||
last_capture_at_ms: None,
|
||||
capture_count: 0,
|
||||
frames: VecDeque::new(),
|
||||
last_context: None,
|
||||
task: None,
|
||||
@@ -432,6 +451,7 @@ impl AccessibilityEngine {
|
||||
};
|
||||
|
||||
push_ephemeral_frame(&mut session.frames, frame.clone());
|
||||
session.capture_count = session.capture_count.saturating_add(1);
|
||||
session.last_capture_at_ms = Some(frame.captured_at_ms);
|
||||
session.last_context = context;
|
||||
if frame.image_ref.is_some() && session.vision_enabled {
|
||||
@@ -795,6 +815,7 @@ impl AccessibilityEngine {
|
||||
image_ref: capture_result.ok(),
|
||||
};
|
||||
push_ephemeral_frame(&mut session.frames, frame.clone());
|
||||
session.capture_count = session.capture_count.saturating_add(1);
|
||||
session.last_capture_at_ms = Some(now);
|
||||
session.last_context = context;
|
||||
if frame.image_ref.is_some() && session.vision_enabled {
|
||||
@@ -968,6 +989,7 @@ mod tests {
|
||||
panic_hotkey: state.config.panic_stop_hotkey.clone(),
|
||||
stop_reason: None,
|
||||
last_capture_at_ms: None,
|
||||
capture_count: 0,
|
||||
frames: VecDeque::new(),
|
||||
last_context: None,
|
||||
task: None,
|
||||
|
||||
@@ -7,9 +7,9 @@ use serde_json::json;
|
||||
|
||||
use crate::openhuman::screen_intelligence::{
|
||||
self, AccessibilityStatus, CaptureImageRefResult, CaptureNowResult, CaptureTestResult,
|
||||
InputActionParams, InputActionResult, PermissionRequestParams, PermissionState,
|
||||
PermissionStatus, SessionStatus, StartSessionParams, StopSessionParams, VisionFlushResult,
|
||||
VisionRecentResult,
|
||||
GlobeHotkeyPollResult, GlobeHotkeyStatus, InputActionParams, InputActionResult,
|
||||
PermissionRequestParams, PermissionState, PermissionStatus, SessionStatus, StartSessionParams,
|
||||
StopSessionParams, VisionFlushResult, VisionRecentResult,
|
||||
};
|
||||
use crate::rpc::RpcOutcome;
|
||||
|
||||
@@ -214,3 +214,30 @@ pub async fn accessibility_capture_test() -> Result<RpcOutcome<CaptureTestResult
|
||||
"screen intelligence capture test completed",
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn accessibility_globe_listener_start() -> Result<RpcOutcome<GlobeHotkeyStatus>, String> {
|
||||
log::info!("[screen_intelligence] globe_listener_start requested");
|
||||
let result = crate::openhuman::accessibility::globe_listener_start()?;
|
||||
Ok(RpcOutcome::single_log(
|
||||
result,
|
||||
"globe listener start processed",
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn accessibility_globe_listener_poll() -> Result<RpcOutcome<GlobeHotkeyPollResult>, String>
|
||||
{
|
||||
let result = crate::openhuman::accessibility::globe_listener_poll()?;
|
||||
Ok(RpcOutcome::single_log(
|
||||
result,
|
||||
"globe listener poll processed",
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn accessibility_globe_listener_stop() -> Result<RpcOutcome<GlobeHotkeyStatus>, String> {
|
||||
log::info!("[screen_intelligence] globe_listener_stop requested");
|
||||
let result = crate::openhuman::accessibility::globe_listener_stop()?;
|
||||
Ok(RpcOutcome::single_log(
|
||||
result,
|
||||
"globe listener stop processed",
|
||||
))
|
||||
}
|
||||
|
||||
@@ -28,6 +28,9 @@ pub fn all_controller_schemas() -> Vec<ControllerSchema> {
|
||||
schemas("vision_recent"),
|
||||
schemas("vision_flush"),
|
||||
schemas("capture_test"),
|
||||
schemas("globe_listener_start"),
|
||||
schemas("globe_listener_poll"),
|
||||
schemas("globe_listener_stop"),
|
||||
]
|
||||
}
|
||||
|
||||
@@ -81,6 +84,18 @@ pub fn all_registered_controllers() -> Vec<RegisteredController> {
|
||||
schema: schemas("capture_test"),
|
||||
handler: handle_capture_test,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: schemas("globe_listener_start"),
|
||||
handler: handle_globe_listener_start,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: schemas("globe_listener_poll"),
|
||||
handler: handle_globe_listener_poll,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: schemas("globe_listener_stop"),
|
||||
handler: handle_globe_listener_stop,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
@@ -208,6 +223,27 @@ pub fn schemas(function: &str) -> ControllerSchema {
|
||||
"Capture test result with diagnostics.",
|
||||
)],
|
||||
},
|
||||
"globe_listener_start" => ControllerSchema {
|
||||
namespace: "screen_intelligence",
|
||||
function: "globe_listener_start",
|
||||
description: "Start the macOS Globe/Fn hotkey listener helper.",
|
||||
inputs: vec![],
|
||||
outputs: vec![json_output("result", "Globe hotkey listener status.")],
|
||||
},
|
||||
"globe_listener_poll" => ControllerSchema {
|
||||
namespace: "screen_intelligence",
|
||||
function: "globe_listener_poll",
|
||||
description: "Drain pending Globe/Fn listener events and return listener status.",
|
||||
inputs: vec![],
|
||||
outputs: vec![json_output("result", "Globe hotkey listener poll result.")],
|
||||
},
|
||||
"globe_listener_stop" => ControllerSchema {
|
||||
namespace: "screen_intelligence",
|
||||
function: "globe_listener_stop",
|
||||
description: "Stop the macOS Globe/Fn hotkey listener helper.",
|
||||
inputs: vec![],
|
||||
outputs: vec![json_output("result", "Globe hotkey listener status.")],
|
||||
},
|
||||
_ => ControllerSchema {
|
||||
namespace: "screen_intelligence",
|
||||
function: "unknown",
|
||||
@@ -319,6 +355,31 @@ fn handle_capture_test(_params: Map<String, Value>) -> ControllerFuture {
|
||||
})
|
||||
}
|
||||
|
||||
fn handle_globe_listener_start(_params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async {
|
||||
to_json(
|
||||
crate::openhuman::screen_intelligence::rpc::accessibility_globe_listener_start()
|
||||
.await?,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
fn handle_globe_listener_poll(_params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async {
|
||||
to_json(
|
||||
crate::openhuman::screen_intelligence::rpc::accessibility_globe_listener_poll().await?,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
fn handle_globe_listener_stop(_params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async {
|
||||
to_json(
|
||||
crate::openhuman::screen_intelligence::rpc::accessibility_globe_listener_stop().await?,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
fn deserialize_params<T: DeserializeOwned>(params: Map<String, Value>) -> Result<T, String> {
|
||||
serde_json::from_value(Value::Object(params)).map_err(|e| format!("invalid params: {e}"))
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ use crate::openhuman::config::ScreenIntelligenceConfig;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
// Permission types are defined in the accessibility middleware; re-export for compatibility.
|
||||
pub use crate::openhuman::accessibility::{GlobeHotkeyPollResult, GlobeHotkeyStatus};
|
||||
pub use crate::openhuman::accessibility::{PermissionKind, PermissionState, PermissionStatus};
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
@@ -20,9 +21,11 @@ pub struct SessionStatus {
|
||||
pub ttl_secs: u64,
|
||||
pub panic_hotkey: String,
|
||||
pub stop_reason: Option<String>,
|
||||
pub capture_count: u64,
|
||||
pub frames_in_memory: usize,
|
||||
pub last_capture_at_ms: Option<i64>,
|
||||
pub last_context: Option<String>,
|
||||
pub last_window_title: Option<String>,
|
||||
pub vision_enabled: bool,
|
||||
pub vision_state: String,
|
||||
pub vision_queue_depth: usize,
|
||||
@@ -42,6 +45,7 @@ pub struct AccessibilityStatus {
|
||||
pub permissions: PermissionStatus,
|
||||
pub features: AccessibilityFeatures,
|
||||
pub session: SessionStatus,
|
||||
pub foreground_context: Option<AppContextInfo>,
|
||||
pub config: ScreenIntelligenceConfig,
|
||||
pub denylist: Vec<String>,
|
||||
pub is_context_blocked: bool,
|
||||
|
||||
Reference in New Issue
Block a user