Files
openhuman/app/src-tauri/src/lib.rs
T
Steven EnamakelandGitHub 99d61f93a7 feat(webui-messaging): multi-provider webview accounts, scanners, and chat runtime (#629)
* feat(accounts): implement accounts management with webview integration

- Added a new Accounts page for managing user accounts, including the ability to add and remove accounts.
- Introduced AddAccountModal for selecting account providers and initiating account setup.
- Implemented WebviewHost to display third-party web applications (e.g., WhatsApp Web) within the app.
- Enhanced routing to include a protected route for the Accounts page.
- Updated the BottomTabBar to include an Accounts tab for easy navigation.
- Integrated Redux for state management of accounts, messages, and logs, ensuring a seamless user experience.
- Updated dependencies in Cargo.lock to version 0.52.9 for compatibility.

This commit significantly enhances the application's functionality by allowing users to manage accounts directly within the app, improving overall user engagement and experience.

* refactor(accounts): clean up Accounts component layout and improve readability

- Removed unnecessary comments and simplified the structure of the Accounts component for better clarity.
- Adjusted the rendering logic to enhance the layout of the active account section, improving user experience.
- Reformatted text in the no accounts message for better readability.
- Streamlined the import statements by consolidating related imports, enhancing code organization.

* feat(accounts): enhance account management with new providers and routing updates

- Introduced support for additional account providers: Telegram, LinkedIn, Gmail, and Slack, expanding user options for account management.
- Updated routing to replace the old /conversations path with /chat, streamlining navigation and improving user experience.
- Refactored the App component to include an AppShell for better layout management, ensuring the bottom tab bar visibility aligns with the selected account.
- Enhanced the BottomTabBar component to reflect the new routing and account options, improving accessibility and usability.
- Implemented fullscreen logic for accounts, allowing for a more immersive experience when interacting with selected accounts.
- Added utility functions for managing fullscreen states and account provider icons, enhancing code organization and maintainability.

This commit significantly improves the application's account management capabilities, providing users with a more flexible and engaging experience.

* feat(cef): integrate Chromium Embedded Framework support and enhance webview functionality

- Added support for the Chromium Embedded Framework (CEF) as an alternative runtime, allowing for improved webview capabilities.
- Updated Cargo.toml to include new dependencies and features for CEF integration, ensuring compatibility with existing Tauri plugins.
- Enhanced the WhatsApp recipe to include ghost-text autocomplete functionality, improving user experience during message composition.
- Implemented WebSocket observation in the WhatsApp recipe to capture and forward relevant message frames, enhancing real-time interaction.
- Introduced user agent spoofing for specific providers to bypass fingerprinting checks, ensuring better compatibility with services like Slack and LinkedIn.
- Refactored various components to accommodate the new runtime and improve overall code organization and maintainability.

This commit significantly enhances the application's webview capabilities and user interaction with messaging services, providing a more robust and flexible experience.

* feat(cef): add development command for CEF and update configuration

- Introduced a new development command `dev:cef` in both package.json files to streamline the development process for the Chromium Embedded Framework (CEF).
- Updated Cargo.toml to include the `tauri/devtools` feature alongside `tauri/cef`, enhancing debugging capabilities.
- Modified tauri.conf.json to adjust visibility settings for the application window and refined the Content Security Policy (CSP) for improved security.
- Enhanced resource paths in tauri.conf.json to support recursive file inclusion for better resource management.
- Updated the Rust code to bypass macOS Keychain prompts when using CEF, improving user experience during development.

This commit enhances the development workflow for CEF integration, providing better tools and configurations for developers.

* fix(cef): update development command for CEF to include signing identity

- Modified the `dev:cef` command in package.json to include the `APPLE_SIGNING_IDENTITY` environment variable, enhancing the development process for CEF on macOS.
- This change improves the build process by ensuring proper code signing during development, streamlining the workflow for developers working with CEF integration.

* feat(cef): enhance development command for CEF with safe storage setup

- Updated the `dev:cef` command in package.json to include a call to a new script, `setup-chromium-safe-storage.sh`, which pre-seeds the "Chromium Safe Storage" keychain entry with a permissive ACL.
- Added the `setup-chromium-safe-storage.sh` script to ensure that CEF/Chromium can read the keychain entry without prompting, improving the development experience on macOS.
- This change streamlines the setup process for developers working with CEF integration, ensuring a smoother workflow.

* feat(whatsapp): enhance message ingestion and IndexedDB integration

- Introduced a new `IngestMessage` interface to standardize message structure for WhatsApp.
- Updated `IngestPayload` to include additional fields for better message handling, including `provider`, `chatId`, and `day`.
- Implemented a new function `persistWhatsappChatDay` to handle the ingestion of chat messages by day, improving data organization and retrieval.
- Enhanced the WhatsApp recipe to utilize IndexedDB for direct data access, eliminating the need for DOM scraping and improving performance.
- Updated the Tauri configuration to enable development tools for easier debugging of webview accounts.

This commit significantly improves the application's ability to manage and ingest WhatsApp messages, providing a more robust and efficient user experience.

* feat(cdp): integrate IndexedDB scanner for WhatsApp via Chrome DevTools Protocol

- Added a new module for scanning IndexedDB using the Chrome DevTools Protocol (CDP), enabling direct access to WhatsApp data without DOM scraping.
- Implemented a scanner that communicates with the embedded CEF instance to read and decrypt messages stored in IndexedDB.
- Updated the Tauri application to manage the new scanner, ensuring it operates seamlessly with existing webview accounts.
- Enhanced the Cargo.toml and Cargo.lock files to include necessary dependencies such as `tokio-tungstenite` and `futures-util` for asynchronous operations.
- Refactored the WhatsApp recipe to utilize the new scanning capabilities, improving performance and data handling.

This commit significantly enhances the application's ability to interact with WhatsApp's IndexedDB, providing a more efficient and robust user experience.

* feat(cdp): enhance message diagnostics in IndexedDB scanner

- Updated the ScanSnapshot struct to include new fields for message diagnostics: `messageKeyUnion`, `messageTypeBreakdown`, and `sampleByType`, providing a comprehensive overview of message structures and types.
- Modified the scanner logic to capture and log detailed information about message types and their shapes, improving debugging capabilities.
- Refactored the JavaScript scanner to aggregate message key signatures and counts, enhancing the analysis of message records.

This commit significantly improves the application's ability to analyze and log message data from WhatsApp's IndexedDB, facilitating better debugging and data handling.

* feat(cdp): implement fast DOM scraping and crypto key extraction for WhatsApp

- Introduced a new fast-tick DOM scraping mechanism to extract rendered WhatsApp message bodies, enabling near real-time message updates without relying on IndexedDB.
- Added scripts for capturing and logging CryptoKey operations within WhatsApp's workers, allowing for better analysis of key derivations and decryptions.
- Enhanced the CDP scanner to interleave fast DOM scans with full IndexedDB scans, optimizing data retrieval and reducing UI spamming during idle periods.
- Updated the ScanSnapshot struct to include new fields for DOM-scraped messages and crypto operation statistics, improving the overall diagnostic capabilities of the application.

This commit significantly enhances the application's ability to interact with WhatsApp's messaging system, providing a more efficient and responsive user experience.

* feat(whatsapp): replace cdp_indexeddb with whatsapp_scanner for enhanced message handling

- Replaced the `cdp_indexeddb` module with `whatsapp_scanner` to streamline the scanning process for WhatsApp messages.
- Updated the application to manage the new `ScannerRegistry` for WhatsApp, improving the integration with the Chrome DevTools Protocol.
- Introduced new scripts for fast DOM scraping and full IndexedDB scanning, optimizing data retrieval and enhancing real-time message updates.
- Added a new `dom_scan.js` for efficient extraction of rendered message bodies directly from the DOM, reducing reliance on IndexedDB.
- Enhanced the `ScanSnapshot` struct to accommodate new fields for DOM-scraped messages, improving diagnostic capabilities.

This commit significantly improves the application's ability to interact with WhatsApp, providing a more efficient and responsive user experience.

* refactor(whatsapp): replace JavaScript DOM scanning with Rust-based DOM snapshot

- Removed the `dom_scan.js` script and replaced it with a new Rust module `dom_snapshot.rs` that captures DOM snapshots directly via the Chrome DevTools Protocol, enhancing performance and reliability.
- Introduced a new `idb.rs` module for scanning WhatsApp's IndexedDB, streamlining data retrieval and improving integration with the Rust backend.
- Updated the `ScanSnapshot` struct to accommodate changes in data handling, ensuring compatibility with the new scanning methods.
- Enhanced overall message handling capabilities, providing a more efficient and responsive user experience.

This commit significantly improves the application's ability to interact with WhatsApp, leveraging Rust for better performance and reducing reliance on JavaScript for DOM operations.

* feat(docs): add webview integration playbook for third-party messaging

- Introduced a comprehensive playbook detailing the process for integrating third-party webviews (e.g., Instagram, Messenger) into the application.
- Documented architecture, workflow, and best practices for building and debugging new integrations, leveraging Rust and Chrome DevTools Protocol.
- Included step-by-step instructions for setting up scanners, monitoring logs, and optimizing message handling, ensuring a streamlined development experience for future integrations.

This addition enhances the documentation, providing developers with a clear guide to implement and maintain webview integrations effectively.

* docs(webview): improve table formatting for clarity

- Enhanced the formatting of tables in the webview integration playbook to improve readability and consistency.
- Adjusted column headers and alignment for better presentation of job intervals and costs, ensuring clearer communication of scanning processes and common pitfalls.

This update aims to provide a more user-friendly documentation experience for developers integrating third-party webviews.

* feat(slack): integrate Slack scanner for message extraction and management

- Updated the OpenHuman package version to 0.52.15 in both Cargo.lock files.
- Introduced a new Slack scanner module to extract messages, users, and channels from Slack's IndexedDB using the Chrome DevTools Protocol.
- Added functionality to manage Slack accounts within the application, allowing for automatic opening of Slack webviews based on environment variables.
- Enhanced the existing webview account management to support Slack integration, ensuring seamless interaction with the Slack API.

This commit significantly improves the application's ability to interact with Slack, providing a robust framework for message handling and account management.

* fix(slack): one-doc-per-channel + omit CDP indexName param

CEF 146's IndexedDB.requestData rejects `indexName: ""` with "Could not
get index"; the CDP spec says empty string means the primary-key index
but this backend only accepts the field unset. Omit it entirely so the
Slack Redux-persist dump actually comes back.

Also switch memory grouping from (channel, day) → channel. Each Slack
channel is now one long-running memory doc keyed by channel name
(e.g. `general`, `team-product`, `elvin516`), falling back to channel
id for non-slug names. Every transcript line carries its own
`YYYY-MM-DD HH:MM` stamp and the header records the full date range.

`infer_team_id` updated to Slack's real DB naming pattern
`objectStore-<TEAM>-<USER>` (not `ReduxPersistIDB:` as initially
assumed).

* fix(onboarding): update navigation and test descriptions in OnboardingOverlay tests

- Changed the navigation path from '/conversations' to '/chat' in the OnboardingOverlay tests to reflect the updated routing logic.
- Updated test descriptions for clarity, ensuring they accurately describe the functionality being tested.

These changes enhance the accuracy and readability of the onboarding tests, aligning them with the current application flow.

* fix(conversations): add return statement to Conversations component

- Introduced a return statement in the Conversations component to ensure proper rendering of the sidebar or page variant.
- This change enhances the component's functionality by ensuring it returns the expected JSX structure.

These modifications improve the overall structure and behavior of the Conversations component.

* feat(accounts): add Discord integration and enhance AddAccountModal

- Introduced Discord as a new account provider, including its icon and service details.
- Updated the AddAccountModal to filter out already connected providers, improving user experience.
- Enhanced the UI to display a message when all providers are connected, ensuring clarity for users.
- Implemented context menu functionality for account management, allowing users to log out directly from the accounts list.

These changes expand the application's capabilities by integrating Discord and refining account management features.

* feat(discord): integrate Discord scanner for HTTP and WebSocket monitoring

- Added a new `discord_scanner` module to capture Discord API calls and WebSocket frames using the Chrome DevTools Protocol (CDP).
- Updated the `lib.rs` to manage the new Discord scanner alongside existing WhatsApp and Slack scanners.
- Enhanced the `webview_accounts` module to support Discord account management, including scanner registration and cleanup.

These changes expand the application's capabilities by enabling real-time monitoring of Discord interactions, enhancing user experience and functionality.

* feat(google-meet): integrate Google Meet as a new account provider

- Added Google Meet as a supported account provider, including its icon and service details.
- Updated the account management logic to handle Google Meet interactions, including recipe integration for call monitoring and notifications.
- Enhanced the UI to accommodate the new provider, ensuring a seamless user experience when managing accounts.

These changes expand the application's capabilities by integrating Google Meet, allowing users to join calls and receive notifications directly within the app.

* refactor(runtime): remove notification handling and composer autocomplete

- Eliminated the notification interception logic and associated functions, streamlining the runtime code.
- Removed composer autocomplete features, transferring responsibility for ghost-text overlays to the UI host.
- Updated comments to reflect the changes and clarify the remaining functionality.

These modifications simplify the runtime script, focusing on core features while delegating UI responsibilities.

* feat(google-meet): enhance Google Meet integration with lifecycle event handling

- Implemented lifecycle event handling for Google Meet, including events for call start, captions, and call end.
- Introduced in-memory storage for caption snapshots during meetings, allowing for the generation of markdown transcripts upon call completion.
- Added interfaces for payload structures related to Google Meet events, improving type safety and clarity in the codebase.
- Updated the webview account service to manage active meetings and flush transcripts to memory, ensuring a seamless user experience.

These changes significantly enhance the Google Meet integration, enabling real-time caption handling and transcript generation, thereby improving the overall functionality of the application.

* feat(cef): integrate CEF-based notification handling and update dependencies

- Added support for native OS notifications through the `tauri-runtime-cef` crate, enabling interception of browser notifications in embedded webviews.
- Introduced a new submodule for `tauri-cef` to manage CEF dependencies and facilitate notification handling.
- Updated the `.gitignore` to exclude CEF-related build artifacts and lock files.
- Removed the deprecated `notification_scanner` module, streamlining the codebase and focusing on the new CEF integration.
- Enhanced the `webview_accounts` module to register and manage CEF browser notifications, improving user experience with real-time alerts.

These changes significantly enhance the application's notification capabilities, leveraging CEF for a more integrated and responsive user experience.

* feat(cef): update CEF integration and dependency management

- Added `cef` and `tauri-runtime-cef` as dependencies to enhance CEF support.
- Updated `Cargo.toml` to reference `tauri-runtime-cef` from a local path, ensuring proper integration with the vendored CEF submodule.
- Removed direct Git references for Tauri packages, streamlining dependency management by using local paths.

These changes improve the application's CEF capabilities and simplify the dependency structure, facilitating better integration and maintenance.

* feat(browserscan): add BrowserScan as a new account provider with associated resources

- Introduced BrowserScan as a development-only account provider, including its icon and service details.
- Updated the account provider types and management logic to accommodate BrowserScan, enhancing the application's capabilities.
- Added a new recipe and manifest for BrowserScan, ensuring it integrates seamlessly into the existing webview account lifecycle.
- Enhanced the UI to display BrowserScan, providing users with a bot-detection sandbox for testing purposes.

These changes expand the application's functionality by integrating BrowserScan, allowing for improved testing and development workflows.

* feat(google-meet): enhance Google Meet integration with new transcript handling and session recovery

- Introduced a new service to handle Google Meet transcripts, enabling structured note extraction and proactive follow-up actions.
- Implemented session recovery logic to manage in-progress meetings when navigating away from the call.
- Updated the webview account service to log call events and captions, improving monitoring and debugging capabilities.
- Enhanced the Google Meet recipe to persist meeting state across navigations, ensuring seamless user experience.

These changes significantly improve the Google Meet integration, allowing for better management of meeting transcripts and user interactions.

* refactor(App): reorganize imports and clean up code structure

- Removed unused imports from App.tsx to streamline the code.
- Adjusted the import order for better readability and consistency.
- Enhanced the BottomTabBar component by simplifying the button rendering logic.
- Cleaned up the AddAccountModal component by consolidating prop destructuring.
- Improved formatting in various components for better code clarity.

These changes enhance code maintainability and readability across the application.

* update tauri

* feat(google-meet): improve caption handling and speaker identification logic

- Enhanced the logic for extracting speaker names from Google Meet rows, adding checks to filter out icon ligatures and irrelevant text.
- Updated the caption processing to better identify and score caption regions, ensuring more accurate transcript generation.
- Introduced new utility functions to differentiate between real captions and icon names, improving the overall reliability of the captioning feature.

These changes significantly enhance the accuracy and usability of the Google Meet integration, providing users with clearer and more relevant caption data.

* chore(dependencies): update Cargo.lock with new package versions and remove obsolete entries

- Bump OpenHuman version to 0.52.20.
- Add new dependencies: cef, futures-util, tauri-plugin-notification, tauri-runtime-cef, tokio-tungstenite, bzip2, clap, console, cookie_store, data-encoding, dioxus-debug-cell, document-features, download-cef, encode_unicode, filetime.
- Remove obsolete packages: alloc-no-stdlib, alloc-stdlib, brotli, brotli-decompressor, gdkx11, gdkx11-sys.
- Update existing dependencies to their latest versions where applicable.

* chore(dependencies): update submodule URLs and pin plugin versions

- Changed the tauri-cef submodule URL from SSH to HTTPS for consistency.
- Updated Cargo.toml to pin tauri plugins to a specific commit for reproducibility.
- Adjusted Cargo.lock to reflect the new plugin source format with pinned revisions.
- Modified the builder configuration in lib.rs to conditionally enable devtools in debug builds only, enhancing security in release builds.
- Updated webview_accounts to restrict devtools access based on build type, ensuring better control over webview inspection.

* feat(ui): enhance BottomTabBar and AddAccountModal interactions

- Added focus and blur event handlers to BottomTabBar for improved visibility management.
- Implemented keyboard accessibility in AddAccountModal by closing the modal on Escape key press and focusing the close button when opened.
- Updated Content Security Policy in tauri.conf.json for enhanced security.
- Improved Gmail recipe to use stable message IDs for better message tracking.
- Introduced account ID sanitization in webview_accounts to prevent path traversal vulnerabilities.

* fix(AddAccountModal): add missing import for improved functionality

* chore(dependencies): bump OpenHuman version to 0.52.20 in Cargo.lock
2026-04-17 15:55:50 -07:00

858 lines
34 KiB
Rust

#[cfg(not(any(target_os = "windows", target_os = "macos", target_os = "linux")))]
compile_error!("src-tauri host is desktop-only. Non-desktop targets are not supported.");
mod core_process;
mod core_update;
#[cfg(feature = "cef")]
mod discord_scanner;
#[cfg(feature = "cef")]
mod slack_scanner;
mod webview_accounts;
#[cfg(feature = "cef")]
mod whatsapp_scanner;
use std::sync::Mutex;
use tauri::{
menu::{Menu, MenuItem},
tray::{MouseButton, MouseButtonState, TrayIconBuilder, TrayIconEvent},
AppHandle, Emitter, Manager, PhysicalPosition, RunEvent, WebviewWindow, WindowEvent,
};
use tauri_plugin_global_shortcut::{GlobalShortcutExt, ShortcutState};
#[cfg(any(windows, target_os = "linux"))]
use tauri_plugin_deep_link::DeepLinkExt;
#[cfg(target_os = "macos")]
use objc2::runtime::{AnyClass, AnyObject};
#[cfg(target_os = "macos")]
use objc2::ClassType;
#[cfg(target_os = "macos")]
use objc2_app_kit::{NSPanel, NSWindowCollectionBehavior, NSWindowStyleMask};
// Runtime backend alias. `AppHandle`/`WebviewWindow` get their default `R`
// (the runtime generic) from whichever runtime feature the tauri crate has
// enabled — and when we drop `tauri/wry` in favor of `tauri/cef`, `Wry`
// disappears entirely, so plain `AppHandle` (no generic) stops resolving.
// Every helper that touches an `AppHandle` or `WebviewWindow` threads this
// alias through its signature; tauri command handlers get the right runtime
// automatically from the `#[tauri::command]` macro.
#[cfg(feature = "cef")]
pub(crate) type AppRuntime = tauri::Cef;
#[cfg(not(feature = "cef"))]
pub(crate) type AppRuntime = tauri::Wry;
/// Tracks the currently registered dictation hotkey string so we can unregister it later.
struct DictationHotkeyState(Mutex<Vec<String>>);
fn expand_dictation_shortcuts(shortcut: &str) -> Vec<String> {
let trimmed = shortcut.trim();
if trimmed.is_empty() {
return vec![];
}
#[cfg(target_os = "macos")]
{
if trimmed.contains("CmdOrCtrl") {
let cmd_variant = trimmed.replace("CmdOrCtrl", "Cmd");
let ctrl_variant = trimmed.replace("CmdOrCtrl", "Ctrl");
if cmd_variant == ctrl_variant {
return vec![cmd_variant];
}
return vec![cmd_variant, ctrl_variant];
}
}
#[cfg(not(target_os = "macos"))]
{
if trimmed.contains("CmdOrCtrl") {
return vec![trimmed.replace("CmdOrCtrl", "Ctrl")];
}
}
vec![trimmed.to_string()]
}
#[tauri::command]
fn core_rpc_url() -> String {
std::env::var("OPENHUMAN_CORE_RPC_URL")
.unwrap_or_else(|_| "http://127.0.0.1:7788/rpc".to_string())
}
#[tauri::command]
fn overlay_parent_rpc_url() -> Option<String> {
let url = std::env::var("OPENHUMAN_CORE_RPC_URL").ok()?;
let trimmed = url.trim();
if trimmed.is_empty() {
return None;
}
Some(trimmed.to_string())
}
fn pin_overlay_bottom_right(window: &WebviewWindow<AppRuntime>) {
let Ok(Some(monitor)) = window.current_monitor() else {
log::warn!("[overlay] could not resolve current monitor for positioning");
return;
};
let Ok(size) = window.outer_size() else {
log::warn!("[overlay] could not resolve overlay size for positioning");
return;
};
let margin = 20i32;
let x = monitor.position().x + monitor.size().width as i32 - size.width as i32 - margin;
let y = monitor.position().y + monitor.size().height as i32 - size.height as i32 - margin;
if let Err(err) = window.set_position(PhysicalPosition::new(x, y)) {
log::warn!("[overlay] failed to pin overlay bottom-right: {err}");
} else {
log::info!("[overlay] pinned overlay bottom-right at {},{}", x, y);
}
}
#[cfg(target_os = "macos")]
fn configure_overlay_window_macos(window: &WebviewWindow<AppRuntime>) {
// Standard NSWindow cannot float above fullscreen apps on macOS because
// fullscreen apps run in a separate Space. Only NSPanel can do this.
//
// Tauri/tao hardcodes NSWindow as the window class, so we use
// object_setClass() to reclass the existing NSWindow into an NSPanel
// at runtime. This avoids creating a new window (which crashes because
// Tao's window delegate is tightly coupled to the original NSWindow).
//
// After reclassing, we set the NonactivatingPanel style mask and
// Transient collection behavior — matching the working Swift overlay
// helper (accessibility/helper.rs OverlayController) which is confirmed
// to float above fullscreen apps on macOS Sonoma.
//
// Previous attempts that FAILED:
// 1. CGShieldingWindowLevel + CanJoinAllSpaces + FullScreenAuxiliary → hidden
// 2. Window level i32::MAX-17 + Stationary → hidden
// 3. CGS private API CGSSetWindowTags sticky bit → hidden
// 4. object_setClass WITHOUT NonactivatingPanel style mask → hidden
// 5. Create new NSPanel + reparent webview → CRASH (Tao delegate panic)
//
// See: https://github.com/tauri-apps/tauri/issues/11488
match window.ns_window() {
Ok(ns_window_raw) => unsafe {
let ns_window = ns_window_raw as *mut AnyObject;
// ── Reclass NSWindow → NSPanel ──────────────────────────
let panel_class: *const AnyClass = NSPanel::class();
objc2::ffi::object_setClass(ns_window, panel_class);
log::info!("[overlay] reclassed NSWindow → NSPanel via object_setClass");
// Cast to NSPanel for method calls
let panel: &NSPanel = &*(ns_window as *const NSPanel);
// ── Style mask: add NonactivatingPanel ──────────────────
// This is the KEY piece the Swift helper uses. Without it,
// the panel doesn't behave as a proper non-activating panel
// and won't float above fullscreen Spaces.
let current_style = panel.styleMask();
panel.setStyleMask(current_style | NSWindowStyleMask::NonactivatingPanel);
// ── Collection behavior ─────────────────────────────────
// The Swift helper uses .canJoinAllSpaces + .transient
// (NOT .stationary or .fullScreenAuxiliary alone).
// Transient means the panel follows the active Space and
// appears above fullscreen apps.
panel.setCollectionBehavior(
NSWindowCollectionBehavior::CanJoinAllSpaces
| NSWindowCollectionBehavior::Transient
| NSWindowCollectionBehavior::FullScreenAuxiliary
| NSWindowCollectionBehavior::IgnoresCycle,
);
// ── Window level: status bar tier ───────────────────────
// NSStatusWindowLevel = 25. The Swift helper uses .statusBar
// which is the same value.
panel.setLevel(25);
// ── Panel-specific properties ───────────────────────────
panel.setFloatingPanel(true);
panel.setHidesOnDeactivate(false);
panel.setBecomesKeyOnlyIfNeeded(true);
panel.setWorksWhenModal(true);
// Make sure it's ordered front
panel.orderFrontRegardless();
log::info!(
"[overlay] NSPanel configured — level=25, \
NonactivatingPanel+canJoinAllSpaces+transient, \
floatingPanel={}, hidesOnDeactivate={}",
panel.isFloatingPanel(),
panel.hidesOnDeactivate(),
);
},
Err(err) => {
log::warn!("[overlay] failed to access native NSWindow handle: {err}");
}
}
}
/// Resolve the core binary, preferring the staged sidecar.
fn resolve_core_bin() -> Result<std::path::PathBuf, String> {
if let Some(bin) = core_process::default_core_bin() {
return Ok(bin);
}
std::env::current_exe().map_err(|e| format!("cannot resolve executable: {e}"))
}
/// Run the core binary with the given CLI args and return its stdout.
async fn run_core_cli(args: Vec<String>) -> Result<String, String> {
tokio::task::spawn_blocking(move || {
let bin = resolve_core_bin()?;
let is_self = {
let current = std::env::current_exe().ok();
current
.as_ref()
.and_then(|c| std::fs::canonicalize(c).ok())
.zip(std::fs::canonicalize(&bin).ok())
.map_or(false, |(a, b)| a == b)
};
let mut cmd = std::process::Command::new(&bin);
if is_self {
cmd.arg("core");
}
cmd.args(&args);
log::info!(
"[service-direct] running {:?} {}{}",
bin,
if is_self { "core " } else { "" },
args.join(" ")
);
let output = cmd
.output()
.map_err(|e| format!("failed to execute core binary: {e}"))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(format!(
"core binary exited with {}: {}",
output.status,
stderr.trim()
));
}
Ok(String::from_utf8_lossy(&output.stdout).to_string())
})
.await
.map_err(|e| format!("task join error: {e}"))?
}
#[tauri::command]
async fn service_install_direct() -> Result<String, String> {
run_core_cli(vec!["service".into(), "install".into()]).await
}
#[tauri::command]
async fn service_start_direct() -> Result<String, String> {
run_core_cli(vec!["service".into(), "start".into()]).await
}
#[tauri::command]
async fn service_stop_direct() -> Result<String, String> {
run_core_cli(vec!["service".into(), "stop".into()]).await
}
#[tauri::command]
async fn service_status_direct() -> Result<String, String> {
run_core_cli(vec!["service".into(), "status".into()]).await
}
#[tauri::command]
async fn service_uninstall_direct() -> Result<String, String> {
run_core_cli(vec!["service".into(), "uninstall".into()]).await
}
/// Check if the core sidecar is outdated and whether a newer version is available on GitHub.
/// Returns version info, compatibility status, and update availability.
#[tauri::command]
async fn check_core_update(
state: tauri::State<'_, core_process::CoreProcessHandle>,
) -> Result<serde_json::Value, String> {
let rpc_url = state.inner().rpc_url();
let info = core_update::check_full(&rpc_url).await?;
serde_json::to_value(&info).map_err(|e| format!("serialize error: {e}"))
}
/// Trigger a full core update: download latest from GitHub, stage, kill old, restart.
/// Uses `force=true` so it updates to the latest release even if the running core
/// meets the minimum version requirement.
#[tauri::command]
async fn apply_core_update(
state: tauri::State<'_, core_process::CoreProcessHandle>,
app: tauri::AppHandle<AppRuntime>,
) -> Result<(), String> {
log::info!("[core-update] manual apply_core_update invoked from frontend");
core_update::check_and_update_core(state.inner().clone(), Some(app), true).await
}
#[tauri::command]
async fn restart_core_process(
state: tauri::State<'_, core_process::CoreProcessHandle>,
) -> Result<(), String> {
log::info!("[core] restart_core_process: command invoked from frontend");
let _guard = state.inner().restart_lock().await;
log::debug!("[core] restart_core_process: acquired restart lock");
state.inner().restart().await
}
/// Register (or re-register) the global dictation toggle hotkey.
/// Emits `dictation://toggle` to all webviews when the shortcut is pressed.
#[tauri::command]
async fn register_dictation_hotkey(
app: AppHandle<AppRuntime>,
shortcut: String,
) -> Result<(), String> {
log::info!("[dictation] register_dictation_hotkey: shortcut={shortcut}");
let old_shortcuts = {
let state = app.state::<DictationHotkeyState>();
let guard = state.0.lock().unwrap();
guard.clone()
};
let expanded_shortcuts = expand_dictation_shortcuts(&shortcut);
if expanded_shortcuts.is_empty() {
return Err("Shortcut cannot be empty".to_string());
}
log::info!(
"[dictation] expanded shortcuts: {}",
expanded_shortcuts.join(", ")
);
let register_shortcut = |shortcut_variant: &str| -> Result<(), String> {
let app_clone = app.clone();
app.global_shortcut()
.on_shortcut(shortcut_variant, move |_app, _sc, event| {
if event.state == ShortcutState::Pressed {
log::debug!("[dictation] hotkey pressed — emitting dictation://toggle");
if let Err(e) = app_clone.emit("dictation://toggle", ()) {
log::warn!("[dictation] emit failed: {e}");
}
}
})
.map_err(|e| format!("Failed to register shortcut '{shortcut_variant}': {e}"))
};
let mut unregistered_old: Vec<String> = Vec::new();
for old in &old_shortcuts {
log::debug!("[dictation] unregistering previous shortcut: {old}");
if let Err(e) = app.global_shortcut().unregister(old.as_str()) {
for restored in &unregistered_old {
if let Err(restore_err) = register_shortcut(restored.as_str()) {
log::warn!(
"[dictation] rollback failed while restoring old shortcut '{restored}': {restore_err}"
);
}
}
return Err(format!(
"Failed to unregister previous shortcut '{old}': {e}"
));
}
unregistered_old.push(old.clone());
}
let mut newly_registered: Vec<String> = Vec::new();
for shortcut_variant in &expanded_shortcuts {
if let Err(err) = register_shortcut(shortcut_variant.as_str()) {
log::error!("[dictation] failed to register shortcut '{shortcut_variant}': {err}");
for registered in &newly_registered {
if let Err(unregister_err) = app.global_shortcut().unregister(registered.as_str()) {
log::warn!(
"[dictation] rollback failed while unregistering '{registered}': {unregister_err}"
);
}
}
for old in &old_shortcuts {
if let Err(restore_err) = register_shortcut(old.as_str()) {
log::warn!(
"[dictation] rollback failed while restoring old shortcut '{old}': {restore_err}"
);
}
}
return Err(err);
}
newly_registered.push(shortcut_variant.clone());
}
// Persist all newly registered shortcuts.
{
let state = app.state::<DictationHotkeyState>();
let mut guard = state.0.lock().unwrap();
*guard = expanded_shortcuts.clone();
}
log::info!(
"[dictation] shortcuts registered: {}",
expanded_shortcuts.join(", ")
);
Ok(())
}
/// Unregister the global dictation hotkey (if any).
#[tauri::command]
async fn unregister_dictation_hotkey(app: AppHandle<AppRuntime>) -> Result<(), String> {
log::info!("[dictation] unregister_dictation_hotkey: called");
let state = app.state::<DictationHotkeyState>();
let mut guard = state.0.lock().unwrap();
if guard.is_empty() {
log::debug!("[dictation] no shortcut registered — nothing to unregister");
} else {
let old_shortcuts = guard.clone();
guard.clear();
for old in old_shortcuts {
log::debug!("[dictation] unregistering shortcut: {old}");
app.global_shortcut()
.unregister(old.as_str())
.map_err(|e| {
log::warn!("[dictation] failed to unregister '{old}': {e}");
format!("Failed to unregister shortcut '{old}': {e}")
})?;
log::info!("[dictation] shortcut unregistered: {old}");
}
}
Ok(())
}
fn is_daemon_mode() -> bool {
std::env::args().any(|arg| arg == "daemon" || arg == "--daemon")
}
/// Tauri command: bring the main window to front from any webview (e.g. overlay orb click).
#[tauri::command]
fn activate_main_window(app: AppHandle<AppRuntime>) -> Result<(), String> {
log::debug!("[window] activate_main_window called from overlay");
show_main_window(&app)
}
fn show_main_window(app: &AppHandle<AppRuntime>) -> Result<(), String> {
let window = app
.get_webview_window("main")
.ok_or_else(|| "main window not found".to_string())?;
window
.show()
.map_err(|err| format!("failed to show main window: {err}"))?;
window
.unminimize()
.map_err(|err| format!("failed to unminimize main window: {err}"))?;
window
.set_focus()
.map_err(|err| format!("failed to focus main window: {err}"))?;
Ok(())
}
fn setup_tray(app: &AppHandle<AppRuntime>) -> tauri::Result<()> {
log::info!("[tray] setting up tray icon");
let show_item = MenuItem::with_id(
app,
"tray_show_window",
"Open OpenHuman",
true,
None::<&str>,
)?;
let quit_item = MenuItem::with_id(app, "tray_quit", "Quit", true, None::<&str>)?;
let menu = Menu::with_items(app, &[&show_item, &quit_item])?;
let icon = app
.default_window_icon()
.cloned()
.ok_or_else(|| tauri::Error::AssetNotFound("default window icon".to_string()))?;
TrayIconBuilder::with_id("openhuman-tray")
.icon(icon)
.menu(&menu)
.on_menu_event(|app, event| match event.id().as_ref() {
"tray_show_window" => {
log::info!("[tray] action=show_window source=menu");
if let Err(err) = show_main_window(app) {
log::error!("[tray] failed to show main window from menu: {err}");
}
}
"tray_quit" => {
log::info!("[tray] action=quit source=menu");
app.exit(0);
}
_ => {}
})
.on_tray_icon_event(|tray, event| {
if let TrayIconEvent::Click {
button: MouseButton::Left,
button_state: MouseButtonState::Up,
..
} = event
{
log::info!("[tray] action=show_window source=left_click");
if let Err(err) = show_main_window(tray.app_handle()) {
log::error!("[tray] failed to show main window from tray click: {err}");
}
}
})
.build(app)?;
log::info!("[tray] tray icon ready");
Ok(())
}
pub fn run() {
let daemon_mode = is_daemon_mode();
let default_filter = std::env::var("RUST_LOG").unwrap_or_else(|_| "info".to_string());
let _ = env_logger::Builder::new()
.parse_filters(&default_filter)
.try_init();
// Runtime selection: default build uses wry (WKWebView on macOS), the
// `cef` feature swaps to Chromium Embedded Framework. The switch is at
// Builder construction only — everything downstream (plugins, commands,
// events, state, tray, deep links, child webviews) uses the shared
// tauri-runtime trait surface and does not care which backend drives it.
#[cfg(not(feature = "cef"))]
let builder = tauri::Builder::<tauri::Wry>::new();
#[cfg(feature = "cef")]
let builder = {
// Bypass macOS Keychain. Without this, every embedded service that
// touches password / cookie / encryption-key storage triggers a
// "Allow access to your keychain?" prompt — WhatsApp Web hits it on
// every cold start, Chromium's own component-update store also does.
// `use-mock-keychain` swaps the Keychain backend for an in-process
// mock; `password-store=basic` is the equivalent for the password
// manager. Both are no-ops on Windows/Linux, so safe to always set.
//
// In debug builds we additionally expose the Chrome DevTools
// Protocol on localhost:9222 so every CEF webview can be inspected
// from a regular browser (right-click "Inspect" does not propagate
// to CEF child webviews on macOS). Release builds intentionally do
// NOT open the CDP port — it would let any process on the machine
// drive the embedded WhatsApp/Slack/etc. webviews.
//
// NOTE: flags must be prefixed with `--`. The runtime's
// `on_before_command_line_processing` dispatch (in
// `tauri-runtime-cef/src/cef_impl.rs`) routes value-less args that
// don't start with `-` to `append_argument` (positional) instead of
// `append_switch`, which means Chromium silently ignores them.
let mut args: Vec<(&str, Option<&str>)> = vec![
("--use-mock-keychain", None),
("--password-store", Some("basic")),
];
if cfg!(debug_assertions) {
args.push(("--remote-debugging-port", Some("9222")));
}
tauri::Builder::<tauri::Cef>::new().command_line_args::<&str, &str>(args)
};
let builder = builder
.plugin(tauri_plugin_opener::init())
.plugin(tauri_plugin_deep_link::init())
.plugin(tauri_plugin_notification::init())
.plugin(tauri_plugin_global_shortcut::Builder::new().build())
.manage(DictationHotkeyState(Mutex::new(Vec::new())))
.manage(webview_accounts::WebviewAccountsState::default());
#[cfg(feature = "cef")]
let builder = builder.manage(whatsapp_scanner::ScannerRegistry::new());
#[cfg(feature = "cef")]
let builder = builder.manage(slack_scanner::ScannerRegistry::new());
#[cfg(feature = "cef")]
let builder = builder.manage(discord_scanner::ScannerRegistry::new());
builder
.setup(move |app| {
#[cfg(any(windows, target_os = "linux"))]
{
if let Err(err) = app.deep_link().register_all() {
log::warn!("[deep-link] register_all failed (non-fatal): {err}");
}
}
let core_run_mode = core_process::default_core_run_mode(daemon_mode);
let core_bin = if matches!(core_run_mode, core_process::CoreRunMode::ChildProcess) {
core_process::default_core_bin()
} else {
None
};
let core_handle = core_process::CoreProcessHandle::new(
core_process::default_core_port(),
core_bin,
core_run_mode,
);
std::env::set_var("OPENHUMAN_CORE_RPC_URL", core_handle.rpc_url());
app.manage(core_handle.clone());
let app_handle_for_update = app.handle().clone();
tauri::async_runtime::spawn(async move {
if let Err(err) = core_handle.ensure_running().await {
log::error!("[core] failed to start core process: {err}");
return;
}
log::info!("[core] core process ready");
// Check if the running core is outdated and auto-update if needed.
let update_handle = core_handle.clone();
if let Err(err) = core_update::check_and_update_core(
update_handle,
Some(app_handle_for_update),
false,
)
.await
{
log::warn!("[core-update] auto-update check failed (non-fatal): {err}");
}
});
if daemon_mode {
if let Some(window) = app.get_webview_window("main") {
let _ = window.hide();
log::info!("[tray] daemon_mode=true window_hidden_on_startup");
}
}
#[cfg(target_os = "macos")]
{
if let Some(window) = app.get_webview_window("overlay") {
configure_overlay_window_macos(&window);
} else {
log::warn!("[overlay] overlay window not found during setup");
}
}
if let Some(window) = app.get_webview_window("overlay") {
pin_overlay_bottom_right(&window);
if let Err(err) = window.show() {
log::warn!("[overlay] failed to show overlay on startup: {err}");
} else {
log::info!("[overlay] overlay shown on startup");
}
}
if let Err(err) = setup_tray(app.handle()) {
log::error!("[tray] failed to setup tray icon: {err}");
}
// Dev convenience: if OPENHUMAN_DEV_AUTO_WHATSAPP=<account-id>
// is set, spawn that account's webview at startup so the
// CDP/IndexedDB scanner can iterate without manual UI clicks.
// The same account-id reuses the persistent data dir, so a
// previously-logged-in WhatsApp session stays logged in.
if let Ok(account_id) = std::env::var("OPENHUMAN_DEV_AUTO_WHATSAPP") {
let account_id = account_id.trim().to_string();
if !account_id.is_empty() {
let app_handle = app.handle().clone();
tauri::async_runtime::spawn(async move {
// Wait for the window to be fully ready.
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
let state = app_handle.state::<webview_accounts::WebviewAccountsState>();
let args = webview_accounts::OpenArgs {
account_id: account_id.clone(),
provider: "whatsapp".to_string(),
url: None,
bounds: Some(webview_accounts::Bounds {
x: 100.0,
y: 100.0,
width: 900.0,
height: 700.0,
}),
};
match webview_accounts::webview_account_open(
app_handle.clone(),
state,
args,
)
.await
{
Ok(label) => log::info!(
"[dev-auto-whatsapp] spawned label={} account={}",
label,
account_id
),
Err(e) => log::error!(
"[dev-auto-whatsapp] failed: {} (account={})",
e,
account_id
),
}
});
}
}
// Same dev helper, Slack flavour. OPENHUMAN_DEV_AUTO_SLACK=<uuid>
// opens the Slack account webview on startup so the CDP scanner
// can iterate without manual UI clicks.
if let Ok(account_id) = std::env::var("OPENHUMAN_DEV_AUTO_SLACK") {
let account_id = account_id.trim().to_string();
if !account_id.is_empty() {
let app_handle = app.handle().clone();
tauri::async_runtime::spawn(async move {
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
let state = app_handle.state::<webview_accounts::WebviewAccountsState>();
let args = webview_accounts::OpenArgs {
account_id: account_id.clone(),
provider: "slack".to_string(),
url: None,
bounds: Some(webview_accounts::Bounds {
x: 100.0,
y: 100.0,
width: 900.0,
height: 700.0,
}),
};
match webview_accounts::webview_account_open(
app_handle.clone(),
state,
args,
)
.await
{
Ok(label) => log::info!(
"[dev-auto-slack] spawned label={} account={}",
label,
account_id
),
Err(e) => log::error!(
"[dev-auto-slack] failed: {} (account={})",
e,
account_id
),
}
});
}
}
// Same dev helper, Google Meet flavour.
// OPENHUMAN_DEV_AUTO_GOOGLE_MEET=<uuid> opens the gmeet account
// webview at startup so the caption-capture recipe runs
// without manual UI clicks. Use in combination with:
// tail -F /tmp/oh-cef.log | grep -E --line-buffered \
// "\[gmeet\]|memory_doc_ingest|orchestrator"
// to verify captions flow → transcript persist → thread handoff.
if let Ok(account_id) = std::env::var("OPENHUMAN_DEV_AUTO_GOOGLE_MEET") {
let account_id = account_id.trim().to_string();
if !account_id.is_empty() {
let app_handle = app.handle().clone();
tauri::async_runtime::spawn(async move {
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
let state = app_handle.state::<webview_accounts::WebviewAccountsState>();
// Dev mode: size the child webview to the parent
// window's inner bounds so Meet controls (CC toggle,
// mic/cam, leave) are reachable without overflowing.
let (w, h) = app_handle
.get_webview_window("main")
.and_then(|main| {
let scale = main.scale_factor().unwrap_or(1.0);
main.inner_size()
.ok()
.map(|s| ((s.width as f64) / scale, (s.height as f64) / scale))
})
.unwrap_or((1100.0, 780.0));
let args = webview_accounts::OpenArgs {
account_id: account_id.clone(),
provider: "google-meet".to_string(),
url: None,
bounds: Some(webview_accounts::Bounds {
x: 0.0,
y: 0.0,
width: w,
height: h,
}),
};
match webview_accounts::webview_account_open(
app_handle.clone(),
state,
args,
)
.await
{
Ok(label) => log::info!(
"[dev-auto-gmeet] spawned label={} account={}",
label,
account_id
),
Err(e) => log::error!(
"[dev-auto-gmeet] failed: {} (account={})",
e,
account_id
),
}
});
}
}
Ok(())
})
.invoke_handler(tauri::generate_handler![
core_rpc_url,
overlay_parent_rpc_url,
check_core_update,
apply_core_update,
restart_core_process,
service_install_direct,
service_start_direct,
service_stop_direct,
service_status_direct,
service_uninstall_direct,
register_dictation_hotkey,
unregister_dictation_hotkey,
webview_accounts::webview_account_open,
webview_accounts::webview_account_close,
webview_accounts::webview_account_purge,
webview_accounts::webview_account_bounds,
webview_accounts::webview_account_hide,
webview_accounts::webview_account_show,
webview_accounts::webview_recipe_event,
webview_accounts::webview_account_eval,
activate_main_window
])
.build(tauri::generate_context!())
.expect("error while building tauri application")
.run(move |app_handle, event| match event {
#[cfg(target_os = "macos")]
RunEvent::WindowEvent {
label,
event: WindowEvent::CloseRequested { api, .. },
..
} if label == "main" => {
log::info!(
"[window] close requested on main window — hiding instead of destroying"
);
api.prevent_close();
if let Some(window) = app_handle.get_webview_window("main") {
let _ = window.hide();
}
}
#[cfg(target_os = "macos")]
RunEvent::Reopen { .. } => {
log::info!("[window] reopen event — showing main window");
if let Err(err) = show_main_window(app_handle) {
log::error!("[macos] failed to show main window on reopen: {err}");
}
}
RunEvent::Exit => {
if let Some(core) = app_handle.try_state::<core_process::CoreProcessHandle>() {
let core = core.inner().clone();
tauri::async_runtime::block_on(async move {
core.shutdown().await;
});
}
}
_ => {}
});
}
pub fn run_core_from_args(args: &[String]) -> Result<(), String> {
let core_bin = crate::core_process::default_core_bin()
.ok_or_else(|| "openhuman-core binary not found".to_string())?;
let status = std::process::Command::new(core_bin)
.args(args)
.status()
.map_err(|e| format!("failed to execute core binary: {e}"))?;
if !status.success() {
return Err(format!("core binary exited with status {status}"));
}
Ok(())
}