feat: consolidate overlay into desktop app and add compact orb demo (#450)

* feat(overlay): implement overlay window functionality and related RPC integration

- Added a new OverlayApp component to handle overlay-specific UI and functionality.
- Introduced an overlay window configuration in tauri.conf.json, allowing for a transparent, always-on-top overlay.
- Implemented parent RPC communication for the overlay to interact with the main application.
- Updated main.tsx to conditionally render the OverlayApp based on the current window context.
- Enhanced CSS styles to support the overlay's visual requirements.

This commit establishes the foundation for overlay functionality, improving user experience with a dedicated interface for specific tasks.

* refactor(overlay): remove overlay functionality and related configurations

- Deleted the overlay module and its associated files, including process management and configuration settings.
- Removed environment variable checks and overlay-related logic from the core process and configuration schema.
- Updated documentation to reflect the removal of overlay features, simplifying the codebase and improving maintainability.

This commit streamlines the application by eliminating unused overlay components, enhancing overall performance.

* feat(overlay): enhance overlay bubble functionality and styling

- Added a new CSS animation for overlay bubble appearance, improving visual feedback.
- Introduced an OverlayBubble interface to manage bubble properties such as tone and text.
- Updated OverlayApp component to include a new OverlayBubbleChip for displaying messages with dynamic styling based on tone.
- Adjusted overlay window dimensions in tauri.conf.json for a more compact design.

This commit improves the user experience by providing visually distinct overlay messages and a refined interface.

* feat(rotating-tetrahedron): add inverted color support and refactor canvas component

- Introduced an optional `inverted` prop to the `RotatingTetrahedronCanvas` component, allowing for dynamic color changes based on the prop value.
- Updated fill and edge materials to reflect the inverted state, enhancing visual customization.
- Refactored the component to improve readability and maintainability by utilizing the new prop in the rendering logic.
- Adjusted the effect dependencies to include the `inverted` prop for proper reactivity.

This commit enhances the user experience by providing a more flexible and visually appealing tetrahedron display.

* fix(rotating-tetrahedron): adjust opacity and emissive intensity for inverted colors

- Updated the opacity of the fill material in the RotatingTetrahedronCanvas component to enhance visual clarity when the inverted prop is true.
- Reduced the emissive intensity for the inverted state to improve the overall appearance of the tetrahedron.

This commit refines the visual representation of the rotating tetrahedron, ensuring better contrast and aesthetics based on user preferences.

* feat(rotating-tetrahedron): enhance dynamic color handling and performance

- Implemented useRef hooks for fill and edge materials in the RotatingTetrahedronCanvas component to optimize rendering performance.
- Updated the useEffect hook to adjust material properties based on the inverted state, improving visual consistency.
- Refactored animation speed handling to utilize a reference for smoother updates.
- Cleaned up resource management by ensuring materials are disposed of correctly when the component unmounts.

This commit enhances the visual fidelity and performance of the rotating tetrahedron, providing a more responsive and visually appealing experience.

* fix(overlay): adjust bubble alignment and overlay positioning

- Changed the text alignment of the OverlayBubbleChip component from left to right for improved readability.
- Updated the vertical positioning logic in the Tauri overlay to account for a right margin, ensuring consistent placement of the overlay window.

These adjustments enhance the visual presentation and positioning of overlay elements, contributing to a better user experience.

* fix(overlay): update overlay dimensions and bubble styling

- Adjusted the overlay dimensions for a more refined appearance.
- Modified the bubble tone classes for improved color consistency and readability.
- Enhanced the text size and line height in the OverlayBubbleChip component for better visual clarity.
- Updated the orb button size and styling to enhance user interaction.

These changes contribute to a more polished and user-friendly overlay experience.

* feat(overlay): enhance overlay scenario handling and text display

- Introduced a new scenario management system in the OverlayApp component, allowing for dynamic cycling between different overlay states.
- Added a new text display feature for scenario two, providing real-time feedback as the text is typed out.
- Refactored the bubble rendering logic to accommodate the new scenario structure, improving the overall user interaction experience.

These changes enhance the functionality and interactivity of the overlay, making it more engaging for users.

* fix(overlay): reorder demo scenarios

* fix(overlay): update overlay dimensions and improve text display logic

- Adjusted overlay dimensions for better visual consistency.
- Enhanced text display in OverlayBubbleChip to show text progressively based on bubble content.
- Refactored the handling of scenario text in OverlayApp to streamline the display logic.

These changes contribute to a more polished and engaging user experience in the overlay.

* refactor(overlay): simplify type parameters in RPC functions

- Removed unnecessary generic type parameter from `unwrapCliCompatibleJson` and `callParentCoreRpc` functions for improved clarity and conciseness.
- These changes enhance code readability and maintainability without altering functionality.
This commit is contained in:
Steven Enamakel
2026-04-09 15:01:14 -07:00
committed by GitHub
parent 0b23ae6b96
commit d66ee0d4de
18 changed files with 546 additions and 271 deletions
+66
View File
@@ -8,6 +8,7 @@ version = "0.52.0"
dependencies = [
"env_logger",
"log",
"objc2-app-kit",
"reqwest 0.12.28",
"semver",
"serde",
@@ -2441,8 +2442,38 @@ checksum = "d49e936b501e5c5bf01fda3a9452ff86dc3ea98ad5f283e1455153142d97518c"
dependencies = [
"bitflags 2.11.0",
"block2",
"libc",
"objc2",
"objc2-cloud-kit",
"objc2-core-data",
"objc2-core-foundation",
"objc2-core-graphics",
"objc2-core-image",
"objc2-core-text",
"objc2-core-video",
"objc2-foundation",
"objc2-quartz-core",
]
[[package]]
name = "objc2-cloud-kit"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "73ad74d880bb43877038da939b7427bba67e9dd42004a18b809ba7d87cee241c"
dependencies = [
"bitflags 2.11.0",
"objc2",
"objc2-foundation",
]
[[package]]
name = "objc2-core-data"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b402a653efbb5e82ce4df10683b6b28027616a2715e90009947d50b8dd298fa"
dependencies = [
"bitflags 2.11.0",
"objc2",
"objc2-foundation",
]
@@ -2470,6 +2501,41 @@ dependencies = [
"objc2-io-surface",
]
[[package]]
name = "objc2-core-image"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5d563b38d2b97209f8e861173de434bd0214cf020e3423a52624cd1d989f006"
dependencies = [
"objc2",
"objc2-foundation",
]
[[package]]
name = "objc2-core-text"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0cde0dfb48d25d2b4862161a4d5fcc0e3c24367869ad306b0c9ec0073bfed92d"
dependencies = [
"bitflags 2.11.0",
"objc2",
"objc2-core-foundation",
"objc2-core-graphics",
]
[[package]]
name = "objc2-core-video"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d425caf1df73233f29fd8a5c3e5edbc30d2d4307870f802d18f00d83dc5141a6"
dependencies = [
"bitflags 2.11.0",
"objc2",
"objc2-core-foundation",
"objc2-core-graphics",
"objc2-io-surface",
]
[[package]]
name = "objc2-encode"
version = "4.1.0"
+2
View File
@@ -40,6 +40,8 @@ semver = "1"
log = "0.4"
env_logger = "0.11"
[target.'cfg(target_os = "macos")'.dependencies]
objc2-app-kit = "0.3.2"
[features]
# This feature is used for production builds or when a dev server is not specified, DO NOT REMOVE!!
+9 -10
View File
@@ -1,22 +1,21 @@
{
"$schema": "../gen/schemas/desktop-schema.json",
"identifier": "default",
"description": "Capability for the main window (desktop only)",
"platforms": [
"linux",
"macOS",
"windows"
],
"windows": [
"main"
],
"description": "Capability for the main and overlay windows (desktop only)",
"platforms": ["linux", "macOS", "windows"],
"windows": ["main", "overlay"],
"permissions": [
"core:default",
"core:window:default",
"core:window:allow-hide",
"core:window:allow-show",
"core:window:allow-set-focus",
"core:window:allow-unminimize",
"core:window:allow-start-dragging",
"core:window:allow-set-always-on-top",
"core:event:default",
"deep-link:default",
"opener:default",
"allow-core-process"
]
}
}
+81 -1
View File
@@ -9,13 +9,16 @@ use std::sync::Mutex;
use tauri::{
menu::{Menu, MenuItem},
tray::{MouseButton, MouseButtonState, TrayIconBuilder, TrayIconEvent},
AppHandle, Emitter, Manager, RunEvent,
AppHandle, Emitter, Manager, PhysicalPosition, RunEvent, WebviewWindow,
};
use tauri_plugin_global_shortcut::{GlobalShortcutExt, ShortcutState};
#[cfg(any(windows, target_os = "linux"))]
use tauri_plugin_deep_link::DeepLinkExt;
#[cfg(target_os = "macos")]
use objc2_app_kit::{NSStatusWindowLevel, NSWindow, NSWindowCollectionBehavior};
/// Tracks the currently registered dictation hotkey string so we can unregister it later.
struct DictationHotkeyState(Mutex<Vec<String>>);
@@ -53,6 +56,64 @@ fn core_rpc_url() -> String {
.unwrap_or_else(|_| "http://127.0.0.1:7788/rpc".to_string())
}
#[tauri::command]
fn overlay_parent_rpc_url() -> Option<String> {
let url = std::env::var("OPENHUMAN_CORE_RPC_URL").ok()?;
let trimmed = url.trim();
if trimmed.is_empty() {
return None;
}
Some(trimmed.to_string())
}
fn pin_overlay_bottom_right(window: &WebviewWindow) {
let Ok(Some(monitor)) = window.current_monitor() else {
log::warn!("[overlay] could not resolve current monitor for positioning");
return;
};
let Ok(size) = window.outer_size() else {
log::warn!("[overlay] could not resolve overlay size for positioning");
return;
};
let margin = 20i32;
let x = monitor.position().x + monitor.size().width as i32 - size.width as i32 - margin;
let y = monitor.position().y + monitor.size().height as i32 - size.height as i32 - margin;
if let Err(err) = window.set_position(PhysicalPosition::new(x, y)) {
log::warn!("[overlay] failed to pin overlay bottom-right: {err}");
} else {
log::info!("[overlay] pinned overlay bottom-right at {},{}", x, y);
}
}
#[cfg(target_os = "macos")]
fn configure_overlay_window_macos(window: &WebviewWindow) {
if let Err(err) = window.set_always_on_top(true) {
log::warn!("[overlay] failed to set always-on-top: {err}");
}
if let Err(err) = window.set_visible_on_all_workspaces(true) {
log::warn!("[overlay] failed to set visible-on-all-workspaces: {err}");
}
match window.ns_window() {
Ok(ns_window) => unsafe {
let window: &NSWindow = &*ns_window.cast();
let mut behavior = window.collectionBehavior();
behavior.insert(NSWindowCollectionBehavior::FullScreenAuxiliary);
behavior.insert(NSWindowCollectionBehavior::CanJoinAllSpaces);
window.setCollectionBehavior(behavior);
window.setLevel(NSStatusWindowLevel);
log::info!(
"[overlay] macOS overlay configured for all spaces/fullscreen auxiliary at status level"
);
},
Err(err) => {
log::warn!("[overlay] failed to access native NSWindow handle: {err}");
}
}
}
/// Resolve the core binary, preferring the staged sidecar.
fn resolve_core_bin() -> Result<std::path::PathBuf, String> {
if let Some(bin) = core_process::default_core_bin() {
@@ -410,6 +471,24 @@ pub fn run() {
}
}
#[cfg(target_os = "macos")]
{
if let Some(window) = app.get_webview_window("overlay") {
configure_overlay_window_macos(&window);
} else {
log::warn!("[overlay] overlay window not found during setup");
}
}
if let Some(window) = app.get_webview_window("overlay") {
pin_overlay_bottom_right(&window);
if let Err(err) = window.show() {
log::warn!("[overlay] failed to show overlay on startup: {err}");
} else {
log::info!("[overlay] overlay shown on startup");
}
}
if let Err(err) = setup_tray(app.handle()) {
log::error!("[tray] failed to setup tray icon: {err}");
}
@@ -418,6 +497,7 @@ pub fn run() {
})
.invoke_handler(tauri::generate_handler![
core_rpc_url,
overlay_parent_rpc_url,
check_core_update,
apply_core_update,
restart_core_process,
+21 -27
View File
@@ -20,23 +20,29 @@
"decorations": true,
"resizable": true,
"center": true
},
{
"label": "overlay",
"title": "OpenHuman Overlay",
"width": 248,
"height": 228,
"minWidth": 248,
"minHeight": 228,
"transparent": true,
"decorations": false,
"alwaysOnTop": true,
"skipTaskbar": true,
"resizable": false,
"visible": true,
"center": false
}
],
"security": {
"csp": null
},
"security": { "csp": null },
"macOSPrivateApi": true
},
"bundle": {
"active": true,
"targets": [
"app",
"dmg",
"deb",
"nsis",
"msi",
"appimage"
],
"targets": ["app", "dmg", "deb", "nsis", "msi", "appimage"],
"icon": [
"icons/32x32.png",
"icons/128x128.png",
@@ -44,19 +50,13 @@
"icons/icon.icns",
"icons/icon.ico"
],
"resources": [
"../../src/openhuman/agent/prompts"
],
"externalBin": [
"binaries/openhuman-core"
],
"resources": ["../../src/openhuman/agent/prompts"],
"externalBin": ["binaries/openhuman-core"],
"createUpdaterArtifacts": false,
"macOS": {
"minimumSystemVersion": "10.15",
"entitlements": "entitlements.sidecar.plist",
"dmg": {
"background": "./images/background-dmg.png"
}
"dmg": { "background": "./images/background-dmg.png" }
}
},
"plugins": {
@@ -67,12 +67,6 @@
"https://github.com/tinyhumansai/openhuman/releases/latest/download/latest.json"
]
},
"deep-link": {
"desktop": {
"schemes": [
"openhuman"
]
}
}
"deep-link": { "desktop": { "schemes": ["openhuman"] } }
}
}
@@ -4,6 +4,10 @@ import * as THREE from 'three';
import { useEffect, useRef, useState } from 'react';
import { ConvexGeometry } from 'three/addons/geometries/ConvexGeometry.js';
interface RotatingTetrahedronCanvasProps {
inverted?: boolean;
}
/** Start from a regular tetrahedron and lightly truncate each corner to create small blunted edges. */
function bluntedTetrahedronPoints(scale: number, bluntness = 0.12): THREE.Vector3[] {
const tetra = [
@@ -25,10 +29,34 @@ function bluntedTetrahedronPoints(scale: number, bluntness = 0.12): THREE.Vector
return points;
}
export default function RotatingTetrahedronCanvas() {
export default function RotatingTetrahedronCanvas({
inverted = false,
}: RotatingTetrahedronCanvasProps) {
const canvasRef = useRef<HTMLCanvasElement | null>(null);
const fillMaterialRef = useRef<THREE.MeshLambertMaterial | null>(null);
const edgeMaterialRef = useRef<THREE.LineBasicMaterial | null>(null);
const speedRef = useRef(2);
const [webglFailed, setWebglFailed] = useState(false);
useEffect(() => {
const fillMaterial = fillMaterialRef.current;
const edgeMaterial = edgeMaterialRef.current;
if (!fillMaterial || !edgeMaterial) {
return;
}
fillMaterial.color.set(inverted ? '#0f172a' : '#dbeafe');
fillMaterial.opacity = inverted ? 0.1 : 0.72;
fillMaterial.emissive.set(inverted ? '#020617' : '#334155');
fillMaterial.emissiveIntensity = inverted ? 0.4 : 0.35;
fillMaterial.needsUpdate = true;
edgeMaterial.color.set(inverted ? '#f8fafc' : '#f8fafc');
edgeMaterial.needsUpdate = true;
speedRef.current = inverted ? 20 : 2;
}, [inverted]);
useEffect(() => {
const canvas = canvasRef.current;
if (!canvas) return;
@@ -75,16 +103,18 @@ export default function RotatingTetrahedronCanvas() {
const geometry = new ConvexGeometry(bluntedTetrahedronPoints(0.98, 0.11));
const fillMaterial = new THREE.MeshLambertMaterial({
color: '#8e86e9',
color: inverted ? '#0f172a' : '#dbeafe',
transparent: true,
opacity: 0.2,
emissive: '#0c1208',
emissiveIntensity: 1,
opacity: inverted ? 0.58 : 0.72,
emissive: inverted ? '#020617' : '#334155',
emissiveIntensity: inverted ? 0.4 : 0.35,
});
fillMaterialRef.current = fillMaterial;
const fillMesh = new THREE.Mesh(geometry, fillMaterial);
const edgeGeometry = new THREE.EdgesGeometry(geometry);
const edgeMaterial = new THREE.LineBasicMaterial({ color: '#868ee9' });
const edgeMaterial = new THREE.LineBasicMaterial({ color: inverted ? '#020617' : '#f8fafc' });
edgeMaterialRef.current = edgeMaterial;
const edges = new THREE.LineSegments(edgeGeometry, edgeMaterial);
fillMesh.rotation.x = 0.35;
@@ -118,8 +148,9 @@ export default function RotatingTetrahedronCanvas() {
if (canvas.parentElement) observer.observe(canvas.parentElement);
resize();
const speed = 2;
speedRef.current = inverted ? 7 : 2;
const animate = () => {
const speed = speedRef.current;
fillMesh.rotation.y += 0.0038 * speed;
fillMesh.rotation.x += 0.0002 * speed;
edges.rotation.y += 0.0038 * speed;
@@ -138,8 +169,11 @@ export default function RotatingTetrahedronCanvas() {
geometry.dispose();
fillMaterial.dispose();
edgeMaterial.dispose();
fillMaterialRef.current = null;
edgeMaterialRef.current = null;
renderer.dispose();
};
// eslint-disable-next-line react-hooks/exhaustive-deps -- Scene created once; inverted changes handled by separate effect.
}, []);
if (webglFailed) {
+20
View File
@@ -41,6 +41,26 @@
bottom: 0;
}
html[data-window='overlay'],
html[data-window='overlay'] body,
html[data-window='overlay'] #root {
background: transparent;
overflow: hidden;
user-select: none;
}
@keyframes overlay-bubble-in {
from {
opacity: 0;
transform: translateY(10px) scale(0.92);
}
to {
opacity: 1;
transform: translateY(0) scale(1);
}
}
* {
color: inherit;
}
+23 -13
View File
@@ -1,4 +1,6 @@
// IMPORTANT: Polyfills must be imported FIRST
import { isTauri as tauriRuntimeAvailable } from '@tauri-apps/api/core';
import { getCurrentWindow } from '@tauri-apps/api/window';
import React from 'react';
import ReactDOM from 'react-dom/client';
@@ -6,6 +8,7 @@ import App from './App';
import ErrorReportNotification from './components/ErrorReportNotification';
import './index.css';
import { getCoreStateSnapshot } from './lib/coreState/store';
import OverlayApp from './overlay/OverlayApp';
import './polyfills';
import { initSentry } from './services/analytics';
import { setStoreForApiClient } from './services/apiClient';
@@ -13,6 +16,9 @@ import { setupDesktopDeepLinkListener } from './utils/desktopDeepLinkListener';
setStoreForApiClient(() => getCoreStateSnapshot().snapshot.sessionToken);
const currentWindowLabel = tauriRuntimeAvailable() ? getCurrentWindow().label : 'main';
const isOverlayWindow = currentWindowLabel === 'overlay';
const ensureDefaultHashRoute = () => {
const hash = window.location.hash;
if (!hash || hash === '#') {
@@ -26,21 +32,25 @@ const ensureDefaultHashRoute = () => {
// Initialize Sentry early (before React renders)
initSentry();
ensureDefaultHashRoute();
document.documentElement.dataset.window = currentWindowLabel;
// Deep link listener — try/catch handles non-Tauri environments
setupDesktopDeepLinkListener().catch(err => {
console.error('[DeepLink] setup error:', err);
});
if (!isOverlayWindow) {
ensureDefaultHashRoute();
// Deep link listener — try/catch handles non-Tauri environments
setupDesktopDeepLinkListener().catch(err => {
console.error('[DeepLink] setup error:', err);
});
}
ReactDOM.createRoot(document.getElementById('root') as HTMLElement).render(
<React.StrictMode>
<App />
</React.StrictMode>
<React.StrictMode>{isOverlayWindow ? <OverlayApp /> : <App />}</React.StrictMode>
);
// Mount error notification in an isolated React root so it survives App crashes
const errorRoot = document.createElement('div');
errorRoot.id = 'error-report-root';
document.body.appendChild(errorRoot);
ReactDOM.createRoot(errorRoot).render(<ErrorReportNotification />);
if (!isOverlayWindow) {
// Mount error notification in an isolated React root so it survives App crashes.
const errorRoot = document.createElement('div');
errorRoot.id = 'error-report-root';
document.body.appendChild(errorRoot);
ReactDOM.createRoot(errorRoot).render(<ErrorReportNotification />);
}
+156
View File
@@ -0,0 +1,156 @@
import { getCurrentWindow, LogicalSize } from '@tauri-apps/api/window';
import { useEffect, useMemo, useRef, useState } from 'react';
import RotatingTetrahedronCanvas from '../components/RotatingTetrahedronCanvas';
const OVERLAY_WIDTH = 248;
const OVERLAY_HEIGHT = 228;
const SCENARIO_THREE_TEXT = '"Noted. Need milk."';
type OverlayStatus = 'idle' | 'active';
type OverlayScenario = 1 | 2 | 3;
interface OverlayBubble {
id: string;
text: string;
tone: 'neutral' | 'accent' | 'success';
compact?: boolean;
}
function bubbleToneClass(tone: OverlayBubble['tone']) {
switch (tone) {
case 'accent':
return 'bg-blue-700 text-white';
case 'success':
return 'bg-emerald-500 text-emerald-950';
default:
return 'bg-slate-700 text-white';
}
}
function OverlayBubbleChip({ bubble }: { bubble: OverlayBubble }) {
const [displayedText, setDisplayedText] = useState('');
const indexRef = useRef(0);
useEffect(() => {
if (!bubble.text) {
return () => {
indexRef.current = 0;
setDisplayedText('');
};
}
const timeoutId = window.setInterval(
() => {
indexRef.current += 1;
setDisplayedText(bubble.text.slice(0, indexRef.current));
if (indexRef.current >= bubble.text.length) {
window.clearInterval(timeoutId);
}
},
bubble.compact ? 28 : 32
);
return () => {
window.clearInterval(timeoutId);
indexRef.current = 0;
setDisplayedText('');
};
}, [bubble.compact, bubble.id, bubble.text]);
return (
<div
className={`max-w-[184px] rounded-[18px] px-3 py-2 text-right transition-all duration-200 ${bubbleToneClass(bubble.tone)} ${bubble.compact ? 'text-[12px] leading-[1.35]' : 'text-[13px] leading-[1.45]'}`}>
{displayedText || ' '}
</div>
);
}
export default function OverlayApp() {
const [scenario, setScenario] = useState<OverlayScenario>(1);
useEffect(() => {
const appWindow = getCurrentWindow();
const size = new LogicalSize(OVERLAY_WIDTH, OVERLAY_HEIGHT);
void appWindow.setSize(size).catch(error => {
console.warn('[overlay] failed to resize overlay window', error);
});
void appWindow.setMinSize(size).catch(error => {
console.warn('[overlay] failed to set overlay min size', error);
});
void appWindow.setMaxSize(size).catch(error => {
console.warn('[overlay] failed to set overlay max size', error);
});
}, []);
useEffect(() => {
const timeoutId = window.setTimeout(() => {
setScenario(current => {
if (current === 1) return 2;
if (current === 2) return 3;
return 1;
});
}, 5000);
return () => {
window.clearTimeout(timeoutId);
};
}, [scenario]);
const status: OverlayStatus = scenario === 1 ? 'idle' : 'active';
const bubbles = useMemo<OverlayBubble[]>(() => {
if (scenario === 1) {
return [];
}
if (scenario === 2) {
return [
{
id: 'assistant',
text: '"Hey I think your coffee is getting cold. Want me to get you a new one?"',
tone: 'accent',
},
];
}
return [{ id: 'stt', text: SCENARIO_THREE_TEXT, tone: 'accent' }];
}, [scenario]);
const orbClassName = useMemo(() => {
if (status === 'active') {
return 'border-blue-950 bg-blue-700';
}
return 'border-slate-950 bg-slate-800';
}, [status]);
const tetrahedronInverted = status === 'active';
return (
<div className="flex h-screen w-screen items-end justify-end bg-transparent px-0 py-0">
<div className="relative flex select-none flex-col items-end gap-3">
<div className="flex max-w-[190px] flex-col items-end gap-2">
{bubbles.map(bubble => (
<div key={bubble.id} className="animate-[overlay-bubble-in_220ms_ease-out]">
<OverlayBubbleChip bubble={bubble} />
</div>
))}
</div>
<div className="relative">
<button
type="button"
aria-label="Activate overlay orb"
onClick={() => {
setScenario(2);
}}
className={`group relative flex h-[56px] w-[56px] cursor-pointer items-center justify-center overflow-hidden rounded-full border transition-all duration-200 ${orbClassName}`}
title="Click to start the demo.">
<div className="pointer-events-none h-[92%] w-[92%] opacity-95 transition-transform duration-300 group-hover:scale-105">
<RotatingTetrahedronCanvas inverted={tetrahedronInverted} />
</div>
</button>
</div>
</div>
</div>
);
}
+118
View File
@@ -0,0 +1,118 @@
/**
* HTTP JSON-RPC to the desktop core sidecar shared by the main app and overlay.
* Mirrors the naming normalization in app/src/services/coreRpcClient.ts (subset).
*/
let nextJsonRpcId = 1;
export const normalizeLegacyMethod = (method: string): string => {
if (method.startsWith('openhuman.accessibility_')) {
return method.replace('openhuman.accessibility_', 'openhuman.screen_intelligence_');
}
return method;
};
/** RpcOutcome with non-empty logs serializes as `{ result, logs }` in the core. */
const unwrapCliCompatibleJson = <T>(raw: unknown): T => {
if (
raw !== null &&
typeof raw === 'object' &&
Object.prototype.hasOwnProperty.call(raw, 'result') &&
Object.prototype.hasOwnProperty.call(raw, 'logs')
) {
const keys = Object.keys(raw);
const { logs } = raw as { logs: unknown };
if (keys.length === 2 && Array.isArray(logs)) {
return (raw as { result: T }).result;
}
}
return raw as T;
};
interface JsonRpcError {
code: number;
message: string;
data?: unknown;
}
interface JsonRpcResponse<T> {
jsonrpc?: string;
id?: number | string | null;
result?: T;
error?: JsonRpcError;
}
const DEFAULT_RPC_TIMEOUT_MS = 10_000;
const isJsonRpcEnvelope = (value: unknown): value is JsonRpcResponse<unknown> => {
if (value === null || typeof value !== 'object') {
return false;
}
return (
Object.prototype.hasOwnProperty.call(value, 'error') ||
Object.prototype.hasOwnProperty.call(value, 'result') ||
Object.prototype.hasOwnProperty.call(value, 'id') ||
Object.prototype.hasOwnProperty.call(value, 'jsonrpc')
);
};
export const callParentCoreRpc = async <T>(
rpcUrl: string,
method: string,
params: Record<string, unknown> = {},
timeoutMs: number = DEFAULT_RPC_TIMEOUT_MS
): Promise<T> => {
const normalizedMethod = normalizeLegacyMethod(method);
const payload = {
jsonrpc: '2.0' as const,
id: nextJsonRpcId++,
method: normalizedMethod,
params,
};
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
let response: Response;
try {
response = await fetch(rpcUrl, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload),
signal: controller.signal,
});
} catch (err) {
clearTimeout(timer);
if (
(err instanceof DOMException && err.name === 'AbortError') ||
(err instanceof Error && err.name === 'AbortError')
) {
throw new Error(
`Core RPC request timed out after ${timeoutMs}ms (method: ${normalizedMethod})`
);
}
throw err;
} finally {
clearTimeout(timer);
}
if (!response.ok) {
const text = await response.text();
throw new Error(`Core RPC HTTP ${response.status}: ${text || response.statusText}`);
}
const json = await response.json();
if (!isJsonRpcEnvelope(json)) {
throw new Error('Invalid JSON-RPC envelope');
}
if (json.error) {
throw new Error(json.error.message || 'Core RPC returned an error');
}
if (!Object.prototype.hasOwnProperty.call(json, 'result')) {
throw new Error('Core RPC response missing result');
}
return unwrapCliCompatibleJson<T>(json.result);
};
+4 -1
View File
@@ -104,7 +104,10 @@ class ApiClient {
}
// Handle abort/timeout specifically
if (error instanceof DOMException && error.name === 'AbortError') {
if (
(error instanceof DOMException && error.name === 'AbortError') ||
(error instanceof Error && error.name === 'AbortError')
) {
throw { success: false, error: `Request timed out after ${timeout / 1000}s` } as ApiError;
}
+5 -36
View File
@@ -574,8 +574,7 @@ pub async fn run_server(
run_server_inner(host, port, socketio_enabled, false).await
}
/// Like [`run_server`] but marks the instance as embedded — the overlay will
/// **not** be auto-spawned, preventing recursive overlay launches.
/// Like [`run_server`] but marks the instance as embedded.
pub async fn run_server_embedded(
host: Option<&str>,
port: Option<u16>,
@@ -585,9 +584,6 @@ pub async fn run_server_embedded(
}
/// Internal server entrypoint.
///
/// When `embedded_core` is `true` the server skips auto-spawning the overlay
/// (useful when the overlay itself hosts an embedded core to avoid recursion).
async fn run_server_inner(
host: Option<&str>,
port: Option<u16>,
@@ -654,30 +650,7 @@ async fn run_server_inner(
log::info!("[rpc:socketio] disabled (--jsonrpc-only)");
}
// Derive the real bound address from the listener so ephemeral port 0,
// wildcard binds, and IPv6 are handled correctly.
let bound_addr = listener.local_addr()?;
let parent_core_rpc_url = {
let (h, p) = (bound_addr.ip(), bound_addr.port());
let effective_ip = if h.is_unspecified() {
if h.is_ipv6() {
std::net::IpAddr::V6(std::net::Ipv6Addr::LOCALHOST)
} else {
std::net::IpAddr::V4(std::net::Ipv4Addr::LOCALHOST)
}
} else {
h
};
if effective_ip.is_ipv6() {
format!("http://[{effective_ip}]:{p}/rpc")
} else {
format!("http://{effective_ip}:{p}/rpc")
}
};
log::debug!("[core] parent_core_rpc_url = {parent_core_rpc_url}");
// Optional background bootstrap for local AI services.
let is_embedded = embedded_core;
tokio::spawn(async move {
match crate::openhuman::config::Config::load_or_init().await {
Ok(config) => {
@@ -686,14 +659,10 @@ async fn run_server_inner(
service.bootstrap(&config).await;
}
// Launch the overlay Tauri app (transparent debug/voice panel) as a child process.
// Skip when running as an embedded core inside the overlay itself.
if is_embedded {
log::info!("[overlay] embedded core — skipping overlay auto-spawn");
} else if config.overlay_enabled {
crate::openhuman::overlay::spawn_overlay(parent_core_rpc_url.as_str());
if embedded_core {
log::debug!("[core] embedded core startup");
} else {
log::info!("[overlay] overlay disabled by config (overlay_enabled = false)");
log::debug!("[core] desktop core startup");
}
// Start the voice server (records + transcribes) and/or the
@@ -758,7 +727,7 @@ async fn run_server_inner(
}
}
Err(err) => {
log::warn!("[core] config load failed, skipping local-ai and overlay: {err}");
log::warn!("[core] config load failed, skipping local-ai startup: {err}");
}
}
});
-9
View File
@@ -840,15 +840,6 @@ impl Config {
}
}
if let Ok(flag) = std::env::var("OPENHUMAN_OVERLAY_ENABLED") {
let normalized = flag.trim().to_ascii_lowercase();
match normalized.as_str() {
"1" | "true" | "yes" | "on" => self.overlay_enabled = true,
"0" | "false" | "no" | "off" => self.overlay_enabled = false,
_ => {}
}
}
if self.proxy.enabled && self.proxy.scope == ProxyScope::Environment {
self.proxy.apply_to_process_env();
}
-1
View File
@@ -8,7 +8,6 @@ mod autocomplete;
mod autonomy;
mod channels;
mod defaults;
pub(crate) use defaults::default_true;
mod dictation;
mod hardware;
mod heartbeat_cron;
-6
View File
@@ -130,11 +130,6 @@ pub struct Config {
#[serde(default)]
pub dictation: DictationConfig,
/// Whether to launch the overlay Tauri app (floating debug/voice panel)
/// when the core RPC server starts. Defaults to `true`.
#[serde(default = "default_true")]
pub overlay_enabled: bool,
/// Whether the user has completed the onboarding flow.
#[serde(default)]
pub onboarding_completed: bool,
@@ -188,7 +183,6 @@ impl Default for Config {
orchestrator: OrchestratorConfig::default(),
update: UpdateConfig::default(),
dictation: DictationConfig::default(),
overlay_enabled: true,
onboarding_completed: false,
}
}
-1
View File
@@ -37,7 +37,6 @@ pub mod learning;
pub mod local_ai;
pub mod memory;
pub mod migration;
pub mod overlay;
pub mod providers;
pub mod referral;
pub mod screen_intelligence;
-7
View File
@@ -1,7 +0,0 @@
//! Overlay process launcher — discovers and spawns the `openhuman-overlay`
//! Tauri application as a child process so the floating debug/voice panel
//! appears automatically when the core RPC server is running.
mod process;
pub use process::spawn_overlay;
-152
View File
@@ -1,152 +0,0 @@
//! Discovery and lifecycle management for the `openhuman-overlay` binary.
//!
//! The overlay is a separate Tauri application that provides a transparent
//! floating panel with voice transcription, autocomplete debug info, and
//! Globe/Fn hotkey toggling. The core process launches it as a fire-and-forget
//! child so that it appears automatically when the core RPC server starts.
use std::path::PathBuf;
use std::process::{Command, Stdio};
/// Attempt to find and spawn the overlay binary.
///
/// `parent_core_rpc_url` must be the JSON-RPC URL of the **same** core process
/// that spawned the overlay (e.g. `http://127.0.0.1:7788/rpc`) so the overlay
/// UI can query autocomplete, screen intelligence, and voice state from the
/// desktop sidecar instead of a separate in-process core instance.
///
/// This is best-effort: if the binary is not found or fails to launch, a
/// warning is logged and the core continues normally.
pub fn spawn_overlay(parent_core_rpc_url: &str) {
let Some(overlay_bin) = find_overlay_binary() else {
log::warn!(
"[overlay] openhuman-overlay binary not found — skipping overlay launch (set OPENHUMAN_OVERLAY_BIN or build overlay/src-tauri)"
);
return;
};
log::info!(
"[overlay] launching overlay: {} (parent RPC {})",
overlay_bin.display(),
parent_core_rpc_url
);
let mut cmd = Command::new(&overlay_bin);
cmd.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.env(
"OPENHUMAN_OVERLAY_PARENT_RPC_URL",
parent_core_rpc_url.trim(),
)
// The desktop core sets OPENHUMAN_CORE_PORT; the overlay must not inherit it or its
// optional embedded JSON-RPC server will try to bind the same port as the parent.
.env_remove("OPENHUMAN_CORE_PORT");
match cmd.spawn() {
Ok(child) => {
log::info!("[overlay] overlay process spawned (pid={})", child.id());
}
Err(err) => {
log::warn!(
"[overlay] failed to spawn overlay at {}: {err}",
overlay_bin.display()
);
}
}
}
/// Search for the `openhuman-overlay` binary in standard locations.
///
/// Search order:
/// 1. `OPENHUMAN_OVERLAY_BIN` env var (explicit override)
/// 2. Next to the current executable (`openhuman-overlay` / `openhuman-overlay.exe`)
/// 3. macOS: inside the `.app` bundle Resources directory
/// 4. Dev builds: `overlay/src-tauri/target/debug/openhuman-overlay`
fn find_overlay_binary() -> Option<PathBuf> {
// 1. Explicit env var override
if let Ok(path) = std::env::var("OPENHUMAN_OVERLAY_BIN") {
let candidate = PathBuf::from(&path);
if candidate.exists() {
log::debug!(
"[overlay] found via OPENHUMAN_OVERLAY_BIN: {}",
candidate.display()
);
return Some(candidate);
}
log::debug!("[overlay] OPENHUMAN_OVERLAY_BIN set but path does not exist: {path}");
}
let exe = std::env::current_exe().ok()?;
let exe_dir = exe.parent()?;
// 2. Next to the current executable
let standalone = exe_dir.join(overlay_binary_name());
if standalone.exists() {
log::debug!("[overlay] found next to exe: {}", standalone.display());
return Some(standalone);
}
// 3. macOS: Resources directory inside the .app bundle
#[cfg(target_os = "macos")]
{
if let Some(resources_dir) = exe_dir.parent().map(|p| p.join("Resources")) {
let in_resources = resources_dir.join(overlay_binary_name());
if in_resources.exists() {
log::debug!("[overlay] found in Resources: {}", in_resources.display());
return Some(in_resources);
}
}
}
// 4. Dev builds: walk up from current exe to find overlay/src-tauri/target
if cfg!(debug_assertions) {
// Try relative to CARGO_MANIFEST_DIR at compile time
let dev_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("overlay")
.join("src-tauri")
.join("target")
.join("debug")
.join(overlay_binary_name());
if dev_path.exists() {
log::debug!("[overlay] found dev build: {}", dev_path.display());
return Some(dev_path);
}
// Also check for the macOS .app bundle in dev
#[cfg(target_os = "macos")]
{
let dev_app_path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("overlay")
.join("src-tauri")
.join("target")
.join("debug")
.join("bundle")
.join("macos")
.join("openhuman-overlay.app")
.join("Contents")
.join("MacOS")
.join("openhuman-overlay");
if dev_app_path.exists() {
log::debug!(
"[overlay] found dev .app bundle: {}",
dev_app_path.display()
);
return Some(dev_app_path);
}
}
}
None
}
fn overlay_binary_name() -> &'static str {
#[cfg(windows)]
{
"openhuman-overlay.exe"
}
#[cfg(not(windows))]
{
"openhuman-overlay"
}
}