diff --git a/.github/workflows/ci-lite.yml b/.github/workflows/ci-lite.yml index 4d65bf31d..cdb5f5636 100644 --- a/.github/workflows/ci-lite.yml +++ b/.github/workflows/ci-lite.yml @@ -443,11 +443,9 @@ jobs: set -euo pipefail EXPECTED=$(cat <<'EOF' core/all_tests.rs - core/autocomplete_cli_adapter.rs core/cli_tests.rs core/jsonrpc_tests.rs core/legacy_aliases.rs - core/runtime/context.rs core/runtime/services.rs openhuman/agent/harness/builtin_definitions.rs openhuman/agent/harness/definition_tests.rs @@ -465,7 +463,7 @@ jobs: openhuman/x402/stub.rs EOF ) - ACTUAL=$(grep -rlE '#\[cfg\((not\()?feature = "(voice|media|web3|meet|mcp|skills|flows|channels|desktop-automation)"' src --include='*.rs' \ + ACTUAL=$(grep -rlE '#\[cfg\((not\()?feature = "(voice|media|web3|meet|mcp|skills|flows|channels)"' src --include='*.rs' \ | xargs grep -lE '#\[test\]|#\[tokio::test\]|fn .*_test' 2>/dev/null | sed 's|^src/||' | sort -u) if ! diff <(echo "$EXPECTED" | sed 's/^ *//' | sort -u) <(echo "$ACTUAL"); then echo "::error::Gated-test file set changed. Update the EXPECTED allowlist in the rust-feature-gate-smoke lane, and extend the scoped 'cargo test' filter if the new module can carry an ungated-assert regression (see #5022)." diff --git a/AGENTS.md b/AGENTS.md index c29e73a17..98e54bc3b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -177,7 +177,7 @@ Embedded provider webviews **must not** grow new JS injection. No new `.js` unde ### Domain layout (`src/openhuman/`) -~130 domain directories — authoritative list: `ls -d src/openhuman/*/`. Major families: agent (`agent`, `agent_experience`, `agent_meetings`, `agent_memory`, `agent_orchestration`, `agent_registry`, `agent_tool_policy`, `agentbox`, `orchestration`), memory (`memory`, `memory_archivist`, `memory_conversations`, `memory_diff`, `memory_goals`, `memory_queue`, `memory_search`, `memory_sources`, `memory_store`, `memory_sync`, `memory_tools`, `memory_tree`, `tinycortex`), skills/flows (`skills`, `skill_registry`, `skill_runtime`, `flows`, `tinyflows`, `tinyagents`, `rhai_workflows`), inference/AI (`inference`, `model_council`, `council_registry`, `embeddings`, `routing`), MCP (`mcp_audit`, `mcp_client`, `mcp_registry`, `mcp_server`), runtimes (`runtime_node`, `runtime_python`, `runtime_python_server`, `javascript`, `sandbox`, `cwd_jail`), channels/webviews (`channels`, `webview_accounts`, `webview_apis`, `webview_notifications`, `whatsapp_data`), meet (`meet`, `meet_agent`), web3 (`wallet`, `web3`, `x402`, `tokenjuice`), plus platform domains (`about_app`, `approval`, `config`, `cron`, `credentials`, `keyring`, `security`, `threads`, `tools`, `update`, `voice`, …). +~130 domain directories — authoritative list: `ls -d src/openhuman/*/`. Major families: agent (`agent`, `agent_experience`, `agent_meetings`, `agent_memory`, `agent_orchestration`, `agent_registry`, `agent_tool_policy`, `agentbox`, `orchestration`), memory (`memory`, `memory_archivist`, `memory_conversations`, `memory_diff`, `memory_goals`, `memory_queue`, `memory_search`, `memory_sources`, `memory_store`, `memory_sync`, `memory_tools`, `memory_tree`, `tinycortex`), skills/flows (`skills`, `skill_registry`, `skill_runtime`, `flows`, `tinyflows`, `tinyagents`, `rhai_workflows`), inference/AI (`inference`, `embeddings`, `routing`), MCP (`mcp_audit`, `mcp_client`, `mcp_registry`, `mcp_server`), runtimes (`runtime_node`, `runtime_python`, `runtime_python_server`, `javascript`, `sandbox`, `cwd_jail`), channels/webviews (`channels`, `whatsapp_data`), meet (`meet`, `meet_agent`), web3 (`wallet`, `web3`, `x402`, `tokenjuice`), plus platform domains (`about_app`, `approval`, `config`, `cron`, `credentials`, `keyring`, `security`, `threads`, `tools`, `update`, `voice`, …). **Skills runtime**: the QuickJS per-skill VM engine is gone. `src/openhuman/skills/` holds skill metadata/tool descriptors; execution of installed `SKILL.md` workflows lives in `src/openhuman/skill_runtime/` (starts/cancels runs, hosts the `skill_executor` agent, reuses `runtime_node`/`runtime_python`). @@ -256,12 +256,11 @@ GGML_NATIVE=OFF cargo check --manifest-path Cargo.toml \ **`meet` gate (#4800)** — uses all three module patterns, one per domain, chosen by the rule *"does always-compiled code reach a non-registration symbol in here?"*: - **`meet` → leaf-gate.** `#[cfg(feature = "meet")] pub mod meet;` outright. Its only outside reference is the registration site in `src/core/all.rs`, so no facade is needed (same shape as `audio_toolkit`). -- **`meet_agent` → facade + carve-out, no stub file.** Every submodule is gated **except `wav`** (below). Nothing outside the Meet domain calls the gated submodules. +- **`meet_agent` → leaf-gate.** Every submodule (including `wav`) is `#[cfg(feature = "meet")]`. Nothing outside the Meet domain calls into it. (`wav` was formerly ungated for the always-on `desktop_companion` STT path; that domain was removed, so the carve-out is gone and `wav` gates with the rest.) - **`agent_meetings` → facade + stub.** Three always-compiled call sites reach in — the heartbeat planner (`calendar::handle_calendar_meeting_candidate`) and two subscriber registrations (`core::jsonrpc`, `channels::runtime::startup`) — so `src/openhuman/agent_meetings/stub.rs` supplies no-op equivalents and those callers need no `#[cfg]`. **No deps to shed (do not re-litigate).** Unlike `voice`, this gate drops **zero** dependencies — the Meet domains have no exclusive crates. `meet_agent::wav` is a hand-rolled 79-line RIFF writer with no `use` statements, written precisely so Meet never needed `hound` (which `voice` already owns and sheds). The dependency shed was pre-paid; this gate's value is compile-time surface and binary size, not the dep tree. -**⚠ The `wav` carve-out is load-bearing.** `meet_agent::wav::pack_pcm16le_mono_wav` is called by `desktop_companion::pipeline::stt`, which is `DomainGroup::Platform` and compiled in **every** build. `pub mod wav;` must stay **ungated** so that call site keeps its real implementation — it costs nothing, since `wav.rs` is dependency-free. If someone gates it, the `--no-default-features` build fails loudly at `desktop_companion`; that failure is correct. Do **not** "fix" it by stubbing `pack_pcm16le_mono_wav` — that trades a compile error for green CI while silently corrupting desktop-companion STT (the STT backend would receive a malformed WAV). Revert the cfg instead. **Both-ways tests.** `src/core/all_tests.rs` pins the gate in both directions (`meet_controllers_registered_when_feature_on` / `meet_controllers_absent_when_feature_off`). The negative half is the one that proves the gate removes anything. Note CI's smoke lane runs `cargo check` only and never compiles test code, so a disabled-build **test** break is invisible to it — run `cargo test --lib --no-default-features --features tokenjuice-treesitter core::all::tests` locally after touching any gated surface. diff --git a/Cargo.lock b/Cargo.lock index bfdee024d..fc449256e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4763,7 +4763,6 @@ dependencies = [ "tracing-appender", "tracing-log", "tracing-subscriber", - "uiautomation", "unicode-width", "url", "urlencoding", @@ -8010,29 +8009,6 @@ version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971" -[[package]] -name = "uiautomation" -version = "0.25.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c68495a701b9f2f21f29353ac446f0d27dd0d7ce97aa9ccf9061bca0446cd744" -dependencies = [ - "chrono", - "uiautomation_derive", - "windows 0.62.2", - "windows-core 0.62.2", -] - -[[package]] -name = "uiautomation_derive" -version = "0.7.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffcc4d404aa1c03a848f95cf5feadc3e63946d7f095bf388770b85550093d388" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "uint" version = "0.9.5" @@ -8886,27 +8862,6 @@ dependencies = [ "windows-targets 0.52.6", ] -[[package]] -name = "windows" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580" -dependencies = [ - "windows-collections", - "windows-core 0.62.2", - "windows-future", - "windows-numerics", -] - -[[package]] -name = "windows-collections" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "23b2d95af1a8a14a3c7367e1ed4fc9c20e0a26e79551b1454d72583c97cc6610" -dependencies = [ - "windows-core 0.62.2", -] - [[package]] name = "windows-core" version = "0.54.0" @@ -8942,30 +8897,6 @@ dependencies = [ "windows-targets 0.52.6", ] -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement 0.60.2", - "windows-interface 0.59.3", - "windows-link", - "windows-result 0.4.1", - "windows-strings 0.5.1", -] - -[[package]] -name = "windows-future" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb" -dependencies = [ - "windows-core 0.62.2", - "windows-link", - "windows-threading", -] - [[package]] name = "windows-implement" version = "0.57.0" @@ -8988,17 +8919,6 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "windows-implement" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "windows-interface" version = "0.57.0" @@ -9021,33 +8941,12 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "windows-interface" -version = "0.59.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "windows-link" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" -[[package]] -name = "windows-numerics" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26" -dependencies = [ - "windows-core 0.62.2", - "windows-link", -] - [[package]] name = "windows-registry" version = "0.6.1" @@ -9213,15 +9112,6 @@ dependencies = [ "windows_x86_64_msvc 0.53.1", ] -[[package]] -name = "windows-threading" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3949bd5b99cafdf1c7ca86b43ca564028dfe27d66958f2470940f73d86d75b37" -dependencies = [ - "windows-link", -] - [[package]] name = "windows_aarch64_gnullvm" version = "0.42.2" diff --git a/Cargo.toml b/Cargo.toml index 17575b1c3..bc27da454 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -318,23 +318,16 @@ reqwest = { version = "0.12", default-features = false, features = ["native-tls" # AppContainer / process-jail backend in `openhuman::cwd_jail`. # Feature list mirrors the Win32 surface used by cwd_jail/windows.rs: # AppContainer profile APIs, ACL editing, STARTUPINFOEXW process spawn, -# and the GENERIC_* file access masks. `Win32_System_Com` is used by the -# UIA accessibility backend (`accessibility::uia_interact`) to initialise COM -# on the worker thread before creating the UI Automation client. +# and the GENERIC_* file access masks. windows-sys = { version = "0.61", features = [ "Win32_Foundation", "Win32_Security", "Win32_Security_Authorization", "Win32_Security_Isolation", "Win32_Storage_FileSystem", - "Win32_System_Com", "Win32_System_Memory", "Win32_System_Threading", ] } -# Microsoft UI Automation (UIA) bindings — the Windows backend for the -# `ax_interact` tool (`accessibility::uia_interact`). Safe Rust wrappers over -# the UIA COM API; the Windows analogue of the macOS AXUIElement Swift helper. -uiautomation = { version = "0.25", optional = true } [target.'cfg(not(windows))'.dependencies] # macOS / Linux: keep rustls + Mozilla webpki-roots — the historical @@ -387,7 +380,7 @@ tokio = { version = "1", features = ["test-util"] } proptest = "1" [features] -default = ["tokenjuice-treesitter", "inference", "voice", "web3", "media", "documents", "meet", "skills", "flows", "mcp", "crash-reporting", "http-server", "desktop-automation", "channels", "tui", "medulla-local"] +default = ["tokenjuice-treesitter", "inference", "voice", "web3", "media", "documents", "meet", "skills", "flows", "mcp", "crash-reporting", "http-server", "channels", "tui", "medulla-local"] # HTTP + Socket.IO server transport (#5048): the `/rpc` JSON-RPC endpoint and # its auth middleware/CORS layer (`core::jsonrpc`, `core::auth`), the `/v1` # OpenAI-compatible router (`inference::http`), the ad-hoc static-dir file @@ -564,27 +557,6 @@ mcp = [] # the crash-reporting feature" error. crash-reporting = ["dep:sentry"] -# Desktop-automation cluster (#5049): the five modules that read/drive the local -# desktop UI — `openhuman::accessibility` (macOS AX / Windows UIA FFI middleware), -# `openhuman::screen_intelligence` (capture + vision loop), `openhuman::autocomplete` -# (inline completion), `openhuman::desktop_companion` (Clicky-style loop), and the -# `computer` agent-tool family (`ax_interact` / `automate` / mouse / keyboard). -# Default-ON — the desktop app always ships with these. Slim / headless builds opt -# out via `--no-default-features --features ""`, -# which drops the exclusive `uiautomation` (Windows UI Automation COM bindings) -# dependency. Composes with the runtime `DomainSet::desktop_automation` flag: the -# feature narrows the compile-time surface, `DomainSet` gates it at runtime. -# -# CARVE-OUT: the inert type modules stay compiled in BOTH builds — -# `accessibility::types` (incl. the `GlobeHotkey*` structs), `autocomplete::types` -# (`AutocompleteStatus`), and `screen_intelligence::types` (`AccessibilityStatus`, -# `CaptureImageRefResult`) — because always-on callers (`text_input`, `voice`, -# `app_state`) name them. Only behaviour is gated; the stubs re-export the one real -# type definition. See AGENTS.md "skills gate — the type carve-out". -# -# NOTE: only `uiautomation` is exclusive and thus shed. `enigo` is shared with the -# `voice` domain; `rdev` / `arboard` are voice-owned — none of those are dropped here. -desktop-automation = ["dep:uiautomation"] # Tabbed terminal UI: the `openhuman tui` (alias `chat`) CLI subcommand, a # ratatui/crossterm front-end for logs, orchestrator chat, curated configuration, # and account settings. Default-ON for the standalone `openhuman-core` binary, but diff --git a/app/scripts/e2e-run-all-flows.sh b/app/scripts/e2e-run-all-flows.sh index 2bfa50d8e..72541c2ae 100755 --- a/app/scripts/e2e-run-all-flows.sh +++ b/app/scripts/e2e-run-all-flows.sh @@ -293,7 +293,6 @@ if should_run_suite "notifications"; then run "test/e2e/specs/memory-roundtrip.spec.ts" "memory-roundtrip" "notifications" run "test/e2e/specs/coding-session-memory.spec.ts" "coding-session-memory" "notifications" run "test/e2e/specs/cron-jobs-flow.spec.ts" "cron-jobs" "notifications" - run "test/e2e/specs/autocomplete-flow.spec.ts" "autocomplete" "notifications" _mini_summary "notifications" fi diff --git a/app/src-tauri/Cargo.lock b/app/src-tauri/Cargo.lock index e1aa752e4..ee5e3291d 100644 --- a/app/src-tauri/Cargo.lock +++ b/app/src-tauri/Cargo.lock @@ -12,6 +12,7 @@ dependencies = [ "block2 0.6.2", "cef", "chrono", + "cpal", "directories 5.0.1", "env_logger", "futures-util", @@ -28,6 +29,7 @@ dependencies = [ "openhuman", "parking_lot", "rand 0.9.4", + "regex", "reqwest 0.12.28", "resvg", "rfd", @@ -53,6 +55,7 @@ dependencies = [ "tokio-util", "toml 0.8.2", "url", + "uuid 1.23.1", "windows-sys 0.59.0", ] @@ -5619,7 +5622,6 @@ dependencies = [ "tracing-appender", "tracing-log", "tracing-subscriber", - "uiautomation", "url", "urlencoding", "uuid 1.23.1", @@ -9656,29 +9658,6 @@ dependencies = [ "windows-sys 0.60.2", ] -[[package]] -name = "uiautomation" -version = "0.25.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c68495a701b9f2f21f29353ac446f0d27dd0d7ce97aa9ccf9061bca0446cd744" -dependencies = [ - "chrono", - "uiautomation_derive", - "windows 0.62.2", - "windows-core 0.62.2", -] - -[[package]] -name = "uiautomation_derive" -version = "0.7.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffcc4d404aa1c03a848f95cf5feadc3e63946d7f095bf388770b85550093d388" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "uint" version = "0.9.5" @@ -10509,23 +10488,11 @@ version = "0.61.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9babd3a767a4c1aef6900409f85f5d53ce2544ccdfaa86dad48c91782c6d6893" dependencies = [ - "windows-collections 0.2.0", + "windows-collections", "windows-core 0.61.2", - "windows-future 0.2.1", + "windows-future", "windows-link 0.1.3", - "windows-numerics 0.2.0", -] - -[[package]] -name = "windows" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580" -dependencies = [ - "windows-collections 0.3.2", - "windows-core 0.62.2", - "windows-future 0.3.2", - "windows-numerics 0.3.1", + "windows-numerics", ] [[package]] @@ -10537,15 +10504,6 @@ dependencies = [ "windows-core 0.61.2", ] -[[package]] -name = "windows-collections" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "23b2d95af1a8a14a3c7367e1ed4fc9c20e0a26e79551b1454d72583c97cc6610" -dependencies = [ - "windows-core 0.62.2", -] - [[package]] name = "windows-core" version = "0.54.0" @@ -10594,19 +10552,6 @@ dependencies = [ "windows-strings 0.4.2", ] -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement 0.60.2", - "windows-interface 0.59.3", - "windows-link 0.2.1", - "windows-result 0.4.1", - "windows-strings 0.5.1", -] - [[package]] name = "windows-future" version = "0.2.1" @@ -10615,18 +10560,7 @@ checksum = "fc6a41e98427b19fe4b73c550f060b59fa592d7d686537eebf9385621bfbad8e" dependencies = [ "windows-core 0.61.2", "windows-link 0.1.3", - "windows-threading 0.1.0", -] - -[[package]] -name = "windows-future" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb" -dependencies = [ - "windows-core 0.62.2", - "windows-link 0.2.1", - "windows-threading 0.2.1", + "windows-threading", ] [[package]] @@ -10717,16 +10651,6 @@ dependencies = [ "windows-link 0.1.3", ] -[[package]] -name = "windows-numerics" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26" -dependencies = [ - "windows-core 0.62.2", - "windows-link 0.2.1", -] - [[package]] name = "windows-registry" version = "0.5.3" @@ -10765,15 +10689,6 @@ dependencies = [ "windows-link 0.1.3", ] -[[package]] -name = "windows-result" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" -dependencies = [ - "windows-link 0.2.1", -] - [[package]] name = "windows-strings" version = "0.1.0" @@ -10793,15 +10708,6 @@ dependencies = [ "windows-link 0.1.3", ] -[[package]] -name = "windows-strings" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" -dependencies = [ - "windows-link 0.2.1", -] - [[package]] name = "windows-sys" version = "0.45.0" @@ -10928,15 +10834,6 @@ dependencies = [ "windows-link 0.1.3", ] -[[package]] -name = "windows-threading" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3949bd5b99cafdf1c7ca86b43ca564028dfe27d66958f2470940f73d86d75b37" -dependencies = [ - "windows-link 0.2.1", -] - [[package]] name = "windows-version" version = "0.1.7" diff --git a/app/src-tauri/Cargo.toml b/app/src-tauri/Cargo.toml index 9bee1ad1a..b10ee436e 100644 --- a/app/src-tauri/Cargo.toml +++ b/app/src-tauri/Cargo.toml @@ -107,9 +107,18 @@ sentry = { version = "0.47.0", default-features = false, features = ["backtrace" # Used by the imessage_scanner module. anyhow = "1.0" +# SQLite for the shell-side whatsapp_data store (relocated from the core). All +# platforms — the store persists on Windows/Linux/macOS. Pinned to match the +# core crate so a single bundled SQLite is linked. (The macOS-only iMessage +# scanner also uses rusqlite; this general dep covers it too.) +rusqlite = { version = "=0.40.0", features = ["bundled"] } parking_lot = "0.12" chrono = "0.4" async-trait = "0.1" +# Desktop companion: native mic capture + POINT-tag parsing + session ids. +cpal = "0.15" +regex = "1" +uuid = { version = "1", features = ["v4"] } # Mascot fake-camera pipeline (meet_call): rasterizes the OpenHuman # mascot SVG to a PNG once, converts it to a YUV420 Y4M frame, and # points CEF's `--use-file-for-fake-video-capture` flag at the cached @@ -177,7 +186,6 @@ openhuman_core = { path = "../..", package = "openhuman", default-features = fal # http://127.0.0.1:/rpc, so it REQUIRES the HTTP + Socket.IO transport # (#5048). Enforced by the HTTP_SERVER_COMPILED_IN compile assert in lib.rs. "http-server", - "desktop-automation", "medulla-local", ] } tinyjuice = { version = "0.2.1", default-features = false } @@ -189,8 +197,8 @@ nix = { version = "0.29", default-features = false, features = ["hostname", "sig objc2 = "0.6" objc2-app-kit = "0.3.2" mac-notification-sys = "0.6" -# iMessage scanner reads ~/Library/Messages/chat.db read-only on macOS. -rusqlite = { version = "=0.40.0", features = ["bundled"] } +# iMessage scanner reads ~/Library/Messages/chat.db read-only on macOS +# (rusqlite is declared once in the general [dependencies] table above). objc2-user-notifications = "0.3.2" block2 = "0.6.2" objc2-foundation = { version = "0.3.2", features = ["NSTimer", "block2"] } diff --git a/app/src-tauri/permissions/allow-core-process.toml b/app/src-tauri/permissions/allow-core-process.toml index 055f17403..396701560 100644 --- a/app/src-tauri/permissions/allow-core-process.toml +++ b/app/src-tauri/permissions/allow-core-process.toml @@ -73,6 +73,30 @@ allow = [ "unregister_ptt_hotkey", "show_ptt_overlay", + # ========================= + # DESKTOP COMPANION + # ========================= + # The desktop companion moved from core RPC into the Tauri shell. Its + # settings panel and hotkey bridge invoke these commands directly, so each + # one must be present in the main-window capability. + "register_companion_hotkey", + "unregister_companion_hotkey", + "companion_activate", + "companion_start_session", + "companion_stop_session", + "companion_status", + "companion_config_get", + "companion_config_set", + + # ========================= + # STRUCTURED WHATSAPP DATA + # ========================= + # The SQLite store moved from the core into the Tauri shell. Intelligence + # views now query it through these read-only invoke commands. + "whatsapp_data_list_chats", + "whatsapp_data_list_messages", + "whatsapp_data_search_messages", + # ========================= # ACCOUNT WEBVIEW # ========================= diff --git a/app/src-tauri/src/companion/audio.rs b/app/src-tauri/src/companion/audio.rs new file mode 100644 index 000000000..518e30611 --- /dev/null +++ b/app/src-tauri/src/companion/audio.rs @@ -0,0 +1,303 @@ +//! Native microphone capture for the desktop companion (shell-side). +//! +//! This is the "shell captures audio natively" half of the migration: while a +//! companion session is Listening we open the default input device with `cpal`, +//! downmix to mono `i16`, and resample to ~16 kHz PCM for upload to the core STT +//! endpoint (`openhuman.voice_transcribe_bytes`). We deliberately do **not** +//! depend on the core voice listener for capture. +//! +//! `cpal::Stream` is `!Send` on several platforms, so the stream is built and +//! owned by a dedicated std thread; the audio callback accumulates samples into +//! a shared buffer and the thread holds the stream alive until `stop()`. + +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::mpsc; +use std::sync::{Arc, Mutex}; +use std::thread::JoinHandle; + +use cpal::traits::{DeviceTrait, HostTrait, StreamTrait}; +use log::{debug, info, warn}; + +const LOG_PREFIX: &str = "[companion_audio]"; + +/// Target sample rate for STT upload. Whisper works at 16 kHz mono. +pub const TARGET_SAMPLE_RATE: u32 = 16_000; + +/// A live microphone capture. Dropping without `stop()` still tears the stream +/// down (via the `running` flag + thread join in `Drop`). +pub struct MicCapture { + running: Arc, + buffer: Arc>>, + device_rate: u32, + handle: Option>, +} + +impl MicCapture { + /// Open the default input device and begin capturing mono `i16` at the + /// device's native rate. Returns once the stream is playing (or an error). + pub fn start() -> Result { + let running = Arc::new(AtomicBool::new(true)); + let buffer: Arc>> = Arc::new(Mutex::new(Vec::new())); + + let (ready_tx, ready_rx) = mpsc::channel::>(); + let thread_running = running.clone(); + let thread_buffer = buffer.clone(); + + let handle = std::thread::Builder::new() + .name("companion-mic".into()) + .spawn(move || { + run_capture_thread(thread_running, thread_buffer, ready_tx); + }) + .map_err(|e| format!("failed to spawn mic thread: {e}"))?; + + let device_rate = match ready_rx.recv() { + Ok(Ok(rate)) => rate, + Ok(Err(e)) => { + running.store(false, Ordering::SeqCst); + let _ = handle.join(); + return Err(e); + } + Err(_) => { + running.store(false, Ordering::SeqCst); + let _ = handle.join(); + return Err("mic capture thread exited before signalling ready".into()); + } + }; + + info!("{LOG_PREFIX} capture started device_rate={device_rate}"); + Ok(Self { + running, + buffer, + device_rate, + handle: Some(handle), + }) + } + + /// Stop capture and return the collected PCM resampled to + /// [`TARGET_SAMPLE_RATE`] (mono `i16`). + pub fn stop(mut self) -> (Vec, u32) { + self.running.store(false, Ordering::SeqCst); + if let Some(handle) = self.handle.take() { + let _ = handle.join(); + } + let raw = std::mem::take(&mut *self.buffer.lock().unwrap()); + debug!( + "{LOG_PREFIX} capture stopped raw_samples={} device_rate={}", + raw.len(), + self.device_rate + ); + let resampled = resample_linear(&raw, self.device_rate, TARGET_SAMPLE_RATE); + (resampled, TARGET_SAMPLE_RATE) + } +} + +impl Drop for MicCapture { + fn drop(&mut self) { + self.running.store(false, Ordering::SeqCst); + if let Some(handle) = self.handle.take() { + let _ = handle.join(); + } + } +} + +/// Body of the capture thread: build the input stream, signal readiness with the +/// device sample rate, then keep the stream alive until told to stop. +fn run_capture_thread( + running: Arc, + buffer: Arc>>, + ready_tx: mpsc::Sender>, +) { + let host = cpal::default_host(); + let device = match host.default_input_device() { + Some(d) => d, + None => { + let _ = ready_tx.send(Err("no default input device".into())); + return; + } + }; + + let config = match device.default_input_config() { + Ok(c) => c, + Err(e) => { + let _ = ready_tx.send(Err(format!("no default input config: {e}"))); + return; + } + }; + + let device_rate = config.sample_rate().0; + let channels = config.channels() as usize; + let sample_format = config.sample_format(); + let stream_config: cpal::StreamConfig = config.into(); + + let err_fn = |e| warn!("{LOG_PREFIX} input stream error: {e}"); + + let build = |buf: Arc>>| -> Result { + match sample_format { + cpal::SampleFormat::F32 => device.build_input_stream( + &stream_config, + move |data: &[f32], _| { + let mut guard = buf.lock().unwrap(); + for frame in data.chunks(channels.max(1)) { + let sum: f32 = frame.iter().copied().sum(); + let mono = sum / channels.max(1) as f32; + guard.push((mono.clamp(-1.0, 1.0) * i16::MAX as f32) as i16); + } + }, + err_fn, + None, + ), + cpal::SampleFormat::I16 => device.build_input_stream( + &stream_config, + move |data: &[i16], _| { + let mut guard = buf.lock().unwrap(); + for frame in data.chunks(channels.max(1)) { + let sum: i32 = frame.iter().map(|&s| s as i32).sum(); + guard.push((sum / channels.max(1) as i32) as i16); + } + }, + err_fn, + None, + ), + cpal::SampleFormat::U16 => device.build_input_stream( + &stream_config, + move |data: &[u16], _| { + let mut guard = buf.lock().unwrap(); + for frame in data.chunks(channels.max(1)) { + let sum: i32 = frame.iter().map(|&s| s as i32 - 32768).sum(); + guard.push((sum / channels.max(1) as i32) as i16); + } + }, + err_fn, + None, + ), + other => Err(cpal::BuildStreamError::BackendSpecific { + err: cpal::BackendSpecificError { + description: format!("unsupported sample format: {other:?}"), + }, + }), + } + }; + + let stream = match build(buffer) { + Ok(s) => s, + Err(e) => { + let _ = ready_tx.send(Err(format!("failed to build input stream: {e}"))); + return; + } + }; + + if let Err(e) = stream.play() { + let _ = ready_tx.send(Err(format!("failed to start input stream: {e}"))); + return; + } + + let _ = ready_tx.send(Ok(device_rate)); + + // Keep the stream alive on this thread until stop() flips the flag. + while running.load(Ordering::SeqCst) { + std::thread::sleep(std::time::Duration::from_millis(50)); + } + drop(stream); + debug!("{LOG_PREFIX} capture thread exiting"); +} + +/// Naive linear-interpolation resampler for mono `i16`. Adequate for speech STT +/// upload; whisper is robust to modest resampling artefacts. +fn resample_linear(input: &[i16], from_rate: u32, to_rate: u32) -> Vec { + if input.is_empty() || from_rate == 0 || to_rate == 0 { + return Vec::new(); + } + if from_rate == to_rate { + return input.to_vec(); + } + let ratio = to_rate as f64 / from_rate as f64; + let out_len = ((input.len() as f64) * ratio).round() as usize; + let mut out = Vec::with_capacity(out_len); + for i in 0..out_len { + let src = i as f64 / ratio; + let idx = src.floor() as usize; + let frac = src - idx as f64; + let a = input[idx.min(input.len() - 1)] as f64; + let b = input[(idx + 1).min(input.len() - 1)] as f64; + out.push((a + (b - a) * frac).round() as i16); + } + out +} + +/// Pack mono `i16` little-endian PCM into a minimal RIFF/WAVE container. +/// +/// The core STT endpoint (`openhuman.voice_transcribe_bytes`) takes container +/// bytes + an `extension`, not raw PCM — and the shell must not call the +/// `meet`-gated `meet_agent::wav` packer, so we ship this tiny dependency-free +/// RIFF writer here. +pub fn pack_wav_pcm16le_mono(samples: &[i16], sample_rate: u32) -> Vec { + let num_samples = samples.len() as u32; + let bytes_per_sample = 2u32; + let byte_rate = sample_rate * bytes_per_sample; // mono + let data_len = num_samples * bytes_per_sample; + let riff_len = 36 + data_len; + + let mut out = Vec::with_capacity(44 + data_len as usize); + out.extend_from_slice(b"RIFF"); + out.extend_from_slice(&riff_len.to_le_bytes()); + out.extend_from_slice(b"WAVE"); + // fmt chunk + out.extend_from_slice(b"fmt "); + out.extend_from_slice(&16u32.to_le_bytes()); // PCM fmt chunk size + out.extend_from_slice(&1u16.to_le_bytes()); // audio format = PCM + out.extend_from_slice(&1u16.to_le_bytes()); // channels = mono + out.extend_from_slice(&sample_rate.to_le_bytes()); + out.extend_from_slice(&byte_rate.to_le_bytes()); + out.extend_from_slice(&(bytes_per_sample as u16).to_le_bytes()); // block align + out.extend_from_slice(&16u16.to_le_bytes()); // bits per sample + // data chunk + out.extend_from_slice(b"data"); + out.extend_from_slice(&data_len.to_le_bytes()); + for &s in samples { + out.extend_from_slice(&s.to_le_bytes()); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn resample_identity_when_rates_match() { + let input = vec![1i16, 2, 3, 4]; + assert_eq!(resample_linear(&input, 16000, 16000), input); + } + + #[test] + fn resample_downsamples_length() { + let input = vec![0i16; 48_000]; + let out = resample_linear(&input, 48_000, 16_000); + assert_eq!(out.len(), 16_000); + } + + #[test] + fn resample_empty_is_empty() { + assert!(resample_linear(&[], 48_000, 16_000).is_empty()); + } + + #[test] + fn wav_header_is_44_bytes_plus_data() { + let samples = vec![0i16, 1, -1, 32767, -32768]; + let wav = pack_wav_pcm16le_mono(&samples, 16_000); + assert_eq!(&wav[0..4], b"RIFF"); + assert_eq!(&wav[8..12], b"WAVE"); + assert_eq!(&wav[36..40], b"data"); + assert_eq!(wav.len(), 44 + samples.len() * 2); + // data length field matches + let data_len = u32::from_le_bytes([wav[40], wav[41], wav[42], wav[43]]); + assert_eq!(data_len as usize, samples.len() * 2); + } + + #[test] + fn wav_sample_rate_field_roundtrips() { + let wav = pack_wav_pcm16le_mono(&[0i16; 4], 16_000); + let rate = u32::from_le_bytes([wav[24], wav[25], wav[26], wav[27]]); + assert_eq!(rate, 16_000); + } +} diff --git a/app/src-tauri/src/companion/events.rs b/app/src-tauri/src/companion/events.rs new file mode 100644 index 000000000..4614b6857 --- /dev/null +++ b/app/src-tauri/src/companion/events.rs @@ -0,0 +1,82 @@ +//! Frontend event delivery for the desktop companion. +//! +//! State changes are emitted as a Tauri event `companion://state_changed` +//! with a camelCase payload (`{ sessionId, state, previousState }`) for the +//! main renderer, and forwarded through the embedded core's transport-only +//! Socket.IO seam for the native notch WKWebView. The session state machine lives +//! deep below any `#[tauri::command]` boundary (it is driven from the +//! pipeline), so we stash the `AppHandle` in a process-global `OnceLock` +//! (mirroring `cdp::in_process::set_cef_app_handle`) and read it from the +//! emit helper. + +use std::sync::OnceLock; + +use log::debug; +use tauri::{AppHandle, Emitter}; + +use super::types::{CompanionState, CompanionStateChangedEvent}; + +const LOG_PREFIX: &str = "[companion]"; + +/// Tauri event name the frontend companion slice + overlay/notch listen to. +pub const STATE_CHANGED_EVENT: &str = "companion://state_changed"; + +/// Process-global `AppHandle`, set once from `Builder::setup`. +static COMPANION_APP_HANDLE: OnceLock> = OnceLock::new(); + +#[cfg(test)] +static RECORDED_EVENTS: std::sync::Mutex> = + std::sync::Mutex::new(Vec::new()); + +/// Store the app handle so the pipeline/session can emit frontend events from +/// outside a command context. Idempotent — a second write is ignored. +pub fn set_app_handle(app: AppHandle) { + if COMPANION_APP_HANDLE.set(app).is_err() { + debug!("{LOG_PREFIX} set_app_handle called more than once — ignoring"); + } +} + +/// Emit a `companion://state_changed` event to the frontend. +/// +/// Fire-and-forget: if the handle is not yet installed (e.g. in unit tests) or +/// the emit fails, we log and move on rather than failing the turn. +pub fn emit_state_changed(session_id: &str, state: CompanionState, previous: CompanionState) { + debug!("{LOG_PREFIX} state_changed session={session_id} {previous} -> {state}"); + let payload = CompanionStateChangedEvent { + session_id: session_id.to_string(), + state, + previous_state: previous, + }; + #[cfg(test)] + RECORDED_EVENTS + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .push(payload.clone()); + let Some(app) = COMPANION_APP_HANDLE.get() else { + debug!("{LOG_PREFIX} no app handle installed — state change not delivered to frontend"); + return; + }; + if let Err(e) = app.emit(STATE_CHANGED_EVENT, payload) { + debug!("{LOG_PREFIX} emit {STATE_CHANGED_EVENT} failed: {e}"); + } + + // The native notch WKWebView has no Tauri IPC bridge; it connects directly + // to the embedded core over Socket.IO. Preserve the legacy snake_case + // payload on that transport without moving companion behavior back to core. + let socket_payload = serde_json::json!({ + "session_id": session_id, + "state": state, + "previous_state": previous, + }); + let delivered = openhuman_core::core::socketio::publish_companion_state_changed(socket_payload); + debug!("{LOG_PREFIX} queued companion state for {delivered} Socket.IO bridge subscriber(s)"); +} + +#[cfg(test)] +pub(crate) fn take_recorded_events() -> Vec { + std::mem::take( + &mut *RECORDED_EVENTS + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()), + ) +} diff --git a/app/src-tauri/src/companion/mod.rs b/app/src-tauri/src/companion/mod.rs new file mode 100644 index 000000000..e47420031 --- /dev/null +++ b/app/src-tauri/src/companion/mod.rs @@ -0,0 +1,339 @@ +//! Desktop companion — Clicky-style interaction loop (shell-side). +//! +//! Migrated out of the Rust core (`src/openhuman/desktop_companion`) into the +//! Tauri shell. Ties hotkey activation, **native** microphone capture, screen +//! context, LLM reasoning, speech synthesis, and visual pointing into one +//! product experience: +//! +//! - `audio` — native `cpal` mic capture (mono i16 PCM ~16 kHz) +//! - `session` — session lifecycle + state machine (idle→listening→…→idle) +//! - `pipeline` — one interaction turn (STT → LLM → TTS → pointing) +//! - `pointing` — `[POINT:x,y:label:screenN]` tag parsing + monitor mapping +//! - `events` — `companion://state_changed` Tauri event delivery +//! +//! STT/TTS run in the embedded core over JSON-RPC; the LLM turn runs shell-side +//! against the backend using creds fetched from core. The five Tauri commands +//! below mirror the five RPC methods the old core `desktop_companion` exposed. + +pub mod audio; +pub mod events; +pub mod pipeline; +pub mod pointing; +pub mod session; +pub mod types; + +use log::{debug, info, warn}; +use parking_lot::Mutex; +use std::sync::atomic::{AtomicU64, Ordering}; +use tauri::{AppHandle, Manager}; +use tokio_util::sync::CancellationToken; + +use crate::AppRuntime; +use audio::MicCapture; +use pointing::ScreenGeometry; +use types::*; + +const LOG_PREFIX: &str = "[companion]"; + +/// Process-global companion configuration (persisted only in-memory for now — +/// mirrors the old core stub, but round-trips get/set instead of erroring). +static CONFIG: Mutex> = Mutex::new(None); + +/// The in-flight mic capture, if the companion is currently Listening. Used to +/// implement both tap-to-talk and push-to-talk activation. +static ACTIVE_CAPTURE: Mutex> = Mutex::new(None); + +/// Cancellation handle for the currently running STT/LLM/TTS turn. +/// +/// A new capture cancels the previous turn before entering `Listening`, so +/// stale turn cleanup cannot race the interrupting utterance. The monotonic id +/// lets a completed old task avoid clearing a newer turn's token. +struct ActiveTurn { + id: u64, + cancel: CancellationToken, +} + +static ACTIVE_TURN: Mutex> = Mutex::new(None); +static NEXT_TURN_ID: AtomicU64 = AtomicU64::new(1); +static SESSION_LIFECYCLE: Mutex<()> = Mutex::new(()); + +/// Install the app handle so the session/pipeline can emit frontend events from +/// outside a command context. Call once from `Builder::setup`. +pub fn setup(app: &AppHandle) { + events::set_app_handle(app.clone()); + info!("{LOG_PREFIX} setup complete"); +} + +pub(super) fn current_config() -> CompanionConfig { + CONFIG.lock().clone().unwrap_or_default() +} + +// ── Tauri commands (mirror the 5 former core RPC methods) ────────────── + +/// Start a desktop companion session with explicit consent. +#[tauri::command] +pub(crate) async fn companion_start_session( + app: AppHandle, + consent: bool, +) -> Result { + debug!("{LOG_PREFIX} companion_start_session consent={consent}"); + let _lifecycle = SESSION_LIFECYCLE.lock(); + cleanup_expired_session(&app); + let ttl_secs = current_config().ttl_secs; + session::start_session(&StartCompanionSessionParams { + consent, + ttl_secs: Some(ttl_secs), + }) +} + +/// Stop the active desktop companion session. +#[tauri::command] +pub(crate) async fn companion_stop_session( + app: AppHandle, +) -> Result { + debug!("{LOG_PREFIX} companion_stop_session"); + let _lifecycle = SESSION_LIFECYCLE.lock(); + stop_session_resources(); + let result = session::stop_session(&StopCompanionSessionParams { + reason: Some("user_requested".into()), + })?; + crate::companion_commands::unregister_companion_hotkey_for_app(&app)?; + Ok(result) +} + +fn stop_session_resources() { + cancel_active_turn(); + if let Some(capture) = ACTIVE_CAPTURE.lock().take() { + let _ = capture.stop(); + } +} + +fn cleanup_expired_session(app: &AppHandle) { + if session::session_is_expired() { + stop_session_resources(); + let _ = session::expire_session_if_needed(); + if let Err(error) = crate::companion_commands::unregister_companion_hotkey_for_app(app) { + warn!("{LOG_PREFIX} failed to unregister expired session hotkey: {error}"); + } + } +} + +/// Get the current desktop companion session status. +#[tauri::command] +pub(crate) async fn companion_status( + app: AppHandle, +) -> Result { + let _lifecycle = SESSION_LIFECYCLE.lock(); + cleanup_expired_session(&app); + Ok(session::session_status()) +} + +/// Get the current desktop companion configuration. +#[tauri::command] +pub(crate) async fn companion_config_get() -> Result { + Ok(current_config()) +} + +/// Update the desktop companion configuration. +#[tauri::command] +pub(crate) async fn companion_config_set( + config: CompanionConfig, +) -> Result { + debug!( + "{LOG_PREFIX} companion_config_set hotkey={} mode={}", + config.hotkey, config.activation_mode + ); + *CONFIG.lock() = Some(config.clone()); + Ok(config) +} + +// ── Activation driven by the hotkey / companion_activate ────────────── + +/// Handle a global-hotkey press according to the configured activation mode. +/// +/// Push mode begins capture on press; tap mode toggles capture on each press. +pub fn handle_hotkey_pressed(app: AppHandle) { + if current_config() + .activation_mode + .eq_ignore_ascii_case("push") + { + start_capture(&app); + } else { + handle_activation(app); + } +} + +/// Handle a global-hotkey release according to the configured activation mode. +/// +/// Only push mode consumes releases. Tap mode intentionally waits for the next +/// press, preserving toggle behavior. +pub fn handle_hotkey_released(app: AppHandle) { + if current_config() + .activation_mode + .eq_ignore_ascii_case("push") + { + finish_capture_and_run(app); + } +} + +/// Handle a companion activation (hotkey press or the `companion_activate` +/// command). Tap-to-talk: the first activation while a session is active starts +/// native mic capture (Listening); the next activation stops capture and spawns +/// the interaction turn. +pub fn handle_activation(app: AppHandle) { + if ACTIVE_CAPTURE.lock().is_some() { + finish_capture_and_run(app); + } else { + start_capture(&app); + } +} + +fn start_capture(app: &AppHandle) { + let _lifecycle = SESSION_LIFECYCLE.lock(); + cleanup_expired_session(app); + let status = session::session_status(); + if !status.active { + debug!("{LOG_PREFIX} activation ignored — no active session"); + return; + } + + let mut cap = ACTIVE_CAPTURE.lock(); + if cap.is_some() { + debug!("{LOG_PREFIX} capture start ignored — already listening"); + return; + } + + // Interrupt any previous STT/LLM/TTS task before the new capture owns the + // Listening state. The pipeline observes this token between every phase. + cancel_active_turn(); + // A cancelled pre-STT turn can still own Listening after its capture was + // consumed. Retire that state while holding the capture lock so its later + // cleanup cannot race a newly-started microphone capture. + session::finish_listening_turn(); + + if let Err(e) = session::transition_state(CompanionState::Listening, None) { + warn!("{LOG_PREFIX} could not enter Listening: {e}"); + return; + } + match MicCapture::start() { + Ok(c) => { + *cap = Some(c); + info!("{LOG_PREFIX} listening — mic capture started"); + } + Err(e) => { + warn!("{LOG_PREFIX} mic capture failed to start: {e}"); + session::recover_after_turn_error(format!("mic capture failed: {e}")); + } + } +} + +fn finish_capture_and_run(app: AppHandle) { + let _lifecycle = SESSION_LIFECYCLE.lock(); + let Some(capture) = ACTIVE_CAPTURE.lock().take() else { + debug!("{LOG_PREFIX} capture stop ignored — not listening"); + return; + }; + let (samples, rate) = capture.stop(); + info!("{LOG_PREFIX} captured {} samples @ {rate}Hz", samples.len()); + let screens = monitors_geometry(&app); + let (turn_id, cancel) = register_active_turn(); + tauri::async_runtime::spawn(async move { + let result = pipeline::run_audio_turn(&samples, rate, &screens, cancel.clone()).await; + if let Err(e) = result { + if cancel.is_cancelled() { + debug!("{LOG_PREFIX} interrupted companion turn stopped: {e}"); + } else { + warn!("{LOG_PREFIX} companion turn failed: {e}"); + session::recover_after_turn_error(e); + } + } + clear_active_turn(turn_id); + }); +} + +pub(super) fn finish_listening_without_active_capture() { + let capture = ACTIVE_CAPTURE.lock(); + if capture.is_none() { + session::finish_listening_turn(); + } +} + +fn register_active_turn() -> (u64, CancellationToken) { + let id = NEXT_TURN_ID.fetch_add(1, Ordering::Relaxed); + let cancel = CancellationToken::new(); + let mut active = ACTIVE_TURN.lock(); + if let Some(previous) = active.take() { + previous.cancel.cancel(); + } + *active = Some(ActiveTurn { + id, + cancel: cancel.clone(), + }); + (id, cancel) +} + +fn cancel_active_turn() { + if let Some(active) = ACTIVE_TURN.lock().take() { + active.cancel.cancel(); + } +} + +fn clear_active_turn(id: u64) { + let mut active = ACTIVE_TURN.lock(); + if active.as_ref().map(|turn| turn.id) == Some(id) { + active.take(); + } +} + +/// Enumerate connected monitors as [`ScreenGeometry`] for POINT-tag coordinate +/// mapping. Best-effort — returns an empty vec if no window/monitor info is +/// available (pointing then falls back to raw coordinates). +fn monitors_geometry(app: &AppHandle) -> Vec { + let Some(window) = app.get_webview_window("main") else { + debug!("{LOG_PREFIX} no main window — cannot enumerate monitors"); + return Vec::new(); + }; + let Ok(monitors) = window.available_monitors() else { + return Vec::new(); + }; + monitors + .iter() + .enumerate() + .map(|(index, m)| { + let pos = m.position(); + let size = m.size(); + ScreenGeometry { + index, + x: pos.x as f64, + y: pos.y as f64, + width: size.width as f64, + height: size.height as f64, + } + }) + .collect() +} + +#[cfg(test)] +mod turn_cancellation_tests { + use super::*; + + #[test] + fn interrupt_cancels_in_flight_turn_and_old_completion_cannot_clear_new_token() { + cancel_active_turn(); + + let (old_id, old_token) = register_active_turn(); + cancel_active_turn(); + assert!(old_token.is_cancelled()); + + let (new_id, new_token) = register_active_turn(); + clear_active_turn(old_id); + assert!(!new_token.is_cancelled()); + assert_eq!( + ACTIVE_TURN.lock().as_ref().map(|turn| turn.id), + Some(new_id) + ); + + cancel_active_turn(); + assert!(new_token.is_cancelled()); + } +} diff --git a/src/openhuman/desktop_companion/pipeline.rs b/app/src-tauri/src/companion/pipeline.rs similarity index 58% rename from src/openhuman/desktop_companion/pipeline.rs rename to app/src-tauri/src/companion/pipeline.rs index 48345dcbe..7c7aafbb7 100644 --- a/src/openhuman/desktop_companion/pipeline.rs +++ b/app/src-tauri/src/companion/pipeline.rs @@ -1,38 +1,43 @@ //! Companion interaction pipeline: STT → screen context → LLM → TTS → pointing. //! -//! Orchestrates a single interaction turn for the desktop companion. Reuses -//! the same cloud STT, LLM, and TTS backends that `meet_agent::brain` uses, -//! but adds screenshot + foreground-app context and POINT-tag parsing for -//! visual pointing. +//! Orchestrates a single interaction turn for the desktop companion. Migrated +//! from the former core `desktop_companion::pipeline`, re-plumbed for the shell: +//! +//! - **STT** and **TTS** are performed by the embedded core over JSON-RPC +//! (`openhuman.voice_stt_dispatch` / `openhuman.voice_tts_dispatch`) via +//! [`crate::core_rpc::call_core_rpc`]. +//! - The **LLM turn** runs shell-side: it fetches the backend session token +//! (`openhuman.auth_get_session_token`) and effective API URL +//! (`openhuman.config_resolve_api_url`) from core, then POSTs +//! `/openai/v1/chat/completions` directly with `reqwest`. +//! - Audio is captured natively by the shell (`super::audio`) and packed into a +//! shell-local RIFF/WAVE container for upload (no dependency on the +//! `meet`-gated `meet_agent::wav`). //! //! The pipeline is cancellable via [`tokio_util::sync::CancellationToken`] so -//! the Tauri shell can interrupt mid-turn (e.g. user presses hotkey again -//! during Speaking). +//! the shell can interrupt mid-turn (e.g. user presses the hotkey again during +//! Speaking). -use base64::{engine::general_purpose::STANDARD as B64, Engine as _}; +use std::time::Duration; + +use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _}; use log::{debug, info, warn}; use serde::{Deserialize, Serialize}; use serde_json::{json, Value}; use tokio_util::sync::CancellationToken; -use super::handoff::{self, HandoffEvent}; use super::pointing::{self, PointTarget, PointingParseResult, ScreenGeometry}; use super::session; use super::types::*; const LOG_PREFIX: &str = "[companion_pipeline]"; -/// Maximum tokens for the companion LLM reply. Longer than meet_agent (220) -/// because the companion can give richer answers when not constrained to -/// live-meeting brevity. +/// Maximum tokens for the companion LLM reply. const REPLY_MAX_TOKENS: u32 = 512; /// Rolling conversation context window (number of turns). const CONTEXT_WINDOW: usize = 20; -/// ElevenLabs TTS model for companion speech. -const TTS_MODEL_ID: &str = "eleven_turbo_v2_5"; - /// Result of a single companion interaction turn. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct TurnResult { @@ -42,10 +47,8 @@ pub struct TurnResult { pub response_text: String, /// Parsed pointing targets from the LLM response. pub targets: Vec, - /// Whether TTS audio was synthesized and enqueued. + /// Whether TTS audio was synthesized. pub tts_synthesized: bool, - /// Provider-surface handoff events detected in the response. - pub handoff_events: Vec, /// Whether the turn was cancelled before completion. pub cancelled: bool, } @@ -53,8 +56,7 @@ pub struct TurnResult { /// Run a text-input companion turn (no STT needed — the user typed their query). /// /// **Precondition**: the session must already be in `Listening` state. The -/// caller (e.g. Tauri hotkey bridge or `companion_activate` RPC) is -/// responsible for the `Idle → Listening` transition before invoking this. +/// caller is responsible for the `Idle → Listening` transition before invoking. /// /// Transitions: Listening → Thinking → Speaking/Pointing → Idle. pub async fn run_text_turn( @@ -80,8 +82,12 @@ pub async fn run_text_turn( let history = session::conversation_history(); let history_window = tail_history(&history, CONTEXT_WINDOW); - // Screen context (best-effort — skip if unavailable). - let screen_context = gather_screen_context().await; + // Foreground app/window context (best-effort and user-configurable). + let screen_context = if super::current_config().include_app_context { + gather_screen_context().await + } else { + None + }; if cancel.is_cancelled() { return Ok(cancelled_result(trimmed)); @@ -91,6 +97,9 @@ pub async fn run_text_turn( let raw_reply = match llm_companion(trimmed, &history_window, screen_context.as_deref()).await { Ok(reply) => reply, Err(err) => { + if cancel.is_cancelled() { + return Ok(cancelled_result(trimmed)); + } warn!("{LOG_PREFIX} LLM failed err={err}"); session::transition_state(CompanionState::Error, Some(format!("LLM failure: {err}")))?; return Err(format!("LLM failure: {err}")); @@ -113,12 +122,6 @@ pub async fn run_text_turn( targets.len() ); - // Check for provider-surface handoff opportunities. - let handoff_events = handoff::check_handoff(&clean_text); - if !handoff_events.is_empty() { - debug!("{LOG_PREFIX} handoff events={}", handoff_events.len()); - } - // Record conversation turns. let now_ms = chrono::Utc::now().timestamp_millis(); let _ = session::push_conversation_turn(ConversationTurn { @@ -138,8 +141,8 @@ pub async fn run_text_turn( let tts_ok = if !clean_text.is_empty() && !cancel.is_cancelled() { session::transition_state(CompanionState::Speaking, None)?; match tts(&clean_text).await { - Ok(_samples) => { - debug!("{LOG_PREFIX} TTS synthesized samples={}", _samples.len()); + Ok(()) => { + debug!("{LOG_PREFIX} TTS synthesized"); true } Err(err) => { @@ -161,14 +164,13 @@ pub async fn run_text_turn( } // Back to idle. - let _ = session::transition_state(CompanionState::Idle, None); + session::finish_turn(); let result = TurnResult { transcript: trimmed.to_string(), response_text: clean_text, targets, tts_synthesized: tts_ok, - handoff_events, cancelled: false, }; @@ -178,9 +180,7 @@ pub async fn run_text_turn( /// Run a full audio-input companion turn: STT → screen context → LLM → TTS → pointing. /// -/// **Precondition**: the session must already be in `Listening` state. The -/// caller (e.g. Tauri hotkey bridge or `companion_activate` RPC) is -/// responsible for the `Idle → Listening` transition before invoking this. +/// **Precondition**: the session must already be in `Listening` state. /// /// Transitions: Listening → Thinking → Speaking/Pointing → Idle. pub async fn run_audio_turn( @@ -190,7 +190,12 @@ pub async fn run_audio_turn( cancel: CancellationToken, ) -> Result { if audio_samples.is_empty() { - return Err("no audio samples".into()); + if cancel.is_cancelled() { + return Ok(cancelled_result("")); + } + let message = "no audio samples".to_string(); + session::recover_after_turn_error(message.clone()); + return Err(message); } info!( @@ -205,20 +210,25 @@ pub async fn run_audio_turn( // STT. let transcript = match stt(audio_samples, sample_rate).await { + Ok(text) if cancel.is_cancelled() => { + return Ok(cancelled_result(text.trim())); + } Ok(text) if text.trim().is_empty() => { info!("{LOG_PREFIX} STT returned empty transcript, skipping turn"); - let _ = session::transition_state(CompanionState::Idle, None); + super::finish_listening_without_active_capture(); return Ok(TurnResult { transcript: String::new(), response_text: String::new(), targets: Vec::new(), tts_synthesized: false, - handoff_events: Vec::new(), cancelled: false, }); } Ok(text) => text, Err(err) => { + if cancel.is_cancelled() { + return Ok(cancelled_result("")); + } warn!("{LOG_PREFIX} STT failed err={err}"); session::transition_state(CompanionState::Error, Some(format!("STT failure: {err}")))?; return Err(format!("STT failure: {err}")); @@ -231,22 +241,79 @@ pub async fn run_audio_turn( run_text_turn(&transcript, screens, cancel).await } -// ─── Real adapters ────────────────────────────────────────────────── +// ─── Core-RPC / backend adapters ──────────────────────────────────── -/// Transcribe audio samples to text via cloud STT. +/// Transcribe audio samples through the core's configured STT provider. +/// +/// The dispatch endpoint accepts base64 container bytes, so we pack the mono +/// PCM into a shell-local WAV container first. async fn stt(samples: &[i16], sample_rate: u32) -> Result { - use crate::openhuman::voice::cloud_transcribe::{transcribe_cloud, CloudTranscribeOptions}; + let wav = super::audio::pack_wav_pcm16le_mono(samples, sample_rate); + let params = stt_dispatch_params(&wav); + let result = crate::core_rpc::call_core_rpc("openhuman.voice_stt_dispatch", params).await?; + debug!( + "{LOG_PREFIX} stt provider={:?}", + result.get("provider").and_then(|p| p.as_str()) + ); + result + .get("text") + .and_then(|t| t.as_str()) + .map(|s| s.to_string()) + .ok_or_else(|| format!("unexpected transcribe response: {result}")) +} - let config = crate::openhuman::config::ops::load_config_with_timeout().await?; - let wav_bytes = crate::openhuman::meet_agent::wav::pack_pcm16le_mono_wav(samples, sample_rate); - let audio_b64 = B64.encode(&wav_bytes); - let opts = CloudTranscribeOptions { - mime_type: Some("audio/wav".to_string()), - file_name: Some("companion.wav".to_string()), - ..Default::default() - }; - let outcome = transcribe_cloud(&config, &audio_b64, &opts).await?; - Ok(outcome.value.text.clone()) +fn stt_dispatch_params(wav: &[u8]) -> Value { + json!({ + "audio_base64": BASE64_STANDARD.encode(wav), + "mime_type": "audio/wav", + "file_name": "companion.wav", + }) +} + +/// Synthesize speech through the core's configured TTS provider. +/// +/// The shell only needs to know synthesis succeeded to hold the Speaking +/// state; native playback remains a follow-up. +async fn tts(text: &str) -> Result<(), String> { + let params = tts_dispatch_params(text); + let result = crate::core_rpc::call_core_rpc("openhuman.voice_tts_dispatch", params).await?; + debug!( + "{LOG_PREFIX} tts audio_mime={:?}", + result.get("audio_mime").and_then(|mime| mime.as_str()) + ); + Ok(()) +} + +fn tts_dispatch_params(text: &str) -> Value { + json!({ "text": text }) +} + +/// Fetch the backend session token + effective API base URL from core so the +/// shell-side LLM turn can authenticate against `/openai/v1/chat/completions`. +async fn fetch_backend_creds() -> Result<(String, String), String> { + let token_v = crate::core_rpc::call_core_rpc("openhuman.auth_get_session_token", json!({})) + .await + .map_err(|e| format!("fetch session token: {e}"))?; + let token = token_v + .get("token") + .and_then(|t| t.as_str()) + .map(str::trim) + .filter(|t| !t.is_empty()) + .ok_or_else(|| "no backend session token".to_string())? + .to_string(); + + let url_v = crate::core_rpc::call_core_rpc("openhuman.config_resolve_api_url", json!({})) + .await + .map_err(|e| format!("resolve api url: {e}"))?; + let backend_url = url_v + .get("api_url") + .and_then(|u| u.as_str()) + .map(str::trim) + .filter(|u| !u.is_empty()) + .ok_or_else(|| "no backend api_url".to_string())? + .to_string(); + + Ok((token, backend_url)) } /// System prompt for the desktop companion LLM. @@ -268,34 +335,21 @@ your response will be spoken aloud via TTS.\n\ "; /// Build a chat-completions request with screen context and conversation -/// history, then return the assistant's reply. +/// history, then return the assistant's reply. Runs shell-side against the +/// backend using creds fetched from core. async fn llm_companion( prompt: &str, history: &[&ConversationTurn], screen_context: Option<&str>, ) -> Result { - use crate::api::config::effective_backend_api_url; - use crate::api::jwt::get_session_token; - use crate::api::BackendOAuthClient; - use reqwest::Method; + let (token, backend_url) = fetch_backend_creds().await?; + let endpoint = companion_chat_endpoint(&backend_url); - let config = crate::openhuman::config::ops::load_config_with_timeout().await?; - let token = get_session_token(&config) - .map_err(|e| e.to_string())? - .filter(|t| !t.trim().is_empty()) - .ok_or_else(|| "no backend session token".to_string())?; - - let api_url = effective_backend_api_url(&config.api_url); - let client = BackendOAuthClient::new(&api_url).map_err(|e| e.to_string())?; - - let mut messages: Vec = Vec::with_capacity(history.len() + 3); + let mut messages: Vec = Vec::with_capacity(history.len() + 2); // System prompt with optional screen context. let system = if let Some(ctx) = screen_context { - format!( - "{COMPANION_SYSTEM_PROMPT}\n\ - Current screen context:\n{ctx}" - ) + format!("{COMPANION_SYSTEM_PROMPT}\nCurrent screen context:\n{ctx}") } else { COMPANION_SYSTEM_PROMPT.to_string() }; @@ -316,68 +370,51 @@ async fn llm_companion( "messages": messages, }); - // `flatten_authed_error` maps the typed `BackendApiError::Unauthorized` - // (expected session-lapse 401) onto the `SESSION_EXPIRED` sentinel so the - // JSON-RPC layer classifies it as session expiry and skips Sentry, matching - // the #3384 team/billing pattern and the voice TTS fix (TAURI-RUST-8X1). - // The previous `e.to_string()` leaked every lapsed-session companion-voice - // 401 to Sentry as a hard error; every other error keeps its `{e:#}` chain. - let raw = client - .authed_json( - &token, - Method::POST, - "/openai/v1/chat/completions", - Some(body), - ) + let client = reqwest::Client::builder() + .timeout(Duration::from_secs(60)) + .build() + .map_err(|e| format!("build http client: {e}"))?; + + let resp = client + .post(&endpoint) + .header("Content-Type", "application/json") + .header("Authorization", format!("Bearer {token}")) + .json(&body) + .send() .await - .map_err(crate::api::flatten_authed_error)?; + .map_err(|e| format!("chat completions request failed: {e}"))?; + + let status = resp.status(); + let raw: Value = resp + .json() + .await + .map_err(|e| format!("chat completions: invalid JSON response: {e}"))?; + + if !status.is_success() { + return Err(format!("chat completions returned {status}: {raw}")); + } extract_chat_completion_text(&raw) .ok_or_else(|| format!("unexpected chat completions response: {raw}")) } -/// Synthesize speech from text via cloud TTS. Returns raw PCM16LE samples. -async fn tts(text: &str) -> Result, String> { - use crate::openhuman::voice::reply_speech::{synthesize_reply, ReplySpeechOptions}; - - let config = crate::openhuman::config::ops::load_config_with_timeout().await?; - let voice_settings = json!({ - "stability": 0.4, - "similarity_boost": 0.75, - "style": 0.35, - "use_speaker_boost": true, - }); - let opts = ReplySpeechOptions { - output_format: Some("pcm_16000".to_string()), - model_id: Some(TTS_MODEL_ID.to_string()), - voice_settings: Some(voice_settings), - ..Default::default() - }; - let outcome = synthesize_reply(&config, text, &opts).await?; - let pcm_bytes = B64 - .decode(outcome.value.audio_base64.as_bytes()) - .map_err(|e| format!("decode tts base64: {e}"))?; - if pcm_bytes.len() % 2 != 0 { - return Err(format!("odd byte length from tts: {}", pcm_bytes.len())); - } - Ok(pcm_bytes - .chunks_exact(2) - .map(|c| i16::from_le_bytes([c[0], c[1]])) - .collect()) +fn companion_chat_endpoint(backend_url: &str) -> String { + format!( + "{}/openai/v1/chat/completions", + backend_url.trim_end_matches('/') + ) } -/// Gather screen context (foreground app info) as a text summary. -/// Returns `None` if screen intelligence is unavailable. +/// Gather foreground app/window context as a text summary for the LLM. +/// +/// The accessibility backbone remains in the embedded core after desktop +/// controls moved out, so the shell can reuse its public, read-only foreground +/// query without restoring any automation commands. async fn gather_screen_context() -> Option { #[cfg(target_os = "macos")] { - let context = crate::openhuman::accessibility::foreground_context(); - context.map(|ctx| { - format!( - "App: {} | Window: {}", - ctx.app_name.as_deref().unwrap_or("unknown"), - ctx.window_title.as_deref().unwrap_or("unknown"), - ) + openhuman_core::openhuman::accessibility::foreground_context().map(|ctx| { + format_foreground_context(ctx.app_name.as_deref(), ctx.window_title.as_deref()) }) } #[cfg(not(target_os = "macos"))] @@ -386,6 +423,15 @@ async fn gather_screen_context() -> Option { } } +#[cfg(any(target_os = "macos", test))] +fn format_foreground_context(app_name: Option<&str>, window_title: Option<&str>) -> String { + format!( + "App: {} | Window: {}", + app_name.unwrap_or("unknown"), + window_title.unwrap_or("unknown"), + ) +} + fn extract_chat_completion_text(raw: &Value) -> Option { raw.get("choices") .and_then(|c| c.as_array()) @@ -404,14 +450,13 @@ fn tail_history(history: &[ConversationTurn], n: usize) -> Vec<&ConversationTurn fn cancelled_result(transcript: &str) -> TurnResult { // Restore session to Idle so it doesn't stay stuck in Thinking/Speaking. - let _ = session::transition_state(CompanionState::Idle, None); + session::finish_turn(); info!("{LOG_PREFIX} turn cancelled, restored session to Idle"); TurnResult { transcript: transcript.to_string(), response_text: String::new(), targets: Vec::new(), tts_synthesized: false, - handoff_events: Vec::new(), cancelled: true, } } diff --git a/src/openhuman/desktop_companion/pipeline_tests.rs b/app/src-tauri/src/companion/pipeline_tests.rs similarity index 81% rename from src/openhuman/desktop_companion/pipeline_tests.rs rename to app/src-tauri/src/companion/pipeline_tests.rs index 70cfa13a1..893cf72e4 100644 --- a/src/openhuman/desktop_companion/pipeline_tests.rs +++ b/app/src-tauri/src/companion/pipeline_tests.rs @@ -2,12 +2,12 @@ //! //! These tests exercise the pipeline's orchestration logic — state //! transitions, cancellation, conversation history, and POINT-tag -//! integration. Real STT/LLM/TTS calls are not made; the pipeline -//! falls back to stubs in a test environment (no backend token). +//! integration. Real STT/LLM/TTS calls are not made; the pipeline's +//! network calls fail fast in a test environment (no embedded core). use super::*; -use crate::openhuman::desktop_companion::pointing::ScreenGeometry; -use crate::openhuman::desktop_companion::session; +use crate::companion::pointing::ScreenGeometry; +use crate::companion::session; /// Serialize tests that touch the process-global session state. Shared with /// `session_tests` via `session::lock_test_state()` so transitions in one test @@ -75,7 +75,6 @@ fn cancelled_result_has_correct_fields() { assert!(r.response_text.is_empty()); assert!(r.targets.is_empty()); assert!(!r.tts_synthesized); - assert!(r.handoff_events.is_empty()); assert!(r.cancelled); } @@ -104,6 +103,31 @@ fn extract_chat_completion_text_malformed() { assert_eq!(extract_chat_completion_text(&json!(42)), None); } +#[test] +fn stt_dispatch_params_encode_wav_for_provider_dispatch() { + assert_eq!( + stt_dispatch_params(b"wav"), + json!({ + "audio_base64": "d2F2", + "mime_type": "audio/wav", + "file_name": "companion.wav", + }) + ); +} + +#[test] +fn tts_dispatch_params_defer_provider_selection_to_core() { + assert_eq!(tts_dispatch_params("hello"), json!({ "text": "hello" })); +} + +#[test] +fn companion_chat_endpoint_joins_only_the_backend_path() { + assert_eq!( + companion_chat_endpoint("https://api.tinyhumans.ai/"), + "https://api.tinyhumans.ai/openai/v1/chat/completions" + ); +} + // ── Text turn tests ────────────────────────────────────────────────── #[tokio::test] @@ -148,6 +172,9 @@ async fn audio_turn_rejects_empty_samples() { let result = run_audio_turn(&[], 16_000, &single_screen(), cancel).await; assert!(result.is_err()); assert!(result.unwrap_err().contains("no audio")); + let status = session::session_status(); + assert_eq!(status.state, CompanionState::Idle); + assert_eq!(status.last_error.as_deref(), Some("no audio samples")); session::reset_for_test(); } @@ -156,10 +183,22 @@ async fn audio_turn_rejects_empty_samples() { #[tokio::test] async fn gather_screen_context_returns_option() { let ctx = gather_screen_context().await; - // Just verify it doesn't panic — value depends on platform. + // Just verify it doesn't panic — value depends on platform/capability. let _ = ctx; } +#[test] +fn foreground_context_summary_preserves_app_and_window() { + assert_eq!( + format_foreground_context(Some("Safari"), Some("OpenHuman")), + "App: Safari | Window: OpenHuman" + ); + assert_eq!( + format_foreground_context(None, None), + "App: unknown | Window: unknown" + ); +} + // ── System prompt ──────────────────────────────────────────────────── #[test] diff --git a/src/openhuman/desktop_companion/pointing.rs b/app/src-tauri/src/companion/pointing.rs similarity index 96% rename from src/openhuman/desktop_companion/pointing.rs rename to app/src-tauri/src/companion/pointing.rs index ab491cf81..5e3248cca 100644 --- a/src/openhuman/desktop_companion/pointing.rs +++ b/app/src-tauri/src/companion/pointing.rs @@ -4,6 +4,9 @@ //! response text (Clicky convention). This module extracts those tags, //! maps screen-relative coordinates to absolute desktop coordinates //! using monitor geometry, and strips the tags from the display text. +//! +//! Migrated verbatim from the former core `desktop_companion::pointing` +//! (the parsing logic has no core dependencies — it is pure regex + math). use log::debug; use serde::{Deserialize, Serialize}; diff --git a/src/openhuman/desktop_companion/pointing_tests.rs b/app/src-tauri/src/companion/pointing_tests.rs similarity index 100% rename from src/openhuman/desktop_companion/pointing_tests.rs rename to app/src-tauri/src/companion/pointing_tests.rs diff --git a/src/openhuman/desktop_companion/session.rs b/app/src-tauri/src/companion/session.rs similarity index 72% rename from src/openhuman/desktop_companion/session.rs rename to app/src-tauri/src/companion/session.rs index ecfec233d..7b8cca2ee 100644 --- a/src/openhuman/desktop_companion/session.rs +++ b/app/src-tauri/src/companion/session.rs @@ -1,4 +1,4 @@ -//! Companion session lifecycle and state machine. +//! Companion session lifecycle and state machine (shell-side). //! //! A companion session represents a single period of desktop companion //! activity. It owns the state machine (idle → listening → thinking → @@ -6,14 +6,20 @@ //! //! Only one session may be active at a time. Sessions are created with //! explicit user consent and can be stopped manually or via TTL expiry. +//! +//! Migrated from the former core `desktop_companion::session`. The only +//! behavioural change: state transitions emit the `companion://state_changed` +//! Tauri event (`super::events`) instead of the old Socket.IO broadcast bus, +//! and the core `event_bus` session-lifecycle publishes are dropped (the +//! frontend now derives session-active purely from state transitions). use log::{debug, info, warn}; use parking_lot::Mutex; -use super::bus; +use super::events; use super::types::*; -const LOG_PREFIX: &str = "[desktop_companion]"; +const LOG_PREFIX: &str = "[companion]"; /// Maximum number of conversation turns retained per session. const MAX_CONVERSATION_TURNS: usize = 50; @@ -28,6 +34,7 @@ struct CompanionSessionInner { state: CompanionState, started_at_ms: i64, expires_at_ms: Option, + #[allow(dead_code)] ttl_secs: u64, conversation: Vec, last_error: Option, @@ -92,14 +99,6 @@ pub fn start_session( *guard = Some(inner); drop(guard); - // Publish session-started event for Socket.IO bridge / subscribers. - crate::core::event_bus::publish_global( - crate::core::event_bus::DomainEvent::CompanionSessionStarted { - session_id: session_id.clone(), - ttl_secs, - }, - ); - Ok(StartCompanionSessionResult { session_id, state: CompanionState::Idle, @@ -123,15 +122,11 @@ pub fn stop_session( info!( "{LOG_PREFIX} stopping session id={session_id} reason={reason} turns={turn_count}", ); + let previous = inner.state; drop(guard); - - crate::core::event_bus::publish_global( - crate::core::event_bus::DomainEvent::CompanionSessionEnded { - session_id, - reason: reason.clone(), - turn_count, - }, - ); + if previous != CompanionState::Idle { + events::emit_state_changed(&session_id, CompanionState::Idle, previous); + } Ok(StopCompanionSessionResult { stopped: true, @@ -148,48 +143,48 @@ pub fn stop_session( } } +/// Return whether the active session has exceeded its TTL. +pub fn session_is_expired() -> bool { + let guard = ACTIVE_SESSION.lock(); + guard + .as_ref() + .and_then(|inner| inner.expires_at_ms) + .is_some_and(|expires_at_ms| chrono::Utc::now().timestamp_millis() >= expires_at_ms) +} + +/// Atomically remove the active session only when its TTL has elapsed. +pub fn expire_session_if_needed() -> bool { + let mut guard = ACTIVE_SESSION.lock(); + let Some(inner) = guard.as_ref() else { + return false; + }; + let expired = inner + .expires_at_ms + .is_some_and(|expires_at_ms| chrono::Utc::now().timestamp_millis() >= expires_at_ms); + if !expired { + return false; + } + + let stale = guard.take().expect("active session checked above"); + let session_id = stale.id.clone(); + let previous = stale.state; + let turn_count = stale.conversation.len(); + drop(guard); + info!("{LOG_PREFIX} auto-expiring stale session id={session_id} turns={turn_count}"); + if previous != CompanionState::Idle { + events::emit_state_changed(&session_id, CompanionState::Idle, previous); + } + true +} + /// Get the current session status. pub fn session_status() -> CompanionSessionStatus { - let mut guard = ACTIVE_SESSION.lock(); + let guard = ACTIVE_SESSION.lock(); match guard.as_ref() { Some(inner) => { let now_ms = chrono::Utc::now().timestamp_millis(); let remaining_ms = inner.expires_at_ms.map(|exp| (exp - now_ms).max(0)); - // Auto-expire if TTL exceeded. - // Clear inline (guard.take) instead of calling stop_session() to - // avoid a TOCTOU race where another thread starts a new session - // between drop(guard) and the stop_session() call. - if let Some(remaining) = remaining_ms { - if remaining == 0 { - let stale = guard.take().expect("checked is_some"); - let stale_id = stale.id.clone(); - let turn_count = stale.conversation.len(); - drop(stale); - drop(guard); - info!( - "{LOG_PREFIX} auto-expiring stale session id={stale_id} turns={turn_count}" - ); - crate::core::event_bus::publish_global( - crate::core::event_bus::DomainEvent::CompanionSessionEnded { - session_id: stale_id, - reason: "ttl_expired".into(), - turn_count, - }, - ); - return CompanionSessionStatus { - active: false, - state: CompanionState::Idle, - session_id: None, - started_at_ms: None, - expires_at_ms: None, - remaining_ms: None, - turn_count: 0, - last_error: Some("session expired".into()), - }; - } - } - CompanionSessionStatus { active: true, state: inner.state, @@ -217,7 +212,8 @@ pub fn session_status() -> CompanionSessionStatus { /// Transition the companion to a new state. /// /// Returns the previous state, or an error if no session is active or the -/// transition is invalid. +/// transition is invalid. Emits `companion://state_changed` to the frontend on +/// success. pub fn transition_state( new_state: CompanionState, message: Option, @@ -253,20 +249,70 @@ pub fn transition_state( inner.last_error = message.clone(); } - // Publish the state change event. let session_id = inner.id.clone(); drop(guard); - bus::publish_state_changed(CompanionStateChangedEvent { - session_id, - state: new_state, - previous_state: previous, - message, - }); + events::emit_state_changed(&session_id, new_state, previous); Ok(previous) } +/// Record a failed turn and return the active session to `Idle` so the next +/// activation can start a fresh capture. +pub fn recover_after_turn_error(message: String) { + let status = session_status(); + if !status.active { + return; + } + if status.state != CompanionState::Error { + let _ = transition_state(CompanionState::Error, Some(message)); + } + let _ = transition_state(CompanionState::Idle, None); +} + +/// Finish a turn without clobbering a newly-started interrupting capture. +/// +/// The state check and update share the session lock, closing the race where an +/// old cancelled task could observe `Thinking`, a new capture could enter +/// `Listening`, and the old task could then reset that fresh capture to `Idle`. +pub fn finish_turn() { + let mut guard = ACTIVE_SESSION.lock(); + let Some(inner) = guard.as_mut() else { + return; + }; + if matches!( + inner.state, + CompanionState::Idle | CompanionState::Listening + ) { + return; + } + let previous = inner.state; + if !is_valid_transition(previous, CompanionState::Idle) { + return; + } + inner.state = CompanionState::Idle; + let session_id = inner.id.clone(); + drop(guard); + events::emit_state_changed(&session_id, CompanionState::Idle, previous); +} + +/// Return a pre-STT/empty-transcript turn from `Listening` to `Idle`. +/// +/// The caller must ensure no newer microphone capture owns `Listening`. +pub fn finish_listening_turn() { + let mut guard = ACTIVE_SESSION.lock(); + let Some(inner) = guard.as_mut() else { + return; + }; + if inner.state != CompanionState::Listening { + return; + } + inner.state = CompanionState::Idle; + let session_id = inner.id.clone(); + drop(guard); + events::emit_state_changed(&session_id, CompanionState::Idle, CompanionState::Listening); +} + /// Add a conversation turn to the session history. pub fn push_conversation_turn(turn: ConversationTurn) -> Result<(), String> { let mut guard = ACTIVE_SESSION.lock(); diff --git a/src/openhuman/desktop_companion/session_tests.rs b/app/src-tauri/src/companion/session_tests.rs similarity index 80% rename from src/openhuman/desktop_companion/session_tests.rs rename to app/src-tauri/src/companion/session_tests.rs index a2e6ea96a..fa60afa4c 100644 --- a/src/openhuman/desktop_companion/session_tests.rs +++ b/app/src-tauri/src/companion/session_tests.rs @@ -8,8 +8,10 @@ use super::*; fn with_clean_session(f: F) { let _lock = lock_test_state(); reset_for_test(); + let _ = events::take_recorded_events(); f(); reset_for_test(); + let _ = events::take_recorded_events(); } fn start_default_session() -> StartCompanionSessionResult { @@ -84,6 +86,27 @@ fn stop_session_succeeds() { }); } +#[test] +fn stop_session_emits_idle_from_active_state() { + with_clean_session(|| { + let session = start_default_session(); + transition_state(CompanionState::Listening, None).unwrap(); + let _ = events::take_recorded_events(); + + let result = stop_session(&StopCompanionSessionParams { + reason: Some("test".into()), + }) + .unwrap(); + + assert!(result.stopped); + let events = events::take_recorded_events(); + assert_eq!(events.len(), 1); + assert_eq!(events[0].session_id, session.session_id); + assert_eq!(events[0].previous_state, CompanionState::Listening); + assert_eq!(events[0].state, CompanionState::Idle); + }); +} + #[test] fn stop_session_with_no_active_session() { with_clean_session(|| { @@ -220,6 +243,19 @@ fn transition_error_to_idle() { }); } +#[test] +fn failed_turn_recovers_to_idle_and_preserves_error() { + with_clean_session(|| { + let _s = start_default_session(); + transition_state(CompanionState::Listening, None).unwrap(); + recover_after_turn_error("no audio samples".into()); + + let status = session_status(); + assert_eq!(status.state, CompanionState::Idle); + assert_eq!(status.last_error.as_deref(), Some("no audio samples")); + }); +} + #[test] fn transition_speaking_to_listening_interrupt() { with_clean_session(|| { @@ -232,6 +268,32 @@ fn transition_speaking_to_listening_interrupt() { }); } +#[test] +fn cancelled_old_turn_does_not_reset_interrupting_capture() { + with_clean_session(|| { + let _s = start_default_session(); + transition_state(CompanionState::Listening, None).unwrap(); + transition_state(CompanionState::Thinking, None).unwrap(); + transition_state(CompanionState::Speaking, None).unwrap(); + transition_state(CompanionState::Listening, None).unwrap(); + + finish_turn(); + assert_eq!(session_status().state, CompanionState::Listening); + }); +} + +#[test] +fn pre_stt_listening_turn_can_return_to_idle() { + with_clean_session(|| { + let _s = start_default_session(); + transition_state(CompanionState::Listening, None).unwrap(); + + finish_listening_turn(); + + assert_eq!(session_status().state, CompanionState::Idle); + }); +} + #[test] fn transition_invalid_idle_to_speaking() { with_clean_session(|| { @@ -343,6 +405,29 @@ fn start_session_auto_expires_stale_session() { }); } +#[test] +fn expire_session_emits_idle_from_active_state() { + with_clean_session(|| { + let session = start_session(&StartCompanionSessionParams { + consent: true, + ttl_secs: Some(1), + }) + .unwrap(); + transition_state(CompanionState::Thinking, None).unwrap_err(); + transition_state(CompanionState::Listening, None).unwrap(); + let _ = events::take_recorded_events(); + std::thread::sleep(std::time::Duration::from_millis(1100)); + + assert!(expire_session_if_needed()); + assert!(!session_status().active); + let events = events::take_recorded_events(); + assert_eq!(events.len(), 1); + assert_eq!(events[0].session_id, session.session_id); + assert_eq!(events[0].previous_state, CompanionState::Listening); + assert_eq!(events[0].state, CompanionState::Idle); + }); +} + #[test] fn start_session_ttl_overflow_guard() { with_clean_session(|| { @@ -364,15 +449,3 @@ fn start_session_ttl_overflow_guard() { ); }); } - -// ── is_active helper ────────────────────────────────────────────────── - -#[test] -fn companion_state_is_active() { - assert!(!CompanionState::Idle.is_active()); - assert!(CompanionState::Listening.is_active()); - assert!(CompanionState::Thinking.is_active()); - assert!(CompanionState::Speaking.is_active()); - assert!(CompanionState::Pointing.is_active()); - assert!(!CompanionState::Error.is_active()); -} diff --git a/src/openhuman/desktop_companion/types.rs b/app/src-tauri/src/companion/types.rs similarity index 87% rename from src/openhuman/desktop_companion/types.rs rename to app/src-tauri/src/companion/types.rs index 8e8fb76bc..f5e25a79b 100644 --- a/src/openhuman/desktop_companion/types.rs +++ b/app/src-tauri/src/companion/types.rs @@ -1,4 +1,10 @@ -//! Shared types for the desktop companion session. +//! Shared types for the desktop companion session (shell-side). +//! +//! Migrated from the former core `desktop_companion::types`. The only +//! behavioural change is the state-changed event: it is now delivered to the +//! frontend as a Tauri event with a **camelCase** payload +//! (`{ sessionId, state, previousState }`) instead of the old Socket.IO +//! `companion:state_changed` snake_case payload. use serde::{Deserialize, Serialize}; @@ -21,13 +27,6 @@ pub enum CompanionState { Error, } -impl CompanionState { - /// Returns `true` for states that represent an active interaction turn. - pub fn is_active(self) -> bool { - !matches!(self, Self::Idle | Self::Error) - } -} - impl std::fmt::Display for CompanionState { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { @@ -143,13 +142,13 @@ pub struct StopCompanionSessionResult { pub reason: Option, } -/// Event emitted when companion state changes (for Socket.IO bridge). +/// Payload of the `companion://state_changed` Tauri event delivered to the +/// frontend. Serialized **camelCase** to match the JS companion slice / +/// overlay contract (`{ sessionId, state, previousState }`). #[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] pub struct CompanionStateChangedEvent { pub session_id: String, pub state: CompanionState, pub previous_state: CompanionState, - /// Optional human-readable message (e.g. error details, response text). - #[serde(default, skip_serializing_if = "Option::is_none")] - pub message: Option, } diff --git a/app/src-tauri/src/companion_commands.rs b/app/src-tauri/src/companion_commands.rs index 1626bbdf4..092f95256 100644 --- a/app/src-tauri/src/companion_commands.rs +++ b/app/src-tauri/src/companion_commands.rs @@ -40,12 +40,17 @@ pub(crate) async fn register_companion_hotkey( let register_shortcut = |variant: &str| -> Result<(), String> { let app_clone = app.clone(); app.global_shortcut() - .on_shortcut(variant, move |_app, _sc, event| { - if event.state == ShortcutState::Pressed { + .on_shortcut(variant, move |_app, _sc, event| match event.state { + ShortcutState::Pressed => { debug!("[companion] hotkey pressed — emitting companion://activate"); if let Err(e) = app_clone.emit("companion://activate", ()) { warn!("[companion] emit failed: {e}"); } + crate::companion::handle_hotkey_pressed(app_clone.clone()); + } + ShortcutState::Released => { + debug!("[companion] hotkey released"); + crate::companion::handle_hotkey_released(app_clone.clone()); } }) .map_err(|e| format!("Failed to register shortcut '{variant}': {e}")) @@ -100,8 +105,9 @@ pub(crate) async fn register_companion_hotkey( } /// Unregister the global companion hotkey (if any). -#[tauri::command] -pub(crate) async fn unregister_companion_hotkey(app: AppHandle) -> Result<(), String> { +pub(crate) fn unregister_companion_hotkey_for_app( + app: &AppHandle, +) -> Result<(), String> { info!("[companion] unregister_companion_hotkey: called"); let state = app.state::(); let mut guard = state.0.lock().unwrap(); @@ -127,10 +133,20 @@ pub(crate) async fn unregister_companion_hotkey(app: AppHandle) -> R Ok(()) } +/// Tauri command wrapper around the shared lifecycle cleanup. +#[tauri::command] +pub(crate) async fn unregister_companion_hotkey(app: AppHandle) -> Result<(), String> { + unregister_companion_hotkey_for_app(&app) +} + /// Programmatic companion activation (e.g. from a "Test" button in settings). #[tauri::command] pub(crate) async fn companion_activate(app: AppHandle) -> Result<(), String> { info!("[companion] companion_activate: called"); app.emit("companion://activate", ()) - .map_err(|e| format!("Failed to emit companion://activate: {e}")) + .map_err(|e| format!("Failed to emit companion://activate: {e}"))?; + // Programmatic activation is a toggle regardless of the configured hotkey + // mode because a button has no corresponding release event. + crate::companion::handle_activation(app.clone()); + Ok(()) } diff --git a/app/src-tauri/src/core_rpc.rs b/app/src-tauri/src/core_rpc.rs index 11e53ab7d..84071eacc 100644 --- a/app/src-tauri/src/core_rpc.rs +++ b/app/src-tauri/src/core_rpc.rs @@ -80,6 +80,19 @@ pub(crate) async fn relay_http_rpc( url: String, token: Option, body: String, +) -> Result { + post_json_rpc(&url, token.as_deref(), body).await +} + +/// Transport core of [`relay_http_rpc`]: POST a JSON body to `url` with an +/// optional bearer, returning the upstream status + body verbatim. Factored out +/// so in-process shell callers (e.g. the desktop companion pipeline) can reach +/// the local core over the same path the renderer's relay uses, without going +/// through the Tauri command boundary. +pub(crate) async fn post_json_rpc( + url: &str, + token: Option<&str>, + body: String, ) -> Result { let client = reqwest::Client::builder() .timeout(Duration::from_secs(30)) @@ -87,16 +100,16 @@ pub(crate) async fn relay_http_rpc( .map_err(|e| format!("failed to build HTTP client: {e}"))?; let mut builder = client - .post(&url) + .post(url) .header("Content-Type", "application/json") .body(body); - let bearer = relay_bearer_header(token.as_deref()); + let bearer = relay_bearer_header(token); if let Some(value) = bearer.as_deref() { builder = builder.header("Authorization", value); } - let safe_url = redact_url_for_log(&url); + let safe_url = redact_url_for_log(url); log::debug!( "[core_rpc][relay] POST {safe_url} (auth={})", bearer.is_some() @@ -118,6 +131,58 @@ pub(crate) async fn relay_http_rpc( Ok(RelayHttpResponse { status, body: text }) } +/// Invoke a core JSON-RPC method against the embedded local core, returning the +/// inner result value. Handles building the JSON-RPC envelope, applying the +/// per-launch bearer, and unwrapping the `{ "result": , "logs": [...] }` +/// `RpcOutcome` envelope some core handlers emit. +pub(crate) async fn call_core_rpc( + method: &str, + params: serde_json::Value, +) -> Result { + let url = core_rpc_url_value(); + let token = crate::core_process::current_rpc_token(); + let envelope = serde_json::json!({ + "jsonrpc": "2.0", + "id": 1, + "method": method, + "params": params, + }); + let resp = post_json_rpc(&url, token.as_deref(), envelope.to_string()).await?; + if !(200..300).contains(&resp.status) { + return Err(format!("core rpc {method} returned status {}", resp.status)); + } + let parsed: serde_json::Value = serde_json::from_str(&resp.body) + .map_err(|e| format!("core rpc {method}: invalid JSON response: {e}"))?; + if let Some(err) = parsed.get("error") { + if !err.is_null() { + return Err(format!("core rpc {method} error: {err}")); + } + } + let result = parsed + .get("result") + .cloned() + .ok_or_else(|| format!("core rpc {method}: response has no result field"))?; + Ok(unwrap_rpc_outcome(result)) +} + +/// Unwrap the `{ "result": , "logs": [...] }` `RpcOutcome` envelope that +/// logged core handlers wrap their payload in; passes bare values through +/// unchanged. +fn unwrap_rpc_outcome(value: serde_json::Value) -> serde_json::Value { + if let Some(obj) = value.as_object() { + if obj.len() == 2 + && obj.contains_key("result") + && obj.get("logs").map(|l| l.is_array()).unwrap_or(false) + { + return obj + .get("result") + .cloned() + .unwrap_or(serde_json::Value::Null); + } + } + value +} + #[cfg(test)] mod tests { use super::relay_bearer_header; diff --git a/app/src-tauri/src/lib.rs b/app/src-tauri/src/lib.rs index 7f4a0d3c5..ac3433b91 100644 --- a/app/src-tauri/src/lib.rs +++ b/app/src-tauri/src/lib.rs @@ -57,6 +57,7 @@ mod cef_singleton_wait; #[cfg(any(target_os = "macos", target_os = "linux", test))] mod cef_stale_reap; mod claude_code; +mod companion; mod companion_commands; mod core_process; mod core_rpc; @@ -100,6 +101,7 @@ mod telegram_scanner; mod webview_accounts; mod webview_apis; mod wechat_scanner; +mod whatsapp_data; mod whatsapp_scanner; mod window_state; mod workspace_paths; @@ -3121,6 +3123,16 @@ pub fn run() { // requires) from generic `` call sites. cdp::set_cef_app_handle(app.handle().clone()); + // Install the app handle for the desktop companion so its session + // state machine can emit `companion://state_changed` events. + companion::setup(app.handle()); + + // Structured WhatsApp Web data store lives shell-side. Register the + // in-process native handlers so the core agent tools (list/search) + // and the scanner ingest path can reach the SQLite store over the + // native request bus. No handler = graceful degradation core-side. + whatsapp_data::register_native_handlers(); + #[cfg(windows)] { // `register_all` writes HKCU\Software\Classes\openhuman so the @@ -3484,48 +3496,6 @@ pub fn run() { // here would flash the pill on every launch even with always-on // listening disabled (the default). - // Synthetic-input main-thread executor. enigo's macOS keyboard-layout - // lookup (TSMGetInputSourceProperty) MUST run on the app main thread - // or it traps (`_dispatch_assert_queue_fail`/EXC_BREAKPOINT) and - // crashes the CEF host (Change 1.15, confirmed via crash report). The - // keyboard/mouse tools run on tokio workers, so they dispatch their - // enigo ops here via the native registry; we run each on the real - // main thread through `run_on_main_thread`. - { - use openhuman_core::core::event_bus::register_native_global; - use openhuman_core::openhuman::tools::{ - MainThreadInputOp, INPUT_ON_MAIN_THREAD_METHOD, - }; - let input_app = app.handle().clone(); - register_native_global::, _, _>( - INPUT_ON_MAIN_THREAD_METHOD, - move |req| { - let input_app = input_app.clone(); - async move { - let (tx, rx) = tokio::sync::oneshot::channel(); - let run = req.run; - input_app - .run_on_main_thread(move || { - // Catch an enigo FFI panic so it can't unwind - // across the app main thread (which would be - // UB / abort). Convert it to a clean Err. - let result = std::panic::catch_unwind( - std::panic::AssertUnwindSafe(run), - ) - .unwrap_or_else(|_| { - Err("synthetic input panicked on the main thread".to_string()) - }); - let _ = tx.send(result); - }) - .map_err(|e| format!("run_on_main_thread dispatch failed: {e}"))?; - rx.await - .map_err(|_| "main-thread input op was cancelled".to_string()) - } - }, - ); - log::info!("[computer] registered main-thread synthetic-input executor"); - } - // Tray icon setup moved to RunEvent::Ready (see below) — GTK is only // initialized after the event loop starts, so we must delay tray creation // until the Ready event fires. Creating the tray here would panic on @@ -3845,6 +3815,10 @@ pub fn run() { // and the Save-As fallback needs it there (CodeRabbit on #4127). artifact_commands::save_artifact_via_dialog, artifact_commands::download_artifact_to_downloads, + // Structured WhatsApp data (store lives shell-side). + whatsapp_data::whatsapp_data_list_chats, + whatsapp_data::whatsapp_data_list_messages, + whatsapp_data::whatsapp_data_search_messages, check_core_update, apply_core_update, check_app_update, @@ -3904,6 +3878,11 @@ pub fn run() { companion_commands::register_companion_hotkey, companion_commands::unregister_companion_hotkey, companion_commands::companion_activate, + companion::companion_start_session, + companion::companion_stop_session, + companion::companion_status, + companion::companion_config_get, + companion::companion_config_set, mcp_commands::mcp_resolve_binary_path, mcp_commands::mcp_open_client_config, loopback_oauth::start_loopback_oauth_listener, diff --git a/app/src-tauri/src/webview_accounts/mod.rs b/app/src-tauri/src/webview_accounts/mod.rs index 2890f2edf..adea2db8a 100644 --- a/app/src-tauri/src/webview_accounts/mod.rs +++ b/app/src-tauri/src/webview_accounts/mod.rs @@ -1145,10 +1145,9 @@ impl From<&NotificationBypassPrefs> for NotificationBypassPrefsPayload { } /// Title prefix applied to every OS toast fired from an embedded webview. -/// Matches `openhuman_core::webview_notifications::OPENHUMAN_TITLE_PREFIX` -/// — kept inline here so the shell crate doesn't take a build-time dep on -/// the core library. Disambiguates from natively-installed apps (Slack, -/// Discord, Telegram desktop) firing the same message twice. +/// Owned by the shell crate so it doesn't take a build-time dep on the core +/// library. Disambiguates from natively-installed apps (Slack, Discord, +/// Telegram desktop) firing the same message twice. const OPENHUMAN_TITLE_PREFIX: &str = "OpenHuman: "; fn slack_scanner_enabled() -> bool { diff --git a/app/src-tauri/src/whatsapp_data/global.rs b/app/src-tauri/src/whatsapp_data/global.rs new file mode 100644 index 000000000..e7e7a542f --- /dev/null +++ b/app/src-tauri/src/whatsapp_data/global.rs @@ -0,0 +1,91 @@ +//! Process-global WhatsApp data store singleton. +//! +//! One workspace-bound `WhatsAppDataStore` is active at a time, shared by +//! native handlers, scanners, and Tauri commands. When the active workspace +//! changes without relaunching the shell, the singleton is reopened for the new +//! path so user data cannot leak across sessions. +//! +//! # Usage +//! +//! ```ignore +//! // At startup: +//! whatsapp_data::global::init(workspace_dir)?; +//! +//! // In RPC handlers: +//! let store = whatsapp_data::global::store()?; +//! ``` + +use std::path::{Path, PathBuf}; +use std::sync::{Arc, RwLock}; + +use super::store::WhatsAppDataStore; + +/// Shared, thread-safe reference to the store. +pub type WhatsAppDataStoreRef = Arc; + +// `RwLock>` rather than `OnceLock` so tests can swap workspaces +// between runs (each test uses its own temp dir; without reset, the second +// test would attach to a dropped sqlite path). Production callers still get +// strict idempotency: `init` is a no-op once a store is set. +struct WorkspaceStore { + workspace_dir: PathBuf, + store: WhatsAppDataStoreRef, +} + +static GLOBAL_STORE: RwLock> = RwLock::new(None); + +/// Initialise the global store for `workspace_dir`. +/// +/// Reuses the current instance only when it is bound to the same workspace. +/// A different path atomically replaces it with a freshly opened store. +pub fn init(workspace_dir: PathBuf) -> Result { + let mut guard = GLOBAL_STORE + .write() + .map_err(|e| format!("[whatsapp_data:global] write lock poisoned: {e}"))?; + if let Some(existing) = guard + .as_ref() + .filter(|entry| same_workspace(&entry.workspace_dir, &workspace_dir)) + { + log::debug!("[whatsapp_data:global] already initialised"); + return Ok(Arc::clone(&existing.store)); + } + log::info!( + "[whatsapp_data:global] opening store workspace={}", + workspace_dir.display() + ); + let store = Arc::new( + WhatsAppDataStore::new(&workspace_dir) + .map_err(|e| format!("[whatsapp_data] store init failed: {e}"))?, + ); + *guard = Some(WorkspaceStore { + workspace_dir, + store: Arc::clone(&store), + }); + Ok(store) +} + +fn same_workspace(current: &Path, requested: &Path) -> bool { + current == requested +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn init_reuses_same_workspace_and_reopens_when_workspace_changes() { + let first_dir = tempfile::tempdir().unwrap(); + let second_dir = tempfile::tempdir().unwrap(); + + let first = init(first_dir.path().to_path_buf()).unwrap(); + let same = init(first_dir.path().to_path_buf()).unwrap(); + assert!(Arc::ptr_eq(&first, &same)); + + let second = init(second_dir.path().to_path_buf()).unwrap(); + assert!(!Arc::ptr_eq(&first, &second)); + assert!(second_dir + .path() + .join("whatsapp_data/whatsapp_data.db") + .exists()); + } +} diff --git a/app/src-tauri/src/whatsapp_data/mod.rs b/app/src-tauri/src/whatsapp_data/mod.rs new file mode 100644 index 000000000..f85965e3e --- /dev/null +++ b/app/src-tauri/src/whatsapp_data/mod.rs @@ -0,0 +1,145 @@ +//! Shell-side structured WhatsApp Web data store (relocated from the core). +//! +//! Owns the SQLite persistence (`store`), the ingest + list/search business +//! logic (`ops`), the busy/corrupt retry layer (`sqlite_retry`), and a +//! process-global store singleton (`global`). The DB lives at +//! `/whatsapp_data/whatsapp_data.db` — the same workspace the +//! core resolves via [`Config`], so the file stays on the internal-path denylist +//! (`security::policy` `WORKSPACE_INTERNAL_DIRS`) and agent tools cannot write it. +//! +//! Two callers reach this store: +//! +//! - **The scanner** (`whatsapp_scanner`) writes via the `whatsapp_data.ingest` +//! native request (see [`register_native_handlers`]). +//! - **The core agent tools** (`openhuman::whatsapp_data::tools`) query via the +//! `whatsapp_data.{list_chats,list_messages,search_messages}` native requests. +//! - **The frontend** reads via the Tauri commands +//! [`whatsapp_data_list_chats`] / [`whatsapp_data_list_messages`] / +//! [`whatsapp_data_search_messages`]. +//! +//! The shared DTOs (request/response/row types) are defined once in the core +//! crate (`openhuman_core::openhuman::whatsapp_data::types`) so both sides agree +//! on a single definition and the native-request `TypeId` checks line up. + +mod global; +mod ops; +mod sqlite_retry; +mod store; + +use std::sync::Arc; + +use openhuman_core::openhuman::whatsapp_data::methods; +use openhuman_core::openhuman::whatsapp_data::types::{ + IngestRequest, IngestResult, ListChatsRequest, ListMessagesRequest, SearchMessagesRequest, + WhatsAppChat, WhatsAppMessage, +}; +use store::WhatsAppDataStore; + +/// Lazily open (or return) the shell's whatsapp_data store, bound to the active +/// OpenHuman workspace dir. +/// +/// The store is process-global so the scanner, native query handlers, and Tauri +/// commands share one write lock and recovery lifecycle. Every call resolves +/// the active workspace; [`global::init`] reuses the store only when that path +/// still matches, and atomically reopens it after user/reset workspace changes. +pub async fn ensure_store() -> Result, String> { + let cfg = openhuman_core::openhuman::config::Config::load_or_init() + .await + .map_err(|e| format!("[whatsapp_data] config load failed: {e:#}"))?; + log::debug!( + "[whatsapp_data] ensuring shell store (workspace={})", + cfg.workspace_dir.display() + ); + global::init(cfg.workspace_dir.clone()) +} + +/// Register the in-process native request handlers that bridge the core (agent +/// tools + scanner) to this shell store. Call once during Tauri `setup`. +/// +/// Keyed by the method-name constants the core owns +/// (`openhuman_core::openhuman::whatsapp_data::methods`) so the two sides never +/// drift on the string key. +pub fn register_native_handlers() { + use openhuman_core::core::event_bus::register_native_global; + + register_native_global::, _, _>( + methods::LIST_CHATS, + |req| async move { + let store = ensure_store().await?; + ops::list_chats(&store, req).map_err(|e| format!("{e:#}")) + }, + ); + register_native_global::, _, _>( + methods::LIST_MESSAGES, + |req| async move { + let store = ensure_store().await?; + ops::list_messages(&store, req).map_err(|e| format!("{e:#}")) + }, + ); + register_native_global::, _, _>( + methods::SEARCH_MESSAGES, + |req| async move { + let store = ensure_store().await?; + ops::search_messages(&store, req).map_err(|e| format!("{e:#}")) + }, + ); + register_native_global::( + methods::INGEST, + |req| async move { + let store = ensure_store().await?; + // `{e:#}` renders the full anyhow chain so the underlying SQLite + // cause (locked / malformed / FK) survives to the scanner's log. + ops::ingest(&store, req).map_err(|e| format!("[whatsapp_data] ingest failed: {e:#}")) + }, + ); + log::info!( + "[whatsapp_data] registered shell native handlers (list_chats / list_messages / search_messages / ingest)" + ); +} + +// ── Tauri commands (frontend read surface) ─────────────────────────────────── + +/// List locally-stored WhatsApp chats. Frontend replacement for the former +/// `openhuman.whatsapp_data_list_chats` core RPC. +#[tauri::command] +pub async fn whatsapp_data_list_chats(req: ListChatsRequest) -> Result, String> { + log::debug!( + "[whatsapp_data][cmd] list_chats has_account={} limit={:?} offset={:?}", + req.account_id.is_some(), + req.limit, + req.offset + ); + let store = ensure_store().await?; + ops::list_chats(&store, req).map_err(|e| format!("[whatsapp_data] list_chats failed: {e:#}")) +} + +/// List messages for a chat. Frontend replacement for the former +/// `openhuman.whatsapp_data_list_messages` core RPC. +#[tauri::command] +pub async fn whatsapp_data_list_messages( + req: ListMessagesRequest, +) -> Result, String> { + log::debug!( + "[whatsapp_data][cmd] list_messages has_account={} (chat redacted)", + req.account_id.is_some() + ); + let store = ensure_store().await?; + ops::list_messages(&store, req) + .map_err(|e| format!("[whatsapp_data] list_messages failed: {e:#}")) +} + +/// Full-text search over stored WhatsApp messages. Frontend-facing companion to +/// the agent's `whatsapp_data_search_messages` tool. +#[tauri::command] +pub async fn whatsapp_data_search_messages( + req: SearchMessagesRequest, +) -> Result, String> { + log::debug!( + "[whatsapp_data][cmd] search_messages has_account={} has_chat={}", + req.account_id.is_some(), + req.chat_id.is_some() + ); + let store = ensure_store().await?; + ops::search_messages(&store, req) + .map_err(|e| format!("[whatsapp_data] search_messages failed: {e:#}")) +} diff --git a/src/openhuman/whatsapp_data/ops.rs b/app/src-tauri/src/whatsapp_data/ops.rs similarity index 95% rename from src/openhuman/whatsapp_data/ops.rs rename to app/src-tauri/src/whatsapp_data/ops.rs index 99c087b81..0b14fe725 100644 --- a/src/openhuman/whatsapp_data/ops.rs +++ b/app/src-tauri/src/whatsapp_data/ops.rs @@ -5,12 +5,10 @@ use anyhow::Result; -use crate::openhuman::whatsapp_data::{ - store::WhatsAppDataStore, - types::{ - IngestRequest, IngestResult, ListChatsRequest, ListMessagesRequest, SearchMessagesRequest, - WhatsAppChat, WhatsAppMessage, - }, +use super::store::WhatsAppDataStore; +use openhuman_core::openhuman::whatsapp_data::types::{ + IngestRequest, IngestResult, ListChatsRequest, ListMessagesRequest, SearchMessagesRequest, + WhatsAppChat, WhatsAppMessage, }; /// Number of seconds in 90 days — the auto-prune horizon. @@ -90,7 +88,7 @@ pub fn search_messages( #[cfg(test)] mod tests { use super::*; - use crate::openhuman::whatsapp_data::types::{ChatMeta, IngestMessage}; + use openhuman_core::openhuman::whatsapp_data::types::{ChatMeta, IngestMessage}; use std::collections::HashMap; use tempfile::tempdir; diff --git a/src/openhuman/whatsapp_data/sqlite_retry.rs b/app/src-tauri/src/whatsapp_data/sqlite_retry.rs similarity index 88% rename from src/openhuman/whatsapp_data/sqlite_retry.rs rename to app/src-tauri/src/whatsapp_data/sqlite_retry.rs index 7f041cdee..2673d2887 100644 --- a/src/openhuman/whatsapp_data/sqlite_retry.rs +++ b/app/src-tauri/src/whatsapp_data/sqlite_retry.rs @@ -1,7 +1,6 @@ //! SQLite busy/locked detection and retry-with-backoff for WhatsApp data writes. //! -//! Modelled on [`crate::openhuman::memory_queue::worker::is_sqlite_busy`] — -//! the configured `busy_timeout` absorbs short waits inside rusqlite; this layer +//! The configured `busy_timeout` absorbs short waits inside rusqlite; this layer //! catches residual `SQLITE_BUSY` / `SQLITE_LOCKED` after that window. use std::thread; @@ -41,16 +40,13 @@ pub fn is_sqlite_busy(err: &anyhow::Error) -> bool { /// re-opens the dead file, re-hits the error, and re-reports to Sentry — /// turning one corrupt file into an escalating flood (Sentry TAURI-RUST-KNH: /// 1,813 events from a single host). Detecting it here is what lets the store -/// drive its quarantine + rebuild recovery -/// ([`WhatsAppDataStore::recover_corrupt_db`](crate::openhuman::whatsapp_data::store::WhatsAppDataStore)) -/// instead of retrying forever. +/// drive its quarantine + rebuild recovery instead of retrying forever. /// /// Matching on the error **code** is rusqlite-version-stable. The text /// fallback covers the case where the rusqlite error was flattened to a plain /// `anyhow!` string across `.context()` layers — SQLite renders these as /// "database disk image is malformed" (code 11) and "file is not a database" -/// (code 26). The `"disk image is malformed"` substring matches both the bare -/// and `Error code 11: The database disk image is malformed` envelope shapes. +/// (code 26). pub fn is_sqlite_corrupt(err: &anyhow::Error) -> bool { if let Some(rusqlite::Error::SqliteFailure(sqlite_err, _)) = err.downcast_ref::() @@ -161,10 +157,6 @@ mod tests { assert!(is_sqlite_corrupt(&anyhow::Error::from(raw))); } - /// The rusqlite error sits under the `[whatsapp_data] ingest failed:` + - /// `upsert wa_chat …` context layers when it bubbles out of the store; the - /// downcast must still find the `DatabaseCorrupt` code (regression guard: - /// don't rely on the top-level error type). #[test] fn is_sqlite_corrupt_matches_through_context_layers() { let raw = rusqlite::Error::SqliteFailure( @@ -180,8 +172,6 @@ mod tests { assert!(is_sqlite_corrupt(&wrapped)); } - /// Text fallback: the exact flattened Sentry string (TAURI-RUST-KNH) must - /// classify even when no rusqlite error is available to downcast. #[test] fn is_sqlite_corrupt_text_fallback() { let err = anyhow::anyhow!( @@ -191,8 +181,6 @@ mod tests { assert!(is_sqlite_corrupt(&err)); } - /// Busy/locked and constraint violations must NOT be swallowed as - /// corruption — quarantining on those would destroy a perfectly good DB. #[test] fn is_sqlite_corrupt_does_not_match_busy_or_constraint() { let busy = rusqlite::Error::SqliteFailure( diff --git a/src/openhuman/whatsapp_data/store.rs b/app/src-tauri/src/whatsapp_data/store.rs similarity index 99% rename from src/openhuman/whatsapp_data/store.rs rename to app/src-tauri/src/whatsapp_data/store.rs index 6025f8ed3..288030016 100644 --- a/src/openhuman/whatsapp_data/store.rs +++ b/app/src-tauri/src/whatsapp_data/store.rs @@ -13,10 +13,8 @@ use std::sync::Mutex; use anyhow::{Context, Result}; use rusqlite::{params, Connection}; -use crate::openhuman::whatsapp_data::sqlite_retry::{ - is_sqlite_corrupt, retry_on_sqlite_busy, BUSY_TIMEOUT, -}; -use crate::openhuman::whatsapp_data::types::{ +use super::sqlite_retry::{is_sqlite_corrupt, retry_on_sqlite_busy, BUSY_TIMEOUT}; +use openhuman_core::openhuman::whatsapp_data::types::{ ChatMeta, IngestMessage, ListChatsRequest, ListMessagesRequest, SearchMessagesRequest, WhatsAppChat, WhatsAppMessage, }; @@ -233,7 +231,7 @@ impl WhatsAppDataStore { // latch the scanner's 2–30s poll re-hits the wedged DB and re-pages on // every tick (TAURI-RUST-KNH: 1,813 events from one host). if !CORRUPT_REPORTED.swap(true, Ordering::Relaxed) { - crate::core::observability::report_error( + openhuman_core::core::observability::report_error( err, "whatsapp_data", "ingest_corrupt", diff --git a/src/openhuman/whatsapp_data/store_tests.rs b/app/src-tauri/src/whatsapp_data/store_tests.rs similarity index 99% rename from src/openhuman/whatsapp_data/store_tests.rs rename to app/src-tauri/src/whatsapp_data/store_tests.rs index b20617f07..91c89a31d 100644 --- a/src/openhuman/whatsapp_data/store_tests.rs +++ b/app/src-tauri/src/whatsapp_data/store_tests.rs @@ -5,10 +5,10 @@ //! for one `account_id` must never appear in queries for a different account. use super::super::sqlite_retry::BUSY_TIMEOUT; -use super::super::types::{ +use super::WhatsAppDataStore; +use openhuman_core::openhuman::whatsapp_data::types::{ ChatMeta, IngestMessage, ListChatsRequest, ListMessagesRequest, SearchMessagesRequest, }; -use super::WhatsAppDataStore; use std::collections::HashMap; use std::path::PathBuf; use std::sync::mpsc; diff --git a/app/src-tauri/src/whatsapp_scanner/mod.rs b/app/src-tauri/src/whatsapp_scanner/mod.rs index bb7105052..8e171f4db 100644 --- a/app/src-tauri/src/whatsapp_scanner/mod.rs +++ b/app/src-tauri/src/whatsapp_scanner/mod.rs @@ -1050,11 +1050,12 @@ async fn post_whatsapp_data_ingest( }) .unwrap_or_default(); - // Split messages into chunks to stay well under the HTTP body size limit. - // Chats are sent only with the first batch (upserts are idempotent). + // Split messages into chunks, preserving the historical batching/dedup + // behavior: chats are sent only with the first batch (upserts are + // idempotent), and the store's 90-day prune runs per ingest call. The + // store now lives in the shell, so each batch dispatches over the + // in-process native request bus instead of an HTTP JSON-RPC POST. const BATCH_SIZE: usize = 500; - let empty_chats = Value::Object(serde_json::Map::new()); - let url = crate::core_rpc::core_rpc_url_value(); // Build at least one batch even when messages is empty (chats-only upsert). let chunks: Vec<&[Value]> = if messages.is_empty() { @@ -1077,65 +1078,33 @@ async fn post_whatsapp_data_ingest( let batch_chats = if batch_idx == 0 { Value::Object(chats_param.clone()) } else { - empty_chats.clone() + Value::Object(serde_json::Map::new()) }; let params = json!({ "account_id": account_id, "chats": batch_chats, "messages": chunk, }); - let body = json!({ - "jsonrpc": "2.0", - "id": 1, - "method": "openhuman.whatsapp_data_ingest", - "params": params, - }); - - let mut last_err = String::new(); - let mut succeeded = false; - for attempt in 1u8..=2 { - let client = reqwest::Client::builder() - .timeout(Duration::from_secs(15)) - .build() - .map_err(|e| format!("http client: {e}"))?; - let req = crate::core_rpc::apply_auth(client.post(&url)) - .map_err(|e| format!("prepare {url}: {e}"))?; - let send_result = req.json(&body).send().await; - match send_result { - Err(e) if e.is_connect() || e.is_timeout() => { - last_err = format!("POST {url}: {e}"); - if attempt < 2 { - log::debug!( - "[wa][{}] whatsapp_data_ingest connect error batch={}/{} attempt {}: {}", - account_id, - batch_idx + 1, - total_batches, - attempt, - last_err - ); - tokio::time::sleep(Duration::from_millis(500)).await; - } - continue; - } - Err(e) => return Err(format!("POST {url}: {e}")), - Ok(resp) => { - let status = resp.status(); - if !status.is_success() { - let body_text = resp.text().await.unwrap_or_default(); - return Err(format!("{status}: {body_text}")); - } - let v: Value = resp.json().await.map_err(|e| format!("decode: {e}"))?; - if let Some(err) = v.get("error") { - return Err(format!("rpc error: {err}")); - } - succeeded = true; - break; - } - } - } - if !succeeded { - return Err(last_err); - } + // Deserialize into the shared core DTO and dispatch to the shell store's + // native handler registered in `whatsapp_data::register_native_handlers`. + let req: openhuman_core::openhuman::whatsapp_data::types::IngestRequest = + serde_json::from_value(params) + .map_err(|e| format!("build ingest request (batch {}): {e}", batch_idx + 1))?; + openhuman_core::core::event_bus::request_native_global::< + openhuman_core::openhuman::whatsapp_data::types::IngestRequest, + openhuman_core::openhuman::whatsapp_data::types::IngestResult, + >( + openhuman_core::openhuman::whatsapp_data::methods::INGEST, + req, + ) + .await + .map_err(|e| { + format!( + "whatsapp_data ingest batch {}/{}: {e}", + batch_idx + 1, + total_batches + ) + })?; } log::debug!( diff --git a/app/src/components/intelligence/ModelCouncilTab.test.tsx b/app/src/components/intelligence/ModelCouncilTab.test.tsx deleted file mode 100644 index 4fcc6ef87..000000000 --- a/app/src/components/intelligence/ModelCouncilTab.test.tsx +++ /dev/null @@ -1,650 +0,0 @@ -import { act, fireEvent, render, screen, waitFor, within } from '@testing-library/react'; -import { beforeEach, describe, expect, it, vi } from 'vitest'; - -import type { CouncilDefinition } from '../../services/api/councilRegistryApi'; -import type { CouncilMemberResult, ModelCouncilResult } from '../../services/api/modelCouncilApi'; -import ModelCouncilTab from './ModelCouncilTab'; - -const mockListCouncils = vi.fn(); -const mockUpsertCouncil = vi.fn(); -const mockDeleteCouncil = vi.fn(); -const mockAnswerMember = vi.fn(); -const mockSynthesizeCouncil = vi.fn(); -const mockLoadAISettings = vi.fn(); -const mockLoadLocalProviderSnapshot = vi.fn(); -const mockListProviderModels = vi.fn(); -const mockDispatch = vi.fn(); - -const mockState = { - agentProfiles: { - profiles: [ - { - id: 'default', - name: 'Default Agent', - description: 'Default', - agentId: 'openhuman.default', - modelOverride: 'profile-model', - builtIn: true, - }, - { - id: 'critic', - name: 'Critic', - description: 'Finds gaps', - agentId: 'critic-agent', - modelOverride: 'critic-model', - builtIn: false, - }, - ], - activeProfileId: 'default', - status: 'idle', - error: null, - }, -}; - -vi.mock('../../services/api/modelCouncilApi', () => ({ - modelCouncilApi: { - answerMember: (...args: unknown[]) => mockAnswerMember(...args), - synthesizeCouncil: (...args: unknown[]) => mockSynthesizeCouncil(...args), - }, -})); - -vi.mock('../../services/api/councilRegistryApi', () => ({ - councilRegistryApi: { - list: (...args: unknown[]) => mockListCouncils(...args), - upsert: (...args: unknown[]) => mockUpsertCouncil(...args), - delete: (...args: unknown[]) => mockDeleteCouncil(...args), - }, -})); - -vi.mock('../../services/api/aiSettingsApi', () => ({ - loadAISettings: (...args: unknown[]) => mockLoadAISettings(...args), - loadLocalProviderSnapshot: (...args: unknown[]) => mockLoadLocalProviderSnapshot(...args), - listProviderModels: (...args: unknown[]) => mockListProviderModels(...args), -})); - -vi.mock('../../store/hooks', () => ({ - useAppDispatch: () => mockDispatch, - useAppSelector: (selector: (state: typeof mockState) => unknown) => selector(mockState), -})); - -vi.mock('../../features/human/Mascot', () => ({ - RiveMascot: ({ face }: { face?: string }) =>
, - getMascotPalette: () => ({ bodyFill: '#F7D145', neckShadowColor: '#B23C05' }), - hexToArgbInt: () => 0xfff7d145, -})); - -const RESULT: ModelCouncilResult = { - question: 'What is the capital of France?', - members: [ - { model: 'gpt-5.2', response: 'Paris is the capital.', error: null }, - { model: 'critic-model', response: null, error: 'rate limited' }, - ], - chair_model: 'claude-opus-4-8', - synthesis: 'Both that answered agree: Paris. One seat failed.', -}; - -const DEFAULT_MEMBERS: CouncilMemberResult[] = [ - { model: 'reasoning-v1', response: 'Paris is the capital.', error: null }, - { model: 'reasoning-v1', response: 'France uses Paris as its capital.', error: null }, - { model: 'reasoning-v1', response: 'The answer is Paris.', error: null }, -]; - -const DEFAULT_COUNCIL: CouncilDefinition = { - id: 'default-council', - name: 'Default council', - description: 'Balanced analyst, builder, and skeptic jury.', - jury_count: 3, - debate_rounds: 3, - seats: [ - { - id: 0, - mode: 'default', - profile_id: '', - name: 'Analyst', - model: 'reasoning-v1', - brief: 'Evidence, assumptions, and risk.', - }, - { - id: 1, - mode: 'default', - profile_id: '', - name: 'Builder', - model: 'reasoning-v1', - brief: 'Practical implementation path.', - }, - { - id: 2, - mode: 'default', - profile_id: '', - name: 'Skeptic', - model: 'reasoning-v1', - brief: 'Failure modes and missing context.', - }, - ], - judge: { mode: 'default', profile_id: '', name: 'Chief Judge', model: 'reasoning-v1' }, - shared_reasoning: [ - '# Shared reasoning', - '- Claims the council agrees on:', - '- Open disagreements:', - '- Evidence or constraints to preserve:', - '- Judge synthesis notes:', - ].join('\n'), - created_at_ms: 1, - updated_at_ms: 1, -}; - -const fillQuestion = () => { - fireEvent.change(screen.getByLabelText('Question'), { - target: { value: 'What is the capital of France?' }, - }); -}; - -const mockProgressiveSuccess = (members: CouncilMemberResult[] = DEFAULT_MEMBERS) => { - mockAnswerMember.mockImplementation(async ({ model }: { model: string }) => { - const index = mockAnswerMember.mock.calls.length - 1; - return members[index] ?? { model, response: `answer ${index + 1}`, error: null }; - }); - mockSynthesizeCouncil.mockResolvedValue(RESULT); -}; - -const renderCouncilList = async () => { - render(); - await screen.findByRole('button', { name: 'Open council' }); -}; - -const renderOpenCouncil = async () => { - await renderCouncilList(); - fireEvent.click(screen.getByRole('button', { name: 'Open council' })); - await screen.findByLabelText('Question'); -}; - -const renderEditCouncil = async () => { - await renderOpenCouncil(); - fireEvent.click(screen.getByRole('button', { name: 'Edit current council' })); - await screen.findByLabelText('Council name'); -}; - -const saveCouncilSettings = async () => { - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Save council' })); - }); - await screen.findByLabelText('Question'); -}; - -describe('ModelCouncilTab', () => { - beforeEach(() => { - mockListCouncils.mockReset(); - mockUpsertCouncil.mockReset(); - mockDeleteCouncil.mockReset(); - mockAnswerMember.mockReset(); - mockSynthesizeCouncil.mockReset(); - mockLoadAISettings.mockReset(); - mockLoadLocalProviderSnapshot.mockReset(); - mockListProviderModels.mockReset(); - mockDispatch.mockReset(); - mockListCouncils.mockResolvedValue([DEFAULT_COUNCIL]); - mockUpsertCouncil.mockImplementation(async council => ({ - ...council, - id: council.id || 'saved', - })); - mockDeleteCouncil.mockResolvedValue(true); - mockLoadAISettings.mockResolvedValue({ - cloudProviders: [ - { - id: 'openai-id', - slug: 'openai', - label: 'OpenAI', - endpoint: 'https://api.openai.com/v1', - auth_style: 'bearer', - has_api_key: true, - }, - { - id: 'anthropic-id', - slug: 'anthropic', - label: 'Anthropic', - endpoint: 'https://api.anthropic.com/v1', - auth_style: 'anthropic', - has_api_key: false, - }, - ], - routing: {}, - }); - mockLoadLocalProviderSnapshot.mockResolvedValue({ - status: null, - diagnostics: null, - presets: null, - installedModels: [{ name: 'llama3.2:latest', chat_capable: true }], - }); - mockListProviderModels.mockImplementation(async (provider: string) => { - if (provider === 'openhuman') return [{ id: 'managed-reasoning' }]; - if (provider === 'openai') return [{ id: 'gpt-4o' }, { id: 'gpt-4o-mini' }]; - return []; - }); - }); - - it('renders the council list first, then opens the default council', async () => { - await renderCouncilList(); - - expect(screen.getByText('Councils')).toBeInTheDocument(); - expect(screen.getByText('Default council')).toBeInTheDocument(); - - fireEvent.click(screen.getByRole('button', { name: 'Open council' })); - - await screen.findByLabelText('Question'); - expect(screen.getByText('Default council')).toBeInTheDocument(); - expect(screen.queryByText('Council settings')).not.toBeInTheDocument(); - expect(screen.queryByLabelText('Debate turns')).not.toBeInTheDocument(); - expect(screen.queryByLabelText('Shared reasoning file')).not.toBeInTheDocument(); - expect(screen.getByRole('button', { name: 'Convene council' })).toBeInTheDocument(); - }); - - it('allows the default council to be deleted from the persisted registry', async () => { - await renderCouncilList(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Delete Default council' })); - }); - - expect(mockDeleteCouncil).toHaveBeenCalledWith('default-council'); - expect(screen.queryByText('Default council')).not.toBeInTheDocument(); - expect(screen.getByText('No councils yet. Add one to get started.')).toBeInTheDocument(); - }); - - it('uses the jury count setting to resize the roster up to five', async () => { - await renderEditCouncil(); - - expect(screen.queryByLabelText('Question')).not.toBeInTheDocument(); - expect(screen.queryByRole('button', { name: 'Convene council' })).not.toBeInTheDocument(); - - fireEvent.click(screen.getByRole('button', { name: '5' })); - - expect(screen.getAllByTestId('rive-mascot')).toHaveLength(5); - expect(screen.getAllByText('Juror 5')).toHaveLength(2); - expect(screen.getByLabelText('Juror 5 name')).toBeInTheDocument(); - }); - - it('disables Convene until a question is filled because seats and judge have defaults', async () => { - await renderOpenCouncil(); - - const run = screen.getByRole('button', { name: 'Convene council' }); - expect(run).toBeDisabled(); - fillQuestion(); - expect(run).not.toBeDisabled(); - }); - - it('shows mascot deliberation and agent thoughts while the council is running', async () => { - let resolveFirst: (value: CouncilMemberResult) => void = () => {}; - let resolveSecond: (value: CouncilMemberResult) => void = () => {}; - let resolveThird: (value: CouncilMemberResult) => void = () => {}; - let resolveSynthesis: (value: ModelCouncilResult) => void = () => {}; - mockAnswerMember - .mockImplementation(async ({ model }: { model: string }) => ({ - model, - response: `follow-up thought ${mockAnswerMember.mock.calls.length}`, - error: null, - })) - .mockReturnValueOnce( - new Promise(resolve => { - resolveFirst = resolve; - }) - ) - .mockReturnValueOnce( - new Promise(resolve => { - resolveSecond = resolve; - }) - ) - .mockReturnValueOnce( - new Promise(resolve => { - resolveThird = resolve; - }) - ); - mockSynthesizeCouncil.mockReturnValueOnce( - new Promise(resolve => { - resolveSynthesis = resolve; - }) - ); - await renderOpenCouncil(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - expect(screen.getByText('Council deliberation')).toBeInTheDocument(); - expect(screen.getAllByText('Thinking')).toHaveLength(3); - expect(screen.getByText('Judge')).toBeInTheDocument(); - expect( - screen.getByText(/Waiting for juror answers, then reading the shared reasoning file/) - ).toBeInTheDocument(); - expect(screen.getAllByTestId('rive-mascot')).toHaveLength(4); - expect(screen.getAllByTestId('rive-mascot')[0]).toHaveAttribute('data-face', 'thinking'); - - await act(async () => { - resolveFirst({ - model: 'reasoning-v1', - response: 'First juror live thought: Paris.', - error: null, - }); - }); - - expect(screen.getByText('First juror live thought: Paris.')).toBeInTheDocument(); - expect(screen.getByText('Round 1')).toBeInTheDocument(); - expect(screen.getAllByText('Thinking')).toHaveLength(2); - expect(screen.getByText('Answered')).toBeInTheDocument(); - expect(screen.getByText('Judge')).toBeInTheDocument(); - - await act(async () => { - resolveSecond({ model: 'reasoning-v1', response: 'Second juror agrees.', error: null }); - resolveThird({ model: 'reasoning-v1', response: 'Third juror agrees.', error: null }); - }); - - await waitFor(() => { - expect(screen.getByText('Synthesizing')).toBeInTheDocument(); - }); - - await act(async () => { - resolveSynthesis(RESULT); - }); - - await waitFor(() => { - expect(screen.queryByText('Council deliberation')).not.toBeInTheDocument(); - }); - }); - - it('streams failed juror status without blocking other juror thoughts', async () => { - let resolveFirst: (value: CouncilMemberResult) => void = () => {}; - let resolveSecond: (value: CouncilMemberResult) => void = () => {}; - let resolveThird: (value: CouncilMemberResult) => void = () => {}; - mockAnswerMember - .mockImplementation(async ({ model }: { model: string }) => ({ - model, - response: `follow-up answer ${mockAnswerMember.mock.calls.length}`, - error: null, - })) - .mockReturnValueOnce( - new Promise(resolve => { - resolveFirst = resolve; - }) - ) - .mockReturnValueOnce( - new Promise(resolve => { - resolveSecond = resolve; - }) - ) - .mockReturnValueOnce( - new Promise(resolve => { - resolveThird = resolve; - }) - ); - mockSynthesizeCouncil.mockResolvedValueOnce(RESULT); - await renderOpenCouncil(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - await act(async () => { - resolveFirst({ model: 'reasoning-v1', response: null, error: 'rate limited' }); - }); - - expect(screen.getByText('rate limited')).toBeInTheDocument(); - expect(screen.getByText('Failed')).toBeInTheDocument(); - expect(screen.getAllByText('Thinking')).toHaveLength(2); - - await act(async () => { - resolveSecond({ model: 'reasoning-v1', response: 'Second juror answer.', error: null }); - resolveThird({ model: 'reasoning-v1', response: 'Third juror answer.', error: null }); - }); - - await waitFor(() => { - expect(mockSynthesizeCouncil).toHaveBeenCalledWith({ - question: expect.any(String), - members: [ - { - model: 'reasoning-v1', - response: expect.stringContaining('[failed: rate limited]'), - error: null, - }, - { - model: 'reasoning-v1', - response: expect.stringContaining('Second juror answer.'), - error: null, - }, - { - model: 'reasoning-v1', - response: expect.stringContaining('Third juror answer.'), - error: null, - }, - ], - chair_model: 'reasoning-v1', - }); - }); - }); - - it('appends juror turns to the shared scratchpad before the next debate round', async () => { - mockAnswerMember.mockImplementation(async ({ model }: { model: string }) => ({ - model, - response: `round ${mockAnswerMember.mock.calls.length} update`, - error: null, - })); - mockSynthesizeCouncil.mockResolvedValueOnce(RESULT); - await renderOpenCouncil(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - await waitFor(() => { - expect(mockSynthesizeCouncil).toHaveBeenCalled(); - }); - expect(mockAnswerMember.mock.calls[3][0].question).toContain('Round 1 updates'); - expect(mockAnswerMember.mock.calls[3][0].question).toContain('round 1 update'); - }); - - it('lets a juror model be selected from routing hints', async () => { - mockProgressiveSuccess(); - await renderEditCouncil(); - - fireEvent.click(screen.getByLabelText('Member model 1')); - expect(screen.getByRole('dialog', { name: 'Member model 1' })).toBeInTheDocument(); - fireEvent.click(screen.getByRole('button', { name: /Reasoning/ })); - await saveCouncilSettings(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - expect(mockAnswerMember.mock.calls.map(call => call[0].model).slice(0, 3)).toEqual([ - 'hint:reasoning', - 'reasoning-v1', - 'reasoning-v1', - ]); - }); - - it('enables provider and model dropdowns only after choosing Custom', async () => { - await renderEditCouncil(); - - fireEvent.click(screen.getByLabelText('Member model 1')); - const dialog = screen.getByRole('dialog', { name: 'Member model 1' }); - const providerSelect = within(dialog).getByLabelText('Model provider'); - const modelSelect = within(dialog).getByLabelText('Model id'); - - expect(providerSelect).toBeDisabled(); - expect(modelSelect).toBeDisabled(); - - fireEvent.click(within(dialog).getByRole('button', { name: /Provider \+ model/ })); - - await waitFor(() => expect(providerSelect).not.toBeDisabled()); - expect( - within(providerSelect).getByRole('option', { name: 'Managed (openhuman)' }) - ).toBeInTheDocument(); - expect( - within(providerSelect).getByRole('option', { name: 'OpenAI (openai)' }) - ).toBeInTheDocument(); - expect( - within(providerSelect).queryByRole('option', { name: 'Anthropic (anthropic)' }) - ).not.toBeInTheDocument(); - - await waitFor(() => expect(modelSelect).not.toBeDisabled()); - expect( - within(modelSelect).getByRole('option', { name: 'managed-reasoning' }) - ).toBeInTheDocument(); - - fireEvent.change(providerSelect, { target: { value: 'openai' } }); - await waitFor(() => { - expect(within(modelSelect).getByRole('option', { name: 'gpt-4o' })).toBeInTheDocument(); - }); - fireEvent.change(modelSelect, { target: { value: 'gpt-4o' } }); - fireEvent.click(within(dialog).getByRole('button', { name: 'Use provider model' })); - - expect(screen.getByLabelText('Member model 1')).toHaveTextContent('openai:gpt-4o'); - }); - - it('lets a council seat use a saved profile and submits that profile model', async () => { - mockProgressiveSuccess(); - await renderEditCouncil(); - - const firstSeat = screen.getByLabelText('Juror 1 name').closest('article'); - expect(firstSeat).not.toBeNull(); - fireEvent.click(within(firstSeat as HTMLElement).getByRole('tab', { name: 'Profile' })); - fireEvent.change(screen.getByLabelText('Juror 1 profile'), { target: { value: 'critic' } }); - await saveCouncilSettings(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - expect(mockAnswerMember.mock.calls.map(call => call[0].model)).toEqual([ - 'critic-model', - 'reasoning-v1', - 'reasoning-v1', - 'critic-model', - 'reasoning-v1', - 'reasoning-v1', - 'critic-model', - 'reasoning-v1', - 'reasoning-v1', - ]); - expect(mockSynthesizeCouncil).toHaveBeenCalledWith({ - question: expect.stringContaining('shared_reasoning.md'), - members: expect.any(Array), - chair_model: 'reasoning-v1', - }); - expect(mockAnswerMember.mock.calls[0][0].question).toContain('User question:'); - expect(mockAnswerMember.mock.calls[0][0].question).toContain('What is the capital of France?'); - expect(mockAnswerMember.mock.calls[0][0].question).toContain('Debate round 1 of 3.'); - expect(mockAnswerMember.mock.calls[8][0].question).toContain('Debate round 3 of 3.'); - }); - - it('lets the judge agent use a saved profile unless a model override is typed', async () => { - mockProgressiveSuccess(); - await renderEditCouncil(); - - fireEvent.change(screen.getByLabelText('Judge agent'), { target: { value: 'profile' } }); - fireEvent.change(screen.getByLabelText('Judge profile'), { target: { value: 'critic' } }); - await saveCouncilSettings(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - expect(mockAnswerMember.mock.calls.map(call => call[0].model)).toEqual([ - 'reasoning-v1', - 'reasoning-v1', - 'reasoning-v1', - 'reasoning-v1', - 'reasoning-v1', - 'reasoning-v1', - 'reasoning-v1', - 'reasoning-v1', - 'reasoning-v1', - ]); - expect(mockSynthesizeCouncil).toHaveBeenCalledWith({ - question: expect.any(String), - members: expect.any(Array), - chair_model: 'critic-model', - }); - }); - - it('renders member answers side-by-side + the synthesis', async () => { - mockProgressiveSuccess(RESULT.members); - await renderOpenCouncil(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - await waitFor(() => { - expect(screen.getByText('Council results')).toBeInTheDocument(); - }); - expect(screen.getByText('Paris is the capital.')).toBeInTheDocument(); - expect(screen.getByText('rate limited')).toBeInTheDocument(); - expect(screen.getByText('Answered')).toBeInTheDocument(); - expect(screen.getByText('Failed')).toBeInTheDocument(); - expect( - screen.getByText('Both that answered agree: Paris. One seat failed.') - ).toBeInTheDocument(); - expect(screen.getByText('by claude-opus-4-8')).toBeInTheDocument(); - expect(screen.getByText('Debate usage')).toBeInTheDocument(); - expect(screen.getByText('Total')).toBeInTheDocument(); - }); - - it('renders council markdown instead of showing raw markdown markers', async () => { - mockProgressiveSuccess([ - { model: 'reasoning-v1', response: '**Paris** is the capital.', error: null }, - { model: 'reasoning-v1', response: '- France\n- Paris', error: null }, - { model: 'reasoning-v1', response: '`Paris` remains the answer.', error: null }, - ]); - mockSynthesizeCouncil.mockResolvedValueOnce({ - ...RESULT, - members: [ - { model: 'reasoning-v1', response: '**Paris** is the capital.', error: null }, - { model: 'reasoning-v1', response: '- France\n- Paris', error: null }, - ], - synthesis: '## Consensus\n\nThe answer is **Paris**.', - }); - await renderOpenCouncil(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - await waitFor(() => { - expect(screen.getByRole('heading', { name: 'Consensus' })).toBeInTheDocument(); - }); - const results = screen.getByText('Council results').closest('section'); - expect(results).not.toBeNull(); - expect(screen.getAllByText('Paris').some(node => node.tagName.toLowerCase() === 'strong')).toBe( - true - ); - expect(within(results as HTMLElement).queryByText(/\*\*Paris\*\*/)).not.toBeInTheDocument(); - }); - - it('surfaces an error alert when the council run fails', async () => { - mockAnswerMember.mockResolvedValue({ - model: 'reasoning-v1', - response: null, - error: 'downstream', - }); - mockSynthesizeCouncil.mockRejectedValueOnce(new Error('all member models failed to respond')); - await renderOpenCouncil(); - fillQuestion(); - - await act(async () => { - fireEvent.click(screen.getByRole('button', { name: 'Convene council' })); - }); - - await waitFor(() => { - const alert = screen.getByRole('alert'); - expect(alert.textContent).toMatch(/all member models failed to respond/); - }); - expect(screen.queryByText('Council results')).not.toBeInTheDocument(); - }); -}); diff --git a/app/src/components/intelligence/ModelCouncilTab.tsx b/app/src/components/intelligence/ModelCouncilTab.tsx deleted file mode 100644 index 2a09b011b..000000000 --- a/app/src/components/intelligence/ModelCouncilTab.tsx +++ /dev/null @@ -1,1837 +0,0 @@ -/** - * Model Council tab — configure a small council of agent-flavored model seats, - * ask one question, then let a judge model synthesize the responses. - * - * The Rust core still owns orchestration through `openhuman.model_council_run`. - * This surface gives each seat an agent profile, Rive presence, and council - * settings, then resolves the roster to model ids for the existing RPC. - */ -import { useCallback, useEffect, useMemo, useState } from 'react'; - -import { BubbleMarkdown } from '../../features/conversations/components/AgentMessageBubble'; -import { - getMascotPalette, - hexToArgbInt, - type MascotFace, - RiveMascot, -} from '../../features/human/Mascot'; -import { useT } from '../../lib/i18n/I18nContext'; -import { - listProviderModels, - loadAISettings, - loadLocalProviderSnapshot, - type ModelInfo, -} from '../../services/api/aiSettingsApi'; -import { type CouncilDefinition, councilRegistryApi } from '../../services/api/councilRegistryApi'; -import { - type CouncilMemberResult, - modelCouncilApi, - type ModelCouncilResult, -} from '../../services/api/modelCouncilApi'; -import { - type AgentProfilesStatus, - loadAgentProfiles, - selectAgentProfiles, -} from '../../store/agentProfileSlice'; -import { useAppDispatch, useAppSelector } from '../../store/hooks'; -import type { AgentProfile } from '../../types/agentProfile'; -import Button from '../ui/Button'; - -/** Matches the server-side MAX_COUNCIL_MEMBERS cap. */ -const MAX_MEMBERS = 5; -const MIN_MEMBERS = 1; -const MAX_DEBATE_ROUNDS = 4; -const MIN_DEBATE_ROUNDS = 2; -const DEFAULT_REASONING_MODEL = 'reasoning-v1'; - -type SeatMode = 'default' | 'profile' | 'custom'; - -interface CouncilSeat { - id: number; - mode: SeatMode; - profileId: string; - name: string; - model: string; - brief: string; -} - -interface ResolvedSeat { - label: string; - model: string; - brief: string; -} - -interface LiveMemberThought { - status: 'pending' | 'answered' | 'failed'; - member: CouncilMemberResult | null; - turns: CouncilDebateTurn[]; -} - -interface CouncilDebateTurn { - round: number; - response: string | null; - error: string | null; -} - -interface DebateUsageEstimate { - inputTokens: number; - outputTokens: number; - totalTokens: number; -} - -interface ModelPickerState { - title: string; - value: string; - onSelect: (model: string) => void; -} - -// `value` is the stable hint identifier; `labelKey` is resolved via `useT()` -// at render (this is a module-level const with no hook scope of its own). -const MODEL_HINTS = [ - { value: 'default', labelKey: 'modelCouncil.hint.default' }, - { value: 'hint:chat', labelKey: 'modelCouncil.hint.chat' }, - { value: 'hint:reasoning', labelKey: 'modelCouncil.hint.reasoning' }, - { value: 'hint:code', labelKey: 'modelCouncil.hint.code' }, - { value: 'hint:summarize', labelKey: 'modelCouncil.hint.summarize' }, -] as const; - -interface ConnectedModelProvider { - slug: string; - label: string; - models?: ModelInfo[]; -} - -function parseProviderModel(value: string): { provider: string; model: string } { - const trimmed = value.trim(); - const colon = trimmed.indexOf(':'); - if (colon <= 0) { - return { provider: '', model: trimmed }; - } - return { provider: trimmed.slice(0, colon), model: trimmed.slice(colon + 1) }; -} - -async function loadConnectedModelProviders(): Promise { - const [settings, localSnapshot] = await Promise.all([ - loadAISettings(), - loadLocalProviderSnapshot().catch(() => null), - ]); - const providers: ConnectedModelProvider[] = [{ slug: 'openhuman', label: 'Managed' }]; - const seen = new Set(providers.map(provider => provider.slug)); - - for (const provider of settings.cloudProviders) { - const slug = provider.slug.trim(); - if (!slug || seen.has(slug)) continue; - if (!provider.has_api_key && provider.auth_style !== 'none') continue; - providers.push({ slug, label: provider.label || slug }); - seen.add(slug); - } - - const localModels = - localSnapshot?.installedModels - .filter(model => model.chat_capable !== false) - .map(model => ({ id: model.name, context_window: model.context_length ?? null })) ?? []; - if (localModels.length > 0 && !seen.has('ollama')) { - providers.push({ slug: 'ollama', label: 'Ollama', models: localModels }); - } - - return providers; -} - -const Icon = ({ - name, - size = 16, -}: { - name: 'arrow-left' | 'plus' | 'settings' | 'trash'; - size?: number; -}) => { - const common = { - fill: 'none', - stroke: 'currentColor', - strokeLinecap: 'round' as const, - strokeLinejoin: 'round' as const, - strokeWidth: 2, - }; - return ( - - ); -}; - -const ModelPickerDialog = ({ - picker, - onClose, -}: { - picker: ModelPickerState; - onClose: () => void; -}) => { - const { t } = useT(); - const initial = parseProviderModel(picker.value); - const initialHint = MODEL_HINTS.some(hint => hint.value === picker.value); - const [selectionMode, setSelectionMode] = useState<'hint' | 'custom'>( - initial.provider && !initialHint ? 'custom' : 'hint' - ); - const [providers, setProviders] = useState([]); - const [providersLoading, setProvidersLoading] = useState(false); - const [providersError, setProvidersError] = useState(null); - const [provider, setProvider] = useState(initial.provider); - const [models, setModels] = useState([]); - const [modelsLoading, setModelsLoading] = useState(false); - const [modelsError, setModelsError] = useState(null); - const [model, setModel] = useState(initial.model); - - useEffect(() => { - let active = true; - setProvidersLoading(true); - setProvidersError(null); - loadConnectedModelProviders() - .then(loaded => { - if (!active) return; - setProviders(loaded); - setProvidersLoading(false); - setProvider(current => { - if (current && loaded.some(item => item.slug === current)) return current; - return loaded[0]?.slug ?? ''; - }); - }) - .catch(err => { - if (!active) return; - setProvidersError(err instanceof Error ? err.message : String(err)); - setProvidersLoading(false); - }); - return () => { - active = false; - }; - }, []); - - useEffect(() => { - if (selectionMode !== 'custom' || !provider) { - setModels([]); - setModelsError(null); - return; - } - - const connectedProvider = providers.find(item => item.slug === provider); - if (!connectedProvider) { - setModels([]); - setModelsError(null); - return; - } - - if (connectedProvider.models) { - setModels(connectedProvider.models); - setModelsError(null); - setModelsLoading(false); - setModel(current => current || connectedProvider.models?.[0]?.id || ''); - return; - } - - let active = true; - setModelsLoading(true); - setModels([]); - setModelsError(null); - listProviderModels(provider) - .then(loaded => { - if (!active) return; - setModels(loaded); - setModelsLoading(false); - setModel(current => { - if (current && loaded.some(item => item.id === current)) return current; - return loaded[0]?.id ?? ''; - }); - }) - .catch(err => { - if (!active) return; - setModelsError(err instanceof Error ? err.message : String(err)); - setModelsLoading(false); - }); - return () => { - active = false; - }; - }, [provider, providers, selectionMode]); - - const saveProviderModel = () => { - const trimmedModel = model.trim(); - if (!trimmedModel) return; - const trimmedProvider = provider.trim(); - picker.onSelect(trimmedProvider ? `${trimmedProvider}:${trimmedModel}` : trimmedModel); - onClose(); - }; - - return ( -
-
-
-
-

- {picker.title} -

-

{t('modelCouncil.modelPickerHelp')}

-
- -
- -
-

- {t('modelCouncil.modelPickerHints')} -

-
- {MODEL_HINTS.map(hint => ( - - ))} -
-
- -
- - -
- - -
- {(providersLoading || modelsLoading) && ( -

{t('skills.resource.preview.loading')}

- )} - {(providersError || modelsError) && ( -

- {providersError || modelsError} -

- )} - -
-
-
- ); -}; - -const DEFAULT_MODEL = DEFAULT_REASONING_MODEL; -const DEFAULT_JUDGE_MODEL = DEFAULT_REASONING_MODEL; -const SHARED_REASONING_FILE = 'shared_reasoning.md'; -const DEFAULT_SHARED_REASONING = [ - '# Shared reasoning', - '- Claims the council agrees on:', - '- Open disagreements:', - '- Evidence or constraints to preserve:', - '- Judge synthesis notes:', -].join('\n'); -const DEFAULT_SEATS: CouncilSeat[] = [ - { - id: 0, - mode: 'default', - profileId: '', - name: 'Analyst', - model: DEFAULT_MODEL, - brief: 'Evidence, assumptions, and risk.', - }, - { - id: 1, - mode: 'default', - profileId: '', - name: 'Builder', - model: DEFAULT_MODEL, - brief: 'Practical implementation path.', - }, - { - id: 2, - mode: 'default', - profileId: '', - name: 'Skeptic', - model: DEFAULT_MODEL, - brief: 'Failure modes and missing context.', - }, -]; - -const SEAT_COLORS = ['yellow', 'burgundy', 'navy', 'black', 'yellow'] as const; -const SEAT_FACES: MascotFace[] = ['thinking', 'writing', 'reading', 'curious', 'proud']; -const ACTIVE_SEAT_FACES: MascotFace[] = ['thinking', 'writing', 'thinking', 'reading', 'curious']; - -const nextSeatId = (seats: CouncilSeat[]): number => - seats.reduce((max, seat) => Math.max(max, seat.id), -1) + 1; - -function profileLabel(profile: AgentProfile): string { - return profile.modelOverride ? `${profile.name} · ${profile.modelOverride}` : profile.name; -} - -function profileModel(profile: AgentProfile | undefined): string { - return profile?.modelOverride?.trim() || profile?.agentId?.trim() || profile?.id?.trim() || ''; -} - -function resolveSeat(seat: CouncilSeat, profiles: AgentProfile[], index: number): ResolvedSeat { - const profile = profiles.find(p => p.id === seat.profileId); - const fallbackName = - seat.mode === 'profile' && profile ? profile.name : seat.name.trim() || `Juror ${index + 1}`; - const fallbackModel = seat.mode === 'profile' ? profileModel(profile) : DEFAULT_MODEL; - - return { - label: fallbackName, - model: seat.model.trim() || fallbackModel, - brief: seat.brief.trim(), - }; -} - -function mascotColors(index: number) { - const palette = getMascotPalette(SEAT_COLORS[index % SEAT_COLORS.length]); - return { - primaryColor: hexToArgbInt(palette.bodyFill), - secondaryColor: hexToArgbInt(palette.neckShadowColor), - }; -} - -function deliberationThought( - seat: ResolvedSeat, - index: number, - t: (key: string) => string -): string { - const brief = seat.brief.trim(); - if (brief) { - return t('modelCouncil.thinkingWithBrief').replace('{brief}', brief); - } - - const keys = [ - 'modelCouncil.thought.evidence', - 'modelCouncil.thought.plan', - 'modelCouncil.thought.risk', - 'modelCouncil.thought.tradeoffs', - 'modelCouncil.thought.synthesis', - ]; - return t(keys[index % keys.length]); -} - -function buildCouncilQuestion( - question: string, - sharedReasoning: string, - seats: ResolvedSeat[], - judgeName: string -): string { - const trimmedQuestion = question.trim(); - const trimmedSharedReasoning = sharedReasoning.trim(); - const roster = seats - .map((seat, index) => { - const brief = seat.brief ? ` — ${seat.brief}` : ''; - return `${index + 1}. ${seat.label} (${seat.model})${brief}`; - }) - .join('\n'); - const commonPrefix = [ - `Council workspace: ${SHARED_REASONING_FILE}`, - 'Use this shared reasoning file as the common deliberation scratchpad.', - '', - 'Council roster:', - roster, - '', - `Judge agent: ${judgeName}`, - ]; - - if (!trimmedSharedReasoning) { - return [...commonPrefix, '', 'User question:', trimmedQuestion].join('\n'); - } - - return [ - ...commonPrefix, - '', - `${SHARED_REASONING_FILE}:`, - trimmedSharedReasoning, - '', - 'User question:', - trimmedQuestion, - ].join('\n'); -} - -function buildDebateTurnQuestion( - baseQuestion: string, - seat: ResolvedSeat, - round: number, - totalRounds: number, - transcript: CouncilDebateTurn[][], - t: (key: string) => string -): string { - const previousTurns = transcript - .map((turns, seatIndex) => { - if (turns.length === 0) return ''; - const body = turns - .map(turn => { - const text = turn.response || `[${turn.error || 'no response'}]`; - return `Round ${turn.round}: ${text}`; - }) - .join('\n'); - return `Juror ${seatIndex + 1} previous turns:\n${body}`; - }) - .filter(Boolean) - .join('\n\n'); - - const phase = - round === totalRounds - ? t('modelCouncil.debateFinalInstruction') - : t('modelCouncil.debateRoundInstruction'); - - return [ - baseQuestion, - '', - `Debate round ${round} of ${totalRounds}.`, - `You are ${seat.label}. Perspective: ${seat.brief || 'independent council juror'}.`, - phase, - previousTurns ? ['', 'Debate so far:', previousTurns].join('\n') : '', - '', - 'Write this turn as a concise council thought plus your current conclusion.', - ] - .filter(Boolean) - .join('\n'); -} - -function appendScratchpadRound( - scratchpad: string, - round: number, - seats: ResolvedSeat[], - roundResults: Array<{ index: number; turn: CouncilDebateTurn }>, - t: (key: string) => string -): string { - const existing = scratchpad.trim() || '# Shared reasoning'; - const lines = [ - '', - '', - `## ${t('modelCouncil.scratchpadRoundHeading').replace('{round}', String(round))}`, - ]; - for (const { index, turn } of [...roundResults].sort((a, b) => a.index - b.index)) { - const seat = seats[index]; - lines.push('', `### ${seat?.label || `Juror ${index + 1}`}`); - if (turn.response) { - lines.push(turn.response.trim()); - } else { - lines.push(`_${t('modelCouncil.scratchpadNoResponse')}: ${turn.error || 'unknown'}_`); - } - } - return `${existing}${lines.join('\n')}`; -} - -function estimateTokens(text: string): number { - return Math.max(1, Math.ceil(text.length / 4)); -} - -function formatTokenCount(value: number): string { - return new Intl.NumberFormat(undefined, { maximumFractionDigits: 0 }).format(value); -} - -function buildMemberSynthesisInput( - seat: ResolvedSeat, - model: string, - turns: CouncilDebateTurn[] -): CouncilMemberResult { - const answeredTurns = turns.filter(turn => turn.response); - if (answeredTurns.length === 0) { - return { - model, - response: null, - error: turns.find(turn => turn.error)?.error || 'no debate turns completed', - }; - } - - return { - model, - response: [ - `${seat.label} debate record:`, - ...turns.map(turn => { - const text = turn.response || `[failed: ${turn.error || 'unknown'}]`; - return `Round ${turn.round}: ${text}`; - }), - ].join('\n\n'), - error: null, - }; -} - -function councilSeatsFromDefinition(council: CouncilDefinition): CouncilSeat[] { - return council.seats.map(seat => ({ - id: seat.id, - mode: seat.mode, - profileId: seat.profile_id, - name: seat.name, - model: seat.model, - brief: seat.brief, - })); -} - -function createDraftCouncil(): CouncilDefinition { - const now = Date.now(); - return { - id: '', - name: 'New council', - description: '', - jury_count: 3, - debate_rounds: 3, - seats: DEFAULT_SEATS.map(seat => ({ - id: seat.id, - mode: seat.mode, - profile_id: seat.profileId, - name: seat.name, - model: seat.model, - brief: seat.brief, - })), - judge: { mode: 'default', profile_id: '', name: 'Chief Judge', model: DEFAULT_JUDGE_MODEL }, - shared_reasoning: DEFAULT_SHARED_REASONING, - created_at_ms: now, - updated_at_ms: now, - }; -} - -const ModelCouncilTab = () => { - const { t } = useT(); - const dispatch = useAppDispatch(); - const profiles = useAppSelector(selectAgentProfiles); - const profileStatus = useAppSelector(state => state.agentProfiles.status as AgentProfilesStatus); - - const [question, setQuestion] = useState(''); - const [view, setView] = useState<'list' | 'run' | 'edit'>('list'); - const [councils, setCouncils] = useState([]); - const [selectedCouncil, setSelectedCouncil] = useState(null); - const [councilName, setCouncilName] = useState('Default council'); - const [councilDescription, setCouncilDescription] = useState(''); - const [registryLoading, setRegistryLoading] = useState(true); - const [registrySaving, setRegistrySaving] = useState(false); - const [registryError, setRegistryError] = useState(null); - const [sharedReasoning, setSharedReasoning] = useState(DEFAULT_SHARED_REASONING); - const [liveScratchpad, setLiveScratchpad] = useState(null); - const [juryCount, setJuryCount] = useState(3); - const [debateRounds, setDebateRounds] = useState(3); - const [seats, setSeats] = useState(DEFAULT_SEATS); - const [judgeMode, setJudgeMode] = useState('default'); - const [judgeProfileId, setJudgeProfileId] = useState(''); - const [judgeName, setJudgeName] = useState('Chief Judge'); - const [judgeModel, setJudgeModel] = useState(DEFAULT_JUDGE_MODEL); - const [running, setRunning] = useState(false); - const [liveMembers, setLiveMembers] = useState([]); - const [judgeSynthesizing, setJudgeSynthesizing] = useState(false); - const [usageEstimate, setUsageEstimate] = useState(null); - const [modelPicker, setModelPicker] = useState(null); - const [result, setResult] = useState(null); - const [error, setError] = useState(null); - - useEffect(() => { - if (profileStatus === 'idle' && profiles.length === 0) { - void dispatch(loadAgentProfiles()); - } - }, [dispatch, profileStatus, profiles.length]); - - const applyCouncilDefinition = useCallback((council: CouncilDefinition) => { - setSelectedCouncil(council); - setCouncilName(council.name || 'Untitled council'); - setCouncilDescription(council.description || ''); - setJuryCount(Math.min(MAX_MEMBERS, Math.max(MIN_MEMBERS, council.jury_count || 3))); - setDebateRounds( - Math.min(MAX_DEBATE_ROUNDS, Math.max(MIN_DEBATE_ROUNDS, council.debate_rounds || 3)) - ); - setSeats(councilSeatsFromDefinition(council).slice(0, council.jury_count || 3)); - setJudgeMode(council.judge.mode); - setJudgeProfileId(council.judge.profile_id || ''); - setJudgeName(council.judge.name || 'Chief Judge'); - setJudgeModel(council.judge.model ?? DEFAULT_JUDGE_MODEL); - setSharedReasoning(council.shared_reasoning || DEFAULT_SHARED_REASONING); - setQuestion(''); - setLiveMembers([]); - setLiveScratchpad(null); - setJudgeSynthesizing(false); - setUsageEstimate(null); - setResult(null); - setError(null); - }, []); - - const loadCouncils = useCallback(async () => { - setRegistryLoading(true); - setRegistryError(null); - try { - const loaded = await councilRegistryApi.list(); - setCouncils(loaded); - } catch (err) { - setRegistryError(err instanceof Error ? err.message : String(err)); - } finally { - setRegistryLoading(false); - } - }, []); - - useEffect(() => { - void loadCouncils(); - }, [loadCouncils]); - - useEffect(() => { - setSeats(prev => { - if (prev.length === juryCount) return prev; - if (prev.length > juryCount) return prev.slice(0, juryCount); - - const next = [...prev]; - while (next.length < juryCount) { - const index = next.length; - next.push({ - id: nextSeatId(next), - mode: 'default', - profileId: '', - name: `${t('modelCouncil.jurorFallback')} ${index + 1}`, - model: DEFAULT_MODEL, - brief: '', - }); - } - return next; - }); - }, [juryCount, t]); - - const judgeProfile = useMemo( - () => profiles.find(profile => profile.id === judgeProfileId), - [profiles, judgeProfileId] - ); - - const resolvedSeats = useMemo( - () => seats.map((seat, index) => resolveSeat(seat, profiles, index)), - [profiles, seats] - ); - - const resolvedJudgeModel = - judgeModel.trim() || - (judgeMode === 'profile' ? profileModel(judgeProfile) : '') || - DEFAULT_JUDGE_MODEL; - const resolvedJudgeName = - judgeMode === 'profile' && judgeProfile ? judgeProfile.name : judgeName.trim() || 'Chief Judge'; - - const canRun = - !running && - question.trim().length > 0 && - resolvedSeats.some(seat => seat.model.trim().length > 0) && - resolvedJudgeModel.trim().length > 0; - - const updateSeat = useCallback((id: number, patch: Partial) => { - setSeats(prev => prev.map(seat => (seat.id === id ? { ...seat, ...patch } : seat))); - }, []); - - const buildCouncilDefinition = useCallback( - (base: CouncilDefinition | null): CouncilDefinition => { - const now = Date.now(); - return { - id: base?.id || '', - name: councilName.trim() || 'Untitled council', - description: councilDescription.trim(), - jury_count: juryCount, - debate_rounds: debateRounds, - seats: seats - .slice(0, juryCount) - .map(seat => ({ - id: seat.id, - mode: seat.mode, - profile_id: seat.profileId, - name: seat.name, - model: seat.model, - brief: seat.brief, - })), - judge: { mode: judgeMode, profile_id: judgeProfileId, name: judgeName, model: judgeModel }, - shared_reasoning: sharedReasoning, - created_at_ms: base?.created_at_ms || now, - updated_at_ms: now, - }; - }, - [ - councilDescription, - councilName, - debateRounds, - judgeMode, - judgeModel, - judgeName, - judgeProfileId, - juryCount, - seats, - sharedReasoning, - ] - ); - - const saveCouncil = useCallback(async () => { - setRegistrySaving(true); - setRegistryError(null); - try { - const saved = await councilRegistryApi.upsert(buildCouncilDefinition(selectedCouncil)); - setCouncils(prev => { - const without = prev.filter(council => council.id !== saved.id); - return [saved, ...without].sort((a, b) => a.name.localeCompare(b.name)); - }); - applyCouncilDefinition(saved); - setView('run'); - } catch (err) { - setRegistryError(err instanceof Error ? err.message : String(err)); - } finally { - setRegistrySaving(false); - } - }, [applyCouncilDefinition, buildCouncilDefinition, selectedCouncil]); - - const handleSelectCouncil = useCallback( - (council: CouncilDefinition) => { - applyCouncilDefinition(council); - setView('run'); - }, - [applyCouncilDefinition] - ); - - const handleCreateCouncil = useCallback(() => { - applyCouncilDefinition(createDraftCouncil()); - setView('edit'); - }, [applyCouncilDefinition]); - - const selectedCouncilId = selectedCouncil?.id; - const handleDeleteCouncil = useCallback( - async (council: CouncilDefinition) => { - setRegistryError(null); - try { - await councilRegistryApi.delete(council.id); - setCouncils(prev => prev.filter(item => item.id !== council.id)); - if (selectedCouncilId === council.id) { - setSelectedCouncil(null); - setView('list'); - } - } catch (err) { - setRegistryError(err instanceof Error ? err.message : String(err)); - } - }, - [selectedCouncilId] - ); - - const setSeatMode = useCallback( - (seat: CouncilSeat, mode: SeatMode) => { - updateSeat(seat.id, { - mode, - profileId: mode === 'profile' ? seat.profileId || profiles[0]?.id || '' : '', - name: mode === 'custom' ? seat.name : seat.name || '', - model: mode === 'profile' ? '' : seat.model || DEFAULT_MODEL, - }); - }, - [profiles, updateSeat] - ); - - const handleRun = useCallback(async () => { - if (running) return; - const memberModels = resolvedSeats.map(seat => seat.model.trim()).filter(Boolean); - const chairModel = resolvedJudgeModel.trim(); - if (question.trim().length === 0 || memberModels.length === 0 || chairModel.length === 0) { - return; - } - setRunning(true); - setJudgeSynthesizing(false); - setLiveMembers(memberModels.map(() => ({ status: 'pending', member: null, turns: [] }))); - setLiveScratchpad(sharedReasoning); - setUsageEstimate(null); - setError(null); - setResult(null); - try { - const transcript: CouncilDebateTurn[][] = memberModels.map(() => []); - let currentScratchpad = sharedReasoning; - let estimatedInputTokens = 0; - let estimatedOutputTokens = 0; - - for (let round = 1; round <= debateRounds; round += 1) { - setLiveMembers(prev => prev.map(entry => ({ ...entry, status: 'pending' }))); - - const roundResults = await Promise.all( - memberModels.map(async (model, index) => { - const councilQuestion = buildCouncilQuestion( - question, - currentScratchpad, - resolvedSeats, - resolvedJudgeName - ); - const turnQuestion = buildDebateTurnQuestion( - councilQuestion, - resolvedSeats[index], - round, - debateRounds, - transcript, - t - ); - estimatedInputTokens += estimateTokens(turnQuestion); - try { - const member = await modelCouncilApi.answerMember({ question: turnQuestion, model }); - const turn: CouncilDebateTurn = { - round, - response: member.response, - error: member.error, - }; - estimatedOutputTokens += estimateTokens(member.response || member.error || ''); - setLiveMembers(prev => - prev.map((entry, entryIndex) => - entryIndex === index - ? { - status: member.error ? 'failed' : 'answered', - member, - turns: [...entry.turns, turn], - } - : entry - ) - ); - return { index, turn }; - } catch (memberError) { - const errorText = - memberError instanceof Error ? memberError.message : String(memberError); - const failedMember: CouncilMemberResult = { model, response: null, error: errorText }; - const turn: CouncilDebateTurn = { round, response: null, error: errorText }; - estimatedOutputTokens += estimateTokens(errorText); - setLiveMembers(prev => - prev.map((entry, entryIndex) => - entryIndex === index - ? { status: 'failed', member: failedMember, turns: [...entry.turns, turn] } - : entry - ) - ); - return { index, turn }; - } - }) - ); - - for (const { index, turn } of roundResults) { - transcript[index].push(turn); - } - currentScratchpad = appendScratchpadRound( - currentScratchpad, - round, - resolvedSeats, - roundResults, - t - ); - setLiveScratchpad(currentScratchpad); - setSharedReasoning(currentScratchpad); - } - - const councilQuestion = buildCouncilQuestion( - question, - currentScratchpad, - resolvedSeats, - resolvedJudgeName - ); - const memberResults = memberModels.map((model, index) => - buildMemberSynthesisInput(resolvedSeats[index], model, transcript[index]) - ); - const synthesisInputTokens = estimateTokens( - `${councilQuestion}\n${JSON.stringify(memberResults)}` - ); - estimatedInputTokens += synthesisInputTokens; - setJudgeSynthesizing(true); - const res = await modelCouncilApi.synthesizeCouncil({ - question: councilQuestion, - members: memberResults, - chair_model: chairModel, - }); - estimatedOutputTokens += estimateTokens(res.synthesis); - const totalTokens = estimatedInputTokens + estimatedOutputTokens; - setUsageEstimate({ - inputTokens: estimatedInputTokens, - outputTokens: estimatedOutputTokens, - totalTokens, - }); - setResult(res); - } catch (err) { - setError(err instanceof Error ? err.message : String(err)); - } finally { - setJudgeSynthesizing(false); - setRunning(false); - setLiveScratchpad(null); - } - }, [ - debateRounds, - resolvedJudgeModel, - resolvedJudgeName, - resolvedSeats, - question, - running, - sharedReasoning, - t, - ]); - - if (view === 'list') { - return ( -
-
-
-

{t('modelCouncil.listTitle')}

-

- {t('modelCouncil.listIntro')} -

-
- -
- - {registryError && ( -

- {t('modelCouncil.registryErrorPrefix')} {registryError} -

- )} - - {registryLoading ? ( -
- {t('modelCouncil.loadingCouncils')} -
- ) : councils.length === 0 ? ( -
- {t('modelCouncil.noCouncils')} -
- ) : ( -
- {councils.map(council => ( -
-
-
-

{council.name}

-

- {council.description || t('modelCouncil.noCouncilDescription')} -

-
-
- - -
-
-
-
-
{t('modelCouncil.juryCountLabel')}
-
{council.jury_count}
-
-
-
{t('modelCouncil.debateRoundsLabel')}
-
- {council.debate_rounds} -
-
-
- -
- ))} -
- )} -
- ); - } - - return ( -
-
-
- -
-

- {view === 'edit' ? t('modelCouncil.editCouncil') : councilName} -

- {selectedCouncil && view === 'run' && ( -

- {councilDescription || t('modelCouncil.noCouncilDescription')} -

- )} -
-
-
- {view === 'edit' ? ( - <> - - - - ) : ( - - )} -
-
- - {registryError && ( -

- {t('modelCouncil.registryErrorPrefix')} {registryError} -

- )} - - {view === 'edit' && ( -
-
- - setCouncilName(e.target.value)} - className="w-full rounded-lg border border-line bg-surface px-3 py-2 text-sm text-content shadow-sm focus:outline-none focus:ring-2 focus:ring-primary-400 dark:bg-surface-canvas" - /> -
-
- -