mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
feat(voice): standalone voice dictation server with hotkey support (#368)
* feat: add standalone voice dictation server with hotkey support - Introduced a new `voice` subcommand to the CLI for running a standalone voice dictation server that listens for a hotkey, records audio, transcribes it using Whisper, and inserts the result into the active text field. - Implemented configuration options for the voice server, including hotkey combination, activation mode (tap or push), and an option to skip LLM post-processing. - Added audio capture functionality using the `cpal` crate and integrated hotkey listening with the `rdev` crate for global key event handling. - Enhanced the configuration schema to include voice server settings and updated the main configuration structure accordingly. - Updated relevant modules and tests to ensure consistent behavior and functionality across the application. This feature enhances user interaction by allowing voice dictation directly into any active text field, improving accessibility and usability. * feat: add voice dictation server with hotkey support - Introduced a standalone voice dictation server that listens for a configurable hotkey to start recording audio, transcribes it using whisper, and inserts the transcribed text into the active text field. - Added CLI support for the `voice` command, allowing users to manage the voice server's configuration, including hotkey and activation mode settings. - Implemented configuration structures for the voice server, including options for automatic start, hotkey combination, activation mode, and cleanup behavior. - Enhanced audio capture functionality using the `cpal` library for microphone input and integrated text insertion using the `enigo` library for simulating keyboard input. - Updated relevant modules and schemas to support the new voice server features, ensuring a cohesive integration within the OpenHuman platform. * refactor: streamline voice server command and enhance audio capture functionality - Updated the `run_voice_server_command` function to initialize the configuration with environment overrides instead of loading from a file, improving performance and flexibility. - Refactored the audio capture logic in `start_recording` to enhance thread management and error handling, ensuring a more robust audio stream setup. - Improved the handling of audio stream creation and playback, ensuring that all cpal objects are managed on the same thread as required, enhancing stability during recording operations. * fix: remove unused import in voice server module - Eliminated the `HotkeyListenerHandle` import from the `server.rs` file, streamlining the code and improving clarity by removing unnecessary dependencies. * feat(voice): auto-enable LLM cleanup when local model is ready The postprocessor now checks the local LLM state and automatically enables transcription cleanup when the model is downloaded and ready, even if not explicitly configured. Falls back gracefully to raw text when the LLM is unavailable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: apply cargo fmt + prettier formatting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(voice): dictation config, hotkey lifecycle, and WebSocket streaming (#332) Add the foundational infrastructure for voice dictation (EPIC #332): **Rust core:** - New `DictationConfig` schema with serde defaults and env var overrides (enabled, hotkey, activation_mode, llm_refinement, streaming, interval) - RPC controllers: `config_get_dictation_settings` / `config_update_dictation_settings` - WebSocket endpoint `/ws/dictation` for streaming PCM16 transcription with periodic partial inference and final LLM refinement - Microphone permission declaration (`NSMicrophoneUsageDescription`) in Tauri macOS bundle config **Frontend:** - `useDictationHotkey` hook: fetches config from core RPC, auto-registers global hotkey, listens for `dictation://toggle` events - `DictationHotkeyManager` headless component mounted in App.tsx - Fix voice RPC response type mismatch: voice handlers return flat results (no `{result, logs}` wrapper), so remove incorrect `CommandResponse<T>` wrapping from `openhumanVoiceStatus`, `openhumanVoiceTranscribe`, `openhumanVoiceTranscribeBytes`, and `openhumanVoiceTts` Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(tauri): remove invalid infoPlist config that breaks tauri dev The `infoPlist` field in tauri.conf.json expects a string path, not an inline object. Remove it for now — microphone permission will be added via a proper Info.plist supplement in the production build pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: apply Prettier formatting to dictation files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * format files * feat(dictation): integrate dictation listener and event broadcasting - Added a global dictation hotkey listener that activates based on configuration. - Implemented a web channel bridge to handle dictation events and broadcast them to connected clients. - Updated the voice module to include the new dictation listener functionality. This enhances the voice dictation capabilities by ensuring real-time event handling and client communication. * update code * format * feat(voice): enhance voice server configuration and functionality - Updated `Cargo.toml` to mark voice-related dependencies as optional. - Introduced `VoiceActivationMode` enum for better control over voice server activation. - Refactored voice server command handling and dictation event broadcasting to support new features. - Added conditional compilation for voice features across various modules, ensuring they are only included when enabled. This commit improves the modularity and configurability of the voice server, allowing for more flexible integration and usage. * refactor: clean up whitespace and formatting in core and voice modules - Removed unnecessary blank lines in `cli.rs`, `jsonrpc.rs`, `schemas.rs`, and `socketio.rs` to improve code readability. - Adjusted import order in `mod.rs` for better organization. This commit enhances the overall code quality by ensuring consistent formatting across multiple files. * chore: update Dockerfile and test workflow to install additional system dependencies - Added installation of system dependencies (cmake, ALSA, X11) in the Dockerfile for improved build support. - Updated the GitHub Actions workflow to reflect the new dependencies, ensuring consistent environment setup for testing. This commit enhances the build environment by including necessary libraries for audio and GUI support. * format * fix claude * format --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: oxoxDev <nikhil@tinyhumans.ai>
This commit is contained in:
Steven Enamakel
co-authored by
Claude Opus 4.6
oxoxDev
Claude Opus 4.6
oxoxDev
parent
07b1df4f24
commit
11f718d8bc
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"attribution": {
|
||||
"commit": "",
|
||||
"pr": ""
|
||||
}
|
||||
}
|
||||
@@ -45,6 +45,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
webkit2gtk-driver \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install system dependencies (cmake, ALSA, X11)
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends cmake libasound2-dev libxdo-dev libxtst-dev libx11-dev libevdev-dev && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# sccache (pre-installed so the action only needs to configure it)
|
||||
RUN curl -fsSL https://github.com/mozilla/sccache/releases/download/v0.10.0/sccache-v0.10.0-x86_64-unknown-linux-musl.tar.gz \
|
||||
| tar xz -C /usr/local/bin --strip-components=1 sccache-v0.10.0-x86_64-unknown-linux-musl/sccache \
|
||||
|
||||
@@ -85,8 +85,8 @@ jobs:
|
||||
app/src-tauri -> target
|
||||
cache-on-failure: true
|
||||
|
||||
- name: Install cmake (for whisper-rs)
|
||||
run: apt-get update && apt-get install -y --no-install-recommends cmake && rm -rf /var/lib/apt/lists/*
|
||||
- name: Install system dependencies (cmake, ALSA, X11)
|
||||
run: apt-get update && apt-get install -y --no-install-recommends cmake libasound2-dev libxdo-dev libxtst-dev libx11-dev libevdev-dev && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
- name: Check formatting (cargo fmt)
|
||||
run: cargo fmt --all -- --check
|
||||
@@ -136,7 +136,8 @@ jobs:
|
||||
libgtk-3-dev libwebkit2gtk-4.1-dev libappindicator3-dev \
|
||||
librsvg2-dev patchelf \
|
||||
xvfb at-spi2-core dbus-x11 \
|
||||
webkit2gtk-driver
|
||||
webkit2gtk-driver \
|
||||
libasound2-dev libxdo-dev libxtst-dev libx11-dev libevdev-dev
|
||||
|
||||
- name: Cargo.lock fingerprint (deps only)
|
||||
id: cargo-lock-fingerprint
|
||||
|
||||
Generated
+599
-46
File diff suppressed because it is too large
Load Diff
@@ -89,6 +89,10 @@ socketioxide = { version = "0.15", features = ["extensions"] }
|
||||
whisper-rs = "0.16"
|
||||
image = { version = "0.25", default-features = false, features = ["png", "jpeg"] }
|
||||
tempfile = "3"
|
||||
cpal = "0.15"
|
||||
hound = "3.5"
|
||||
enigo = "0.3"
|
||||
rdev = "0.5"
|
||||
|
||||
matrix-sdk = { version = "0.16", optional = true, default-features = false, features = ["e2e-encryption", "rustls-tls", "markdown"] }
|
||||
fantoccini = { version = "0.22.0", optional = true, default-features = false, features = ["rustls-tls"] }
|
||||
|
||||
@@ -6,6 +6,7 @@ import { PersistGate } from 'redux-persist/integration/react';
|
||||
import AppRoutes from './AppRoutes';
|
||||
import BottomTabBar from './components/BottomTabBar';
|
||||
import ServiceBlockingGate from './components/daemon/ServiceBlockingGate';
|
||||
import DictationHotkeyManager from './components/DictationHotkeyManager';
|
||||
import ErrorFallbackScreen from './components/ErrorFallbackScreen';
|
||||
import LocalAIDownloadSnackbar from './components/LocalAIDownloadSnackbar';
|
||||
import MeshGradient from './components/MeshGradient';
|
||||
@@ -40,6 +41,7 @@ function App() {
|
||||
</div>
|
||||
</div>
|
||||
<OnboardingOverlay />
|
||||
<DictationHotkeyManager />
|
||||
<LocalAIDownloadSnackbar />
|
||||
</ServiceBlockingGate>
|
||||
</Router>
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
/**
|
||||
* DictationHotkeyManager
|
||||
*
|
||||
* Headless component that auto-registers the global dictation hotkey on mount
|
||||
* and logs toggle events. Mount inside ServiceBlockingGate so the core RPC
|
||||
* is available when it initialises.
|
||||
*/
|
||||
import { useEffect } from 'react';
|
||||
|
||||
import { useDictationHotkey } from '../hooks/useDictationHotkey';
|
||||
|
||||
export default function DictationHotkeyManager() {
|
||||
const { dictationEnabled, hotkeyRegistered, toggleCount, hotkey } = useDictationHotkey();
|
||||
|
||||
useEffect(() => {
|
||||
if (toggleCount === 0) return;
|
||||
console.debug(`[dictation] toggle #${toggleCount} — dictation overlay should show/hide`);
|
||||
}, [toggleCount]);
|
||||
|
||||
useEffect(() => {
|
||||
if (hotkeyRegistered) {
|
||||
console.debug(`[dictation] global hotkey active: ${hotkey}`);
|
||||
}
|
||||
}, [hotkeyRegistered, hotkey]);
|
||||
|
||||
useEffect(() => {
|
||||
console.debug(`[dictation] enabled=${dictationEnabled}`);
|
||||
}, [dictationEnabled]);
|
||||
|
||||
return null;
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* useDictationHotkey
|
||||
*
|
||||
* Fetches dictation config from the core RPC on mount and listens for
|
||||
* `dictation:toggle` Socket.IO events emitted by the Rust core when
|
||||
* the global hotkey is pressed. The hotkey listener runs in the core
|
||||
* process (via rdev), not in the Tauri shell.
|
||||
*
|
||||
* Consumers receive:
|
||||
* - `dictationEnabled`: whether dictation is configured on
|
||||
* - `hotkeyRegistered`: true once the core confirms the hotkey is active
|
||||
* - `toggleCount`: increments each time the hotkey fires (use to trigger effects)
|
||||
* - `activationMode`: "toggle" or "push"
|
||||
* - `hotkey`: the configured hotkey string
|
||||
*/
|
||||
import { useEffect, useState } from 'react';
|
||||
|
||||
import { callCoreRpc } from '../services/coreRpcClient';
|
||||
import { socketService } from '../services/socketService';
|
||||
|
||||
interface DictationSettings {
|
||||
enabled: boolean;
|
||||
hotkey: string;
|
||||
activation_mode: string;
|
||||
llm_refinement: boolean;
|
||||
streaming: boolean;
|
||||
streaming_interval_ms: number;
|
||||
}
|
||||
|
||||
export interface DictationHotkeyState {
|
||||
/** Whether dictation is enabled in the core config. */
|
||||
dictationEnabled: boolean;
|
||||
/** Whether the core hotkey listener is active. */
|
||||
hotkeyRegistered: boolean;
|
||||
/** Increments each time the hotkey is pressed (consumers can use as a trigger). */
|
||||
toggleCount: number;
|
||||
/** The configured activation mode ("toggle" or "push"). */
|
||||
activationMode: string;
|
||||
/** The configured hotkey string. */
|
||||
hotkey: string;
|
||||
}
|
||||
|
||||
export function useDictationHotkey(): DictationHotkeyState {
|
||||
const [dictationEnabled, setDictationEnabled] = useState(false);
|
||||
const [hotkeyRegistered, setHotkeyRegistered] = useState(false);
|
||||
const [toggleCount, setToggleCount] = useState(0);
|
||||
const [activationMode, setActivationMode] = useState('toggle');
|
||||
const [hotkey, setHotkey] = useState('');
|
||||
|
||||
// Fetch config from core RPC on mount.
|
||||
useEffect(() => {
|
||||
let disposed = false;
|
||||
|
||||
const init = async () => {
|
||||
try {
|
||||
const settings = await callCoreRpc<DictationSettings>({
|
||||
method: 'openhuman.config_get_dictation_settings',
|
||||
});
|
||||
|
||||
if (disposed) return;
|
||||
|
||||
if (!settings || typeof settings !== 'object') {
|
||||
console.debug('[dictation] no dictation settings from core');
|
||||
return;
|
||||
}
|
||||
|
||||
// Handle RpcOutcome wrapper — the result may be nested in .result
|
||||
const s = (
|
||||
'result' in settings ? (settings as Record<string, unknown>).result : settings
|
||||
) as DictationSettings;
|
||||
|
||||
setDictationEnabled(s.enabled);
|
||||
setActivationMode(s.activation_mode ?? 'toggle');
|
||||
setHotkey(s.hotkey ?? '');
|
||||
|
||||
if (s.enabled && s.hotkey) {
|
||||
// The core process registers the hotkey via rdev — we just note it.
|
||||
setHotkeyRegistered(true);
|
||||
console.debug(`[dictation] core hotkey active: ${s.hotkey}`);
|
||||
} else {
|
||||
console.debug('[dictation] dictation disabled or no hotkey configured');
|
||||
}
|
||||
} catch (err) {
|
||||
console.warn('[dictation] failed to fetch dictation settings', err);
|
||||
}
|
||||
};
|
||||
|
||||
void init();
|
||||
|
||||
return () => {
|
||||
disposed = true;
|
||||
};
|
||||
}, []);
|
||||
|
||||
// Listen for hotkey events from the core via Socket.IO.
|
||||
useEffect(() => {
|
||||
const handleToggle = () => {
|
||||
console.debug('[dictation] hotkey toggle event received via socket');
|
||||
setToggleCount(c => c + 1);
|
||||
};
|
||||
|
||||
socketService.on('dictation:toggle', handleToggle);
|
||||
socketService.on('dictation_toggle', handleToggle);
|
||||
|
||||
return () => {
|
||||
socketService.off('dictation:toggle', handleToggle);
|
||||
socketService.off('dictation_toggle', handleToggle);
|
||||
};
|
||||
}, []);
|
||||
|
||||
return { dictationEnabled, hotkeyRegistered, toggleCount, activationMode, hotkey };
|
||||
}
|
||||
@@ -331,9 +331,8 @@ const Conversations = () => {
|
||||
let cancelled = false;
|
||||
void (async () => {
|
||||
try {
|
||||
const resp = await openhumanVoiceStatus();
|
||||
const status = await openhumanVoiceStatus();
|
||||
if (cancelled) return;
|
||||
const status = resp.result;
|
||||
if (!status.stt_available) {
|
||||
setVoiceStatus(
|
||||
'Speech-to-text unavailable: whisper-cli binary or STT model not found. Check Settings > Local Models.'
|
||||
@@ -688,7 +687,7 @@ const Conversations = () => {
|
||||
: undefined;
|
||||
|
||||
const result = await openhumanVoiceTranscribeBytes(audioBytes, extension, context);
|
||||
const transcript = result.result.text.trim();
|
||||
const transcript = result.text.trim();
|
||||
|
||||
if (!transcript) {
|
||||
setVoiceStatus('No speech detected. Try again.');
|
||||
@@ -790,7 +789,7 @@ const Conversations = () => {
|
||||
const ttsResult = await openhumanVoiceTts(latestAgentMessage.content);
|
||||
if (cancelled) return;
|
||||
|
||||
const audioSrc = convertFileSrc(ttsResult.result.output_path);
|
||||
const audioSrc = convertFileSrc(ttsResult.output_path);
|
||||
const audio = new window.Audio(audioSrc);
|
||||
replyAudioRef.current?.pause();
|
||||
replyAudioRef.current = audio;
|
||||
|
||||
@@ -2456,19 +2456,16 @@ export interface VoiceStatus {
|
||||
llm_cleanup_enabled: boolean;
|
||||
}
|
||||
|
||||
export async function openhumanVoiceStatus(): Promise<CommandResponse<VoiceStatus>> {
|
||||
return await callCoreRpc<CommandResponse<VoiceStatus>>({
|
||||
method: 'openhuman.voice_status',
|
||||
params: {},
|
||||
});
|
||||
export async function openhumanVoiceStatus(): Promise<VoiceStatus> {
|
||||
return await callCoreRpc<VoiceStatus>({ method: 'openhuman.voice_status', params: {} });
|
||||
}
|
||||
|
||||
export async function openhumanVoiceTranscribe(
|
||||
audioPath: string,
|
||||
context?: string,
|
||||
skipCleanup?: boolean
|
||||
): Promise<CommandResponse<VoiceSpeechResult>> {
|
||||
return await callCoreRpc<CommandResponse<VoiceSpeechResult>>({
|
||||
): Promise<VoiceSpeechResult> {
|
||||
return await callCoreRpc<VoiceSpeechResult>({
|
||||
method: 'openhuman.voice_transcribe',
|
||||
params: { audio_path: audioPath, context, skip_cleanup: skipCleanup },
|
||||
});
|
||||
@@ -2479,8 +2476,8 @@ export async function openhumanVoiceTranscribeBytes(
|
||||
extension?: string,
|
||||
context?: string,
|
||||
skipCleanup?: boolean
|
||||
): Promise<CommandResponse<VoiceSpeechResult>> {
|
||||
return await callCoreRpc<CommandResponse<VoiceSpeechResult>>({
|
||||
): Promise<VoiceSpeechResult> {
|
||||
return await callCoreRpc<VoiceSpeechResult>({
|
||||
method: 'openhuman.voice_transcribe_bytes',
|
||||
params: { audio_bytes: audioBytes, extension, context, skip_cleanup: skipCleanup },
|
||||
});
|
||||
@@ -2489,8 +2486,8 @@ export async function openhumanVoiceTranscribeBytes(
|
||||
export async function openhumanVoiceTts(
|
||||
text: string,
|
||||
outputPath?: string
|
||||
): Promise<CommandResponse<VoiceTtsResult>> {
|
||||
return await callCoreRpc<CommandResponse<VoiceTtsResult>>({
|
||||
): Promise<VoiceTtsResult> {
|
||||
return await callCoreRpc<VoiceTtsResult>({
|
||||
method: 'openhuman.voice_tts',
|
||||
params: { text, output_path: outputPath },
|
||||
});
|
||||
|
||||
@@ -58,6 +58,7 @@ pub fn run_from_cli_args(args: &[String]) -> Result<()> {
|
||||
"screen-intelligence" => {
|
||||
crate::core::screen_intelligence_cli::run_screen_intelligence_command(&args[1..])
|
||||
}
|
||||
"voice" | "dictate" => run_voice_server_command(&args[1..]),
|
||||
"text-input" => crate::core::text_input_cli::run_text_input_command(&args[1..]),
|
||||
namespace => run_namespace_command(namespace, &args[1..], &grouped),
|
||||
}
|
||||
@@ -189,6 +190,98 @@ fn run_call_command(args: &[String]) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Handles the `voice` subcommand to run the standalone voice dictation server.
|
||||
///
|
||||
/// Listens for a hotkey, records audio, transcribes via whisper, and inserts
|
||||
/// the result into the active text field.
|
||||
|
||||
fn run_voice_server_command(args: &[String]) -> Result<()> {
|
||||
use crate::openhuman::voice::hotkey::ActivationMode;
|
||||
use crate::openhuman::voice::server::{run_standalone, VoiceServerConfig};
|
||||
|
||||
let mut hotkey: Option<String> = None;
|
||||
let mut mode: Option<String> = None;
|
||||
let mut skip_cleanup = false;
|
||||
let mut verbose = false;
|
||||
let mut i = 0usize;
|
||||
|
||||
while i < args.len() {
|
||||
match args[i].as_str() {
|
||||
"--hotkey" => {
|
||||
hotkey = Some(
|
||||
args.get(i + 1)
|
||||
.ok_or_else(|| anyhow::anyhow!("missing value for --hotkey"))?
|
||||
.clone(),
|
||||
);
|
||||
i += 2;
|
||||
}
|
||||
"--mode" => {
|
||||
mode = Some(
|
||||
args.get(i + 1)
|
||||
.ok_or_else(|| anyhow::anyhow!("missing value for --mode"))?
|
||||
.clone(),
|
||||
);
|
||||
i += 2;
|
||||
}
|
||||
"--skip-cleanup" => {
|
||||
skip_cleanup = true;
|
||||
i += 1;
|
||||
}
|
||||
"-v" | "--verbose" => {
|
||||
verbose = true;
|
||||
i += 1;
|
||||
}
|
||||
"-h" | "--help" => {
|
||||
println!("Usage: openhuman voice [--hotkey <combo>] [--mode <tap|push>] [--skip-cleanup] [-v]");
|
||||
println!();
|
||||
println!(" --hotkey <combo> Key combination (default: ctrl+shift+space)");
|
||||
println!(
|
||||
" --mode <tap|push> Activation: tap to toggle, push to hold (default: tap)"
|
||||
);
|
||||
println!(" --skip-cleanup Skip LLM post-processing on transcriptions");
|
||||
println!(" -v, --verbose Enable debug logging");
|
||||
println!();
|
||||
println!("Standalone voice dictation server. Press the hotkey to dictate,");
|
||||
println!("transcribed text is inserted into the active text field.");
|
||||
return Ok(());
|
||||
}
|
||||
other => return Err(anyhow::anyhow!("unknown voice arg: {other}")),
|
||||
}
|
||||
}
|
||||
|
||||
crate::core::logging::init_for_cli_run(verbose);
|
||||
|
||||
let rt = tokio::runtime::Builder::new_multi_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
|
||||
rt.block_on(async {
|
||||
let mut config = crate::openhuman::config::Config::load_or_init()
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
config.apply_env_overrides();
|
||||
|
||||
let activation_mode = match mode.as_deref() {
|
||||
Some("push") => ActivationMode::Push,
|
||||
_ => ActivationMode::Tap,
|
||||
};
|
||||
|
||||
let server_config = VoiceServerConfig {
|
||||
hotkey: hotkey.unwrap_or_else(|| config.voice_server.hotkey.clone()),
|
||||
activation_mode,
|
||||
skip_cleanup,
|
||||
context: None,
|
||||
min_duration_secs: config.voice_server.min_duration_secs,
|
||||
};
|
||||
|
||||
run_standalone(config, server_config)
|
||||
.await
|
||||
.map_err(anyhow::Error::msg)
|
||||
})?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Dispatches commands that fall under a specific namespace (e.g., `openhuman <namespace> <function>`).
|
||||
///
|
||||
/// It looks up the function schema for validation and executes the request.
|
||||
@@ -430,6 +523,7 @@ fn print_general_help(grouped: &BTreeMap<String, Vec<ControllerSchema>>) {
|
||||
println!(" openhuman repl [--verbose] [--eval '<cmd>'] [--batch]");
|
||||
println!(" openhuman call --method <name> [--params '<json>']");
|
||||
println!(" openhuman skills <subcommand> [options] (skill development runtime)");
|
||||
println!(" openhuman voice [--hotkey <combo>] [--mode <tap|push>] (voice dictation server)");
|
||||
println!(" openhuman <namespace> <function> [--param value ...]\n");
|
||||
println!("Available namespaces:");
|
||||
for namespace in grouped.keys() {
|
||||
|
||||
+23
-1
@@ -6,7 +6,9 @@
|
||||
//! - SSE (Server-Sent Events) for real-time event streaming.
|
||||
//! - Helper routes for health checks, schema discovery, and Telegram authentication.
|
||||
|
||||
use axum::extract::{Query, State};
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::extract::{Query, State, WebSocketUpgrade};
|
||||
use axum::http::{header, HeaderValue, Method, StatusCode};
|
||||
use axum::middleware::{self, Next};
|
||||
use axum::response::sse::{Event, KeepAlive, Sse};
|
||||
@@ -326,6 +328,21 @@ async fn telegram_auth_handler(Query(query): Query<TelegramAuthQuery>) -> impl I
|
||||
html_response(StatusCode::OK, success_html())
|
||||
}
|
||||
|
||||
/// WebSocket upgrade handler for streaming voice dictation.
|
||||
async fn dictation_ws_handler(ws: WebSocketUpgrade) -> Response {
|
||||
log::info!("[ws] dictation WebSocket upgrade requested");
|
||||
ws.on_upgrade(|socket| async move {
|
||||
let config = match crate::openhuman::config::rpc::load_config_with_timeout().await {
|
||||
Ok(c) => Arc::new(c),
|
||||
Err(e) => {
|
||||
log::error!("[ws] failed to load config for dictation: {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
crate::openhuman::voice::streaming::handle_dictation_ws(socket, config).await;
|
||||
})
|
||||
}
|
||||
|
||||
/// Builds the main Axum router for the core HTTP server.
|
||||
///
|
||||
/// Includes routes for health, schema, SSE events, JSON-RPC, and Telegram auth.
|
||||
@@ -338,6 +355,7 @@ pub fn build_core_http_router(socketio_enabled: bool) -> Router {
|
||||
.route("/events", get(events_handler))
|
||||
.route("/events/webhooks", get(webhook_events_handler))
|
||||
.route("/rpc", post(rpc_handler))
|
||||
.route("/ws/dictation", get(dictation_ws_handler))
|
||||
.route("/auth/telegram", get(telegram_auth_handler))
|
||||
.fallback(not_found_handler)
|
||||
.layer(middleware::from_fn(http_request_log_middleware))
|
||||
@@ -628,6 +646,10 @@ pub async fn run_server(
|
||||
} else {
|
||||
log::info!("[overlay] overlay disabled by config (overlay_enabled = false)");
|
||||
}
|
||||
|
||||
// Start the global dictation hotkey listener (rdev-based, core-side).
|
||||
|
||||
crate::openhuman::voice::dictation_listener::start_if_enabled(&config).await;
|
||||
}
|
||||
Err(err) => {
|
||||
log::warn!("[core] config load failed, skipping local-ai and overlay: {err}");
|
||||
|
||||
+28
-1
@@ -196,6 +196,8 @@ pub fn attach_socketio() -> (socketioxide::layer::SocketIoLayer, SocketIo) {
|
||||
}
|
||||
|
||||
pub fn spawn_web_channel_bridge(io: SocketIo) {
|
||||
// Web channel events → per-client rooms.
|
||||
let io_web = io.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut rx = crate::openhuman::channels::providers::web::subscribe_web_channel_events();
|
||||
loop {
|
||||
@@ -211,10 +213,35 @@ pub fn spawn_web_channel_bridge(io: SocketIo) {
|
||||
Err(tokio::sync::broadcast::error::RecvError::Closed) => break,
|
||||
};
|
||||
|
||||
emit_web_channel_event(&io, event);
|
||||
emit_web_channel_event(&io_web, event);
|
||||
}
|
||||
log::debug!("[socketio] web_channel bridge stopped");
|
||||
});
|
||||
|
||||
// Dictation hotkey events → broadcast to all connected clients.
|
||||
tokio::spawn(async move {
|
||||
let mut rx = crate::openhuman::voice::dictation_listener::subscribe_dictation_events();
|
||||
loop {
|
||||
let event = match rx.recv().await {
|
||||
Ok(event) => event,
|
||||
Err(tokio::sync::broadcast::error::RecvError::Lagged(skipped)) => {
|
||||
log::warn!("[socketio] dropped {} dictation events due to lag", skipped);
|
||||
continue;
|
||||
}
|
||||
Err(tokio::sync::broadcast::error::RecvError::Closed) => break,
|
||||
};
|
||||
|
||||
if let Ok(payload) = serde_json::to_value(&event) {
|
||||
log::debug!(
|
||||
"[socketio] broadcast dictation:{} to all clients",
|
||||
event.event_type
|
||||
);
|
||||
let _ = io.emit("dictation:toggle", &payload);
|
||||
let _ = io.emit("dictation_toggle", &payload);
|
||||
}
|
||||
}
|
||||
log::debug!("[socketio] dictation bridge stopped");
|
||||
});
|
||||
}
|
||||
|
||||
fn emit_web_channel_event(io: &SocketIo, event: WebChannelEvent) {
|
||||
|
||||
+11
-11
@@ -26,17 +26,17 @@ pub use schema::{
|
||||
build_runtime_proxy_client_with_timeouts, runtime_proxy_config, set_runtime_proxy_config,
|
||||
AgentConfig, ArchetypeConfig, AuditConfig, AutocompleteConfig, AutonomyConfig,
|
||||
BrowserComputerUseConfig, BrowserConfig, ChannelsConfig, ClassificationRule, ComposioConfig,
|
||||
Config, CostConfig, CronConfig, DelegateAgentConfig, DiscordConfig, DockerRuntimeConfig,
|
||||
EmbeddingRouteConfig, HardwareConfig, HardwareTransport, HeartbeatConfig, HttpRequestConfig,
|
||||
IMessageConfig, IdentityConfig, IntegrationToggle, IntegrationsConfig, LarkConfig,
|
||||
LearningConfig, LocalAiConfig, MatrixConfig, MemoryConfig, ModelRouteConfig, MultimodalConfig,
|
||||
ObservabilityConfig, OrchestratorConfig, PeripheralBoardConfig, PeripheralsConfig, ProxyConfig,
|
||||
ProxyScope, QueryClassificationConfig, ReflectionSource, ReliabilityConfig,
|
||||
ResourceLimitsConfig, RuntimeConfig, SandboxBackend, SandboxConfig, SchedulerConfig,
|
||||
ScreenIntelligenceConfig, SecretsConfig, SecurityConfig, SlackConfig, StorageConfig,
|
||||
StorageProviderConfig, StorageProviderSection, StreamMode, TelegramConfig, UpdateConfig,
|
||||
WebSearchConfig, WebhookConfig, DEFAULT_MODEL, MODEL_AGENTIC_V1, MODEL_CODING_V1,
|
||||
MODEL_REASONING_V1,
|
||||
Config, CostConfig, CronConfig, DelegateAgentConfig, DictationActivationMode, DictationConfig,
|
||||
DiscordConfig, DockerRuntimeConfig, EmbeddingRouteConfig, HardwareConfig, HardwareTransport,
|
||||
HeartbeatConfig, HttpRequestConfig, IMessageConfig, IdentityConfig, IntegrationToggle,
|
||||
IntegrationsConfig, LarkConfig, LearningConfig, LocalAiConfig, MatrixConfig, MemoryConfig,
|
||||
ModelRouteConfig, MultimodalConfig, ObservabilityConfig, OrchestratorConfig,
|
||||
PeripheralBoardConfig, PeripheralsConfig, ProxyConfig, ProxyScope, QueryClassificationConfig,
|
||||
ReflectionSource, ReliabilityConfig, ResourceLimitsConfig, RuntimeConfig, SandboxBackend,
|
||||
SandboxConfig, SchedulerConfig, ScreenIntelligenceConfig, SecretsConfig, SecurityConfig,
|
||||
SlackConfig, StorageConfig, StorageProviderConfig, StorageProviderSection, StreamMode,
|
||||
TelegramConfig, UpdateConfig, VoiceActivationMode, VoiceServerConfig, WebSearchConfig,
|
||||
WebhookConfig, DEFAULT_MODEL, MODEL_AGENTIC_V1, MODEL_CODING_V1, MODEL_REASONING_V1,
|
||||
};
|
||||
pub use schema::{
|
||||
clear_active_user, default_root_openhuman_dir, read_active_user_id, user_openhuman_dir,
|
||||
|
||||
@@ -534,6 +534,80 @@ pub async fn set_onboarding_completed(value: bool) -> Result<RpcOutcome<bool>, S
|
||||
))
|
||||
}
|
||||
|
||||
// ── Dictation settings ───────────────────────────────────────────────
|
||||
|
||||
pub struct DictationSettingsPatch {
|
||||
pub enabled: Option<bool>,
|
||||
pub hotkey: Option<String>,
|
||||
pub activation_mode: Option<String>,
|
||||
pub llm_refinement: Option<bool>,
|
||||
pub streaming: Option<bool>,
|
||||
pub streaming_interval_ms: Option<u64>,
|
||||
}
|
||||
|
||||
pub async fn get_dictation_settings() -> Result<RpcOutcome<serde_json::Value>, String> {
|
||||
let config = load_config_with_timeout().await?;
|
||||
let result = json!({
|
||||
"enabled": config.dictation.enabled,
|
||||
"hotkey": config.dictation.hotkey,
|
||||
"activation_mode": config.dictation.activation_mode,
|
||||
"llm_refinement": config.dictation.llm_refinement,
|
||||
"streaming": config.dictation.streaming,
|
||||
"streaming_interval_ms": config.dictation.streaming_interval_ms,
|
||||
});
|
||||
Ok(RpcOutcome::new(
|
||||
result,
|
||||
vec!["dictation settings read".to_string()],
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn load_and_apply_dictation_settings(
|
||||
update: DictationSettingsPatch,
|
||||
) -> Result<RpcOutcome<serde_json::Value>, String> {
|
||||
let mut config = load_config_with_timeout().await?;
|
||||
if let Some(enabled) = update.enabled {
|
||||
config.dictation.enabled = enabled;
|
||||
}
|
||||
if let Some(hotkey) = update.hotkey {
|
||||
config.dictation.hotkey = hotkey;
|
||||
}
|
||||
if let Some(mode) = update.activation_mode {
|
||||
match mode.as_str() {
|
||||
"toggle" => {
|
||||
config.dictation.activation_mode =
|
||||
crate::openhuman::config::DictationActivationMode::Toggle;
|
||||
}
|
||||
"push" => {
|
||||
config.dictation.activation_mode =
|
||||
crate::openhuman::config::DictationActivationMode::Push;
|
||||
}
|
||||
_ => {
|
||||
return Err(format!(
|
||||
"invalid activation_mode: {mode} (valid: toggle, push)"
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(llm_refinement) = update.llm_refinement {
|
||||
config.dictation.llm_refinement = llm_refinement;
|
||||
}
|
||||
if let Some(streaming) = update.streaming {
|
||||
config.dictation.streaming = streaming;
|
||||
}
|
||||
if let Some(interval) = update.streaming_interval_ms {
|
||||
config.dictation.streaming_interval_ms = interval;
|
||||
}
|
||||
config.save().await.map_err(|e| e.to_string())?;
|
||||
let snapshot = snapshot_config_json(&config)?;
|
||||
Ok(RpcOutcome::new(
|
||||
snapshot,
|
||||
vec![format!(
|
||||
"dictation settings saved to {}",
|
||||
config.config_path.display()
|
||||
)],
|
||||
))
|
||||
}
|
||||
|
||||
pub fn agent_server_status() -> RpcOutcome<serde_json::Value> {
|
||||
let running = crate::openhuman::service::mock::mock_agent_running().unwrap_or(true);
|
||||
log::info!("[config] agent_server_status requested: running={running}");
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
//! Voice dictation configuration.
|
||||
|
||||
use schemars::JsonSchema;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Activation mode for the dictation hotkey.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, PartialEq, Eq)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum DictationActivationMode {
|
||||
/// Press once to start, press again to stop.
|
||||
Toggle,
|
||||
/// Hold to record, release to stop (push-to-talk).
|
||||
Push,
|
||||
}
|
||||
|
||||
impl Default for DictationActivationMode {
|
||||
fn default() -> Self {
|
||||
Self::Toggle
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct DictationConfig {
|
||||
/// Whether voice dictation is enabled.
|
||||
#[serde(default = "default_enabled")]
|
||||
pub enabled: bool,
|
||||
|
||||
/// Global hotkey for activating dictation (e.g. "CmdOrCtrl+Shift+D").
|
||||
#[serde(default = "default_hotkey")]
|
||||
pub hotkey: String,
|
||||
|
||||
/// Activation mode: "toggle" (press to start/stop) or "push" (hold to record).
|
||||
#[serde(default)]
|
||||
pub activation_mode: DictationActivationMode,
|
||||
|
||||
/// Whether to refine raw transcription through a local LLM for grammar/punctuation.
|
||||
#[serde(default = "default_llm_refinement")]
|
||||
pub llm_refinement: bool,
|
||||
|
||||
/// Whether to use WebSocket streaming transcription (chunks sent in real-time)
|
||||
/// instead of batch transcription after recording stops.
|
||||
#[serde(default = "default_streaming")]
|
||||
pub streaming: bool,
|
||||
|
||||
/// Interval in milliseconds between streaming inference passes on accumulated audio.
|
||||
#[serde(default = "default_streaming_interval_ms")]
|
||||
pub streaming_interval_ms: u64,
|
||||
}
|
||||
|
||||
fn default_enabled() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn default_hotkey() -> String {
|
||||
"CmdOrCtrl+Shift+D".to_string()
|
||||
}
|
||||
|
||||
fn default_llm_refinement() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn default_streaming() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn default_streaming_interval_ms() -> u64 {
|
||||
2000
|
||||
}
|
||||
|
||||
impl Default for DictationConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: default_enabled(),
|
||||
hotkey: default_hotkey(),
|
||||
activation_mode: DictationActivationMode::default(),
|
||||
llm_refinement: default_llm_refinement(),
|
||||
streaming: default_streaming(),
|
||||
streaming_interval_ms: default_streaming_interval_ms(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -776,6 +776,62 @@ impl Config {
|
||||
}
|
||||
}
|
||||
|
||||
// Dictation overrides
|
||||
if let Ok(flag) = std::env::var("OPENHUMAN_DICTATION_ENABLED") {
|
||||
let normalized = flag.trim().to_ascii_lowercase();
|
||||
match normalized.as_str() {
|
||||
"1" | "true" | "yes" | "on" => self.dictation.enabled = true,
|
||||
"0" | "false" | "no" | "off" => self.dictation.enabled = false,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if let Ok(hotkey) = std::env::var("OPENHUMAN_DICTATION_HOTKEY") {
|
||||
let hotkey = hotkey.trim();
|
||||
if !hotkey.is_empty() {
|
||||
self.dictation.hotkey = hotkey.to_string();
|
||||
}
|
||||
}
|
||||
if let Ok(mode) = std::env::var("OPENHUMAN_DICTATION_ACTIVATION_MODE") {
|
||||
let normalized = mode.trim().to_ascii_lowercase();
|
||||
match normalized.as_str() {
|
||||
"toggle" => {
|
||||
self.dictation.activation_mode =
|
||||
crate::openhuman::config::DictationActivationMode::Toggle
|
||||
}
|
||||
"push" => {
|
||||
self.dictation.activation_mode =
|
||||
crate::openhuman::config::DictationActivationMode::Push
|
||||
}
|
||||
_ => {
|
||||
tracing::warn!(
|
||||
mode = %mode,
|
||||
"ignoring invalid OPENHUMAN_DICTATION_ACTIVATION_MODE (valid: toggle, push)"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Ok(flag) = std::env::var("OPENHUMAN_DICTATION_LLM_REFINEMENT") {
|
||||
let normalized = flag.trim().to_ascii_lowercase();
|
||||
match normalized.as_str() {
|
||||
"1" | "true" | "yes" | "on" => self.dictation.llm_refinement = true,
|
||||
"0" | "false" | "no" | "off" => self.dictation.llm_refinement = false,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if let Ok(flag) = std::env::var("OPENHUMAN_DICTATION_STREAMING") {
|
||||
let normalized = flag.trim().to_ascii_lowercase();
|
||||
match normalized.as_str() {
|
||||
"1" | "true" | "yes" | "on" => self.dictation.streaming = true,
|
||||
"0" | "false" | "no" | "off" => self.dictation.streaming = false,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if let Ok(val) = std::env::var("OPENHUMAN_DICTATION_STREAMING_INTERVAL_MS") {
|
||||
if let Ok(ms) = val.trim().parse::<u64>() {
|
||||
self.dictation.streaming_interval_ms = ms;
|
||||
}
|
||||
}
|
||||
|
||||
if let Ok(flag) = std::env::var("OPENHUMAN_OVERLAY_ENABLED") {
|
||||
let normalized = flag.trim().to_ascii_lowercase();
|
||||
match normalized.as_str() {
|
||||
|
||||
@@ -9,6 +9,7 @@ mod autonomy;
|
||||
mod channels;
|
||||
mod defaults;
|
||||
pub(crate) use defaults::default_true;
|
||||
mod dictation;
|
||||
mod hardware;
|
||||
mod heartbeat_cron;
|
||||
mod identity_cost;
|
||||
@@ -38,6 +39,7 @@ pub use channels::{
|
||||
SandboxBackend, SandboxConfig, SecurityConfig, SignalConfig, SlackConfig, StreamMode,
|
||||
TelegramConfig, WebhookConfig, WhatsAppConfig,
|
||||
};
|
||||
pub use dictation::{DictationActivationMode, DictationConfig};
|
||||
pub use hardware::{HardwareConfig, HardwareTransport};
|
||||
pub use heartbeat_cron::{CronConfig, HeartbeatConfig};
|
||||
pub use identity_cost::{
|
||||
@@ -64,5 +66,7 @@ pub use tools::{
|
||||
IntegrationsConfig, MultimodalConfig, SecretsConfig, WebSearchConfig,
|
||||
};
|
||||
pub use update::UpdateConfig;
|
||||
mod voice_server;
|
||||
pub use voice_server::{VoiceActivationMode, VoiceServerConfig};
|
||||
mod types;
|
||||
pub use types::*;
|
||||
|
||||
@@ -112,6 +112,9 @@ pub struct Config {
|
||||
#[serde(default)]
|
||||
pub local_ai: LocalAiConfig,
|
||||
|
||||
#[serde(default)]
|
||||
pub voice_server: VoiceServerConfig,
|
||||
|
||||
#[serde(default)]
|
||||
pub integrations: IntegrationsConfig,
|
||||
|
||||
@@ -124,6 +127,9 @@ pub struct Config {
|
||||
#[serde(default)]
|
||||
pub update: UpdateConfig,
|
||||
|
||||
#[serde(default)]
|
||||
pub dictation: DictationConfig,
|
||||
|
||||
/// Whether to launch the overlay Tauri app (floating debug/voice panel)
|
||||
/// when the core RPC server starts. Defaults to `true`.
|
||||
#[serde(default = "default_true")]
|
||||
@@ -175,11 +181,13 @@ impl Default for Config {
|
||||
agents: HashMap::new(),
|
||||
hardware: HardwareConfig::default(),
|
||||
local_ai: LocalAiConfig::default(),
|
||||
voice_server: VoiceServerConfig::default(),
|
||||
query_classification: QueryClassificationConfig::default(),
|
||||
integrations: IntegrationsConfig::default(),
|
||||
learning: LearningConfig::default(),
|
||||
orchestrator: OrchestratorConfig::default(),
|
||||
update: UpdateConfig::default(),
|
||||
dictation: DictationConfig::default(),
|
||||
overlay_enabled: true,
|
||||
onboarding_completed: false,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
//! Voice server configuration.
|
||||
|
||||
use schemars::JsonSchema;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Activation mode for the voice server hotkey.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum VoiceActivationMode {
|
||||
/// Single press toggles recording on/off.
|
||||
Tap,
|
||||
/// Hold to record, release to stop.
|
||||
Push,
|
||||
}
|
||||
|
||||
impl Default for VoiceActivationMode {
|
||||
fn default() -> Self {
|
||||
Self::Tap
|
||||
}
|
||||
}
|
||||
|
||||
/// Configuration for the voice dictation server.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
|
||||
pub struct VoiceServerConfig {
|
||||
/// Whether the voice server should start automatically with the core.
|
||||
#[serde(default)]
|
||||
pub auto_start: bool,
|
||||
|
||||
/// Hotkey combination to trigger recording (e.g. "ctrl+shift+space").
|
||||
#[serde(default = "default_hotkey")]
|
||||
pub hotkey: String,
|
||||
|
||||
/// Activation mode: "tap" (toggle) or "push" (hold-to-record).
|
||||
#[serde(default)]
|
||||
pub activation_mode: VoiceActivationMode,
|
||||
|
||||
/// Skip LLM post-processing for transcriptions.
|
||||
#[serde(default)]
|
||||
pub skip_cleanup: bool,
|
||||
|
||||
/// Minimum recording duration in seconds. Recordings shorter than
|
||||
/// this are discarded.
|
||||
#[serde(default = "default_min_duration")]
|
||||
pub min_duration_secs: f32,
|
||||
}
|
||||
|
||||
fn default_hotkey() -> String {
|
||||
"ctrl+shift+space".to_string()
|
||||
}
|
||||
|
||||
fn default_min_duration() -> f32 {
|
||||
0.3
|
||||
}
|
||||
|
||||
impl Default for VoiceServerConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
auto_start: false,
|
||||
hotkey: default_hotkey(),
|
||||
activation_mode: VoiceActivationMode::default(),
|
||||
skip_cleanup: false,
|
||||
min_duration_secs: default_min_duration(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -76,6 +76,16 @@ struct OnboardingCompletedSetParams {
|
||||
value: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct DictationSettingsUpdate {
|
||||
enabled: Option<bool>,
|
||||
hotkey: Option<String>,
|
||||
activation_mode: Option<String>,
|
||||
llm_refinement: Option<bool>,
|
||||
streaming: Option<bool>,
|
||||
streaming_interval_ms: Option<u64>,
|
||||
}
|
||||
|
||||
pub fn all_controller_schemas() -> Vec<ControllerSchema> {
|
||||
vec![
|
||||
schemas("get_config"),
|
||||
@@ -95,6 +105,8 @@ pub fn all_controller_schemas() -> Vec<ControllerSchema> {
|
||||
schemas("reset_local_data"),
|
||||
schemas("get_onboarding_completed"),
|
||||
schemas("set_onboarding_completed"),
|
||||
schemas("get_dictation_settings"),
|
||||
schemas("update_dictation_settings"),
|
||||
]
|
||||
}
|
||||
|
||||
@@ -168,6 +180,14 @@ pub fn all_registered_controllers() -> Vec<RegisteredController> {
|
||||
schema: schemas("set_onboarding_completed"),
|
||||
handler: handle_set_onboarding_completed,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: schemas("get_dictation_settings"),
|
||||
handler: handle_get_dictation_settings,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: schemas("update_dictation_settings"),
|
||||
handler: handle_update_dictation_settings,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
@@ -410,6 +430,32 @@ pub fn schemas(function: &str) -> ControllerSchema {
|
||||
required: true,
|
||||
}],
|
||||
},
|
||||
"get_dictation_settings" => ControllerSchema {
|
||||
namespace: "config",
|
||||
function: "get_dictation_settings",
|
||||
description: "Read current voice dictation settings.",
|
||||
inputs: vec![],
|
||||
outputs: vec![json_output("settings", "Dictation settings payload.")],
|
||||
},
|
||||
"update_dictation_settings" => ControllerSchema {
|
||||
namespace: "config",
|
||||
function: "update_dictation_settings",
|
||||
description: "Update voice dictation settings.",
|
||||
inputs: vec![
|
||||
optional_bool("enabled", "Enable voice dictation."),
|
||||
optional_string("hotkey", "Global hotkey string (e.g. CmdOrCtrl+Shift+D)."),
|
||||
optional_string("activation_mode", "Activation mode: toggle or push."),
|
||||
optional_bool("llm_refinement", "Enable LLM post-processing of transcription."),
|
||||
optional_bool("streaming", "Enable WebSocket streaming transcription."),
|
||||
FieldSchema {
|
||||
name: "streaming_interval_ms",
|
||||
ty: TypeSchema::Option(Box::new(TypeSchema::U64)),
|
||||
comment: "Interval between streaming inference passes (ms).",
|
||||
required: false,
|
||||
},
|
||||
],
|
||||
outputs: vec![json_output("snapshot", "Updated config snapshot.")],
|
||||
},
|
||||
"set_onboarding_completed" => ControllerSchema {
|
||||
namespace: "config",
|
||||
function: "set_onboarding_completed",
|
||||
@@ -589,6 +635,25 @@ fn handle_get_onboarding_completed(_params: Map<String, Value>) -> ControllerFut
|
||||
Box::pin(async { to_json(config_rpc::get_onboarding_completed().await?) })
|
||||
}
|
||||
|
||||
fn handle_get_dictation_settings(_params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async { to_json(config_rpc::get_dictation_settings().await?) })
|
||||
}
|
||||
|
||||
fn handle_update_dictation_settings(params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async move {
|
||||
let update = deserialize_params::<DictationSettingsUpdate>(params)?;
|
||||
let patch = config_rpc::DictationSettingsPatch {
|
||||
enabled: update.enabled,
|
||||
hotkey: update.hotkey,
|
||||
activation_mode: update.activation_mode,
|
||||
llm_refinement: update.llm_refinement,
|
||||
streaming: update.streaming,
|
||||
streaming_interval_ms: update.streaming_interval_ms,
|
||||
};
|
||||
to_json(config_rpc::load_and_apply_dictation_settings(patch).await?)
|
||||
})
|
||||
}
|
||||
|
||||
fn handle_set_onboarding_completed(params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async move {
|
||||
let payload = deserialize_params::<OnboardingCompletedSetParams>(params)?;
|
||||
|
||||
@@ -0,0 +1,387 @@
|
||||
//! Microphone audio capture using cpal.
|
||||
//!
|
||||
//! Records audio from the default input device and produces 16-kHz mono WAV
|
||||
//! bytes suitable for whisper transcription.
|
||||
|
||||
use std::io::Cursor;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
use cpal::traits::{DeviceTrait, HostTrait, StreamTrait};
|
||||
use cpal::{SampleFormat, SampleRate, StreamConfig};
|
||||
use hound::{SampleFormat as HoundFormat, WavSpec, WavWriter};
|
||||
use log::{debug, error, info, warn};
|
||||
use tokio::sync::oneshot;
|
||||
|
||||
const LOG_PREFIX: &str = "[voice_capture]";
|
||||
|
||||
/// Target sample rate for whisper (16 kHz mono).
|
||||
const TARGET_SAMPLE_RATE: u32 = 16_000;
|
||||
|
||||
/// Result of a completed recording.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RecordingResult {
|
||||
/// WAV-encoded audio bytes (16 kHz, mono, 16-bit PCM).
|
||||
pub wav_bytes: Vec<u8>,
|
||||
/// Duration of the recording in seconds.
|
||||
pub duration_secs: f32,
|
||||
/// Number of samples captured.
|
||||
pub sample_count: usize,
|
||||
}
|
||||
|
||||
/// Handle to a recording in progress. Drop or call `stop()` to end recording.
|
||||
pub struct RecordingHandle {
|
||||
stop_flag: Arc<AtomicBool>,
|
||||
result_rx: Option<oneshot::Receiver<Result<RecordingResult, String>>>,
|
||||
}
|
||||
|
||||
impl RecordingHandle {
|
||||
/// Signal the recording to stop and return the captured audio.
|
||||
pub async fn stop(mut self) -> Result<RecordingResult, String> {
|
||||
self.stop_flag.store(true, Ordering::SeqCst);
|
||||
debug!("{LOG_PREFIX} stop signal sent");
|
||||
|
||||
match self.result_rx.take() {
|
||||
Some(rx) => rx
|
||||
.await
|
||||
.map_err(|_| "recording task dropped before completing".to_string())?,
|
||||
None => Err("recording already stopped".to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Start recording from the default microphone.
|
||||
///
|
||||
/// Returns a `RecordingHandle` that must be `.stop().await`-ed to get
|
||||
/// the captured audio. Recording runs on a dedicated OS thread because
|
||||
/// `cpal::Stream` is `!Send` (it must be created and dropped on the
|
||||
/// same thread).
|
||||
pub fn start_recording() -> Result<RecordingHandle, String> {
|
||||
let stop_flag = Arc::new(AtomicBool::new(false));
|
||||
let stop_flag_clone = stop_flag.clone();
|
||||
let (result_tx, result_rx) = oneshot::channel();
|
||||
|
||||
// Use a oneshot to report whether stream setup succeeded.
|
||||
let (setup_tx, setup_rx) = std::sync::mpsc::sync_channel::<Result<(), String>>(1);
|
||||
|
||||
std::thread::Builder::new()
|
||||
.name("voice-capture".into())
|
||||
.spawn(move || {
|
||||
// All cpal objects are created and used on this thread.
|
||||
let result = record_on_thread(stop_flag_clone, setup_tx);
|
||||
let _ = result_tx.send(result);
|
||||
})
|
||||
.map_err(|e| format!("failed to spawn capture thread: {e}"))?;
|
||||
|
||||
// Wait for the stream to be set up (or an error).
|
||||
match setup_rx.recv() {
|
||||
Ok(Ok(())) => {
|
||||
info!("{LOG_PREFIX} recording started");
|
||||
Ok(RecordingHandle {
|
||||
stop_flag,
|
||||
result_rx: Some(result_rx),
|
||||
})
|
||||
}
|
||||
Ok(Err(e)) => Err(e),
|
||||
Err(_) => Err("capture thread exited before signalling readiness".to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs the entire recording lifecycle on a single thread (cpal requirement).
|
||||
fn record_on_thread(
|
||||
stop_flag: Arc<AtomicBool>,
|
||||
setup_tx: std::sync::mpsc::SyncSender<Result<(), String>>,
|
||||
) -> Result<RecordingResult, String> {
|
||||
let host = cpal::default_host();
|
||||
let device = host
|
||||
.default_input_device()
|
||||
.ok_or_else(|| "no default audio input device found".to_string())?;
|
||||
|
||||
let device_name = device.name().unwrap_or_else(|_| "<unknown>".into());
|
||||
info!("{LOG_PREFIX} using input device: {device_name}");
|
||||
|
||||
let supported_configs = device
|
||||
.supported_input_configs()
|
||||
.map_err(|e| format!("failed to query input configs: {e}"))?;
|
||||
|
||||
let config = find_best_config(supported_configs)?;
|
||||
let source_sample_rate = config.sample_rate().0;
|
||||
let source_channels = config.channels() as usize;
|
||||
|
||||
debug!(
|
||||
"{LOG_PREFIX} recording config: rate={source_sample_rate} channels={source_channels} format={:?}",
|
||||
config.sample_format()
|
||||
);
|
||||
|
||||
let samples: Arc<parking_lot::Mutex<Vec<f32>>> = Arc::new(parking_lot::Mutex::new(
|
||||
Vec::with_capacity(TARGET_SAMPLE_RATE as usize * 30),
|
||||
));
|
||||
|
||||
let sample_format = config.sample_format();
|
||||
let stream_config: StreamConfig = config.into();
|
||||
|
||||
let stream = {
|
||||
let samples_writer = samples.clone();
|
||||
match sample_format {
|
||||
SampleFormat::F32 => device
|
||||
.build_input_stream(
|
||||
&stream_config,
|
||||
move |data: &[f32], _: &cpal::InputCallbackInfo| {
|
||||
let mono = to_mono(data, source_channels);
|
||||
samples_writer.lock().extend_from_slice(&mono);
|
||||
},
|
||||
|err| error!("{LOG_PREFIX} audio stream error: {err}"),
|
||||
None,
|
||||
)
|
||||
.map_err(|e| format!("failed to build f32 input stream: {e}")),
|
||||
SampleFormat::I16 => device
|
||||
.build_input_stream(
|
||||
&stream_config,
|
||||
move |data: &[i16], _: &cpal::InputCallbackInfo| {
|
||||
let floats: Vec<f32> = data.iter().map(|&s| s as f32 / 32768.0).collect();
|
||||
let mono = to_mono(&floats, source_channels);
|
||||
samples_writer.lock().extend_from_slice(&mono);
|
||||
},
|
||||
|err| error!("{LOG_PREFIX} audio stream error: {err}"),
|
||||
None,
|
||||
)
|
||||
.map_err(|e| format!("failed to build i16 input stream: {e}")),
|
||||
SampleFormat::U16 => device
|
||||
.build_input_stream(
|
||||
&stream_config,
|
||||
move |data: &[u16], _: &cpal::InputCallbackInfo| {
|
||||
let floats: Vec<f32> = data
|
||||
.iter()
|
||||
.map(|&s| (s as f32 - 32768.0) / 32768.0)
|
||||
.collect();
|
||||
let mono = to_mono(&floats, source_channels);
|
||||
samples_writer.lock().extend_from_slice(&mono);
|
||||
},
|
||||
|err| error!("{LOG_PREFIX} audio stream error: {err}"),
|
||||
None,
|
||||
)
|
||||
.map_err(|e| format!("failed to build u16 input stream: {e}")),
|
||||
other => Err(format!("unsupported sample format: {other:?}")),
|
||||
}
|
||||
};
|
||||
|
||||
let stream = match stream {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
let _ = setup_tx.send(Err(e.clone()));
|
||||
return Err(e);
|
||||
}
|
||||
};
|
||||
|
||||
if let Err(e) = stream.play() {
|
||||
let msg = format!("failed to start audio stream: {e}");
|
||||
let _ = setup_tx.send(Err(msg.clone()));
|
||||
return Err(msg);
|
||||
}
|
||||
|
||||
// Signal success so start_recording() returns.
|
||||
let _ = setup_tx.send(Ok(()));
|
||||
|
||||
// Poll stop flag while keeping the stream alive on this thread.
|
||||
while !stop_flag.load(Ordering::SeqCst) {
|
||||
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||
}
|
||||
|
||||
debug!("{LOG_PREFIX} stop flag detected, finalizing recording");
|
||||
drop(stream);
|
||||
|
||||
let raw_samples = samples.lock().clone();
|
||||
finalize_recording(raw_samples, source_sample_rate)
|
||||
}
|
||||
|
||||
/// List available input devices.
|
||||
pub fn list_input_devices() -> Result<Vec<String>, String> {
|
||||
let host = cpal::default_host();
|
||||
let devices = host
|
||||
.input_devices()
|
||||
.map_err(|e| format!("failed to enumerate input devices: {e}"))?;
|
||||
|
||||
let names: Vec<String> = devices.filter_map(|d| d.name().ok()).collect();
|
||||
|
||||
debug!("{LOG_PREFIX} found {} input devices", names.len());
|
||||
Ok(names)
|
||||
}
|
||||
|
||||
/// Convert interleaved multi-channel samples to mono by averaging channels.
|
||||
fn to_mono(samples: &[f32], channels: usize) -> Vec<f32> {
|
||||
if channels <= 1 {
|
||||
return samples.to_vec();
|
||||
}
|
||||
|
||||
samples
|
||||
.chunks_exact(channels)
|
||||
.map(|frame| frame.iter().sum::<f32>() / channels as f32)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Resample mono f32 samples from `source_rate` to `TARGET_SAMPLE_RATE` using
|
||||
/// linear interpolation. Good enough for voice dictation quality.
|
||||
fn resample(samples: &[f32], source_rate: u32) -> Vec<f32> {
|
||||
if source_rate == TARGET_SAMPLE_RATE {
|
||||
return samples.to_vec();
|
||||
}
|
||||
|
||||
let ratio = source_rate as f64 / TARGET_SAMPLE_RATE as f64;
|
||||
let output_len = (samples.len() as f64 / ratio).ceil() as usize;
|
||||
let mut output = Vec::with_capacity(output_len);
|
||||
|
||||
for i in 0..output_len {
|
||||
let src_idx = i as f64 * ratio;
|
||||
let idx0 = src_idx.floor() as usize;
|
||||
let idx1 = (idx0 + 1).min(samples.len().saturating_sub(1));
|
||||
let frac = (src_idx - idx0 as f64) as f32;
|
||||
output.push(samples[idx0] * (1.0 - frac) + samples[idx1] * frac);
|
||||
}
|
||||
|
||||
output
|
||||
}
|
||||
|
||||
/// Finalize recorded samples into a 16-kHz mono WAV.
|
||||
fn finalize_recording(
|
||||
raw_samples: Vec<f32>,
|
||||
source_sample_rate: u32,
|
||||
) -> Result<RecordingResult, String> {
|
||||
if raw_samples.is_empty() {
|
||||
warn!("{LOG_PREFIX} no audio samples captured");
|
||||
return Err("no audio samples captured".to_string());
|
||||
}
|
||||
|
||||
let resampled = resample(&raw_samples, source_sample_rate);
|
||||
let sample_count = resampled.len();
|
||||
let duration_secs = sample_count as f32 / TARGET_SAMPLE_RATE as f32;
|
||||
|
||||
debug!(
|
||||
"{LOG_PREFIX} finalizing: {sample_count} samples, {duration_secs:.1}s, \
|
||||
resampled from {source_sample_rate} to {TARGET_SAMPLE_RATE}"
|
||||
);
|
||||
|
||||
let spec = WavSpec {
|
||||
channels: 1,
|
||||
sample_rate: TARGET_SAMPLE_RATE,
|
||||
bits_per_sample: 16,
|
||||
sample_format: HoundFormat::Int,
|
||||
};
|
||||
|
||||
let mut buf = Cursor::new(Vec::new());
|
||||
{
|
||||
let mut writer =
|
||||
WavWriter::new(&mut buf, spec).map_err(|e| format!("WAV writer error: {e}"))?;
|
||||
|
||||
for &sample in &resampled {
|
||||
let clamped = sample.clamp(-1.0, 1.0);
|
||||
let i16_sample = (clamped * 32767.0) as i16;
|
||||
writer
|
||||
.write_sample(i16_sample)
|
||||
.map_err(|e| format!("WAV write error: {e}"))?;
|
||||
}
|
||||
|
||||
writer
|
||||
.finalize()
|
||||
.map_err(|e| format!("WAV finalize error: {e}"))?;
|
||||
}
|
||||
|
||||
let wav_bytes = buf.into_inner();
|
||||
info!(
|
||||
"{LOG_PREFIX} recording finalized: {duration_secs:.1}s, {} bytes WAV",
|
||||
wav_bytes.len()
|
||||
);
|
||||
|
||||
Ok(RecordingResult {
|
||||
wav_bytes,
|
||||
duration_secs,
|
||||
sample_count,
|
||||
})
|
||||
}
|
||||
|
||||
/// Find the best input config — prefer 16 kHz mono, else closest match.
|
||||
fn find_best_config(
|
||||
configs: impl Iterator<Item = cpal::SupportedStreamConfigRange>,
|
||||
) -> Result<cpal::SupportedStreamConfig, String> {
|
||||
let mut configs_vec: Vec<cpal::SupportedStreamConfigRange> = configs.collect();
|
||||
if configs_vec.is_empty() {
|
||||
return Err("no supported audio input configurations found".to_string());
|
||||
}
|
||||
|
||||
// Sort: prefer configs whose range includes 16kHz, then by fewer channels.
|
||||
configs_vec.sort_by(|a, b| {
|
||||
let a_has_target = a.min_sample_rate().0 <= TARGET_SAMPLE_RATE
|
||||
&& a.max_sample_rate().0 >= TARGET_SAMPLE_RATE;
|
||||
let b_has_target = b.min_sample_rate().0 <= TARGET_SAMPLE_RATE
|
||||
&& b.max_sample_rate().0 >= TARGET_SAMPLE_RATE;
|
||||
|
||||
b_has_target
|
||||
.cmp(&a_has_target)
|
||||
.then(a.channels().cmp(&b.channels()))
|
||||
});
|
||||
|
||||
let best = &configs_vec[0];
|
||||
let rate = if best.min_sample_rate().0 <= TARGET_SAMPLE_RATE
|
||||
&& best.max_sample_rate().0 >= TARGET_SAMPLE_RATE
|
||||
{
|
||||
SampleRate(TARGET_SAMPLE_RATE)
|
||||
} else {
|
||||
// Use the maximum supported rate and resample later.
|
||||
best.max_sample_rate()
|
||||
};
|
||||
|
||||
Ok(best.clone().with_sample_rate(rate))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn to_mono_passthrough_single_channel() {
|
||||
let input = vec![0.1, 0.2, 0.3];
|
||||
assert_eq!(to_mono(&input, 1), input);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn to_mono_averages_stereo() {
|
||||
let input = vec![0.0, 1.0, 0.5, 0.5];
|
||||
let mono = to_mono(&input, 2);
|
||||
assert_eq!(mono.len(), 2);
|
||||
assert!((mono[0] - 0.5).abs() < 1e-6);
|
||||
assert!((mono[1] - 0.5).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resample_same_rate_passthrough() {
|
||||
let input = vec![0.1, 0.2, 0.3];
|
||||
let output = resample(&input, TARGET_SAMPLE_RATE);
|
||||
assert_eq!(output, input);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resample_downsamples() {
|
||||
// 32kHz -> 16kHz should roughly halve the samples.
|
||||
let input: Vec<f32> = (0..3200).map(|i| (i as f32 / 3200.0).sin()).collect();
|
||||
let output = resample(&input, 32_000);
|
||||
// Should be approximately 1600 samples.
|
||||
assert!(output.len() >= 1590 && output.len() <= 1610);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn finalize_produces_valid_wav() {
|
||||
let samples: Vec<f32> = (0..16000)
|
||||
.map(|i| (i as f32 * 440.0 * 2.0 * std::f32::consts::PI / 16000.0).sin())
|
||||
.collect();
|
||||
let result = finalize_recording(samples, 16_000).unwrap();
|
||||
assert!(result.wav_bytes.len() > 44); // WAV header is 44 bytes
|
||||
assert!((result.duration_secs - 1.0).abs() < 0.1);
|
||||
// Check WAV magic bytes.
|
||||
assert_eq!(&result.wav_bytes[..4], b"RIFF");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn finalize_empty_samples_errors() {
|
||||
let result = finalize_recording(vec![], 16_000);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
//! Core-side dictation hotkey listener.
|
||||
//!
|
||||
//! Reads the `DictationConfig` from config, starts an `rdev`-based global
|
||||
//! hotkey listener on the core process, and broadcasts `dictation:toggle`
|
||||
//! events over a `tokio::sync::broadcast` channel that the Socket.IO
|
||||
//! bridge subscribes to — so the frontend receives hotkey presses without
|
||||
//! any Tauri-side shortcut registration.
|
||||
|
||||
use once_cell::sync::Lazy;
|
||||
use serde::Serialize;
|
||||
use tokio::sync::broadcast;
|
||||
|
||||
use crate::openhuman::config::Config;
|
||||
use crate::openhuman::voice::hotkey::{self, ActivationMode, HotkeyEvent};
|
||||
|
||||
const LOG_PREFIX: &str = "[dictation_listener]";
|
||||
|
||||
// ── Broadcast channel for dictation events ────────────────────────────
|
||||
|
||||
/// A dictation event broadcast to Socket.IO clients.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct DictationEvent {
|
||||
/// Event type: `"pressed"` or `"released"`.
|
||||
#[serde(rename = "type")]
|
||||
pub event_type: String,
|
||||
/// The hotkey that triggered this event.
|
||||
pub hotkey: String,
|
||||
/// The activation mode in use.
|
||||
pub activation_mode: String,
|
||||
}
|
||||
|
||||
static DICTATION_BUS: Lazy<broadcast::Sender<DictationEvent>> = Lazy::new(|| {
|
||||
let (tx, _rx) = broadcast::channel(64);
|
||||
tx
|
||||
});
|
||||
|
||||
/// Subscribe to dictation events (used by the Socket.IO bridge).
|
||||
pub fn subscribe_dictation_events() -> broadcast::Receiver<DictationEvent> {
|
||||
DICTATION_BUS.subscribe()
|
||||
}
|
||||
|
||||
fn publish_dictation_event(event: DictationEvent) {
|
||||
let _ = DICTATION_BUS.send(event);
|
||||
}
|
||||
|
||||
// ── Listener lifecycle ────────────────────────────────────────────────
|
||||
|
||||
/// Start the dictation hotkey listener if enabled in config.
|
||||
///
|
||||
/// Intended to be called once from `run_server()` as a background task.
|
||||
/// Reads the `dictation` config section and registers the global hotkey.
|
||||
/// When the hotkey fires, publishes a `DictationEvent` to the broadcast
|
||||
/// channel that the Socket.IO bridge forwards to all connected clients.
|
||||
pub async fn start_if_enabled(config: &Config) {
|
||||
if !config.dictation.enabled {
|
||||
log::info!("{LOG_PREFIX} dictation disabled in config, skipping hotkey listener");
|
||||
return;
|
||||
}
|
||||
|
||||
let hotkey_str = config.dictation.hotkey.clone();
|
||||
if hotkey_str.is_empty() {
|
||||
log::warn!("{LOG_PREFIX} dictation enabled but no hotkey configured");
|
||||
return;
|
||||
}
|
||||
|
||||
// Map DictationActivationMode to our hotkey ActivationMode.
|
||||
let mode = match config.dictation.activation_mode {
|
||||
crate::openhuman::config::DictationActivationMode::Push => ActivationMode::Push,
|
||||
crate::openhuman::config::DictationActivationMode::Toggle => ActivationMode::Tap,
|
||||
};
|
||||
|
||||
// Normalize the hotkey string for rdev (CmdOrCtrl → cmd on macOS, ctrl on others).
|
||||
let normalized = normalize_hotkey_for_rdev(&hotkey_str);
|
||||
|
||||
log::info!(
|
||||
"{LOG_PREFIX} starting dictation hotkey listener: hotkey={normalized} (raw={hotkey_str}) mode={mode:?}"
|
||||
);
|
||||
|
||||
let combo = match hotkey::parse_hotkey(&normalized) {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
log::error!("{LOG_PREFIX} failed to parse hotkey '{normalized}': {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let (listener_handle, mut hotkey_rx) = match hotkey::start_listener(combo, mode) {
|
||||
Ok(pair) => pair,
|
||||
Err(e) => {
|
||||
log::error!("{LOG_PREFIX} failed to start hotkey listener: {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let mode_str = match mode {
|
||||
ActivationMode::Tap => "toggle",
|
||||
ActivationMode::Push => "push",
|
||||
};
|
||||
|
||||
log::info!("{LOG_PREFIX} dictation hotkey active: {normalized}");
|
||||
|
||||
// Forward hotkey events to the broadcast channel.
|
||||
tokio::spawn(async move {
|
||||
// Keep the listener handle alive for the lifetime of this task.
|
||||
let _handle = listener_handle;
|
||||
|
||||
while let Some(event) = hotkey_rx.recv().await {
|
||||
let event_type = match event {
|
||||
HotkeyEvent::Pressed => "pressed",
|
||||
HotkeyEvent::Released => "released",
|
||||
};
|
||||
|
||||
log::debug!("{LOG_PREFIX} hotkey {event_type}");
|
||||
|
||||
publish_dictation_event(DictationEvent {
|
||||
event_type: event_type.to_string(),
|
||||
hotkey: normalized.clone(),
|
||||
activation_mode: mode_str.to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
log::warn!("{LOG_PREFIX} hotkey event channel closed, listener stopping");
|
||||
});
|
||||
}
|
||||
|
||||
/// Normalize a Tauri-style hotkey string to rdev-compatible format.
|
||||
///
|
||||
/// Converts `CmdOrCtrl+Shift+D` → `cmd+shift+d` (macOS) or `ctrl+shift+d` (other).
|
||||
fn normalize_hotkey_for_rdev(hotkey: &str) -> String {
|
||||
let parts: Vec<&str> = hotkey.split('+').map(|s| s.trim()).collect();
|
||||
let mut result = Vec::new();
|
||||
|
||||
for part in parts {
|
||||
let lower = part.to_lowercase();
|
||||
let mapped = match lower.as_str() {
|
||||
"cmdorctrl" | "commandorcontrol" => {
|
||||
if cfg!(target_os = "macos") {
|
||||
"cmd"
|
||||
} else {
|
||||
"ctrl"
|
||||
}
|
||||
}
|
||||
"cmd" | "command" => "cmd",
|
||||
"ctrl" | "control" => "ctrl",
|
||||
other => other,
|
||||
};
|
||||
result.push(mapped.to_string());
|
||||
}
|
||||
|
||||
result.join("+")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn normalize_cmdorctrl_macos() {
|
||||
let result = normalize_hotkey_for_rdev("CmdOrCtrl+Shift+D");
|
||||
if cfg!(target_os = "macos") {
|
||||
assert_eq!(result, "cmd+shift+d");
|
||||
} else {
|
||||
assert_eq!(result, "ctrl+shift+d");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_plain_keys() {
|
||||
assert_eq!(normalize_hotkey_for_rdev("Ctrl+Space"), "ctrl+space");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn normalize_preserves_structure() {
|
||||
assert_eq!(normalize_hotkey_for_rdev("Alt+Shift+F5"), "alt+shift+f5");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn subscribe_returns_receiver() {
|
||||
let _rx = subscribe_dictation_events();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,345 @@
|
||||
//! Global hotkey listener using rdev.
|
||||
//!
|
||||
//! Monitors keyboard events system-wide and fires callbacks when a
|
||||
//! configurable key combination is pressed/released. Supports two
|
||||
//! activation modes: **tap** (toggle on press) and **push** (hold to
|
||||
//! record, release to stop).
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
use log::{debug, error, info, warn};
|
||||
use parking_lot::Mutex;
|
||||
use rdev::{listen, Event, EventType, Key};
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
const LOG_PREFIX: &str = "[voice_hotkey]";
|
||||
|
||||
/// Activation mode for the voice hotkey.
|
||||
#[derive(
|
||||
Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize, schemars::JsonSchema,
|
||||
)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ActivationMode {
|
||||
/// Single press toggles recording on/off.
|
||||
Tap,
|
||||
/// Hold to record, release to stop.
|
||||
Push,
|
||||
}
|
||||
|
||||
impl Default for ActivationMode {
|
||||
fn default() -> Self {
|
||||
Self::Tap
|
||||
}
|
||||
}
|
||||
|
||||
/// Events emitted by the hotkey listener.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum HotkeyEvent {
|
||||
/// The hotkey was pressed (start recording).
|
||||
Pressed,
|
||||
/// The hotkey was released (stop recording — only relevant in Push mode).
|
||||
Released,
|
||||
}
|
||||
|
||||
/// Parsed hotkey combination (e.g. Ctrl+Shift+Space).
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct HotkeyCombination {
|
||||
/// Modifier keys that must be held.
|
||||
pub modifiers: HashSet<Key>,
|
||||
/// The primary trigger key.
|
||||
pub trigger: Key,
|
||||
}
|
||||
|
||||
/// Handle to a running hotkey listener. Drop to stop.
|
||||
pub struct HotkeyListenerHandle {
|
||||
stop_flag: Arc<AtomicBool>,
|
||||
_thread: Option<std::thread::JoinHandle<()>>,
|
||||
}
|
||||
|
||||
impl HotkeyListenerHandle {
|
||||
/// Signal the listener to ignore further events.
|
||||
///
|
||||
/// Note: this does **not** terminate the listener thread. `rdev::listen`
|
||||
/// blocks in the platform event loop and provides no cancellation API
|
||||
/// (rdev 0.5). The thread stays alive until the process exits; the
|
||||
/// stop flag merely causes the callback to discard all events.
|
||||
pub fn stop(&self) {
|
||||
self.stop_flag.store(true, Ordering::SeqCst);
|
||||
info!("{LOG_PREFIX} hotkey listener signaled to skip events");
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for HotkeyListenerHandle {
|
||||
fn drop(&mut self) {
|
||||
self.stop_flag.store(true, Ordering::SeqCst);
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a hotkey string like "ctrl+shift+space" into a `HotkeyCombination`.
|
||||
pub fn parse_hotkey(hotkey_str: &str) -> Result<HotkeyCombination, String> {
|
||||
let parts: Vec<&str> = hotkey_str
|
||||
.split('+')
|
||||
.map(|s| s.trim())
|
||||
.filter(|s| !s.is_empty())
|
||||
.collect();
|
||||
|
||||
if parts.is_empty() {
|
||||
return Err("hotkey string is empty".to_string());
|
||||
}
|
||||
|
||||
let mut modifiers = HashSet::new();
|
||||
let mut trigger = None;
|
||||
|
||||
for (i, part) in parts.iter().enumerate() {
|
||||
let key = string_to_key(part)?;
|
||||
if i < parts.len() - 1 {
|
||||
// Everything except the last part is a modifier.
|
||||
modifiers.insert(key);
|
||||
} else {
|
||||
trigger = Some(key);
|
||||
}
|
||||
}
|
||||
|
||||
let trigger = trigger.ok_or_else(|| "no trigger key specified".to_string())?;
|
||||
|
||||
debug!(
|
||||
"{LOG_PREFIX} parsed hotkey: modifiers={:?} trigger={:?}",
|
||||
modifiers, trigger
|
||||
);
|
||||
|
||||
Ok(HotkeyCombination { modifiers, trigger })
|
||||
}
|
||||
|
||||
/// Start the global hotkey listener.
|
||||
///
|
||||
/// Returns a handle (drop to stop) and a receiver for hotkey events.
|
||||
/// The listener runs on a dedicated OS thread since rdev::listen is blocking.
|
||||
pub fn start_listener(
|
||||
hotkey: HotkeyCombination,
|
||||
mode: ActivationMode,
|
||||
) -> Result<(HotkeyListenerHandle, mpsc::UnboundedReceiver<HotkeyEvent>), String> {
|
||||
let stop_flag = Arc::new(AtomicBool::new(false));
|
||||
let (tx, rx) = mpsc::unbounded_channel();
|
||||
|
||||
let stop_flag_clone = stop_flag.clone();
|
||||
let pressed_keys: Arc<Mutex<HashSet<Key>>> = Arc::new(Mutex::new(HashSet::new()));
|
||||
let is_active = Arc::new(AtomicBool::new(false));
|
||||
|
||||
info!(
|
||||
"{LOG_PREFIX} starting hotkey listener, mode={mode:?}, trigger={:?}, modifiers={:?}",
|
||||
hotkey.trigger, hotkey.modifiers
|
||||
);
|
||||
|
||||
let thread = std::thread::Builder::new()
|
||||
.name("voice-hotkey".into())
|
||||
.spawn(move || {
|
||||
let callback = move |event: Event| {
|
||||
if stop_flag_clone.load(Ordering::SeqCst) {
|
||||
return;
|
||||
}
|
||||
|
||||
match event.event_type {
|
||||
EventType::KeyPress(key) => {
|
||||
let mut keys = pressed_keys.lock();
|
||||
keys.insert(key);
|
||||
|
||||
// Check if all modifiers + trigger are held.
|
||||
if key == hotkey.trigger
|
||||
&& hotkey.modifiers.iter().all(|m| keys.contains(m))
|
||||
{
|
||||
match mode {
|
||||
ActivationMode::Tap => {
|
||||
let was_active = is_active.fetch_xor(true, Ordering::SeqCst);
|
||||
let event = if was_active {
|
||||
HotkeyEvent::Released
|
||||
} else {
|
||||
HotkeyEvent::Pressed
|
||||
};
|
||||
debug!("{LOG_PREFIX} tap hotkey → {event:?}");
|
||||
if tx.send(event).is_err() {
|
||||
warn!("{LOG_PREFIX} event receiver dropped");
|
||||
}
|
||||
}
|
||||
ActivationMode::Push => {
|
||||
if !is_active.swap(true, Ordering::SeqCst) {
|
||||
debug!("{LOG_PREFIX} push hotkey → Pressed");
|
||||
if tx.send(HotkeyEvent::Pressed).is_err() {
|
||||
warn!("{LOG_PREFIX} event receiver dropped");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EventType::KeyRelease(key) => {
|
||||
let mut keys = pressed_keys.lock();
|
||||
keys.remove(&key);
|
||||
|
||||
// In push mode, release when the trigger key is released.
|
||||
if mode == ActivationMode::Push
|
||||
&& key == hotkey.trigger
|
||||
&& is_active.swap(false, Ordering::SeqCst)
|
||||
{
|
||||
debug!("{LOG_PREFIX} push hotkey → Released");
|
||||
if tx.send(HotkeyEvent::Released).is_err() {
|
||||
warn!("{LOG_PREFIX} event receiver dropped");
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
};
|
||||
|
||||
// rdev::listen blocks in the platform event loop and has no
|
||||
// graceful cancellation API (rdev 0.5.3). The thread remains
|
||||
// alive until the process exits; the stop_flag only causes the
|
||||
// callback above to discard events. This is a known limitation.
|
||||
if let Err(e) = listen(callback) {
|
||||
error!("{LOG_PREFIX} rdev listen error: {e:?}");
|
||||
}
|
||||
})
|
||||
.map_err(|e| format!("failed to spawn hotkey listener thread: {e}"))?;
|
||||
|
||||
Ok((
|
||||
HotkeyListenerHandle {
|
||||
stop_flag,
|
||||
_thread: Some(thread),
|
||||
},
|
||||
rx,
|
||||
))
|
||||
}
|
||||
|
||||
/// Convert a string key name to an rdev Key.
|
||||
fn string_to_key(s: &str) -> Result<Key, String> {
|
||||
match s.to_lowercase().as_str() {
|
||||
// Modifiers
|
||||
"ctrl" | "control" | "leftcontrol" => Ok(Key::ControlLeft),
|
||||
"rctrl" | "rightcontrol" => Ok(Key::ControlRight),
|
||||
"shift" | "leftshift" => Ok(Key::ShiftLeft),
|
||||
"rshift" | "rightshift" => Ok(Key::ShiftRight),
|
||||
"alt" | "option" | "leftalt" => Ok(Key::Alt),
|
||||
"ralt" | "rightaltoption" => Ok(Key::AltGr),
|
||||
"meta" | "super" | "cmd" | "command" | "leftmeta" => Ok(Key::MetaLeft),
|
||||
"rmeta" | "rsuper" | "rcmd" | "rightmeta" => Ok(Key::MetaRight),
|
||||
|
||||
// Common keys
|
||||
"space" => Ok(Key::Space),
|
||||
"enter" | "return" => Ok(Key::Return),
|
||||
"tab" => Ok(Key::Tab),
|
||||
"escape" | "esc" => Ok(Key::Escape),
|
||||
"backspace" => Ok(Key::Backspace),
|
||||
"delete" | "del" => Ok(Key::Delete),
|
||||
"capslock" => Ok(Key::CapsLock),
|
||||
|
||||
// F-keys
|
||||
"f1" => Ok(Key::F1),
|
||||
"f2" => Ok(Key::F2),
|
||||
"f3" => Ok(Key::F3),
|
||||
"f4" => Ok(Key::F4),
|
||||
"f5" => Ok(Key::F5),
|
||||
"f6" => Ok(Key::F6),
|
||||
"f7" => Ok(Key::F7),
|
||||
"f8" => Ok(Key::F8),
|
||||
"f9" => Ok(Key::F9),
|
||||
"f10" => Ok(Key::F10),
|
||||
"f11" => Ok(Key::F11),
|
||||
"f12" => Ok(Key::F12),
|
||||
|
||||
// Navigation
|
||||
"up" | "uparrow" => Ok(Key::UpArrow),
|
||||
"down" | "downarrow" => Ok(Key::DownArrow),
|
||||
"left" | "leftarrow" => Ok(Key::LeftArrow),
|
||||
"right" | "rightarrow" => Ok(Key::RightArrow),
|
||||
"home" => Ok(Key::Home),
|
||||
"end" => Ok(Key::End),
|
||||
"pageup" | "pgup" => Ok(Key::PageUp),
|
||||
"pagedown" | "pgdn" => Ok(Key::PageDown),
|
||||
"insert" | "ins" => Ok(Key::Insert),
|
||||
|
||||
// Letters
|
||||
"a" => Ok(Key::KeyA),
|
||||
"b" => Ok(Key::KeyB),
|
||||
"c" => Ok(Key::KeyC),
|
||||
"d" => Ok(Key::KeyD),
|
||||
"e" => Ok(Key::KeyE),
|
||||
"f" => Ok(Key::KeyF),
|
||||
"g" => Ok(Key::KeyG),
|
||||
"h" => Ok(Key::KeyH),
|
||||
"i" => Ok(Key::KeyI),
|
||||
"j" => Ok(Key::KeyJ),
|
||||
"k" => Ok(Key::KeyK),
|
||||
"l" => Ok(Key::KeyL),
|
||||
"m" => Ok(Key::KeyM),
|
||||
"n" => Ok(Key::KeyN),
|
||||
"o" => Ok(Key::KeyO),
|
||||
"p" => Ok(Key::KeyP),
|
||||
"q" => Ok(Key::KeyQ),
|
||||
"r" => Ok(Key::KeyR),
|
||||
"s" => Ok(Key::KeyS),
|
||||
"t" => Ok(Key::KeyT),
|
||||
"u" => Ok(Key::KeyU),
|
||||
"v" => Ok(Key::KeyV),
|
||||
"w" => Ok(Key::KeyW),
|
||||
"x" => Ok(Key::KeyX),
|
||||
"y" => Ok(Key::KeyY),
|
||||
"z" => Ok(Key::KeyZ),
|
||||
|
||||
// Numbers
|
||||
"0" => Ok(Key::Num0),
|
||||
"1" => Ok(Key::Num1),
|
||||
"2" => Ok(Key::Num2),
|
||||
"3" => Ok(Key::Num3),
|
||||
"4" => Ok(Key::Num4),
|
||||
"5" => Ok(Key::Num5),
|
||||
"6" => Ok(Key::Num6),
|
||||
"7" => Ok(Key::Num7),
|
||||
"8" => Ok(Key::Num8),
|
||||
"9" => Ok(Key::Num9),
|
||||
|
||||
other => Err(format!("unknown key: '{other}'")),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parse_simple_hotkey() {
|
||||
let combo = parse_hotkey("ctrl+shift+space").unwrap();
|
||||
assert_eq!(combo.trigger, Key::Space);
|
||||
assert!(combo.modifiers.contains(&Key::ControlLeft));
|
||||
assert!(combo.modifiers.contains(&Key::ShiftLeft));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_single_key() {
|
||||
let combo = parse_hotkey("f5").unwrap();
|
||||
assert_eq!(combo.trigger, Key::F5);
|
||||
assert!(combo.modifiers.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_cmd_key() {
|
||||
let combo = parse_hotkey("cmd+space").unwrap();
|
||||
assert_eq!(combo.trigger, Key::Space);
|
||||
assert!(combo.modifiers.contains(&Key::MetaLeft));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_empty_errors() {
|
||||
assert!(parse_hotkey("").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_unknown_key_errors() {
|
||||
assert!(parse_hotkey("ctrl+unknownkey").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn activation_mode_default_is_tap() {
|
||||
assert_eq!(ActivationMode::default(), ActivationMode::Tap);
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,18 @@
|
||||
//! Voice domain — speech-to-text (whisper.cpp) and text-to-speech (piper).
|
||||
//!
|
||||
//! Provides RPC endpoints under the `openhuman.voice_*` namespace for
|
||||
//! transcription, synthesis, and proactive availability checking.
|
||||
//! transcription, synthesis, proactive availability checking, and a
|
||||
//! standalone voice dictation server (hotkey → record → transcribe → insert).
|
||||
|
||||
pub mod audio_capture;
|
||||
pub mod dictation_listener;
|
||||
pub mod hotkey;
|
||||
mod ops;
|
||||
mod postprocess;
|
||||
mod schemas;
|
||||
pub mod server;
|
||||
pub mod streaming;
|
||||
pub mod text_input;
|
||||
mod types;
|
||||
|
||||
pub use ops::*;
|
||||
|
||||
@@ -20,6 +20,13 @@ nothing else.";
|
||||
|
||||
/// Clean up raw transcription text using a local LLM.
|
||||
///
|
||||
/// Cleanup is enabled when **either** of these conditions holds:
|
||||
/// - `config.local_ai.voice_llm_cleanup_enabled` is `true` (default), **or**
|
||||
/// - the local LLM state is `"ready"` or `"degraded"`.
|
||||
///
|
||||
/// Even when enabled by config, cleanup is **skipped** if the LLM is not
|
||||
/// in a ready/degraded state (i.e. not yet downloaded or bootstrapped).
|
||||
///
|
||||
/// Returns the cleaned text on success, or the original raw text if the
|
||||
/// LLM is unavailable or cleanup fails (graceful degradation).
|
||||
pub async fn cleanup_transcription(
|
||||
@@ -31,13 +38,27 @@ pub async fn cleanup_transcription(
|
||||
return raw_text.to_string();
|
||||
}
|
||||
|
||||
if !config.local_ai.voice_llm_cleanup_enabled {
|
||||
debug!("{LOG_PREFIX} LLM cleanup disabled in config");
|
||||
let service = local_ai::global(config);
|
||||
let llm_state = service.status.lock().state.clone();
|
||||
let llm_ready = matches!(llm_state.as_str(), "ready" | "degraded");
|
||||
|
||||
// Enable cleanup when:
|
||||
// 1. Explicitly enabled in config (default: true), OR
|
||||
// 2. The local LLM is already downloaded and ready.
|
||||
let should_cleanup = config.local_ai.voice_llm_cleanup_enabled || llm_ready;
|
||||
|
||||
if !should_cleanup {
|
||||
debug!("{LOG_PREFIX} LLM cleanup skipped: config disabled and LLM not ready (state={llm_state})");
|
||||
return raw_text.to_string();
|
||||
}
|
||||
|
||||
if !llm_ready {
|
||||
debug!("{LOG_PREFIX} LLM cleanup enabled but LLM not ready (state={llm_state}), skipping");
|
||||
return raw_text.to_string();
|
||||
}
|
||||
|
||||
debug!(
|
||||
"{LOG_PREFIX} cleaning up transcription ({} chars, context={})",
|
||||
"{LOG_PREFIX} cleaning up transcription ({} chars, context={}, llm_state={llm_state})",
|
||||
raw_text.len(),
|
||||
conversation_context.is_some()
|
||||
);
|
||||
@@ -52,8 +73,6 @@ pub async fn cleanup_transcription(
|
||||
_ => raw_text.to_string(),
|
||||
};
|
||||
|
||||
let service = local_ai::global(config);
|
||||
|
||||
let result: Result<String, String> = service
|
||||
.inference(config, CLEANUP_SYSTEM_PROMPT, &prompt, Some(512), true)
|
||||
.await;
|
||||
|
||||
@@ -54,6 +54,9 @@ pub fn all_voice_controller_schemas() -> Vec<ControllerSchema> {
|
||||
voice_schemas("voice_transcribe"),
|
||||
voice_schemas("voice_transcribe_bytes"),
|
||||
voice_schemas("voice_tts"),
|
||||
voice_schemas("voice_server_start"),
|
||||
voice_schemas("voice_server_stop"),
|
||||
voice_schemas("voice_server_status"),
|
||||
]
|
||||
}
|
||||
|
||||
@@ -75,6 +78,18 @@ pub fn all_voice_registered_controllers() -> Vec<RegisteredController> {
|
||||
schema: voice_schemas("voice_tts"),
|
||||
handler: handle_voice_tts,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: voice_schemas("voice_server_start"),
|
||||
handler: handle_voice_server_start,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: voice_schemas("voice_server_stop"),
|
||||
handler: handle_voice_server_stop,
|
||||
},
|
||||
RegisteredController {
|
||||
schema: voice_schemas("voice_server_status"),
|
||||
handler: handle_voice_server_status,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
@@ -139,6 +154,35 @@ pub fn voice_schemas(function: &str) -> ControllerSchema {
|
||||
],
|
||||
outputs: vec![json_output("tts", "TTS result with output path.")],
|
||||
},
|
||||
"voice_server_start" => ControllerSchema {
|
||||
namespace: "voice",
|
||||
function: "server_start",
|
||||
description:
|
||||
"Start the voice dictation server (hotkey → record → transcribe → insert text).",
|
||||
inputs: vec![
|
||||
optional_string("hotkey", "Hotkey combination (default: ctrl+shift+space)."),
|
||||
optional_string(
|
||||
"activation_mode",
|
||||
"Activation mode: tap or push (default: tap).",
|
||||
),
|
||||
optional_bool("skip_cleanup", "Skip LLM post-processing."),
|
||||
],
|
||||
outputs: vec![json_output("status", "Voice server status after start.")],
|
||||
},
|
||||
"voice_server_stop" => ControllerSchema {
|
||||
namespace: "voice",
|
||||
function: "server_stop",
|
||||
description: "Stop the voice dictation server.",
|
||||
inputs: vec![],
|
||||
outputs: vec![json_output("status", "Voice server status after stop.")],
|
||||
},
|
||||
"voice_server_status" => ControllerSchema {
|
||||
namespace: "voice",
|
||||
function: "server_status",
|
||||
description: "Get the current voice dictation server status.",
|
||||
inputs: vec![],
|
||||
outputs: vec![json_output("status", "Current voice server status.")],
|
||||
},
|
||||
_ => ControllerSchema {
|
||||
namespace: "voice",
|
||||
function: "unknown",
|
||||
@@ -208,6 +252,127 @@ fn handle_voice_tts(params: Map<String, Value>) -> ControllerFuture {
|
||||
})
|
||||
}
|
||||
|
||||
fn handle_voice_server_start(params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async move {
|
||||
use crate::openhuman::voice::hotkey::ActivationMode;
|
||||
use crate::openhuman::voice::server::{global_server, VoiceServerConfig};
|
||||
|
||||
let config = config_rpc::load_config_with_timeout().await?;
|
||||
|
||||
let hotkey = params
|
||||
.get("hotkey")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or(&config.voice_server.hotkey)
|
||||
.to_string();
|
||||
|
||||
let activation_mode = match params.get("activation_mode").and_then(|v| v.as_str()) {
|
||||
Some("push") => ActivationMode::Push,
|
||||
Some("tap") => ActivationMode::Tap,
|
||||
Some(other) => {
|
||||
log::warn!(
|
||||
"[voice_server] unrecognized activation_mode '{}', defaulting to Tap",
|
||||
other
|
||||
);
|
||||
ActivationMode::Tap
|
||||
}
|
||||
None => match config.voice_server.activation_mode {
|
||||
crate::openhuman::config::VoiceActivationMode::Push => ActivationMode::Push,
|
||||
crate::openhuman::config::VoiceActivationMode::Tap => ActivationMode::Tap,
|
||||
},
|
||||
};
|
||||
|
||||
let skip_cleanup = params
|
||||
.get("skip_cleanup")
|
||||
.and_then(|v| v.as_bool())
|
||||
.unwrap_or(config.voice_server.skip_cleanup);
|
||||
|
||||
let server_config = VoiceServerConfig {
|
||||
hotkey,
|
||||
activation_mode,
|
||||
skip_cleanup,
|
||||
context: None,
|
||||
min_duration_secs: config.voice_server.min_duration_secs,
|
||||
};
|
||||
|
||||
// Check if a server is already running with a different config.
|
||||
if let Some(existing) = crate::openhuman::voice::server::try_global_server() {
|
||||
let existing_status = existing.status().await;
|
||||
if existing_status.state != crate::openhuman::voice::server::ServerState::Stopped {
|
||||
if existing_status.hotkey != server_config.hotkey
|
||||
|| existing_status.activation_mode != server_config.activation_mode
|
||||
{
|
||||
return Err(format!(
|
||||
"voice server already running (hotkey={}, mode={:?}); \
|
||||
stop it first before starting with different config",
|
||||
existing_status.hotkey, existing_status.activation_mode
|
||||
));
|
||||
}
|
||||
// Same config, already running — return current status.
|
||||
return serde_json::to_value(existing_status)
|
||||
.map_err(|e| format!("serialize error: {e}"));
|
||||
}
|
||||
}
|
||||
|
||||
let server = global_server(server_config);
|
||||
let config_clone = config.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = server.run(&config_clone).await {
|
||||
log::error!("[voice_server] server exited with error: {e}");
|
||||
}
|
||||
});
|
||||
|
||||
// Give the server a moment to start.
|
||||
tokio::time::sleep(std::time::Duration::from_millis(200)).await;
|
||||
|
||||
if let Some(s) = crate::openhuman::voice::server::try_global_server() {
|
||||
let status = s.status().await;
|
||||
serde_json::to_value(status).map_err(|e| format!("serialize error: {e}"))
|
||||
} else {
|
||||
Err("voice server failed to initialize".to_string())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn handle_voice_server_stop(_params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async move {
|
||||
if let Some(server) = crate::openhuman::voice::server::try_global_server() {
|
||||
server.stop().await;
|
||||
tokio::time::sleep(std::time::Duration::from_millis(200)).await;
|
||||
let status = server.status().await;
|
||||
serde_json::to_value(status).map_err(|e| format!("serialize error: {e}"))
|
||||
} else {
|
||||
// Not running — return a stopped status rather than an error.
|
||||
let status = crate::openhuman::voice::server::VoiceServerStatus {
|
||||
state: crate::openhuman::voice::server::ServerState::Stopped,
|
||||
hotkey: String::new(),
|
||||
activation_mode: crate::openhuman::voice::hotkey::ActivationMode::Tap,
|
||||
transcription_count: 0,
|
||||
last_error: None,
|
||||
};
|
||||
serde_json::to_value(status).map_err(|e| format!("serialize error: {e}"))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn handle_voice_server_status(_params: Map<String, Value>) -> ControllerFuture {
|
||||
Box::pin(async move {
|
||||
if let Some(server) = crate::openhuman::voice::server::try_global_server() {
|
||||
let status = server.status().await;
|
||||
serde_json::to_value(status).map_err(|e| format!("serialize error: {e}"))
|
||||
} else {
|
||||
let status = crate::openhuman::voice::server::VoiceServerStatus {
|
||||
state: crate::openhuman::voice::server::ServerState::Stopped,
|
||||
hotkey: String::new(),
|
||||
activation_mode: crate::openhuman::voice::hotkey::ActivationMode::Tap,
|
||||
transcription_count: 0,
|
||||
last_error: None,
|
||||
};
|
||||
serde_json::to_value(status).map_err(|e| format!("serialize error: {e}"))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,360 @@
|
||||
//! Standalone voice server — hotkey → record → transcribe → insert text.
|
||||
//!
|
||||
//! Can run as part of the core process or independently via the CLI.
|
||||
//! The server listens for a configurable hotkey, records audio from the
|
||||
//! microphone, transcribes via whisper, and inserts the result into the
|
||||
//! active text field.
|
||||
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::sync::Arc;
|
||||
|
||||
use log::{debug, error, info, warn};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use crate::openhuman::config::Config;
|
||||
|
||||
use super::audio_capture::{self, RecordingHandle};
|
||||
use super::hotkey::{self, ActivationMode, HotkeyEvent};
|
||||
use super::text_input;
|
||||
|
||||
const LOG_PREFIX: &str = "[voice_server]";
|
||||
|
||||
/// Running state of the voice server.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ServerState {
|
||||
/// Server is not running.
|
||||
Stopped,
|
||||
/// Server is running and idle, waiting for hotkey.
|
||||
Idle,
|
||||
/// Actively recording audio.
|
||||
Recording,
|
||||
/// Transcribing recorded audio.
|
||||
Transcribing,
|
||||
}
|
||||
|
||||
/// Status snapshot of the voice server.
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct VoiceServerStatus {
|
||||
pub state: ServerState,
|
||||
pub hotkey: String,
|
||||
pub activation_mode: ActivationMode,
|
||||
pub transcription_count: u64,
|
||||
pub last_error: Option<String>,
|
||||
}
|
||||
|
||||
/// Configuration for the voice server.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct VoiceServerConfig {
|
||||
pub hotkey: String,
|
||||
pub activation_mode: ActivationMode,
|
||||
/// Skip LLM post-processing on transcriptions.
|
||||
pub skip_cleanup: bool,
|
||||
/// Optional conversation context for better transcription accuracy.
|
||||
pub context: Option<String>,
|
||||
/// Minimum recording duration in seconds. Shorter recordings are discarded.
|
||||
pub min_duration_secs: f32,
|
||||
}
|
||||
|
||||
impl Default for VoiceServerConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
hotkey: "ctrl+shift+space".to_string(),
|
||||
activation_mode: ActivationMode::Tap,
|
||||
skip_cleanup: false,
|
||||
context: None,
|
||||
min_duration_secs: 0.3,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The voice server runtime.
|
||||
pub struct VoiceServer {
|
||||
state: Arc<Mutex<ServerState>>,
|
||||
cancel: CancellationToken,
|
||||
config: VoiceServerConfig,
|
||||
transcription_count: Arc<std::sync::atomic::AtomicU64>,
|
||||
last_error: Arc<Mutex<Option<String>>>,
|
||||
}
|
||||
|
||||
impl VoiceServer {
|
||||
pub fn new(config: VoiceServerConfig) -> Self {
|
||||
Self {
|
||||
state: Arc::new(Mutex::new(ServerState::Stopped)),
|
||||
cancel: CancellationToken::new(),
|
||||
config,
|
||||
transcription_count: Arc::new(std::sync::atomic::AtomicU64::new(0)),
|
||||
last_error: Arc::new(Mutex::new(None)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Get the current server status.
|
||||
pub async fn status(&self) -> VoiceServerStatus {
|
||||
VoiceServerStatus {
|
||||
state: *self.state.lock().await,
|
||||
hotkey: self.config.hotkey.clone(),
|
||||
activation_mode: self.config.activation_mode,
|
||||
transcription_count: self.transcription_count.load(Ordering::Relaxed),
|
||||
last_error: self.last_error.lock().await.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Run the voice server. Blocks until stopped.
|
||||
///
|
||||
/// This is the main entry point for both embedded and standalone modes.
|
||||
pub async fn run(&self, app_config: &Config) -> Result<(), String> {
|
||||
info!(
|
||||
"{LOG_PREFIX} starting voice server: hotkey={} mode={:?}",
|
||||
self.config.hotkey, self.config.activation_mode
|
||||
);
|
||||
|
||||
let combo = hotkey::parse_hotkey(&self.config.hotkey)?;
|
||||
let (listener_handle, mut hotkey_rx) =
|
||||
hotkey::start_listener(combo, self.config.activation_mode)?;
|
||||
|
||||
*self.state.lock().await = ServerState::Idle;
|
||||
|
||||
info!("{LOG_PREFIX} voice server ready, listening for hotkey");
|
||||
|
||||
let mut recording: Option<RecordingHandle> = None;
|
||||
|
||||
loop {
|
||||
let event = tokio::select! {
|
||||
ev = hotkey_rx.recv() => {
|
||||
match ev {
|
||||
Some(e) => e,
|
||||
None => {
|
||||
warn!("{LOG_PREFIX} hotkey channel closed");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
_ = self.cancel.cancelled() => {
|
||||
debug!("{LOG_PREFIX} cancellation received");
|
||||
break;
|
||||
}
|
||||
};
|
||||
|
||||
match event {
|
||||
HotkeyEvent::Pressed => {
|
||||
if recording.is_some() {
|
||||
// In tap mode, second press stops recording.
|
||||
debug!("{LOG_PREFIX} hotkey pressed while recording → stopping");
|
||||
if let Some(handle) = recording.take() {
|
||||
self.process_recording(handle, app_config).await;
|
||||
}
|
||||
} else {
|
||||
debug!("{LOG_PREFIX} hotkey pressed → starting recording");
|
||||
match audio_capture::start_recording() {
|
||||
Ok(handle) => {
|
||||
*self.state.lock().await = ServerState::Recording;
|
||||
recording = Some(handle);
|
||||
info!("{LOG_PREFIX} recording started");
|
||||
}
|
||||
Err(e) => {
|
||||
error!("{LOG_PREFIX} failed to start recording: {e}");
|
||||
*self.last_error.lock().await = Some(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
HotkeyEvent::Released => {
|
||||
// In push mode, release stops recording.
|
||||
if let Some(handle) = recording.take() {
|
||||
debug!("{LOG_PREFIX} hotkey released → stopping recording");
|
||||
self.process_recording(handle, app_config).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
listener_handle.stop();
|
||||
*self.state.lock().await = ServerState::Stopped;
|
||||
info!("{LOG_PREFIX} voice server stopped");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Stop the voice server.
|
||||
pub async fn stop(&self) {
|
||||
info!("{LOG_PREFIX} stopping voice server");
|
||||
self.cancel.cancel();
|
||||
}
|
||||
|
||||
/// Process a completed recording: transcribe and insert text.
|
||||
async fn process_recording(&self, handle: RecordingHandle, config: &Config) {
|
||||
*self.state.lock().await = ServerState::Transcribing;
|
||||
|
||||
match handle.stop().await {
|
||||
Ok(result) => {
|
||||
info!(
|
||||
"{LOG_PREFIX} recording stopped: {:.1}s, {} bytes",
|
||||
result.duration_secs,
|
||||
result.wav_bytes.len()
|
||||
);
|
||||
|
||||
if result.duration_secs < self.config.min_duration_secs {
|
||||
warn!(
|
||||
"{LOG_PREFIX} recording too short ({:.1}s), skipping",
|
||||
result.duration_secs
|
||||
);
|
||||
*self.state.lock().await = ServerState::Idle;
|
||||
return;
|
||||
}
|
||||
|
||||
match crate::openhuman::voice::voice_transcribe_bytes(
|
||||
config,
|
||||
&result.wav_bytes,
|
||||
Some("wav".to_string()),
|
||||
self.config.context.as_deref(),
|
||||
self.config.skip_cleanup,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(outcome) => {
|
||||
let text = &outcome.value.text;
|
||||
info!(
|
||||
"{LOG_PREFIX} transcription: '{}' ({} chars)",
|
||||
truncate_for_log(text, 80),
|
||||
text.len()
|
||||
);
|
||||
|
||||
if !text.trim().is_empty() {
|
||||
if let Err(e) = text_input::insert_text(text) {
|
||||
error!("{LOG_PREFIX} failed to insert text: {e}");
|
||||
*self.last_error.lock().await = Some(e);
|
||||
} else {
|
||||
self.transcription_count.fetch_add(1, Ordering::Relaxed);
|
||||
info!("{LOG_PREFIX} text inserted into active field");
|
||||
}
|
||||
} else {
|
||||
debug!("{LOG_PREFIX} transcription was empty, nothing to insert");
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
error!("{LOG_PREFIX} transcription failed: {e}");
|
||||
*self.last_error.lock().await = Some(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
error!("{LOG_PREFIX} failed to stop recording: {e}");
|
||||
*self.last_error.lock().await = Some(e);
|
||||
}
|
||||
}
|
||||
|
||||
*self.state.lock().await = ServerState::Idle;
|
||||
}
|
||||
}
|
||||
|
||||
/// Global voice server instance, lazily initialized.
|
||||
static VOICE_SERVER: once_cell::sync::OnceCell<Arc<VoiceServer>> = once_cell::sync::OnceCell::new();
|
||||
|
||||
/// Get or initialize the global voice server instance.
|
||||
pub fn global_server(config: VoiceServerConfig) -> Arc<VoiceServer> {
|
||||
VOICE_SERVER
|
||||
.get_or_init(|| Arc::new(VoiceServer::new(config)))
|
||||
.clone()
|
||||
}
|
||||
|
||||
/// Get the global voice server if already initialized.
|
||||
pub fn try_global_server() -> Option<Arc<VoiceServer>> {
|
||||
VOICE_SERVER.get().cloned()
|
||||
}
|
||||
|
||||
/// Run the voice server standalone (blocking). Intended for CLI usage.
|
||||
///
|
||||
/// Creates a fresh `VoiceServer` that is **not** registered in the global
|
||||
/// singleton used by `voice_server_status` RPC. This keeps CLI-started
|
||||
/// instances isolated from the core RPC lifecycle.
|
||||
pub async fn run_standalone(
|
||||
app_config: Config,
|
||||
server_config: VoiceServerConfig,
|
||||
) -> Result<(), String> {
|
||||
info!("{LOG_PREFIX} starting standalone voice server");
|
||||
info!("{LOG_PREFIX} hotkey: {}", server_config.hotkey);
|
||||
info!("{LOG_PREFIX} mode: {:?}", server_config.activation_mode);
|
||||
info!("{LOG_PREFIX} press the hotkey to start dictating");
|
||||
|
||||
let server = VoiceServer::new(server_config);
|
||||
|
||||
// Handle Ctrl+C gracefully.
|
||||
let server_arc = Arc::new(server);
|
||||
let server_for_signal = server_arc.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
if let Ok(()) = tokio::signal::ctrl_c().await {
|
||||
info!("{LOG_PREFIX} Ctrl+C received, shutting down");
|
||||
server_for_signal.stop().await;
|
||||
}
|
||||
});
|
||||
|
||||
// This is safe because we hold the Arc and nothing else moves it.
|
||||
// The server.run() borrows &self, and we await it to completion.
|
||||
server_arc.run(&app_config).await
|
||||
}
|
||||
|
||||
fn truncate_for_log(s: &str, max: usize) -> String {
|
||||
let truncated: String = s.chars().take(max).collect();
|
||||
if truncated.len() < s.len() {
|
||||
format!("{truncated}...")
|
||||
} else {
|
||||
truncated
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn default_server_config() {
|
||||
let cfg = VoiceServerConfig::default();
|
||||
assert_eq!(cfg.hotkey, "ctrl+shift+space");
|
||||
assert_eq!(cfg.activation_mode, ActivationMode::Tap);
|
||||
assert!(!cfg.skip_cleanup);
|
||||
assert!(cfg.context.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn server_status_initial() {
|
||||
let server = VoiceServer::new(VoiceServerConfig::default());
|
||||
let status = server.status().await;
|
||||
assert_eq!(status.state, ServerState::Stopped);
|
||||
assert_eq!(status.transcription_count, 0);
|
||||
assert!(status.last_error.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn server_state_serializes() {
|
||||
let json = serde_json::to_string(&ServerState::Recording).unwrap();
|
||||
assert_eq!(json, "\"recording\"");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn voice_server_status_serializes() {
|
||||
let status = VoiceServerStatus {
|
||||
state: ServerState::Idle,
|
||||
hotkey: "ctrl+shift+space".into(),
|
||||
activation_mode: ActivationMode::Tap,
|
||||
transcription_count: 5,
|
||||
last_error: None,
|
||||
};
|
||||
let v = serde_json::to_value(&status).unwrap();
|
||||
assert_eq!(v["state"], "idle");
|
||||
assert_eq!(v["transcription_count"], 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_for_log_short() {
|
||||
assert_eq!(truncate_for_log("hello", 10), "hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncate_for_log_long() {
|
||||
let result = truncate_for_log("hello world this is long", 10);
|
||||
assert!(result.ends_with("..."));
|
||||
assert!(result.len() <= 14); // 10 + "..."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
//! WebSocket streaming transcription endpoint.
|
||||
//!
|
||||
//! Accepts a WebSocket connection that receives PCM16 audio chunks (16kHz mono)
|
||||
//! and periodically runs whisper inference on the accumulated buffer, sending
|
||||
//! back partial transcription results as JSON messages.
|
||||
//!
|
||||
//! Protocol:
|
||||
//! Client → Server: binary frames containing PCM16 LE audio bytes (16kHz mono)
|
||||
//! Server → Client: JSON text frames:
|
||||
//! { "type": "partial", "text": "..." } — interim transcription
|
||||
//! { "type": "final", "text": "...", "raw_text": "..." } — after client sends
|
||||
//! `{"type":"stop"}` text frame
|
||||
//! { "type": "error", "message": "..." } — on error
|
||||
//! Client → Server: text frame `{"type":"stop"}` — end recording, get final result
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::extract::ws::{Message, WebSocket};
|
||||
use serde::Deserialize;
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
use super::postprocess;
|
||||
use crate::openhuman::config::Config;
|
||||
use crate::openhuman::local_ai;
|
||||
use crate::openhuman::local_ai::whisper_engine;
|
||||
|
||||
const LOG_PREFIX: &str = "[voice-stream]";
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct ClientCommand {
|
||||
#[serde(rename = "type")]
|
||||
cmd_type: String,
|
||||
}
|
||||
|
||||
/// Handle an upgraded WebSocket connection for streaming dictation.
|
||||
pub async fn handle_dictation_ws(mut socket: WebSocket, config: Arc<Config>) {
|
||||
log::info!("{LOG_PREFIX} new streaming dictation connection");
|
||||
|
||||
let audio_buf: Arc<Mutex<Vec<i16>>> = Arc::new(Mutex::new(Vec::new()));
|
||||
let interval_ms = config.dictation.streaming_interval_ms;
|
||||
let do_streaming = config.dictation.streaming;
|
||||
|
||||
// Periodic inference task — runs every `interval_ms` on the accumulated buffer
|
||||
let buf_clone = audio_buf.clone();
|
||||
let config_clone = config.clone();
|
||||
let (partial_tx, mut partial_rx) = tokio::sync::mpsc::channel::<String>(8);
|
||||
|
||||
let inference_handle = if do_streaming {
|
||||
let handle = tokio::spawn(async move {
|
||||
let mut interval =
|
||||
tokio::time::interval(std::time::Duration::from_millis(interval_ms.max(500)));
|
||||
let mut last_len: usize = 0;
|
||||
|
||||
loop {
|
||||
interval.tick().await;
|
||||
|
||||
let samples: Vec<i16> = {
|
||||
let guard = buf_clone.lock().await;
|
||||
if guard.len() <= last_len || guard.len() < 8000 {
|
||||
// No new data or less than 0.5s of audio — skip
|
||||
continue;
|
||||
}
|
||||
last_len = guard.len();
|
||||
guard.clone()
|
||||
};
|
||||
|
||||
let service = local_ai::global(&config_clone);
|
||||
match whisper_engine::transcribe_pcm_i16(&service.whisper, &samples, None) {
|
||||
Ok(text) => {
|
||||
let trimmed = text.trim().to_string();
|
||||
if !trimmed.is_empty() {
|
||||
log::debug!(
|
||||
"{LOG_PREFIX} partial transcription ({} samples): {}",
|
||||
samples.len(),
|
||||
&trimmed[..trimmed.len().min(80)]
|
||||
);
|
||||
if partial_tx.send(trimmed).await.is_err() {
|
||||
break; // receiver dropped
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
log::warn!("{LOG_PREFIX} partial inference error: {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
Some(handle)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
loop {
|
||||
tokio::select! {
|
||||
// Forward partial results to the client
|
||||
Some(partial_text) = partial_rx.recv() => {
|
||||
let msg = serde_json::json!({
|
||||
"type": "partial",
|
||||
"text": partial_text,
|
||||
});
|
||||
if socket.send(Message::Text(msg.to_string().into())).await.is_err() {
|
||||
log::debug!("{LOG_PREFIX} client disconnected while sending partial");
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Receive audio data or commands from the client
|
||||
msg = socket.recv() => {
|
||||
match msg {
|
||||
Some(Ok(Message::Binary(data))) => {
|
||||
// PCM16 LE bytes → i16 samples
|
||||
if data.len() % 2 != 0 {
|
||||
log::warn!("{LOG_PREFIX} received odd-length binary frame, skipping");
|
||||
continue;
|
||||
}
|
||||
let samples: Vec<i16> = data
|
||||
.chunks_exact(2)
|
||||
.map(|chunk| i16::from_le_bytes([chunk[0], chunk[1]]))
|
||||
.collect();
|
||||
|
||||
let mut buf = audio_buf.lock().await;
|
||||
buf.extend_from_slice(&samples);
|
||||
log::trace!(
|
||||
"{LOG_PREFIX} buffered {} new samples, total {}",
|
||||
samples.len(),
|
||||
buf.len()
|
||||
);
|
||||
}
|
||||
|
||||
Some(Ok(Message::Text(text))) => {
|
||||
if let Ok(cmd) = serde_json::from_str::<ClientCommand>(&text) {
|
||||
if cmd.cmd_type == "stop" {
|
||||
log::info!("{LOG_PREFIX} stop command received, running final inference");
|
||||
break; // fall through to final transcription
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Some(Ok(Message::Close(_))) | None => {
|
||||
log::info!("{LOG_PREFIX} client disconnected");
|
||||
if let Some(h) = inference_handle {
|
||||
h.abort();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
Some(Err(e)) => {
|
||||
log::warn!("{LOG_PREFIX} websocket error: {e}");
|
||||
if let Some(h) = inference_handle {
|
||||
h.abort();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Stop the periodic inference task
|
||||
if let Some(h) = inference_handle {
|
||||
h.abort();
|
||||
}
|
||||
|
||||
// Run final transcription on the complete buffer
|
||||
let final_samples = audio_buf.lock().await.clone();
|
||||
if final_samples.is_empty() {
|
||||
let msg = serde_json::json!({
|
||||
"type": "final",
|
||||
"text": "",
|
||||
"raw_text": "",
|
||||
});
|
||||
let _ = socket.send(Message::Text(msg.to_string().into())).await;
|
||||
return;
|
||||
}
|
||||
|
||||
log::info!(
|
||||
"{LOG_PREFIX} running final inference on {} samples ({:.1}s)",
|
||||
final_samples.len(),
|
||||
final_samples.len() as f64 / 16000.0
|
||||
);
|
||||
|
||||
let service = local_ai::global(&config);
|
||||
let raw_text = match whisper_engine::transcribe_pcm_i16(&service.whisper, &final_samples, None)
|
||||
{
|
||||
Ok(text) => text.trim().to_string(),
|
||||
Err(e) => {
|
||||
log::error!("{LOG_PREFIX} final inference error: {e}");
|
||||
let msg = serde_json::json!({
|
||||
"type": "error",
|
||||
"message": format!("Transcription failed: {e}"),
|
||||
});
|
||||
let _ = socket.send(Message::Text(msg.to_string().into())).await;
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
// LLM refinement if enabled
|
||||
let refined_text = if config.dictation.llm_refinement && !raw_text.is_empty() {
|
||||
postprocess::cleanup_transcription(&config, &raw_text, None).await
|
||||
} else {
|
||||
raw_text.clone()
|
||||
};
|
||||
|
||||
let msg = serde_json::json!({
|
||||
"type": "final",
|
||||
"text": refined_text,
|
||||
"raw_text": raw_text,
|
||||
});
|
||||
let _ = socket.send(Message::Text(msg.to_string().into())).await;
|
||||
log::info!("{LOG_PREFIX} streaming session complete");
|
||||
// Socket is dropped here, which sends a close frame automatically
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
//! Text insertion into the currently active text field.
|
||||
//!
|
||||
//! Uses enigo to simulate keyboard input so that transcribed text
|
||||
//! appears in whatever application has focus.
|
||||
|
||||
use log::{debug, info, warn};
|
||||
|
||||
use enigo::{Enigo, Keyboard, Settings};
|
||||
|
||||
const LOG_PREFIX: &str = "[voice_input]";
|
||||
|
||||
/// Insert text into the currently active text field via enigo.
|
||||
///
|
||||
/// Skips empty or whitespace-only input. Uses enigo's `text()` method
|
||||
/// which handles Unicode and platform-appropriate input simulation.
|
||||
pub fn insert_text(text: &str) -> Result<(), String> {
|
||||
if text.trim().is_empty() {
|
||||
warn!("{LOG_PREFIX} transcription was empty/whitespace, skipping insertion");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
info!(
|
||||
"{LOG_PREFIX} inserting {} chars into active field",
|
||||
text.len()
|
||||
);
|
||||
|
||||
let mut enigo = Enigo::new(&Settings::default())
|
||||
.map_err(|e| format!("failed to create enigo instance: {e}"))?;
|
||||
|
||||
enigo
|
||||
.text(text)
|
||||
.map_err(|e| format!("failed to insert text: {e}"))?;
|
||||
|
||||
debug!("{LOG_PREFIX} text inserted successfully");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn empty_text_is_noop() {
|
||||
assert!(insert_text("").is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn whitespace_only_skips_insertion() {
|
||||
assert!(insert_text(" ").is_ok());
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user