mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
feat: local model support and SocketManager WebSocket transport (#36)
* chore: add simulation function for deep link testing in development - Introduced a simulation function for deep links in the development environment to facilitate testing. - The function allows developers to simulate deep link URLs without affecting production behavior. * feat: implement QuickJS skill runtime integration - Added a new QuickJS skill runtime engine to manage skill execution within the application. - Introduced commands for skill management, including discovery, starting, stopping, and querying skill states. - Implemented a SQLite database bridge for each skill to handle data storage. - Enhanced the application structure with new modules for runtime management, skill instances, and manifest parsing. - Updated dependencies in Cargo.toml and Cargo.lock to support new features. * refactor: improve code readability and formatting in DownloadScreen and deviceDetection - Enhanced the button className formatting in DownloadScreen for better readability. - Reformatted the fetchLatestRelease and parseReleaseAssetsByArchitecture functions in deviceDetection for improved clarity and maintainability. - Utilized multiline formatting for complex conditions to enhance code structure. * feat: integrate Rust-native Socket.io client for persistent connections - Implemented a SocketManager in Rust to handle Socket.io connections, ensuring persistence across app backgrounding. - Updated SocketProvider to connect/disconnect using Rust-native methods in Tauri mode. - Enhanced Tauri event listeners for socket state changes and server events. - Refactored socket handling logic to differentiate between web and Tauri modes, improving maintainability and clarity. - Added new commands for connecting, disconnecting, and emitting events through the Rust socket. * refactor: remove Telegram login commands and related functionality - Deleted the startTelegramLogin and startTelegramLoginWithUrl functions from the Tauri commands. - Removed associated references in the Rust command module and utility file. - This cleanup simplifies the codebase by eliminating unused Telegram login features. * feat: add cron scheduling functionality for skills - Introduced a new CronScheduler to manage scheduled tasks for skills, allowing for cron expression parsing and execution. - Added a cron bridge to expose scheduling methods to skill contexts, enabling skills to register, unregister, and list cron schedules. - Updated the RuntimeEngine to initialize and manage the cron scheduler, ensuring it runs in the background. - Enhanced skill instances to support cron scheduling through a new BridgeDeps structure. - Updated dependencies in Cargo.toml and Cargo.lock to include the cron crate. * feat: add Android support and enhance dependencies - Introduced Android-specific build and run commands in package.json for Tauri. - Updated Cargo.toml to include OpenSSL for Android and adjusted reqwest for cross-platform TLS support. - Added new mobile capabilities configuration for Android and iOS. - Implemented a foreground service in the Android app to maintain the Rust backend process. - Added various Android resources including layouts, icons, and notification handling. - Updated Cargo.lock with new dependencies for improved functionality. * chore: update .gitignore and AndroidManifest.xml for deep link support - Added .kotlin and .cargo to .gitignore to exclude Kotlin and Cargo build artifacts. - Updated AndroidManifest.xml with comments for the deep link plugin, clarifying its auto-generated nature. * feat: enhance store configuration for development testing - Added functionality to auto-inject a JWT token from environment variables during development. - Implemented automatic onboarding for users once their profile is fetched, improving the testing experience. * feat: enhance socket connection handling and add Android logging support - Updated the Rust socket connection command to accept an optional URL parameter, allowing for dynamic backend URL configuration. - Integrated Android logging capabilities using the android_logger crate, ensuring logs are properly routed to logcat. - Improved error handling in the SocketManager to log connection errors and successful connections for better debugging. * feat: implement notification permission handling and enhance foreground service in Android - Added runtime permission request for POST_NOTIFICATIONS in MainActivity to comply with API 33+ requirements. - Updated RuntimeService to specify foreground service type for API 34+ compatibility. - Improved logging levels in the Rust backend for better debugging and monitoring of socket connections and skill discovery. * feat: integrate QuickJS skill management and service - Added QuickJS skill hooks for retrieving and managing QuickJS skills from Redux. - Implemented a QuickJS service to handle skill lifecycle, preferences, and IPC calls with the Rust backend. - Enhanced the skills state in Redux to include QuickJS skills, enabling better management and state tracking. - Updated the store configuration to persist QuickJS skills state across sessions. - Introduced new commands in the Rust backend for enabling/disabling skills and managing preferences. - Improved the SkillProvider to initialize the QuickJS service during app startup. * refactor: transition skill management to QuickJS runtime - Removed legacy skill catalog and loading logic, replacing it with QuickJS runtime integration for skill discovery and management. - Updated SkillProvider to utilize QuickJS for skill registration and lifecycle management. - Simplified skill data handling by directly invoking the Rust backend for skill operations. - Enhanced error handling and logging for skill loading processes. - Cleaned up unused interfaces and functions related to previous skill management methods. * refactor: remove unused skill commands and plugins - Deleted the skills command module and related functions to streamline the codebase. - Removed dependencies on the tauri-plugin-shell and other unused plugins from Cargo.toml and Cargo.lock. - Updated the authentication module by removing the exchange_token function and its associated structures. - Simplified the capabilities configuration by eliminating shell-related permissions. - Enhanced the runtime engine to support new JSON-RPC commands for skill data management. * feat: expose whitelisted environment values to skills - Added `platform.env(key)` function to retrieve whitelisted environment values for skills. - Implemented `get_skill_env` function to provide access to `BACKEND_URL` and `PLATFORM`. - Updated `get_backend_url` to check for `VITE_BACKEND_URL` before falling back to `BACKEND_URL`. - Enhanced the QuickJS runtime to support the new environment functionality. * feat: fix skills enable/disable flow, setup pipeline, and status derivation - Add SkillSetup struct to Rust manifest and include setup field in discovery response so the frontend knows which skills need setup - Map setup field in SkillProvider discoverSkills() - Fix SkillsGrid to use real hasSetup from manifest instead of hardcoding false - Add contextual Enable/Setup/Configure/Retry buttons in management modal - Add status indicator dots to compact skill table rows - Fix deriveConnectionStatus to return "connected" for ready skills with completed setup that don't push host state (e.g. cron-based skills) - Add select option renderer in SkillManagementPanel for non-boolean options - Add dotenvy crate to load .env file at Rust startup so env vars like VITE_BACKEND_URL are available to the runtime engine and skills Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add platform support for skills and enhance manifest handling - Introduced platform filtering for skills in the SkillManifest, allowing skills to specify supported platforms. - Updated QuickJSManifest and runtime engine to handle platform checks, ensuring skills are only loaded on compatible platforms. - Added new build and watch commands for skills in package.json to streamline development. - Enhanced the Rust backend to log unsupported skills based on platform restrictions. * feat: update skills submodule with TypeScript pipeline and test harness Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: transition from QuickJS to V8 runtime for skill management - Replaced QuickJS with V8 (via deno_core) for improved JavaScript execution and WASM support. - Updated skill management logic to utilize the new V8 runtime, including changes to the RuntimeEngine and skill instance handling. - Enhanced manifest handling to support the new runtime and added platform compatibility checks. - Introduced new dependencies and updated Cargo.toml to reflect changes in the skill execution environment. - Refactored related modules and commands to align with the V8 integration, ensuring a seamless transition for skill operations. * refactor: clean up dead code in IdbStorage and enhance ops module - Removed unused `#[allow(dead_code)]` annotations from the IdbStorage struct and its methods to improve code clarity. - Introduced new timer and WebSocket state management structures in the ops module, laying groundwork for future enhancements. - Added timer and WebSocket operation functions, including `op_timer_start`, `op_timer_cancel`, and WebSocket connection handling, to support asynchronous operations. * feat: implement high-level TDLib service with V8 integration - Introduced a new `TdlibV8Service` to manage TDLib client instances using the V8 runtime. - Added a `MockTdClient` for development and testing, simulating TDLib responses for various queries. - Enhanced the bootstrap script to initialize and manage TDLib clients, including methods for sending queries and retrieving authentication states. - Updated the `mod.rs` file to include the new service module and improved documentation for clarity. - Refactored existing code to support the new TDLib integration, ensuring a seamless experience for skill management. * refactor: update V8 runtime integration and platform handling - Updated the V8 runtime integration to ensure it is only available on desktop platforms, with appropriate error handling for mobile. - Refactored the `SocketManager` and command implementations to conditionally include desktop-only features, enhancing clarity and maintainability. - Cleaned up the `Cargo.toml` to reflect the changes in V8 runtime availability and added relevant documentation. - Removed dead code related to mobile platform handling in the TDLib integration, ensuring a streamlined codebase. * chore: update skills submodule to latest commit - Updated the skills submodule to the latest commit (19a18e8), ensuring alignment with recent changes and improvements in the codebase. * refactor: transition from QuickJS to V8 runtime for skill management - Updated the skills submodule to the latest commit, reflecting the transition from QuickJS to V8 for improved JavaScript execution. - Refactored skill management logic to utilize the V8 runtime, including changes to the RuntimeEngine and skill instance handling. - Enhanced manifest handling to support the new runtime and updated platform compatibility checks. - Removed dead code related to QuickJS, ensuring a streamlined codebase. - Updated comments and documentation to reflect the changes in runtime integration. * feat: enhance logging and update timer operations in V8 integration - Added logging for skill discovery and manifest processing in the V8 runtime, improving traceability during skill management. - Updated timer operation functions to use new prefixed names (`op_ah_timer_start` and `op_ah_timer_cancel`) to avoid conflicts with deno_core built-ins. - Implemented a mechanism to load .env files from various locations, ensuring environment variables are available for configuration. - Enhanced the `get_backend_url` function to include debug logging for better visibility of the backend URL resolution process. * chore: update skills submodule to latest commit and enhance logging - Updated the skills submodule to the latest commit (54c40a1), ensuring alignment with recent changes. - Added console logging for skill manifests during discovery in the SkillsGrid component to improve debugging and traceability. * refactor: clean up and format code in DownloadScreen and SkillsGrid components - Improved code readability by formatting multi-line statements in the DownloadScreen and SkillsGrid components. - Removed unnecessary type imports in DownloadScreen for clarity. - Enhanced the structure of the SkillsGrid component by adjusting the layout of JSX elements for better maintainability. - Updated socketService to ensure consistent import order and improved logging in TauriSocket utility functions. * chore: update development script in package.json - Replaced the existing setup-python-sidecar script with a new dev:app script to streamline the development process for Tauri applications, enabling better debugging with RUST_BACKTRACE and RUST_LOG settings. * refactor: update load method to accept additional parameters - Modified the load method in SkillRuntime to accept an optional additionalParams argument, enhancing flexibility for skill loading. - Ensured compatibility by using a fallback to an empty object when additionalParams is not provided. * refactor: improve JSX structure in SkillsGrid component - Enhanced the formatting of the connection status indicator in the SkillsGrid component for better readability. - Adjusted the indentation and spacing to maintain consistent code style and improve maintainability. * Refactor invoke_handler to consolidate command registration across platforms - Simplified the command registration process by merging desktop and mobile command handlers into a single `invoke_handler` call. - Improved code readability and maintainability by reducing duplication in command definitions. - Ensured that window commands remain desktop-only while maintaining common functionality for all platforms. * updated icon * Update skills submodule to latest commit and remove redundant logging statements in V8 engine skill discovery * Update skills submodule to reflect dirty state * feat: expose BACKEND_URL environment variable for skills - Added "BACKEND_URL" to the list of whitelisted environment variables accessible to skills, allowing for broader usage without the "VITE_" prefix. * Enhance V8 runtime with async event loop and timer management - Implemented an async event loop in the V8 skill instance to handle timers and messages efficiently. - Introduced a new TimerState structure for managing scheduled timers, allowing for better integration with the V8 event loop. - Updated the dotenv loading mechanism to check for environment variables in the current and parent directories. - Enhanced logging for timer operations and skill management processes to improve traceability. * Update skills submodule to latest commit and expose additional environment variables for skills - Updated the skills submodule to commit 3793fdc, ensuring alignment with recent changes. - Added "TELEGRAM_API_ID" and "TELEGRAM_API_HASH" to the list of whitelisted environment variables, allowing skills to access these without the "VITE_" prefix. * Integrate TDLib support for Telegram skill - Added TDLib commands for creating, sending, receiving, and destroying clients, enabling Telegram functionality. - Implemented a TdLibManager for managing TDLib client lifecycle and asynchronous operations on desktop platforms. - Introduced JNI bridge for Android to facilitate TDLib interactions. - Updated Cargo.toml and Cargo.lock to include tdlib-rs and related dependencies. - Enhanced the V8 runtime to support TDLib operations, ensuring compatibility across platforms. * Enhance TDLib manager with update queue and async polling - Introduced an update queue and notification channel in ClientState for managing TDLib updates. - Implemented a separate polling task for TDLib updates to improve responsiveness and prevent blocking the main event loop. - Updated lifecycle function handling in the V8 runtime to avoid waiting for the event loop, allowing for long-running async operations. - Improved error handling and logging for better traceability during TDLib operations. * Enhance documentation and improve code formatting - Updated CLAUDE.md to include new core commands and runtime management features for the Rust backend, as well as Android support and platform-specific details. - Refactored SettingsHeader component in SettingsHeader.tsx for improved readability by consolidating props and cleaning up JSX formatting. - Enhanced SkillProvider.tsx with better logging for skill loading errors, improving traceability in the development process. * Integrate local LLM inference support and update dependencies - Added local LLM inference capabilities for desktop and Android platforms, excluding iOS. - Introduced new model commands for checking availability, generating text, and summarizing content. - Updated Cargo.toml and Cargo.lock to include necessary dependencies such as `llama-cpp-2` and `encoding_rs`. - Enhanced the V8 runtime and JavaScript API to support model operations, improving the overall functionality of the application. * Enhance SocketManager with WebSocket transport type - Added support for WebSocket transport type in SocketManager for improved connection handling. - Updated connection configuration to utilize WebSocketUpgrade, enhancing real-time communication capabilities. * style: fix Prettier formatting in ModelDownloadProgress * chore: update skills submodule to latest commit --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Steven Enamakel
co-authored by
Claude Opus 4.5
Claude Opus 4.5
parent
f0e3c03544
commit
3fb5e1dd11
+1
-1
Submodule skills updated: 79e21914f4...d9dfb8d974
Generated
+76
-1
@@ -15,10 +15,12 @@ dependencies = [
|
||||
"deno_core",
|
||||
"dirs 5.0.1",
|
||||
"dotenvy",
|
||||
"encoding_rs",
|
||||
"env_logger",
|
||||
"futures",
|
||||
"futures-util",
|
||||
"keyring",
|
||||
"llama-cpp-2",
|
||||
"log",
|
||||
"once_cell",
|
||||
"openssl",
|
||||
@@ -503,6 +505,26 @@ dependencies = [
|
||||
"which 4.4.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "bindgen"
|
||||
version = "0.72.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895"
|
||||
dependencies = [
|
||||
"bitflags 2.10.0",
|
||||
"cexpr",
|
||||
"clang-sys",
|
||||
"itertools",
|
||||
"log",
|
||||
"prettyplease",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"regex",
|
||||
"rustc-hash 2.1.1",
|
||||
"shlex",
|
||||
"syn 2.0.114",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "bit-set"
|
||||
version = "0.5.3"
|
||||
@@ -814,6 +836,15 @@ dependencies = [
|
||||
"libloading 0.8.9",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cmake"
|
||||
version = "0.1.57"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "75443c44cd6b379beb8c5b45d85d0773baf31cce901fe7bb252f4eff3008ef7d"
|
||||
dependencies = [
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "colorchoice"
|
||||
version = "1.0.4"
|
||||
@@ -1564,6 +1595,15 @@ version = "0.1.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8591b0bcc8a98a64310a2fae1bb3e9b8564dd10e381e6e28010fde8e8e8568db"
|
||||
|
||||
[[package]]
|
||||
name = "find_cuda_helper"
|
||||
version = "0.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f9f9e65c593dd01ac77daad909ea4ad17f0d6d1776193fc8ea766356177abdad"
|
||||
dependencies = [
|
||||
"glob",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "flate2"
|
||||
version = "1.1.8"
|
||||
@@ -2804,6 +2844,34 @@ version = "0.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77"
|
||||
|
||||
[[package]]
|
||||
name = "llama-cpp-2"
|
||||
version = "0.1.133"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "888c8805527f4c35ec16f26003d54a318cde1629e7439da8e9ef2d6d3883e106"
|
||||
dependencies = [
|
||||
"encoding_rs",
|
||||
"enumflags2",
|
||||
"llama-cpp-sys-2",
|
||||
"thiserror 1.0.69",
|
||||
"tracing",
|
||||
"tracing-core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "llama-cpp-sys-2"
|
||||
version = "0.1.133"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a180dfa6d6f9d1df1e031bcdf0464bbad4f9b326395bfd28f2fa539d8cbc9c2b"
|
||||
dependencies = [
|
||||
"bindgen 0.72.1",
|
||||
"cc",
|
||||
"cmake",
|
||||
"find_cuda_helper",
|
||||
"glob",
|
||||
"walkdir",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lock_api"
|
||||
version = "0.4.14"
|
||||
@@ -5885,6 +5953,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
|
||||
dependencies = [
|
||||
"once_cell",
|
||||
"valuable",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -6114,7 +6183,7 @@ version = "0.106.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a381badc47c6f15acb5fe0b5b40234162349ed9d4e4fd7c83a7f5547c0fc69c5"
|
||||
dependencies = [
|
||||
"bindgen",
|
||||
"bindgen 0.69.5",
|
||||
"bitflags 2.10.0",
|
||||
"fslock",
|
||||
"gzip-header",
|
||||
@@ -6125,6 +6194,12 @@ dependencies = [
|
||||
"which 6.0.3",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "valuable"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
|
||||
|
||||
[[package]]
|
||||
name = "vcpkg"
|
||||
version = "0.2.15"
|
||||
|
||||
@@ -97,6 +97,11 @@ tdlib-rs = { version = "1.2", features = ["download-tdlib"] }
|
||||
# TDLib build configuration
|
||||
tdlib-rs = { version = "1.2", features = ["download-tdlib"] }
|
||||
|
||||
# Local LLM inference - available on desktop and Android (not iOS)
|
||||
[target.'cfg(not(target_os = "ios"))'.dependencies]
|
||||
llama-cpp-2 = "0.1"
|
||||
encoding_rs = "0.8"
|
||||
|
||||
[features]
|
||||
# This feature is used for production builds or when a dev server is not specified, DO NOT REMOVE!!
|
||||
custom-protocol = ["tauri/custom-protocol"]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
pub mod auth;
|
||||
pub mod model;
|
||||
pub mod runtime;
|
||||
pub mod socket;
|
||||
pub mod tdlib;
|
||||
@@ -8,6 +9,7 @@ pub mod window;
|
||||
|
||||
// Re-export all commands for registration
|
||||
pub use auth::*;
|
||||
pub use model::*;
|
||||
pub use runtime::*;
|
||||
pub use socket::*;
|
||||
pub use tdlib::*;
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
//! Model Tauri Commands
|
||||
//!
|
||||
//! These commands provide local LLM access via Tauri's invoke() system.
|
||||
//! Available on desktop and Android (not iOS).
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Model status response for frontend.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ModelStatusResponse {
|
||||
/// Whether the model API is available on this platform.
|
||||
pub available: bool,
|
||||
/// Whether the model is currently loaded in memory.
|
||||
pub loaded: bool,
|
||||
/// Whether the model is currently being loaded or downloaded.
|
||||
pub loading: bool,
|
||||
/// Download progress (0.0 to 1.0) if downloading.
|
||||
pub download_progress: Option<f32>,
|
||||
/// Error message if loading failed.
|
||||
pub error: Option<String>,
|
||||
/// Model file path if known.
|
||||
pub model_path: Option<String>,
|
||||
}
|
||||
|
||||
/// Generation configuration from frontend.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct GenerateRequest {
|
||||
/// Input prompt.
|
||||
pub prompt: String,
|
||||
/// Maximum tokens to generate (default: 2048).
|
||||
#[serde(default = "default_max_tokens")]
|
||||
pub max_tokens: u32,
|
||||
/// Sampling temperature (default: 0.7).
|
||||
#[serde(default = "default_temperature")]
|
||||
pub temperature: f32,
|
||||
/// Top-p sampling (default: 0.9).
|
||||
#[serde(default = "default_top_p")]
|
||||
pub top_p: f32,
|
||||
}
|
||||
|
||||
fn default_max_tokens() -> u32 {
|
||||
2048
|
||||
}
|
||||
fn default_temperature() -> f32 {
|
||||
0.7
|
||||
}
|
||||
fn default_top_p() -> f32 {
|
||||
0.9
|
||||
}
|
||||
|
||||
/// Check if the local model API is available on this platform.
|
||||
#[tauri::command]
|
||||
pub fn model_is_available() -> bool {
|
||||
#[cfg(not(target_os = "ios"))]
|
||||
{
|
||||
true
|
||||
}
|
||||
|
||||
#[cfg(target_os = "ios")]
|
||||
{
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
/// Get the current model status.
|
||||
#[tauri::command]
|
||||
pub fn model_get_status() -> ModelStatusResponse {
|
||||
#[cfg(not(target_os = "ios"))]
|
||||
{
|
||||
let status = crate::services::llama::LLAMA_MANAGER.get_status();
|
||||
ModelStatusResponse {
|
||||
available: status.available,
|
||||
loaded: status.loaded,
|
||||
loading: status.loading,
|
||||
download_progress: status.download_progress,
|
||||
error: status.error,
|
||||
model_path: status.model_path,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(target_os = "ios")]
|
||||
{
|
||||
ModelStatusResponse {
|
||||
available: false,
|
||||
loaded: false,
|
||||
loading: false,
|
||||
download_progress: None,
|
||||
error: Some("Model not available on iOS".to_string()),
|
||||
model_path: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Ensure the model is loaded (downloads if necessary).
|
||||
/// This is useful for preloading the model.
|
||||
#[tauri::command]
|
||||
pub async fn model_ensure_loaded() -> Result<(), String> {
|
||||
#[cfg(not(target_os = "ios"))]
|
||||
{
|
||||
crate::services::llama::LLAMA_MANAGER.ensure_loaded().await
|
||||
}
|
||||
|
||||
#[cfg(target_os = "ios")]
|
||||
{
|
||||
Err("Model not available on iOS".to_string())
|
||||
}
|
||||
}
|
||||
|
||||
/// Generate text from a prompt.
|
||||
#[tauri::command]
|
||||
pub async fn model_generate(request: GenerateRequest) -> Result<String, String> {
|
||||
#[cfg(not(target_os = "ios"))]
|
||||
{
|
||||
use crate::services::llama::GenerateConfig;
|
||||
|
||||
let config = GenerateConfig {
|
||||
max_tokens: request.max_tokens,
|
||||
temperature: request.temperature,
|
||||
top_p: request.top_p,
|
||||
};
|
||||
|
||||
crate::services::llama::LLAMA_MANAGER
|
||||
.generate(&request.prompt, config)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(target_os = "ios")]
|
||||
{
|
||||
let _ = request;
|
||||
Err("Model not available on iOS".to_string())
|
||||
}
|
||||
}
|
||||
|
||||
/// Summarize text using a built-in prompt.
|
||||
#[tauri::command]
|
||||
pub async fn model_summarize(text: String, max_tokens: Option<u32>) -> Result<String, String> {
|
||||
#[cfg(not(target_os = "ios"))]
|
||||
{
|
||||
let tokens = max_tokens.unwrap_or(500);
|
||||
crate::services::llama::LLAMA_MANAGER
|
||||
.summarize(&text, tokens)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(target_os = "ios")]
|
||||
{
|
||||
let _ = (text, max_tokens);
|
||||
Err("Model not available on iOS".to_string())
|
||||
}
|
||||
}
|
||||
|
||||
/// Unload the model from memory to free resources.
|
||||
#[tauri::command]
|
||||
pub fn model_unload() -> Result<(), String> {
|
||||
#[cfg(not(target_os = "ios"))]
|
||||
{
|
||||
crate::services::llama::LLAMA_MANAGER.unload();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(target_os = "ios")]
|
||||
{
|
||||
Err("Model not available on iOS".to_string())
|
||||
}
|
||||
}
|
||||
+39
-2
@@ -304,6 +304,11 @@ pub fn run() {
|
||||
});
|
||||
let skills_data_dir = data_dir.join("skills");
|
||||
|
||||
// Initialize local model service (for skills to use)
|
||||
let model_dir = data_dir.join("models");
|
||||
services::llama::LLAMA_MANAGER.set_data_dir(model_dir);
|
||||
log::info!("[runtime] Local model service initialized");
|
||||
|
||||
match runtime::v8_engine::RuntimeEngine::new(skills_data_dir) {
|
||||
Ok(engine) => {
|
||||
engine.set_app_handle(app.handle().clone());
|
||||
@@ -334,9 +339,27 @@ pub fn run() {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(any(target_os = "android", target_os = "ios"))]
|
||||
#[cfg(target_os = "android")]
|
||||
{
|
||||
log::info!("[runtime] V8 runtime disabled on mobile platform");
|
||||
log::info!("[runtime] V8 runtime disabled on Android");
|
||||
|
||||
// Initialize local model service for Android
|
||||
let data_dir = app
|
||||
.path()
|
||||
.app_data_dir()
|
||||
.unwrap_or_else(|_| {
|
||||
dirs::home_dir()
|
||||
.unwrap_or_else(|| std::path::PathBuf::from("."))
|
||||
.join(".alphahuman")
|
||||
});
|
||||
let model_dir = data_dir.join("models");
|
||||
services::llama::LLAMA_MANAGER.set_data_dir(model_dir);
|
||||
log::info!("[runtime] Local model service initialized for Android");
|
||||
}
|
||||
|
||||
#[cfg(target_os = "ios")]
|
||||
{
|
||||
log::info!("[runtime] V8 runtime and local model disabled on iOS");
|
||||
}
|
||||
|
||||
// Store SocketManager as Tauri state
|
||||
@@ -450,6 +473,13 @@ pub fn run() {
|
||||
tdlib_receive,
|
||||
tdlib_destroy,
|
||||
tdlib_is_available,
|
||||
// Model commands (local LLM)
|
||||
model_is_available,
|
||||
model_get_status,
|
||||
model_ensure_loaded,
|
||||
model_generate,
|
||||
model_summarize,
|
||||
model_unload,
|
||||
]
|
||||
}
|
||||
#[cfg(not(desktop))]
|
||||
@@ -531,6 +561,13 @@ pub fn run() {
|
||||
tdlib_receive,
|
||||
tdlib_destroy,
|
||||
tdlib_is_available,
|
||||
// Model commands (local LLM)
|
||||
model_is_available,
|
||||
model_get_status,
|
||||
model_ensure_loaded,
|
||||
model_generate,
|
||||
model_summarize,
|
||||
model_unload,
|
||||
]
|
||||
}
|
||||
})
|
||||
|
||||
@@ -135,6 +135,7 @@ impl SocketManager {
|
||||
.auth(json!({"token": token}))
|
||||
.reconnect(true)
|
||||
.max_reconnect_attempts(0) // unlimited
|
||||
.transport_type(rust_socketio::TransportType::WebsocketUpgrade)
|
||||
// --- Connection established ---
|
||||
.on("connect", move |_payload, _client: Client| {
|
||||
let shared = Arc::clone(&s_connect);
|
||||
@@ -288,6 +289,7 @@ impl SocketManager {
|
||||
.auth(json!({"token": token}))
|
||||
.reconnect(true)
|
||||
.max_reconnect_attempts(0) // unlimited
|
||||
.transport_type(rust_socketio::TransportType::WebsocketUpgrade)
|
||||
// --- Connection established ---
|
||||
.on("connect", move |_payload, _client: Client| {
|
||||
let shared = Arc::clone(&s_connect);
|
||||
|
||||
@@ -0,0 +1,464 @@
|
||||
//! LlamaManager — singleton manager for local LLM inference.
|
||||
//!
|
||||
//! Provides:
|
||||
//! - Lazy model loading on first use
|
||||
//! - Automatic model download if not present
|
||||
//! - Thread-safe inference with dedicated thread pool
|
||||
//! - Generate and summarize API for skills
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::Arc;
|
||||
|
||||
use llama_cpp_2::context::params::LlamaContextParams;
|
||||
use llama_cpp_2::llama_backend::LlamaBackend;
|
||||
use llama_cpp_2::llama_batch::LlamaBatch;
|
||||
use llama_cpp_2::model::params::LlamaModelParams;
|
||||
use llama_cpp_2::model::LlamaModel;
|
||||
use llama_cpp_2::sampling::LlamaSampler;
|
||||
use llama_cpp_2::token::data_array::LlamaTokenDataArray;
|
||||
use once_cell::sync::Lazy;
|
||||
use parking_lot::RwLock;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Global LLama manager instance.
|
||||
pub static LLAMA_MANAGER: Lazy<LlamaManager> = Lazy::new(LlamaManager::new);
|
||||
|
||||
/// Model file name (Gemma 3n E2B Q4_K_M quantization)
|
||||
const MODEL_FILENAME: &str = "gemma-3n-E2B-it-Q4_K_M.gguf";
|
||||
|
||||
/// HuggingFace model URL for download
|
||||
const MODEL_URL: &str = "https://huggingface.co/bartowski/google_gemma-3n-E2B-it-GGUF/resolve/main/google_gemma-3n-E2B-it-Q4_K_M.gguf";
|
||||
|
||||
/// Expected SHA256 hash for model verification (first 16 chars for quick check)
|
||||
const MODEL_SHA256_PREFIX: &str = ""; // Will be verified on first download
|
||||
|
||||
/// Status of the local model.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct ModelStatus {
|
||||
/// Whether the model API is available on this platform.
|
||||
pub available: bool,
|
||||
/// Whether the model is currently loaded in memory.
|
||||
pub loaded: bool,
|
||||
/// Whether the model is currently being loaded or downloaded.
|
||||
pub loading: bool,
|
||||
/// Download progress (0.0 to 1.0) if downloading.
|
||||
pub download_progress: Option<f32>,
|
||||
/// Error message if loading failed.
|
||||
pub error: Option<String>,
|
||||
/// Model file path if known.
|
||||
pub model_path: Option<String>,
|
||||
}
|
||||
|
||||
impl Default for ModelStatus {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
available: true,
|
||||
loaded: false,
|
||||
loading: false,
|
||||
download_progress: None,
|
||||
error: None,
|
||||
model_path: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Configuration for text generation.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub struct GenerateConfig {
|
||||
/// Maximum tokens to generate (default: 2048).
|
||||
#[serde(default = "default_max_tokens")]
|
||||
pub max_tokens: u32,
|
||||
/// Sampling temperature (default: 0.7).
|
||||
#[serde(default = "default_temperature")]
|
||||
pub temperature: f32,
|
||||
/// Top-p sampling (default: 0.9).
|
||||
#[serde(default = "default_top_p")]
|
||||
pub top_p: f32,
|
||||
}
|
||||
|
||||
fn default_max_tokens() -> u32 {
|
||||
2048
|
||||
}
|
||||
fn default_temperature() -> f32 {
|
||||
0.7
|
||||
}
|
||||
fn default_top_p() -> f32 {
|
||||
0.9
|
||||
}
|
||||
|
||||
/// Internal state for the loaded model.
|
||||
struct LoadedModel {
|
||||
backend: LlamaBackend,
|
||||
model: LlamaModel,
|
||||
}
|
||||
|
||||
// Safety: LlamaBackend and LlamaModel are thread-safe through their C API
|
||||
unsafe impl Send for LoadedModel {}
|
||||
unsafe impl Sync for LoadedModel {}
|
||||
|
||||
/// LLama Manager for local model inference.
|
||||
pub struct LlamaManager {
|
||||
/// Directory for model files.
|
||||
data_dir: RwLock<PathBuf>,
|
||||
/// Loaded model (lazy-loaded on first use).
|
||||
model: RwLock<Option<Arc<LoadedModel>>>,
|
||||
/// Current status.
|
||||
status: RwLock<ModelStatus>,
|
||||
/// Lock to prevent concurrent loading.
|
||||
loading: AtomicBool,
|
||||
}
|
||||
|
||||
impl LlamaManager {
|
||||
/// Create a new LlamaManager (model not loaded yet).
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
data_dir: RwLock::new(PathBuf::new()),
|
||||
model: RwLock::new(None),
|
||||
status: RwLock::new(ModelStatus::default()),
|
||||
loading: AtomicBool::new(false),
|
||||
}
|
||||
}
|
||||
|
||||
/// Set the data directory for model storage.
|
||||
pub fn set_data_dir(&self, dir: PathBuf) {
|
||||
log::info!("[llama] Setting data dir: {:?}", dir);
|
||||
*self.data_dir.write() = dir.clone();
|
||||
|
||||
// Update status with model path
|
||||
let model_path = dir.join(MODEL_FILENAME);
|
||||
self.status.write().model_path = Some(model_path.to_string_lossy().to_string());
|
||||
}
|
||||
|
||||
/// Get the current model status.
|
||||
pub fn get_status(&self) -> ModelStatus {
|
||||
self.status.read().clone()
|
||||
}
|
||||
|
||||
/// Get the model file path.
|
||||
fn model_path(&self) -> PathBuf {
|
||||
self.data_dir.read().join(MODEL_FILENAME)
|
||||
}
|
||||
|
||||
/// Check if the model file exists.
|
||||
fn model_exists(&self) -> bool {
|
||||
self.model_path().exists()
|
||||
}
|
||||
|
||||
/// Download the model from HuggingFace.
|
||||
async fn download_model(&self) -> Result<(), String> {
|
||||
let model_path = self.model_path();
|
||||
|
||||
// Ensure parent directory exists
|
||||
if let Some(parent) = model_path.parent() {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| format!("Failed to create model directory: {}", e))?;
|
||||
}
|
||||
|
||||
log::info!("[llama] Downloading model from {}", MODEL_URL);
|
||||
self.status.write().download_progress = Some(0.0);
|
||||
|
||||
// Use reqwest for download
|
||||
let client = reqwest::Client::new();
|
||||
let response = client
|
||||
.get(MODEL_URL)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| format!("Failed to start download: {}", e))?;
|
||||
|
||||
if !response.status().is_success() {
|
||||
return Err(format!("Download failed with status: {}", response.status()));
|
||||
}
|
||||
|
||||
let total_size = response.content_length().unwrap_or(0);
|
||||
let mut downloaded: u64 = 0;
|
||||
|
||||
// Create temp file for download
|
||||
let temp_path = model_path.with_extension("download");
|
||||
let mut file = std::fs::File::create(&temp_path)
|
||||
.map_err(|e| format!("Failed to create temp file: {}", e))?;
|
||||
|
||||
// Stream the download
|
||||
use std::io::Write;
|
||||
let mut stream = response.bytes_stream();
|
||||
use futures::StreamExt;
|
||||
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let chunk = chunk.map_err(|e| format!("Download error: {}", e))?;
|
||||
file.write_all(&chunk)
|
||||
.map_err(|e| format!("Failed to write chunk: {}", e))?;
|
||||
|
||||
downloaded += chunk.len() as u64;
|
||||
|
||||
if total_size > 0 {
|
||||
let progress = downloaded as f32 / total_size as f32;
|
||||
self.status.write().download_progress = Some(progress);
|
||||
|
||||
// Log progress every 10%
|
||||
if (progress * 10.0) as u32 > ((downloaded - chunk.len() as u64) as f32 / total_size as f32 * 10.0) as u32 {
|
||||
log::info!("[llama] Download progress: {:.1}%", progress * 100.0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Flush and close file
|
||||
file.flush()
|
||||
.map_err(|e| format!("Failed to flush file: {}", e))?;
|
||||
drop(file);
|
||||
|
||||
// Rename temp file to final path
|
||||
std::fs::rename(&temp_path, &model_path)
|
||||
.map_err(|e| format!("Failed to rename temp file: {}", e))?;
|
||||
|
||||
log::info!("[llama] Model downloaded successfully to {:?}", model_path);
|
||||
self.status.write().download_progress = None;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Ensure the model is loaded into memory.
|
||||
pub async fn ensure_loaded(&self) -> Result<(), String> {
|
||||
// Already loaded?
|
||||
if self.model.read().is_some() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Prevent concurrent loading
|
||||
if self.loading.swap(true, Ordering::SeqCst) {
|
||||
// Another thread is loading, wait for it
|
||||
while self.loading.load(Ordering::SeqCst) {
|
||||
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
|
||||
}
|
||||
// Check if loading succeeded
|
||||
if self.model.read().is_some() {
|
||||
return Ok(());
|
||||
}
|
||||
return Err("Model loading failed".to_string());
|
||||
}
|
||||
|
||||
// Update status
|
||||
{
|
||||
let mut status = self.status.write();
|
||||
status.loading = true;
|
||||
status.error = None;
|
||||
}
|
||||
|
||||
let result = self.load_model_internal().await;
|
||||
|
||||
// Update status based on result
|
||||
{
|
||||
let mut status = self.status.write();
|
||||
status.loading = false;
|
||||
match &result {
|
||||
Ok(_) => {
|
||||
status.loaded = true;
|
||||
status.error = None;
|
||||
}
|
||||
Err(e) => {
|
||||
status.loaded = false;
|
||||
status.error = Some(e.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
self.loading.store(false, Ordering::SeqCst);
|
||||
result
|
||||
}
|
||||
|
||||
/// Internal model loading logic.
|
||||
async fn load_model_internal(&self) -> Result<(), String> {
|
||||
// Check if model exists, download if not
|
||||
if !self.model_exists() {
|
||||
log::info!("[llama] Model not found, downloading...");
|
||||
self.download_model().await?;
|
||||
}
|
||||
|
||||
let model_path = self.model_path();
|
||||
log::info!("[llama] Loading model from {:?}", model_path);
|
||||
|
||||
// Load model in blocking thread
|
||||
let path = model_path.clone();
|
||||
let loaded = tokio::task::spawn_blocking(move || -> Result<LoadedModel, String> {
|
||||
// Initialize llama backend
|
||||
let backend = LlamaBackend::init()
|
||||
.map_err(|e| format!("Failed to initialize llama backend: {}", e))?;
|
||||
|
||||
// Set up model parameters
|
||||
let model_params = LlamaModelParams::default();
|
||||
|
||||
// Load the model
|
||||
let model = LlamaModel::load_from_file(&backend, &path, &model_params)
|
||||
.map_err(|e| format!("Failed to load model: {}", e))?;
|
||||
|
||||
Ok(LoadedModel { backend, model })
|
||||
})
|
||||
.await
|
||||
.map_err(|e| format!("Task join error: {}", e))??;
|
||||
|
||||
// Store the loaded model
|
||||
*self.model.write() = Some(Arc::new(loaded));
|
||||
log::info!("[llama] Model loaded successfully");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Generate text from a prompt.
|
||||
pub async fn generate(&self, prompt: &str, config: GenerateConfig) -> Result<String, String> {
|
||||
// Ensure model is loaded
|
||||
self.ensure_loaded().await?;
|
||||
|
||||
let model_arc = self
|
||||
.model
|
||||
.read()
|
||||
.clone()
|
||||
.ok_or_else(|| "Model not loaded".to_string())?;
|
||||
|
||||
let prompt = prompt.to_string();
|
||||
let max_tokens = config.max_tokens;
|
||||
let temperature = config.temperature;
|
||||
|
||||
// Run inference in blocking thread
|
||||
tokio::task::spawn_blocking(move || {
|
||||
Self::generate_sync(&model_arc, &prompt, max_tokens, temperature)
|
||||
})
|
||||
.await
|
||||
.map_err(|e| format!("Task join error: {}", e))?
|
||||
}
|
||||
|
||||
/// Synchronous text generation (runs on blocking thread).
|
||||
fn generate_sync(
|
||||
loaded: &LoadedModel,
|
||||
prompt: &str,
|
||||
max_tokens: u32,
|
||||
temperature: f32,
|
||||
) -> Result<String, String> {
|
||||
// Create context for inference
|
||||
let ctx_params = LlamaContextParams::default()
|
||||
.with_n_ctx(std::num::NonZeroU32::new(8192));
|
||||
|
||||
let mut ctx = loaded
|
||||
.model
|
||||
.new_context(&loaded.backend, ctx_params)
|
||||
.map_err(|e| format!("Failed to create context: {}", e))?;
|
||||
|
||||
// Tokenize the prompt
|
||||
let tokens = loaded
|
||||
.model
|
||||
.str_to_token(prompt, llama_cpp_2::model::AddBos::Always)
|
||||
.map_err(|e| format!("Failed to tokenize: {}", e))?;
|
||||
|
||||
if tokens.is_empty() {
|
||||
return Err("Empty prompt".to_string());
|
||||
}
|
||||
|
||||
// Create batch with initial tokens
|
||||
let mut batch = LlamaBatch::new(8192, 1);
|
||||
|
||||
for (i, token) in tokens.iter().enumerate() {
|
||||
let is_last = i == tokens.len() - 1;
|
||||
batch
|
||||
.add(*token, i as i32, &[0], is_last)
|
||||
.map_err(|e| format!("Failed to add token to batch: {}", e))?;
|
||||
}
|
||||
|
||||
// Decode initial tokens
|
||||
ctx.decode(&mut batch)
|
||||
.map_err(|e| format!("Failed to decode batch: {}", e))?;
|
||||
|
||||
// Generate tokens
|
||||
let mut output_tokens = Vec::new();
|
||||
let mut n_cur = tokens.len();
|
||||
|
||||
// Create sampler chain for temperature sampling
|
||||
let seed = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_millis() as u32)
|
||||
.unwrap_or(42);
|
||||
|
||||
for _ in 0..max_tokens {
|
||||
// Get logits for the last token
|
||||
let logits = ctx.candidates_ith(batch.n_tokens() - 1);
|
||||
|
||||
// Create token data array for sampling
|
||||
let mut candidates = LlamaTokenDataArray::from_iter(logits, false);
|
||||
|
||||
// Apply temperature sampler
|
||||
let mut temp_sampler = LlamaSampler::temp(temperature);
|
||||
candidates.apply_sampler(&mut temp_sampler);
|
||||
|
||||
// Sample token with random seed
|
||||
let new_token = candidates.sample_token(seed);
|
||||
|
||||
// Check for end of generation
|
||||
if loaded.model.is_eog_token(new_token) {
|
||||
break;
|
||||
}
|
||||
|
||||
output_tokens.push(new_token);
|
||||
|
||||
// Prepare next batch
|
||||
batch.clear();
|
||||
batch
|
||||
.add(new_token, n_cur as i32, &[0], true)
|
||||
.map_err(|e| format!("Failed to add token: {}", e))?;
|
||||
|
||||
n_cur += 1;
|
||||
|
||||
// Decode
|
||||
ctx.decode(&mut batch)
|
||||
.map_err(|e| format!("Failed to decode: {}", e))?;
|
||||
}
|
||||
|
||||
// Convert tokens to string using token_to_piece
|
||||
let mut decoder = encoding_rs::UTF_8.new_decoder();
|
||||
let mut output = String::new();
|
||||
|
||||
for token in &output_tokens {
|
||||
match loaded.model.token_to_piece(*token, &mut decoder, false, None) {
|
||||
Ok(piece) => output.push_str(&piece),
|
||||
Err(e) => {
|
||||
log::warn!("[llama] Failed to decode token: {}", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
/// Summarize text using a built-in prompt.
|
||||
pub async fn summarize(&self, text: &str, max_tokens: u32) -> Result<String, String> {
|
||||
let prompt = format!(
|
||||
"<start_of_turn>user\nPlease provide a concise summary of the following text:\n\n{}\n<end_of_turn>\n<start_of_turn>model\n",
|
||||
text
|
||||
);
|
||||
|
||||
self.generate(
|
||||
&prompt,
|
||||
GenerateConfig {
|
||||
max_tokens,
|
||||
temperature: 0.5, // Lower temperature for more focused summarization
|
||||
top_p: 0.9,
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Unload the model from memory.
|
||||
pub fn unload(&self) {
|
||||
log::info!("[llama] Unloading model");
|
||||
*self.model.write() = None;
|
||||
self.status.write().loaded = false;
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for LlamaManager {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure LlamaManager is Send + Sync
|
||||
unsafe impl Send for LlamaManager {}
|
||||
unsafe impl Sync for LlamaManager {}
|
||||
@@ -0,0 +1,11 @@
|
||||
//! Local LLM Service using llama-cpp-2
|
||||
//!
|
||||
//! Provides local AI model inference for skills using llama.cpp.
|
||||
//! Desktop only - not available on Android/iOS.
|
||||
|
||||
mod manager;
|
||||
|
||||
pub use manager::GenerateConfig;
|
||||
pub use manager::LlamaManager;
|
||||
pub use manager::ModelStatus;
|
||||
pub use manager::LLAMA_MANAGER;
|
||||
@@ -5,3 +5,7 @@ pub mod tdlib_v8;
|
||||
|
||||
#[cfg(desktop)]
|
||||
pub mod notification_service;
|
||||
|
||||
// Local LLM inference - available on desktop and Android (not iOS)
|
||||
#[cfg(not(target_os = "ios"))]
|
||||
pub mod llama;
|
||||
|
||||
+74
@@ -912,4 +912,78 @@ globalThis.tdlib = {
|
||||
},
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
// Model Bridge API (local LLM inference)
|
||||
// ============================================================================
|
||||
|
||||
globalThis.__model = {
|
||||
isAvailable: function () {
|
||||
try {
|
||||
return typeof Deno?.core?.ops?.op_model_is_available === 'function'
|
||||
? Deno.core.ops.op_model_is_available()
|
||||
: false;
|
||||
} catch (e) {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
getStatus: function () {
|
||||
return Deno.core.ops.op_model_get_status();
|
||||
},
|
||||
generate: async function (prompt, configJson) {
|
||||
return await Deno.core.ops.op_model_generate(prompt, configJson);
|
||||
},
|
||||
summarize: async function (text, maxTokens) {
|
||||
return await Deno.core.ops.op_model_summarize(text, maxTokens);
|
||||
},
|
||||
};
|
||||
|
||||
globalThis.model = {
|
||||
/**
|
||||
* Check if local model is available (desktop only).
|
||||
* @returns {boolean}
|
||||
*/
|
||||
isAvailable: function () {
|
||||
return __model.isAvailable();
|
||||
},
|
||||
|
||||
/**
|
||||
* Get model status.
|
||||
* @returns {{ available: boolean, loaded: boolean, loading: boolean, downloadProgress?: number, error?: string, modelPath?: string }}
|
||||
*/
|
||||
getStatus: function () {
|
||||
return __model.getStatus();
|
||||
},
|
||||
|
||||
/**
|
||||
* Generate text from a prompt.
|
||||
* @param {string} prompt - Input prompt
|
||||
* @param {object} [options] - Generation options
|
||||
* @param {number} [options.maxTokens=2048] - Max output tokens
|
||||
* @param {number} [options.temperature=0.7] - Sampling temperature
|
||||
* @param {number} [options.topP=0.9] - Top-p sampling
|
||||
* @returns {Promise<string>}
|
||||
*/
|
||||
generate: async function (prompt, options) {
|
||||
var config = {
|
||||
max_tokens: (options && options.maxTokens) || 2048,
|
||||
temperature: (options && options.temperature) || 0.7,
|
||||
top_p: (options && options.topP) || 0.9,
|
||||
};
|
||||
return await __model.generate(prompt, config);
|
||||
},
|
||||
|
||||
/**
|
||||
* Summarize text locally.
|
||||
* @param {string} text - Text to summarize
|
||||
* @param {object} [options] - Options
|
||||
* @param {number} [options.maxTokens=500] - Target summary length
|
||||
* @returns {Promise<string>}
|
||||
*/
|
||||
summarize: async function (text, options) {
|
||||
var maxTokens = (options && options.maxTokens) || 500;
|
||||
return await __model.summarize(text, maxTokens);
|
||||
},
|
||||
};
|
||||
|
||||
console.log('[bootstrap] Model API initialized');
|
||||
console.log('[bootstrap] V8 browser APIs initialized');
|
||||
|
||||
@@ -79,6 +79,11 @@ extension!(
|
||||
op_tdlib_receive,
|
||||
op_tdlib_destroy,
|
||||
op_tdlib_is_available,
|
||||
// Model ops (local LLM inference)
|
||||
op_model_is_available,
|
||||
op_model_get_status,
|
||||
op_model_generate,
|
||||
op_model_summarize,
|
||||
],
|
||||
state = |state| {
|
||||
// State will be initialized when runtime is created
|
||||
@@ -1182,3 +1187,50 @@ async fn op_tdlib_destroy(
|
||||
.await
|
||||
.map_err(|e| deno_core::error::generic_error(e))
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Model Ops (local LLM inference)
|
||||
// ============================================================================
|
||||
|
||||
/// Check if local model API is available (desktop only).
|
||||
#[op2(fast)]
|
||||
fn op_model_is_available() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
/// Get model status (loading, ready, error).
|
||||
#[op2]
|
||||
#[serde]
|
||||
fn op_model_get_status() -> serde_json::Value {
|
||||
let status = crate::services::llama::LLAMA_MANAGER.get_status();
|
||||
serde_json::to_value(status).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Generate text from prompt (async, blocking inference on thread pool).
|
||||
#[op2(async)]
|
||||
#[string]
|
||||
async fn op_model_generate(
|
||||
#[string] prompt: String,
|
||||
#[serde] config: serde_json::Value,
|
||||
) -> Result<String, deno_core::error::AnyError> {
|
||||
let cfg: crate::services::llama::GenerateConfig =
|
||||
serde_json::from_value(config).unwrap_or_default();
|
||||
|
||||
crate::services::llama::LLAMA_MANAGER
|
||||
.generate(&prompt, cfg)
|
||||
.await
|
||||
.map_err(|e| deno_core::error::generic_error(e))
|
||||
}
|
||||
|
||||
/// Summarize text (async).
|
||||
#[op2(async)]
|
||||
#[string]
|
||||
async fn op_model_summarize(
|
||||
#[string] text: String,
|
||||
max_tokens: u32,
|
||||
) -> Result<String, deno_core::error::AnyError> {
|
||||
crate::services::llama::LLAMA_MANAGER
|
||||
.summarize(&text, max_tokens)
|
||||
.await
|
||||
.map_err(|e| deno_core::error::generic_error(e))
|
||||
}
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
import { useModelStatus } from '../hooks/useModelStatus';
|
||||
|
||||
interface ModelDownloadProgressProps {
|
||||
className?: string;
|
||||
showWhenLoaded?: boolean;
|
||||
}
|
||||
|
||||
const ModelDownloadProgress = ({
|
||||
className = '',
|
||||
showWhenLoaded = false,
|
||||
}: ModelDownloadProgressProps) => {
|
||||
const { isAvailable, isLoaded, isLoading, downloadProgress, error, ensureLoaded } =
|
||||
useModelStatus();
|
||||
|
||||
// Don't render if not available on this platform
|
||||
if (!isAvailable) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Don't render if loaded and showWhenLoaded is false
|
||||
if (isLoaded && !showWhenLoaded && !isLoading) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Format download progress percentage
|
||||
const progressPercent = downloadProgress !== null ? Math.round(downloadProgress * 100) : 0;
|
||||
|
||||
// Determine status display
|
||||
const getStatusDisplay = () => {
|
||||
if (error) {
|
||||
return {
|
||||
icon: (
|
||||
<svg
|
||||
className="w-5 h-5 text-coral-500"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
viewBox="0 0 24 24">
|
||||
<path
|
||||
strokeLinecap="round"
|
||||
strokeLinejoin="round"
|
||||
strokeWidth={2}
|
||||
d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z"
|
||||
/>
|
||||
</svg>
|
||||
),
|
||||
title: 'Model Error',
|
||||
description: error,
|
||||
color: 'coral',
|
||||
};
|
||||
}
|
||||
|
||||
if (isLoading && downloadProgress !== null) {
|
||||
return {
|
||||
icon: (
|
||||
<svg
|
||||
className="w-5 h-5 text-primary-500 animate-pulse"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
viewBox="0 0 24 24">
|
||||
<path
|
||||
strokeLinecap="round"
|
||||
strokeLinejoin="round"
|
||||
strokeWidth={2}
|
||||
d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4"
|
||||
/>
|
||||
</svg>
|
||||
),
|
||||
title: 'Downloading Local AI Model',
|
||||
description: `${progressPercent}% complete (~1.2 GB).`,
|
||||
color: 'primary',
|
||||
};
|
||||
}
|
||||
|
||||
if (isLoading) {
|
||||
return {
|
||||
icon: (
|
||||
<svg
|
||||
className="w-5 h-5 text-primary-500 animate-spin"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
viewBox="0 0 24 24">
|
||||
<path
|
||||
strokeLinecap="round"
|
||||
strokeLinejoin="round"
|
||||
strokeWidth={2}
|
||||
d="M4 4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003 0 01-15.357-2m15.357 2H15"
|
||||
/>
|
||||
</svg>
|
||||
),
|
||||
title: 'Loading Local AI Model',
|
||||
description: 'Initializing local inference engine...',
|
||||
color: 'primary',
|
||||
};
|
||||
}
|
||||
|
||||
if (isLoaded) {
|
||||
return {
|
||||
icon: (
|
||||
<svg
|
||||
className="w-5 h-5 text-sage-500"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
viewBox="0 0 24 24">
|
||||
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M5 13l4 4L19 7" />
|
||||
</svg>
|
||||
),
|
||||
title: 'AI Model Ready',
|
||||
description: 'Local inference available',
|
||||
color: 'sage',
|
||||
};
|
||||
}
|
||||
|
||||
// Not loaded, show download prompt
|
||||
return {
|
||||
icon: (
|
||||
<svg
|
||||
className="w-5 h-5 text-stone-400"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
viewBox="0 0 24 24">
|
||||
<path
|
||||
strokeLinecap="round"
|
||||
strokeLinejoin="round"
|
||||
strokeWidth={2}
|
||||
d="M9.75 17L9 20l-1 1h8l-1-1-.75-3M3 13h18M5 17h14a2 2 0 002-2V5a2 2 0 00-2-2H5a2 2 0 00-2 2v10a2 2 0 002 2z"
|
||||
/>
|
||||
</svg>
|
||||
),
|
||||
title: 'Local AI Model',
|
||||
description: 'Download for offline summarization',
|
||||
color: 'stone',
|
||||
};
|
||||
};
|
||||
|
||||
const statusDisplay = getStatusDisplay();
|
||||
|
||||
return (
|
||||
<div className={`glass rounded-2xl p-3 shadow-large animate-fade-up ${className}`}>
|
||||
<div className="flex items-center gap-3">
|
||||
{/* Icon */}
|
||||
<div className="flex-shrink-0">{statusDisplay.icon}</div>
|
||||
|
||||
{/* Content */}
|
||||
<div className="flex-1 min-w-0">
|
||||
<div className="flex items-center justify-between mb-1">
|
||||
<span className="font-medium text-sm">{statusDisplay.title}</span>
|
||||
{!isLoaded && !isLoading && !error && (
|
||||
<button
|
||||
onClick={ensureLoaded}
|
||||
className="text-xs text-primary-500 hover:text-primary-400 transition-colors">
|
||||
Download
|
||||
</button>
|
||||
)}
|
||||
{error && (
|
||||
<button
|
||||
onClick={ensureLoaded}
|
||||
className="text-xs text-coral-500 hover:text-coral-400 transition-colors">
|
||||
Retry
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
<p className="text-xs opacity-60 truncate">{statusDisplay.description}</p>
|
||||
|
||||
{/* Progress bar */}
|
||||
{isLoading && downloadProgress !== null && (
|
||||
<div className="mt-2 w-full bg-stone-700/50 rounded-full h-1.5 overflow-hidden">
|
||||
<div
|
||||
className="bg-primary-500 h-full rounded-full transition-all duration-300 ease-out"
|
||||
style={{ width: `${progressPercent}%` }}
|
||||
/>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
export default ModelDownloadProgress;
|
||||
@@ -0,0 +1,124 @@
|
||||
import { invoke } from '@tauri-apps/api/core';
|
||||
import { useCallback, useEffect, useState } from 'react';
|
||||
|
||||
/**
|
||||
* Model status from Rust backend
|
||||
*/
|
||||
export interface ModelStatus {
|
||||
available: boolean;
|
||||
loaded: boolean;
|
||||
loading: boolean;
|
||||
downloadProgress: number | null;
|
||||
error: string | null;
|
||||
modelPath: string | null;
|
||||
}
|
||||
|
||||
const DEFAULT_STATUS: ModelStatus = {
|
||||
available: false,
|
||||
loaded: false,
|
||||
loading: false,
|
||||
downloadProgress: null,
|
||||
error: null,
|
||||
modelPath: null,
|
||||
};
|
||||
|
||||
/**
|
||||
* Hook to monitor and control local AI model status
|
||||
*/
|
||||
export const useModelStatus = (pollInterval = 1000) => {
|
||||
const [status, setStatus] = useState<ModelStatus>(DEFAULT_STATUS);
|
||||
const [isPolling, setIsPolling] = useState(false);
|
||||
|
||||
// Fetch current status from backend
|
||||
const fetchStatus = useCallback(async () => {
|
||||
try {
|
||||
const result = await invoke<ModelStatus>('model_get_status');
|
||||
setStatus(result);
|
||||
return result;
|
||||
} catch (error) {
|
||||
console.error('[useModelStatus] Failed to fetch status:', error);
|
||||
setStatus(prev => ({
|
||||
...prev,
|
||||
error: error instanceof Error ? error.message : 'Failed to fetch status',
|
||||
}));
|
||||
return null;
|
||||
}
|
||||
}, []);
|
||||
|
||||
// Check if model API is available
|
||||
const checkAvailability = useCallback(async () => {
|
||||
try {
|
||||
const available = await invoke<boolean>('model_is_available');
|
||||
setStatus(prev => ({ ...prev, available }));
|
||||
return available;
|
||||
} catch (error) {
|
||||
console.error('[useModelStatus] Failed to check availability:', error);
|
||||
return false;
|
||||
}
|
||||
}, []);
|
||||
|
||||
// Start loading/downloading the model
|
||||
const ensureLoaded = useCallback(async () => {
|
||||
try {
|
||||
setStatus(prev => ({ ...prev, loading: true, error: null }));
|
||||
setIsPolling(true);
|
||||
await invoke('model_ensure_loaded');
|
||||
await fetchStatus();
|
||||
setIsPolling(false);
|
||||
} catch (error) {
|
||||
console.error('[useModelStatus] Failed to load model:', error);
|
||||
setStatus(prev => ({
|
||||
...prev,
|
||||
loading: false,
|
||||
error: error instanceof Error ? error.message : 'Failed to load model',
|
||||
}));
|
||||
setIsPolling(false);
|
||||
}
|
||||
}, [fetchStatus]);
|
||||
|
||||
// Unload the model from memory
|
||||
const unload = useCallback(async () => {
|
||||
try {
|
||||
await invoke('model_unload');
|
||||
await fetchStatus();
|
||||
} catch (error) {
|
||||
console.error('[useModelStatus] Failed to unload model:', error);
|
||||
}
|
||||
}, [fetchStatus]);
|
||||
|
||||
// Initial check and polling setup
|
||||
useEffect(() => {
|
||||
// Initial fetch
|
||||
fetchStatus();
|
||||
|
||||
// Check availability
|
||||
checkAvailability();
|
||||
}, [fetchStatus, checkAvailability]);
|
||||
|
||||
// Polling when loading/downloading
|
||||
useEffect(() => {
|
||||
if (!isPolling && !status.loading) return;
|
||||
|
||||
const interval = setInterval(async () => {
|
||||
const newStatus = await fetchStatus();
|
||||
// Stop polling when loading is done
|
||||
if (newStatus && !newStatus.loading) {
|
||||
setIsPolling(false);
|
||||
}
|
||||
}, pollInterval);
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, [isPolling, status.loading, pollInterval, fetchStatus]);
|
||||
|
||||
return {
|
||||
status,
|
||||
isAvailable: status.available,
|
||||
isLoaded: status.loaded,
|
||||
isLoading: status.loading,
|
||||
downloadProgress: status.downloadProgress,
|
||||
error: status.error,
|
||||
ensureLoaded,
|
||||
unload,
|
||||
refresh: fetchStatus,
|
||||
};
|
||||
};
|
||||
@@ -1,6 +1,7 @@
|
||||
import { useNavigate } from 'react-router-dom';
|
||||
|
||||
import ConnectionIndicator from '../components/ConnectionIndicator';
|
||||
import ModelDownloadProgress from '../components/ModelDownloadProgress';
|
||||
import SkillsGrid from '../components/SkillsGrid';
|
||||
import { useUser } from '../hooks/useUser';
|
||||
import { TELEGRAM_BOT_USERNAME } from '../utils/config';
|
||||
@@ -137,6 +138,8 @@ const Home = () => {
|
||||
|
||||
{/* Skills Grid */}
|
||||
<SkillsGrid />
|
||||
|
||||
<ModelDownloadProgress className="mb-4" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user