mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-30 23:14:37 +00:00
* feat(tree-summarizer): implement hierarchical summarization engine and event handling - Introduced a new `tree_summarizer` module to manage hierarchical time-based summaries, organizing data into a tree structure (root → year → month → day → hour). - Added functionality to ingest raw content, summarize it into hour leaves, and propagate summaries upward through the tree. - Implemented event handling for summarization completion and tree rebuild events, enhancing observability and modularity. - Created RPC operations for ingesting content, triggering summarization, querying the tree, and retrieving tree status. - Added comprehensive tests to ensure the reliability of the summarization process and event handling. This update significantly enhances the summarization capabilities of the system, allowing for efficient data organization and retrieval. * feat(tree-summarizer): add CLI support for tree summarization commands - Introduced a new `tree-summarizer` command to the CLI, allowing users to ingest content, run summarization jobs, query the summary tree, check status, and rebuild the tree. - Updated the CLI help documentation to include the new command and its subcommands. - Added a new module `tree_summarizer_cli` to encapsulate the tree summarization functionality. This enhancement improves the usability of the summarization features, providing a streamlined interface for managing hierarchical summaries directly from the command line. * style: apply cargo fmt to tree_summarizer module * feat(tree-summarizer): implement TreeSummarizerEventSubscriber for observability logging - Added a new `TreeSummarizerEventSubscriber` to log events related to tree summarization, enhancing observability. - Updated the `start_channels` function to register the new subscriber. - Refactored the `run_summarization` function to group buffered entries by hour and publish events upon completion of summarization. - Improved documentation and added tests for the new subscriber functionality. This update aims to provide better insights into the summarization process and facilitate future cross-module workflows. * refactor(tree_summarizer): streamline buffer backup and function signature - Simplified the buffer backup process by consolidating the rename operation with context handling for better error reporting. - Cleaned up the function signature of `derive_node_ids_from_hour_id` for improved readability. These changes enhance code clarity and maintainability within the tree summarization engine. * feat(tree_summarizer): enhance tree summarization with metadata support - Updated the `tree_summarizer_ingest` function to accept an optional metadata parameter, allowing users to include additional context during content ingestion. - Refactored related functions to validate and handle metadata, improving the overall robustness of the summarization process. - Adjusted the buffer write functionality to store metadata alongside content, enhancing the data structure for future retrieval and processing. These changes aim to enrich the summarization capabilities and provide more context for ingested content. * refactor(tree_summarizer): improve error message formatting in node ID validation - Enhanced the formatting of error messages in the `validate_node_id` function for better readability and consistency. - Adjusted the string formatting to use multi-line syntax, improving clarity in error reporting. - Minor formatting changes in the `strip_buffer_frontmatter` function to enhance code readability. These changes aim to improve the maintainability and clarity of error handling within the tree summarization module. * refactor(tree_summarizer): enhance buffer management and summarization process - Replaced the buffer draining mechanism with a non-destructive read approach, allowing for safer data handling during summarization. - Introduced a new `buffer_delete` function to explicitly manage the deletion of buffer entries after successful processing. - Updated the `run_summarization` function to reflect these changes, ensuring that buffer entries are only deleted after durable writes are confirmed. - Improved the backup process for the buffer directory during tree rebuilds, ensuring it is preserved outside the tree structure. These modifications aim to improve data integrity and clarity in the summarization workflow. * refactor(tree_summarizer): improve markdown parsing and timestamp handling - Updated the `parse_node_markdown` function to trim trailing whitespace from the body after splitting frontmatter, enhancing data cleanliness. - Modified test cases to use specific timestamps instead of the current time, ensuring consistent and predictable test results. - Adjusted assertions in tests to reflect the new timestamp-based ordering of entries. These changes aim to improve the robustness of markdown parsing and the reliability of test outcomes in the tree summarization module.
376 lines
16 KiB
Rust
376 lines
16 KiB
Rust
use std::future::Future;
|
|
use std::pin::Pin;
|
|
use std::sync::OnceLock;
|
|
|
|
use serde_json::{Map, Value};
|
|
|
|
use crate::core::ControllerSchema;
|
|
|
|
pub type ControllerFuture = Pin<Box<dyn Future<Output = Result<Value, String>> + Send + 'static>>;
|
|
pub type ControllerHandler = fn(Map<String, Value>) -> ControllerFuture;
|
|
|
|
#[derive(Clone)]
|
|
pub struct RegisteredController {
|
|
pub schema: ControllerSchema,
|
|
pub handler: ControllerHandler,
|
|
}
|
|
|
|
impl RegisteredController {
|
|
pub fn rpc_method_name(&self) -> String {
|
|
rpc_method_name(&self.schema)
|
|
}
|
|
}
|
|
|
|
static REGISTRY: OnceLock<Vec<RegisteredController>> = OnceLock::new();
|
|
|
|
fn registry() -> &'static [RegisteredController] {
|
|
REGISTRY
|
|
.get_or_init(|| {
|
|
let registered = build_registered_controllers();
|
|
let declared = build_declared_controller_schemas();
|
|
validate_registry(®istered, &declared).unwrap_or_else(|err| {
|
|
panic!("invalid controller registry: {err}");
|
|
});
|
|
registered
|
|
})
|
|
.as_slice()
|
|
}
|
|
|
|
fn build_registered_controllers() -> Vec<RegisteredController> {
|
|
let mut controllers = Vec::new();
|
|
controllers.extend(crate::openhuman::about_app::all_about_app_registered_controllers());
|
|
controllers.extend(crate::openhuman::app_state::all_app_state_registered_controllers());
|
|
controllers.extend(crate::openhuman::cron::all_cron_registered_controllers());
|
|
controllers.extend(crate::openhuman::agent::all_agent_registered_controllers());
|
|
controllers.extend(crate::openhuman::health::all_health_registered_controllers());
|
|
controllers.extend(crate::openhuman::doctor::all_doctor_registered_controllers());
|
|
controllers.extend(crate::openhuman::encryption::all_encryption_registered_controllers());
|
|
controllers.extend(crate::openhuman::heartbeat::all_heartbeat_registered_controllers());
|
|
controllers.extend(crate::openhuman::cost::all_cost_registered_controllers());
|
|
controllers.extend(crate::openhuman::autocomplete::all_autocomplete_registered_controllers());
|
|
controllers.extend(
|
|
crate::openhuman::channels::providers::web::all_web_channel_registered_controllers(),
|
|
);
|
|
controllers
|
|
.extend(crate::openhuman::channels::controllers::all_channels_registered_controllers());
|
|
controllers.extend(crate::openhuman::config::all_config_registered_controllers());
|
|
controllers.extend(crate::openhuman::credentials::all_credentials_registered_controllers());
|
|
controllers.extend(crate::openhuman::service::all_service_registered_controllers());
|
|
controllers.extend(crate::openhuman::migration::all_migration_registered_controllers());
|
|
controllers.extend(crate::openhuman::local_ai::all_local_ai_registered_controllers());
|
|
controllers.extend(
|
|
crate::openhuman::screen_intelligence::all_screen_intelligence_registered_controllers(),
|
|
);
|
|
controllers.extend(crate::openhuman::skills::all_skills_registered_controllers());
|
|
controllers.extend(crate::openhuman::socket::all_socket_registered_controllers());
|
|
controllers.extend(crate::openhuman::workspace::all_workspace_registered_controllers());
|
|
controllers.extend(crate::openhuman::tools::all_tools_registered_controllers());
|
|
controllers.extend(crate::openhuman::memory::all_memory_registered_controllers());
|
|
controllers.extend(crate::openhuman::billing::all_billing_registered_controllers());
|
|
controllers.extend(crate::openhuman::team::all_team_registered_controllers());
|
|
controllers.extend(crate::openhuman::text_input::all_text_input_registered_controllers());
|
|
controllers.extend(crate::openhuman::voice::all_voice_registered_controllers());
|
|
controllers.extend(crate::openhuman::subconscious::all_subconscious_registered_controllers());
|
|
controllers.extend(crate::openhuman::webhooks::all_webhooks_registered_controllers());
|
|
controllers.extend(crate::openhuman::update::all_update_registered_controllers());
|
|
controllers
|
|
.extend(crate::openhuman::tree_summarizer::all_tree_summarizer_registered_controllers());
|
|
controllers
|
|
}
|
|
|
|
fn build_declared_controller_schemas() -> Vec<ControllerSchema> {
|
|
let mut schemas = Vec::new();
|
|
schemas.extend(crate::openhuman::about_app::all_about_app_controller_schemas());
|
|
schemas.extend(crate::openhuman::app_state::all_app_state_controller_schemas());
|
|
schemas.extend(crate::openhuman::cron::all_cron_controller_schemas());
|
|
schemas.extend(crate::openhuman::agent::all_agent_controller_schemas());
|
|
schemas.extend(crate::openhuman::health::all_health_controller_schemas());
|
|
schemas.extend(crate::openhuman::doctor::all_doctor_controller_schemas());
|
|
schemas.extend(crate::openhuman::encryption::all_encryption_controller_schemas());
|
|
schemas.extend(crate::openhuman::heartbeat::all_heartbeat_controller_schemas());
|
|
schemas.extend(crate::openhuman::cost::all_cost_controller_schemas());
|
|
schemas.extend(crate::openhuman::autocomplete::all_autocomplete_controller_schemas());
|
|
schemas
|
|
.extend(crate::openhuman::channels::providers::web::all_web_channel_controller_schemas());
|
|
schemas.extend(crate::openhuman::channels::controllers::all_channels_controller_schemas());
|
|
schemas.extend(crate::openhuman::config::all_config_controller_schemas());
|
|
schemas.extend(crate::openhuman::credentials::all_credentials_controller_schemas());
|
|
schemas.extend(crate::openhuman::service::all_service_controller_schemas());
|
|
schemas.extend(crate::openhuman::migration::all_migration_controller_schemas());
|
|
schemas.extend(crate::openhuman::local_ai::all_local_ai_controller_schemas());
|
|
schemas.extend(
|
|
crate::openhuman::screen_intelligence::all_screen_intelligence_controller_schemas(),
|
|
);
|
|
schemas.extend(crate::openhuman::skills::all_skills_controller_schemas());
|
|
schemas.extend(crate::openhuman::socket::all_socket_controller_schemas());
|
|
schemas.extend(crate::openhuman::workspace::all_workspace_controller_schemas());
|
|
schemas.extend(crate::openhuman::tools::all_tools_controller_schemas());
|
|
schemas.extend(crate::openhuman::memory::all_memory_controller_schemas());
|
|
schemas.extend(crate::openhuman::billing::all_billing_controller_schemas());
|
|
schemas.extend(crate::openhuman::team::all_team_controller_schemas());
|
|
schemas.extend(crate::openhuman::text_input::all_text_input_controller_schemas());
|
|
schemas.extend(crate::openhuman::voice::all_voice_controller_schemas());
|
|
schemas.extend(crate::openhuman::subconscious::all_subconscious_controller_schemas());
|
|
schemas.extend(crate::openhuman::webhooks::all_webhooks_controller_schemas());
|
|
schemas.extend(crate::openhuman::update::all_update_controller_schemas());
|
|
schemas.extend(crate::openhuman::tree_summarizer::all_tree_summarizer_controller_schemas());
|
|
schemas
|
|
}
|
|
|
|
pub fn all_registered_controllers() -> Vec<RegisteredController> {
|
|
registry().to_vec()
|
|
}
|
|
|
|
pub fn all_controller_schemas() -> Vec<ControllerSchema> {
|
|
let _ = registry();
|
|
build_declared_controller_schemas()
|
|
}
|
|
|
|
pub fn rpc_method_name(schema: &ControllerSchema) -> String {
|
|
format!("openhuman.{}_{}", schema.namespace, schema.function)
|
|
}
|
|
|
|
pub fn namespace_description(namespace: &str) -> Option<&'static str> {
|
|
match namespace {
|
|
"about_app" => Some("Catalog the app's user-facing capabilities and where to find them."),
|
|
"app_state" => Some("Expose core-owned app shell state for frontend polling."),
|
|
"auth" => Some("Manage app session and provider credentials."),
|
|
"autocomplete" => Some("Inline autocomplete engine controls and style settings."),
|
|
"channels" => Some("Channel definitions, connections, and lifecycle management."),
|
|
"config" => Some("Read and update persisted runtime configuration."),
|
|
"cron" => Some("Manage scheduled jobs and run history."),
|
|
"decrypt" => Some("Decrypt secure values managed by secret storage."),
|
|
"doctor" => Some("Run diagnostics for workspace and runtime health."),
|
|
"encrypt" => Some("Encrypt secure values managed by secret storage."),
|
|
"health" => Some("Process and component health snapshots."),
|
|
"local_ai" => Some("Local AI chat, inference, downloads, and media operations."),
|
|
"migrate" => Some("Data migration utilities."),
|
|
"screen_intelligence" => Some("Screen capture, permissions, and accessibility automation."),
|
|
"service" => Some("Desktop service lifecycle management."),
|
|
"skills" => Some("Skill registry, runtime lifecycle, setup, tools, and sync."),
|
|
"socket" => Some("Skills runtime socket bridge controls."),
|
|
"memory" => Some("Document storage, vector search, key-value store, and knowledge graph."),
|
|
"billing" => Some("Subscription plan, payment links, and credit top-up via the backend."),
|
|
"team" => Some("Team member management, invites, and role changes via the backend."),
|
|
"voice" => Some("Speech-to-text and text-to-speech using local models."),
|
|
"subconscious" => Some("Periodic local-model background awareness loop."),
|
|
"text_input" => Some("Read, insert, and preview text in the OS-focused input field."),
|
|
"webhooks" => {
|
|
Some("Webhook tunnel registrations and captured request/response debug logs.")
|
|
}
|
|
"update" => {
|
|
Some("Self-update: check GitHub Releases for newer core binary and stage updates.")
|
|
}
|
|
"tree_summarizer" => {
|
|
Some("Hierarchical time-based summarization tree for background knowledge compression.")
|
|
}
|
|
_ => None,
|
|
}
|
|
}
|
|
|
|
pub fn rpc_method_from_parts(namespace: &str, function: &str) -> Option<String> {
|
|
registry()
|
|
.iter()
|
|
.find(|r| r.schema.namespace == namespace && r.schema.function == function)
|
|
.map(|r| r.rpc_method_name())
|
|
}
|
|
|
|
pub fn schema_for_rpc_method(method: &str) -> Option<ControllerSchema> {
|
|
registry()
|
|
.iter()
|
|
.find(|r| r.rpc_method_name() == method)
|
|
.map(|r| r.schema.clone())
|
|
}
|
|
|
|
pub fn validate_params(
|
|
schema: &ControllerSchema,
|
|
params: &Map<String, Value>,
|
|
) -> Result<(), String> {
|
|
for input in &schema.inputs {
|
|
if input.required && !params.contains_key(input.name) {
|
|
return Err(format!(
|
|
"missing required param '{}': {}",
|
|
input.name, input.comment
|
|
));
|
|
}
|
|
}
|
|
|
|
for key in params.keys() {
|
|
if !schema.inputs.iter().any(|f| f.name == key) {
|
|
return Err(format!(
|
|
"unknown param '{}' for {}.{}",
|
|
key, schema.namespace, schema.function
|
|
));
|
|
}
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
|
|
pub async fn try_invoke_registered_rpc(
|
|
method: &str,
|
|
params: Map<String, Value>,
|
|
) -> Option<Result<Value, String>> {
|
|
for controller in registry() {
|
|
if controller.rpc_method_name() == method {
|
|
return Some((controller.handler)(params).await);
|
|
}
|
|
}
|
|
None
|
|
}
|
|
|
|
fn validate_registry(
|
|
registered: &[RegisteredController],
|
|
declared: &[ControllerSchema],
|
|
) -> Result<(), String> {
|
|
use std::collections::{BTreeMap, BTreeSet};
|
|
|
|
let mut errors: Vec<String> = Vec::new();
|
|
let mut declared_keys = BTreeSet::new();
|
|
let mut declared_rpc_methods = BTreeSet::new();
|
|
let mut registered_keys = BTreeSet::new();
|
|
let mut registered_rpc_methods = BTreeSet::new();
|
|
|
|
for schema in declared {
|
|
let key = format!("{}.{}", schema.namespace, schema.function);
|
|
if !declared_keys.insert(key.clone()) {
|
|
errors.push(format!("duplicate declared controller `{key}`"));
|
|
}
|
|
|
|
let rpc_method = rpc_method_name(schema);
|
|
if !declared_rpc_methods.insert(rpc_method.clone()) {
|
|
errors.push(format!("duplicate declared rpc method `{rpc_method}`"));
|
|
}
|
|
|
|
if schema.namespace.trim().is_empty() {
|
|
errors.push(format!(
|
|
"invalid declared controller `{key}`: namespace must not be empty"
|
|
));
|
|
}
|
|
if schema.function.trim().is_empty() {
|
|
errors.push(format!(
|
|
"invalid declared controller `{key}`: function must not be empty"
|
|
));
|
|
}
|
|
|
|
let mut required_inputs = BTreeSet::new();
|
|
let mut required_dupes: BTreeMap<String, usize> = BTreeMap::new();
|
|
for input in schema.inputs.iter().filter(|input| input.required) {
|
|
if !required_inputs.insert(input.name.to_string()) {
|
|
*required_dupes.entry(input.name.to_string()).or_default() += 1;
|
|
}
|
|
}
|
|
for (name, _) in required_dupes {
|
|
errors.push(format!(
|
|
"duplicate required input `{name}` in `{}`",
|
|
schema.method_name()
|
|
));
|
|
}
|
|
}
|
|
|
|
for controller in registered {
|
|
let key = format!(
|
|
"{}.{}",
|
|
controller.schema.namespace, controller.schema.function
|
|
);
|
|
if !registered_keys.insert(key.clone()) {
|
|
errors.push(format!("duplicate registered controller `{key}`"));
|
|
}
|
|
|
|
let rpc_method = controller.rpc_method_name();
|
|
if !registered_rpc_methods.insert(rpc_method.clone()) {
|
|
errors.push(format!("duplicate registered rpc method `{rpc_method}`"));
|
|
}
|
|
}
|
|
|
|
for key in declared_keys.difference(®istered_keys) {
|
|
errors.push(format!(
|
|
"declared controller `{key}` has no registered handler"
|
|
));
|
|
}
|
|
for key in registered_keys.difference(&declared_keys) {
|
|
errors.push(format!(
|
|
"registered controller `{key}` has no declared schema"
|
|
));
|
|
}
|
|
|
|
if errors.is_empty() {
|
|
Ok(())
|
|
} else {
|
|
Err(errors.join("; "))
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use serde_json::Map;
|
|
|
|
use super::*;
|
|
use crate::core::{ControllerSchema, FieldSchema, TypeSchema};
|
|
|
|
fn schema(
|
|
namespace: &'static str,
|
|
function: &'static str,
|
|
inputs: Vec<FieldSchema>,
|
|
) -> ControllerSchema {
|
|
ControllerSchema {
|
|
namespace,
|
|
function,
|
|
description: "test",
|
|
inputs,
|
|
outputs: vec![],
|
|
}
|
|
}
|
|
|
|
fn noop_handler(_params: Map<String, Value>) -> ControllerFuture {
|
|
Box::pin(async { Ok(Value::Null) })
|
|
}
|
|
|
|
#[test]
|
|
fn validate_registry_rejects_duplicate_namespace_function() {
|
|
let declared = vec![schema("dup", "fn", vec![]), schema("dup", "fn", vec![])];
|
|
let registered = vec![
|
|
RegisteredController {
|
|
schema: declared[0].clone(),
|
|
handler: noop_handler,
|
|
},
|
|
RegisteredController {
|
|
schema: declared[1].clone(),
|
|
handler: noop_handler,
|
|
},
|
|
];
|
|
|
|
let err = validate_registry(®istered, &declared).expect_err("expected duplicate error");
|
|
assert!(err.contains("duplicate declared controller `dup.fn`"));
|
|
}
|
|
|
|
#[test]
|
|
fn validate_registry_rejects_duplicate_required_inputs() {
|
|
let declared = vec![schema(
|
|
"doctor",
|
|
"models",
|
|
vec![
|
|
FieldSchema {
|
|
name: "use_cache",
|
|
ty: TypeSchema::Bool,
|
|
comment: "x",
|
|
required: true,
|
|
},
|
|
FieldSchema {
|
|
name: "use_cache",
|
|
ty: TypeSchema::Bool,
|
|
comment: "x",
|
|
required: true,
|
|
},
|
|
],
|
|
)];
|
|
let registered = vec![RegisteredController {
|
|
schema: declared[0].clone(),
|
|
handler: noop_handler,
|
|
}];
|
|
|
|
let err = validate_registry(®istered, &declared).expect_err("expected duplicate input");
|
|
assert!(err.contains("duplicate required input `use_cache` in `doctor.models`"));
|
|
}
|
|
}
|