mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-29 14:02:19 +00:00
a98817917c
* feat(screen-intelligence): add new commands for diagnostics and vision processing - Introduced `doctor` command for system readiness diagnostics, checking permissions and platform support. - Added `vision` command to analyze frames and persist vision summaries. - Updated CLI usage documentation to reflect new command options and improved verbosity handling. - Enhanced the `run_server` function to provide detailed endpoint information for better user guidance. This update improves the functionality and usability of the screen intelligence CLI, enabling better diagnostics and vision processing capabilities. * feat(screen-intelligence): update CLI endpoints for status monitoring - Added `tower-http` dependency for enhanced HTTP capabilities. - Updated endpoint documentation in `run_server` to reflect changes from SSE to long-polling for status updates. - Renamed `/events` endpoint to `/watch` with a query parameter for interval control, improving clarity and usability. This update enhances the screen intelligence CLI by providing more flexible status monitoring options. * feat(screen-intelligence): integrate standalone server for screen intelligence - Added a new `server` module to handle the standalone screen intelligence server functionality. - Updated CLI commands to include an `--auto-start` option for initiating capture sessions on server boot. - Enhanced the `run_server` function to provide detailed endpoint information and improved logging for server status. - Integrated the screen intelligence engine with JSON-RPC and REST endpoints for better debugging and usability. This update significantly enhances the screen intelligence capabilities by allowing it to run independently and providing more flexible configuration options. * refactor(screen-intelligence): simplify CLI options and server configuration - Removed the `--port` and `--auto-start` options from the CLI for the `screen-intelligence run` command, streamlining the command usage. - Updated the server configuration to focus on session duration and logging, enhancing clarity and usability. - Adjusted the documentation to reflect the new command structure and improved logging details for the screen intelligence server. This refactor improves the user experience by simplifying command options and enhancing the clarity of server operations. * feat(screen-intelligence): enhance screenshot management and CLI options - Updated the CLI usage documentation to include the new `--keep` option for retaining screenshots after processing. - Modified the server configuration to support the `keep_screenshots` flag, allowing for immediate saving of screenshots to disk. - Enhanced the `run_server` function to reflect the updated configuration and logging details regarding screenshot retention. - Improved the capture logic to prioritize window ID for more reliable screenshot capturing on macOS. These changes improve the usability and functionality of the screen intelligence feature by providing better control over screenshot management. * refactor(accessibility): improve capture logic and window ID handling - Updated the capture mode logic to prioritize reliable window ID capture on macOS, falling back to fullscreen capture to avoid issues with region-based capture. - Refactored the method for resolving the frontmost window ID to utilize Swift for better performance and reliability, replacing the previous JavaScript-based approach. - Enhanced the frame processing in the accessibility engine to track processed timestamps, preventing re-analysis of the same screenshot and improving efficiency. - Adjusted the vision summary parsing to support both JSON and plain text formats, ensuring better compatibility and usability. These changes enhance the reliability and performance of the accessibility features, particularly in macOS environments. * feat(accessibility): integrate Apple Vision OCR and enhance LLM context analysis - Implemented Apple Vision OCR for extracting text from screenshots, improving accuracy and reducing hallucination. - Refactored the vision processing flow to include structured prompts for LLM context analysis, enhancing the quality of the output summary. - Updated the vision summary structure to include detailed fields such as APP, DOING, FOCUS, and MOOD, providing clearer insights into the captured content. - Improved error handling and logging for image processing and OCR operations, ensuring better reliability and debugging capabilities. These changes significantly enhance the accessibility engine's ability to analyze and summarize screen content effectively. * refactor(screen-intelligence): enhance capture logic and summary structure - Improved capture mode handling to prioritize window ID availability, with a fallback to bounds-based capture before defaulting to fullscreen. - Updated the vision summary structure to clearly separate synthesized summaries, visual context, and raw OCR text for better clarity and usability. - Enhanced logging to provide more informative output during capture operations, aiding in debugging and performance monitoring. These changes improve the reliability and clarity of the screen intelligence features, particularly in managing capture contexts and summarizing visual information. * refactor(accessibility): enhance capture logic to reject fullscreen fallback - Updated the screen capture logic to prevent falling back to fullscreen mode when no valid window ID or bounds are available, improving privacy and user intent. - Added detailed logging for cases where fullscreen capture is refused, aiding in debugging and understanding capture decisions. - Introduced tests to ensure that the new logic correctly rejects fullscreen capture under invalid conditions, enhancing reliability. These changes improve the accessibility engine's handling of screen captures, ensuring that only relevant content is captured. * feat(screen-intelligence): implement capture and processing workers for enhanced vision analysis - Introduced a dedicated `capture_worker` to manage screenshot capturing from the foreground context, ensuring efficient frame handling and immediate saving of screenshots when configured. - Added a `processing_worker` to analyze captured frames using Apple Vision OCR and LLM, synthesizing insights and persisting results to unified memory. - Refactored the `AccessibilityEngine` to integrate these workers, improving the overall architecture and separation of concerns in the screen intelligence module. - Enhanced logging throughout the capture and processing workflows for better debugging and performance monitoring. These changes significantly improve the screen intelligence capabilities by enabling real-time capture and analysis of visual data, enhancing user experience and functionality. * Merge remote-tracking branch 'upstream/main' into fix/screen-intgellignce-2 * refactor(screen-intelligence): improve type handling and visibility in capture and processing modules - Updated the calculation of baseline milliseconds in `capture_worker` to ensure proper type handling with floating-point division. - Made several fields in `SessionRuntime` and `EngineState` public for better accessibility within the crate. - Changed the visibility of the `analyze_frame` function in `processing_worker` to public within the crate, allowing it to be called from `engine.rs`. These changes enhance type safety and improve the modularity of the screen intelligence components, facilitating better integration and usage across the codebase. * refactor(screen-intelligence): update visibility and helper function usage in engine and processing modules - Changed the `SessionRuntime` struct to be public within the crate, enhancing accessibility. - Updated the `analyze_frame` function parameter to use a reference instead of a direct reference to the `AccessibilityEngine`, improving clarity. - Refactored calls to `persist_vision_summary` to utilize the helper function from the `super::helpers` module, promoting better organization and code reuse. These changes streamline the code structure and improve the modularity of the screen intelligence components. * feat(screen-intelligence): introduce input handling and autocomplete features - Added a new `input.rs` module to manage input actions, including keyboard/mouse automation and predictive text functionalities. - Implemented methods for handling input actions, generating autocomplete suggestions, and committing selected suggestions within the `AccessibilityEngine`. - Created a `state.rs` module to encapsulate engine state management, including session runtime and engine state structures. - Enhanced the `engine.rs` module to support session lifecycle management and integrate new input functionalities. - Introduced a `vision.rs` module for vision-related query methods, improving the overall architecture and modularity of the screen intelligence components. These changes significantly enhance user interaction capabilities and streamline the management of input actions and vision processing. * fix(screen-intelligence): update worker imports to use state module Workers imported AccessibilityEngine from engine.rs but it was moved to state.rs during the refactor. Fix the import paths. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(screen-intelligence): clean up formatting and improve readability in various modules - Reformatted print statements in `screen_intelligence_cli.rs` for better readability. - Simplified assertions in tests within `capture.rs` to enhance clarity. - Streamlined function definitions and calls in `focus.rs`, `engine.rs`, and `processing_worker.rs` for improved code organization. - Updated import statements in `mod.rs` and `server.rs` to maintain consistency and clarity. These changes enhance the overall readability and maintainability of the codebase, promoting better coding practices across the screen intelligence components. * refactor(screen-intelligence): update import paths for AccessibilityEngine and EngineState - Changed the import statements in `tests.rs` to source `AccessibilityEngine` and `EngineState` from the `state` module instead of the `engine` module. - This adjustment aligns with recent refactoring efforts to improve module organization and maintainability. These changes enhance code clarity and ensure consistency in module usage across the screen intelligence components. * fix(screen-intelligence): fix test compilation and assertions - Update test imports from engine to state module - Add mock env var path to processing_worker::analyze_frame for test support - Update parser tests for plain-text mode (non-JSON fallback) - Handle missing window_id gracefully in capture_scheduler test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(screen-intelligence): enhance code clarity and performance in various modules - Updated command-line usage documentation in `screen_intelligence_cli.rs` to reflect new options. - Improved string handling for truncation in `run_start_session` and `run_vision` functions to use character counts instead of byte lengths, ensuring accurate truncation for multi-byte characters. - Added intentional blocking sleep in `resolve_frontmost_window_id` to minimize impact on the Tokio runtime during app switches. - Implemented a consistent default confidence score in vision processing and improved YAML escaping in `persist_vision_summary`. - Refactored session management in `capture_worker` and `engine` modules to reduce lock contention and improve performance during I/O operations. These changes enhance the overall performance, maintainability, and user experience of the screen intelligence components. * fix(tests): update confidence assertion in vision summary test - Adjusted the expected confidence value in the `parse_vision_missing_fields` test to reflect the new default confidence score of 0.8, ensuring consistency across JSON and plain-text branches. This change improves the accuracy of the test and aligns it with recent updates in the vision processing logic. * refactor(screen-intelligence): improve code formatting and readability in various modules - Enhanced string formatting for truncation in `run_vision` to improve clarity. - Reformatted variable declarations in `capture_worker` for better readability. - Updated YAML escaping logic in `helpers.rs` for consistency and clarity. These changes contribute to improved maintainability and readability of the codebase. * refactor(screen-intelligence): reorganize analyze_frame function for improved flow - Moved configuration validation to the beginning of the `analyze_frame` function to ensure proper setup before processing. - Reintroduced the Apple Vision OCR and Vision LLM steps in the correct order, enhancing the logical flow of the analysis process. These changes improve the readability and maintainability of the code, ensuring a clearer structure for future modifications. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OpenHuman
The age of super intelligence is here. OpenHuman is your Personal AI super intelligence. Private, Simple and extremely powerful.
Discord • Reddit • X/Twitter • Docs
"The Tet. What a brilliant machine" — Morgan Freeman as he reminisces about alien superintelligence in the movie Oblivion
Early Beta — Under active development. Expect rough edges.
To install or get started, either download from the website over at tinyhumans.ai/openhuman or run
# For MacOS/Linux
curl -fsSL https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.sh | bash
# For Windows
irm https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.ps1 | iex
What is OpenHuman?
OpenHuman is an open-source agentic assistant that is designed to integrate with you in your daily life. Here's what makes OpenHuman special:
-
Simple, UI-first — A clean desktop experience and short onboarding paths so you can go from install to a working agent in a few clicks, without a config-first setup. You don't need a terminal to run OpenHuman.
-
One subscription, many providers — You only need one account to get access to many agentic APIs (AI Models, Search, Webhooks/Tunnels and other 3rd party APIs etc..), simplifying the experience to get a powerful agent going.
-
Rich Skills — Plug into Gmail, Slack, Notion, and the rest of your stack via rich, feature-backed skills. Connections are typically one click through setup wizards instead of wiring APIs by hand. Workflow data is kept on device, encrypted locally, and treated as yours: encryption and sensitive context stay on your machine. Webhooks give instant feedback into the agent when external systems or skills emit events, so the loop stays tight without constant polling.
-
Local knowledge base — Built from your data and your activity. How you work across tools, sessions, and connected services—so the agent gets rich, workflow-aware context, not a one-off chat transcript. Everything is stored on your machine and compounding over time without becoming a cloud dossier. Channels, skills and ongoing conversations feed the same loop so day-to-day context does not reset every session.
-
Local AI model — The Rust core exposes local AI paths (and the desktop bundle can ship local/bundled runners where applicable) for the workloads above—vision snippets, speech helpers, summarization, tooling—so sensitive steps can stay off the cloud when you choose.
-
Deep desktop integrations — OpenHuman is a native desktop assistant, not a web-only chat: memory-aware keyboard autocomplete, voice (STT listening and TTS replies), screen intelligence that understands what is on screen and feeds your local context, plus windowing and OS-level permissions—so the agent meets you on the machine, not trapped in a browser tab.
Architecture: docs/ARCHITECTURE.md. Contributor orientation: CONTRIBUTING.md.
OpenHuman vs other agents
High-level comparison (products evolve—verify against each vendor). OpenHuman is built to minimize vendor sprawl, keep workflow knowledge on-device, and ship deep desktop features—not only chat.
| Claude Code/Cowork | OpenClaw | Hermes Agent | OpenHuman | |
|---|---|---|---|---|
| Open-source: Is the codebase open to review? | 🚫 Proprietary client | ✅ MIT License | ✅ MIT License | ✅ GNU License |
| Simple: Is it simple to get started? | ✅ Simple Desktop App + CLI | ⚠️ Terminal first and often complex | ⚠️ Terminal first and often complex | ✅ Simple, Clean UI/UX. Get started within minutes |
| Cost: How expensive is to run? | ⚠️ Subscription + add-on tool/API costs | ⚠️ Tied to models & hosting you choose | ⚠️ Tied to models & hosting you choose | ✅ Cost optimized with the option to run many things locally for free |
| Memory & Knowledge Base (KB): Does the agent know you and your world? | ✅ Built-in memory; mostly chat/session scoped | ⚠️ Has a local memory but often needs plugins for richer behavior | ✅ Self-learning / task loops (typical) | 🚀 Local KB + Self-learning from your activity & data (GMail, Notion etc... via skills) & prompts |
| API spagetti: How complex is it to hook mulitple features together? | 🚫 Claude bill + often extra keys for MCP/tools | 🚫 BYOK / multi-vendor common | 🚫 Multiple providers common | ✅ One account get access to many bundled platform APIs |
| Extensibility: Can you add rich features into it? | ✅ MCP (different model than sandboxed skills) | ✅ Plugin Architecture (SKILL.md) | ✅ Plugin Architecture (SKILL.md) | 🚀 Rich Skills with ability to have realtime updates, local DB & more |
| Desktop integrations: Can it integrate into your desktop completely? | ⚠️ Desktop app & access to folders | ⚠️ Often lighter native surface | ⚠️ Often lighter native surface | ✅ STT, TTS, screen intelligence, memory-aware autocomplete and a whole lot more |
Contributors Hall of Fame
Show some love and end up in the hall of fame