2545ae374a
* feat(autocomplete): add target_role to EngineState for improved context validation - Introduced a new field `target_role` in `EngineState` to store the AXRole of the text element when suggestions are generated. - Updated the `AutocompleteEngine` to validate the focused element against the expected app and role before applying suggestions. - Enhanced the `apply_text_to_focused_field` function to include role validation, ensuring better accuracy in text insertion. - Improved the handling of suggestion context and error states during autocomplete operations. This update enhances the reliability of the autocomplete feature by ensuring that suggestions are applied only when the correct context is maintained. * fix(package): update dev:app script to include core staging step - Modified the `dev:app` script in `package.json` to run `yarn core:stage` before executing `tauri dev`, ensuring that the core sidecar is properly staged during development. - This change enhances the development workflow by automating the staging process, reducing manual steps for developers. This update improves the reliability of the development environment setup. * feat(autocomplete): implement core autocomplete engine and supporting modules - Introduced a new `AutocompleteEngine` struct to manage the state and operations of the autocomplete feature, including starting, stopping, and refreshing suggestions. - Added `EngineState` to track the current status, phase, and context of the autocomplete process. - Implemented helper functions for managing focus and overlay notifications, enhancing user interaction with suggestions. - Created utility functions for terminal context extraction and text sanitization, improving the accuracy of suggestions. - Developed a comprehensive set of types and structures to support autocomplete operations, including suggestion handling and status reporting. This update lays the foundation for a robust autocomplete feature, enhancing user experience through improved context awareness and interaction. * refactor(autocomplete): remove unused functions and improve clipboard handling - Deleted the `normalize_ax_value` and `parse_ax_number` functions as they were not utilized in the codebase, streamlining the autocomplete module. - Enhanced the `clipboard_save` function to improve readability by formatting the string conversion of clipboard output, ensuring better handling of empty or "missing value" cases. This update simplifies the code and improves the overall maintainability of the autocomplete functionality. * refactor(autocomplete): remove unused functions and improve clipboard handling - Deleted the `normalize_ax_value` and `parse_ax_number` functions as they were not utilized in the codebase, streamlining the autocomplete module. - Enhanced the `clipboard_save` function to improve readability by formatting the string conversion of clipboard output, ensuring better handling of empty or "missing value" cases. This update cleans up the code and optimizes the clipboard handling logic for the autocomplete feature. * feat(autocomplete): enhance focus handling and text insertion methods - Implemented a unified Swift helper for querying focused text elements, improving performance and reliability on macOS. - Added fallback mechanisms to use osascript for focus queries when the helper is unavailable, ensuring consistent functionality. - Refactored text insertion logic to prioritize the unified helper, with osascript and AXValue as fallback options, enhancing user experience during text application. - Cleaned up and organized focus-related functions, improving code readability and maintainability. This update significantly enhances the autocomplete feature's ability to interact with focused text elements, providing a more robust and responsive user experience. * feat(accessibility): introduce comprehensive accessibility module for macOS - Added a new `accessibility` module that centralizes focus queries, screen capture, key state detection, and permission management for macOS. - Implemented a unified Swift helper process to enhance performance and reliability in querying focused text elements and managing overlays. - Introduced various functionalities including screen capture, text insertion into focused fields, and permission detection for accessibility features. - Enhanced user experience by providing robust methods for interacting with accessibility APIs, ensuring consistent behavior across different contexts. This update significantly improves the accessibility capabilities of the application, providing a more responsive and user-friendly interface for macOS users. * feat(accessibility): introduce screen capture and focus query modules - Added a new `accessibility` module to centralize platform-specific accessibility functionalities, including screen capture and focus queries. - Implemented `capture.rs` for screen capture using platform-native tools, supporting both windowed and fullscreen modes. - Developed `focus.rs` to handle accessibility focus queries, utilizing a unified Swift helper for improved performance on macOS. - Introduced helper functions for managing overlays and permissions, enhancing user interaction and accessibility features. This update significantly enhances the application's accessibility capabilities, providing robust tools for screen capture and focus management. * refactor(accessibility): remove unused accessibility functions and streamline modules - Deleted unused functions from the accessibility module, including `focused_text_context`, `is_text_role`, `normalize_ax_value`, and `parse_ax_number`, to enhance code clarity and maintainability. - Updated module documentation to reflect the current structure and purpose, ensuring consistency across the accessibility and screen intelligence modules. This update simplifies the codebase and improves the overall organization of accessibility-related functionalities. * feat(image-processing): implement image compression and resizing for vision LLM - Added a new module `image_processing` to handle the compression and resizing of screenshots before sending them to the vision LLM. - Implemented the `compress_screenshot` function, which decodes PNG data-URIs, resizes images to fit within a specified maximum dimension, and re-encodes them as JPEGs. - Updated the `AccessibilityEngine` to utilize the new image processing functionality, ensuring that images sent for analysis are optimized for size and quality. - Introduced new dependencies in `Cargo.toml` for image handling and compression. This update enhances the efficiency of image processing in the application, reducing token usage and improving inference speed. * feat(tests): add end-to-end tests for screen intelligence vision pipeline - Introduced a new test file `screen_intelligence_vision_e2e.rs` to validate the complete flow of the screen intelligence vision pipeline. - Implemented tests that cover generating images, compressing and resizing them, simulating LLM responses, and persisting results to memory. - Utilized temporary directories and environment variable management to ensure test isolation and reliability. - Enhanced the testing framework by including helper functions for image creation and mock responses, improving the overall test coverage and robustness. This update significantly strengthens the testing capabilities of the screen intelligence module, ensuring that the entire pipeline functions correctly under various scenarios. * style: apply cargo fmt and import ordering fixes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(accessibility): add non-macOS stub for validate_focused_target The function was gated with #[cfg(target_os = "macos")] but exported unconditionally from mod.rs, causing compilation failure on non-macOS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(accessibility): enhance screen capture and error handling - Updated the screen capture functionality to include a timestamp helper in the documentation. - Improved error handling when reading resized screenshots, ensuring temporary files are removed on failure. - Added logging for raw errors returned by the helper in the focus context, providing better debugging information. - Refactored clipboard handling in the paste functionality to preserve multi-line text, enhancing usability. - Introduced a constant for terminal application names to simplify terminal detection logic. This update improves the robustness and clarity of the accessibility module, enhancing both screen capture and focus handling capabilities. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OpenHuman
The age of super intelligence is here. OpenHuman is your Personal AI super intelligence. Private, Simple and extremely powerful.
Discord • Reddit • X/Twitter • Docs
"The Tet. What a brilliant machine" — Morgan Freeman as he reminisces about alien superintelligence in the movie Oblivion
Early Beta — Under active development. Expect rough edges.
To install or get started, either download from the website over at tinyhumans.ai/openhuman or run
# For MacOS/Linux
curl -fsSL https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.sh | bash
# For Windows
irm https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.ps1 | iex
What is OpenHuman?
OpenHuman is an open-source agentic assistant that is designed to integrate with you in your daily life. Here's what makes OpenHuman special:
-
One subscription, many providers — One assistant wired to skills and backend models so you are not juggling a separate subscription stack for every integration surface.
-
Incredible memory — Rust-side memory (store / recall / namespaces) plus optional TinyHumans Neocortex-backed context when configured, so the agent can retain and retrieve more than a single chat window. Channels and ongoing conversations feed the same loop so day-to-day context does not reset every session.
-
Screen intelligence — Regular screen capture (on a cadence or when triggered) feeds an on-device pipeline that understands what is on screen, distills it into memory (facts, UI state, workflows), and can propose actions the agent executes for you. OS permissions and capture APIs vary by platform; the goal is your machine first, not shipping raw frames to the cloud by default.
-
Voice & meetings — A Local-model speech stack (listen / TTS) let the assistant talk back and capture or work with meeting audio with a privacy-first default when you route inference locally. Transcripts and summaries land in the same memory + agent loop so OpenHuman can follow up: tasks, drafts, calendar nudges, or skill-backed workflows—without treating a meeting as a one-off chat.
-
Memory-aware autocomplete — Keyboard autocomplete is built for right-context suggestions: it consults memory namespaces and recent context so completions stay aligned with you, your workspace, and prior sessions—not a blank model every keystroke.
-
Runs a local AI model — The Rust core exposes local AI paths (and the desktop bundle can ship local/bundled runners where applicable) for the workloads above—vision snippets, speech helpers, summarization, tooling—so sensitive steps can stay off the cloud when you choose.
-
Simple or advanced — Skill setup wizards and defaults for common tools, with room to go deeper via settings, credentials, and core RPC when you need control and privacy.
Architecture: docs/ARCHITECTURE.md. Contributor orientation: CONTRIBUTING.md.
Under the hood (Architecture)
OpenHuman is a desktop monorepo: Rust owns business logic and execution; the UI owns interaction, layout, and OS integration.
Rust (openhuman / openhuman_core). The repo root src/ crate is the brain: JSON-RPC over HTTP (core_server), domain modules (auth, config, memory, skills, channels, screen intelligence, local AI, cron, …), and a QuickJS runtime for sandboxed JavaScript skills. The openhuman binary is built and staged next to the Tauri app so the desktop shell can spawn it as a sidecar. Heavy work—SQLite, sockets, crypto, skill lifecycle—runs there under Tokio, not in the WebView.
UI (app/). Vite + React (TypeScript) implements screens, onboarding, settings, and realtime UX. Redux Toolkit holds client state; Socket.io and the MCP-style client stack stay in sync with the core’s realtime surface. Tauri v2 (app/src-tauri/) is a thin Rust host: windowing, filesystem hooks where needed, and core_rpc_relay—forwarding JSON-RPC from the WebView to the openhuman process so the UI never re-implements domain rules.
Controllers and the RPC surface. Features are exposed as registered controllers: each domain declares schemas (namespace, function name, parameter shapes) and a handler. At runtime, calls are validated, dispatched by method name (e.g. openhuman.auth_get_state, openhuman.local_ai_agent_chat), and return structured outcomes. CLI and HTTP share the same controller catalog, so automation, tests, and the app all hit one contract.
What ties it together: one registry of controllers, one sidecar process for execution, Tauri IPC for shell-only capabilities, and HTTP JSON-RPC for everything else—plus skills and dual-socket behavior documented in the architecture guide.
Read more: docs/ARCHITECTURE.md · Frontend tree: docs/src/README.md · Tauri commands: docs/src-tauri/README.md
Contributors Hall of Fame
Show some love and end up in the hall of fame