* test(screen_intelligence): add E2E pipeline proof tests for #343
Add layered test coverage proving the full capture → vision → memory
pipeline: screenshot save/cleanup disk paths, VisionSummary serde
roundtrip, JSON-RPC shape tests for status and vision_recent endpoints.
- tests/screen_intelligence_vision_e2e.rs: save_screenshot_to_disk
creates a PNG and keep_screenshots=false cleanup removes it;
VisionSummary struct serializes/persists/is queryable end-to-end;
platform support table + macOS checklist added to module doc
- tests/json_rpc_e2e.rs: screen_intelligence_status shape test
(platform_supported, session.active, permissions.screen_recording);
vision_recent returns empty summaries without an active session
- src/openhuman/screen_intelligence/tests.rs: save_screenshot_to_disk
unit tests for the write path and the no-image-ref error path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(screen_intelligence): enhance vision summary persistence and error handling
- Added new fields to track vision persistence count, last persisted key, and last persist error in SessionRuntime and SessionStatus.
- Implemented error handling for vision summary persistence, ensuring errors are logged and state is updated accordingly.
- Introduced a new method to analyze a frame and persist the summary, improving the vision processing pipeline.
- Updated tests to validate the new functionality and ensure proper behavior with mocked vision outputs.
This commit improves the robustness of the screen intelligence pipeline by enhancing the tracking and handling of vision summary persistence.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(autocomplete): add target_role to EngineState for improved context validation
- Introduced a new field `target_role` in `EngineState` to store the AXRole of the text element when suggestions are generated.
- Updated the `AutocompleteEngine` to validate the focused element against the expected app and role before applying suggestions.
- Enhanced the `apply_text_to_focused_field` function to include role validation, ensuring better accuracy in text insertion.
- Improved the handling of suggestion context and error states during autocomplete operations.
This update enhances the reliability of the autocomplete feature by ensuring that suggestions are applied only when the correct context is maintained.
* fix(package): update dev:app script to include core staging step
- Modified the `dev:app` script in `package.json` to run `yarn core:stage` before executing `tauri dev`, ensuring that the core sidecar is properly staged during development.
- This change enhances the development workflow by automating the staging process, reducing manual steps for developers.
This update improves the reliability of the development environment setup.
* feat(autocomplete): implement core autocomplete engine and supporting modules
- Introduced a new `AutocompleteEngine` struct to manage the state and operations of the autocomplete feature, including starting, stopping, and refreshing suggestions.
- Added `EngineState` to track the current status, phase, and context of the autocomplete process.
- Implemented helper functions for managing focus and overlay notifications, enhancing user interaction with suggestions.
- Created utility functions for terminal context extraction and text sanitization, improving the accuracy of suggestions.
- Developed a comprehensive set of types and structures to support autocomplete operations, including suggestion handling and status reporting.
This update lays the foundation for a robust autocomplete feature, enhancing user experience through improved context awareness and interaction.
* refactor(autocomplete): remove unused functions and improve clipboard handling
- Deleted the `normalize_ax_value` and `parse_ax_number` functions as they were not utilized in the codebase, streamlining the autocomplete module.
- Enhanced the `clipboard_save` function to improve readability by formatting the string conversion of clipboard output, ensuring better handling of empty or "missing value" cases.
This update simplifies the code and improves the overall maintainability of the autocomplete functionality.
* refactor(autocomplete): remove unused functions and improve clipboard handling
- Deleted the `normalize_ax_value` and `parse_ax_number` functions as they were not utilized in the codebase, streamlining the autocomplete module.
- Enhanced the `clipboard_save` function to improve readability by formatting the string conversion of clipboard output, ensuring better handling of empty or "missing value" cases.
This update cleans up the code and optimizes the clipboard handling logic for the autocomplete feature.
* feat(autocomplete): enhance focus handling and text insertion methods
- Implemented a unified Swift helper for querying focused text elements, improving performance and reliability on macOS.
- Added fallback mechanisms to use osascript for focus queries when the helper is unavailable, ensuring consistent functionality.
- Refactored text insertion logic to prioritize the unified helper, with osascript and AXValue as fallback options, enhancing user experience during text application.
- Cleaned up and organized focus-related functions, improving code readability and maintainability.
This update significantly enhances the autocomplete feature's ability to interact with focused text elements, providing a more robust and responsive user experience.
* feat(accessibility): introduce comprehensive accessibility module for macOS
- Added a new `accessibility` module that centralizes focus queries, screen capture, key state detection, and permission management for macOS.
- Implemented a unified Swift helper process to enhance performance and reliability in querying focused text elements and managing overlays.
- Introduced various functionalities including screen capture, text insertion into focused fields, and permission detection for accessibility features.
- Enhanced user experience by providing robust methods for interacting with accessibility APIs, ensuring consistent behavior across different contexts.
This update significantly improves the accessibility capabilities of the application, providing a more responsive and user-friendly interface for macOS users.
* feat(accessibility): introduce screen capture and focus query modules
- Added a new `accessibility` module to centralize platform-specific accessibility functionalities, including screen capture and focus queries.
- Implemented `capture.rs` for screen capture using platform-native tools, supporting both windowed and fullscreen modes.
- Developed `focus.rs` to handle accessibility focus queries, utilizing a unified Swift helper for improved performance on macOS.
- Introduced helper functions for managing overlays and permissions, enhancing user interaction and accessibility features.
This update significantly enhances the application's accessibility capabilities, providing robust tools for screen capture and focus management.
* refactor(accessibility): remove unused accessibility functions and streamline modules
- Deleted unused functions from the accessibility module, including `focused_text_context`, `is_text_role`, `normalize_ax_value`, and `parse_ax_number`, to enhance code clarity and maintainability.
- Updated module documentation to reflect the current structure and purpose, ensuring consistency across the accessibility and screen intelligence modules.
This update simplifies the codebase and improves the overall organization of accessibility-related functionalities.
* feat(image-processing): implement image compression and resizing for vision LLM
- Added a new module `image_processing` to handle the compression and resizing of screenshots before sending them to the vision LLM.
- Implemented the `compress_screenshot` function, which decodes PNG data-URIs, resizes images to fit within a specified maximum dimension, and re-encodes them as JPEGs.
- Updated the `AccessibilityEngine` to utilize the new image processing functionality, ensuring that images sent for analysis are optimized for size and quality.
- Introduced new dependencies in `Cargo.toml` for image handling and compression.
This update enhances the efficiency of image processing in the application, reducing token usage and improving inference speed.
* feat(tests): add end-to-end tests for screen intelligence vision pipeline
- Introduced a new test file `screen_intelligence_vision_e2e.rs` to validate the complete flow of the screen intelligence vision pipeline.
- Implemented tests that cover generating images, compressing and resizing them, simulating LLM responses, and persisting results to memory.
- Utilized temporary directories and environment variable management to ensure test isolation and reliability.
- Enhanced the testing framework by including helper functions for image creation and mock responses, improving the overall test coverage and robustness.
This update significantly strengthens the testing capabilities of the screen intelligence module, ensuring that the entire pipeline functions correctly under various scenarios.
* style: apply cargo fmt and import ordering fixes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(accessibility): add non-macOS stub for validate_focused_target
The function was gated with #[cfg(target_os = "macos")] but exported
unconditionally from mod.rs, causing compilation failure on non-macOS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(accessibility): enhance screen capture and error handling
- Updated the screen capture functionality to include a timestamp helper in the documentation.
- Improved error handling when reading resized screenshots, ensuring temporary files are removed on failure.
- Added logging for raw errors returned by the helper in the focus context, providing better debugging information.
- Refactored clipboard handling in the paste functionality to preserve multi-line text, enhancing usability.
- Introduced a constant for terminal application names to simplify terminal detection logic.
This update improves the robustness and clarity of the accessibility module, enhancing both screen capture and focus handling capabilities.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
