7.7 KiB
description, icon
| description | icon |
|---|---|
| On-device screen capture, OCR + vision summarization, input automation, and inline autocomplete — gated behind explicit macOS privacy permissions. | scan-eye |
Screen Intelligence
Screen Intelligence lets the agent see what you're working on. When you start a consent-gated session, OpenHuman periodically screenshots your active window, runs it through on-device OCR and a local vision model, and synthesizes a short "what the user is doing right now" note into memory. On top of that capture loop it offers input automation (text staging, keyboard actions, a panic stop) and a separate system-wide autocomplete that suggests inline text completions in any focused field.
Everything runs locally. Capture, OCR, and summarization happen on your machine using the local model (Ollama); nothing is sent to the cloud as part of this feature.
Platform: Screen Intelligence is macOS-only in V1. On Windows and Linux the engine reports
platform_supported: false,start_sessionreturns amacOS-onlyerror, and the embedded capture server does not autostart. Microphone permission detection is the only cross-platform piece.
What it does
A session runs two cooperating background workers:
| Worker | Role |
|---|---|
| Capture worker | Polls the foreground window at baseline_fps (default 1 fps), screenshots the active window only via screencapture -l <windowID> (no fullscreen fallback), applies the allow/deny policy, optionally saves a PNG, and enqueues the frame. |
| Processing worker | Drains to the latest frame, compresses it (PNG → JPEG, longest edge ≤ 1024 px, quality 72), runs Apple Vision OCR, then a vision LLM, then a synthesis LLM, and persists a VisionSummary to memory. |
Capture only proceeds when the foreground context exposes a window_id. Stale queued frames are discarded — only the most recent frame is analyzed, deduped by capture timestamp.
Sessions
Sessions are explicitly consent-gated and time-boxed:
- You start a session from Settings (or the
screen_intelligence.start_sessionRPC) withconsent: true. - TTL is clamped to 30–3600 seconds (the Settings panel defaults to 300 s). When the TTL expires the session stops on its own.
- Analyze now flushes the pending frame through the vision pipeline immediately.
- Stop ends the session; the
panic_stopinput action force-stops it instantly.
Input automation
While a session is active, input_action lets the agent stage text and signal keyboard intent into the session's autocomplete context. Actions are blocked when no session is active or when the foreground app is denylisted by the active policy. The special panic_stop action immediately tears down the session regardless of state — a hard kill switch.
The capture-side autocomplete helpers (autocomplete_suggest / autocomplete_commit) maintain a small in-memory context buffer (capped at 256 chars) and return heuristic suggestions; they're gated on autocomplete_enabled.
Autocomplete
Separate from the capture session, OpenHuman ships a system-wide inline autocomplete engine (the autocomplete domain). It is also macOS-only at runtime.
- It captures your currently-focused text field through the macOS accessibility (AX) layer, runs local inference to generate a short single-line continuation, and renders it in a floating overlay badge.
- Press Tab to accept (it inserts the text and cleans up any stray indentation the app added), or Escape to reject.
- Accepted completions are saved as personalization examples in a local KV store and a local memory-doc namespace, and feed back into later suggestions.
- It special-cases terminals (extracting just the input line), skips blocked/disabled apps and OpenHuman's own window, and filters low-quality suggestions (too short, no alphanumerics, or an echo of what you just typed).
- Debounce is clamped 50–2000 ms; the displayed/applied suggestion is capped at 64 characters. After 5 consecutive inference failures the engine auto-stops to avoid notification floods.
There is also an in-app path: the OpenHuman composer passes an explicit context to the engine, bypassing AX capture entirely.
Permissions
Capture and automation require macOS privacy grants. OpenHuman detects each one and can open the relevant System Settings → Privacy & Security pane and trigger the system prompt.
| Permission | Why it's needed | Detected via |
|---|---|---|
| Screen Recording | Screenshot the active window for OCR + vision. | CGPreflightScreenCaptureAccess |
| Accessibility | Read the foreground window/element, capture focused text, and insert text (required to start a session and to run autocomplete). | AXIsProcessTrusted |
| Input Monitoring | Listen for accept/reject key edges (Tab/Escape) and the Globe/Fn hotkey. | IOHIDCheckAccess |
| Microphone | Voice features (cross-platform; the only permission detected off macOS). | CPAL device probe |
Restart after granting. macOS TCC grants are per-executable and per-process — a running core never sees a freshly granted permission. After you grant a permission you must restart the core for it to take effect. The status payload carries
permission_check_process_pathand the core process pid/start time so the UI can confirm a restart actually happened. The panel exposes a "refresh with restart" action for this.
Privacy considerations
- Local-only processing. OCR (Apple Vision) and both LLM passes run on-device via the local model. The vision pipeline requires
local_ai.runtime_enabled = truewith theollamaprovider; without it, analysis errors out rather than falling back to a cloud call. - What's stored. Each summary is written to unified memory in the
backgroundnamespace (source typescreenshot, taggedscreen_intelligence) as a small markdown doc with the app name, window title, capture timestamp, and confidence. Raw frames are not persisted to memory. - Screenshots on disk are opt-in. PNGs are only saved to
{workspace}/screenshots/when Keep screenshots is enabled; otherwise a temp file is written for OCR and deleted immediately. - Allow/deny policy. A policy mode (
all_except_blacklistorwhitelist_only) plus allow/deny app lists control which windows are ever captured. - Time-boxed + consent-gated. Nothing captures until you start a session, and every session expires on its TTL.
Enabling & configuring
Configure it under Settings → Screen awareness (the ScreenIntelligencePanel):
- Permissions section — per-permission status (granted / denied / unknown), request buttons, and the restart-to-refresh flow.
- Enabled — master toggle for the feature.
- Mode —
All except blacklistorWhitelist only, backed by the allow/deny lists. - Screen monitoring — toggle the capture loop for a session.
- Session controls — Start / Stop / Analyze now, with a live remaining-time countdown. Start is disabled until Accessibility is granted and the platform is supported.
Under the hood these map to the [screen_intelligence] config block (enabled, vision_enabled, use_vision_model, keep_screenshots, baseline_fps, session_ttl_secs, policy_mode, allow/deny lists, autocomplete_enabled) and the screen_intelligence.* JSON-RPC surface. The same capabilities are available from the openhuman screen-intelligence CLI (status, start, stop, capture, doctor, run).
See also
- Memory Tree — where vision summaries land.
- Privacy & Security — the broader permission and data model.
- Voice — the other feature that uses the Microphone permission.