mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-30 23:14:37 +00:00
refactor: remove screen awareness and capture surfaces (#5186)
This commit is contained in:
Steven Enamakel
@@ -51,9 +51,8 @@ app/src-tauri/src/
|
||||
├── webview_apis/ # WS bridge for webview-side APIs
|
||||
├── discord_scanner/ … whatsapp_scanner/ … # Per-provider scanners (slack, telegram, wechat,
|
||||
│ # gmessages, imessage, meet, …) driving CDP
|
||||
├── meet_audio/ meet_call/ meet_video/ # Google Meet call window + media capture
|
||||
├── meet_audio/ meet_call/ meet_video/ # Google Meet call window + media integration
|
||||
├── fake_camera/ # Virtual camera support
|
||||
├── screen_capture/ # Screen share picker sessions (getDisplayMedia shim backend)
|
||||
├── mascot_native_window.rs / notch_window.rs / window_state.rs
|
||||
├── dictation_hotkeys.rs / ptt_hotkeys.rs / ptt_overlay.rs / companion_commands.rs
|
||||
├── native_notifications/ notification_settings/
|
||||
@@ -168,16 +167,6 @@ Hide-to-tray / reopen behavior is **not** an IPC command — it lives in the `Ru
|
||||
| `start_loopback_oauth_listener` / `stop_loopback_oauth_listener` | Localhost OAuth redirect listener (`loopback_oauth.rs`) |
|
||||
| `claude_code_login_launch` | Launch the Claude Code login flow (`claude_code.rs`) |
|
||||
|
||||
### Screen share picker (CEF / macOS)
|
||||
|
||||
From **`screen_capture/mod.rs`**. Backs the in-page `getDisplayMedia` shim in `webview_accounts/runtime.js`. Session-gated: the shim must open a session with a live user gesture before enumeration / thumbnail captures succeed. See issue #713 (picker UX) + #812 (session gating).
|
||||
|
||||
| Command | Purpose |
|
||||
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `screen_share_begin_session` | Open a 30s session from an account webview, after a `navigator.userActivation.isActive` gesture. Returns `{ token, sources }`. Rate-limited to 10/minute per account. |
|
||||
| `screen_share_thumbnail` | Capture a single source's thumbnail as base64 PNG. Requires a live token and an `id` that the session was issued for. macOS only; other platforms return an error. |
|
||||
| `screen_share_finalize_session` | Close the session. Called by the shim on Share or Cancel; safe to call with an unknown/expired token (no-op). |
|
||||
|
||||
### Workspace file links
|
||||
|
||||
From **`workspace_paths.rs`** (closes `#1402`). These commands accept workspace-relative paths only. The shell resolves each path against the active OpenHuman workspace, canonicalizes the target, and rejects traversal, absolute paths, URI-like prefixes, and symlink escapes before opening or reading anything.
|
||||
|
||||
@@ -11,13 +11,13 @@ How OpenHuman tests its product. Source of truth for "where does my test go?". C
|
||||
|
||||
## Layers
|
||||
|
||||
| Layer | Where it lives | What it tests | Driver |
|
||||
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Rust unit** | `#[cfg(test)] mod tests` inside the same `*.rs` file, or sibling `tests.rs`, or `tests/` subdir under a domain (e.g. `src/openhuman/channels/tests/`) | Pure domain logic, schemas, RPC handler shape, in-memory state machines | `cargo test` |
|
||||
| **Rust integration** | `tests/*.rs` at repo root | Full domain wiring with real Tokio runtime, mock external services, JSON-RPC end-to-end (`tests/json_rpc_e2e.rs`), domain × domain interactions | `pnpm test:rust` (which calls `bash scripts/test-rust-with-mock.sh`) |
|
||||
| **Vitest unit** | Co-located as `*.test.ts(x)` next to source under `app/src/**`, or under `app/src/**/__tests__/` | React components, hooks, store slices, pure utilities, service-layer adapters | `pnpm test:unit` |
|
||||
| **WDIO E2E** | `app/test/e2e/specs/*.spec.ts` | Full desktop flow: UI → Tauri → in-process core → JSON-RPC; user-visible behaviour | All platforms: Appium Chromium driver (port 4723) against the CEF runtime. See [E2E Testing](e2e-testing.md). |
|
||||
| **Manual smoke** | [`docs/RELEASE-MANUAL-SMOKE.md`](../../docs/RELEASE-MANUAL-SMOKE.md) | OS-level surfaces drivers cannot assert: TCC permission prompts, Gatekeeper, code signing, DMG install, OS-native toasts | Human at release-cut, signed off in release PR |
|
||||
| Layer | Where it lives | What it tests | Driver |
|
||||
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| **Rust unit** | `#[cfg(test)] mod tests` inside the same `*.rs` file, or sibling `tests.rs`, or `tests/` subdir under a domain (e.g. `src/openhuman/channels/tests/`) | Pure domain logic, schemas, RPC handler shape, in-memory state machines | `cargo test` |
|
||||
| **Rust integration** | `tests/*.rs` at repo root | Full domain wiring with real Tokio runtime, mock external services, JSON-RPC end-to-end (`tests/json_rpc_e2e.rs`), domain × domain interactions | `pnpm test:rust` (which calls `bash scripts/test-rust-with-mock.sh`) |
|
||||
| **Vitest unit** | Co-located as `*.test.ts(x)` next to source under `app/src/**`, or under `app/src/**/__tests__/` | React components, hooks, store slices, pure utilities, service-layer adapters | `pnpm test:unit` |
|
||||
| **WDIO E2E** | `app/test/e2e/specs/*.spec.ts` | Full desktop flow: UI → Tauri → in-process core → JSON-RPC; user-visible behaviour | All platforms: Appium Chromium driver (port 4723) against the CEF runtime. See [E2E Testing](e2e-testing.md). |
|
||||
| **Manual smoke** | [`docs/RELEASE-MANUAL-SMOKE.md`](../../docs/RELEASE-MANUAL-SMOKE.md) | OS-level surfaces drivers cannot assert: TCC permission prompts, Gatekeeper, code signing, DMG install, OS-native toasts | Human at release-cut, signed off in release PR |
|
||||
|
||||
---
|
||||
|
||||
@@ -127,7 +127,7 @@ bash app/scripts/e2e-run-spec.sh test/e2e/specs/<your-spec>.spec.ts <id>
|
||||
|
||||
Some surfaces cannot be driven by WDIO / Appium because they cross OS-level trust boundaries or hardware paths. The complete checklist + sign-off block lives in [`docs/RELEASE-MANUAL-SMOKE.md`](../../docs/RELEASE-MANUAL-SMOKE.md), that file is the source of truth for what must be verified per release. Examples of what it covers:
|
||||
|
||||
- macOS TCC permission prompts (Accessibility, Input Monitoring, Screen Recording, Microphone)
|
||||
- macOS TCC permission prompts (Accessibility, Input Monitoring, Microphone)
|
||||
- Gatekeeper signature validation on first launch
|
||||
- Code-sign integrity (`codesign --verify --deep --strict`)
|
||||
- DMG install / drag-to-Applications flow
|
||||
|
||||
Reference in New Issue
Block a user