--- description: How OpenHuman tests its product - Vitest, cargo test, WDIO E2E. Where each test goes. icon: vial --- # Testing Strategy How OpenHuman tests its product. Source of truth for "where does my test go?". Companion to [`TEST-COVERAGE-MATRIX.md`](../../docs/TEST-COVERAGE-MATRIX.md). --- ## Layers | Layer | Where it lives | What it tests | Driver | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | **Rust unit** | `#[cfg(test)] mod tests` inside the same `*.rs` file, or sibling `tests.rs`, or `tests/` subdir under a domain (e.g. `src/openhuman/channels/tests/`) | Pure domain logic, schemas, RPC handler shape, in-memory state machines | `cargo test` | | **Rust integration** | `tests/*.rs` at repo root | Full domain wiring with real Tokio runtime, mock external services, JSON-RPC end-to-end (`tests/json_rpc_e2e.rs`), domain × domain interactions | `pnpm test:rust` (which calls `bash scripts/test-rust-with-mock.sh`) | | **Vitest unit** | Co-located as `*.test.ts(x)` next to source under `app/src/**`, or under `app/src/**/__tests__/` | React components, hooks, store slices, pure utilities, service-layer adapters | `pnpm test:unit` | | **WDIO E2E** | `app/test/e2e/specs/*.spec.ts` | Full desktop flow: UI → Tauri → in-process core → JSON-RPC; user-visible behaviour | All platforms: Appium Chromium driver (port 4723) against the CEF runtime. See [E2E Testing](e2e-testing.md). | | **Manual smoke** | [`docs/RELEASE-MANUAL-SMOKE.md`](../../docs/RELEASE-MANUAL-SMOKE.md) | OS-level surfaces drivers cannot assert: TCC permission prompts, Gatekeeper, code signing, DMG install, OS-native toasts | Human at release-cut, signed off in release PR | --- ## Decision tree - where does my test go? ```text Is the change behind the JSON-RPC boundary (in `src/`)? ├─ YES - does it cross domains or talk to external services? │ ├─ YES → Rust integration (tests/*.rs) │ └─ NO → Rust unit (next to source) └─ NO - change is in `app/` ├─ Is it a pure function, hook, slice, or component in isolation? │ └─ YES → Vitest unit (*.test.tsx co-located) └─ Is it user-visible AND it crosses UI ⇄ Tauri ⇄ sidecar ⇄ JSON-RPC? ├─ YES → WDIO E2E (app/test/e2e/specs/*.spec.ts) └─ Is it OS-level (TCC, Gatekeeper, install, OS toasts)? └─ YES → Manual smoke checklist ``` If a change touches more than one of these, write a test in **each** layer it touches. Don't substitute one for another. --- ## Failure-path requirement Every feature leaf in the coverage matrix must have **at least one failure / edge** assertion in addition to the happy path. Examples: - File-write tool: happy = wrote bytes; failure = path-restriction denial. - OAuth flow: happy = token issued; edge = expired refresh token recovery. - Memory store: happy = stored + recalled; edge = forget-then-recall returns nothing. A spec that asserts only the happy path is incomplete. --- ## Mock policy - **No real network in unit / integration / E2E.** Use the shared mock backend (`scripts/mock-api-core.mjs`, `scripts/mock-api-server.mjs`, `app/test/e2e/mock-server.ts`). - Admin endpoints for tests: `GET /__admin/health`, `POST /__admin/reset`, `POST /__admin/behavior`, `GET /__admin/requests`. - **External services** (Telegram, Slack, Gmail, Notion, Ollama, OpenAI, etc.) are stubbed at the mock backend level; tests assert the request shape via `getRequestLog()`. - The only acceptable exception is a documented release-cut manual smoke step. --- ## Determinism rules - No wall-clock waits, use `waitForApp`, `waitForAppReady`, `waitForWebView` helpers, or explicit element-readiness predicates. - No shared filesystem state, every E2E spec runs inside an isolated `OPENHUMAN_WORKSPACE` (created/cleaned by `app/scripts/e2e-run-spec.sh`). - No order-dependent specs, each spec must pass when run alone. - No reliance on absolute coordinates or animation timing. - Prefer synthesizing keyboard input via `browser.execute(...)` over `browser.keys()` (see `command-palette.spec.ts` for the pattern). --- ## What the existing harness gives you - **Mock backend bootstrapping**: `startMockServer` / `stopMockServer` in `app/test/e2e/mock-server.ts`. - **Auth shortcut**: `triggerAuthDeepLink` / `triggerAuthDeepLinkBypass` in `helpers/deep-link-helpers.ts` skips real OAuth. - **Element helpers**: `clickNativeButton`, `waitForWebView`, `clickToggle` in `helpers/element-helpers.ts`, use these instead of raw `XCUIElementType*` selectors. - **Shared flows**: `completeOnboardingIfVisible`, `navigateViaHash`, `navigateToSkills`, `walkOnboarding` in `helpers/shared-flows.ts`. - **Core RPC from spec**: `callOpenhumanRpc` in `helpers/core-rpc.ts`, drives the sidecar directly when a UI step would be brittle. - **Platform guards**: `isTauriDriver`, `isMac2`, `supportsExecuteScript` in `helpers/platform.ts` (the first two are legacy shims — everything runs on the Appium Chromium driver now). - **Artifact capture on failure**: `captureFailureArtifacts` runs from `wdio.conf.ts`, screenshots + DOM dumps land under `app/test/e2e/artifacts/`. --- ## Naming + structure conventions - WDIO specs: `-flow.spec.ts` for end-to-end product flows; `.spec.ts` for narrower surfaces. - Vitest co-location: prefer `Component.tsx` + `Component.test.tsx` siblings; use `__tests__/` only when grouping multiple related tests. - Rust integration tests: snake_case file matching the surface, `_e2e.rs` for JSON-RPC-driven flows, `_integration.rs` for cross-domain. - Each `describe` / `mod tests` block maps to a feature-list ID range, link the matrix row in a comment if the mapping is non-obvious. --- ## Pre-merge gates Run before opening a PR. CI runs the same set, but local runs are faster: ```bash # Rust core cargo fmt --check cargo check --manifest-path Cargo.toml cargo clippy --manifest-path Cargo.toml -- -D warnings cargo test --manifest-path Cargo.toml # Tauri shell cargo check --manifest-path app/src-tauri/Cargo.toml # Frontend pnpm typecheck pnpm lint pnpm format:check pnpm test:unit # Rust integration with mock backend pnpm test:rust # E2E (slow - run when behaviour changes user-visibly) pnpm test:e2e:build bash app/scripts/e2e-run-spec.sh test/e2e/specs/.spec.ts ``` --- ## Not driver-automatable - manual smoke required Some surfaces cannot be driven by WDIO / Appium because they cross OS-level trust boundaries or hardware paths. The complete checklist + sign-off block lives in [`docs/RELEASE-MANUAL-SMOKE.md`](../../docs/RELEASE-MANUAL-SMOKE.md), that file is the source of truth for what must be verified per release. Examples of what it covers: - macOS TCC permission prompts (Accessibility, Input Monitoring, Microphone) - Gatekeeper signature validation on first launch - Code-sign integrity (`codesign --verify --deep --strict`) - DMG install / drag-to-Applications flow - Auto-update download + relaunch - OS-native notification toasts on Linux (no display server visible to the driver beyond Xvfb) If a feature has no automated coverage AND is not on the manual smoke list, treat it as untested, open a coverage gap. --- ## Coverage matrix as the contract Every feature leaf in the [coverage matrix](../../docs/TEST-COVERAGE-MATRIX.md) maps to: 1. A test path or paths, **or** 2. A justified `🚫` with a manual-smoke entry. When you add / remove / rename a feature, **update the matrix row in the same PR**. CI will guard this contract once #965 lands. --- ## When in doubt - Push the test as low in the layer stack as possible (Rust unit > Rust integration > Vitest > WDIO). Lower layers are faster, more deterministic, and cheaper to run. - WDIO is for behaviours that genuinely cross UI ⇄ Tauri ⇄ sidecar ⇄ JSON-RPC. Don't drive a unit-testable concern through WDIO just because the UI exists. - A failing happy path is a regression. A missing failure-path test is a gap. Both are bugs.