openhuman/docs/e2e-status.md

E2E Test Suite Status

Living tracking document for the OpenHuman E2E test suite. Updated whenever specs are added, fixed, or start failing.

Last updated: 2026-05-20 Total specs: 66 (11 categories) Runner: WDIO + Appium Chromium on the CEF desktop binary

---

Suite health overview

Category	Specs	Known issues
auth	6	Hardcoded pauses replaced with condition waits (2026-05-20)
navigation	6	channels-smoke and insights-dashboard are shallow/smoke only
chat	10	chat-harness-wallet-flow has 6 sequential 30s waits
skills	6	skill-execution-flow is RC-7 (ghost RPCs); 4 specs are shallow stubs
notifications	4	memory-roundtrip has async indexing race
webhooks	5	webhooks-ingress-flow missing payload delivery assertion
providers	8	telegram-flow is describe.skip; gmail/slack/whatsapp miss multi-account
payments	4	rewards-progression-persistence has hardcoded pauses
settings	7	settings-ai-skills uses OR-chain assertions
system	4+1L	local-model-runtime is describe.skip; voice-mode has hardcoded pauses
journeys	3	All moderate depth

L = Linux-only spec

---

How to update this document

• Adding a spec: add it to the coverage matrix below and to e2e-run-all-flows.sh
• Fixing an issue: strike through the entry or remove it from Known Issues
• A spec starts failing: add it to the Known Issues section with severity + status tag
• Pre-flight check: bash app/scripts/e2e-preflight.sh

---

Coverage matrix

Auth (6 specs)

Spec	Feature covered	Coverage depth	Known issues
smoke.spec.ts	Harness bootstrap, app loads	deep
login-flow.spec.ts	Deep-link auth → onboarding → home	deep
auth-access-control.spec.ts	Billing dashboard handoff	moderate	Previously had hardcoded 5s/8s pauses — replaced 2026-05-20
logout-relogin-onboarding.spec.ts	Logout + re-login round-trip	moderate
onboarding-modes.spec.ts	Onboarding step sequence	moderate	config.toml write race on slow CI
runtime-picker-login.spec.ts	Core mode selection + login	moderate	Deep-link bootstrap race

Navigation (6 specs)

Spec	Feature covered	Coverage depth	Known issues
navigation.spec.ts	Tab bar + route rendering	deep
navigation-smoothness.spec.ts	Transition timing	moderate
navigation-settings-panels.spec.ts	Settings panel routing	moderate
command-palette.spec.ts	Command search	moderate
channels-smoke.spec.ts	Channels surface mount	shallow	No channel feature validation
insights-dashboard.spec.ts	Insights panel	shallow	No data validation

Chat (10 specs)

Spec	Feature covered	Coverage depth	Known issues
chat-harness-send-stream.spec.ts	Send → SSE stream → UI render	deep
chat-harness-cancel.spec.ts	Cancel mid-stream	deep
chat-harness-scroll-render.spec.ts	Scroll + render correctness	moderate
chat-harness-subagent.spec.ts	Subagent invocation	moderate
chat-harness-wallet-flow.spec.ts	Chat + wallet state	moderate	6 sequential 30s waits; should use condition waits
chat-tool-call-flow.spec.ts	Function calling roundtrip	deep
chat-multi-tool-round.spec.ts	Multi-turn tool loop	deep
chat-tool-error-recovery.spec.ts	Tool error handling	deep
agent-review.spec.ts	Agent review + feedback	moderate
mega-flow.spec.ts	Full journey (auth/oauth/chat/logout)	deep

Skills (6 specs)

Spec	Feature covered	Coverage depth	Known issues
skills-registry.spec.ts	Install from URL	moderate	Post-install state not verified
skill-execution-flow.spec.ts	Ghost RPCs (RC-7)	skipped	[RC-7 OPEN] Runtime removed; spec calls non-existent RPC methods
skill-lifecycle.spec.ts	/skills page loads	shallow	No feature validation beyond page mount
skill-multi-round.spec.ts	/chat page loads	shallow	No multi-round skill behavior tested
skill-oauth.spec.ts	/skills page loads	shallow	No OAuth flow tested
skill-socket-reconnect.spec.ts	Home page loads	shallow	No socket reconnect behavior tested

Notifications (4 specs)

Spec	Feature covered	Coverage depth	Known issues
notifications.spec.ts	Ingest + list + mark-read + UI	deep
memory-roundtrip.spec.ts	Doc store + cross-namespace recall	moderate	Async indexing race on slow CI
cron-jobs-flow.spec.ts	Job creation UI	moderate
autocomplete-flow.spec.ts	Chat autocomplete	shallow

Webhooks & Tools (5 specs)

Spec	Feature covered	Coverage depth	Known issues
webhooks-ingress-flow.spec.ts	RPC endpoints + debug panel	moderate	No actual payload delivery assertion
webhooks-tunnel-flow.spec.ts	Tunneling	moderate
tool-browser-flow.spec.ts	Browser tool	moderate
tool-filesystem-flow.spec.ts	Filesystem security	deep
tool-shell-git-flow.spec.ts	Shell + git	moderate

Providers (8 specs)

Spec	Feature covered	Coverage depth	Known issues
telegram-flow.spec.ts	Telegram integration	skipped	[SKIPPED OPEN] describe.skip — no replacement spec
gmail-flow.spec.ts	Gmail OAuth	moderate	Token refresh path untested
accounts-provider-modal.spec.ts	Account connection modal	moderate
slack-flow.spec.ts	Slack OAuth + Redux state	moderate	Multi-account scenario untested
whatsapp-flow.spec.ts	WhatsApp OAuth + state	moderate	Multi-account scenario untested
notion-flow.spec.ts	Notion OAuth	moderate	Scope upgrade path untested
conversations-web-channel-flow.spec.ts	Web channel messaging	moderate	Linux skip reason is stale
composio-triggers-flow.spec.ts	Trigger enable/disable + UI	moderate	No trigger event delivery tested

Payments (4 specs)

Spec	Feature covered	Coverage depth	Known issues
card-payment-flow.spec.ts	Card payment + error handling	moderate
crypto-payment-flow.spec.ts	Crypto payment	moderate
rewards-unlock-flow.spec.ts	Rewards unlock	moderate
rewards-progression-persistence.spec.ts	Rewards persistence	moderate	Hardcoded pauses; should use condition waits

Settings (7 specs)

Spec	Feature covered	Coverage depth	Known issues
settings-channels-permissions.spec.ts	Channels + privacy settings	moderate
settings-data-management.spec.ts	Data management	moderate
settings-dev-options.spec.ts	Developer options	moderate
settings-ai-skills.spec.ts	LLM config	shallow	OR-chain assertions (passes if any one LLM panel is present)
settings-account-preferences.spec.ts	Account preferences	moderate
settings-advanced-config.spec.ts	Advanced config	moderate
settings-feature-preferences.spec.ts	Feature toggles	moderate

System (6 specs + 1 Linux-only)

Spec	Feature covered	Coverage depth	Known issues
local-model-runtime.spec.ts	Ollama integration	skipped	[SKIPPED OPEN] describe.skip
voice-mode.spec.ts	Voice I/O	shallow	Hardcoded pauses
screen-intelligence.spec.ts	Screen awareness	shallow
audio-toolkit-flow.spec.ts	Audio toolkit	shallow
tauri-commands.spec.ts	Tauri IPC surface	moderate
service-connectivity-flow.spec.ts	Service discovery	moderate	Requires OPENHUMAN_SERVICE_MOCK=1
linux-cef-deb-runtime.spec.ts	Linux /usr/bin path	moderate	Linux only

User Journeys (3 specs)

Spec	Feature covered	Coverage depth	Known issues
user-journey-full-task.spec.ts	Task completion end-to-end	moderate
user-journey-settings-round-trip.spec.ts	Settings persistence round-trip	moderate
chat-conversation-history.spec.ts	Conversation history	moderate

---

Known Issues

ID	Spec	Severity	Status	Description
RC-7	skill-execution-flow.spec.ts	HIGH	[RC-7 OPEN]	Calls RPC methods that were removed when the QuickJS runtime was stripped. Spec will ghost-fail silently until updated or deleted.
SKIP-1	telegram-flow.spec.ts	MEDIUM	[SKIPPED OPEN]	Entire suite is describe.skip. No replacement coverage.
SKIP-2	local-model-runtime.spec.ts	LOW	[SKIPPED OPEN]	Entire suite is describe.skip. Ollama is optional — acceptable.
RACE-1	memory-roundtrip.spec.ts	LOW	[RACE]	Async indexing race on slow CI machines. Intermittent.
RACE-2	onboarding-modes.spec.ts	LOW	[RACE]	config.toml write race during core restart. Intermittent.
SHALLOW-1	skill-lifecycle.spec.ts	MEDIUM	[SHALLOW]	Only asserts page mount, not any skill lifecycle behavior.
SHALLOW-2	skill-multi-round.spec.ts	MEDIUM	[SHALLOW]	Only asserts /chat page loads.
SHALLOW-3	skill-oauth.spec.ts	MEDIUM	[SHALLOW]	Only asserts /skills page loads. No OAuth.
SHALLOW-4	skill-socket-reconnect.spec.ts	MEDIUM	[SHALLOW]	Only asserts home page loads. No socket reconnect.
PAUSE-1	chat-harness-wallet-flow.spec.ts	LOW	[PAUSE]	Six sequential browser.pause(30_000) calls. Should be replaced with condition waits.
PAUSE-2	rewards-progression-persistence.spec.ts	LOW	[PAUSE]	Hardcoded pauses. Should be replaced with condition waits.
PAUSE-3	voice-mode.spec.ts	LOW	[PAUSE]	Hardcoded pauses in voice I/O flow.
STALE-1	conversations-web-channel-flow.spec.ts	LOW	[STALE]	Linux skip condition uses a reason that no longer applies.
ASSERT-1	settings-ai-skills.spec.ts	LOW	[SHALLOW]	OR-chain assertions: passes if any one LLM provider panel is present.

---

Mock API behavior flags

These flags are set via setMockBehavior(key, value) from mock-server.ts and control the shared mock backend at http://127.0.0.1:18473.

Flag	Type	Description
seed	string	Fuzzy randomization seed for mock data generation
forceError503	'true' / 'false'	Force HTTP 503 on all non-admin endpoints
llmStreamScript	JSON string	Custom LLM response delta sequence. Array of {delta: string} objects
composioConnections	JSON string	Override Composio connections list (e.g. '[]' for empty)
composioAvailableTriggers	JSON string	Override available triggers returned by the API
composioActiveTriggers	JSON string	Override active triggers state
purchaseError	string	Trigger payment failure (value becomes the error message)
plan	'FREE' / 'BASIC' / 'PRO'	Override the billing plan returned by /settings
planActive	'true' / 'false'	Override whether the plan is active
planExpiry	ISO date string	Override the plan expiry date
session	'revoked' / 'active'	Force 401 on auth endpoints when set to 'revoked'

Reset all flags to defaults: resetMockBehavior().

---

How to run

# Full suite (all 66 specs)
bash app/scripts/e2e-run-all-flows.sh

# Single suite category
bash app/scripts/e2e-run-all-flows.sh --suite chat

# Stop after first failure
bash app/scripts/e2e-run-all-flows.sh --bail

# Single spec (fastest iteration)
bash app/scripts/e2e-run-session.sh test/e2e/specs/smoke.spec.ts smoke

# Pre-flight check only
bash app/scripts/e2e-preflight.sh

# With Appium/WDIO debug output
WDIO_LOG_LEVEL=debug bash app/scripts/e2e-run-all-flows.sh --suite auth

# Skip preflight (e.g. in CI where it ran as a separate step)
bash app/scripts/e2e-run-all-flows.sh --skip-preflight

# Use the debug runner (summary output + log tee)
pnpm debug e2e test/e2e/specs/smoke.spec.ts
pnpm debug e2e test/e2e/specs/notifications.spec.ts notifications --verbose

---

How to add a new spec

1. Create the spec file in app/test/e2e/specs/YOUR-SPEC.spec.ts.

2. Scaffold the harness:

   import { resetApp } from '../helpers/reset-app';
   import { startMockServer, stopMockServer } from '../mock-server';
   
   describe('Your feature', () => {
     before(async () => {
       await startMockServer();
       await resetApp('e2e-your-spec');
     });
     after(async () => {
       await stopMockServer();
     });
   
     it('does the thing', async () => { /* ... */ });
   });
   

3. Register in the orchestrator — add a run(...) call in the correct suite section of app/scripts/e2e-run-all-flows.sh.

4. Add to this tracking doc — add a row to the coverage matrix table for the appropriate category with an honest coverage depth.

5. Add any new RPC methods to REQUIRED_RPC_METHODS in app/test/e2e/helpers/rpc-preflight.ts if the spec calls RPC methods not already listed there.

6. Run pre-flight before executing: bash app/scripts/e2e-preflight.sh.