openhuman/gitbooks/developing/agent-observability.md

description, icon

description	icon
Artifact-capture layer that makes E2E tests debuggable. Logs, traces, screenshots.	eye

Agent Observability for E2E

This doc describes the artifact-capture layer that makes the desktop app inspectable by coding agents (Codex, Claude Code, Cursor) through the existing WDIO/Appium/tauri-driver harness.

It is intentionally narrow: one canonical onboarding + privacy flow with on-disk screenshots, page-source dumps, and mock backend request logs. See AGENT_OBSERVABILITY_PLAN.md at the repo root for the broader plan.

TL;DR

bash app/scripts/e2e-agent-review.sh

Artifacts land under:

app/test/e2e/artifacts/<ISO-timestamp>-agent-review/
  01-welcome.png
  01-welcome.source.xml
  02-post-welcome.png
  02-post-welcome.source.xml
  03-post-onboarding.png
  03-post-onboarding.source.xml
  04-privacy-panel.png
  04-privacy-panel.source.xml
  mock-requests-after-welcome.json
  mock-requests-after-onboarding.json
  mock-requests-after-privacy.json
  failure-<test>.png              # only on failure
  failure-<test>.source.xml       # only on failure
  meta.json                       # run metadata + checkpoint index

The script prints the resolved artifact directory at the end.

Pieces

Piece	Path	Role
Helper	app/test/e2e/helpers/artifacts.ts	Run dir, captureCheckpoint, captureFailureArtifacts, saveMockRequestLog
WDIO hook	app/test/wdio.conf.ts (afterTest)	Always dumps screenshot + source on any failing test
Canonical spec	app/test/e2e/specs/agent-review.spec.ts	Welcome → onboarding → privacy panel with named checkpoints
Wrapper script	app/scripts/e2e-agent-review.sh	Build + run + print artifact dir
Stable selectors	data-testid on OnboardingNextButton, Onboarding overlay + skip button, WelcomeStep, PrivacyPanel	Agent-reliable navigation anchors

Environment overrides

Variable	Effect
E2E_ARTIFACT_DIR	Force a specific run dir (skips auto-timestamped name)
E2E_ARTIFACT_ROOT	Parent dir for auto-generated run dirs (default: app/test/e2e/artifacts)
E2E_ARTIFACT_LABEL	Label used in the auto-generated run dir name (default: run; wrapper sets agent-review)

Using the helper from new specs

import {
  captureCheckpoint,
  saveMockRequestLog,
} from '../helpers/artifacts';
import { getRequestLog } from '../mock-server';

await captureCheckpoint('after-connect-click');
saveMockRequestLog('after-connect-click', getRequestLog());

captureCheckpoint numbers captures so the run dir reads chronologically. captureFailureArtifacts is wired into wdio.conf.ts and fires automatically on any failing test, specs should not call it directly.

What is intentionally out of scope

• Visual baselines / image diffs across every component state.
• Screenshot capture on every click (too noisy).
• Live integrations (Gmail, Notion, Telegram); mock server only.
• New test framework / reporter.

Widen to more flows only after this loop proves out.