OpenJarvis/instr.md

# Agent Runtime Manual Test Plan

**Branch:** `main`
**PR Reference:** [#32](https://github.com/open-jarvis/OpenJarvis/pull/32)

---

## Setup

```bash
git checkout main && git pull
uv sync --extra dev
```

Create `~/.openjarvis/config.toml`:

```toml
[engine]
type = "cloud"

[intelligence]
default_model = "Qwen/Qwen3.5-35B-A3B"

[engine.cloud]
provider = "openai"
api_key = "sk-..."
```

For every test case, record: **Pass / Fail / Partial / Blocked**, what you actually saw, and screenshots for any UI issues.

---

## Part 1: CLI (`jarvis agents`)

### 1.1 Commands exist

| # | Test | Expected |
|---|------|----------|
| 1 | `jarvis agents --help` | Shows all subcommands: `launch`, `start`, `stop`, `run`, `status`, `logs`, `daemon`, `watch`, `recover`, `errors`, `ask`, `instruct`, `messages`, `list`, `info`, `create`, `pause`, `resume`, `delete`, `bind`, `channels`, `search`, `templates`, `tasks` |

### 1.2 Agent lifecycle: create → run → pause → resume → delete

| # | Test | Expected |
|---|------|----------|
| 2 | `jarvis agents launch` | Wizard: template list → name/schedule/tools/budget/learning prompts → creates agent, prints ID |
| 3 | `jarvis agents list` | Agent appears, status=`idle` |
| 4 | `jarvis agents status` | Table: name, status dot, schedule, last run, runs=0, cost=$0 |
| 5 | `jarvis agents run <id>` | Prints progress then "Tick complete. Status: idle, runs: 1" |
| 6 | `jarvis agents status` | runs=1, last run time updated |
| 7 | `jarvis agents pause <id>` then `status` | Status shows `paused` |
| 8 | `jarvis agents resume <id>` then `status` | Status back to `idle` |
| 9 | `jarvis agents delete <id>` then `list` | Agent gone (soft-deleted/archived, not in list) |

### 1.3 Agent creation variants

| # | Test | Expected |
|---|------|----------|
| 10 | `jarvis agents create "Test Agent"` | Creates agent by name, prints ID |
| 11 | `jarvis agents create --template <template_name>` | Creates from template, inherits template config |
| 12 | `jarvis agents launch` → pick a template | Wizard pre-fills config from template |
| 13 | `jarvis agents launch` → pick "Custom Agent" | Wizard starts with blank config |
| 14 | `jarvis agents templates` | Lists built-in + user templates with descriptions |

### 1.4 Scheduling

| # | Test | Expected |
|---|------|----------|
| 15 | Create agent with `schedule_type=interval`, `schedule_value=30` | Created |
| 16 | `jarvis agents start <id>` | "Agent registered with scheduler" |
| 17 | `jarvis agents stop <id>` | "Agent deregistered from scheduler" |
| 18 | `jarvis agents daemon` | Starts, prints agent count, blocks. Ctrl+C → "Daemon stopped." clean exit |
| 19 | Create agent with `schedule_type=cron`, `schedule_value="*/5 * * * *"` | Created |
| 20 | `jarvis agents start <id>` (cron agent) | Registered, next fire time displayed or logged |
| 21 | Create agent with `schedule_type=manual` then `start <id>` | Agent registered but never auto-fires |

### 1.5 Interaction: ask / instruct / messages

| # | Test | Expected |
|---|------|----------|
| 22 | `jarvis agents ask <id> "What is 2+2?"` | Runs tick, prints agent response inline |
| 23 | `jarvis agents messages <id>` | Shows user→agent ask + agent→user response |
| 24 | `jarvis agents instruct <id> "Focus on ML papers"` | "Instruction queued for next tick" |
| 25 | `jarvis agents messages <id>` | Queued instruction shows `[queued]`, status=pending |
| 26 | `jarvis agents run <id>` then `messages <id>` | Queued message now delivered, status changes from pending |
| 27 | `jarvis agents ask <id> ""` (empty message) | Graceful error or rejection, no crash |
| 28 | `jarvis agents instruct <id>` with very long message (>1000 chars) | Accepted and stored correctly |

### 1.6 Error recovery & monitoring

| # | Test | Expected |
|---|------|----------|
| 29 | `jarvis agents errors` | Lists agents in error/needs_attention/stalled/budget_exceeded (or empty table) |
| 30 | `jarvis agents recover <id>` (on errored agent) | Restores checkpoint, status → `idle` |
| 31 | `jarvis agents recover <id>` (on idle agent) | Clear message: "Agent is not in error state" or similar |
| 32 | `jarvis agents logs <id>` | Recent traces with tick IDs and timestamps |
| 33 | `jarvis agents logs <nonexistent_id>` | Clear error: "Agent not found" |
| 34 | `jarvis agents watch` (then run a tick in another terminal) | Events stream live: AGENT_TICK_START, AGENT_TICK_END visible. Ctrl+C to stop. |
| 35 | `jarvis agents watch <id>` | Same, filtered to one agent only |
| 36 | `jarvis agents watch` then Ctrl+C | Clean exit, no traceback, no hanging threads |

### 1.7 Agent info & inspection

| # | Test | Expected |
|---|------|----------|
| 37 | `jarvis agents info <id>` | Shows agent type, status, memory snippet, tasks, channels, config details |
| 38 | `jarvis agents tasks <id>` | Lists tasks with statuses (or empty state) |
| 39 | `jarvis agents channels <id>` | Lists channel bindings (or empty state) |
| 40 | `jarvis agents search "keyword"` | Searches across agent traces, returns relevant results |

### 1.8 Edge cases & invalid input

| # | Test | Expected |
|---|------|----------|
| 41 | `jarvis agents run <nonexistent_id>` | Clear error: "Agent not found" — no Python traceback |
| 42 | `jarvis agents pause <id>` twice | Second pause is no-op or clear message, no crash |
| 43 | `jarvis agents resume <id>` (already idle) | No-op or clear message, no crash |
| 44 | `jarvis agents run <id>` while another tick is running | Concurrency guard: "Agent is already running" error |
| 45 | `jarvis agents delete <id>` then `run <id>` | Clear error about deleted/archived agent |
| 46 | Create agent with invalid cron expression | Rejected with clear validation error |
| 47 | Create agent with negative budget | Rejected or clamped to 0 |

### 1.9 CLI aesthetics

| # | Check | Expected |
|---|-------|----------|
| 48 | `status` table formatting | Columns aligned, readable at 80-char terminal width |
| 49 | Error messages (run with no engine configured) | Clear human-readable message, no Python tracebacks |
| 50 | `launch` wizard prompts | Clear labels, sensible defaults, no confusing jargon |
| 51 | `watch` event stream | Color-coded, event type + agent name visible, timestamps |
| 52 | `list` table with 0 agents | "No agents found" or empty table — not a crash |
| 53 | `list` table with 10+ agents | Table remains readable, no column overflow |
| 54 | All commands with `--help` | Every subcommand has a help string |

---

## Part 2: Web Frontend

### 2.0 Setup

```bash
# Terminal 1                    # Terminal 2
uv run jarvis serve             cd frontend && npm install && npm run dev
```

Open http://localhost:5173, navigate to **Agents** page via sidebar.

### 2.1 Navigation & routing

| # | Test | Expected |
|---|------|----------|
| 55 | Click "Agents" in sidebar | AgentsPage renders, URL is `/agents` |
| 56 | Direct navigation to `/agents` | Page loads correctly (no blank screen) |
| 57 | Browser back/forward after visiting agent detail | Navigation works, state preserved |

### 2.2 List view

| # | Test | Expected |
|---|------|----------|
| 58 | Page loads with backend running | No console errors, agent list renders |
| 59 | Page loads with backend **down** | User-visible error message (not blank white screen), no console exceptions |
| 60 | Agent cards | Name, color status dot, schedule description, last run time, runs count, cost |
| 61 | "Run Now" button | Triggers tick, card updates (runs count increments, last run time updates) |
| 62 | Pause/Resume button | Toggles status, dot color changes immediately |
| 63 | Agent list auto-refresh | After running a tick via CLI, the web list eventually reflects the updated state |
| 64 | 10+ agents in list | Cards render without performance issues, scroll works |

### 2.3 Launch wizard

| # | Test | Expected |
|---|------|----------|
| 65 | Click "Launch Agent" | Modal appears: Step 1 template picker with templates + "Custom Agent" option |
| 66 | Templates load from API | Template cards display with names and descriptions |
| 67 | Select template → Next → Step 2 | Config form: name (pre-filled from template), schedule_type dropdown, schedule_value, tools checkboxes, budget, learning toggle (off) |
| 68 | Select "Custom Agent" → Next → Step 2 | Config form with blank name, no pre-filled values |
| 69 | Next → Step 3 | Review summary of all config values |
| 70 | Click Launch | Agent created, modal closes, new agent appears in list |
| 71 | Back button at Step 2 | Returns to Step 1, template selection preserved |
| 72 | Back button at Step 3 | Returns to Step 2, all form inputs preserved |
| 73 | Launch with empty name | Inline error: "Agent name is required" — modal stays open |
| 74 | Launch with all tools selected | All tools included in review and in created agent config |
| 75 | Click outside modal / press Escape | Modal closes (or stays open — document behavior) |
| 76 | Schedule type = "Manual" | schedule_value input is disabled/hidden |
| 77 | Schedule type = "Cron" | schedule_value placeholder shows cron example |
| 78 | Schedule type = "Interval" | schedule_value placeholder shows seconds example |

### 2.4 Detail view (click an agent)

| # | Test | Expected |
|---|------|----------|
| 79 | Click agent card | Detail view opens with tabbed interface |
| 80 | **Overview** tab | Stat cards (Total Runs, Success Rate, Total Cost), config display, channels list, action buttons |
| 81 | **Overview** action buttons | Run Now, Pause, Resume visible and functional |
| 82 | **Interact** tab | Chat message list, textarea, "Immediate" and "Queue" send buttons |
| 83 | Send immediate message | Appears in chat with user styling, agent responds after tick |
| 84 | Send queued message | Shows with "queued" badge, status=pending |
| 85 | Send empty message | Button disabled or graceful rejection — no empty message sent |
| 86 | Rapid-fire send (click Send multiple times quickly) | No duplicate messages, no race condition errors |
| 87 | Chat auto-scroll | New messages scroll into view automatically |
| 88 | **Tasks** tab | Task list with status badges (completed=green, failed=red, active=blue, pending=gray) |
| 89 | **Tasks** tab (no tasks) | Empty state: "No tasks assigned." |
| 90 | **Memory** tab | summary_memory text displayed in readable format |
| 91 | **Memory** tab (no memory) | Empty state: "Agent has no stored memory yet." |
| 92 | **Learning** tab | Toggle switch (read-only, off by default), placeholder text for future events |
| 93 | **Logs** tab | Placeholder / empty state message (not a crash or blank) |
| 94 | Tab switching — rapid clicks | All 6 tabs render instantly, no layout shift, no flash of wrong content |

### 2.5 Error states

| # | Test | Expected |
|---|------|----------|
| 95 | Agent in `error` status | Red status dot/badge, "Recover" button visible |
| 96 | Click Recover | Status resets to `idle`, dot turns green |
| 97 | Agent in `needs_attention` status | Amber badge visible |
| 98 | Agent in `budget_exceeded` status | Orange badge visible |
| 99 | Agent in `stalled` status | Yellow badge visible |
| 100 | Backend goes down while page is open | Next refresh/action shows error — not silent failure |
| 101 | Delete agent → confirm it disappears from list | Agent removed from list immediately (or on next refresh) |
| 102 | Delete agent (no confirmation dialog in web) | **Document:** Is instant delete OK or should there be a confirm? |

### 2.6 Overflow menu

| # | Test | Expected |
|---|------|----------|
| 103 | Click "..." menu on agent card | Dropdown with Delete + other options |
| 104 | Click Delete from menu | Agent deleted, list updates |
| 105 | Click outside dropdown | Dropdown closes |

### 2.7 Web aesthetics & UX

| # | Check | Expected |
|---|-------|----------|
| 106 | Status dot colors | idle=#22c55e, running=#3b82f6, paused=#6b7280, error=#ef4444, needs_attention=#f59e0b, budget_exceeded=#f97316, stalled=#eab308 |
| 107 | Launch wizard spacing/alignment | Modal centered, steps clearly numbered, form inputs aligned, no overlap |
| 108 | Detail view tab switching | Instant, no layout shift or flash |
| 109 | Interact tab chat feel | Messages visually distinct (user=right vs agent=left or different colors), auto-scroll, clear input area |
| 110 | Responsive at 1024px width | No overflow or cut-off content, agent cards reflow |
| 111 | Responsive at 1440px width | Proper use of space, no excessive stretching |
| 112 | Responsive at 768px width (tablet) | Still usable, no broken layout |
| 113 | Empty states | "No agents yet" + CTA button / "No messages" / "No tasks" — not blank white space |
| 114 | Loading states | "Loading agents..." shown during fetch, spinner or skeleton |
| 115 | Page title / browser tab | Meaningful title (not just "Vite App") |
| 116 | Console errors | Zero console errors during normal usage flow |

---

## Part 3: Desktop App

### 3.0 Setup

```bash
# Terminal 1                    # Terminal 2
uv run jarvis serve             cd desktop && npm install && npm run tauri dev
```

Navigate to the **Agents** tab.

### 3.1 Functionality

| # | Test | Expected |
|---|------|----------|
| 117 | Left panel: agent list | Status dots, schedule descriptions, last run times |
| 118 | Click agent → right panel | Tabbed detail view (Overview, Interact, Tasks, Memory, Learning, Logs) |
| 119 | No agent selected | Right panel shows "Select an agent to view details" |
| 120 | "Launch Agent" button | Opens wizard, same 3-step flow as web |
| 121 | Launch wizard → Create agent | Agent appears in left panel list |
| 122 | **Overview** tab | Key-value stats (Status, Agent Type, Schedule, Last Run, Total Runs, Total Cost, Budget) + action buttons (Run Now, Pause, Resume, Recover) |
| 123 | **Interact** tab | Chat UI, mode toggle (immediate/queued), Enter shortcut sends message |
| 124 | Send immediate message | Response appears in chat |
| 125 | Send queued message | Shows as pending |
| 126 | **Tasks** tab | Task list with colored status badges + created-at timestamps |
| 127 | **Memory** tab | summary_memory in monospace font |
| 128 | **Learning** tab | Shows enabled/disabled status + placeholder text |
| 129 | **Logs** tab | Placeholder: "Log streaming not yet connected." |
| 130 | Auto-refresh | Agent list refreshes on ~10s interval (verify with CLI-triggered state change) |
| 131 | Delete agent via desktop | Confirmation dialog appears, agent removed on confirm |

### 3.2 Desktop edge cases

| # | Test | Expected |
|---|------|----------|
| 132 | Backend not running → open desktop app | Error state shown, not a crash |
| 133 | Backend dies while desktop is open | Graceful degradation on next action/refresh |
| 134 | Selected agent deleted via CLI → desktop refreshes | Selected agent deselects, list updates |

### 3.3 Desktop aesthetics

| # | Check | Expected |
|---|-------|----------|
| 135 | Catppuccin color scheme consistent | idle=#a6e3a1, running=#89b4fa, paused=#6c7086, error=#f38ba8, needs_attention=#fab387, stalled=#f9e2af |
| 136 | Left/right panel split | Resizable or fixed at reasonable ratio, no overlap |
| 137 | Tab switching | Smooth, no flicker |
| 138 | Launch wizard modal | Properly overlays content, dismissible with Escape or outside click |
| 139 | Text readability | Font sizes consistent, sufficient contrast against dark background |
| 140 | Window resize | Layout adapts, no overflow or clipping |
| 141 | Status badge consistency with web | Same statuses map to same semantic colors (green=idle, blue=running, etc.) |

---

## Part 4: API Backend (Direct)

### 4.1 REST endpoint smoke tests

Run with `uv run jarvis serve` and test via curl or Postman.

| # | Test | Expected |
|---|------|----------|
| 142 | `GET /v1/managed-agents` | 200, returns `[]` or agent list JSON |
| 143 | `POST /v1/managed-agents` with valid body | 200/201, returns created agent JSON with `id` |
| 144 | `POST /v1/managed-agents` with empty body | 422 or 400 with validation error |
| 145 | `GET /v1/managed-agents/<id>` | 200, returns single agent |
| 146 | `GET /v1/managed-agents/<bad_id>` | 404, returns error JSON |
| 147 | `POST /v1/managed-agents/<id>/run` | 200, tick executes |
| 148 | `POST /v1/managed-agents/<id>/pause` | 200, status changes to paused |
| 149 | `POST /v1/managed-agents/<id>/resume` | 200, status changes to idle |
| 150 | `POST /v1/managed-agents/<id>/recover` | 200 if errored, appropriate error if not |
| 151 | `DELETE /v1/managed-agents/<id>` | 200, agent archived |
| 152 | `GET /v1/templates` | 200, returns template list |
| 153 | `POST /v1/templates/<id>/instantiate` | 200, creates agent from template |
| 154 | `GET /v1/agents/errors` | 200, returns list of problem agents |
| 155 | `GET /v1/agents/health` | 200, returns health summary |

### 4.2 Message endpoints

| # | Test | Expected |
|---|------|----------|
| 156 | `POST /v1/managed-agents/<id>/messages` with `{"content":"hi","direction":"user_to_agent","mode":"immediate"}` | 200, message stored |
| 157 | `GET /v1/managed-agents/<id>/messages` | 200, returns message list |
| 158 | `POST /v1/managed-agents/<id>/messages` with `{"content":"","direction":"user_to_agent","mode":"immediate"}` | 422 or graceful handling |
| 159 | `POST /v1/managed-agents/<id>/messages` with `{"content":"cmd","direction":"user_to_agent","mode":"queued"}` | 200, message has status=pending |

### 4.3 Task & channel endpoints

| # | Test | Expected |
|---|------|----------|
| 160 | `GET /v1/managed-agents/<id>/tasks` | 200, returns task list |
| 161 | `POST /v1/managed-agents/<id>/tasks` | 200, creates task |
| 162 | `GET /v1/managed-agents/<id>/channels` | 200, returns channel bindings |
| 163 | `GET /v1/managed-agents/<id>/state` | 200, returns full agent state |

### 4.4 WebSocket events

| # | Test | Expected |
|---|------|----------|
| 164 | Connect to `ws://localhost:8222/v1/agents/events` | Connection established |
| 165 | Trigger a tick → observe WS messages | Receive AGENT_TICK_START and AGENT_TICK_END events |
| 166 | Connect with `?agent_id=<id>` filter | Only events for that agent |
| 167 | Disconnect cleanly | No server error logs |

---

## Part 5: Cross-Platform Consistency

| # | Test | Expected |
|---|------|----------|
| 168 | Create agent via CLI → check web + desktop | Same name, status, config everywhere |
| 169 | Run tick via CLI → check web + desktop | Run count and last run time update in both UIs |
| 170 | Send message via web Interact → check CLI `messages` | Same content, direction, mode |
| 171 | Pause via desktop → check CLI `status` + web | `paused` everywhere |
| 172 | Delete via web → check CLI `list` + desktop | Gone everywhere |
| 173 | Create via web wizard → check CLI `list` + desktop | Agent visible in all three |
| 174 | Recover via CLI → check web + desktop | Status back to idle in all UIs |
| 175 | Send queued message via CLI `instruct` → check web Interact | Message shows with pending/queued status |
| 176 | Multiple agents created from different clients | All agents appear correctly in all views |

---

## Part 6: Stress & Concurrency

| # | Test | Expected |
|---|------|----------|
| 177 | Run tick on same agent from two terminals simultaneously | Concurrency guard blocks second tick: "Agent is already running" |
| 178 | Create 20+ agents → check list performance | All clients render list without lag |
| 179 | Rapidly pause/resume same agent | All state transitions correct, no stuck states |
| 180 | Run daemon + manual `run` at same time | No double-ticking, concurrency guard holds |
| 181 | Delete agent while tick is in progress | Tick completes or fails gracefully, agent ends up archived |

---

## Part 7: Deferred Features (Placeholder Verification)

Confirm these show placeholders (not crashes):

| # | Feature | CLI | Web | Desktop |
|---|---------|-----|-----|---------|
| 182 | Budget enforcement | `run` still works even if cost > budget | No enforcement, budget is display-only | Same |
| 183 | Stall detection | No automatic stall detection fires | N/A | N/A |
| 184 | Learning event timeline | `Learning` tab shows placeholder text | Same | Same |
| 185 | Logs trace replay | `Logs` tab shows placeholder text | Same | Same |
| 186 | `POST /v1/skills` | N/A | N/A | Returns `"not_implemented"` |
| 187 | `POST /v1/optimize/runs` | N/A | N/A | Returns placeholder `run_id` |
| 188 | `GET /v1/feedback/stats` | N/A | N/A | Returns `{total: 0, mean_score: 0.0}` |

---

## Deliverables

**1. Test results** — Spreadsheet with columns: #, Status (Pass/Fail/Partial/Blocked), Actual Behavior, Screenshot (for UI issues).

**2. Bug list** — Each bug: steps to reproduce, expected vs actual, severity (Critical/Major/Minor), screenshot.

**3. UX & aesthetics feedback** — Is the launch wizard clear? Are status colors distinguishable? Does the Interact tab feel like chat? Is CLI output readable? Are error messages helpful? Is the delete-without-confirm behavior in web acceptable?

**4. API error handling audit** — Document all cases where the frontend silently swallows errors (currently: agent list fetch, interact tab sends). Recommend which should show user-visible errors.

**5. Deferred features check** — Confirm placeholder items in Part 7 show graceful stubs (not crashes or blank screens).

---

## Notes

- Backend (`jarvis serve`) must be running for web and desktop (default port 8222).
- Without an engine configured, `run`/`ask` will error — document whether the error message is clear.
- `daemon` and `watch` block — Ctrl+C to exit.
- Web frontend API client has unused functions (`updateManagedAgent`, `createAgentTask`, `fetchAgentState`, `fetchErrorAgents`) — not a bug, but note for future.
- Desktop API client is missing some endpoints that the web client has (`fetchAgentChannels`, `fetchAgentState`, `fetchErrorAgents`) — may affect feature parity.
- Frontend has **no automated tests** — all testing is manual per this plan.
- Both frontends silently catch API errors (`.catch(() => {})`) — this is a known UX gap to evaluate.