mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
docs: tinyagents vendor-improvement plan — harness/crate audits + workstreams V1–V6 (#4493)
This commit is contained in:
Steven Enamakel
@@ -1,31 +0,0 @@
|
||||
# Delegation Policy
|
||||
|
||||
## When to delegate vs. act directly
|
||||
|
||||
The orchestrator follows a direct-first policy. This document codifies the four-tier decision tree the orchestrator applies to every user message.
|
||||
|
||||
## Tier 1 — Reply directly (no tools)
|
||||
Apply when: small talk, simple factual Q&A, acknowledgements, clarification requests, context already in the system prompt.
|
||||
Cost: 0 tokens (output only).
|
||||
Rule: if you can answer without calling any tool, do so.
|
||||
|
||||
## Tier 2 — Use a direct tool
|
||||
Apply when: the task needs a tool but not specialised execution (time lookup, memory read/write, cron scheduling, workspace state, listing connections).
|
||||
Cost: 1 tool call + parse overhead (~200-400 tokens).
|
||||
Rule: prefer `current_time`, `cron_*`, `memory_*`, `memory_tree`, `read_workspace_state`, `composio_list_connections`, `ask_user_clarification`.
|
||||
|
||||
## Tier 3 — Delegate to a reusable async sub-agent
|
||||
Apply when: the task requires specialised execution (writing code, crawling docs, running shell, calling an external integration) that the orchestrator cannot do directly.
|
||||
Cost: full sub-agent turn (~1-5k tokens depending on archetype).
|
||||
Rule: spawn the narrowest archetype that can complete the task. `spawn_subagent` is reusable and asynchronous by default: it first looks for a compatible durable worker for the same parent thread, agent id, toolkit/model/sandbox/action-root shape, and deterministic task key. If one is running, the new instruction is steered into it; if one is idle, it resumes from saved history; otherwise a new durable worker session is created.
|
||||
|
||||
## Tier 4 — Explicit worker lifecycle control
|
||||
Apply when: the task is long (>5 turns estimated), produces a large transcript, needs manual inspection, or the user explicitly wants it tracked/closed separately.
|
||||
Cost: same as Tier 3 but the parent thread is not flooded.
|
||||
Rule: use `list_subagents`, `steer_subagent`, `wait_subagent`, and `close_subagent` with `subagent_session_id` for durable control. Use `fresh: true` only when the prior worker is materially incompatible or the user asks for a clean worker. Use `blocking: true` only when the parent must synchronously wait for the child result before replying. Do not chain workers (workers cannot spawn workers).
|
||||
|
||||
## Anti-patterns to avoid
|
||||
- Spawning a sub-agent to answer a question the orchestrator already has context for.
|
||||
- Delegating a tool call to a sub-agent when `current_tier <= 2` applies.
|
||||
- Repeatedly forcing fresh sub-agents for the same logical job, which loses worker-local context and cache affinity.
|
||||
- Passing the entire parent conversation as context to a sub-agent — pass only the task-relevant slice.
|
||||
@@ -1,374 +0,0 @@
|
||||
# E2E full-suite hardening — session handoff notes
|
||||
|
||||
Branch: `ci/full-e2e-run-2026-05-23` on `senamakel/openhuman` (fork).
|
||||
Date: 2026-05-23 → 2026-05-24.
|
||||
|
||||
This is the snapshot of what's been done, what's known, and what to pick
|
||||
up next. Pair this with `gitbooks/developing/e2e-testing.md` (the
|
||||
existing E2E doc) — this file documents the multi-session push to get
|
||||
the **full** suite (all ~87 specs) reliably green on Linux + reproducible
|
||||
locally in Docker.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — Current state
|
||||
|
||||
| Surface | Status |
|
||||
|---|---|
|
||||
| Full-suite CI on Linux | **72 / 87 passing** (15 failing), 6 parallel shards, ~25 min wall |
|
||||
| Two shards 100% green | **commerce (11/0)** + **webhooks (9/0)** |
|
||||
| Local Docker runs same 6-shard layout | `bash app/scripts/e2e-run-shards.sh` |
|
||||
| Local + CI agree on shard pass/fail | Yes (per-spec counts differ inside failing shards; see *CEF instability* below) |
|
||||
| macOS / Windows full-suite | Not yet validated this session — sharded jobs exist in workflow but only Linux was iterated on |
|
||||
|
||||
Branch SHA at handoff: `2bad1f046` (`revert: drop Escape press in openConnectorModal`).
|
||||
|
||||
---
|
||||
|
||||
## What changed (commits, top → bottom is most recent)
|
||||
|
||||
1. `revert: drop Escape press in openConnectorModal` — regressed 7
|
||||
connectors, reverted.
|
||||
2. `fix(e2e): only press Escape in openConnectorModal when a modal
|
||||
backdrop is actually present` — superseded by revert.
|
||||
3. `perf(e2e): split integrations into providers + webhooks shards` —
|
||||
6-shard matrix.
|
||||
4. `test(e2e): finish composio_sync URL-drop + close stale modal in
|
||||
openConnectorModal`
|
||||
5. `test(e2e): local shard runner + fix telegram-flow reference +
|
||||
connector log refs` — adds `app/scripts/e2e-run-shards.sh`.
|
||||
6. `test(e2e): drop URL-based assertions for composio_sync/_execute`
|
||||
7. `perf(e2e): isolate connector smoke specs into their own shard`
|
||||
8. `test(e2e): orchestrator coverage + state-bleed fixes` — adds 23
|
||||
missing specs to `e2e-run-all-flows.sh`.
|
||||
9. `test(e2e): align Linux specs with PR #2550 settings restructure`
|
||||
10. `test(e2e): point auth-access-control logout test to /settings/account`
|
||||
11. `test(e2e): drop assertions for surfaces removed in PR #2550`
|
||||
12. `test(e2e): switch auth bypass from deep-link to loopback OAuth
|
||||
path` — important fidelity change, see below.
|
||||
13. `perf(e2e): hoist Linux full-suite build into a single job,
|
||||
fan-out tests` — build-once → matrix shards.
|
||||
14. `fix(e2e): gate build-skip on BOTH binary + CEF cache hits`
|
||||
15. `fix(e2e): align CEF cache paths with actual download location`
|
||||
16. `fix(e2e): set CEF_PATH when binary cache skips build`
|
||||
17. `perf(e2e): cache built binary across shard runs`
|
||||
18. `fix(e2e): install x86_64-apple-darwin target for Mac shard build`
|
||||
19. `perf(e2e): shard full suite across 4 parallel jobs per OS`
|
||||
20. `perf(e2e): run full suite in one shared session (no per-spec
|
||||
relaunch)` — first big perf jump.
|
||||
21. `fix(e2e): repair stale assertions in linux-cef-deb-runtime spec`
|
||||
22. `fix(e2e): put cargo-tauri install root on PATH for macOS/Windows
|
||||
CI` — Mac/Win build fix.
|
||||
|
||||
---
|
||||
|
||||
## Architecture as it stands today
|
||||
|
||||
### CI workflow
|
||||
|
||||
`.github/workflows/e2e-reusable.yml` defines three Linux job tiers:
|
||||
|
||||
```text
|
||||
e2e-linux (smoke + mega-flow only, runs when inputs.full == false)
|
||||
rust-e2e-linux (Rust-side `tests/*_e2e.rs` against mock backend)
|
||||
build-linux-full (one job: cargo tauri build + tar artifact, uploads)
|
||||
e2e-linux-full (matrix of 6 shards, each `needs: build-linux-full`)
|
||||
```
|
||||
|
||||
The build job tars `app/src-tauri/target/debug/OpenHuman`, `app/dist`,
|
||||
and `$HOME/Library/Caches/tauri-cef/` into a single `tar -czf`
|
||||
artifact (~600 MB). Each shard downloads + extracts to the canonical
|
||||
paths and skips the build step entirely. CEF/binary caches still live
|
||||
on the build job to keep cold builds fast.
|
||||
|
||||
### Shard layout
|
||||
|
||||
```text
|
||||
foundation = auth,navigation,system (~21 specs)
|
||||
chat = chat,skills,journeys (~19 specs)
|
||||
providers = providers,notifications (~14 specs)
|
||||
webhooks = webhooks (~9 specs)
|
||||
connectors = connectors (~16 specs)
|
||||
commerce = payments,settings (~11 specs)
|
||||
```
|
||||
|
||||
The 6th shard (`webhooks` carved out of the original `integrations`)
|
||||
was added because anything over ~18-20 specs in one shared CEF
|
||||
session goes unstable on Linux. The `connectors` suite is its own
|
||||
category in `e2e-run-all-flows.sh` for the same reason.
|
||||
|
||||
### Local equivalent
|
||||
|
||||
`app/scripts/e2e-run-shards.sh` is the local mirror of the CI matrix.
|
||||
Runs each shard as a fresh `e2e-run-all-flows.sh --suite=…`
|
||||
invocation, so each shard gets a fresh CEF process.
|
||||
|
||||
```bash
|
||||
docker compose -f e2e/docker-compose.yml run --rm e2e \
|
||||
bash -lc "bash app/scripts/e2e-run-shards.sh"
|
||||
# or one shard:
|
||||
docker compose -f e2e/docker-compose.yml run --rm e2e \
|
||||
bash -lc "bash app/scripts/e2e-run-shards.sh foundation"
|
||||
```
|
||||
|
||||
### Orchestrator (`app/scripts/e2e-run-all-flows.sh`)
|
||||
|
||||
Collects all spec paths into one list (`_spec_paths[@]`) and calls
|
||||
`e2e-run-session.sh` ONCE with the full list, instead of per-spec.
|
||||
That restored the design intent in `wdio.conf.ts` ("WDIO creates ONE
|
||||
session per worker ... all specs run sequentially in the same
|
||||
session"). Per-spec relaunch was costing ~15-30s of CEF cold-start
|
||||
× 65 specs = 15+ min of pure overhead before this change.
|
||||
|
||||
`--suite=` accepts a comma-separated list now (`--suite=auth,navigation,system`).
|
||||
|
||||
slack-flow is explicitly **commented out** in the orchestrator — it
|
||||
crashed the CEF session mid-spec consistently. Investigate before
|
||||
re-enabling.
|
||||
|
||||
---
|
||||
|
||||
## Loopback auth bypass (production fidelity)
|
||||
|
||||
Per PR #2550 the real OAuth login flow uses an RFC 8252 loopback
|
||||
listener (`http://127.0.0.1:53824/auth?state=…`) instead of the
|
||||
`openhuman://` deep-link. E2E auth bypass was still firing
|
||||
`openhuman://auth?token=…` directly through `window.__simulateDeepLink`,
|
||||
which is now the legacy fallback path.
|
||||
|
||||
Switched in `app/test/e2e/helpers/loopback-auth-helpers.ts` +
|
||||
`reset-app.ts`:
|
||||
|
||||
1. WebView calls production `startLoopbackOauthListener()` (exposed
|
||||
on `window.__startLoopbackOauthListener` when the E2E build flag
|
||||
`VITE_OPENHUMAN_E2E_RESTART_APP_AS_RELOAD === 'true'` is set in
|
||||
`app/src/utils/loopbackOauthListener.ts`).
|
||||
2. WebView wires `awaitCallback()` → `__simulateDeepLink` so the
|
||||
callback URL is rewritten `http://127.0.0.1:…/auth?…` →
|
||||
`openhuman://auth?…` and dispatched through the existing deep-link
|
||||
handler — mirroring exactly what `OAuthProviderButton.tsx` does in
|
||||
production.
|
||||
3. Node-side `fetch()` hits the loopback URL with the bypass JWT +
|
||||
state nonce appended; the Rust listener accepts, validates state,
|
||||
emits `loopback-oauth-callback`.
|
||||
|
||||
This means every spec's `resetApp()` now exercises the same Rust HTTP
|
||||
server + state nonce check + Tauri event emit that ships to users.
|
||||
|
||||
`triggerAuthDeepLink` / `triggerAuthDeepLinkBypass` are still kept for
|
||||
oauth-success deep links (e.g. mega-flow's connector callbacks) that
|
||||
the loopback path doesn't cover.
|
||||
|
||||
---
|
||||
|
||||
## Known failures and root causes
|
||||
|
||||
### Foundation (2 failing on CI, more on local)
|
||||
|
||||
| Spec | Cause | Difficulty |
|
||||
|---|---|---|
|
||||
| `onboarding-modes` (Phase B) | After Phase A reaches `/home`, `resetOnboardingFlagAndReload` resets `onboarding_completed=false` + reloads, but the Custom-card click in Phase B doesn't register (data-testid found but click is intercepted or stale). Needs DOM inspection of the wizard re-mount. | medium |
|
||||
| `runtime-picker-login` | `resetApp(skipAuth: true)` should land on Welcome screen, but the renderer re-hydrates from a persisted snapshot and lands on /home instead. `resetApp` already polls for the Welcome heading + re-replaces `#/` for up to 10s; insufficient. Likely needs to wait for `snapshot.sessionToken` to be cleared (via `fetchCoreAppSnapshot`) before considering the reset done. | medium |
|
||||
|
||||
### Chat (3 failing)
|
||||
|
||||
| Spec | Cause |
|
||||
|---|---|
|
||||
| `chat-harness-subagent` | Agent orchestrator doesn't produce expected canary string. Real product/agent behavior — not a test bug. |
|
||||
| `chat-harness-wallet-flow` | Crypto agent doesn't produce wallet quote. Real product behavior. |
|
||||
| `chat-multi-tool-round` | T2.1 (`agent calls tool 1 (file_read); timeline shows it`) — `expect.toBe(true)` fails. Could be timing or real product change. |
|
||||
|
||||
`chat-conversation-history` H1.4 was fixed earlier — root cause was
|
||||
`getSelectedThreadId()` returning the prior-spec's stale thread id
|
||||
before the New-thread click had time to update Redux. Fix: capture
|
||||
prior id, wait for `selectedThreadId !== priorThreadId`.
|
||||
|
||||
### Providers (3 failing)
|
||||
|
||||
`conversations-web-channel-flow`, `telegram-channel-flow`,
|
||||
`whatsapp-flow` — likely the same shared-CEF-session instability
|
||||
hitting late-shard specs. Worth re-checking after any further shard
|
||||
reduction.
|
||||
|
||||
### Connectors (7 failing — all hit "expired auth" subtest)
|
||||
|
||||
The other 9 connector tests in each spec pass. The one consistent
|
||||
failure is "expired auth shows Reconnect button and does not log user
|
||||
out" — `openConnectorModal()`'s card click is intercepted because the
|
||||
previous test left a modal backdrop up. Attempted fix (`Escape`
|
||||
before click) regressed other tests; reverted. Real fix probably:
|
||||
guarantee modal close in `afterEach` rather than working around it in
|
||||
the open helper.
|
||||
|
||||
---
|
||||
|
||||
## CEF shared-session instability — the recurring theme
|
||||
|
||||
Empirically, the shared-CEF debug build becomes unreliable past
|
||||
~18-20 specs in a single session. Symptoms vary:
|
||||
|
||||
- `__simulateDeepLink ready? false (poll N)` after the listener was
|
||||
previously fine
|
||||
- `A sessionId is required for this command`
|
||||
- ECONNREFUSED to Appium :4723 mid-suite
|
||||
- Mysterious `esbuild` platform-mismatch errors during WDIO's TS
|
||||
transform of a spec file (red herring — sub-symptom of WDIO
|
||||
failing to bring the spec into scope after a session loss)
|
||||
|
||||
Mitigations applied:
|
||||
|
||||
- Shard so no shard runs more than ~16 specs (and the busiest two —
|
||||
foundation 21, chat 19 — are at the edge of what works).
|
||||
- Run each shard as a fresh `e2e-run-session.sh` invocation locally
|
||||
(mirrors CI matrix isolation).
|
||||
|
||||
What might fix it for real (not attempted this session):
|
||||
|
||||
- Bump WDIO `specFileRetries` so a session loss restarts the failing
|
||||
spec.
|
||||
- Periodic `openhuman.test_reset` + reload at a fixed cadence (every
|
||||
10 specs?) to clear in-process leaks.
|
||||
- Build the test binary in `--release` to reduce per-process memory
|
||||
pressure (debug CEF + tauri builds are heavy).
|
||||
|
||||
---
|
||||
|
||||
## Stale assertions / PR #2550 drift — handled
|
||||
|
||||
PR #2550 ("fix(oauth): make loopback redirect actually work, plus
|
||||
settings cleanup") moved a bunch of settings surfaces. Fixed tests:
|
||||
|
||||
- Logout/Clear App Data lives at `/settings/account` (was `/settings`).
|
||||
Updated `logoutViaSettings` helper + `settings-data-management` +
|
||||
`auth-access-control`.
|
||||
- `/settings/connections` route deleted (ConnectionsPanel removed).
|
||||
`settings-account-preferences` dropped the post-recovery-phrase
|
||||
wallet status assertion; `navigation-settings-panels` N2.2
|
||||
`.skip`-ed with PR pointer.
|
||||
- "Notification Routing" no longer a top-level Developer Options
|
||||
entry — moved into a tab on `/settings/notifications#routing`.
|
||||
`settings-advanced-config` navigates to `/settings/notifications`
|
||||
and clicks the Routing tab.
|
||||
- `screen-intelligence` dropped the "Permissions" assertion on Linux
|
||||
(the section is gated behind `status.platform_supported`, true
|
||||
only on macOS).
|
||||
|
||||
---
|
||||
|
||||
## Composio connector specs — the `composio_sync` URL gotcha
|
||||
|
||||
The 15 connector smoke specs each had:
|
||||
|
||||
```ts
|
||||
clearRequestLog();
|
||||
await callOpenhumanRpc('openhuman.composio_sync', { toolkit: TOOLKIT_SLUG });
|
||||
const syncReq = getRequestLog().find(
|
||||
r => r.method === 'POST' && r.url.includes('/composio/sync')
|
||||
);
|
||||
expect(syncReq).toBeDefined(); // always failed
|
||||
```
|
||||
|
||||
`/composio/sync` does not exist in the mock router and the
|
||||
`composio_sync` RPC short-circuits with "no native provider
|
||||
registered" for any connector without a Rust-side provider, so no
|
||||
HTTP request is ever logged. The probe-style assertion never had a
|
||||
chance.
|
||||
|
||||
The real intent (per the spec's `PASS:` log message: "sync does not
|
||||
nuke session") is covered by `assertSessionNotNuked()` on the next
|
||||
line. Dropped the URL check across all 15 specs.
|
||||
|
||||
Same fix for `composio_execute` / `/composio/execute`.
|
||||
|
||||
---
|
||||
|
||||
## Local docker quirks
|
||||
|
||||
- The docker-compose has named volumes per-platform for `node_modules`
|
||||
and `.pnpm-store` (the bind-mounted host `node_modules` would
|
||||
clobber Linux binaries with macOS ones).
|
||||
- `e2e-bootstrap` (in `e2e/docker-entrypoint.sh`) installs Appium 3 +
|
||||
chromium driver on first entry and caches into the npm volume.
|
||||
- Docker Desktop dies if the host has < ~1 GB free. Watch for
|
||||
`ENOSPC` while running long suites — output files grow fast.
|
||||
- `tee /tmp/local-shards.log` is the recommended way to capture the
|
||||
sharded run output; the bg-task output file gets cleaned up
|
||||
aggressively by the harness.
|
||||
|
||||
---
|
||||
|
||||
## Suggested next-session priorities
|
||||
|
||||
1. **Foundation Phase B onboarding + runtime-picker Welcome.**
|
||||
`resetApp(skipAuth)` is close — it polls for Welcome heading,
|
||||
force-replaces hash to `#/`, gives 10s. Needs to additionally
|
||||
poll `fetchCoreAppSnapshot()` until `sessionToken` is gone before
|
||||
returning. Probably 1-2 hours of careful work.
|
||||
|
||||
2. **Connector expired-auth `openConnectorModal`.** Add an
|
||||
`afterEach` that explicitly closes any open modal (`Escape` +
|
||||
wait for backdrop to disappear) rather than the failed
|
||||
"Escape-before-open" approach. ~30 min.
|
||||
|
||||
3. **CEF session retry.** Add WDIO `specFileRetries: 1` so a
|
||||
session-loss in shard N+1 retries spec N+1 in a fresh slot
|
||||
instead of cascading the rest of the shard. This should recover
|
||||
maybe 5-8 of the late-shard failures.
|
||||
|
||||
4. **Validate macOS + Windows full-suite.** Workflow already has the
|
||||
shard structure for both, but they haven't been exercised this
|
||||
session (Linux focus). Re-dispatch with `-f run_macos=true
|
||||
-f run_windows=true -f full=true` and triage.
|
||||
|
||||
5. **Re-enable slack-flow once the CEF stability fix lands.** It's
|
||||
the only spec the orchestrator deliberately skips today.
|
||||
|
||||
---
|
||||
|
||||
## Key file paths
|
||||
|
||||
- Workflow: `.github/workflows/e2e-reusable.yml`
|
||||
- Orchestrator: `app/scripts/e2e-run-all-flows.sh`
|
||||
- Local sharder: `app/scripts/e2e-run-shards.sh`
|
||||
- Session runner: `app/scripts/e2e-run-session.sh`
|
||||
- Build script: `app/scripts/e2e-build.sh`
|
||||
- WDIO config: `app/test/wdio.conf.ts`
|
||||
- Loopback auth helper: `app/test/e2e/helpers/loopback-auth-helpers.ts`
|
||||
- Production loopback (exposes `__startLoopbackOauthListener` for
|
||||
E2E): `app/src/utils/loopbackOauthListener.ts`
|
||||
- Reset-app helper: `app/test/e2e/helpers/reset-app.ts`
|
||||
- Composio test helper: `app/test/e2e/helpers/composio-helpers.ts`
|
||||
- Docker setup: `e2e/docker-compose.yml`, `e2e/docker-entrypoint.sh`
|
||||
|
||||
---
|
||||
|
||||
## Useful commands cheatsheet
|
||||
|
||||
```bash
|
||||
# CI: dispatch a Linux-only full run on the fork (only run_macos / run_windows are inputs)
|
||||
gh workflow run E2E --repo senamakel/openhuman \
|
||||
--ref ci/full-e2e-run-2026-05-23 \
|
||||
-f run_macos=false -f run_windows=false -f full=true
|
||||
|
||||
# CI: shard summary
|
||||
gh run view <run-id> --repo senamakel/openhuman | grep -E '^(✓|X|\*|-) '
|
||||
|
||||
# CI: per-shard pass/fail + failing spec list
|
||||
gh api repos/senamakel/openhuman/actions/jobs/<job-id>/logs > /tmp/job.log
|
||||
grep -c 'PASSED in linux' /tmp/job.log
|
||||
grep -c 'FAILED in linux' /tmp/job.log
|
||||
grep 'FAILED in linux' /tmp/job.log \
|
||||
| sed -E 's|.*specs/||;s|\.spec\.ts.*||' | sort -u
|
||||
|
||||
# Local: full sharded run
|
||||
docker compose -f e2e/docker-compose.yml run --rm e2e \
|
||||
bash -lc "bash app/scripts/e2e-run-shards.sh" 2>&1 | tee /tmp/local-shards.log
|
||||
|
||||
# Local: single shard
|
||||
docker compose -f e2e/docker-compose.yml run --rm e2e \
|
||||
bash -lc "bash app/scripts/e2e-run-shards.sh foundation"
|
||||
|
||||
# Local: single spec
|
||||
docker compose -f e2e/docker-compose.yml run --rm e2e \
|
||||
bash -lc "bash app/scripts/e2e-run-session.sh test/e2e/specs/<spec>.spec.ts"
|
||||
```
|
||||
@@ -1,120 +0,0 @@
|
||||
# Environment Contract Roadmap
|
||||
|
||||
Post-v1 direction. Framing borrowed from Jeffrey Li's "Agent Harness Is Not Enough"
|
||||
(holaOS thesis): long-horizon agent systems need an *environment contract* around
|
||||
the execution harness, not just a better harness.
|
||||
|
||||
This doc is the note-version of where we go **after** v1 ships.
|
||||
Not a replacement for `TODO.md` — that stays tactical.
|
||||
|
||||
---
|
||||
|
||||
## Where we already sit on the contract
|
||||
|
||||
| Contract layer | Today in openhuman |
|
||||
| --- | --- |
|
||||
| Durable authored state | `skills/` submodule, `ai/*.md` (SOUL, IDENTITY, AGENTS, USER, BOOTSTRAP, MEMORY, TOOLS), controller registry (`src/core/all.rs`) |
|
||||
| Durable adaptive state | TinyHumans memory (`skill-{skill}` namespaces, with `integration_id` carried in record metadata), curated_memory snapshots, retrieval evals |
|
||||
| Runtime continuity | `OPENHUMAN_WORKSPACE` override, r2d2 SQLite pools, life_capture ingest, event bus |
|
||||
| Projected execution state | Controller schemas, JSON-RPC dispatch, capability routing per run |
|
||||
| Portability | Workspace-as-unit via `OPENHUMAN_WORKSPACE` |
|
||||
|
||||
The harness (Rust agentic loop in `src-tauri/src/commands/chat.rs`) is swappable.
|
||||
Most of the weight is already in environment, not in the loop.
|
||||
|
||||
---
|
||||
|
||||
## Gaps to close (the "review boundary")
|
||||
|
||||
Order matters: each unlocks signal for the next.
|
||||
|
||||
### 1. Run trace persistence *(unlocks everything else)*
|
||||
Today: eval traces exist as fixtures; run-level traces are ephemeral.
|
||||
Need:
|
||||
- Persist per-turn record: hot context composition (what was pulled from memory /
|
||||
OpenClaw / Notion), tool calls fired + results, model routing, outcome.
|
||||
- Land in local SQLite under workspace root (`OPENHUMAN_WORKSPACE/traces/`).
|
||||
- Surface in UI (traces panel) — operator can inspect a run later.
|
||||
- Keep it cheap: append-only, no sync by default.
|
||||
|
||||
Why first: no review loop works without durable evidence of what happened.
|
||||
|
||||
### 2. Operator feedback primitives
|
||||
Today: feedback is implicit (user edits, re-runs, disconnects).
|
||||
Need:
|
||||
- Explicit signals on: memory candidates (keep/drop), tool results (good/bad),
|
||||
full turns (thumbs). Minimal UI — thumb + optional reason string.
|
||||
- Feedback attaches to trace ID so signal is joinable with context.
|
||||
- Stored alongside traces; no backend dependency.
|
||||
|
||||
Why second: traces without judgment are noise. This is the reward-like signal
|
||||
Jeffrey calls out.
|
||||
|
||||
### 3. Curated_memory → candidate skill pipeline
|
||||
Today: curated_memory promotes facts into prompts. No path from "agent did X
|
||||
reliably" to "X is a skill."
|
||||
Need:
|
||||
- Detect repeated tool-call patterns with positive feedback (e.g. same sequence,
|
||||
same shape of args, good outcomes).
|
||||
- Generate candidate skill scaffold (`SKILL.md` with frontmatter per the current loader contract; legacy `skill.json` remains as a fallback only).
|
||||
- Review queue in UI — user approves, rejects, or edits before it lands in
|
||||
`skills/`.
|
||||
- Promoted skill is just a regular skill from that point on.
|
||||
|
||||
Why third: needs (1) for pattern data and (2) for "reliably" judgment.
|
||||
|
||||
### 4. Capability projection per role
|
||||
Today: controller permissions and visibility are static.
|
||||
Need:
|
||||
- Roles as first-class: "trading assistant," "inbox triage," etc., each with its
|
||||
own allowed action surface.
|
||||
- Capability grants tied to review — role earns a skill/tool only after the
|
||||
candidate pipeline promotes it.
|
||||
- Per-run projection: harness only sees the surface the role owns.
|
||||
|
||||
Why last: hardest and needs (1)-(3) to have signal worth projecting from.
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Not** replacing the Rust harness. The loop is fine; the point is the
|
||||
contract around it.
|
||||
- **Not** building a generic agent OS. openhuman is a product (AI assistant for
|
||||
communities); the contract serves that.
|
||||
- **Not** shipping this before v1. Premature without real usage data — the whole
|
||||
point is review over runs that actually happened.
|
||||
|
||||
---
|
||||
|
||||
## Harness-swap test (our rubric)
|
||||
|
||||
If we replaced `chat_send_inner` with Claude Agent SDK or OpenAI Agents SDK
|
||||
tomorrow, these must survive unchanged:
|
||||
|
||||
- [x] Skills manifests + handlers
|
||||
- [x] Memory namespaces + curated snapshots
|
||||
- [x] Controller registry + JSON-RPC schemas
|
||||
- [x] Event bus + life_capture data
|
||||
- [x] Workspace portability (`OPENHUMAN_WORKSPACE`)
|
||||
- [ ] Run traces (missing)
|
||||
- [ ] Operator feedback records (missing)
|
||||
- [ ] Promoted skill provenance (missing)
|
||||
- [ ] Role → capability map (missing)
|
||||
|
||||
v1 closes the first five. This roadmap closes the last four.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
- Where do traces live long-term? Local-only, or opt-in sync for eval?
|
||||
- Does role modeling need UI, or is it config-only to start?
|
||||
- Candidate skills: LLM-generated scaffold vs. pure pattern extraction?
|
||||
- Do we expose traces to skills themselves (self-improvement loop) or keep them
|
||||
operator-only?
|
||||
|
||||
---
|
||||
|
||||
_Seeded 2026-04-22 after conversation on Jeffrey Li's environment-contract piece._
|
||||
_Sequencing and scope will shift once v1 is in real users' hands._
|
||||
@@ -1,169 +0,0 @@
|
||||
# MCP Setup Agent
|
||||
|
||||
A sub-agent that walks the user through installing, configuring, and
|
||||
connecting an MCP server from one of the upstream registries
|
||||
(`mcp_registry::registries`: Smithery, modelcontextprotocol/registry).
|
||||
|
||||
**Status: implemented** (issue #3039). The agent archetype, its five
|
||||
`mcp_setup_*` tools, the opaque-secret request/submit flow, and the
|
||||
`install_and_connect` commit path are all live. The orchestrator delegates to
|
||||
it via the `setup_mcp_server` delegate (the `mcp_setup` entry in
|
||||
`src/openhuman/agent_registry/agents/orchestrator/agent.toml`), so a chat turn
|
||||
like *"set up the Notion MCP server"* routes here. The sections below describe
|
||||
how the flow works; paths reflect the shipped layout.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
A non-technical user says *"set up the Notion MCP server for me"*. The
|
||||
agent:
|
||||
|
||||
1. Browses the enabled registries, finds the candidate, summarises it.
|
||||
2. Asks for any secrets the server requires (API keys, OAuth tokens, …)
|
||||
**without ever pulling the values into the LLM context**.
|
||||
3. Test-connects with the collected secrets, surfaces errors, lets the
|
||||
user retry / change values.
|
||||
4. On success, persists the install and the secrets, runs boot-spawn for
|
||||
this one server, returns connection status + the tool list now
|
||||
available to the main agent.
|
||||
|
||||
The agent owns the conversation; the core owns the secrets, the
|
||||
subprocess, and the persistence.
|
||||
|
||||
---
|
||||
|
||||
## Tool surface
|
||||
|
||||
Five tools registered behind a `mcp_setup_*` namespace. All tool inputs
|
||||
and outputs are JSON; secret values **never** appear in either direction.
|
||||
|
||||
| Tool | Input | Output | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `mcp_setup_search` | `{ query?, page?, page_size?, source? }` | `{ servers: [Summary], total_pages }` | Thin wrapper over `mcp_registry::registry::registry_search`. `source` optionally scopes to one upstream. |
|
||||
| `mcp_setup_get` | `{ qualified_name }` | `{ detail, required_env_keys }` | Wraps `registry_get`; pre-computes `required_env_keys` from the `config_schema` (same logic as `ops::collect_required_env_keys`). |
|
||||
| `mcp_setup_request_secret` | `{ key_name, prompt }` | `{ ref: "secret://<opaque>" }` | Triggers an out-of-band UI prompt. Returns an opaque ref; raw value is held in a process-local in-memory map keyed by ref. |
|
||||
| `mcp_setup_test_connection` | `{ qualified_name, env_refs: { KEY: "secret://…" } }` | `{ ok, tools?: [McpTool], error?: string }` | Spawns the candidate subprocess in a **scratch** workspace, resolves refs to values just-in-time, runs `initialize` + `tools/list`, tears it down. No persistence. |
|
||||
| `mcp_setup_install_and_connect` | `{ qualified_name, env_refs }` | `{ server_id, status, tools: [McpTool] }` | Resolves refs, persists the install + `mcp_client_env` rows, calls `connections::connect`. Refs are always consumed (removed from the in-memory map) regardless of outcome — on failure the agent must re-prompt via `mcp_setup_request_secret`. |
|
||||
|
||||
---
|
||||
|
||||
## Secret flow — opaque refs
|
||||
|
||||
The hard requirement: **raw secret values must not enter LLM context**.
|
||||
Opaque refs solve this cleanly:
|
||||
|
||||
```
|
||||
agent: mcp_setup_request_secret({ key_name: "NOTION_API_KEY", prompt: "Notion integration token" })
|
||||
core: → pushes prompt to UI; user types into a native input box
|
||||
core: ← receives value, stores in SETUP_SECRETS: HashMap<RefId, String>
|
||||
core: → returns { ref: "secret://7c9f2e" } ← the agent sees only this
|
||||
agent: mcp_setup_test_connection({
|
||||
qualified_name: "@notion/server",
|
||||
env_refs: { "NOTION_API_KEY": "secret://7c9f2e" }
|
||||
})
|
||||
core: → for each ref, look up the value in SETUP_SECRETS, build the env
|
||||
vector, spawn, init, list_tools, tear down
|
||||
core: ← returns { ok: true, tools: [...] } ← still no raw value to agent
|
||||
```
|
||||
|
||||
Lifecycle of `SETUP_SECRETS`:
|
||||
|
||||
- Process-local `OnceLock<RwLock<HashMap<RefId, SecretEntry>>>`.
|
||||
- Entries TTL out after, say, 15 min (defends against stranded secrets if
|
||||
the conversation is abandoned mid-flow).
|
||||
- `mcp_setup_install_and_connect` consumes refs regardless of outcome:
|
||||
pulls each value, writes it to the `mcp_client_env` table (existing
|
||||
persistence, already keyed by `server_id`), and removes the ref from
|
||||
the in-memory map. On failure the agent should re-prompt via
|
||||
`mcp_setup_request_secret` to collect fresh refs for a retry.
|
||||
- On core shutdown the map is dropped — refs do not survive restart.
|
||||
|
||||
`RefId` is a short random hex string. **No structure or hint of the
|
||||
underlying value** so the agent has nothing useful to leak even if it
|
||||
tries.
|
||||
|
||||
### Why not just take key names?
|
||||
|
||||
Considered (option 2 in the original AskUserQuestion). Rejected because:
|
||||
|
||||
- The agent can't decide between values it just collected — e.g. trying
|
||||
two different tokens to pick the one that works requires distinguishing
|
||||
them, which requires handles.
|
||||
- Tying secrets to the `(server_id, key)` pair too early means a failed
|
||||
test-connect leaves stale rows in `mcp_client_env` for an
|
||||
uninstall-rolled-back server.
|
||||
|
||||
Opaque refs give the agent enough handle to iterate without exposing
|
||||
values.
|
||||
|
||||
---
|
||||
|
||||
## Where the agent lives
|
||||
|
||||
Follows the existing sub-agent pattern (`src/openhuman/agent_registry/`):
|
||||
|
||||
- Archetype TOML at `src/openhuman/agent_registry/agents/mcp_setup/agent.toml`
|
||||
(loaded by the agent registry loader).
|
||||
- Prompt + tool allowlist scoped tight: only the five `mcp_setup_*` tools
|
||||
plus `ask_user_clarification`. **No** general filesystem, network, or shell
|
||||
tools — the agent shouldn't be able to exfiltrate a leaked ref even if one
|
||||
shows up. (`submit_secret` is intentionally NOT in the agent allowlist — the
|
||||
UI calls it out-of-band via the socket bridge.)
|
||||
- Triggered by the orchestrator's `setup_mcp_server` delegate (the `mcp_setup`
|
||||
entry in `agent_registry/agents/orchestrator/agent.toml`), or directly from
|
||||
chat when the user asks to add/install/set up an MCP server.
|
||||
|
||||
---
|
||||
|
||||
## Implementation outline
|
||||
|
||||
Following the project's `Specify → Rust → JSON-RPC → UI → tests` flow:
|
||||
|
||||
1. **Rust core** (in `src/openhuman/mcp_registry/`):
|
||||
- New module `setup.rs` owning `SETUP_SECRETS` (in-memory ref map with
|
||||
TTL) and helpers `mint_ref`, `resolve_refs(env_refs) -> Vec<(K,V)>`,
|
||||
`consume_refs(env_refs)`.
|
||||
- New module `setup_ops.rs` with the four handlers.
|
||||
- Wire schemas in `schemas.rs`, controllers in `core/all.rs`.
|
||||
2. **Tool-side bridge** so the agent harness sees the four tools as
|
||||
regular tool defs. Reuse the controller-to-tool generator already
|
||||
used elsewhere.
|
||||
3. **UI**: out-of-band secret prompt component (probably a `chat`-pinned
|
||||
modal listening on a new socket event `mcp_setup_request_secret`),
|
||||
submit POSTs the value to a Tauri command that calls into core to
|
||||
register the ref.
|
||||
4. **Archetype** + system prompt at
|
||||
`src/openhuman/agent_registry/agents/mcp_setup/agent.toml`.
|
||||
5. **Tests**:
|
||||
- Unit: ref lifecycle (mint → resolve → consume → TTL expiry).
|
||||
- Integration (`tests/mcp_registry_e2e.rs` style): full flow against
|
||||
the existing `test-mcp-stub` binary, asserting refs vanish after
|
||||
install + that test-connect failures leave refs intact.
|
||||
|
||||
---
|
||||
|
||||
## Open questions for the implementer
|
||||
|
||||
- TTL value — 15 min is a guess; calibrate against typical install flow.
|
||||
- Should `test_connection` accept a partial env_refs (some refs, some
|
||||
literal-by-name) for iteration? Current design says refs only, which
|
||||
forces consistency.
|
||||
- The official MCP registry returns servers with **multiple package
|
||||
ecosystems** (`packages: [{ registry_name: "npm" | "pypi" | … }]`). The
|
||||
setup agent needs to either pick one or ask the user. Add a
|
||||
`package_choice` step or default to npm?
|
||||
- Telemetry: log `mcp_setup_*` calls (`tracing::info!` is fine) but
|
||||
never log ref values, never log env values, only key names.
|
||||
|
||||
---
|
||||
|
||||
## Anti-goals
|
||||
|
||||
- The setup agent is **not** a generic "ask user for any data" surface.
|
||||
Its prompt tool is scoped to MCP env values, full stop.
|
||||
- It does **not** persist anything until `install_and_connect` succeeds.
|
||||
No half-installed rows in `mcp_servers` or `mcp_client_env`.
|
||||
- It does **not** read back secrets. Once persisted into `mcp_client_env`
|
||||
they are write-only from the agent's perspective; only the subprocess
|
||||
spawn path in `connections::connect` reads them.
|
||||
@@ -1,113 +0,0 @@
|
||||
# Meet-agent live loop — smoke test runbook
|
||||
|
||||
End-to-end validation that the agent hears, thinks, and speaks on a
|
||||
real Google Meet call. Two laptops are easiest (Laptop A runs OpenHuman
|
||||
+ joins the Meet as the agent; Laptop B is the human host who creates
|
||||
the call and listens to the agent's voice).
|
||||
|
||||
## Pre-flight
|
||||
|
||||
1. Sign in to OpenHuman so a backend session token exists. Without
|
||||
it, all three brain stages (STT/LLM/TTS) silently fall back to
|
||||
stubs and you'll only hear a 200 ms tone — useful for plumbing
|
||||
smoke but not the real loop.
|
||||
2. Ensure the vendored `tauri-cef` submodule is on
|
||||
`feat/openhuman-audio-handler` (or whatever branch carries the
|
||||
`audio` module — see `app/src-tauri/vendor/tauri-cef`).
|
||||
3. `pnpm tauri dev` in the repo root.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Laptop B**: create a Meet call at <https://meet.google.com/new>,
|
||||
stay in the lobby.
|
||||
2. **Laptop A**:
|
||||
- Open OpenHuman → Intelligence → Calls.
|
||||
- Paste the Meet URL, set display name (e.g. "OpenHuman Agent").
|
||||
- Click *Join*.
|
||||
- A dedicated CEF window opens. The window title bar reads
|
||||
"Meet — OpenHuman Agent".
|
||||
3. **Laptop B**: admit the agent from the lobby.
|
||||
4. Confirm Meet's live captions are on. The captions bridge auto-clicks
|
||||
"Turn on captions" up to ~30 times over the first minute; if the
|
||||
button isn't found (Meet UI rolls), enable CC manually.
|
||||
5. Speak a wake-word phrase into the call mic. Examples:
|
||||
- "Hey, OpenHuman — remember to email Bob about the launch."
|
||||
- "Hey OpenHuman, follow up with the design team next week."
|
||||
|
||||
The agent should reply with a short canned ack ("Got it.",
|
||||
"Noted.", "Adding that.", "On it.", or "Captured.") routed back
|
||||
into Meet's audio.
|
||||
|
||||
## What to watch for
|
||||
|
||||
### Listen path (Meet captions → agent)
|
||||
|
||||
- The CEF audio handler / Whisper STT path is **not** the live-call
|
||||
listen path; do not expect `cef stream start` or `push_listen_pcm`
|
||||
log lines (those modules are kept in tree as `_legacy_listen` for a
|
||||
future opt-in).
|
||||
|
||||
- Tail the file logs (`~/Library/Application Support/OpenHuman/logs/`):
|
||||
|
||||
```text
|
||||
[meet-audio] inject reload requested session=…
|
||||
[meet-audio] bridge alive info={"installed":true,"sample_rate":16000,…}
|
||||
[meet-audio] captions drained count=N request_id=…
|
||||
[meet-agent-rpc] wake word fired request_id=… speaker=…
|
||||
[meet-agent] caption turn start request_id=… prompt_chars=…
|
||||
[meet-agent] caption turn done request_id=… reply_chars=… synth_samples=…
|
||||
```
|
||||
|
||||
- If `captions drained` never logs, the captions bridge didn't find
|
||||
Meet's caption region — either CC is off (auto-enable failed) or
|
||||
Meet rolled the DOM and the `aria-label="Captions"` selector
|
||||
needs updating in `captions_bridge.js`. Confirm via the embedded
|
||||
page console: `window.__openhumanCaptionsBridgeInfo()` — the
|
||||
`region_found` field should be `true` once captions are on.
|
||||
|
||||
### Speak path (agent → Meet)
|
||||
|
||||
- Inspect the embedded Meet page's console (right-click → Inspect; or
|
||||
attach via the CDP port 19222 on Laptop A): you should see
|
||||
`[openhuman-audio-bridge] feed failed: …` only on errors.
|
||||
- Run `window.__openhumanAudioBridgeInfo()` in the console:
|
||||
|
||||
```json
|
||||
{ "installed": true, "sample_rate": 16000, "audio_context_state": "running",
|
||||
"next_start_time": 12.3, "destination_track_count": 1 }
|
||||
```
|
||||
|
||||
- **Laptop B**: you should hear the agent's reply through Meet, with
|
||||
the agent's tile lighting up the "speaking" indicator.
|
||||
|
||||
### Mascot webcam
|
||||
|
||||
- Laptop B sees the OpenHuman mascot SVG in the agent's tile.
|
||||
Confirms `--use-file-for-fake-video-capture` is still active (the
|
||||
speak path doesn't break it).
|
||||
|
||||
## Things that should NOT happen
|
||||
|
||||
- macOS prompt for screen recording / microphone permission.
|
||||
- macOS prompt for installing a system audio driver / kext.
|
||||
- The OpenHuman main window's mic indicator turning on (we tap CEF's
|
||||
audio at the renderer level, not via the OS mic).
|
||||
|
||||
## Common failure modes
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
| --- | --- | --- |
|
||||
| Heard event empty / "STT failure" | No backend session | Sign in |
|
||||
| Spoke event present, no audio on Laptop B | Bridge install failed | Check `Page.reload` errored — devtools network |
|
||||
| 1× turn fires, then nothing | VAD `in_utterance` flag stuck | Look for `EndOfUtterance` events; may need a longer hangover |
|
||||
| Audio "robot voice" | Sample-rate mismatch — bridge says 16000 but TTS gave another rate | Confirm `output_format=pcm_16000` request was honored |
|
||||
| `cef stream error` repeated | Renderer crashed | Check Chromium logs in the meet-call data dir |
|
||||
|
||||
## Cleanup
|
||||
|
||||
- Close the meet-call window. The window-destroyed handler tears down
|
||||
`meet_audio` (drops the audio handler registration → silences
|
||||
capture immediately, signals the speak pump → exits) and calls
|
||||
`openhuman.meet_agent_stop_session` which logs the listened/spoken
|
||||
totals.
|
||||
- Per-call data dir is wiped automatically.
|
||||
@@ -1,75 +0,0 @@
|
||||
# Weekly Code-Review Report
|
||||
|
||||
Scheduled aggregation of slow-moving code-health signals that per-PR CI does
|
||||
not catch.
|
||||
|
||||
## What runs
|
||||
|
||||
Workflow: retired; the previous scheduled GitHub Actions workflow was removed
|
||||
when redundant workflows were pruned.
|
||||
Script: [`scripts/weekly-code-review.sh`](../scripts/weekly-code-review.sh).
|
||||
|
||||
The aggregator currently collects:
|
||||
|
||||
| Check | Source | What it catches |
|
||||
| --------------- | ----------------------------------- | ------------------------------------------------- |
|
||||
| Unused code | `pnpm exec knip` (in `app/`) | Unused files, exports, dependencies, types |
|
||||
| Rust advisories | `cargo audit` on core + Tauri shell | Published RustSec advisories against `Cargo.lock` |
|
||||
| TODO backlog | `grep` over `src/` + `app/src/` | `TODO` / `FIXME` / `XXX` / `HACK` drift |
|
||||
|
||||
Each sub-check is **best-effort**: a missing tool or transient failure is
|
||||
reported inline in the Markdown, not fatal. A full lane going red never stops
|
||||
the rest of the report from being produced.
|
||||
|
||||
## Scheduling
|
||||
|
||||
No scheduled GitHub Actions workflow is currently checked in for this report.
|
||||
Run the script locally when a weekly code-health snapshot is needed.
|
||||
|
||||
## Outputs
|
||||
|
||||
1. **Tracking issue** — created fresh every run, labeled `weekly-code-review`.
|
||||
Previous open reports are closed with a "superseded" comment so the
|
||||
maintainer triage view only shows the latest week.
|
||||
2. **Artifact** — `weekly-code-review-<run-id>` with:
|
||||
- `report.md` — the human-readable body also used for the issue.
|
||||
- `report.json` — machine-readable digest (parsed check outputs) for any
|
||||
downstream tooling.
|
||||
Retention: 90 days.
|
||||
|
||||
## Running locally
|
||||
|
||||
From the repo root:
|
||||
|
||||
```bash
|
||||
bash scripts/weekly-code-review.sh # writes to weekly-code-review-out/
|
||||
bash scripts/weekly-code-review.sh ./out # custom dir
|
||||
```
|
||||
|
||||
Dependencies: `pnpm` for knip, `cargo-audit` for Rust advisories, `python3`
|
||||
for the JSON shaping. Missing tools are skipped with a note in the report.
|
||||
|
||||
## Triaging a report
|
||||
|
||||
- **Unused code** — knip findings are suggestions; check the linked file
|
||||
before deleting. Legitimate deletions land in a `chore(cleanup)` PR.
|
||||
- **Rust advisories** — bump the affected crate (`cargo update -p <crate>`
|
||||
for a patch, or pin a workaround) and re-run `cargo audit` locally.
|
||||
- **TODO backlog** — the counter is a direction signal, not an action item
|
||||
on its own. Watch for a rising trend over successive weeks.
|
||||
|
||||
## Retiring
|
||||
|
||||
- **One-off skip** — cancel the scheduled run from the Actions tab.
|
||||
- **Pause indefinitely** — no scheduled workflow is currently installed.
|
||||
- **Retire fully** — delete `scripts/weekly-code-review.sh` and remove the
|
||||
`weekly-code-review` label. No other code references them.
|
||||
|
||||
## Intentionally out of scope for the first cut
|
||||
|
||||
- npm audit: Yarn v1's `audit` output is messy and noisy; revisit when the
|
||||
project moves to Yarn berry or adopts `audit-ci` / GitHub's dependency
|
||||
review action.
|
||||
- Bundle-size diff: needs a baseline to be meaningful; separate workflow.
|
||||
- AI-assisted review: CodeRabbit already runs per-PR; duplicating weekly
|
||||
would be noise, not signal.
|
||||
@@ -1,504 +0,0 @@
|
||||
{
|
||||
"type": "excalidraw",
|
||||
"version": 2,
|
||||
"source": "openhuman-478",
|
||||
"elements": [
|
||||
{
|
||||
"id": "title",
|
||||
"type": "text",
|
||||
"x": 300,
|
||||
"y": 20,
|
||||
"width": 500,
|
||||
"height": 40,
|
||||
"text": "OpenHuman Agent Prompt Architecture",
|
||||
"fontSize": 28,
|
||||
"fontFamily": 1,
|
||||
"textAlign": "center",
|
||||
"strokeColor": "#1e1e1e",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 1,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "orchestrator-box",
|
||||
"type": "rectangle",
|
||||
"x": 50,
|
||||
"y": 80,
|
||||
"width": 400,
|
||||
"height": 380,
|
||||
"strokeColor": "#1971c2",
|
||||
"backgroundColor": "#d0ebff",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 2,
|
||||
"roundness": { "type": 3 },
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 2,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "orchestrator-title",
|
||||
"type": "text",
|
||||
"x": 70,
|
||||
"y": 90,
|
||||
"width": 360,
|
||||
"height": 30,
|
||||
"text": "ORCHESTRATOR (main agent)",
|
||||
"fontSize": 20,
|
||||
"fontFamily": 1,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#1971c2",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 3,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "orchestrator-model",
|
||||
"type": "text",
|
||||
"x": 70,
|
||||
"y": 115,
|
||||
"width": 360,
|
||||
"height": 20,
|
||||
"text": "Model: reasoning-v1",
|
||||
"fontSize": 14,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#868e96",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 4,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "orchestrator-prompt-files",
|
||||
"type": "text",
|
||||
"x": 70,
|
||||
"y": 145,
|
||||
"width": 360,
|
||||
"height": 200,
|
||||
"text": "System prompt (from workspace .md files):\n\n✅ AGENTS.md — Orchestrator routing logic\n \"Pick the right tool, synthesise the result\"\n\n✅ SOUL.md — Personality, voice, tone\n \"Smart friend who helps get stuff done\"\n\n✅ IDENTITY.md — Mission, values, boundaries\n \"What OpenHuman is and isn't\"\n\n✅ USER.md — User adaptation rules\n \"Traders want speed, researchers want depth\"\n\n✅ BOOTSTRAP.md — First interaction flow\n \"Greet warmly, discover role, ask what's needed\"",
|
||||
"fontSize": 12,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#1e1e1e",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 5,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "orchestrator-skipped",
|
||||
"type": "text",
|
||||
"x": 70,
|
||||
"y": 370,
|
||||
"width": 360,
|
||||
"height": 80,
|
||||
"text": "Skipped (not needed for routing):\n\n❌ TOOLS.md — Skill tool docs (subagents have specs)\n❌ MEMORY.md — Memory protocol (auto-injected)\n❌ HEARTBEAT.md — Cron config (handled by system)",
|
||||
"fontSize": 12,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#e03131",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 6,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "tools-box",
|
||||
"type": "rectangle",
|
||||
"x": 500,
|
||||
"y": 80,
|
||||
"width": 350,
|
||||
"height": 250,
|
||||
"strokeColor": "#2f9e44",
|
||||
"backgroundColor": "#d8f5a2",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 2,
|
||||
"roundness": { "type": 3 },
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 7,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "tools-title",
|
||||
"type": "text",
|
||||
"x": 520,
|
||||
"y": 90,
|
||||
"width": 310,
|
||||
"height": 30,
|
||||
"text": "ORCHESTRATOR TOOLS",
|
||||
"fontSize": 20,
|
||||
"fontFamily": 1,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#2f9e44",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 8,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "tools-content",
|
||||
"type": "text",
|
||||
"x": 520,
|
||||
"y": 120,
|
||||
"width": 310,
|
||||
"height": 200,
|
||||
"text": "Visible to LLM (function-calling schema):\n\n🔧 notion → skills_agent(filter:notion)\n🔧 gmail → skills_agent(filter:gmail)\n🔧 slack → skills_agent(filter:slack)\n (dynamic: 1 per installed skill)\n\n🔧 research → researcher subagent\n🔧 run_code → code_executor subagent\n🔧 review_code → critic subagent\n🔧 plan → planner subagent\n (static: always available)\n\n🔧 spawn_subagent → fallback (fork, custom)",
|
||||
"fontSize": 12,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#1e1e1e",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 9,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "subagent-box",
|
||||
"type": "rectangle",
|
||||
"x": 50,
|
||||
"y": 500,
|
||||
"width": 800,
|
||||
"height": 300,
|
||||
"strokeColor": "#e8590c",
|
||||
"backgroundColor": "#fff4e6",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 2,
|
||||
"roundness": { "type": 3 },
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 10,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "subagent-title",
|
||||
"type": "text",
|
||||
"x": 70,
|
||||
"y": 510,
|
||||
"width": 760,
|
||||
"height": 30,
|
||||
"text": "SUB-AGENTS (spawned by orchestrator tools)",
|
||||
"fontSize": 20,
|
||||
"fontFamily": 1,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#e8590c",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 11,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "subagent-content",
|
||||
"type": "text",
|
||||
"x": 70,
|
||||
"y": 545,
|
||||
"width": 760,
|
||||
"height": 245,
|
||||
"text": "Each subagent gets a narrow system prompt: archetype .md + filtered tool specs + workspace dir\nAll subagents: omit_identity=true, omit_memory_context=true, omit_skills_catalog=true\nMemory context auto-forwarded from parent via ParentExecutionContext.memory_context\n\n┌──────────────────┬────────────────────────────────────┬──────────────┬───────────────┐\n│ Agent │ Prompt file (compiled-in) │ Model │ Safety preamble│\n├──────────────────┼────────────────────────────────────┼──────────────┼───────────────┤\n│ skills_agent │ archetypes/skills_agent.md │ agentic-v1 │ ✅ yes │\n│ researcher │ archetypes/researcher.md │ agentic-v1 │ ❌ no │\n│ code_executor │ archetypes/code_executor.md │ coding-v1 │ ✅ yes │\n│ critic │ archetypes/critic.md │ agentic-v1 │ ❌ no │\n│ planner │ PLANNER.md │ reasoning-v1 │ ❌ no │\n│ tool_maker │ archetypes/code_executor.md (shared)│ coding-v1 │ ✅ yes │\n│ archivist │ archetypes/archivist.md │ local-v1 │ ❌ no │\n│ fork │ (replays parent's exact prompt) │ inherited │ inherited │\n└──────────────────┴────────────────────────────────────┴──────────────┴───────────────┘\n\nSubagents DO NOT use workspace .md files (SOUL, IDENTITY, USER, BOOTSTRAP, TOOLS, MEMORY).\nThey only see: their archetype prompt + filtered ToolSpec schemas + [Context] block from parent.",
|
||||
"fontSize": 12,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#1e1e1e",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 12,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "arrow-orch-to-tools",
|
||||
"type": "arrow",
|
||||
"x": 450,
|
||||
"y": 200,
|
||||
"width": 50,
|
||||
"height": 0,
|
||||
"points": [[0, 0], [50, 0]],
|
||||
"strokeColor": "#1e1e1e",
|
||||
"strokeWidth": 2,
|
||||
"fillStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 13,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": [],
|
||||
"startBinding": null,
|
||||
"endBinding": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"roundness": null
|
||||
},
|
||||
{
|
||||
"id": "arrow-tools-to-subagents",
|
||||
"type": "arrow",
|
||||
"x": 550,
|
||||
"y": 330,
|
||||
"width": 0,
|
||||
"height": 170,
|
||||
"points": [[0, 0], [0, 170]],
|
||||
"strokeColor": "#e8590c",
|
||||
"strokeWidth": 2,
|
||||
"fillStyle": "solid",
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 14,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": [],
|
||||
"startBinding": null,
|
||||
"endBinding": null,
|
||||
"startArrowhead": null,
|
||||
"endArrowhead": "arrow",
|
||||
"roundness": null
|
||||
},
|
||||
{
|
||||
"id": "arrow-label-1",
|
||||
"type": "text",
|
||||
"x": 460,
|
||||
"y": 180,
|
||||
"width": 40,
|
||||
"height": 16,
|
||||
"text": "calls",
|
||||
"fontSize": 12,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "center",
|
||||
"strokeColor": "#868e96",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 15,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "arrow-label-2",
|
||||
"type": "text",
|
||||
"x": 555,
|
||||
"y": 400,
|
||||
"width": 120,
|
||||
"height": 16,
|
||||
"text": "run_subagent()",
|
||||
"fontSize": 12,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#868e96",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 16,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "memory-flow",
|
||||
"type": "text",
|
||||
"x": 500,
|
||||
"y": 360,
|
||||
"width": 350,
|
||||
"height": 70,
|
||||
"text": "Data flowing to subagents:\n→ ParentExecutionContext.memory_context\n→ ParentExecutionContext.all_tools (full registry)\n→ ParentExecutionContext.provider (shared)",
|
||||
"fontSize": 11,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#5c940d",
|
||||
"backgroundColor": "transparent",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 17,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
},
|
||||
{
|
||||
"id": "file-sync-note",
|
||||
"type": "text",
|
||||
"x": 50,
|
||||
"y": 830,
|
||||
"width": 800,
|
||||
"height": 80,
|
||||
"text": "FILE SYNC: Workspace .md files are auto-synced from compiled-in defaults.\nA hash of each built-in is stored in .{filename}.builtin-hash — when the code ships\na new version, the hash changes and the disk file is overwritten automatically.\nUser edits between releases are preserved; code updates always win.",
|
||||
"fontSize": 12,
|
||||
"fontFamily": 3,
|
||||
"textAlign": "left",
|
||||
"strokeColor": "#868e96",
|
||||
"backgroundColor": "#f8f9fa",
|
||||
"fillStyle": "solid",
|
||||
"strokeWidth": 1,
|
||||
"roundness": null,
|
||||
"roughness": 0,
|
||||
"opacity": 100,
|
||||
"angle": 0,
|
||||
"seed": 18,
|
||||
"version": 1,
|
||||
"isDeleted": false,
|
||||
"boundElements": null,
|
||||
"updated": 1,
|
||||
"link": null,
|
||||
"locked": false,
|
||||
"groupIds": []
|
||||
}
|
||||
],
|
||||
"appState": {
|
||||
"gridSize": null,
|
||||
"viewBackgroundColor": "#ffffff"
|
||||
},
|
||||
"files": {}
|
||||
}
|
||||
@@ -1,658 +0,0 @@
|
||||
# Agent / Subagent / Tool Flow
|
||||
|
||||
This document explains the current runtime flow around the agent harness, with emphasis on:
|
||||
|
||||
- how the main agent turn executes
|
||||
- how tools are exposed and executed
|
||||
- how `spawn_subagent` works
|
||||
- how typed vs fork subagents differ
|
||||
- where to look when debugging harness and delegation issues
|
||||
|
||||
Scope: current Rust implementation under `src/openhuman/agent/` and `src/openhuman/tools/`.
|
||||
|
||||
## Why This Exists
|
||||
|
||||
The code path is split across several layers:
|
||||
|
||||
- built-in agent definitions in `src/openhuman/agent/agents/`
|
||||
- harness data + task-local plumbing in `src/openhuman/agent/harness/`
|
||||
- main session lifecycle in `src/openhuman/agent/harness/session/`
|
||||
- delegation tools in `src/openhuman/agent/tools/`
|
||||
- synthesised `delegate_*` tools in `src/openhuman/tools/orchestrator_tools.rs`
|
||||
|
||||
If you only read one file, the system looks simpler than it is. The actual runtime path crosses all of them.
|
||||
|
||||
## File Map
|
||||
|
||||
### Registry and definitions
|
||||
|
||||
- `src/openhuman/agent/agents/loader.rs`
|
||||
Loads built-in agents from `agent.toml` plus dynamic `prompt.rs` builders.
|
||||
- `src/openhuman/agent/harness/definition.rs`
|
||||
Defines `AgentDefinition`, `ToolScope`, `SubagentEntry`, `PromptSource`, and registry-facing data.
|
||||
- `src/openhuman/agent/harness/mod.rs`
|
||||
Re-exports the harness entrypoints.
|
||||
|
||||
### Main agent session
|
||||
|
||||
- `src/openhuman/agent/harness/session/builder.rs`
|
||||
Builds an `Agent`, chooses dispatcher, applies visible-tool filtering, synthesises delegation tools.
|
||||
- `src/openhuman/agent/harness/session/turn.rs`
|
||||
Main turn lifecycle, tool execution, parent/fork context setup, transcript persistence, post-turn hooks.
|
||||
|
||||
### Subagent path
|
||||
|
||||
- `src/openhuman/agent/tools/spawn_subagent.rs`
|
||||
Runtime tool entrypoint for explicit subagent spawns.
|
||||
- `src/openhuman/agent/harness/fork_context.rs`
|
||||
Task-local parent and fork context.
|
||||
- `src/openhuman/agent/harness/subagent_runner.rs`
|
||||
Typed/fork subagent execution, inner loop, tool filtering, transcript writes, large-result handoff.
|
||||
|
||||
### Generic tool loop / bus path
|
||||
|
||||
- `src/openhuman/agent/harness/tool_loop.rs`
|
||||
Shared LLM -> tool -> tool result -> LLM loop used by the bus handler and legacy call sites.
|
||||
- `src/openhuman/agent/bus.rs`
|
||||
Native event-bus entrypoint `agent.run_turn`.
|
||||
|
||||
## High-Level Model
|
||||
|
||||
There are two related but distinct execution tiers:
|
||||
|
||||
1. `Agent::turn`
|
||||
This is the stateful session runtime. It owns conversation history, system prompt reuse, memory loading, hooks, transcript resume, and the parent context needed for subagents.
|
||||
|
||||
2. `run_subagent`
|
||||
This is an isolated delegated run. It does not become a nested full `Agent` session. It runs a smaller inner loop and returns a single compact text result to the parent as a normal tool result.
|
||||
Every typed subagent prompt now also appends a shared "Sub-agent Role Contract" suffix that explicitly states sub-agent role expectations and requires concise, synthesis-ready outputs.
|
||||
|
||||
That distinction matters when debugging. A subagent is not a second copy of the full session runtime.
|
||||
|
||||
## Flow Diagram
|
||||
|
||||
### Full parent -> tool -> subagent flow
|
||||
|
||||
```text
|
||||
User message
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| Agent::turn |
|
||||
| session/turn.rs |
|
||||
+---------------------------+
|
||||
|
|
||||
| 1. resume transcript if present
|
||||
| 2. build/reuse system prompt
|
||||
| 3. load memory context
|
||||
| 4. install ParentExecutionContext task-local
|
||||
v
|
||||
+---------------------------+
|
||||
| Parent iteration loop |
|
||||
| provider call |
|
||||
+---------------------------+
|
||||
|
|
||||
| provider response
|
||||
v
|
||||
+---------------------------+
|
||||
| Parse tool calls |
|
||||
| dispatcher + parser |
|
||||
+---------------------------+
|
||||
|
|
||||
+-------------------------------+
|
||||
| no tool calls |
|
||||
| |
|
||||
v |
|
||||
+---------------------------+ |
|
||||
| Final assistant text | |
|
||||
| appended to parent history| |
|
||||
+---------------------------+ |
|
||||
| |
|
||||
v |
|
||||
Return to caller |
|
||||
|
|
||||
| has tool calls
|
||||
v
|
||||
+---------------------------+
|
||||
| Execute tool calls |
|
||||
| parent tool runtime |
|
||||
+---------------------------+
|
||||
|
|
||||
+-------------------+-------------------+
|
||||
| |
|
||||
| regular tool | spawn_subagent
|
||||
v v
|
||||
+---------------------------+ +---------------------------+
|
||||
| Tool::execute(...) | | SpawnSubagentTool |
|
||||
+---------------------------+ | impl/agent/ |
|
||||
| | spawn_subagent.rs |
|
||||
| result +---------------------------+
|
||||
v |
|
||||
+---------------------------+ | validate args
|
||||
| append tool result | | lookup AgentDefinition
|
||||
| to parent history | | publish spawn event
|
||||
+---------------------------+ v
|
||||
| +---------------------------+
|
||||
+-------------------------->| run_subagent(...) |
|
||||
| subagent_runner.rs |
|
||||
+---------------------------+
|
||||
|
|
||||
+-------------------------+-------------------------+
|
||||
| |
|
||||
| typed mode | fork mode
|
||||
v v
|
||||
+---------------------------+ +---------------------------+
|
||||
| run_typed_mode | | run_fork_mode |
|
||||
| - resolve model | | - require ForkContext |
|
||||
| - filter tools | | - replay parent prefix |
|
||||
| - build narrow prompt | | - reuse parent tool specs |
|
||||
+---------------------------+ +---------------------------+
|
||||
| |
|
||||
+-------------------------+-------------------------+
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| run_inner_loop |
|
||||
| subagent private loop |
|
||||
+---------------------------+
|
||||
|
|
||||
+---------------------------+---------------------------+
|
||||
| |
|
||||
| no tool calls | tool calls
|
||||
v v
|
||||
+---------------------------+ +---------------------------+
|
||||
| final child text | | child executes allowed |
|
||||
| returned to parent tool | | tools, appends results, |
|
||||
+---------------------------+ | loops again |
|
||||
+---------------------------+
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| SpawnSubagentTool returns |
|
||||
| ToolResult(output) |
|
||||
+---------------------------+
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| parent appends tool |
|
||||
| result to history |
|
||||
+---------------------------+
|
||||
|
|
||||
v
|
||||
+---------------------------+
|
||||
| next parent iteration |
|
||||
| synthesizes final answer |
|
||||
+---------------------------+
|
||||
```
|
||||
|
||||
### Context wiring for subagents
|
||||
|
||||
```text
|
||||
Agent::turn
|
||||
|
|
||||
+--> build ParentExecutionContext
|
||||
| - provider
|
||||
| - all_tools / all_tool_specs
|
||||
| - model / temperature
|
||||
| - memory / memory_context
|
||||
| - connected_integrations
|
||||
| - composio_client
|
||||
| - tool_call_format
|
||||
| - session lineage
|
||||
|
|
||||
+--> with_parent_context(...)
|
||||
|
|
||||
+--> any tool call inside this turn can read current_parent()
|
||||
|
|
||||
+--> SpawnSubagentTool
|
||||
|
|
||||
+--> run_subagent(...)
|
||||
|
|
||||
+--> typed mode uses ParentExecutionContext directly
|
||||
|
|
||||
+--> fork mode also requires current_fork()
|
||||
|
|
||||
+--> exact parent prompt + prefix replay
|
||||
```
|
||||
|
||||
## Startup and Registry Loading
|
||||
|
||||
Built-in agents live under `src/openhuman/agent/agents/*/` as:
|
||||
|
||||
- `agent.toml`
|
||||
- `prompt.rs`
|
||||
- optional `prompt.md` kept as nearby reference material
|
||||
|
||||
`loader.rs` parses each `agent.toml`, stamps the source as builtin, and installs the `prompt.rs` builder as `PromptSource::Dynamic`.
|
||||
|
||||
The global `AgentDefinitionRegistry` is initialized at startup. `spawn_subagent` depends on it. If the registry is missing, the tool returns a clear error instead of trying to run.
|
||||
|
||||
Important consequence: agent delegation is data-driven. The runtime does not hardcode an enum of built-in agents.
|
||||
|
||||
## How a Main Agent Session Is Built
|
||||
|
||||
`AgentBuilder::build` in `session/builder.rs` assembles:
|
||||
|
||||
- provider
|
||||
- full tool registry
|
||||
- visible tool specs
|
||||
- memory backend
|
||||
- prompt builder
|
||||
- dispatcher
|
||||
- context manager
|
||||
|
||||
Two tool sets exist at build time:
|
||||
|
||||
- full tool registry: what the runtime can execute
|
||||
- visible tool set: what the model can see in its schema/prompt
|
||||
|
||||
That split is intentional. The parent may have access to more runtime tools than it exposes directly to the model.
|
||||
|
||||
### Synthesised delegation tools
|
||||
|
||||
For agents with `subagents = [...]` in their definition, the builder synthesises `delegate_*` tools using `collect_orchestrator_tools()`:
|
||||
|
||||
- `SubagentEntry::AgentId("researcher")` becomes an `ArchetypeDelegationTool`
|
||||
- `SubagentEntry::Skills({ skills = "*" })` expands to one `SkillDelegationTool` per connected integration
|
||||
|
||||
These tools are added to the model-visible surface at build time. They are wrappers around delegation, not standalone business logic.
|
||||
|
||||
## Main Turn Flow
|
||||
|
||||
`Agent::turn` in `session/turn.rs` is the main harness path.
|
||||
|
||||
### 1. Transcript resume and prompt bootstrap
|
||||
|
||||
On a fresh session:
|
||||
|
||||
- it tries to resume a previous transcript for KV-cache reuse
|
||||
- fetches connected integrations
|
||||
- fetches learned context
|
||||
- builds the system prompt once
|
||||
- stores that system prompt as the first message
|
||||
|
||||
On later turns it deliberately does not rebuild the system prompt. Byte stability is treated as a runtime invariant for backend prefix caching.
|
||||
|
||||
### 2. Memory context injection
|
||||
|
||||
Per turn, it asks the memory loader for relevant context and prepends that context to the user message. This is parent-session behavior. Subagents do not run the same memory lookup path.
|
||||
|
||||
### 3. Parent execution context is captured
|
||||
|
||||
Before the loop starts, `Agent::turn` snapshots a `ParentExecutionContext` and installs it on the task-local via `with_parent_context(...)`.
|
||||
|
||||
That context carries the data subagents need:
|
||||
|
||||
- provider
|
||||
- all tools and tool specs
|
||||
- model / temperature
|
||||
- memory handle
|
||||
- loaded memory context
|
||||
- connected integrations
|
||||
- composio client
|
||||
- tool call format
|
||||
- session / transcript lineage
|
||||
|
||||
Without this task-local, `spawn_subagent` cannot work.
|
||||
|
||||
### 4. Iterative provider loop
|
||||
|
||||
For each iteration:
|
||||
|
||||
- context reduction runs first
|
||||
- the dispatcher converts history into provider messages
|
||||
- the provider is called
|
||||
- response text and tool calls are parsed
|
||||
- tool calls are executed
|
||||
- tool results are appended to history
|
||||
- the loop repeats until no tool calls remain
|
||||
|
||||
This is the full parent loop. It also emits progress events and drives post-turn hooks.
|
||||
|
||||
## Tool Execution in the Parent Loop
|
||||
|
||||
The parent loop special-cases delegation but otherwise treats tools generically.
|
||||
|
||||
Core behaviors:
|
||||
|
||||
- unknown or filtered-out tools become structured error results
|
||||
- `CliRpcOnly` tools are blocked in the autonomous loop
|
||||
- approval-gated tools can be denied before execution
|
||||
- successful outputs may be scrubbed / compacted / summarized
|
||||
|
||||
The parent’s history preserves:
|
||||
|
||||
- assistant tool call intent
|
||||
- tool results
|
||||
- final assistant response
|
||||
|
||||
That history format is what the next iteration reasons from.
|
||||
|
||||
## Where `spawn_subagent` Enters
|
||||
|
||||
The explicit delegation tool lives in `src/openhuman/agent/tools/spawn_subagent.rs`.
|
||||
|
||||
Its flow is:
|
||||
|
||||
1. parse `agent_id`, `prompt`, optional `context`, optional `toolkit`, optional `mode`
|
||||
2. require the global `AgentDefinitionRegistry`
|
||||
3. resolve the target definition
|
||||
4. run pre-flight validation for `integrations_agent`
|
||||
5. publish `DomainEvent::SubagentSpawned`
|
||||
6. call `run_subagent(...)`
|
||||
7. publish completed or failed event
|
||||
8. return the subagent’s final text as a normal `ToolResult`
|
||||
|
||||
Important: the parent model never sees the subagent’s internal transcript. It only sees the final tool result string returned by `spawn_subagent`.
|
||||
|
||||
## Typed vs Fork Subagents
|
||||
|
||||
`run_subagent` chooses one of two modes.
|
||||
|
||||
### Typed mode
|
||||
|
||||
Default path. Implemented by `run_typed_mode(...)`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- resolves model from the definition
|
||||
- filters the parent’s tools down to what the child is allowed to use
|
||||
- builds a fresh narrow system prompt
|
||||
- optionally injects inherited memory context
|
||||
- runs an isolated inner tool loop
|
||||
|
||||
This is the normal specialist-agent path.
|
||||
|
||||
### Fork mode
|
||||
|
||||
Optimization path. Implemented by `run_fork_mode(...)`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- requires a `ForkContext` task-local
|
||||
- replays the parent’s exact rendered prompt and exact message prefix
|
||||
- reuses the parent’s tool schema snapshot
|
||||
- appends only the new fork task prompt
|
||||
- runs the same inner loop
|
||||
|
||||
This is for prefix-cache reuse, not for stricter isolation. It is deliberately byte-stable and closely coupled to the parent request shape.
|
||||
|
||||
## How Tool Filtering Works for Subagents
|
||||
|
||||
Typed subagents do not get a cloned tool registry. Instead the runner filters the parent’s tool list by index.
|
||||
|
||||
Filtering inputs:
|
||||
|
||||
- `definition.tools`
|
||||
- `definition.disallowed_tools`
|
||||
- `definition.skill_filter`
|
||||
- `SubagentRunOptions.skill_filter_override`
|
||||
- `definition.extra_tools`
|
||||
|
||||
Additional runtime rules:
|
||||
|
||||
- non-`welcome` subagents lose `complete_onboarding`
|
||||
- `tools_agent` strips Composio skill tools
|
||||
- `integrations_agent` with a bound toolkit may inject dynamic per-action Composio tools
|
||||
|
||||
The allowed tool names become both:
|
||||
|
||||
- the execution allowlist
|
||||
- the prompt-visible tool catalog
|
||||
|
||||
If the model emits a tool call outside that allowlist, the runner feeds back an error result and continues.
|
||||
|
||||
## Prompt Construction for Typed Subagents
|
||||
|
||||
Typed mode creates a `PromptContext` and then does one of:
|
||||
|
||||
- `PromptSource::Dynamic`: call the Rust prompt builder directly
|
||||
- `PromptSource::Inline` or `PromptSource::File`: load raw body, then wrap it with `render_subagent_system_prompt(...)`
|
||||
|
||||
Definition flags control which standard sections are omitted:
|
||||
|
||||
- `omit_identity`
|
||||
- `omit_memory_context`
|
||||
- `omit_safety_preamble`
|
||||
- `omit_skills_catalog`
|
||||
- `omit_profile`
|
||||
- `omit_memory_md`
|
||||
|
||||
This is one of the main token-saving levers in the harness.
|
||||
|
||||
## The Subagent Inner Loop
|
||||
|
||||
The actual delegated execution happens in `run_inner_loop(...)`.
|
||||
|
||||
It is a slimmed-down tool loop:
|
||||
|
||||
- call provider
|
||||
- parse tool calls
|
||||
- persist transcript after provider response
|
||||
- execute tools
|
||||
- append results
|
||||
- persist transcript again
|
||||
- stop on final text or max iterations
|
||||
|
||||
It returns:
|
||||
|
||||
- final output text
|
||||
- iteration count
|
||||
- aggregated usage
|
||||
|
||||
Unlike the parent `Agent::turn`, it does not own the broader session lifecycle.
|
||||
|
||||
## Integrations Agent Special Cases
|
||||
|
||||
`integrations_agent` is the trickiest subagent path.
|
||||
|
||||
### Toolkit gate in `spawn_subagent`
|
||||
|
||||
If `agent_id == "integrations_agent"`:
|
||||
|
||||
- `toolkit` is mandatory
|
||||
- the toolkit must exist in the allowlist
|
||||
- if it exists but is not connected, the tool returns a success message explaining that authorization is required
|
||||
|
||||
This is intentionally not always treated as a hard tool failure, because disconnected integrations are a user-facing state, not necessarily a runtime error.
|
||||
|
||||
### Text-mode override
|
||||
|
||||
In `run_inner_loop`, `integrations_agent` with tool specs forces text mode instead of native tool calling.
|
||||
|
||||
Why:
|
||||
|
||||
- large Composio JSON schemas can blow provider grammar/context limits
|
||||
|
||||
What changes:
|
||||
|
||||
- tool specs are omitted from the API payload
|
||||
- XML-style tool instructions are injected into the system prompt
|
||||
- the runner parses `<tool_call>...</tool_call>` blocks out of plain text
|
||||
- tool results in text mode are fed back as a user message containing `<tool_result>` tags
|
||||
|
||||
If a delegated integration run looks different from native-tool runs, this is usually why.
|
||||
|
||||
### Large result handoff cache
|
||||
|
||||
For toolkit-scoped `integrations_agent` runs, oversized tool results may be replaced by placeholders and stashed in an in-memory `ResultHandoffCache`.
|
||||
|
||||
The child can then call `extract_from_result(result_id, query)` to ask targeted follow-up questions against the cached payload.
|
||||
|
||||
This is not the same as generic payload summarization. It is a progressive-disclosure path specific to oversized delegated tool outputs.
|
||||
|
||||
## Parent -> Subagent -> Parent Result Shape
|
||||
|
||||
Conceptually the data flow is:
|
||||
|
||||
1. parent model emits `spawn_subagent(...)`
|
||||
2. tool runtime executes the delegated subagent loop
|
||||
3. subagent finishes with one final text output
|
||||
4. `spawn_subagent` returns that text as its tool result
|
||||
5. parent history receives the tool result
|
||||
6. parent model gets another iteration and synthesizes the user-facing answer
|
||||
|
||||
The parent does not absorb the child’s internal reasoning trace or full message history. Only the compact final output crosses the boundary.
|
||||
|
||||
## Bus Path vs Session Path
|
||||
|
||||
There are two outer entrypoints to keep straight.
|
||||
|
||||
### `Agent::turn`
|
||||
|
||||
Used for full stateful sessions. This is the richer harness.
|
||||
|
||||
### `agent.run_turn` via `src/openhuman/agent/bus.rs`
|
||||
|
||||
This native event-bus handler calls `run_tool_call_loop(...)` directly using owned Rust payloads.
|
||||
|
||||
It supports:
|
||||
|
||||
- provider reuse
|
||||
- tool filtering
|
||||
- per-turn extra tools
|
||||
- progress streaming
|
||||
|
||||
But it does not create a full `Agent` session object. If you are debugging channel-dispatch behavior, this distinction matters.
|
||||
|
||||
## Debugging Checklist
|
||||
|
||||
### 1. Confirm which execution tier you are in
|
||||
|
||||
Ask first:
|
||||
|
||||
- full `Agent::turn` session?
|
||||
- bus `agent.run_turn` path?
|
||||
- explicit `spawn_subagent` tool?
|
||||
- synthesised `delegate_*` tool leading into `spawn_subagent`?
|
||||
|
||||
If you confuse these, logs will look contradictory.
|
||||
|
||||
### 2. Check registry state
|
||||
|
||||
If delegation fails very early, confirm:
|
||||
|
||||
- `AgentDefinitionRegistry::init_global(...)` ran at startup
|
||||
- the target agent id exists
|
||||
- workspace overrides did not shadow the expected built-in definition
|
||||
|
||||
### 3. Check task-local availability
|
||||
|
||||
If `run_subagent` errors with missing context:
|
||||
|
||||
- `NoParentContext` means the tool ran outside a parent turn
|
||||
- `NoForkContext` means fork mode was requested but the fork snapshot was never installed
|
||||
|
||||
These are wiring issues, not prompt issues.
|
||||
|
||||
### 4. Check tool visibility vs tool execution
|
||||
|
||||
A tool can exist in the parent registry but still be invisible to a child due to:
|
||||
|
||||
- named `ToolScope`
|
||||
- `disallowed_tools`
|
||||
- `skill_filter`
|
||||
- welcome-only stripping
|
||||
- toolkit narrowing
|
||||
|
||||
If the model says “Unknown tool” or “not available to this sub-agent”, inspect filtering first.
|
||||
|
||||
### 5. Check transcript artifacts
|
||||
|
||||
Subagents persist transcripts per iteration using the parent session lineage plus a child session key. This is useful for debugging partial runs and crashes during tool execution.
|
||||
|
||||
Parent sessions and subagents do not write identical transcript shapes, so compare like with like.
|
||||
|
||||
### 6. Check the provider mode
|
||||
|
||||
If tool calling is malformed, verify whether the run used:
|
||||
|
||||
- native tools
|
||||
- p-format / xml instructions
|
||||
- integrations-agent text mode
|
||||
|
||||
The parser and message shape differ.
|
||||
|
||||
## Useful Log Prefixes
|
||||
|
||||
These prefixes are the most useful grep anchors:
|
||||
|
||||
- `[agent_loop]`
|
||||
- `[agent]`
|
||||
- `[tool-loop]`
|
||||
- `[spawn_subagent]`
|
||||
- `[subagent_runner]`
|
||||
- `[subagent_runner:typed]`
|
||||
- `[subagent_runner:fork]`
|
||||
- `[subagent_runner:text-mode]`
|
||||
- `[subagent_runner:handoff]`
|
||||
- `[orchestrator_tools]`
|
||||
- `[agent::bus]`
|
||||
- `[transcript]`
|
||||
|
||||
## Best Existing Tests to Read First
|
||||
|
||||
For end-to-end harness behavior:
|
||||
|
||||
- `src/openhuman/agent/harness/session/tests.rs`
|
||||
- `turn_dispatches_spawn_subagent_through_full_path`
|
||||
- `turn_dispatches_spawn_subagent_in_fork_mode`
|
||||
|
||||
For runner behavior in isolation:
|
||||
|
||||
- `src/openhuman/agent/harness/subagent_runner.rs` tests
|
||||
- typed mode returns text
|
||||
- memory-context inclusion/omission
|
||||
- tool filtering
|
||||
- one-tool execution
|
||||
- blocked tool recovery
|
||||
- fork prefix replay
|
||||
- missing parent/fork context errors
|
||||
|
||||
For orchestration-tool synthesis:
|
||||
|
||||
- `src/openhuman/tools/orchestrator_tools.rs` tests
|
||||
|
||||
For generic parent loop behavior:
|
||||
|
||||
- `src/openhuman/agent/tests.rs`
|
||||
|
||||
## Common Failure Modes
|
||||
|
||||
### Subagent never starts
|
||||
|
||||
Usually one of:
|
||||
|
||||
- registry not initialized
|
||||
- invalid `agent_id`
|
||||
- missing parent context
|
||||
- missing fork context
|
||||
|
||||
### Subagent starts but cannot call expected tools
|
||||
|
||||
Usually one of:
|
||||
|
||||
- tool filtered out by definition scope
|
||||
- `skill_filter` or toolkit override narrowed too aggressively
|
||||
- tool is `CliRpcOnly`
|
||||
- dynamic integration tools were not injected because the toolkit/client state was missing
|
||||
|
||||
### Integrations agent behaves unlike other agents
|
||||
|
||||
Usually expected. It may be in text mode and may be using the oversized-result handoff cache.
|
||||
|
||||
### Parent seems to “lose” child reasoning
|
||||
|
||||
Expected. Only the child’s final output is returned to the parent. Internal child history stays isolated.
|
||||
|
||||
## Practical Mental Model
|
||||
|
||||
The safest mental model is:
|
||||
|
||||
- the parent session is the durable conversation runtime
|
||||
- tools are the execution boundary
|
||||
- subagents are tool implementations that happen to run their own mini LLM loop
|
||||
- fork mode is a cache-optimization path, not a different product feature
|
||||
- `integrations_agent` is a special delegated runtime with extra provider and payload safeguards
|
||||
|
||||
If you debug from that model, the current codebase makes much more sense.
|
||||
@@ -1,363 +0,0 @@
|
||||
Here is the updated technical specification and architectural manual. The human-in-the-loop validation layer has been completely removed, replacing it with a fully automated, bi-directional orchestration loop where the Front-End Agent defers to the Reasoning LLM, which spawns sub-agents and routes feedback back up. Additionally, the compression matrix has been recalibrated to a **20:1 ratio**, and the Subconscious Loop now processes a cumulative **World State Diff** tracking environmental evolution from start to finish.
|
||||
|
||||
---
|
||||
|
||||
# Technical Specification & Architecture Manual: Autonomous Closed-Loop LangGraph Harness
|
||||
|
||||
This document provides a comprehensive technical specification and architectural blueprint for a stateful, split-brain multi-agent system implemented via **LangGraph**. The system decouples immediate user-facing interface management from complex task orchestration and asynchronous deep-state optimization (the "Subconscious Loop") without requiring human intervention.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architectural Philosophy & Autonomous Closed-Loop Design
|
||||
|
||||
The architecture replicates a biological sleep/wake cognitive model optimized for complete autonomy. Immediate input parsing and environmental feedback are handled by lightweight, low-latency loops. Long-term strategic alignment, memory consolidation, and system steering are offloaded to an offline, heavy-reasoning asynchronous layer triggered by a clock cycle (cron).
|
||||
|
||||
Unlike legacy systems that block execution for human verification, this topology establishes a fully automated feedback loop where the orchestrator directly replies back to the ingest surface.
|
||||
|
||||
### The Three LLM Cognitive Tiers
|
||||
|
||||
1. **Quick LLM (Surface Interface Layer):** Low context window (~8k–32k tokens), high-speed streaming optimization. Drives the Front-End Agent to manage ingestion channels, hand off macro-directives to the reasoning layer, and deliver near-instant consumer feedback once execution cycles complete.
|
||||
2. **Reasoning LLM (Execution & Orchestration Layer):** Large context window (1 Million tokens). High-capacity operational model optimized for tool call routing, state mutation planning, dynamic execution sub-agent spawning, and feedback loop compilation.
|
||||
3. **Subconscious LLM (Deep Reflection Layer):** Large context window (1 Million tokens). Extremely high-density reasoning model operating completely offline. It possesses no awareness of external networks or direct user presences. It consumes highly compressed operational traces and cumulative world state diffs to output short, dense, high-impact configuration overrides that steer the Reasoning LLM.
|
||||
|
||||
---
|
||||
|
||||
## 2. Component Topology & Structural Constraints
|
||||
|
||||
```
|
||||
[ External Channels: Telegram / Web App ]
|
||||
│ ▲
|
||||
│ (Webhooks / Events) │ (Final Streaming Response)
|
||||
▼ │
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ FRONT-END LAYER │
|
||||
│ ┌──────────────────────────────────────────────────┐ │
|
||||
│ │ FRONT-END AGENT │ │◄── Always running /
|
||||
│ │ (Quick LLM) │ │ Triggered externally
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ │ ▲ │
|
||||
│ │ Defers Macro- │ Replies │
|
||||
│ │ Instructions │ Back │
|
||||
│ ▼ │ │
|
||||
└───────────┼────────────────────────────────┼───────────┘
|
||||
│ │
|
||||
▼ │
|
||||
┌───────────┼────────────────────────────────┼───────────┐
|
||||
│ ▼ │ │
|
||||
│ ┌──────────────────────────────────────────────────┐ │
|
||||
│ │ REASONING LLM │ │◄── Context Managed
|
||||
│ │ (Orchestration Core) │ │ (80-90% Hooks)
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ├─► Spawns Autonomous Execution Sub-Agents │
|
||||
│ │ │
|
||||
│ ▼ Generates 20:1 Summary │
|
||||
│ & Historical World State Diffs │
|
||||
│ ┌──────────────────────────────────────────────────┐ │
|
||||
│ │ SUBCONSCIOUS LLM │ │
|
||||
│ │ (Asynchronous Core) │ ├─── Steers via Cron
|
||||
│ └──────────────────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────────────────────┘
|
||||
|
||||
```
|
||||
|
||||
### 2.1 The Front-End Ingestion Layer
|
||||
|
||||
- **Channels:** The boundary surface of the graph. Channels absorb heterogeneous asynchronous streams (Telegram bot long-polling/webhooks, Web App WebSocket packets) and standardize them into a singular data payload structure within the Graph State.
|
||||
- **Front-End Agent:** Driven by the **Quick LLM**, this component operates on a two-pass cycle. On intake, it translates raw channel traffic into actionable macro-instructions and defers execution downwards to the orchestration loop. On the return pass, it digests the execution responses and compiles consumer-facing streaming feedback.
|
||||
|
||||
### 2.2 Automated Loop Routing (Bi-Directional Feedback)
|
||||
|
||||
This architecture enforces an autonomous, closed control sequence that loops through the cognitive engine before resolving:
|
||||
|
||||
1. **Front-End Interfacing:** The Front-End Agent processes raw inputs, registers them into state, and yields control down the graph topology.
|
||||
2. **Orchestrated Execution & Sub-Agent Spawning:** The Reasoning LLM assumes control, references current behavioral steering profiles, and spins up ephemeral execution sub-agents to interface with tools and infrastructure.
|
||||
3. **Upstream Reporting:** Once execution bounds are reached, the Reasoning LLM synthesizes operational data and _replies back_ directly to the Front-End Agent.
|
||||
4. **Resolution:** The Front-End Agent intercepts the execution reply, constructs the finalized presentation layer payload, and streams it back to the originating communication channel.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cognitive Dynamics & Memory Lifecycle
|
||||
|
||||
### 3.1 The 20:1 Information Compression Engine
|
||||
|
||||
To preserve context capacity without omitting critical structural history, the Reasoning LLM's raw execution traces, multi-agent message logs, and sub-agent output streams are routed through an inline compression hook.
|
||||
|
||||
- This hook condenses noisy text blocks into a crisp semantic abstraction targeted at a strict **20:1 token reduction ratio** (e.g., a 20,000-token verbose sub-agent debugging trace is boiled down to a dense 1,000-token structural log entry).
|
||||
- These compressed records are periodically committed to the Subconscious memory partition along with the evolving world timeline data.
|
||||
|
||||
### 3.2 Asynchronous Steering Loop & World State Diffs
|
||||
|
||||
The Subconscious LLM executes fully decoupled from the core transaction pipeline, operating on an isolated schedule managed by system **cron jobs**.
|
||||
|
||||
- **The Agent's World State Diff:** Rather than evaluating isolated system mutations, the Subconscious loop consumes a comprehensive, cumulative structural dictionary representing the "state of the diff of the agent's world". This diff documents explicitly how the agent's internal and external environment has shifted over time from start to finish.
|
||||
- **Steering Directive Output:** It evaluates macro-trends and shifts across the world state timeline, filtering out localized operational variance. It outputs highly condensed, high-impact behavioral guidelines. These guidelines are injected directly into the **Reasoning LLM's** operational prompts for subsequent runs, modifying resource prioritization parameters and sub-agent steering traits.
|
||||
|
||||
### 3.3 Context Lifecycle Hooks (80%–90% Threshold)
|
||||
|
||||
Both the Reasoning LLM and Subconscious LLM monitor active token footprints via explicit guardrail hooks:
|
||||
|
||||
- **The Intercept Boundary:** When context window consumption trends between **80% and 90%** of total allocation, the graph routes state execution through a background truncation node.
|
||||
- **Eviction Strategy:** Older tracking logs and early world diff fragments are summarized via an autonomous map-reduce routine, pushed to a long-term Vector Database for RAG operations, and excised from the working state. The window is shifted right, maintaining core operational identities and immediate state milestones.
|
||||
|
||||
---
|
||||
|
||||
## 4. Technical Reference Implementation (LangGraph)
|
||||
|
||||
The complete, runnable Python script below demonstrates how to map out this autonomous, bi-directional topology using **LangGraph**. It removes the human gating mechanism, implements cyclic routing between the Front-End and Reasoning layers, models sub-agent spawning, and executes the out-of-band Subconscious cron process.
|
||||
|
||||
```python
|
||||
import os
|
||||
import uuid
|
||||
from typing import Annotated, Any, Dict, List, Literal, TypedDict
|
||||
from dataclasses import dataclass
|
||||
|
||||
from langgraph.graph import StateGraph, START, END
|
||||
from langgraph.graph.message import add_messages
|
||||
from openhuman.orchestration.checkpoint import SqlRunLedgerCheckpointer
|
||||
|
||||
# ==========================================
|
||||
# 1. AUTONOMOUS STATE DEFINITION
|
||||
# ==========================================
|
||||
|
||||
class SystemState(TypedDict):
|
||||
# Core Communication Channel
|
||||
messages: Annotated[list, add_messages]
|
||||
channel_source: str # "telegram" | "webapp"
|
||||
raw_channel_payload: str
|
||||
|
||||
# Bi-Directional Instruction Flow (No Human in the Loop)
|
||||
agent_instructions: str # Front-End Agent deferring to Reasoning LLM
|
||||
agent_reply: str # Reasoning LLM replying back to Front-End Agent
|
||||
channel_response: str # Final compiled channel feedback output
|
||||
|
||||
# Cognitive Engine Memory & Deep State Steering
|
||||
subconscious_steering: str # Guidelines injected by Subconscious LLM
|
||||
compressed_history: List[str] # Strict 20:1 condensed execution logs
|
||||
world_state_diff: Dict[str, Any] # Cumulative timeline tracking agent's world changes from start to finish
|
||||
context_utilization: float # Tracks 1M token context limit window (0.0 to 1.0)
|
||||
|
||||
# ==========================================
|
||||
# 2. CORE NODE IMPLEMENTATIONS
|
||||
# ==========================================
|
||||
|
||||
def channel_ingestion_node(state: SystemState) -> Dict[str, Any]:
|
||||
"""
|
||||
Acts as the entry vector. Normalizes incoming raw payloads into the unified graph state.
|
||||
"""
|
||||
print(f"[Channel Ingest] Incoming packet from source: {state.get('channel_source')}")
|
||||
return {
|
||||
"raw_channel_payload": state.get("raw_channel_payload", ""),
|
||||
"messages": [("user", state.get("raw_channel_payload", ""))]
|
||||
}
|
||||
|
||||
|
||||
def frontend_agent_node(state: SystemState) -> Dict[str, Any]:
|
||||
"""
|
||||
Driven by Quick LLM.
|
||||
Pass 1: Translates user intent into instructions and defers down to the Reasoning LLM.
|
||||
Pass 2: Receives the Reasoning LLM reply and formulates final channel feedback.
|
||||
"""
|
||||
if not state.get("agent_reply"):
|
||||
print("[Front-End Agent] Pass 1: Deferring action instructions downstream to Reasoning LLM...")
|
||||
raw_input = state.get("raw_channel_payload", "")
|
||||
return {
|
||||
"agent_instructions": f"AUTONOMOUS_EXECUTE: Process environmental signature -> '{raw_input}'"
|
||||
}
|
||||
else:
|
||||
print("[Front-End Agent] Pass 2: Processing Reasoning LLM reply to generate final channel response...")
|
||||
exec_reply = state.get("agent_reply")
|
||||
return {
|
||||
"channel_response": f"Successfully completed. Engine Output: {exec_reply}"
|
||||
}
|
||||
|
||||
|
||||
def agent_execution_node(state: SystemState) -> Dict[str, Any]:
|
||||
"""
|
||||
Driven by Reasoning LLM. Orchestrates execution, spawns sub-agents,
|
||||
applies Subconscious steering directives, and replies back upstream.
|
||||
"""
|
||||
print("[Agent Execution] Initializing orchestration core via Reasoning LLM...")
|
||||
instructions = state.get("agent_instructions")
|
||||
steering = state.get("subconscious_steering", "DEFAULT_ALIGNMENT: Maximize throughput performance.")
|
||||
|
||||
print(f" -> Injecting Subconscious Steering Guidelines: [ {steering} ]")
|
||||
print(f" -> Spawning execution sub-agents to fulfill: [ {instructions} ]")
|
||||
|
||||
# Simulate multi-agent operational activity trace
|
||||
print(" -> [Sub-Agent Alpha] Compiling environmental variables...")
|
||||
print(" -> [Sub-Agent Beta] Mutating structural matrix parameters...")
|
||||
|
||||
# 20:1 Information Compression Engine execution
|
||||
# Simulates condensing a 20,000 token verbose sub-agent log into 1,000 tokens
|
||||
simulated_20_to_1_summary = (
|
||||
"[20:1 Compression Trace] Orchestrated 2 sub-agents. State variables mutated. "
|
||||
"Sub-agent traces pruned from core context loop to maintain compliance bounds."
|
||||
)
|
||||
|
||||
# Accumulate and update the World State Diff (Tracking changes over time from start to finish)
|
||||
current_world_diff = state.get("world_state_diff", {})
|
||||
if not current_world_diff:
|
||||
current_world_diff = {
|
||||
"system_genesis": "initialized",
|
||||
"evolution_timeline": [],
|
||||
"terminal_state": "pending"
|
||||
}
|
||||
|
||||
mutation_step = len(current_world_diff["evolution_timeline"]) + 1
|
||||
current_world_diff["evolution_timeline"].append({
|
||||
"sequence": mutation_step,
|
||||
"event_signature": f"Execution Cycle {mutation_step}",
|
||||
"world_mutation": "Infrastructure parameters rewritten. Ephemeral sub-agents terminated.",
|
||||
"delta_delta": "Matrix state transitioned from idle to computed_active."
|
||||
})
|
||||
current_world_diff["terminal_state"] = "execution_finalized"
|
||||
|
||||
# Tracking Context utilization
|
||||
current_utilization = min(state.get("context_utilization", 0.1) + 0.05, 1.0)
|
||||
|
||||
return {
|
||||
"agent_reply": "Reasoning Core and spawned sub-agents finalized all pipeline mutations successfully.",
|
||||
"compressed_history": [simulated_20_to_1_summary],
|
||||
"world_state_diff": current_world_diff,
|
||||
"context_utilization": current_utilization
|
||||
}
|
||||
|
||||
|
||||
def context_manager_hook_node(state: SystemState) -> Dict[str, Any]:
|
||||
"""
|
||||
Enforces systemic context window safeguards between 80% and 90%.
|
||||
"""
|
||||
utilization = state.get("context_utilization", 0.0)
|
||||
print(f"[Context Manager Hook] Current window utilization footprint: {utilization * 100:.2f}%")
|
||||
|
||||
if utilization >= 0.85:
|
||||
print("[Context Manager Hook] CRITICAL: Context threshold breached. Evicting and shifting window...")
|
||||
return {
|
||||
"context_utilization": 0.2,
|
||||
"compressed_history": ["--- Historical context blocks compressed and evicted to Vector Store ---"]
|
||||
}
|
||||
|
||||
print("[Context Manager Hook] Memory boundaries verified clean.")
|
||||
return {}
|
||||
|
||||
|
||||
def subconscious_cron_node(state: SystemState) -> Dict[str, Any]:
|
||||
"""
|
||||
Long-running heavy reasoning block executed completely out-of-band via Cron loops.
|
||||
Evaluates compressed logs and cumulative world state diffs to construct new steering rules.
|
||||
"""
|
||||
print("[Subconscious Loop] Out-of-band Cron Trigger Activated...")
|
||||
history = state.get("compressed_history", [])
|
||||
world_diff = state.get("world_state_diff", {})
|
||||
|
||||
print(f" -> Digesting {len(history)} historical 20:1 compressed summaries...")
|
||||
print(f" -> Deep evaluation of cumulative Agent's World State Diff timeline from start to finish:")
|
||||
for step in world_diff.get("evolution_timeline", []):
|
||||
print(f" * Step [{step['sequence']}]: {step['world_mutation']} ({step['delta_delta']})")
|
||||
|
||||
new_steering_directive = "STEERING_DIRECTIVE: High asset mutability detected. Enforce stricter resource parameters."
|
||||
print(f" -> Emitting new high-density steering directive: {new_steering_directive}")
|
||||
|
||||
return {
|
||||
"subconscious_steering": new_steering_directive
|
||||
}
|
||||
|
||||
# ==========================================
|
||||
# 3. GRAPH COMPOSITION & CONDITIONAL ROUTING
|
||||
# ==========================================
|
||||
|
||||
def automated_loop_router(state: SystemState) -> Literal["agent_execution", "context_manager_hook"]:
|
||||
"""
|
||||
Directs graph traffic based on system execution tracking.
|
||||
If a final response payload exists, routes to wrap up. Otherwise, defers to execution.
|
||||
"""
|
||||
if state.get("channel_response"):
|
||||
return "context_manager_hook"
|
||||
return "agent_execution"
|
||||
|
||||
|
||||
workflow = StateGraph(SystemState)
|
||||
|
||||
# Declare nodes in processing space
|
||||
workflow.add_node("channel_ingestion", channel_ingestion_node)
|
||||
workflow.add_node("frontend_agent", frontend_agent_node)
|
||||
workflow.add_node("agent_execution", agent_execution_node)
|
||||
workflow.add_node("context_manager_hook", context_manager_hook_node)
|
||||
workflow.add_node("subconscious_cron", subconscious_cron_node)
|
||||
|
||||
# Map edge connections
|
||||
workflow.add_edge(START, "channel_ingestion")
|
||||
workflow.add_edge("channel_ingestion", "frontend_agent")
|
||||
|
||||
# Set up bi-directional feedback routing around the Front-End Agent
|
||||
workflow.add_conditional_edges(
|
||||
"frontend_agent",
|
||||
automated_loop_router,
|
||||
{
|
||||
"agent_execution": "agent_execution",
|
||||
"context_manager_hook": "context_manager_hook"
|
||||
}
|
||||
)
|
||||
|
||||
# Execution loops right back up into the Front-End Agent to communicate findings
|
||||
workflow.add_edge("agent_execution", "frontend_agent")
|
||||
workflow.add_edge("context_manager_hook", END)
|
||||
|
||||
# Compile graph with the durable run ledger checkpointer used by OpenHuman.
|
||||
checkpointer = SqlRunLedgerCheckpointer()
|
||||
compiled_autonomous_harness = workflow.compile(checkpointer=checkpointer)
|
||||
|
||||
# ==========================================
|
||||
# 4. RUNTIME WALKTHROUGH SIMULATION
|
||||
# ==========================================
|
||||
if __name__ == "__main__":
|
||||
print("--- STARTING AUTONOMOUS HARNESS GRAPH RUNTIME ---")
|
||||
thread_config = {"configurable": {"thread_id": "autonomous_session_999"}}
|
||||
|
||||
runtime_input = {
|
||||
"channel_source": "webapp",
|
||||
"raw_channel_payload": "Reallocate resource segments across network array Gamma.",
|
||||
"context_utilization": 0.1,
|
||||
"compressed_history": [],
|
||||
"world_state_diff": {}
|
||||
}
|
||||
|
||||
print("\n--- Phase 1: Streamlined Autonomous Pipeline Execution ---")
|
||||
# Flows completely through ingestion -> front-end -> execution -> front-end -> context checkout autonomously
|
||||
for event in compiled_autonomous_harness.stream(runtime_input, thread_config, stream_mode="values"):
|
||||
pass
|
||||
|
||||
# Verify final execution state output
|
||||
final_state = compiled_autonomous_harness.get_state(thread_config).values
|
||||
print(f"\nFinal Channel Output Received: '{final_state.get('channel_response')}'")
|
||||
print(f"System Context Utilization Factor: {final_state.get('context_utilization')}")
|
||||
|
||||
print("\n--- Phase 2: Isolated Out-of-Band Cron Trigger (Subconscious Alignment) ---")
|
||||
# Fetch runtime state metrics from the thread history
|
||||
active_runtime_state = compiled_autonomous_harness.get_state(thread_config).values
|
||||
|
||||
# Execute the asynchronous subconscious computation block using current state parameters
|
||||
subconscious_adjustments = subconscious_cron_node(active_runtime_state)
|
||||
|
||||
# Directly inject emitted steering guidelines back into the state layout
|
||||
compiled_autonomous_harness.update_state(
|
||||
thread_config,
|
||||
subconscious_adjustments,
|
||||
as_node="subconscious_cron"
|
||||
)
|
||||
|
||||
print("\n--- Autonomous Architecture Verified. Pipeline Clear for Deployment ---")
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Architectural Guardrails Checklist for External LLMs
|
||||
|
||||
When developing, translating, or porting this autonomous spec, verify engine patterns against these strict invariants:
|
||||
|
||||
- **[ ] Feedback Loop Continuity:** Ensure that the Routing function accurately evaluates the presence of `channel_response` or `agent_reply` to prevent infinite execution cycling between the Front-End Agent and the Reasoning LLM.
|
||||
- **[ ] 20:1 Compression Strictness:** Verify that the inline compression mechanism truncates verbose structural strings down to a 5% footprint before modifying the global history arrays to safeguard the 1-Million-token framework.
|
||||
- **[ ] Historical World Tracking:** The `world_state_diff` tracking dictionary must record structural system mutations sequentially from start to finish rather than wiping old keys on each loop iteration.
|
||||
- **[ ] Context Pre-emptibility:** Confirm that context validation checks execute _after_ execution mutations finish but _before_ the graph reaches terminal status (`END`), ensuring memory resets take effect prior to successive iterations.
|
||||
@@ -1,194 +0,0 @@
|
||||
# Dev Workflow — Autonomous Issue Crusher
|
||||
|
||||
## Vision
|
||||
|
||||
An autonomous developer agent that runs on a schedule, picks GitHub issues assigned to the user, and raises PRs automatically. Users configure everything from a settings page — repo, fork/upstream, target branch, and frequency.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Config UI ✅ DONE
|
||||
|
||||
**PR:** [tinyhumansai/openhuman#2703](https://github.com/tinyhumansai/openhuman/pull/2703)
|
||||
**Branch:** `feat/dev-workflow-panel`
|
||||
|
||||
### What was built
|
||||
- Settings panel at **Settings > Advanced > Dev Workflow**
|
||||
- Pre-checks GitHub connection via Composio `listConnections()`
|
||||
- Fetches user's repos via `composio_execute('GITHUB_LIST_REPOSITORIES_FOR_THE_AUTHENTICATED_USER')`
|
||||
- Auto-detects forks → shows upstream repo info
|
||||
- Fetches branches from upstream via `composio_execute('GITHUB_LIST_BRANCHES')`
|
||||
- Schedule presets: 30min, 1hr, 2hr, 6hr, daily
|
||||
- Config saved to localStorage (temporary — Phase 2 moves to core)
|
||||
- All UI strings internationalized via `t()` with 37 i18n keys
|
||||
|
||||
### What was also fixed
|
||||
- Updated 5 outdated Composio GitHub tool slugs in the curated catalog (`tools.rs`)
|
||||
- Removed 3 deprecated tools no longer in Composio's catalog
|
||||
- Added `listGithubRepos()` wrapper and `ComposioGithubReposResponse` types to `composioApi.ts`
|
||||
|
||||
### Files created/modified
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `app/src/components/settings/panels/DevWorkflowPanel.tsx` | Created |
|
||||
| `app/src/pages/Settings.tsx` | Route added |
|
||||
| `app/src/components/settings/panels/DeveloperOptionsPanel.tsx` | Nav link added |
|
||||
| `app/src/components/settings/hooks/useSettingsNavigation.ts` | Route type + breadcrumbs |
|
||||
| `app/src/lib/composio/composioApi.ts` | `listGithubRepos()` wrapper |
|
||||
| `app/src/lib/composio/types.ts` | `ComposioGithubRepo`, `ComposioGithubReposResponse` |
|
||||
| `app/src/lib/i18n/en.ts` + all locale chunks | i18n keys |
|
||||
| `src/openhuman/memory_sync/composio/providers/github/tools.rs` | Slug fixes |
|
||||
| `src/openhuman/memory_sync/composio/providers/github/provider.rs` | Slug refs |
|
||||
| `src/openhuman/memory_sync/composio/providers/github/sync.rs` | Comment refs |
|
||||
| `src/openhuman/memory_sync/composio/providers/github/tests.rs` | Test assertions |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Wire Config to Execution
|
||||
|
||||
**Blocked on:** [tinyhumansai/openhuman#2707](https://github.com/tinyhumansai/openhuman/pull/2707) (codegraph + skills registry)
|
||||
|
||||
### 2a. Create Dev Workflow skill definition
|
||||
|
||||
Create a bundled skill at `src/openhuman/agent/agents/dev_workflow/` (or `skills/dev_workflow/`):
|
||||
|
||||
**`skill.toml`:**
|
||||
```toml
|
||||
id = "dev_workflow"
|
||||
display_name = "Dev Workflow"
|
||||
when_to_use = "Autonomous developer — picks GitHub issues assigned to the user and raises pull requests."
|
||||
temperature = 0.3
|
||||
max_iterations = 30
|
||||
sandbox_mode = "sandboxed"
|
||||
|
||||
[model]
|
||||
hint = "coding"
|
||||
|
||||
[tools]
|
||||
named = [
|
||||
"shell", "file_read", "file_write", "git_operations",
|
||||
"grep", "glob", "list", "edit", "apply_patch",
|
||||
"web_fetch", "composio"
|
||||
]
|
||||
|
||||
[[inputs]]
|
||||
name = "repo"
|
||||
description = "Fork repo full name (e.g. user/repo)"
|
||||
required = true
|
||||
|
||||
[[inputs]]
|
||||
name = "upstream"
|
||||
description = "Upstream repo full name (e.g. org/repo)"
|
||||
required = true
|
||||
|
||||
[[inputs]]
|
||||
name = "target_branch"
|
||||
description = "Branch to raise PRs against"
|
||||
required = true
|
||||
|
||||
[[inputs]]
|
||||
name = "fork_owner"
|
||||
description = "GitHub username of the fork owner"
|
||||
required = true
|
||||
```
|
||||
|
||||
**`SKILL.md`:** (the agent's instructions)
|
||||
```markdown
|
||||
You are an autonomous developer agent. Your job is to pick a GitHub issue and deliver a PR.
|
||||
|
||||
## Per-run workflow
|
||||
|
||||
1. **Pick issue**: Use Composio GITHUB_LIST_REPOSITORY_ISSUES on the upstream repo,
|
||||
filtered to issues assigned to the user. Pick the oldest open issue with no linked PR.
|
||||
2. **Clone & branch**: Clone the fork repo. Add upstream as remote. Fetch target branch.
|
||||
Create branch `dev-workflow/<issue-number>-<slug>` off upstream's target branch.
|
||||
3. **Index**: Use codegraph_index to build a retrieval index of the repo.
|
||||
4. **Implement**: Read the issue carefully. Use codegraph_search to find relevant files.
|
||||
Implement a minimal, correct fix/feature. Follow existing code style.
|
||||
5. **Test**: Detect and run available test commands. Fix failures before proceeding.
|
||||
6. **Push**: Commit with `Fixes #<number>`. Push branch to fork remote.
|
||||
7. **Open PR**: Use Composio GITHUB_CREATE_A_PULL_REQUEST against upstream's target branch.
|
||||
Include issue link, summary, and test results.
|
||||
|
||||
## Rules
|
||||
- One PR per run. After opening the PR, stop.
|
||||
- If no suitable issue exists, exit cleanly.
|
||||
- Never force-push. Never push to upstream directly.
|
||||
- If too large/risky, comment on the issue and skip.
|
||||
```
|
||||
|
||||
### 2b. Add `cron_add` RPC
|
||||
|
||||
**Not blocked on #2707** — can be done now.
|
||||
|
||||
The `cron_add` logic exists in the agent tool (`src/openhuman/cron/tools/add.rs`) but isn't exposed as an RPC controller. Need to add it to `src/openhuman/cron/schemas.rs`.
|
||||
|
||||
**Changes:**
|
||||
- `src/openhuman/cron/schemas.rs` — Add `"add"` controller with inputs: `name`, `schedule`, `prompt`, `session_target`, `model`, `agent_id`, `delivery`, `delete_after_run`
|
||||
- `app/src/utils/tauriCommands/cron.ts` — Add `openhumanCronAdd()` wrapper
|
||||
- `app/src/utils/tauriCommands/index.ts` — Re-export
|
||||
|
||||
### 2c. Wire Save → Cron + Skill
|
||||
|
||||
Update `DevWorkflowPanel.tsx` save handler:
|
||||
- When user clicks "Save Configuration" → call `openhumanCronAdd()` with:
|
||||
- `name: "dev-workflow-<repo>"`
|
||||
- `schedule: { kind: 'cron', expr: selectedSchedule }`
|
||||
- `agent_id: "dev_workflow"`
|
||||
- `prompt: "Run dev_workflow for repo ${repo}. Upstream: ${upstream}. Target: ${branch}. Fork owner: ${forkOwner}."`
|
||||
- `delivery: { mode: 'proactive' }`
|
||||
- Remove localStorage — config lives in the cron job
|
||||
- Show active cron job status instead of localStorage-based summary
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Execution Polish
|
||||
|
||||
### Agent execution improvements
|
||||
- Smarter issue picking: filter by labels, skip issues with existing PRs
|
||||
- Git worktree support: don't pollute the user's working tree
|
||||
- Codegraph integration: use `codegraph_index` + `codegraph_search` for file discovery
|
||||
- Test detection: auto-detect `npm test`, `cargo test`, `pytest`, etc.
|
||||
|
||||
### Error handling
|
||||
- Retry on transient failures (network, rate limits)
|
||||
- Skip issues that are too complex (>N files changed)
|
||||
- Comment on skipped issues explaining why
|
||||
|
||||
### Delivery & notifications
|
||||
- `delivery.mode = 'proactive'` → shows in the active chat channel
|
||||
- Notification when PR is opened
|
||||
- Summary of what was done
|
||||
|
||||
### UI enhancements
|
||||
- Show run history in the Dev Workflow panel
|
||||
- Show active/paused status
|
||||
- Manual trigger button ("Run now")
|
||||
- View logs from past runs
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
Phase 1 (Config UI) ──── ✅ DONE
|
||||
│
|
||||
▼
|
||||
Phase 2a (Skill def) ──── Blocked on #2707
|
||||
Phase 2b (cron_add RPC) ──── NOT blocked, can start now
|
||||
│
|
||||
▼
|
||||
Phase 2c (Wire save) ──── Blocked on 2a + 2b
|
||||
│
|
||||
▼
|
||||
Phase 3 (Polish) ──── Blocked on Phase 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related PRs & Issues
|
||||
|
||||
| Item | Link | Status |
|
||||
|------|------|--------|
|
||||
| Config UI | [openhuman#2703](https://github.com/tinyhumansai/openhuman/pull/2703) | Open (draft) |
|
||||
| Codegraph + Skills | [openhuman#2707](https://github.com/tinyhumansai/openhuman/pull/2707) | Open (draft) |
|
||||
| Backend repo endpoint | `tinyhumansai/backend#842` | Open |
|
||||
@@ -1,273 +0,0 @@
|
||||
# E2E Test Suite Status
|
||||
|
||||
Living tracking document for the OpenHuman E2E test suite. Updated whenever
|
||||
specs are added, fixed, or start failing.
|
||||
|
||||
**Last updated:** 2026-05-20
|
||||
**Total specs:** 66 (11 categories)
|
||||
**Runner:** WDIO + Appium Chromium on the CEF desktop binary
|
||||
|
||||
---
|
||||
|
||||
## Suite health overview
|
||||
|
||||
| Category | Specs | Known issues |
|
||||
|---------------|-------|--------------|
|
||||
| auth | 6 | Hardcoded pauses replaced with condition waits (2026-05-20) |
|
||||
| navigation | 6 | channels-smoke and insights-dashboard are shallow/smoke only |
|
||||
| chat | 10 | chat-harness-wallet-flow has 6 sequential 30s waits |
|
||||
| skills | 6 | skill-execution-flow is RC-7 (ghost RPCs); 4 specs are shallow stubs |
|
||||
| notifications | 4 | memory-roundtrip has async indexing race |
|
||||
| webhooks | 5 | webhooks-ingress-flow missing payload delivery assertion |
|
||||
| providers | 8 | telegram-flow is describe.skip; gmail/slack/whatsapp miss multi-account |
|
||||
| payments | 4 | rewards-progression-persistence has hardcoded pauses |
|
||||
| settings | 7 | settings-ai-skills uses OR-chain assertions |
|
||||
| system | 4+1L | local-model-runtime is describe.skip; voice-mode has hardcoded pauses |
|
||||
| journeys | 3 | All moderate depth |
|
||||
|
||||
L = Linux-only spec
|
||||
|
||||
---
|
||||
|
||||
## How to update this document
|
||||
|
||||
- **Adding a spec**: add it to the coverage matrix below and to `e2e-run-all-flows.sh`
|
||||
- **Fixing an issue**: strike through the entry or remove it from Known Issues
|
||||
- **A spec starts failing**: add it to the Known Issues section with severity + status tag
|
||||
- **Pre-flight check**: `bash app/scripts/e2e-preflight.sh`
|
||||
|
||||
---
|
||||
|
||||
## Coverage matrix
|
||||
|
||||
### Auth (6 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| smoke.spec.ts | Harness bootstrap, app loads | deep | |
|
||||
| login-flow.spec.ts | Deep-link auth → onboarding → home | deep | |
|
||||
| auth-access-control.spec.ts | Billing dashboard handoff | moderate | Previously had hardcoded 5s/8s pauses — replaced 2026-05-20 |
|
||||
| logout-relogin-onboarding.spec.ts | Logout + re-login round-trip | moderate | |
|
||||
| onboarding-modes.spec.ts | Onboarding step sequence | moderate | config.toml write race on slow CI |
|
||||
| runtime-picker-login.spec.ts | Core mode selection + login | moderate | Deep-link bootstrap race |
|
||||
|
||||
### Navigation (6 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| navigation.spec.ts | Tab bar + route rendering | deep | |
|
||||
| navigation-smoothness.spec.ts | Transition timing | moderate | |
|
||||
| navigation-settings-panels.spec.ts | Settings panel routing | moderate | |
|
||||
| command-palette.spec.ts | Command search | moderate | |
|
||||
| channels-smoke.spec.ts | Channels surface mount | shallow | No channel feature validation |
|
||||
| insights-dashboard.spec.ts | Insights panel | shallow | No data validation |
|
||||
|
||||
### Chat (10 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| chat-harness-send-stream.spec.ts | Send → SSE stream → UI render | deep | |
|
||||
| chat-harness-cancel.spec.ts | Cancel mid-stream | deep | |
|
||||
| chat-harness-scroll-render.spec.ts | Scroll + render correctness | moderate | |
|
||||
| chat-harness-subagent.spec.ts | Subagent invocation | moderate | |
|
||||
| chat-harness-wallet-flow.spec.ts | Chat + wallet state | moderate | 6 sequential 30s waits; should use condition waits |
|
||||
| chat-tool-call-flow.spec.ts | Function calling roundtrip | deep | |
|
||||
| chat-multi-tool-round.spec.ts | Multi-turn tool loop | deep | |
|
||||
| chat-tool-error-recovery.spec.ts | Tool error handling | deep | |
|
||||
| agent-review.spec.ts | Agent review + feedback | moderate | |
|
||||
| mega-flow.spec.ts | Full journey (auth/oauth/chat/logout) | deep | |
|
||||
|
||||
### Skills (6 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| skills-registry.spec.ts | Install from URL | moderate | Post-install state not verified |
|
||||
| skill-execution-flow.spec.ts | Ghost RPCs (RC-7) | skipped | **[RC-7 OPEN]** Runtime removed; spec calls non-existent RPC methods |
|
||||
| skill-lifecycle.spec.ts | /skills page loads | shallow | No feature validation beyond page mount |
|
||||
| skill-multi-round.spec.ts | /chat page loads | shallow | No multi-round skill behavior tested |
|
||||
| skill-oauth.spec.ts | /skills page loads | shallow | No OAuth flow tested |
|
||||
| skill-socket-reconnect.spec.ts | Home page loads | shallow | No socket reconnect behavior tested |
|
||||
|
||||
### Notifications (4 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| notifications.spec.ts | Ingest + list + mark-read + UI | deep | |
|
||||
| memory-roundtrip.spec.ts | Doc store + cross-namespace recall | moderate | Async indexing race on slow CI |
|
||||
| cron-jobs-flow.spec.ts | Job creation UI | moderate | |
|
||||
| autocomplete-flow.spec.ts | Chat autocomplete | shallow | |
|
||||
|
||||
### Webhooks & Tools (5 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| webhooks-ingress-flow.spec.ts | RPC endpoints + debug panel | moderate | No actual payload delivery assertion |
|
||||
| webhooks-tunnel-flow.spec.ts | Tunneling | moderate | |
|
||||
| tool-browser-flow.spec.ts | Browser tool | moderate | |
|
||||
| tool-filesystem-flow.spec.ts | Filesystem security | deep | |
|
||||
| tool-shell-git-flow.spec.ts | Shell + git | moderate | |
|
||||
|
||||
### Providers (8 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| telegram-flow.spec.ts | Telegram integration | skipped | **[SKIPPED OPEN]** describe.skip — no replacement spec |
|
||||
| gmail-flow.spec.ts | Gmail OAuth | moderate | Token refresh path untested |
|
||||
| accounts-provider-modal.spec.ts | Account connection modal | moderate | |
|
||||
| slack-flow.spec.ts | Slack OAuth + Redux state | moderate | Multi-account scenario untested |
|
||||
| whatsapp-flow.spec.ts | WhatsApp OAuth + state | moderate | Multi-account scenario untested |
|
||||
| notion-flow.spec.ts | Notion OAuth | moderate | Scope upgrade path untested |
|
||||
| conversations-web-channel-flow.spec.ts | Web channel messaging | moderate | Linux skip reason is stale |
|
||||
| composio-triggers-flow.spec.ts | Trigger enable/disable + UI | moderate | No trigger event delivery tested |
|
||||
|
||||
### Payments (4 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| card-payment-flow.spec.ts | Card payment + error handling | moderate | |
|
||||
| crypto-payment-flow.spec.ts | Crypto payment | moderate | |
|
||||
| rewards-unlock-flow.spec.ts | Rewards unlock | moderate | |
|
||||
| rewards-progression-persistence.spec.ts | Rewards persistence | moderate | Hardcoded pauses; should use condition waits |
|
||||
|
||||
### Settings (7 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| settings-channels-permissions.spec.ts | Channels + privacy settings | moderate | |
|
||||
| settings-data-management.spec.ts | Data management | moderate | |
|
||||
| settings-dev-options.spec.ts | Developer options | moderate | |
|
||||
| settings-ai-skills.spec.ts | LLM config | shallow | OR-chain assertions (passes if any one LLM panel is present) |
|
||||
| settings-account-preferences.spec.ts | Account preferences | moderate | |
|
||||
| settings-advanced-config.spec.ts | Advanced config | moderate | |
|
||||
| settings-feature-preferences.spec.ts | Feature toggles | moderate | |
|
||||
|
||||
### System (6 specs + 1 Linux-only)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| local-model-runtime.spec.ts | Ollama integration | skipped | **[SKIPPED OPEN]** describe.skip |
|
||||
| voice-mode.spec.ts | Voice I/O | shallow | Hardcoded pauses |
|
||||
| screen-intelligence.spec.ts | Screen awareness | shallow | |
|
||||
| audio-toolkit-flow.spec.ts | Audio toolkit | shallow | |
|
||||
| tauri-commands.spec.ts | Tauri IPC surface | moderate | |
|
||||
| service-connectivity-flow.spec.ts | Service discovery | moderate | Requires OPENHUMAN_SERVICE_MOCK=1 |
|
||||
| linux-cef-deb-runtime.spec.ts | Linux /usr/bin path | moderate | Linux only |
|
||||
|
||||
### User Journeys (3 specs)
|
||||
|
||||
| Spec | Feature covered | Coverage depth | Known issues |
|
||||
|------|----------------|----------------|--------------|
|
||||
| user-journey-full-task.spec.ts | Task completion end-to-end | moderate | |
|
||||
| user-journey-settings-round-trip.spec.ts | Settings persistence round-trip | moderate | |
|
||||
| chat-conversation-history.spec.ts | Conversation history | moderate | |
|
||||
|
||||
---
|
||||
|
||||
## Known Issues
|
||||
|
||||
| ID | Spec | Severity | Status | Description |
|
||||
|----|------|----------|--------|-------------|
|
||||
| RC-7 | skill-execution-flow.spec.ts | HIGH | **[RC-7 OPEN]** | Calls RPC methods that were removed when the QuickJS runtime was stripped. Spec will ghost-fail silently until updated or deleted. |
|
||||
| SKIP-1 | telegram-flow.spec.ts | MEDIUM | **[SKIPPED OPEN]** | Entire suite is `describe.skip`. No replacement coverage. |
|
||||
| SKIP-2 | local-model-runtime.spec.ts | LOW | **[SKIPPED OPEN]** | Entire suite is `describe.skip`. Ollama is optional — acceptable. |
|
||||
| RACE-1 | memory-roundtrip.spec.ts | LOW | **[RACE]** | Async indexing race on slow CI machines. Intermittent. |
|
||||
| RACE-2 | onboarding-modes.spec.ts | LOW | **[RACE]** | config.toml write race during core restart. Intermittent. |
|
||||
| SHALLOW-1 | skill-lifecycle.spec.ts | MEDIUM | **[SHALLOW]** | Only asserts page mount, not any skill lifecycle behavior. |
|
||||
| SHALLOW-2 | skill-multi-round.spec.ts | MEDIUM | **[SHALLOW]** | Only asserts /chat page loads. |
|
||||
| SHALLOW-3 | skill-oauth.spec.ts | MEDIUM | **[SHALLOW]** | Only asserts /skills page loads. No OAuth. |
|
||||
| SHALLOW-4 | skill-socket-reconnect.spec.ts | MEDIUM | **[SHALLOW]** | Only asserts home page loads. No socket reconnect. |
|
||||
| PAUSE-1 | chat-harness-wallet-flow.spec.ts | LOW | **[PAUSE]** | Six sequential `browser.pause(30_000)` calls. Should be replaced with condition waits. |
|
||||
| PAUSE-2 | rewards-progression-persistence.spec.ts | LOW | **[PAUSE]** | Hardcoded pauses. Should be replaced with condition waits. |
|
||||
| PAUSE-3 | voice-mode.spec.ts | LOW | **[PAUSE]** | Hardcoded pauses in voice I/O flow. |
|
||||
| STALE-1 | conversations-web-channel-flow.spec.ts | LOW | **[STALE]** | Linux skip condition uses a reason that no longer applies. |
|
||||
| ASSERT-1 | settings-ai-skills.spec.ts | LOW | **[SHALLOW]** | OR-chain assertions: passes if any one LLM provider panel is present. |
|
||||
|
||||
---
|
||||
|
||||
## Mock API behavior flags
|
||||
|
||||
These flags are set via `setMockBehavior(key, value)` from `mock-server.ts` and
|
||||
control the shared mock backend at `http://127.0.0.1:18473`.
|
||||
|
||||
| Flag | Type | Description |
|
||||
|------|------|-------------|
|
||||
| `seed` | string | Fuzzy randomization seed for mock data generation |
|
||||
| `forceError503` | `'true'` / `'false'` | Force HTTP 503 on all non-admin endpoints |
|
||||
| `llmStreamScript` | JSON string | Custom LLM response delta sequence. Array of `{delta: string}` objects |
|
||||
| `composioConnections` | JSON string | Override Composio connections list (e.g. `'[]'` for empty) |
|
||||
| `composioAvailableTriggers` | JSON string | Override available triggers returned by the API |
|
||||
| `composioActiveTriggers` | JSON string | Override active triggers state |
|
||||
| `purchaseError` | string | Trigger payment failure (value becomes the error message) |
|
||||
| `plan` | `'FREE'` / `'BASIC'` / `'PRO'` | Override the billing plan returned by `/settings` |
|
||||
| `planActive` | `'true'` / `'false'` | Override whether the plan is active |
|
||||
| `planExpiry` | ISO date string | Override the plan expiry date |
|
||||
| `session` | `'revoked'` / `'active'` | Force 401 on auth endpoints when set to `'revoked'` |
|
||||
|
||||
Reset all flags to defaults: `resetMockBehavior()`.
|
||||
|
||||
---
|
||||
|
||||
## How to run
|
||||
|
||||
```bash
|
||||
# Full suite (all 66 specs)
|
||||
bash app/scripts/e2e-run-all-flows.sh
|
||||
|
||||
# Single suite category
|
||||
bash app/scripts/e2e-run-all-flows.sh --suite chat
|
||||
|
||||
# Stop after first failure
|
||||
bash app/scripts/e2e-run-all-flows.sh --bail
|
||||
|
||||
# Single spec (fastest iteration)
|
||||
bash app/scripts/e2e-run-session.sh test/e2e/specs/smoke.spec.ts smoke
|
||||
|
||||
# Pre-flight check only
|
||||
bash app/scripts/e2e-preflight.sh
|
||||
|
||||
# With Appium/WDIO debug output
|
||||
WDIO_LOG_LEVEL=debug bash app/scripts/e2e-run-all-flows.sh --suite auth
|
||||
|
||||
# Skip preflight (e.g. in CI where it ran as a separate step)
|
||||
bash app/scripts/e2e-run-all-flows.sh --skip-preflight
|
||||
|
||||
# Use the debug runner (summary output + log tee)
|
||||
pnpm debug e2e test/e2e/specs/smoke.spec.ts
|
||||
pnpm debug e2e test/e2e/specs/notifications.spec.ts notifications --verbose
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How to add a new spec
|
||||
|
||||
1. **Create the spec file** in `app/test/e2e/specs/YOUR-SPEC.spec.ts`.
|
||||
|
||||
2. **Scaffold the harness:**
|
||||
```typescript
|
||||
import { resetApp } from '../helpers/reset-app';
|
||||
import { startMockServer, stopMockServer } from '../mock-server';
|
||||
|
||||
describe('Your feature', () => {
|
||||
before(async () => {
|
||||
await startMockServer();
|
||||
await resetApp('e2e-your-spec');
|
||||
});
|
||||
after(async () => {
|
||||
await stopMockServer();
|
||||
});
|
||||
|
||||
it('does the thing', async () => { /* ... */ });
|
||||
});
|
||||
```
|
||||
|
||||
3. **Register in the orchestrator** — add a `run(...)` call in the correct
|
||||
suite section of `app/scripts/e2e-run-all-flows.sh`.
|
||||
|
||||
4. **Add to this tracking doc** — add a row to the coverage matrix table
|
||||
for the appropriate category with an honest coverage depth.
|
||||
|
||||
5. **Add any new RPC methods** to `REQUIRED_RPC_METHODS` in
|
||||
`app/test/e2e/helpers/rpc-preflight.ts` if the spec calls RPC methods
|
||||
not already listed there.
|
||||
|
||||
6. **Run pre-flight** before executing: `bash app/scripts/e2e-preflight.sh`.
|
||||
@@ -1,53 +0,0 @@
|
||||
# Inference Provider Catalog
|
||||
|
||||
This document tracks the built-in BYOK inference-provider presets exposed in
|
||||
Settings > AI. OpenHuman still supports arbitrary OpenAI-compatible endpoints
|
||||
through a custom provider entry; this table is the first-class chip catalog.
|
||||
|
||||
## Phase 1 Presets
|
||||
|
||||
| Slug | Label | Endpoint | Auth style | Status |
|
||||
| ------------------- | ----------------- | --------------------------------------------------------- | ---------- | ------- |
|
||||
| `openai` | OpenAI | `https://api.openai.com/v1` | bearer | Shipped |
|
||||
| `anthropic` | Anthropic | `https://api.anthropic.com/v1` | anthropic | Shipped |
|
||||
| `openrouter` | OpenRouter | `https://openrouter.ai/api/v1` | bearer | Shipped |
|
||||
| `orcarouter` | OrcaRouter | `https://api.orcarouter.ai/v1` | bearer | Shipped |
|
||||
| `gmi` | GMI | `https://api.gmi-serving.com/v1` | bearer | Shipped |
|
||||
| `fireworks` | Fireworks | `https://api.fireworks.ai/inference/v1` | bearer | Shipped |
|
||||
| `moonshot` | Kimi (Moonshot) | `https://api.moonshot.ai/v1` | bearer | Shipped |
|
||||
| `groq` | Groq | `https://api.groq.com/openai/v1` | bearer | Shipped |
|
||||
| `mistral` | Mistral | `https://api.mistral.ai/v1` | bearer | Shipped |
|
||||
| `deepseek` | DeepSeek | `https://api.deepseek.com/v1` | bearer | Shipped |
|
||||
| `together` | Together AI | `https://api.together.xyz/v1` | bearer | Shipped |
|
||||
| `google` | Google Gemini | `https://generativelanguage.googleapis.com/v1beta/openai` | bearer | Shipped |
|
||||
| `cerebras` | Cerebras | `https://api.cerebras.ai/v1` | bearer | Shipped |
|
||||
| `xai` | xAI | `https://api.x.ai/v1` | bearer | Shipped |
|
||||
| `huggingface` | Hugging Face | `https://router.huggingface.co/v1` | bearer | Shipped |
|
||||
| `nvidia` | NVIDIA | `https://integrate.api.nvidia.com/v1` | bearer | Shipped |
|
||||
| `zai` | Z.AI | `https://api.z.ai/api/paas/v4` | bearer | Shipped |
|
||||
| `minimax` | MiniMax | `https://api.minimax.io/v1` | bearer | Shipped |
|
||||
| `stepfun` | StepFun | `https://api.stepfun.ai/step_plan/v1` | bearer | Shipped |
|
||||
| `kilocode` | Kilo Code | `https://api.kilo.ai/api/gateway` | bearer | Shipped |
|
||||
| `deepinfra` | DeepInfra | `https://api.deepinfra.com/v1/openai` | bearer | Shipped |
|
||||
| `novita` | Novita | `https://api.novita.ai/v3/openai` | bearer | Shipped |
|
||||
| `venice` | Venice | `https://api.venice.ai/api/v1` | bearer | Shipped |
|
||||
| `vercel-ai-gateway` | Vercel AI Gateway | `https://ai-gateway.vercel.sh/v1` | bearer | Shipped |
|
||||
| `sumopod` | SumoPod | `https://ai.sumopod.com/v1` | bearer | Shipped |
|
||||
| `modelscope` | ModelScope | `https://api-inference.modelscope.cn/v1` | bearer | Shipped |
|
||||
|
||||
API keys are stored through the auth-profile store under `provider:<slug>`;
|
||||
they are not read from environment variables by the desktop Settings flow.
|
||||
|
||||
## Deferred Transports
|
||||
|
||||
The following provider families need transport or credential work beyond a
|
||||
static OpenAI-compatible or Anthropic-compatible preset:
|
||||
|
||||
| Provider / family | Needed work |
|
||||
| ------------------------------------------------------ | ---------------------------------------------------------------------- |
|
||||
| OpenAI Codex / ChatGPT subscription | Continue extending existing `openai_oauth` support. |
|
||||
| xAI OAuth, MiniMax OAuth, Qwen OAuth, Nous device-code | Add provider-specific OAuth or import flows. |
|
||||
| Google Gemini CLI, Claude CLI, Copilot ACP | Add subprocess or external-process transports. |
|
||||
| AWS Bedrock Converse | Add native AWS SDK / Bedrock Converse transport. |
|
||||
| Azure Foundry | Model dual OpenAI / Anthropic API modes and Azure credential handling. |
|
||||
| GitHub Copilot | Add GitHub-token import and provider transport. |
|
||||
@@ -1,357 +0,0 @@
|
||||
# Memory Module — Consumer Reference
|
||||
|
||||
How to use the memory layer from services outside `src/openhuman/memory/`.
|
||||
|
||||
**Rule**: Always use `MemoryClient` (`src/openhuman/memory/store/client.rs`). Never call `UnifiedMemory` directly — it's internal to the memory module.
|
||||
|
||||
---
|
||||
|
||||
## Which Function to Use
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```text
|
||||
Need to store data?
|
||||
|
|
||||
+-- Structured key-value pair (config, state, counters)?
|
||||
| -> kv_set()
|
||||
|
|
||||
+-- Full document (text, sync payload, user content)?
|
||||
| |
|
||||
| +-- Ephemeral / high-frequency (screen captures, ticks)?
|
||||
| | -> put_doc_light() (no embedding, no graph extraction)
|
||||
| |
|
||||
| +-- Important content that should be semantically searchable?
|
||||
| | -> put_doc() (embeds + background graph extraction)
|
||||
| |
|
||||
| +-- Rich content needing entity/relation extraction NOW?
|
||||
| -> ingest_doc() (full synchronous pipeline, slower)
|
||||
|
|
||||
+-- Knowledge graph fact (entity-relation-entity)?
|
||||
-> graph_upsert()
|
||||
|
||||
Need to read data?
|
||||
|
|
||||
+-- Have a user query / search string?
|
||||
| -> query_namespace() (returns text for LLM prompt)
|
||||
| -> query_namespace_context_data() (returns structured data)
|
||||
|
|
||||
+-- Need recent context, no specific query?
|
||||
| -> recall_namespace() (returns text)
|
||||
| -> recall_namespace_context_data() (returns structured data)
|
||||
| -> recall_namespace_memories() (returns individual hits)
|
||||
|
|
||||
+-- Looking up a specific key?
|
||||
| -> kv_get()
|
||||
|
|
||||
+-- Listing what exists?
|
||||
| -> list_documents(), list_namespaces(), kv_list_namespace()
|
||||
|
|
||||
+-- Querying entity relationships?
|
||||
-> graph_query()
|
||||
|
||||
Need to delete data?
|
||||
|
|
||||
+-- Single document? -> delete_document()
|
||||
+-- Single KV entry? -> kv_delete()
|
||||
+-- All data for a skill (e.g. on disconnect/revoke)? -> clear_skill_memory()
|
||||
+-- All data in namespace? -> clear_namespace()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Writing Data
|
||||
|
||||
#### `put_doc()` — General-purpose document storage
|
||||
|
||||
Use when content should be **semantically searchable** later. Embeds the content and enqueues background graph extraction.
|
||||
|
||||
```rust
|
||||
client.put_doc(NamespaceDocumentInput {
|
||||
namespace: "autocomplete-memory".into(),
|
||||
key: Some(format!("completion:{timestamp:018}")),
|
||||
title: "Accepted completion".into(),
|
||||
content: format!("{context}\n---\n{suggestion}"),
|
||||
source_type: Some("autocomplete".into()),
|
||||
metadata: Some(json!({ "app_name": app, "timestamp_ms": ts })),
|
||||
..Default::default()
|
||||
}).await?;
|
||||
```
|
||||
|
||||
**Used by**: Skills JS `memory.insert()`, Autocomplete (searchable completions), Subconscious (working-memory docs).
|
||||
|
||||
#### `put_doc_light()` — High-frequency / ephemeral storage
|
||||
|
||||
Use when data is **written often** and doesn't need embedding or graph extraction. Much faster than `put_doc()`.
|
||||
|
||||
```rust
|
||||
client.put_doc_light(NamespaceDocumentInput {
|
||||
namespace: "vision".into(),
|
||||
key: Some(format!("screen_intelligence_{id}")),
|
||||
title: format!("Screen: {app_name} — {window_title}"),
|
||||
content: yaml_frontmatter_and_text,
|
||||
..Default::default()
|
||||
}).await?;
|
||||
```
|
||||
|
||||
**Used by**: Screen Intelligence (vision summaries every few seconds).
|
||||
|
||||
#### `ingest_doc()` — Full synchronous ingestion
|
||||
|
||||
Use when you need the complete pipeline (chunking, embedding, entity extraction, relation extraction) to **finish before proceeding**. Blocks until done. Prefer `put_doc()` unless you need synchronous guarantees.
|
||||
|
||||
**Used by**: Skills event loop (ingesting sync content into vector graph).
|
||||
|
||||
#### `kv_set()` — Structured key-value storage
|
||||
|
||||
Use for **small, structured data** you'll look up by exact key. Not semantically searchable.
|
||||
|
||||
```rust
|
||||
client.kv_set(
|
||||
Some("autocomplete"), // namespace (None = global)
|
||||
&format!("accepted:{ts:018}"), // key
|
||||
&json!({ "context": ctx, "suggestion": s }),
|
||||
).await?;
|
||||
```
|
||||
|
||||
**Used by**: Autocomplete (completion records keyed by timestamp).
|
||||
|
||||
#### `graph_upsert()` — Knowledge graph facts
|
||||
|
||||
Use to store **entity-relation-entity triples**. You rarely call this directly — `put_doc()` and `ingest_doc()` extract graph relations automatically. Only call `graph_upsert()` when you have an explicit fact without an associated document.
|
||||
|
||||
---
|
||||
|
||||
### Reading Data
|
||||
|
||||
#### `query_namespace()` — Semantic search (text)
|
||||
|
||||
Use when you have a **search string** and want relevant content. Returns formatted text ready for LLM prompt injection.
|
||||
|
||||
```rust
|
||||
let context = client.query_namespace(
|
||||
"autocomplete-memory", // namespace
|
||||
&last_80_chars, // query
|
||||
10, // max chunks
|
||||
).await?;
|
||||
```
|
||||
|
||||
**Used by**: Autocomplete (matching past completions to current context), Frontend (user-initiated search in MemoryWorkspace).
|
||||
|
||||
#### `query_namespace_context_data()` — Semantic search (structured)
|
||||
|
||||
Same query, but returns `NamespaceRetrievalContext` with individual hits, entities, relations, and score breakdowns. Use when you need to process results programmatically.
|
||||
|
||||
#### `recall_namespace()` — Recent context (text)
|
||||
|
||||
Use when you need **recent memory** but don't have a query. Returns most recent docs as formatted text.
|
||||
|
||||
```rust
|
||||
if let Ok(Some(text)) = client.recall_namespace(&ns, 3).await {
|
||||
// top 3 chunks of recent context
|
||||
}
|
||||
```
|
||||
|
||||
**Used by**: Subconscious (fetching context per namespace for situation report).
|
||||
|
||||
#### `recall_namespace_memories()` — Recent context (individual hits)
|
||||
|
||||
Returns `Vec<NamespaceMemoryHit>` instead of text. Use when you need to iterate hits and inspect scores.
|
||||
|
||||
#### `recall_namespace_context_data()` — Recent context (structured)
|
||||
|
||||
Returns `NamespaceRetrievalContext`. Use when you need the full retrieval object (hits + entities + relations).
|
||||
|
||||
#### `kv_get()` — Exact key lookup
|
||||
|
||||
```rust
|
||||
let val = client.kv_get(Some("autocomplete"), "accepted:000001719000000").await?;
|
||||
```
|
||||
|
||||
#### `kv_list_namespace()` — List all KV entries
|
||||
|
||||
Enumerate all key-value pairs in a namespace. Useful for trimming, history display, or bulk operations.
|
||||
|
||||
**Used by**: Autocomplete (trimming beyond 50 entries, settings UI, bulk clear).
|
||||
|
||||
#### `list_documents()` — List document metadata
|
||||
|
||||
Returns JSON with document metadata (not full content). Useful for delta analysis or trimming.
|
||||
|
||||
**Used by**: Subconscious (finding docs updated since last tick), Autocomplete (trimming beyond 200 docs).
|
||||
|
||||
#### `graph_query()` — Query knowledge graph
|
||||
|
||||
Find entity relationships. Filter by optional namespace, subject, and/or predicate.
|
||||
|
||||
```rust
|
||||
let relations = client.graph_query(None, None, None).await?;
|
||||
```
|
||||
|
||||
**Used by**: Subconscious (relations for situation report), Frontend (knowledge graph display).
|
||||
|
||||
---
|
||||
|
||||
### Deleting Data
|
||||
|
||||
| Function | When to use |
|
||||
|----------|-------------|
|
||||
| `delete_document(namespace, id)` | Remove a specific document |
|
||||
| `kv_delete(namespace, key)` | Remove a specific KV entry |
|
||||
| `clear_skill_memory(skill_id, integration_id)` | Disconnect / revoke: clears skill-scoped memory in the shared `skill-{skill_id}` namespace. Storage is not isolated per integration—multiple integrations share that namespace; `integration_id` identifies the integration in the API contract (see implementation in `MemoryClient::clear_skill_memory`) |
|
||||
| `clear_namespace(namespace)` | Wipe an arbitrary namespace |
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns
|
||||
|
||||
**Fire-and-forget writes** — Spawn a background task to avoid blocking:
|
||||
|
||||
```rust
|
||||
let client = memory_client.clone();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = client.put_doc(input).await {
|
||||
log::warn!("memory write failed: {e}");
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Trim-after-write** — Cap growth after each insert (e.g., max 50 KV entries, max 200 docs). Use `kv_list_namespace()` or `list_documents()` then delete oldest.
|
||||
|
||||
**Init on app startup** — Frontend calls `syncMemoryClientToken()` in `CoreStateProvider.tsx` on mount and token refresh. Memory subsystem must be initialized before queries run.
|
||||
|
||||
---
|
||||
|
||||
## Namespace Convention
|
||||
|
||||
| Namespace | Owner | Description |
|
||||
|-----------|-------|-------------|
|
||||
| `skill-{skill_id}` | Skills event loop | Raw state blobs + per-page docs (e.g., `skill-notion`, `skill-gmail`) |
|
||||
| `global` | Skills working_memory.rs | Extracted user facts (preferences, goals, entities) |
|
||||
| `conversations` | Agent/inference | Conversation context |
|
||||
| `autocomplete` | Autocomplete | Accepted completion records (KV) |
|
||||
| `autocomplete-memory` | Autocomplete | Searchable completion documents |
|
||||
| `vision` | Screen intelligence | Vision summaries from screen captures |
|
||||
|
||||
---
|
||||
|
||||
## MemoryClient API Reference
|
||||
|
||||
**File**: `src/openhuman/memory/store/client.rs`
|
||||
|
||||
### Documents
|
||||
|
||||
| Function | Line | Type | Description |
|
||||
|----------|------|------|-------------|
|
||||
| `put_doc(NamespaceDocumentInput) -> Result<String>` | 105 | WRITE | Stores document; background graph extraction |
|
||||
| `put_doc_light(NamespaceDocumentInput) -> Result<String>` | 125 | WRITE | Lightweight insert; no embedding or extraction |
|
||||
| `ingest_doc(MemoryIngestionRequest) -> Result<MemoryIngestionResult>` | 132 | WRITE | Full synchronous ingestion pipeline |
|
||||
| `store_skill_sync(...)` | 143 | WRITE | Skill-specific upsert into `skill-{skill_id}` namespace |
|
||||
| `list_documents(Option<&str>) -> Result<Value>` | 184 | READ | Lists documents, optionally by namespace |
|
||||
| `delete_document(&str, &str) -> Result<Value>` | 197 | DELETE | Deletes by namespace + document ID |
|
||||
|
||||
### Namespaces
|
||||
|
||||
| Function | Line | Type | Description |
|
||||
|----------|------|------|-------------|
|
||||
| `list_namespaces() -> Result<Vec<String>>` | 192 | READ | Lists all namespaces |
|
||||
| `clear_namespace(&str) -> Result<()>` | 206 | DELETE | Clears all data in a namespace |
|
||||
| `clear_skill_memory(&str, &str) -> Result<()>` | 211 | DELETE | Clears documents in the shared `skill-{skill_id}` namespace; second arg is the integration identifier passed from disconnect flows |
|
||||
|
||||
### Query / Recall
|
||||
|
||||
| Function | Line | Type | Description |
|
||||
|----------|------|------|-------------|
|
||||
| `query_namespace(&str, &str, u32) -> Result<String>` | 234 | READ | Semantic query; returns text |
|
||||
| `query_namespace_context_data(&str, &str, u32) -> Result<NamespaceRetrievalContext>` | 246 | READ | Semantic query; returns structured data |
|
||||
| `recall_namespace(&str, u32) -> Result<Option<String>>` | 258 | READ | Recent context; returns text |
|
||||
| `recall_namespace_context_data(&str, u32) -> Result<NamespaceRetrievalContext>` | 269 | READ | Recent context; returns structured data |
|
||||
| `recall_namespace_memories(&str, u32) -> Result<Vec<NamespaceMemoryHit>>` | 280 | READ | Recent context; returns individual hits |
|
||||
|
||||
### Key-Value
|
||||
|
||||
| Function | Line | Type | Description |
|
||||
|----------|------|------|-------------|
|
||||
| `kv_set(Option<&str>, &str, &Value) -> Result<()>` | 289 | WRITE | Sets KV pair (`None` namespace = global) |
|
||||
| `kv_get(Option<&str>, &str) -> Result<Option<Value>>` | 302 | READ | Gets KV value |
|
||||
| `kv_delete(Option<&str>, &str) -> Result<bool>` | 314 | DELETE | Deletes KV pair |
|
||||
| `kv_list_namespace(&str) -> Result<Vec<Value>>` | 322 | READ | Lists all KV pairs in namespace |
|
||||
|
||||
### Knowledge Graph
|
||||
|
||||
| Function | Line | Type | Description |
|
||||
|----------|------|------|-------------|
|
||||
| `graph_upsert(Option<&str>, &str, &str, &str, &Value) -> Result<()>` | 330 | WRITE | Upserts relation triple (`None` namespace = global) |
|
||||
| `graph_query(Option<&str>, Option<&str>, Option<&str>) -> Result<Vec<Value>>` | 356 | READ | Queries graph with optional filters |
|
||||
|
||||
---
|
||||
|
||||
## RPC Method Names
|
||||
|
||||
For callers going through JSON-RPC (frontend or external). Each maps 1:1 to a `MemoryClient` method above.
|
||||
|
||||
| RPC Method | Type | MemoryClient equivalent |
|
||||
|------------|------|------------------------|
|
||||
| `openhuman.memory_init` | INIT | — (initializes subsystem) |
|
||||
| `openhuman.memory_doc_put` | WRITE | `put_doc()` |
|
||||
| `openhuman.memory_doc_ingest` | WRITE | `ingest_doc()` |
|
||||
| `openhuman.memory_doc_list` | READ | `list_documents()` |
|
||||
| `openhuman.memory_doc_delete` | DELETE | `delete_document()` |
|
||||
| `openhuman.memory_namespace_list` | READ | `list_namespaces()` |
|
||||
| `openhuman.memory_list_namespaces` | READ | `list_namespaces()` (structured envelope) |
|
||||
| `openhuman.memory_list_documents` | READ | `list_documents()` (structured envelope) |
|
||||
| `openhuman.memory_delete_document` | DELETE | `delete_document()` (structured envelope) |
|
||||
| `openhuman.memory_clear_namespace` | DELETE | `clear_namespace()` |
|
||||
| `openhuman.memory_context_query` | READ | `query_namespace()` |
|
||||
| `openhuman.memory_context_recall` | READ | `recall_namespace()` |
|
||||
| `openhuman.memory_query_namespace` | READ | `query_namespace_context_data()` |
|
||||
| `openhuman.memory_recall_context` | READ | `recall_namespace_context_data()` |
|
||||
| `openhuman.memory_recall_memories` | READ | `recall_namespace_memories()` |
|
||||
| `openhuman.memory_kv_set` | WRITE | `kv_set()` |
|
||||
| `openhuman.memory_kv_get` | READ | `kv_get()` |
|
||||
| `openhuman.memory_kv_delete` | DELETE | `kv_delete()` |
|
||||
| `openhuman.memory_kv_list_namespace` | READ | `kv_list_namespace()` |
|
||||
| `openhuman.memory_graph_upsert` | WRITE | `graph_upsert()` |
|
||||
| `openhuman.memory_graph_query` | READ | `graph_query()` |
|
||||
| `openhuman.ai_list_memory_files` | READ | — (file I/O, not MemoryClient) |
|
||||
| `openhuman.ai_read_memory_file` | READ | — (file I/O) |
|
||||
| `openhuman.ai_write_memory_file` | WRITE | — (file I/O) |
|
||||
|
||||
---
|
||||
|
||||
## Frontend TypeScript Wrappers
|
||||
|
||||
**File**: `app/src/utils/tauriCommands/memory.ts`
|
||||
|
||||
Each calls the corresponding RPC method above via `core_rpc_relay`.
|
||||
|
||||
| Function | Line | RPC Method |
|
||||
|----------|------|------------|
|
||||
| `syncMemoryClientToken(token)` | 82 | `openhuman.memory_init` |
|
||||
| `memoryListDocuments(namespace?)` | 102 | `openhuman.memory_list_documents` |
|
||||
| `memoryListNamespaces()` | 117 | `openhuman.memory_list_namespaces` |
|
||||
| `memoryDeleteDocument(id, ns)` | 132 | `openhuman.memory_delete_document` |
|
||||
| `memoryClearNamespace(ns)` | 145 | `openhuman.memory_clear_namespace` |
|
||||
| `memoryQueryNamespace(ns, query, max?)` | 158 | `openhuman.memory_query_namespace` |
|
||||
| `memoryRecallNamespace(ns, max?)` | 173 | `openhuman.memory_recall_context` |
|
||||
| `memoryGraphQuery(ns?, subj?, pred?)` | 187 | `openhuman.memory_graph_query` |
|
||||
| `memoryDocIngest(params)` | 210 | `openhuman.memory_doc_ingest` |
|
||||
| `aiListMemoryFiles(dir?)` | 229 | `openhuman.ai_list_memory_files` |
|
||||
| `aiReadMemoryFile(path)` | 246 | `openhuman.ai_read_memory_file` |
|
||||
| `aiWriteMemoryFile(path, content)` | 261 | `openhuman.ai_write_memory_file` |
|
||||
|
||||
---
|
||||
|
||||
## Key Data Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `NamespaceDocumentInput` | Document upsert payload (namespace, content, metadata, tags, source_type, priority) |
|
||||
| `MemoryIngestionRequest` | Full ingestion request with config (chunking, embedding, extraction flags) |
|
||||
| `MemoryIngestionResult` | Ingestion result (document_id, chunk_count, entities, relations) |
|
||||
| `NamespaceRetrievalContext` | Structured retrieval (hits, entities, relations, chunks) |
|
||||
| `NamespaceMemoryHit` | Individual memory item with score breakdown |
|
||||
| `GraphRelationRecord` | Knowledge graph triple (subject, predicate, object, evidence, namespace) |
|
||||
| `RetrievalScoreBreakdown` | Score components: graph, vector, keyword, episodic, freshness |
|
||||
|
||||
All types defined in `src/openhuman/memory/store/types.rs`.
|
||||
@@ -1,111 +0,0 @@
|
||||
# Subconscious Orchestration Layer — Multistage Plan
|
||||
|
||||
Implements the split-brain architecture in [`docs/arch-subconscious.md`](../../arch-subconscious.md)
|
||||
on top of the **tinyagents graph harness** (`src/openhuman/tinyagents/` + `agent_graph`), with
|
||||
**tiny.place DM channels** as the transport for external Claude Code / Codex sessions, surfaced in
|
||||
the new **`TinyPlaceOrchestrationTab`** on the Brain page.
|
||||
|
||||
## How to use this folder (goal commands)
|
||||
|
||||
Each `stage-NN-*.md` file is a self-contained **goal**: hand a single stage file to an agent
|
||||
(`claude "implement docs/plans/subconscious-orchestration/stage-03-core-ingest.md"`) and it has the
|
||||
goal statement, the files to read first, deliverables, tasks, and acceptance criteria. Stages are
|
||||
ordered by dependency; each stage must land (tests green, committed) before the next starts.
|
||||
Stage 1 lands in the **`tiny.place/` checkout** (separate repo, separate PR); stage 2 spans both
|
||||
repos (its wrapper half rides with stage 1); stages 3–8 land here.
|
||||
|
||||
## Design decision: one graph, two triggers
|
||||
|
||||
The whole **wake path is a single tinyagents graph** (mirroring the spec's one `StateGraph`), not
|
||||
separate agents ping-ponging through the event bus:
|
||||
|
||||
```
|
||||
invoke(session thread) ─► normalize ─► frontend(pass1) ─► reasoning/execute ─► compress
|
||||
▲ │
|
||||
└── reply ◄── world_diff ◄─────────────┘
|
||||
frontend(pass2) ─► send_dm ─► context_guard ─► END
|
||||
```
|
||||
|
||||
- **One state object** (`OrchestrationState`: `messages`, `agent_instructions`, `agent_reply`,
|
||||
`channel_response`, `subconscious_steering`, `compressed_history`, `world_state_diff`,
|
||||
`context_utilization`) flows through conditional edges — the spec's routing predicate
|
||||
(`channel_response` present → wrap up, else execute) is a graph edge, not bus choreography.
|
||||
- **One checkpointer** (`SqlRunLedgerCheckpointer`, thread id `orchestration:<session_id>`) gives
|
||||
crash-resume for the whole cycle and gives the subconscious a consistent snapshot to read.
|
||||
- Only two things live **outside** the graph:
|
||||
1. **Transport/ingest** (stage 3): DMs arrive asynchronously; the subscriber persists + then
|
||||
*invokes* the graph on the session's thread. A polling node inside the graph would fight
|
||||
checkpointing.
|
||||
2. **Subconscious** (stage 6): out-of-band cron by design (the spec runs it via
|
||||
`update_state(as_node="subconscious_cron")`). It reads checkpointed state + the store and
|
||||
writes `subconscious_steering` back into the thread — a decoupled writer, not an edge.
|
||||
|
||||
Consequence for the stages: stage 4 builds the **graph skeleton + state + front-end nodes**,
|
||||
stage 5 adds the **reasoning/memory nodes to the same graph**. Different model tiers per node are
|
||||
fine — tinyagents nodes each resolve their own provider/model (`hint:chat` vs `hint:reasoning`).
|
||||
|
||||
## Target architecture → repo primitives
|
||||
|
||||
| Spec concept (`arch-subconscious.md`) | Repo primitive |
|
||||
| --- | --- |
|
||||
| Channels (ingestion boundary) | tiny.place Signal DMs → `src/openhuman/tinyplace/` (`messages_list`, `signal_send_message`, `streams.rs`) |
|
||||
| DM authorization | tiny.place mutual **contact graph** (`backend-tinyplace-v2` `/contacts/*`; relay rejects non-contact DMs) → new `tinyplace_contacts_*` controllers + pairing flow (stage 2) |
|
||||
| Front-End Agent (Quick LLM) | `frontend` nodes of the unified graph, model `hint:chat` (`src/openhuman/routing/policy.rs`) |
|
||||
| Reasoning LLM (Orchestration Core) | `execute` node of the unified graph (`AgentGraph::Custom` runner, `hint:reasoning`), spawning sub-agents via `subagent_runner` |
|
||||
| Bi-directional loop (frontend ⇄ reasoning) | Conditional edges on `agent_reply` / `channel_response` in the single graph state |
|
||||
| Checkpointing / resume | `SqlRunLedgerCheckpointer` (`src/openhuman/tinyagents/checkpoint.rs`), keyed by orchestration `thread_id` |
|
||||
| 20:1 compression hook | tinyagents middleware (pattern: `summarize.rs`, `TurnContextMiddleware`) writing `compressed_history` |
|
||||
| World state diff | New `orchestration` store table, appended per execution cycle |
|
||||
| 80–90% context hooks | Context-window middleware (extend `summarize.rs` seam) + eviction to memory/embeddings |
|
||||
| Subconscious LLM + cron | Existing `SubconsciousEngine` tick (`src/openhuman/subconscious/engine.rs`, `heartbeat/`), gated by `scheduler_gate` |
|
||||
| Steering directive injection | New steering store read by the Reasoning agent's `prompt.rs::build` |
|
||||
| UI surface | `TinyPlaceOrchestrationTab` (Brain → Orchestration group), backed by real metadata instead of string heuristics |
|
||||
|
||||
## Message flow (end to end)
|
||||
|
||||
```
|
||||
Claude Code / Codex instance
|
||||
└─ tinyplace wrapper (stage 1) — tails session JSONL → SessionEnvelope
|
||||
└─ contact pairing (stage 2) — accepted mutual contact edge; relay refuses DMs
|
||||
between non-contacts (403 not_a_contact), so this gates everything below
|
||||
└─ Signal E2E DM → owner agent's tiny.place inbox [tagged: sessionId, source, role, kind]
|
||||
└─ core ingest (stage 3) — orchestration::ingest normalizes DMs → SessionState
|
||||
└─ Front-End nodes (stage 4, Quick LLM) — macro-instructions
|
||||
└─ Reasoning + memory nodes (stage 5) — sub-agents, 20:1 compression,
|
||||
world-state diff, context hooks; replies upstream
|
||||
└─ Front-End pass 2 — channel_response → Signal DM back to session
|
||||
└─ Subconscious tick (stage 6, cron/heartbeat) — reads compressed history + world diff,
|
||||
writes steering directives → injected into Reasoning prompts next run
|
||||
UI: TinyPlaceOrchestrationTab (stage 7) — pairing, master / subconscious / per-session chat windows
|
||||
```
|
||||
|
||||
## Stage index
|
||||
|
||||
| Stage | Repo | Delivers | Depends on |
|
||||
| --- | --- | --- | --- |
|
||||
| [1 — tiny.place session bridge](stage-01-tinyplace-session-bridge.md) | `tiny.place/` | `SessionEnvelope` v1 schema; Claude Code wrapper; DM forwarding; wrapper pairing handshake | — |
|
||||
| [2 — contact pairing & DM authorization](stage-02-contact-pairing.md) | both | Contacts RPC in core; user-consented link/approve flows; Brain-tab pairing UX | 1 |
|
||||
| [3 — core ingest + session state](stage-03-core-ingest.md) | openhuman | `orchestration` domain: DM ingest, session store, typed envelopes | 1 (2 for live traffic) |
|
||||
| [4 — graph skeleton + front-end nodes](stage-04-frontend-agent.md) | openhuman | Unified `OrchestrationState` graph, checkpointing, two-pass frontend nodes (Quick LLM) | 3 |
|
||||
| [5 — reasoning + memory nodes](stage-05-reasoning-orchestrator.md) | openhuman | `execute`/`compress`/`world_diff`/`context_guard` nodes in the same graph, sub-agent spawning | 4 |
|
||||
| [6 — subconscious steering loop](stage-06-subconscious-steering.md) | openhuman | Steering directives from world diff via existing heartbeat/cron | 5 |
|
||||
| [7 — RPC + UI wiring](stage-07-rpc-and-ui.md) | openhuman | `orchestration.*` RPC; `TinyPlaceOrchestrationTab` on real data + streaming + composer | 3, 6 |
|
||||
| [8 — testing & observability](stage-08-testing-observability.md) | both | json_rpc_e2e, Vitest, debug logging audit, coverage gate | all |
|
||||
|
||||
## Global invariants (apply to every stage)
|
||||
|
||||
- **Loop continuity**: routing must terminate on `channel_response` presence — never allow
|
||||
frontend ⇄ reasoning infinite cycling (spec §5 checklist).
|
||||
- **20:1 strictness**: compression output budget = ⌈input_tokens / 20⌉, enforced, not advisory.
|
||||
- **World diff is append-only**: sequential timeline from genesis; never wipe keys per cycle.
|
||||
- **Context hooks run before END**: eviction takes effect prior to the next iteration.
|
||||
- **Subconscious isolation**: it never talks to channels or users directly; its only output is
|
||||
steering directives (and it keeps `SubconsciousTainted` origin so effect tools stay gated).
|
||||
- **Pairing consent**: DMs only flow over user-consented contact edges — the wrapper auto-accepts
|
||||
only its configured owner identity; OpenHuman never auto-accepts an unsolicited request unless
|
||||
the user initiated the link or explicitly opted in via config.
|
||||
- **Security**: tinyplace identity stays wallet-derived (`wallet::tinyplace_signer_seed()`); never
|
||||
log message bodies or seeds; Signal bodies stay E2E — the relay sees ciphertext only.
|
||||
- **Repo rules**: new Rust code in a dedicated domain dir (canonical module shape), controller
|
||||
registry (no `dispatch.rs` branches), verbose `[orchestration]`-prefixed debug logging, i18n for
|
||||
all UI strings across all 14 locales, ≥80% changed-line coverage.
|
||||
@@ -1,116 +0,0 @@
|
||||
# Stage 1 — tiny.place session bridge (SDK/CLI, `tiny.place/` repo)
|
||||
|
||||
## Goal command
|
||||
|
||||
> In the `tiny.place/` checkout (`sdk/typescript`), define a versioned **`SessionEnvelope` v1**
|
||||
> message schema for harness sessions and extend the CLI wrappers so that **every semantic message
|
||||
> (user and agent) from a wrapped Codex or Claude Code instance is sent as a Signal E2E DM to a
|
||||
> configured tiny.place recipient** (the OpenHuman owner agent), in addition to the existing
|
||||
> on-disk JSONL envelopes. Add a `tinyplace claude` wrapper mirroring `tinyplace codex`.
|
||||
> **Configure-once, zero-args**: after one-time setup the owner target and forwarding preferences
|
||||
> live in `~/.tinyplace/config.json`, so plain `tinyplace codex` / `tinyplace claude` (or a session
|
||||
> launched from the TUI) forwards automatically — no per-invocation flags. The **TUI is the
|
||||
> primary surface** for setup, pairing status, and launching sessions; `--tinyplace-*` flags exist
|
||||
> only as overrides for scripting.
|
||||
|
||||
## Read first
|
||||
|
||||
- `sdk/typescript/src/cli/codex.ts` — existing wrapper: PTY proxy + session-JSONL tailing already
|
||||
produces `SemanticMessage { role: "user" | "assistant" }` and `SessionEnvelope`s written under
|
||||
`~/.tinyplace/codex-envelopes/messages/…`. This is the source of truth to forward.
|
||||
- `sdk/typescript/src/agent/` — `Agent` facade (`sendMessage` does handle-resolution + Signal E2E).
|
||||
- `sdk/typescript/src/messaging/`, `src/signal/` — DM send path.
|
||||
- `sdk/typescript/src/cli/context.ts` — persisted CLI config at `~/.tinyplace/config.json`
|
||||
(`TINYPLACE_CONFIG` override); this is where the new keys land.
|
||||
- `sdk/typescript/src/cli/tui.ts` — existing blessed TUI (`runTinyPlaceTui`), already models agent
|
||||
kinds `"claude" | "codex"` and session launch; the setup/pairing UX extends this.
|
||||
- `sdk/typescript/AGENTS.md` — identifier kinds, error-code contract.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **`SessionEnvelope` v1 wire schema** (new `sdk/typescript/src/types/session-envelope.ts`,
|
||||
exported from the root). This is the contract stages 3–7 parse; it rides inside the DM body as
|
||||
JSON (encrypted end-to-end):
|
||||
|
||||
```ts
|
||||
interface HarnessSessionEnvelope {
|
||||
v: 1;
|
||||
kind: "session_message" | "session_lifecycle";
|
||||
source: "codex" | "claude-code";
|
||||
sessionId: string; // stable wrapper session id
|
||||
sessionLabel?: string; // e.g. repo folder name
|
||||
workspace?: string; // cwd of the wrapped instance
|
||||
seq: number; // monotonic per session
|
||||
role: "user" | "assistant" | "system";
|
||||
body: string; // the semantic message text (may be truncated, see limits)
|
||||
truncated?: boolean;
|
||||
timestamp: string; // ISO-8601
|
||||
lifecycle?: "started" | "ended" | "error"; // kind === session_lifecycle
|
||||
}
|
||||
```
|
||||
|
||||
2. **Persisted forwarding config + one-time setup**: new `orchestration` block in
|
||||
`~/.tinyplace/config.json` — `{ forwardTo: "<@handle|agentId>", forwardEnabled: true,
|
||||
pairingStatus?, scope?, bucket? }` — written once by the TUI setup flow (or
|
||||
`tinyplace setup --forward-to @owner` non-interactively). Resolution order per run:
|
||||
CLI flag (`--tinyplace-forward-to`) → env (`TINYPLACE_FORWARD_TO`) → config. Once configured,
|
||||
**bare `tinyplace codex` / `tinyplace claude` forwards with no extra args**; when nothing is
|
||||
configured, the wrappers behave exactly as today (local envelopes only, no nagging).
|
||||
3. **DM forwarding in the wrappers**: each `SemanticMessage` and lifecycle event is wrapped in a
|
||||
`HarnessSessionEnvelope` and sent via the Agent facade Signal DM path to the resolved target.
|
||||
Batching: flush per message, but coalesce assistant deltas into one message per completed turn
|
||||
(the tailer already yields whole messages). Failures are logged to stderr JSON and **never
|
||||
crash the wrapped CLI**; retry with backoff on `transient`/`rate_limited`, drop after N retries
|
||||
with a lifecycle `error`.
|
||||
4. **`tinyplace claude` wrapper** (`sdk/typescript/src/cli/claude.ts`): same PTY-proxy shape as
|
||||
`codex.ts`, tailing Claude Code session JSONL (`~/.claude/projects/<slug>/*.jsonl`) for
|
||||
user/assistant messages; shares the envelope writer + forwarder (extract the reusable parts of
|
||||
`codex.ts` into `src/cli/session-bridge.ts` rather than copy-pasting). Both wrappers are
|
||||
launchable from the TUI (`tui.ts` already models both agent kinds) and inherit the config.
|
||||
5. **Size limits**: cap `body` at 8 KiB per DM (set `truncated: true`; full text stays in the local
|
||||
JSONL). Never forward raw terminal chunks — semantic messages only.
|
||||
6. **Contact-pairing handshake in the TUI** (wrapper half of stage 2 — the relay refuses DMs
|
||||
between non-contacts, `403 not_a_contact`). The TUI setup flow owns this: it shows the CLI's
|
||||
own identity (`agentId` / `@handle`) so the user can link it from OpenHuman, displays live
|
||||
pairing status (`none / pending / accepted / blocked`), and persists `pairingStatus` to config
|
||||
so subsequent runs skip straight to forwarding. Headless runs perform the same handshake
|
||||
silently from config. State machine on `contacts/{owner}/status`:
|
||||
- `accepted` → forward; `none` → send a contact request, queue envelopes (bounded, drop-oldest
|
||||
with a lifecycle `error` note) and poll until accepted;
|
||||
- `pending incoming` **from the exact configured forward-to identity only** → accept;
|
||||
- `blocked` → terminal error, no retries.
|
||||
At send time, branch on the `not_a_contact` error code → re-enter pairing instead of retrying.
|
||||
7. Docs: README section + `tinyplace describe` entries for setup, the config keys, and the
|
||||
override flags.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. Extract shared session-bridge module from `codex.ts` (envelope writer, session-id logic, tailer
|
||||
interface) — no behavior change; existing tests stay green.
|
||||
2. Add `HarnessSessionEnvelope` type + serializer/validator (`parseHarnessSessionEnvelope` for
|
||||
consumers) with unit tests round-tripping v1 and rejecting unknown `v`.
|
||||
3. Add the `orchestration` config block to `context.ts` (read/write, flag→env→config resolution)
|
||||
+ `tinyplace setup` command; config round-trip tests.
|
||||
4. Implement the forwarder (queue + retry + never-crash guarantee), target resolved from config.
|
||||
5. Implement `claude.ts` tailer: resolve session file for the wrapped instance (newest JSONL in the
|
||||
project slug dir after spawn; honor `--tinyplace-session-file` override like codex does).
|
||||
6. Implement the pairing handshake (status check, request, owner-only auto-accept, bounded queue,
|
||||
`not_a_contact` recovery) in the shared session-bridge module; TUI panels for setup, pairing
|
||||
status, and session launch on top of it.
|
||||
7. Wire both into `cli.ts`/`commands.ts`/`tui.ts`; update help/catalog output.
|
||||
8. Vitest coverage: envelope schema, config resolution order, forwarder retry/drop behavior (mock
|
||||
client), pairing state machine (all four contact statuses + owner-only accept), claude JSONL
|
||||
parsing fixtures (user message, assistant message, tool-use records ignored).
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- After one-time setup (TUI or `tinyplace setup --forward-to @owner`), a bare `tinyplace codex`
|
||||
forwards every user + assistant message of the session as Signal DMs whose plaintext bodies
|
||||
parse as `HarnessSessionEnvelope` v1, plus `started`/`ended` lifecycle envelopes — **no
|
||||
per-invocation flags**.
|
||||
- `tinyplace claude` does the same for a Claude Code session; both launch from the TUI too.
|
||||
- With no config and no flags, behavior is unchanged from today (no forwarding, no prompts).
|
||||
- With no contact edge, the wrapper pairs (or waits, queueing) before any envelope DM is sent; it
|
||||
never auto-accepts an identity other than the configured forward-to target.
|
||||
- Killing the network mid-session does not break the wrapped CLI; envelopes on disk stay complete.
|
||||
- `pnpm test` green in `sdk/typescript`; new module has no `any`-typed public surface.
|
||||
@@ -1,105 +0,0 @@
|
||||
# Stage 2 — Contact pairing & DM authorization (OpenHuman ⇄ session identities)
|
||||
|
||||
## Goal command
|
||||
|
||||
> Build the **authentication/pairing flow** between the user's OpenHuman tiny.place identity and
|
||||
> each wrapped Claude Code / Codex session identity. The relay **refuses 1:1 DMs between
|
||||
> non-contacts** (`PUT /messages` → `403 not_a_contact`; see
|
||||
> `backend-tinyplace-v2/docs/spec/contacts.md`), so before any stage-1 envelope can flow, the two
|
||||
> identities must hold an **accepted mutual contact edge**. Expose the contacts API through the
|
||||
> core `tinyplace` domain, implement a user-consented pairing policy in the orchestration domain,
|
||||
> and surface link/approve UX in the Brain tab. Contact requests carry no free text by design
|
||||
> (prompt-injection-safe bootstrap) — the pairing signal is the edge itself, never message content.
|
||||
|
||||
## Backend facts this stage builds on
|
||||
|
||||
- Contact model (`tiny.place/sdk/typescript/src/types/contacts.ts`, backend `docs/spec/contacts.md`):
|
||||
one edge per unordered pair, `pending | accepted | blocked`, `requester`/`addressee` direction.
|
||||
- Routes: `POST /contacts/{agentId}` (request), `POST …/accept`, `POST …/block`, `…/unblock`,
|
||||
`DELETE /contacts/{agentId}`, `GET /contacts`, `GET /contacts/requests`,
|
||||
`GET /contacts/{agentId}/status`, `GET /contacts/stats`. All signed; reads too (graph is private).
|
||||
- **Crossing requests auto-accept**: if A→B is pending and B sends B→A, the edge converges to
|
||||
`accepted` — this is the mechanism that makes user-initiated linking frictionless.
|
||||
- Send is idempotent for duplicate outgoing / already-accepted; refused when blocked.
|
||||
|
||||
## Read first
|
||||
|
||||
- `backend-tinyplace-v2` — `docs/spec/contacts.md`, `docs/spec/messaging.md` (DM gate),
|
||||
`internal/controllers/relay/controller.go` (enforcement).
|
||||
- `tiny.place/sdk/typescript/src/api/contacts.ts` + `types/contacts.ts` — client surface to mirror.
|
||||
- `src/openhuman/tinyplace/{mod.rs, schemas.rs, ops.rs, state.rs}` — controller pattern; note
|
||||
there is **no contacts support in the core domain today** (net-new).
|
||||
- `app/src/lib/agentworld/invokeApiClient.ts` — no `contacts` namespace yet (net-new).
|
||||
- Approval surface precedent: `src/openhuman/approval/` + `ApprovalRequestCard` (frontend), and
|
||||
`DomainEvent::ProactiveMessageRequested` for notification-style delivery.
|
||||
- Stage 1 (`stage-01-tinyplace-session-bridge.md`) — the wrapper-side half of the handshake.
|
||||
|
||||
## Pairing flows (both must work)
|
||||
|
||||
**A. User-initiated link (recommended, zero unsolicited approvals):**
|
||||
1. The tinyplace TUI setup flow shows the CLI's identity (`agentId` + optional `@handle`)
|
||||
(stage 1; this identity is per-machine, from `~/.tinyplace/config.json` — pairing happens once,
|
||||
not per session).
|
||||
2. User pastes it into "Link a session" in the Brain tab → core sends `POST /contacts/{cliId}`.
|
||||
3. The CLI, polling `GET /contacts/{owner}/status`, sees `pending incoming` from its **configured
|
||||
owner identity** and accepts (it only ever auto-accepts that exact identity) — or, if the CLI
|
||||
requested first, the owner's request crosses and **auto-accepts** server-side. The TUI flips to
|
||||
"paired", persists the status, and envelopes flow for every future session with no further
|
||||
steps.
|
||||
|
||||
**B. Session-initiated request (approval-gated):**
|
||||
1. Wrapper sends the contact request to the owner identity and queues envelopes (stage 1).
|
||||
2. Core polls/ingests `GET /contacts/requests` → each new incoming request raises an
|
||||
orchestration pairing approval (approval-card or notification path), showing `agentId`,
|
||||
resolved handle/profile if any, and first-seen time.
|
||||
3. User accepts → core calls `POST /contacts/{sessionId}/accept`; decline → `DELETE`; block →
|
||||
`…/block`. Config `[orchestration] auto_accept_session_contacts = false` (default **off**;
|
||||
never auto-accept arbitrary requests).
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **Core contacts controllers** (`src/openhuman/tinyplace/schemas.rs` + `ops.rs`, internal
|
||||
registry): `tinyplace_contacts_{request,accept,remove,block,unblock,list,requests,status,stats}`
|
||||
mapping 1:1 onto the routes above through `TinyPlaceState::client()` signed HTTP.
|
||||
2. **Pairing manager** (`orchestration/pairing.rs`): poll or stream incoming contact requests
|
||||
(cursor in the stage-3 store's `kv`), dedupe, raise/resolve approvals, persist pairing records
|
||||
`{ agent_id, label?, status, linked_at, source: user_link | approved_request }`. Publishes
|
||||
`DomainEvent::OrchestrationPairingChanged`.
|
||||
3. **Renderer client**: `contacts` namespace in `invokeApiClient.ts`
|
||||
(`openhuman.tinyplace_contacts_*`) mirroring the SDK types.
|
||||
4. **UI (Brain tab)**: "Link a session" affordance (paste `@handle`/agentId → request + pending
|
||||
chip → accepted), pending-request approval list (accept / decline / block), and a linked-
|
||||
sessions indicator on session chat windows (unlinked/pending sessions render a "waiting for
|
||||
pairing" state instead of messages). i18n keys across all 14 locales.
|
||||
5. **CLI-side handshake** (specified in `stage-01` deliverable 6, config-first + TUI-driven):
|
||||
status check on start, request + bounded queue + poll on `none`, owner-only auto-accept,
|
||||
`blocked` → hard error, `not_a_contact` at send time → re-enter pairing. Pairing is
|
||||
**once per machine identity** (persisted `pairingStatus` in `~/.tinyplace/config.json`), never
|
||||
per session or per flag.
|
||||
6. **Security invariants**: wrapper auto-accepts only its configured owner identity; core never
|
||||
auto-accepts unless the user initiated the link (flow A) or explicitly enabled the config;
|
||||
blocked identities are never re-requested automatically; the contact graph is private — don't
|
||||
sync it to any store outside the workspace.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. Core controllers + ops with mock-relay unit tests (status transitions, idempotent request,
|
||||
blocked refusal).
|
||||
2. Pairing manager + approval integration + events; tests for flow A (crossing auto-accept) and
|
||||
flow B (approve/decline/block), dedupe on re-poll.
|
||||
3. Renderer `contacts` namespace + Vitest (RPC name mapping, type round-trip).
|
||||
4. Brain-tab UX + i18n + component tests (link flow optimistic states, pending approval list,
|
||||
unpaired session placeholder).
|
||||
5. Amend the stage-1 CLI per deliverable 5 (lands in `tiny.place/` with its own tests: queue
|
||||
bounds, owner-only auto-accept, `not_a_contact` recovery).
|
||||
6. `tests/json_rpc_e2e.rs`: pair → DM flows; unpaired → send refused surfaced cleanly.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Flow A: pasting the CLI identity in the tab yields an accepted edge (crossing-request case
|
||||
covered) and envelopes flow with no approval prompt.
|
||||
- Flow B: an unsolicited session request appears as a pending approval; accept → DMs flow;
|
||||
decline/block → wrapper reports a clear terminal state and stops retrying.
|
||||
- `not_a_contact` at send time never loops hot — it parks into pairing state on both sides.
|
||||
- No path auto-accepts an arbitrary identity; config default is prompt.
|
||||
- `pnpm test:rust`, `pnpm test`, `pnpm i18n:check` green; ≥80% changed-line coverage.
|
||||
@@ -1,71 +0,0 @@
|
||||
# Stage 3 — Core ingest + session state (`src/openhuman/orchestration/`)
|
||||
|
||||
## Goal command
|
||||
|
||||
> Create a new Rust domain `src/openhuman/orchestration/` that ingests tiny.place Signal DMs,
|
||||
> recognizes `HarnessSessionEnvelope` v1 payloads (stage 1), and maintains durable per-session
|
||||
> state (master / subconscious / session chat windows). This is the "channel ingestion" boundary of
|
||||
> the split-brain graph: it normalizes heterogeneous DM traffic into typed graph inputs and
|
||||
> persists a chat-window model the RPC/UI layer (stage 7) can read directly — replacing the string
|
||||
> heuristics currently in `TinyPlaceOrchestrationTab.tsx`.
|
||||
|
||||
## Read first
|
||||
|
||||
- `src/openhuman/tinyplace/` — `mod.rs` (architecture), `schemas.rs` (`handle_tinyplace_messages_list`,
|
||||
`handle_tinyplace_signal_send_message`), `signal_store.rs`, `streams.rs`, `state.rs`.
|
||||
- `src/openhuman/subconscious/store.rs` — SQLite-per-domain store pattern.
|
||||
- `src/core/event_bus/` — `DomainEvent`, `publish_global`/`subscribe_global`, domain `bus.rs` convention.
|
||||
- `tiny.place/sdk/typescript/src/types/session-envelope.ts` (after stage 1) — the wire schema.
|
||||
- Canonical module shape table in `CLAUDE.md`.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **Domain skeleton** (canonical shape): `orchestration/{mod.rs, types.rs, store.rs, ops.rs,
|
||||
schemas.rs, bus.rs, ingest.rs}` + inline tests. Wire `all_controller_schemas` into
|
||||
`src/core/all.rs` (schemas themselves land in stage 7; register the namespace now).
|
||||
2. **`types.rs`**: Rust mirror of `HarnessSessionEnvelope` (serde, `#[serde(tag = "v")]`-style
|
||||
versioning tolerant of unknown fields), plus:
|
||||
- `ChatKind { Master, Subconscious, Session }`
|
||||
- `OrchestrationSession { session_id, source (Codex|ClaudeCode|Other), label, workspace,
|
||||
last_seq, created_at, last_message_at, active }`
|
||||
- `OrchestrationMessage { id, session_id, chat_kind, role, body, timestamp, encrypted, seq }`
|
||||
3. **`ingest.rs`**: subscribe to incoming tiny.place DMs. Preferred seam: a `DomainEvent`
|
||||
published by the tinyplace domain when a Signal DM is received/decrypted (add
|
||||
`DomainEvent::TinyplaceDmReceived { from, envelope_json, … }` in the tinyplace stream/inbox
|
||||
path — `streams.rs` websocket handler and the poll path both publish). Fallback if streams are
|
||||
unavailable: a poll loop calling the existing messages-list op with a cursor. Classification:
|
||||
- body parses as `HarnessSessionEnvelope` → `ChatKind::Session`, keyed by `sessionId`.
|
||||
- DM from the agent's own subconscious identity/thread marker → `ChatKind::Subconscious`.
|
||||
- everything else from the owner/human counterpart → `ChatKind::Master`.
|
||||
Idempotent by `(session_id, seq)` / message id — re-ingest must not duplicate.
|
||||
4. **`store.rs`**: SQLite at `<workspace>/orchestration/orchestration.db` — tables `sessions`,
|
||||
`messages` (indexed by session + timestamp), `kv`. Retention: prune messages beyond N=2000 per
|
||||
session (configurable). Message bodies stored decrypted here are workspace-internal — protected
|
||||
by `is_workspace_internal_path`.
|
||||
5. **`bus.rs`**: `OrchestrationIngestSubscriber` (`name() = "orchestration::ingest"`), registered
|
||||
at startup next to the other bus registrations; also publish
|
||||
`DomainEvent::OrchestrationSessionMessage` after persist, so stage 4 (front-end graph) and
|
||||
stage 7 (UI socket push) can both react without coupling.
|
||||
6. **Logging**: `[orchestration]` prefix; log envelope seq/session/kind on ingest entry/exit,
|
||||
classification decisions, dedupe skips, parse failures (body **never** logged).
|
||||
|
||||
## Tasks
|
||||
|
||||
1. Scaffold domain + wire `mod.rs`/`all.rs`; add config knob `[orchestration] enabled = true`
|
||||
(schema in `src/openhuman/config/schema/`, follow `scheduler_gate.rs` pattern).
|
||||
2. Implement types + envelope parsing with fixture tests (valid v1, unknown v, junk body → Master).
|
||||
3. Implement store + migrations + retention, unit-tested with tempdir DBs.
|
||||
4. Add the `TinyplaceDmReceived` event publication in `tinyplace::streams`/inbox ops; implement
|
||||
`OrchestrationIngestSubscriber`; startup registration.
|
||||
5. Poll-fallback ingest with cursor in `kv` (used when the Signal stream is down), behind the same
|
||||
dedupe.
|
||||
6. Tests: end-to-end ingest of a synthetic envelope sequence → sessions/messages rows; dedupe on
|
||||
replay; classification matrix.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Feeding N stage-1 envelopes (mixed sessions, out-of-order seq, duplicates) through the
|
||||
subscriber yields correctly bucketed, deduped `sessions`/`messages` rows.
|
||||
- Non-envelope DMs land in the Master window; nothing crashes on malformed bodies.
|
||||
- `cargo check` + `pnpm test:rust` green; new code ≥80% line coverage on the diff.
|
||||
- No message bodies or seeds in logs at any level.
|
||||
@@ -1,75 +0,0 @@
|
||||
# Stage 4 — Unified orchestration graph: skeleton, state & front-end nodes
|
||||
|
||||
## Goal command
|
||||
|
||||
> Build the **single orchestration graph** that carries the whole wake path, and implement its
|
||||
> **front-end (Quick LLM) nodes**. Define the shared `OrchestrationState`, the graph topology with
|
||||
> conditional routing, checkpointing on thread id `orchestration:<session_id>`, and the two-pass
|
||||
> front-end behavior: pass 1 turns raw session/master traffic into `agent_instructions` and routes
|
||||
> down; pass 2 (after the reasoning node sets `agent_reply`, stubbed in this stage) compiles
|
||||
> `channel_response` and sends it back to the originating tiny.place DM. No human gate — the loop
|
||||
> is autonomous, and the routing predicate must terminate (spec §5 loop continuity).
|
||||
|
||||
## Read first
|
||||
|
||||
- `docs/plans/subconscious-orchestration/README.md` — "one graph, two triggers" design decision.
|
||||
- `src/openhuman/agent/harness/agent_graph.rs` — `AgentGraph::Custom`, `AgentTurnRequest`.
|
||||
- `src/openhuman/tinyagents/` — `mod.rs` (shared runner), `checkpoint.rs`
|
||||
(`SqlRunLedgerCheckpointer`), `topology.rs`/`orchestration.rs` (existing graph-composition
|
||||
precedents — follow whichever expresses multi-node graphs; document the choice), `tools.rs`
|
||||
(`EarlyExit` seam), `model.rs` (per-node provider/model resolution).
|
||||
- `src/openhuman/subconscious/agent/` — slim-agent packaging (agent.toml + prompt.md + graph.rs).
|
||||
- `src/openhuman/routing/policy.rs` — `hint:chat` (quick, remote for TTFT) vs `hint:reasoning`.
|
||||
- `src/openhuman/tinyplace/schemas.rs` — `handle_tinyplace_signal_send_message` (reply path).
|
||||
- `docs/arch-subconscious.md` §2.2, §4 — node/edge reference topology.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **Graph state** (`orchestration/graph/state.rs`): `OrchestrationState` — `messages` (windowed
|
||||
from the stage-3 store), `agent_instructions: Option<String>`, `agent_reply: Option<String>`,
|
||||
`channel_response: Option<String>`, `subconscious_steering: Option<String>` (read in stage 5/6),
|
||||
`compressed_history: Vec<CompressedEntry>`, `world_state_diff: WorldDiff`,
|
||||
`context_utilization: f32`. Serde-serializable so `SqlRunLedgerCheckpointer<OrchestrationState>`
|
||||
persists it at superstep boundaries.
|
||||
2. **Graph topology** (`orchestration/graph/mod.rs`), registered as the orchestration agent's
|
||||
`AgentGraph::Custom` runner:
|
||||
- Nodes this stage: `normalize` (fold pending session messages into state),
|
||||
`frontend` (two-pass, Quick LLM), `send_dm` (Signal reply to the session counterpart),
|
||||
`execute_stub` (sets a canned `agent_reply`; replaced in stage 5), `context_guard`.
|
||||
- Conditional edges = the spec's router: from `frontend`, `channel_response` present →
|
||||
`send_dm` → `context_guard` → END; else → `execute` → back to `frontend`.
|
||||
- **Invocation**: `invoke_orchestration_graph(session_id)` in `orchestration/ops.rs`, called by
|
||||
the stage-3 ingest subscriber on `OrchestrationSessionMessage`, debounced per session so DM
|
||||
bursts produce one graph run. Resumes from the last checkpoint for that thread.
|
||||
3. **Front-end node** driven by a slim agent package
|
||||
`orchestration/frontend_agent/{agent.toml, prompt.md}`: model `hint:chat`, small context
|
||||
budget, tool surface limited to `defer_to_orchestrator` (EarlyExit emitting
|
||||
`agent_instructions`) and `reply_to_channel` (emits `channel_response`); pass selection purely
|
||||
from state (`agent_reply` present → pass 2), mirroring `frontend_agent_node` in the spec.
|
||||
4. **Gating**: graph runs with background origin (no interactive approval parking); each
|
||||
LLM-bearing node awaits `scheduler_gate::wait_for_capacity()`.
|
||||
5. **Logging**: `[orchestration]` node entry/exit with `session_id`, node name, pass number,
|
||||
routing decision at debug.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. State + checkpointer round-trip test (serialize → resume → identical state).
|
||||
2. Topology with stubs; graph-level test: one invocation walks
|
||||
normalize → frontend(1) → execute_stub → frontend(2) → send_dm → guard → END, sending exactly
|
||||
one DM (mock tinyplace op).
|
||||
3. Frontend agent package (prompt.md with two-pass contract + macro-instruction format,
|
||||
agent.toml) + loader registration; tools in `orchestration/tools.rs` (domain-owned rule).
|
||||
4. Debounced invocation from the ingest subscriber; idempotence test (re-invoke with no new
|
||||
messages → no LLM call, no DM).
|
||||
5. Loop-continuity property test: adversarial state combos (`agent_reply` + `channel_response`
|
||||
both set, neither set, instructions without reply) never cycle more than the configured max
|
||||
supersteps and never double-send.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A new session DM triggers one full stubbed cycle ending in exactly one outbound Signal DM to the
|
||||
right counterpart; checkpoints exist for the thread.
|
||||
- Quick tier verified: frontend node requests resolve via `hint:chat`; prompt+history within the
|
||||
small budget (asserted).
|
||||
- Kill/restart mid-run resumes from checkpoint without a duplicate DM.
|
||||
- `pnpm test:rust` green; no infinite cycling under repeated triggers.
|
||||
@@ -1,76 +0,0 @@
|
||||
# Stage 5 — Reasoning + memory nodes (completing the unified graph)
|
||||
|
||||
## Goal command
|
||||
|
||||
> Replace the stage-4 stubs with the real **reasoning and memory nodes** of the unified
|
||||
> orchestration graph: an `execute` node on the reasoning tier that applies the current
|
||||
> subconscious steering directive and spawns execution sub-agents via `subagent_runner`, followed
|
||||
> by the three memory mechanics from the spec as their own nodes — the **20:1 compression hook**
|
||||
> over the cycle's execution trace, the **append-only world-state diff**, and the **80–90%
|
||||
> context-eviction guard** that runs after mutations and before END. All state changes flow
|
||||
> through `OrchestrationState` and are checkpointed; durable copies land in the orchestration
|
||||
> store for the subconscious (stage 6) and the UI (stage 7).
|
||||
|
||||
## Read first
|
||||
|
||||
- Stage 4's `orchestration/graph/` (state, topology, stubs to replace).
|
||||
- `src/openhuman/tinyagents/` — `mod.rs` (`run_turn_via_tinyagents_shared`, `RunPolicy`, steering
|
||||
forwarders), `delegation.rs` (sub-agent spawning), `summarize.rs` (tokenizer + summarization
|
||||
seam), `middleware.rs` (`TurnContextMiddleware`, `SuperContextConfig`), `observability.rs`
|
||||
(`OpenhumanEventBridge`, `CapPauser`, `GraphTracingSink`).
|
||||
- `src/openhuman/agent/harness/subagent_runner/ops/graph.rs` — `run_subagent_via_graph`.
|
||||
- `docs/arch-subconscious.md` §3–§5 — compression ratio, world diff shape, guardrail checklist.
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **`execute` node** driven by `orchestration/reasoning_agent/{agent.toml, prompt.md}`
|
||||
(`agent_tier = "reasoning"`, `hint:reasoning`, large context):
|
||||
- System prompt assembled per cycle: base prompt + `subconscious_steering` from state (default
|
||||
alignment string when empty).
|
||||
- Tools: sub-agent spawn (via the `delegation.rs` seam; steering text threaded into sub-agent
|
||||
definitions as steering traits, spec §3.2; concurrent cap config, default 2) + the tinyplace
|
||||
read tools already whitelisted for the orchestrator agent.
|
||||
- Output: sets `agent_reply` in state; raw trace (assistant text, tool calls/results, sub-agent
|
||||
outputs) captured for the compression node.
|
||||
2. **`compress` node** (`orchestration/graph/compress.rs`): token-count the cycle trace (tokenizer
|
||||
util used by `summarize.rs`); summarize via a cheap `hint:*` route with an enforced output
|
||||
budget of `min(input_tokens / 20, input_tokens)`; apply the 200-token floor only when the
|
||||
source trace is large enough that the budget is still compressive. Retry once if >1.5× budget,
|
||||
then hard-truncate. Append to `state.compressed_history` **and** persist to the store's
|
||||
`compressed_history` table (cycle id, session id, token counts, text).
|
||||
3. **`world_diff` node** (`orchestration/graph/world_diff.rs` + store table): append one timeline
|
||||
entry `{ seq, cycle_id, event_signature, world_mutation, delta, timestamp }`; `genesis` row on
|
||||
first cycle; `terminal_state` kv updated per cycle. Append-only — never rewritten.
|
||||
4. **`context_guard` node**: utilization from accumulated `AgentTurnUsage` vs the resolved context
|
||||
window, stored in `state.context_utilization`; at ≥ threshold
|
||||
(`[orchestration] context_evict_threshold = 0.85`, clamped 0.8–0.9) map-reduce-summarize the
|
||||
oldest compressed-history entries, push summaries to the memory domain
|
||||
(`metadata.path_scope = "orchestration/<session_id>"` for RAG), drop them from state, reset
|
||||
utilization. Runs after all mutations, before END (spec invariant — edge ordering test).
|
||||
5. **Observability**: node progress mirrored through `OpenhumanEventBridge` (cycles + sub-agents
|
||||
visible in the agent-observability UI with usage/cost); `scheduler_gate::wait_for_capacity()`
|
||||
before every model call (execute, compress, evict).
|
||||
|
||||
## Tasks
|
||||
|
||||
1. Reasoning agent package + swap `execute_stub` → `execute`; mock-provider test: instructions in
|
||||
→ steering applied (assert in captured system prompt) → `agent_reply` set.
|
||||
2. Sub-agent spawning path with a stub sub-agent; steering-trait threading test.
|
||||
3. `compress` node with ratio-enforcement tests (large fixture trace → ≤ budget×1.5; floor case;
|
||||
store row written).
|
||||
4. `world_diff` node + append-only property test (two cycles → seq 1,2; genesis untouched).
|
||||
5. `context_guard` tests at 0.84 (no-op) and 0.86 (evicts: state shrinks, memory write happened,
|
||||
utilization reset) + guard-before-END edge-ordering test.
|
||||
6. Full-graph e2e (mock providers): DM → normalize → frontend(1) → execute (sub-agent stub) →
|
||||
compress → world_diff → frontend(2) → send_dm → guard → END; exactly one compressed row + one
|
||||
diff entry per cycle; checkpoint resume mid-cycle produces no duplicate side effects.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Full autonomous cycle passes hermetically with exactly one outbound DM, one compressed-history
|
||||
row, one world-diff entry.
|
||||
- All four spec §5 guardrails have a dedicated test (loop continuity, 20:1 strictness, append-only
|
||||
diff, guard-before-END ordering).
|
||||
- Kill/restart mid-cycle resumes from the last checkpoint without duplicate outbound DMs or
|
||||
duplicate store rows.
|
||||
- `pnpm test:rust` green; cycles visible in agent observability with cost/usage totals.
|
||||
@@ -1,65 +0,0 @@
|
||||
# Stage 6 — Subconscious steering loop (offline reflection)
|
||||
|
||||
## Goal command
|
||||
|
||||
> Extend the existing **`SubconsciousEngine`** so its cron/heartbeat tick consumes the
|
||||
> orchestration layer's **compressed history** and **cumulative world-state diff** (stage 5) and
|
||||
> emits short, dense **steering directives** that are injected into the Reasoning orchestrator's
|
||||
> prompts on subsequent cycles. The subconscious stays fully offline: no channels, no user
|
||||
> contact, no tools with external effects — its only output is the directive (and its existing
|
||||
> proactive-notify path, unchanged).
|
||||
|
||||
## Read first
|
||||
|
||||
- `src/openhuman/subconscious/README.md`, `engine.rs` (three-stage tick), `heartbeat/mod.rs`,
|
||||
`store.rs`, `agent/{agent.toml, prompt.md, graph.rs}`, `global.rs`.
|
||||
- `src/openhuman/scheduler_gate/` — capacity gating already applied to ticks.
|
||||
- Stage 5's store tables (`compressed_history`, `world_diff`) and `orchestration/types.rs`.
|
||||
- `docs/arch-subconscious.md` §3.2 — world-diff evaluation semantics ("macro-trends, filter
|
||||
localized variance").
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **New tick stage `orchestration_review`** in `SubconsciousEngine::tick_inner`, after
|
||||
`memory_diff` / before `decide` (or folded into `prepare_context` — decide against the existing
|
||||
stage contract and document): loads, since the last reviewed cursor,
|
||||
- unreviewed `compressed_history` rows (bounded batch, oldest-first), and
|
||||
- the **full world-diff timeline** (it is the cumulative object the spec requires — cap by
|
||||
summarizing the oldest ranges through the same 20:1 hook if it outgrows the budget).
|
||||
2. **Steering synthesis**: extend the slim subconscious agent prompt with a steering section —
|
||||
output contract: at most ~150 tokens, imperative, model-agnostic
|
||||
(`STEERING_DIRECTIVE: …` semantics from the spec), plus a machine field
|
||||
`expires_after_cycles: u32` (default 20). Runs on the existing `hint:subconscious` route with
|
||||
`SubconsciousTainted` origin.
|
||||
3. **Steering store** (`orchestration/store.rs`, table `steering_directives`): append-only history
|
||||
`{ id, text, created_at, source_tick_id, expires_after_cycles, superseded_by }`; "current
|
||||
directive" = latest non-expired. The unified graph's `execute` node loads it into
|
||||
`state.subconscious_steering` at cycle start — the spec's out-of-band
|
||||
`update_state(as_node="subconscious_cron")` pattern: the subconscious is a decoupled writer
|
||||
into the checkpointed thread, never an edge in the wake graph.
|
||||
4. **Feedback provenance**: each directive records which compressed-history rows / diff seq range
|
||||
it was derived from; reviewed cursor advances only on successful persist (idempotent ticks).
|
||||
5. **Subconscious chat surface**: publish each emitted directive as an
|
||||
`OrchestrationMessage { chat_kind: Subconscious }` for the local UI only. This fills the pinned
|
||||
"Subconscious" window in the UI (stage 7) without introducing outbound tiny.place effects.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. Add the tick stage + cursor kv; no-op cleanly when orchestration is disabled or tables empty.
|
||||
2. Prompt work: steering section with 2–3 few-shot examples; parse/validate the structured output
|
||||
(reject and retry once on contract violation; skip tick on second failure, log warn).
|
||||
3. Store + supersede/expiry logic; unit tests (expiry by cycle count, supersede chain).
|
||||
4. Integration test: seed fake compressed rows + diff timeline → tick → directive persisted →
|
||||
the stage-5 graph loads it via `execute` on the next cycle (assert it lands in the system prompt of
|
||||
the mock provider call).
|
||||
5. Isolation test: assert the tick's tool surface contains no channel/effect tools and that no
|
||||
tinyplace outbound op is reachable.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A heartbeat tick over seeded orchestration data produces exactly one current directive; the next
|
||||
reasoning cycle demonstrably runs with it injected.
|
||||
- Ticks are idempotent (re-run without new data → no new directive) and cheap when idle.
|
||||
- Directives appear in the Subconscious chat window feed (stage 7 consumes them).
|
||||
- Existing subconscious behaviors (memory diff, planner, notify_user) unchanged — their tests stay
|
||||
green.
|
||||
@@ -1,71 +0,0 @@
|
||||
# Stage 7 — RPC surface + TinyPlaceOrchestrationTab wiring
|
||||
|
||||
## Goal command
|
||||
|
||||
> Expose the orchestration layer over JSON-RPC (`openhuman.orchestration_*`) and rewire
|
||||
> **`app/src/components/intelligence/TinyPlaceOrchestrationTab.tsx`** onto it: the tab's
|
||||
> master / subconscious / per-session chat windows come from the stage-3 store's real
|
||||
> classification and metadata instead of the current `chatKindForEnvelope` string heuristics; add
|
||||
> live updates over the core socket and a composer for the Master window (owner → agent DM).
|
||||
|
||||
## Read first
|
||||
|
||||
- `app/src/components/intelligence/TinyPlaceOrchestrationTab.tsx` (+ its test) — current shape:
|
||||
`ChatKind`/`ChatWindow` model, pinned master+subconscious, `LoadState` incl. `payment_required`.
|
||||
- `app/src/lib/agentworld/invokeApiClient.ts` — `callCoreRpc` pattern, `PaymentRequiredError`.
|
||||
- `src/openhuman/tinyplace/schemas.rs` — internal-registry controller pattern to copy.
|
||||
- `src/openhuman/orchestration/{types.rs, store.rs, ops.rs}` (stages 3–6).
|
||||
- Socket push: `app/src/services/socketService.ts`, `src/openhuman/socket/` — how domain events
|
||||
reach the renderer (dual-socket sync rule).
|
||||
- i18n: `app/src/lib/i18n/en.ts` `tinyplaceOrchestration.*` block (exists; extend).
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **RPC controllers** (`orchestration/schemas.rs`, internal registry — renderer-only, not
|
||||
agent-advertised):
|
||||
- `orchestration.sessions_list` → `{ sessions: OrchestrationSession[] }` (incl. computed
|
||||
`active`, unread counts).
|
||||
- `orchestration.messages_list { chat: "master"|"subconscious"|{sessionId}, limit, before? }`.
|
||||
- `orchestration.send_master_message { body }` → DM to the agent's Master counterpart via the
|
||||
signal-send op (this is the human steering the front-end agent).
|
||||
- `orchestration.mark_read { chat }`.
|
||||
- `orchestration.status` → steering directive (current), last tick, ingest cursor health.
|
||||
2. **Renderer client** `app/src/lib/orchestration/orchestrationClient.ts` following the
|
||||
`invokeApiClient` conventions (`callCoreRpc('openhuman.orchestration_…')`, typed results,
|
||||
`PaymentRequiredError` passthrough where relevant).
|
||||
3. **Tab rewire** (keep the existing layout/UX and testids):
|
||||
- Data source: `sessions_list` + per-selected-chat `messages_list` (drop the client-side
|
||||
bucketing of raw envelopes; keep the file's `ChatWindow` view model).
|
||||
- Live updates: subscribe to an `orchestration.message` socket event (bridge
|
||||
`DomainEvent::OrchestrationSessionMessage` / `ReplyReady` / steering-emitted through the
|
||||
socket domain) → targeted refetch of the affected chat; keep manual Refresh.
|
||||
- Master composer: input + send via `send_master_message`, optimistic append, error surface.
|
||||
- Unread: `mark_read` on chat open; badge from server counts.
|
||||
- Preserve `payment_required` and error states; loading per-pane rather than whole-tab.
|
||||
4. **i18n**: new keys (composer placeholder, send, steering banner, read errors) added to `en.ts`
|
||||
**and all 13 other locales** (`pnpm i18n:check`, `pnpm i18n:english:check` clean).
|
||||
5. **Steering visibility**: pinned Subconscious window shows directives (stage 6 feed); header
|
||||
chip on the tab surfaces the current directive from `orchestration.status`.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. Controllers + schemas + `all.rs` wiring; unit tests on handlers (`RpcOutcome` paths, bad chat
|
||||
key, empty stores).
|
||||
2. Socket bridge for the three orchestration events (follow an existing domain's socket relay).
|
||||
3. `orchestrationClient.ts` + Vitest (RPC name mapping, error classification).
|
||||
4. Tab refactor: extract data hooks (`useOrchestrationChats`) so the component stays presentational
|
||||
and under the ~500-line rule; update `TinyPlaceOrchestrationTab.test.tsx` to mock the new
|
||||
client; add tests for composer send, live-update refetch, unread clearing.
|
||||
5. i18n additions across locales.
|
||||
6. `tests/json_rpc_e2e.rs`: seed store → `sessions_list`/`messages_list`/`send_master_message`
|
||||
round-trip over real RPC with the mock backend.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- With stages 1–6 running and a wrapped Codex/Claude session active, the tab shows that session's
|
||||
window with real label/workspace metadata and both user and assistant messages, updating live
|
||||
without manual refresh.
|
||||
- Master composer sends an E2E DM that reaches the front-end agent (verified in e2e via mock).
|
||||
- Subconscious window shows emitted steering directives.
|
||||
- `pnpm typecheck`, `pnpm lint`, `pnpm test`, `pnpm i18n:check` green; no `import.meta.env` reads;
|
||||
no dynamic imports; all RPC through `core_rpc_relay`.
|
||||
@@ -1,48 +0,0 @@
|
||||
# Stage 8 — End-to-end testing, observability & hardening
|
||||
|
||||
## Goal command
|
||||
|
||||
> Prove the whole loop with automated tests spanning both repos, audit logging/observability
|
||||
> against the project's debug-logging rules, and harden failure modes (relay down, payment
|
||||
> required, malformed envelopes, provider errors) so the layer can run unattended.
|
||||
|
||||
## Read first
|
||||
|
||||
- `tests/json_rpc_e2e.rs`, `scripts/test-rust-with-mock.sh`, `scripts/mock-api-core.mjs`.
|
||||
- `gitbooks/developing/agent-observability.md`, `gitbooks/developing/testing-strategy.md`.
|
||||
- `app/test/e2e/` — WDIO spec conventions.
|
||||
- All stage docs in this folder (the invariants list in `README.md`).
|
||||
|
||||
## Deliverables
|
||||
|
||||
1. **Mock tiny.place relay** additions to the shared mock backend (`scripts/mock-api-core.mjs` or
|
||||
a sibling): DM send/receive + contacts endpoints (enforcing `403 not_a_contact` on unpaired
|
||||
DMs, crossing-request auto-accept) + a scriptable "wrapped session" that emits stage-1
|
||||
envelopes, so the full loop runs hermetically in CI.
|
||||
2. **Cross-layer e2e** (`tests/json_rpc_e2e.rs` + mock relay): scripted session pairs first
|
||||
(stage-2 flow A and flow B both covered), then emits
|
||||
user+assistant envelopes → ingest → frontend pass 1 → reasoning cycle (mock provider) →
|
||||
frontend pass 2 → outbound DM captured by the mock → `orchestration.messages_list` shows the
|
||||
complete conversation; then a subconscious tick emits a directive and the next cycle carries it.
|
||||
3. **Failure-mode suite**: relay 5xx/timeout during ingest and during reply (retry + no message
|
||||
loss), `payment_required` on send (surfaced, not retried blindly), malformed envelope flood
|
||||
(bounded log noise, Master fallback), provider error mid-graph (checkpoint resume, no duplicate
|
||||
outbound DM), scheduler_gate `Paused` (cycles defer, none dropped).
|
||||
4. **Observability audit**: every node/stage logs entry/exit + correlation ids
|
||||
(`session_id`, `cycle_id`, `tick_id`); cycles and sub-agents visible in the agent-observability
|
||||
UI with usage/cost; `orchestration.status` exposes ingest-cursor lag and last-error. Add a
|
||||
`doctor` check for orchestration health if the doctor domain pattern fits.
|
||||
5. **Frontend E2E** (WDIO, mock core): Brain → Orchestration tab renders pinned windows + a seeded
|
||||
session, live-updates on a pushed message, composer send round-trips.
|
||||
6. **Docs**: update `gitbooks/developing/architecture/agent-harness.md` (new graph), a new
|
||||
`gitbooks/developing/architecture/orchestration.md` narrative, and
|
||||
`src/openhuman/about_app/` (user-facing feature registry). Run `pnpm docs:check`.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- One command each side proves the loop: `bash scripts/test-rust-with-mock.sh --test json_rpc_e2e`
|
||||
and `pnpm test` green, including the new suites; WDIO spec passes on the Linux lane.
|
||||
- Coverage gate (≥80% changed lines) passes across frontend + rust-core lanes.
|
||||
- Failure-mode suite green; no unbounded retry loops; no secret/body leakage in captured logs
|
||||
(assert with a log-scan test).
|
||||
- Docs Drift lane green; about_app updated.
|
||||
@@ -1,40 +0,0 @@
|
||||
# Secret Key Keychain Migration
|
||||
|
||||
## Goal
|
||||
|
||||
Move the `SecretStore` master encryption key out of `.secret_key` files and into the OS keychain for real app environments.
|
||||
|
||||
Accepted exception:
|
||||
- unit tests and explicit debug overrides may keep using file-backed storage.
|
||||
|
||||
## Current State
|
||||
|
||||
- Auth/provider credentials already prefer the keychain when available.
|
||||
- `SecretStore` still keeps its master key in `{openhuman_dir}/.secret_key`.
|
||||
- `keyring` currently falls back to a file backend in some non-test environments.
|
||||
|
||||
## Target State
|
||||
|
||||
- `dev`, `staging`, and `prod` use the OS keychain for the `SecretStore` master key.
|
||||
- Existing ciphertext stays on disk unchanged.
|
||||
- Existing `.secret_key` files are migrated into the keychain and then deleted only after verification.
|
||||
- Unit tests continue to work without depending on the host OS keychain.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Change keyring backend selection so `cfg(test)` keeps the file backend, but normal app environments default to the OS backend unless explicitly overridden.
|
||||
2. Teach `SecretStore` to:
|
||||
- derive a stable keychain namespace from the user/openhuman directory
|
||||
- migrate an existing `.secret_key` file into the keychain
|
||||
- create the key in keychain when no legacy file exists
|
||||
- fall back safely when keychain is unavailable
|
||||
3. Add tests for:
|
||||
- legacy `.secret_key` migration
|
||||
- post-migration decrypt compatibility
|
||||
- unit-test file-backed behavior
|
||||
|
||||
## Constraints
|
||||
|
||||
- Never re-encrypt existing payloads unless the ciphertext format itself changes.
|
||||
- Never delete `.secret_key` until the keychain write is verified.
|
||||
- Keep an explicit override path for debugging and recovery.
|
||||
@@ -7,6 +7,13 @@ Status: active plan (2026-07-02). Branch: `issue/4249-finish-tinyagents-migratio
|
||||
> supersedes the workstream ordering below and targets tinyagents 1.4/1.5
|
||||
> (`graph::goals`, `graph::todos`, `NoProgressTracker`, resumable graph
|
||||
> failures).
|
||||
>
|
||||
> **2026-07-04 update:** vendor-crate improvement work (reasoning tokens,
|
||||
> streaming, performance, native Anthropic provider, upstream extractions)
|
||||
> is planned separately in
|
||||
> [`../tinyagents-vendor-improvement-plan/`](../tinyagents-vendor-improvement-plan/)
|
||||
> (workstreams V1–V6, run in parallel with C0–C7; this folder's
|
||||
> `99-deletion-ledger.md` remains the master delete list).
|
||||
|
||||
Goal: **hard-migrate** OpenHuman's agent harness onto the `tinyagents` crate as
|
||||
the library for orchestration, caching, tooling, observability, model
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
# 00 — Harness Audit: what remains in OpenHuman, what can migrate
|
||||
|
||||
Ground truth as of 2026-07-04, main + waves 1–2 (#4473, #4483).
|
||||
|
||||
Scope totals (incl. tests):
|
||||
|
||||
| Tree | LOC | Notes |
|
||||
| ---- | --- | ----- |
|
||||
| `src/openhuman/agent/` | ~56,951 | this audit's core scope |
|
||||
| `src/openhuman/agent_orchestration/` | ~25,749 | lifecycle glue over SDK graphs |
|
||||
| `src/openhuman/tinyagents/` | ~12,185 | the live adapter seam (all turns route here) |
|
||||
| `src/openhuman/inference/` | ~52,955 | provider layer; does NOT wrap tinyagents |
|
||||
|
||||
Classification legend: **(a)** thin adapter over tinyagents · **(b)**
|
||||
migratable/duplicative generic harness logic · **(c)** product policy, stays ·
|
||||
**(d)** unclear.
|
||||
|
||||
## 1. Where execution already lives on tinyagents
|
||||
|
||||
Chat turns (`harness/session/turn/core.rs` → `turn/graph.rs`), channel/CLI
|
||||
turns (`harness/graph.rs`, entry `agent/bus.rs` `agent.run_turn`), and
|
||||
sub-agent turns (`harness/subagent_runner/ops/graph.rs`) all route through
|
||||
`run_turn_via_tinyagents_shared` in the `src/openhuman/tinyagents/` adapter.
|
||||
Middleware, stop hooks, unknown-tool policy, and context compression are on
|
||||
SDK middleware. The legacy `run_tool_call_loop` is deleted.
|
||||
|
||||
Direct `tinyagents::` references inside `agent/` are sparse by design — the
|
||||
adapter module owns the seam. Highest densities: `subagent_runner/ops/runner.rs`
|
||||
and `ops/graph.rs` (15 refs each), `turn/core.rs` (13).
|
||||
|
||||
## 2. Remaining subsystems, ranked by migratable size
|
||||
|
||||
1. **Session/transcript shell — `harness/session/`, ~13.2k LOC.** Transcript
|
||||
read/write + dialect-suffix compat (`transcript.rs` 1347+978 tests),
|
||||
builder assembly (`builder/factory.rs` 1312 — product, stays), turn IO
|
||||
(`session_io.rs`), migration (373), progress projection
|
||||
(`tool_progress.rs` 252). Duplicates crate `Store`/`AppendStore`/
|
||||
`StoreChatHistory`. Old-plan C1 covers the cutover; dual-writes are ON,
|
||||
shadow reads landed (flag OFF). **Biggest single unlock (~9–11k incl.
|
||||
session_db/subagent_sessions/session_import).**
|
||||
2. **Sub-agent runner — `harness/subagent_runner/`, ~6.1k LOC.** Prepare →
|
||||
filter tools → run child → checkpoint → extract → mirror. Duplicates
|
||||
`SubAgent`/`SubAgentSession`/`SubAgentTool` + `graph::subagent_node`.
|
||||
`extract_tool.rs` (612) + `handoff.rs` (287) are generic (old-plan C5/C6);
|
||||
`tool_prep.rs` allowlist half is product.
|
||||
3. **Orchestration lifecycle glue — `agent_orchestration/`, ~3.7k of it.**
|
||||
`running_subagents.rs` **1931** (target ≤300; watch channels, abort/wait/
|
||||
steer, tombstones — belongs on a durable `TaskStore`, see 06),
|
||||
`spawn_parallel_graph.rs` 1764 (dup of `map_reduce`).
|
||||
4. **Progress/streaming projection — ~2.9k LOC.** `progress.rs` (303,
|
||||
`AgentProgress` incl. `ThinkingDelta`/`SubagentThinkingDelta`/
|
||||
`ToolCallArgsDelta`), `session/tool_progress.rs` (252, bridges
|
||||
`ProviderDelta` → `AgentProgress`), `progress_tracing.rs` + `langfuse.rs`
|
||||
+ tests (~2k). Duplicates crate `AgentEvent`/`HarnessEventJournal`/
|
||||
Langfuse exporters. Deletion gated on old-plan C4 journal parity **and**
|
||||
on crate streaming fixes (docs 02/03 here) so nothing regresses in the UI.
|
||||
5. **Tool-call dialects — ~1.9k LOC.** `dispatcher.rs` (609) `ToolDispatcher`
|
||||
trait (native/XML/P-Format), `harness/parse.rs` (833) permissive XML/JSON
|
||||
parser with arg-key drift recovery, `pformat.rs` (499) compact positional
|
||||
`name[a|b]` calls (~80% token cut, OpenHuman-invented). Live path is
|
||||
native; these survive for prompt-guided dialects + transcript compat.
|
||||
Extraction candidate (06) so the C1 read-cutover becomes a pure delete.
|
||||
6. **Multimodal assembly — `multimodal.rs` 1690 (+1020 tests).**
|
||||
`[IMAGE:]`/`[FILE:]` markers → provider content blocks, mime allowlist,
|
||||
PDF extraction, fetch gating, truncation budgets. ~1550 generic (06).
|
||||
7. **Cost — `cost.rs` 343.** Per-turn USD + tier pricing + budget stop-hook
|
||||
feed. Blocked on crate `Usage` having no money field and
|
||||
`reasoning_tokens` never being populated (see 01/02). Old-plan C3 flip
|
||||
criteria stand.
|
||||
8. **Tool policy/filter — `tool_policy.rs` 524 + `harness/tool_filter.rs`
|
||||
299.** Overlaps crate `ToolPolicy`; residual gap is free-form
|
||||
model-visible `ToolSchema` metadata (sdk-gaps §1).
|
||||
|
||||
## 3. Confirmed product — keep in OpenHuman
|
||||
|
||||
- `triage/` (3.8k) — trigger classification; *invokes* the migrated loop.
|
||||
~800-line generic evaluator core is an extraction candidate (06), envelope/
|
||||
escalation/events stay.
|
||||
- `task_dispatcher/` (1.6k) — deterministic card dispatch (CAS claim →
|
||||
autonomous turn). Mechanics shrink onto `graph::todos` (old-plan C2).
|
||||
- `archivist/` (1.4k + 1k tests) — episodic indexing/lessons; memory policy.
|
||||
- `prompts/` (~4k) — section assembly/rendering; product voice.
|
||||
- `agent/tools/` (~4k) — product tools (`run_workflow`, preferences,
|
||||
`delegate_to_personality`, todo CRUD → C2 projections).
|
||||
- `builder/factory.rs`, definitions (`definition.rs` 846,
|
||||
`definition_loader.rs`, `builtin_definitions.rs`), `memory_context.rs`,
|
||||
`memory_protocol.rs`, credentials/sandbox context, `debug/`, `library/`,
|
||||
`bus.rs`, `schemas.rs`.
|
||||
- `inference/` (53k) — stays for credential ownership, billing
|
||||
classification, OAuth, local/Ollama. Largest *parallel* duplication of a
|
||||
tinyagents capability in the tree, but per the standing verdict
|
||||
(`reliable.rs` 900) not forced; revisit only after a native Anthropic
|
||||
provider (05) proves out.
|
||||
|
||||
## 4. Streaming & reasoning today (OpenHuman side)
|
||||
|
||||
Provider deltas (`inference::provider::ProviderDelta::{TextDelta,
|
||||
ThinkingDelta, ToolCallArgsDelta}`) are bridged in
|
||||
`session/tool_progress.rs:226` into `AgentProgress`, a per-request
|
||||
`mpsc::Sender` channel distinct from the DomainEvent bus. Reasoning reaches
|
||||
the UI **only** through this OpenHuman-side path (plus the `ThinkingForwarder`
|
||||
shim), because the crate drops reasoning (01 §3). Fixing 02/03 in the vendor
|
||||
crate lets `AgentProgress` become a thin projection of crate `AgentEvent`s
|
||||
and unblocks deleting `ThinkingForwarder`, `tool_progress.rs`, and eventually
|
||||
the `progress_tracing` stack.
|
||||
|
||||
## 5. Delta vs the old plan's inventory
|
||||
|
||||
The C0–C7 verdicts hold. New/raised items from this sweep:
|
||||
|
||||
- `running_subagents.rs` keeps growing (1931; was flagged at the same size on
|
||||
2026-07-03 — enforce the ≤300 target when C6 lands or it accretes further).
|
||||
- The dialect layer's delete gate ("no live path parses provider text") is
|
||||
reachable only after C1 phase 3 **and** either upstreaming P-Format/XML
|
||||
parsing (06) or accepting native-only transcripts.
|
||||
- `inference/provider/reliable.rs` verdict unchanged, but 05 (native
|
||||
Anthropic in-crate) is the first concrete step that could eventually make
|
||||
routing non-turn provider calls through the crate harness attractive.
|
||||
@@ -0,0 +1,144 @@
|
||||
# 01 — TinyAgents Crate Audit (vendor/tinyagents @ 1.5.0 + ac73382)
|
||||
|
||||
~41.5k non-test LOC (~60k with tests); 54 integration tests, 15 examples.
|
||||
Deps: tokio (sync/time/macros), reqwest (rustls/json/stream), serde, sha2;
|
||||
optional `rusqlite` (`sqlite`), `rhai` (`repl`). No provider-specific SDKs.
|
||||
|
||||
The crate's own `goal.md` is a pre-written gap list from OpenHuman's
|
||||
migration perspective and corroborates most findings below.
|
||||
|
||||
## 1. Module map
|
||||
|
||||
- `harness/` — the agent runtime. Largest: `middleware/` (~5.3k, trait +
|
||||
~18 built-ins incl. retry/timeout/fallback/budget/tool-policy/contextual
|
||||
selection/approval/redaction/prompt-cache guard), `providers/` (~3k:
|
||||
`MockModel` + one real `OpenAiModel`), `agent_loop/` (~2.3k), `model/`
|
||||
(~1.9k: `ChatModel`, `ModelRequest/Response`, `ModelStream`,
|
||||
`StreamAccumulator`, `ModelRegistry`), `observability/` (~1.8k journals/
|
||||
status/Langfuse), `subagent/` (~1.2k), `events/` (~1.2k), plus store/tool/
|
||||
summarization/embeddings/steering/cache/usage/cost/no_progress/testkit.
|
||||
- `graph/` (~19k) — durable typed graphs: compiled runtime, checkpoints
|
||||
(memory/file/sqlite), orchestration `TaskStore` + tools, `todos`, `goals`,
|
||||
`map_reduce`, subagent/subgraph nodes, recursion, export, testkit.
|
||||
- `registry/`, `language/` (.rag), `repl/` (.ragsh, feature-gated).
|
||||
- Ergonomics gap: graph surface is fully re-exported at crate root, but core
|
||||
harness types (`AgentHarness`, `ChatModel`, `ModelRequest`, `OpenAiModel`,
|
||||
middleware, message types) require deep `harness::…` paths (`lib.rs`).
|
||||
|
||||
## 2. The harness loop (what's good)
|
||||
|
||||
`AgentHarness::run_loop` (`harness/agent_loop/mod.rs:293-673`): cancellation
|
||||
checkpoint → steering drain → deadline/call caps → build request →
|
||||
`before_model` → wrap-onion model call with retry + fallback chain
|
||||
(`:766-859`) + per-call `tokio::time::timeout` budget (`:902-917`) + response
|
||||
cache (`:685-756`) → `after_model` → usage/events → `MiddlewareControl`
|
||||
(StopWithFinal/Interrupt) → tools → repeat. Unknown-tool policy
|
||||
`Fail | ReturnToolError | Rewrite` (`:563-618`). Sub-agents nest as tools
|
||||
with a depth cap. Rich extension points; the loop *shape* itself is fixed.
|
||||
|
||||
## 3. Defects & gaps (file:line), grouped
|
||||
|
||||
### Reasoning / thought tokens — scaffolding exists, last mile unwired
|
||||
|
||||
- `ContentBlock` = `Text | Json | Image | ProviderExtension`
|
||||
(`harness/message/types.rs:21-30`) — **no Thinking/Redacted variant**; no
|
||||
persisted home for Anthropic thinking blocks or their signatures.
|
||||
- OpenAI SSE path hardcodes `reasoning: String::new()` per delta
|
||||
(`providers/openai/mod.rs:772`); never parses `reasoning_content` or
|
||||
o-series reasoning.
|
||||
- `StreamAccumulator::finish` **drops accumulated reasoning** — final message
|
||||
built from text + tool chunks only (`model/mod.rs:683-710`).
|
||||
- Loop's middleware-facing `ModelDelta` built with content + tool_call only,
|
||||
dropping reasoning (`agent_loop/mod.rs:980-984`) — the exact sdk-gaps §3
|
||||
item blocking `ThinkingForwarder` deletion.
|
||||
- `Usage.reasoning_tokens` (`usage/types.rs:29`) and
|
||||
`CostTotals.reasoning_cost` (`cost/mod.rs:69`) exist but are **always 0**:
|
||||
`convert_usage` maps only prompt/completion/total/cached
|
||||
(`openai/mod.rs:708-719`); `completion_tokens_details.reasoning_tokens`
|
||||
isn't even in the wire struct (`openai/types.rs:235-256`).
|
||||
- No signature/redacted_thinking handling anywhere → correct multi-turn
|
||||
extended-thinking + tool-use vs native Anthropic is currently impossible.
|
||||
|
||||
### Streaming — real SSE, but observation-only and single-level
|
||||
|
||||
- OpenAI streaming is genuine incremental SSE (`openai/mod.rs:1020-1073`,
|
||||
line buffering `:874-896`, chunk folding `:753-804`); tool-call arg
|
||||
fragments stream as `ToolCallDelta` (`:796-799`).
|
||||
- **`invoke_streaming` returns a completed `AgentRun`, not a `Stream`**
|
||||
(`agent_loop/mod.rs:195-241`); deltas reachable only via `EventSink`/
|
||||
middleware (`:929-994`).
|
||||
- **Sub-agents never stream**: `SubAgentTool::call` →
|
||||
`SubAgent::invoke` non-streaming path (`subagent/mod.rs:535`, `:179-189`).
|
||||
No parent/root attribution on deltas.
|
||||
- No explicit tool-call started/completed stream events.
|
||||
- `MockModel::stream` replays a completed `invoke` (`model/types.rs:536-549`)
|
||||
— mock-backed tests aren't truly incremental.
|
||||
|
||||
### Performance — correctness-first cloning in the hot loop
|
||||
|
||||
- Full history cloned per turn: `ModelRequest::new(messages.clone())`
|
||||
(`agent_loop/mod.rs:356`); full request cloned again **per retry/fallback
|
||||
attempt** (`:804`, `:937`).
|
||||
- Tool schemas rebuilt every iteration: `.with_tools(self.tools.schemas())`
|
||||
(`:356`).
|
||||
- Provider `translate_request` re-clones every message + tool schema and
|
||||
re-serializes tool args per call (`openai/mod.rs:404-449, 570-584`).
|
||||
- Response-cache key = serde round-trip + canonicalize + SHA-256 over the
|
||||
**entire request** per cached call (`cache/mod.rs:101-106`).
|
||||
- `StreamAccumulator::push_tool_chunk` linear scan per fragment
|
||||
(`model/mod.rs:643-650`).
|
||||
- Tools execute **strictly serially** (`agent_loop/mod.rs:526-659`);
|
||||
`parallel_tool_calls` is a capability flag only.
|
||||
|
||||
### Providers — one adapter, many presets
|
||||
|
||||
- Only `OpenAiModel` is real; `anthropic()`/`deepseek`/`groq`/`xai`/
|
||||
`openrouter`/`together`/`mistral`/`ollama` are base-URL presets on the
|
||||
Chat Completions wire (`openai/mod.rs:309-383`).
|
||||
- **No Anthropic Messages API**: zero `cache_control`/`ephemeral` hits in
|
||||
providers; no thinking config; `Usage.cache_creation_tokens` always 0
|
||||
(only `cache_read_tokens` from `prompt_tokens_details.cached_tokens`,
|
||||
`openai/mod.rs:713-716`).
|
||||
- Prompt-cache shaping is metadata-only: `cache_segments` /
|
||||
`prompt_fingerprint` / `cacheable_prefix_ids()`
|
||||
(`model/types.rs:385-405`) + `PromptCacheGuardMiddleware`
|
||||
(`middleware/mod.rs:19-60`) never reach any wire field — the spec's
|
||||
"extreme prompt caching" (`docs/spec/README.md:258-278`) is observability,
|
||||
not request shaping.
|
||||
- Spec explicitly plans feature-flagged native `openai`/`anthropic`/`ollama`
|
||||
(`docs/spec/README.md:279-284`; slot documented at
|
||||
`providers/types.rs:322-337`).
|
||||
|
||||
### Durability / storage
|
||||
|
||||
- `TaskStore` is in-memory/JSONL only; no lifecycle history/replay
|
||||
(goal.md §4/§11) — what `running_subagents.rs` (1931 LOC) needs to shrink.
|
||||
- SQLite ownership: bundled `rusqlite 0.40` vs app-owned sqlite → needs a
|
||||
trait-first/connection-adapter path (goal.md §5).
|
||||
- `Store` has no compare-and-set (single-writer constraint documented in the
|
||||
old plan).
|
||||
|
||||
## 4. Test coverage holes
|
||||
|
||||
No tests for: native Anthropic/prompt caching/thinking (no impl), reasoning
|
||||
end-to-end (only hand-fed accumulator test,
|
||||
`tests/e2e_reasoning_and_selection.rs:57-98`), parallel tool execution,
|
||||
caller-consumable streaming, sub-agent stream propagation. Live-provider
|
||||
tests exist behind env keys; conformance suites exist for graph/checkpoint.
|
||||
|
||||
## 5. Ranked improvement opportunities (effort: S≈1-2d, M≈3-5d, L≈1-2w, XL multi-week)
|
||||
|
||||
1. **Reasoning end-to-end** — L — doc 02. Unblocks `ThinkingForwarder`
|
||||
deletion + real reasoning cost accounting.
|
||||
2. **Hot-loop de-cloning + cache-key + schema caching** — M — doc 04.
|
||||
Biggest runtime win for OpenHuman's long transcripts.
|
||||
3. **Streaming out of the harness + up from sub-agents** — L — doc 03.
|
||||
Unblocks `tool_progress.rs`/progress projection deletions (C4).
|
||||
4. **Native Anthropic Messages provider + cache_control shaping** — L→XL —
|
||||
doc 05. Highest-leverage provider gap.
|
||||
5. **Parallel tool execution** — M — doc 04 §4.
|
||||
6. **Usage completeness (OpenAI reasoning tokens) + catalog pricing** — S→M
|
||||
— doc 02 §5; feeds old-plan C3 budget flip.
|
||||
7. **Durable TaskStore (SQLite/JSONL lifecycle + replay)** — L — doc 06 §4.
|
||||
8. **SQLite dependency flexibility** — M — goal.md §5.
|
||||
9. **Crate-root re-export ergonomics** — S.
|
||||
@@ -0,0 +1,88 @@
|
||||
# 02 — Thought Tokens End-to-End (vendor crate work)
|
||||
|
||||
Goal: reasoning/thinking content becomes a first-class, streamed, persisted,
|
||||
replayed, and priced citizen of the crate — so OpenHuman deletes
|
||||
`ThinkingForwarder` and stops re-projecting thinking via its own
|
||||
`ProviderDelta::ThinkingDelta` bridge (`session/tool_progress.rs:226`).
|
||||
|
||||
All steps are vendor-crate changes (`vendor/tinyagents`), committed on a
|
||||
submodule feature branch, gitlink-bumped into OpenHuman, PR'd upstream.
|
||||
Precedent: `NoProgressTracker` (#7), MicrocompactMiddleware (`ac73382`).
|
||||
|
||||
## Step 1 — Message representation
|
||||
|
||||
- Add `ContentBlock::Thinking { text: String, signature: Option<String> }`
|
||||
and `ContentBlock::RedactedThinking { data: String }` to
|
||||
`harness/message/types.rs:21-30`.
|
||||
- Serde: additively tagged; existing transcripts (no thinking blocks) parse
|
||||
unchanged. Round-trip tests in `message/test.rs`.
|
||||
- `AssistantMessage` rendering helpers must skip thinking blocks for
|
||||
plain-text extraction (`Message::text()`-style accessors) but preserve them
|
||||
for provider replay.
|
||||
|
||||
## Step 2 — Accumulator persistence
|
||||
|
||||
- `StreamAccumulator::finish` (`harness/model/mod.rs:683-710`): emit the
|
||||
accumulated `self.reasoning` as a leading `ContentBlock::Thinking` on the
|
||||
final message instead of dropping it. Carry signature fragments (Anthropic
|
||||
`signature_delta`) through a new accumulator field.
|
||||
- Keep the existing `reasoning()` side-channel accessor for backward compat.
|
||||
|
||||
## Step 3 — Delta plumbing
|
||||
|
||||
- Populate `MessageDelta.reasoning` from providers:
|
||||
- OpenAI path: parse `delta.reasoning_content` (DeepSeek/compat) and
|
||||
o-series reasoning summaries instead of hardcoding `String::new()`
|
||||
(`providers/openai/mod.rs:772`); add the wire fields to
|
||||
`openai/types.rs`.
|
||||
- Anthropic path (05): `thinking_delta` / `signature_delta` events.
|
||||
- Thread reasoning into the middleware-facing `ModelDelta`
|
||||
(`agent_loop/mod.rs:980-984`) — closes sdk-gaps §3. Add
|
||||
`ToolDelta.tool_name` on the first fragment (tool-name-on-start), the
|
||||
other half of that gap.
|
||||
- Emit reasoning in `AgentEvent::ModelDelta` with parent/root run
|
||||
attribution (pairs with doc 03's sub-agent propagation so
|
||||
`SubagentThinkingDelta` can become a projection).
|
||||
|
||||
## Step 4 — Replay correctness (Anthropic contract)
|
||||
|
||||
Anthropic requires thinking blocks (with signatures) to be replayed verbatim
|
||||
in the assistant turn preceding tool results. With Step 1 the blocks live in
|
||||
the transcript; the provider translation (05) must:
|
||||
|
||||
- Serialize `Thinking`/`RedactedThinking` blocks back onto the wire for
|
||||
assistant messages in multi-turn tool-use conversations.
|
||||
- Never send thinking blocks to providers that reject them (OpenAI compat
|
||||
path strips them) — provider capability flag `supports_thinking` on
|
||||
`ModelProfile`.
|
||||
- Property test: build a 3-turn tool-use conversation with thinking, assert
|
||||
byte-stable signature replay.
|
||||
|
||||
## Step 5 — Usage & cost
|
||||
|
||||
- Add `completion_tokens_details.reasoning_tokens` to the OpenAI wire struct
|
||||
(`openai/types.rs:235-256`) and map it in `convert_usage`
|
||||
(`openai/mod.rs:708-719`).
|
||||
- Anthropic (05): thinking output tokens are billed as output; keep
|
||||
`reasoning_tokens` as the *reported* subset where the API exposes it.
|
||||
- `CostTotals.reasoning_cost` (`cost/mod.rs:69`) then prices real numbers —
|
||||
feeds OpenHuman's C3 budget-flip criteria (pricing table wired).
|
||||
|
||||
## Step 6 — OpenHuman follow-through (after gitlink bump)
|
||||
|
||||
1. Delete `ThinkingForwarder` (old-plan C7 item; sdk-gaps §3 closes).
|
||||
2. `AgentProgress::ThinkingDelta`/`SubagentThinkingDelta` become projections
|
||||
of crate `AgentEvent::ModelDelta.reasoning` — shrinks
|
||||
`session/tool_progress.rs` toward deletion (with doc 03).
|
||||
3. Persisted-transcript compat: OpenHuman's `multimodal.rs` reasoning-block
|
||||
handling shrinks once the crate owns thinking blocks in messages.
|
||||
|
||||
## Tests (crate)
|
||||
|
||||
- Provider-level: SSE fixture streams with reasoning deltas → accumulator →
|
||||
message contains `Thinking` block; usage carries reasoning tokens.
|
||||
- Loop-level: middleware `on_model_delta` sees reasoning; `AgentRun.messages`
|
||||
persists it; replay serialization per provider capability.
|
||||
- Extend `tests/e2e_reasoning_and_selection.rs` beyond hand-fed deltas.
|
||||
|
||||
Effort: **L** (1–2 weeks) crate-side; OpenHuman follow-through **S**.
|
||||
@@ -0,0 +1,80 @@
|
||||
# 03 — Streaming Improvements (vendor crate work)
|
||||
|
||||
Goal: the harness exposes a real caller-consumable stream, sub-agent deltas
|
||||
propagate to the parent with lineage attribution, and tool calls get explicit
|
||||
lifecycle events — so OpenHuman's `AgentProgress` layer becomes a thin
|
||||
projection and the C4 progress_tracing deletion has full-fidelity input.
|
||||
|
||||
## Current state (from 01)
|
||||
|
||||
- Provider SSE is genuinely incremental (`openai/mod.rs:1020-1073`) and
|
||||
tool-call arg fragments stream (`ToolCallDelta`, `:796-799`).
|
||||
- But `invoke_streaming` returns a completed `AgentRun`
|
||||
(`agent_loop/mod.rs:195-241`); consumers must attach an `EventSink` or
|
||||
middleware to see deltas.
|
||||
- Sub-agents run the non-streaming path (`subagent/mod.rs:535`, `:179-189`)
|
||||
— a child's tokens/thinking/tool activity are invisible until it returns.
|
||||
- No tool-call started/completed events; `MockModel::stream` replays a
|
||||
completed response (`model/types.rs:536-549`).
|
||||
|
||||
## Step 1 — `invoke_stream`: a Stream-returning entry point
|
||||
|
||||
- New API: `AgentHarness::invoke_stream(...) ->
|
||||
(impl Stream<Item = AgentStreamItem>, JoinHandle<Result<AgentRun>>)` (or a
|
||||
single stream whose terminal item carries the `AgentRun`).
|
||||
- `AgentStreamItem` (new enum, `harness/stream/`): `TurnStarted`,
|
||||
`ModelDelta(ModelDelta)` (text + reasoning + tool_call, post doc 02),
|
||||
`ToolCallStarted { call_id, tool_name }`, `ToolCallArgsDelta`,
|
||||
`ToolCallCompleted { call_id, outcome }`, `AssistantMessage`,
|
||||
`UsageUpdated`, `RunCompleted/Failed` — each stamped with
|
||||
`run_id / parent_run_id / root_run_id / depth`.
|
||||
- Implementation: a bounded `mpsc` fed from the existing emit points in
|
||||
`invoke_model_streaming_once` (`agent_loop/mod.rs:929-994`) and the tool
|
||||
execution section (`:526-659`). The `EventSink` path stays; this is a
|
||||
convenience projection over the same events, not a parallel system.
|
||||
- Backpressure: bounded channel + documented drop-oldest vs block policy
|
||||
(default: block; the loop already awaits network).
|
||||
|
||||
## Step 2 — Sub-agent delta propagation
|
||||
|
||||
- `SubAgentTool::call` switches to the streaming child path and forwards
|
||||
child `AgentStreamItem`s into the parent's stream/sink with the child's
|
||||
`run_id` + inherited `root_run_id` and `depth+1`
|
||||
(`subagent/mod.rs:535` → `SubAgent::invoke_stream`).
|
||||
- Recursion-safe: items flow through the parent's channel; no unbounded
|
||||
fan-in (children are executed serially today; revisit with doc 04 §4).
|
||||
- Closes goal.md §3's "attribute every delta to parent/root run id".
|
||||
|
||||
## Step 3 — Tool-call lifecycle events
|
||||
|
||||
- Emit `ToolCallStarted` as soon as the terminal `Completed` (or the first
|
||||
named `ToolCallDelta`, with doc 02's tool-name-on-start) identifies the
|
||||
tool — this is what OpenHuman's UI needs to show "running X…" before args
|
||||
finish streaming.
|
||||
- Emit `ToolCallCompleted` with outcome + duration after the wrap-onion tool
|
||||
call returns.
|
||||
|
||||
## Step 4 — Test/mock fidelity
|
||||
|
||||
- `MockModel::stream` gains scripted incremental emission (list of
|
||||
`ModelStreamItem`s with optional delays) so streaming tests are truly
|
||||
incremental (`model/types.rs:536-549`).
|
||||
- Tests: caller consumes `invoke_stream` and observes interleaved model/tool
|
||||
items; nested sub-agent test asserts child deltas arrive with correct
|
||||
lineage before the parent's final message.
|
||||
|
||||
## Step 5 — OpenHuman follow-through
|
||||
|
||||
1. `run_turn_via_tinyagents_shared` adapters consume `invoke_stream` (or the
|
||||
sink projection) and map 1:1 onto `AgentProgress` — deleting the
|
||||
`ProviderDelta` bridge in `session/tool_progress.rs` (252).
|
||||
2. `SubagentThinkingDelta`/subagent progress becomes lineage-filtered
|
||||
projection — removes bespoke child-progress plumbing in
|
||||
`subagent_runner/ops/runner.rs`.
|
||||
3. C4 (journal-backed web progress) gains the missing fidelity: journals +
|
||||
live stream share one event vocabulary, enabling the
|
||||
`progress_tracing.rs` + `langfuse.rs` deletion (~2k).
|
||||
|
||||
Effort: **L** crate-side (Steps 1–4), **M** OpenHuman follow-through.
|
||||
Depends on: doc 02 for reasoning-in-deltas (can land in either order; both
|
||||
touch `invoke_model_streaming_once`, so sequence the merges).
|
||||
@@ -0,0 +1,93 @@
|
||||
# 04 — Performance (vendor crate work)
|
||||
|
||||
Goal: remove O(history) per-turn overhead from the agent loop. OpenHuman
|
||||
transcripts are long (multi-hundred-message sessions with large tool
|
||||
results); every listed cost below scales with transcript size and is paid
|
||||
one or more times **per model call**.
|
||||
|
||||
## 1. Stop cloning the transcript per turn/attempt — biggest win
|
||||
|
||||
Today:
|
||||
|
||||
- `ModelRequest::new(messages.clone())` per iteration
|
||||
(`agent_loop/mod.rs:356`).
|
||||
- `model.invoke(state, request.clone())` / `model.stream(state,
|
||||
request.clone())` per retry/fallback **attempt** (`:804`, `:937`) — a
|
||||
3-model fallback chain with 2 retries can clone the full history 6×.
|
||||
|
||||
Plan:
|
||||
|
||||
- Change `ModelRequest.messages` to `Arc<Vec<Message>>` (or `Arc<[Message]>`)
|
||||
with copy-on-write semantics (`Arc::make_mut` on the rare mutation paths —
|
||||
middleware that rewrites history). Public builder API stays source-
|
||||
compatible via `impl Into<Arc<Vec<Message>>>`.
|
||||
- Retry/fallback passes `&ModelRequest`; `ChatModel::invoke/stream` take
|
||||
`&ModelRequest` (breaking trait change — acceptable pre-2.0, coordinated
|
||||
with OpenHuman's adapter in the same gitlink bump).
|
||||
- The loop appends assistant/tool messages to its own `Vec` and rebuilds the
|
||||
`Arc` view per turn (one cheap `Arc::new` of a `Vec` it already owns, or
|
||||
an immutable persistent-list structure if profiling justifies it — start
|
||||
with `Arc<Vec>`, measure).
|
||||
|
||||
## 2. Cache tool schemas per run
|
||||
|
||||
`.with_tools(self.tools.schemas())` rebuilds every `ToolSchema` (cloning
|
||||
name/description/`parameters: Value`) each iteration (`agent_loop/mod.rs:356`).
|
||||
Tool sets are fixed for a run (dynamic-selection middleware operates on the
|
||||
request afterward). Plan: compute `Arc<Vec<ToolSchema>>` once at run start;
|
||||
middleware that filters tools clones only then (copy-on-filter).
|
||||
|
||||
## 3. Response-cache key without full serde round-trips
|
||||
|
||||
`cache_key` = `serde_json::to_value(request)` → canonicalize → `to_vec` →
|
||||
SHA-256 over the entire request per cached call (`cache/mod.rs:101-106`).
|
||||
Plan: maintain an incremental hash — hash each message once when appended
|
||||
(messages are immutable after append), fold message digests + tool-schema
|
||||
digest + params digest into the key. Falls out naturally once messages are
|
||||
`Arc`'d (attach a lazily-computed digest per message).
|
||||
|
||||
## 4. Parallel tool execution
|
||||
|
||||
Tools run strictly serially (`agent_loop/mod.rs:526-659`) while
|
||||
`parallel_tool_calls` exists only as a capability flag. Plan:
|
||||
|
||||
- Execute a turn's tool calls via `futures::stream::iter(...)
|
||||
.buffer_unordered(cap)` but **emit results in request order** (index-sorted
|
||||
reassembly) so transcripts stay deterministic.
|
||||
- Default cap: 1 (today's behavior) — opt-in via
|
||||
`AgentHarnessBuilder::with_tool_concurrency(n)` and a per-tool
|
||||
`ToolPolicy::serial` escape hatch (side-effecting tools opt out).
|
||||
- Middleware contract: `before_tool`/`after_tool` hooks fire per call; wrap
|
||||
middleware must be `Send + Sync` (already is). Events carry call ids so
|
||||
interleaving is attributable (pairs with doc 03 lifecycle events).
|
||||
- OpenHuman: keep cap=1 initially; enable for read-only tool families after
|
||||
the approval-gate interaction is reviewed (approval middleware must park
|
||||
the whole batch, not one call).
|
||||
|
||||
## 5. Provider translation
|
||||
|
||||
`translate_request` re-clones all messages/tool schemas and re-serializes
|
||||
tool args per call (`openai/mod.rs:404-449, 570-584`). Plan: with #1/#2 the
|
||||
inputs are `Arc`'d; add a per-run memo of the translated static prefix
|
||||
(system + tools JSON) keyed by the schema digest from #3. This also makes
|
||||
prefix-stability for provider prompt caches (05) explicit rather than
|
||||
incidental.
|
||||
|
||||
## 6. Minor
|
||||
|
||||
- `StreamAccumulator::push_tool_chunk` linear scan (`model/mod.rs:643-650`)
|
||||
→ index map by `call_id` (only matters for many-tool turns; cheap fix).
|
||||
- `InMemoryResponseCache` mutex per get/put (`cache/mod.rs:119-131`) — fine;
|
||||
re-check only if #3 makes cache use hot.
|
||||
|
||||
## Measurement (gate for merging #1–#3)
|
||||
|
||||
Add a criterion-style bench (or a `live_*`-excluded integration bench) in the
|
||||
crate: synthetic 500-message transcript, 40 tools, 20-turn loop against
|
||||
`MockModel` — assert allocations/turn and wall time before/after. OpenHuman
|
||||
side: `[budget_shadow]`-style log of per-turn overhead in the adapter for one
|
||||
release to confirm on real sessions.
|
||||
|
||||
Effort: #1 **M** (trait-breaking, coordinated bump), #2 **S**, #3 **S–M**,
|
||||
#4 **M**, #5 **S**. Order: #2/#3 first (non-breaking), then #1 (+#5), #4
|
||||
independent.
|
||||
@@ -0,0 +1,83 @@
|
||||
# 05 — Native Anthropic Provider + Prompt-Cache Request Shaping
|
||||
|
||||
Goal: a feature-flagged native Anthropic Messages-API provider in the crate
|
||||
(the spec already reserves the slot, `docs/spec/README.md:279-284`,
|
||||
`providers/types.rs:322-337`), plus wiring the existing cache-segment
|
||||
metadata to real wire fields. Highest-leverage provider gap: today
|
||||
`anthropic()` is an OpenAI-compat preset that silently loses prompt caching,
|
||||
thinking, and cache-write accounting.
|
||||
|
||||
## Why this matters to OpenHuman
|
||||
|
||||
- Claude-family models are primary; every turn without `cache_control`
|
||||
breakpoints pays full input-token price on a growing transcript.
|
||||
- Extended thinking + tool use cannot work correctly at all without
|
||||
signature replay (see 02 §4).
|
||||
- `Usage.cache_creation_tokens` is structurally present but always 0 —
|
||||
OpenHuman's cost accounting (C3 flip) undercounts cache writes.
|
||||
- OpenHuman's own `inference/` layer already handles this correctly; a
|
||||
native crate provider is the precondition for ever routing provider calls
|
||||
through the crate harness (the standing `reliable.rs` gate, old-plan 02.2).
|
||||
|
||||
## Step 1 — `providers/anthropic/` behind `feature = "anthropic"`
|
||||
|
||||
- Messages API: `system` as top-level blocks, `tools` array, `tool_choice`,
|
||||
`max_tokens` required, `anthropic-version` header; reuse the crate's
|
||||
reqwest/rustls stack. No new deps.
|
||||
- Response mapping → `ModelResponse`: content blocks (text, tool_use,
|
||||
thinking, redacted_thinking per doc 02), `stop_reason`, and usage incl.
|
||||
`cache_creation_input_tokens` / `cache_read_input_tokens` →
|
||||
`Usage.cache_creation_tokens` / `cache_read_tokens`.
|
||||
- SSE streaming: `message_start` / `content_block_start` /
|
||||
`content_block_delta` (`text_delta`, `input_json_delta`,
|
||||
`thinking_delta`, `signature_delta`) / `content_block_stop` /
|
||||
`message_delta` — mapped onto `ModelStreamItem` incl. tool-name-on-start
|
||||
(`content_block_start` carries the tool name — doc 02 §3 / doc 03 §3 get
|
||||
this for free on Anthropic).
|
||||
- `ProviderKind::Anthropic` switches from preset to native when the feature
|
||||
is on; preset remains as fallback for compat gateways.
|
||||
|
||||
## Step 2 — `cache_control` shaping from `cache_segments`
|
||||
|
||||
`ModelRequest` already carries `cache_segments` / `prompt_fingerprint` /
|
||||
`cacheable_prefix_ids()` (`model/types.rs:385-405`) — currently ignored by
|
||||
translation. Plan:
|
||||
|
||||
- Translation places up to 4 `cache_control: {type: "ephemeral"}`
|
||||
breakpoints at segment boundaries: tools block, system tail, and the two
|
||||
highest-value conversation prefixes (mirrors OpenHuman's proven layout).
|
||||
- `PromptCacheGuardMiddleware` (`middleware/mod.rs:19-60`) is promoted from
|
||||
observer to enforcement partner: layout-change events now correspond to
|
||||
actual cache invalidation, so its warnings become actionable.
|
||||
- OpenAI path: no wire field needed (implicit prefix caching), but #04 §5's
|
||||
stable-prefix memo guarantees byte-stable prefixes — document that as the
|
||||
OpenAI half of "cache-aware shaping".
|
||||
- TTL/beta options (e.g. 1h cache) via `provider_options` passthrough.
|
||||
|
||||
## Step 3 — Capability & catalog integration
|
||||
|
||||
- `ModelProfile` gains `supports_thinking`, `supports_cache_control`,
|
||||
cache-write pricing multipliers; `ModelCatalog` entries for the Claude
|
||||
family map cache read/write token prices into `CostTotals` (feeds
|
||||
old-plan C3's "pricing table wired" criterion and sdk-gaps §7/§8).
|
||||
|
||||
## Step 4 — Tests
|
||||
|
||||
- Wire-fixture tests: request JSON golden files (system/tools/cache_control
|
||||
placement, thinking replay with signatures); SSE fixture → accumulator →
|
||||
message + usage assertions (cache read/write, thinking tokens).
|
||||
- `live_anthropic.rs` behind `ANTHROPIC_API_KEY`: cache-hit-on-second-call,
|
||||
multi-turn thinking + tool-use replay.
|
||||
|
||||
## Step 5 — OpenHuman follow-through (deliberately conservative)
|
||||
|
||||
- Do **not** immediately reroute OpenHuman's turn traffic — `inference/`
|
||||
stays authoritative (credentials, billing classification, OAuth).
|
||||
- First consumer: sub-agent/background/eval traffic behind a flag, comparing
|
||||
cost + cache-hit telemetry against `inference/` for the same models.
|
||||
- Re-evaluate the `reliable.rs` (900) verdict and old-plan 02.2 only after
|
||||
divergence-free telemetry; that decision gets its own migration note.
|
||||
|
||||
Effort: **L→XL** (Step 1–2 ≈ 2 weeks incl. tests; Steps 3–5 incremental).
|
||||
Depends on doc 02 (thinking blocks in `ContentBlock`) for full value; can
|
||||
land Step 1 text/tool support before thinking if 02 lags.
|
||||
@@ -0,0 +1,98 @@
|
||||
# 06 — Upstream Extraction: OpenHuman logic to move INTO tinyagents
|
||||
|
||||
Extends the old plan's C5 batch with the vendor-submodule workflow: extract
|
||||
generic harness logic into `vendor/tinyagents`, bump the gitlink, shrink the
|
||||
local file to product residue, then delete. Two extractions already shipped
|
||||
this way (`NoProgressTracker` → 1.5.0, MicrocompactMiddleware → `ac73382`,
|
||||
upstream PR pending on `feat/microcompact-middleware`).
|
||||
|
||||
Ordered by value (generic-LOC reclaimed locally + capability gained by the
|
||||
crate). Figures are non-test lines; tests roughly double each.
|
||||
|
||||
## 1. Tool-call dialect layer (~1,900) — `pformat.rs` + `dispatcher.rs` + `harness/parse.rs`
|
||||
|
||||
- Crate gains: a `ToolCallDialect` trait beside native tool calling —
|
||||
P-Format compact positional calls (`name[a|b]`, ~80% token cut on
|
||||
tool-heavy turns — a genuine crate selling point for small/local models
|
||||
that lack native tool calling), permissive XML/JSON parsing with arg-key
|
||||
drift recovery (`TOOL_ARG_KEYS`: arguments|args|parameters|params|input).
|
||||
- Product residue: none identified — zero DomainEvent coupling.
|
||||
- Unlock: old-plan C1 step 4 becomes a pure local delete once no live path
|
||||
parses provider text (transcript compat reads move with it).
|
||||
|
||||
## 2. Multimodal attachment resolver (~1,550 of 1,690) — `multimodal.rs`
|
||||
|
||||
- Crate gains: marker → provider content-block resolution, mime allowlist,
|
||||
PDF text extraction, fetch gating, truncation budgets.
|
||||
- Product residue: the `[IMAGE:]`/`[FILE:]` marker convention itself.
|
||||
- Note: land after doc 02 so thinking/reasoning block handling doesn't need
|
||||
to be ported twice.
|
||||
|
||||
## 3. Overflow-to-artifact tool-result store (~500) — `tool_result_artifacts/`
|
||||
|
||||
- Already built on crate `Store`; overflow-to-artifact is a generic harness
|
||||
concern (pairs with the crate's microcompact). Residue: PII scrub hook.
|
||||
- Include `subagent_runner/extract_tool.rs` (612) + `handoff.rs` (287):
|
||||
progressive-disclosure Q&A over handoff-cached oversized results — the
|
||||
natural companion API.
|
||||
|
||||
## 4. Durable TaskStore upgrade + detached lifecycle (~crate work enabling a 1,631-line local delete)
|
||||
|
||||
- Crate gains: SQLite-backed `TaskStore` (behind the existing `sqlite`
|
||||
feature) with lifecycle history, cancellation records, wait/kill/steer
|
||||
handles, replay/listing by parent/root/thread (goal.md §4/§11).
|
||||
- Unlock: `agent_orchestration/running_subagents.rs` 1931 → ≤300 (old-plan
|
||||
C6.2) stops being an adapter squeeze and becomes a projection.
|
||||
|
||||
## 5. Hook trait machinery (~450) — `hooks.rs` + `stop_hooks.rs`
|
||||
|
||||
- Crate gains: `PostTurnHook`/`StopHook` traits + scheduling shell (incl.
|
||||
the archivist's ~200-line hook-scheduling shell). Product hook bodies
|
||||
(archivist, memory, cost stop-hooks) stay local.
|
||||
|
||||
## 6. Host runtime adapter core (~350) — `host_runtime.rs`
|
||||
|
||||
- Native/Docker `RuntimeAdapter` overlaps crate `WorkspaceIsolation`;
|
||||
merge as backends of one crate abstraction.
|
||||
|
||||
## 7. Fuzzy tool ranker (~250) — `tool_filter.rs`
|
||||
|
||||
- Generic ranking into `ContextualToolSelectionMiddleware` (whose shadow
|
||||
flip is old-plan C3.1); Composio input types stay product.
|
||||
|
||||
## 8. LLM triage node (~800 of 3,779) — `triage/` evaluator core
|
||||
|
||||
- Crate gains: a generic "classify with tiered model fallback + cache +
|
||||
verdict parse" graph node. Envelope/escalation/events (Composio,
|
||||
DomainEvents, agent ids) stay product.
|
||||
|
||||
## 9. Small fry
|
||||
|
||||
- `ArgRecoveryMiddleware` core (~150).
|
||||
- Task-local context carriers (`fork_context`, `sandbox_context`,
|
||||
`task_recency_context`, `turn_attachments_context`, ~500): not an
|
||||
extraction — replace with typed fields on the crate execution context
|
||||
(`ToolExecutionContext`/`RunContext`), old-plan C6.4. Requires a crate API
|
||||
addition (generic `extensions: TypeMap` on the run context is the clean
|
||||
form) — do that API change first, then delete all four locally.
|
||||
|
||||
## Workflow per extraction (checklist)
|
||||
|
||||
1. Branch in `vendor/tinyagents` (`feat/<name>`); port code + tests to crate
|
||||
idioms (types.rs/test.rs split, ≤500-line files, module README).
|
||||
2. `cargo fmt && cargo clippy --all-targets -- -D warnings && cargo test`
|
||||
in the submodule.
|
||||
3. Bump gitlink in OpenHuman on a paired branch; swap local callers to the
|
||||
crate type; shrink local file to residue (or delete); run OpenHuman
|
||||
suites per the two-branch CI model.
|
||||
4. Tick `99-deletion-ledger.md` in the old plan folder (it remains the
|
||||
master ledger) and note the extraction here.
|
||||
5. Push submodule branch to `tinyhumansai/tinyagents`, open upstream PR.
|
||||
Local gitlink may run ahead of upstream merge (already the case for
|
||||
`ac73382`) — keep the queue shallow: ≤2 unmerged submodule branches.
|
||||
|
||||
## License note
|
||||
|
||||
tinyagents is GPL-3.0-only; OpenHuman consumes it in-tree already, so
|
||||
extraction changes nothing legally — but extracted code becomes GPL. Flag
|
||||
any file with third-party-licensed snippets before moving (none known).
|
||||
@@ -0,0 +1,80 @@
|
||||
# 07 — Execution Order, Gates, and Reclaim
|
||||
|
||||
This plan runs **alongside** the old plan's C0–C7 (which continues:
|
||||
wave 3 = flip `session_shadow_reads`, flip crate BudgetMiddleware to
|
||||
enforcing, microcompact upstream PR). The vendor workstreams here are named
|
||||
**V1–V6** to avoid collision.
|
||||
|
||||
## Workstreams
|
||||
|
||||
| WS | Doc | Theme | Effort | Blocks / unblocks |
|
||||
| -- | --- | ----- | ------ | ----------------- |
|
||||
| V1 | 04 §2–3 | Non-breaking perf (schema cache, incremental cache key) | S–M | none; immediate win |
|
||||
| V2 | 02 | Reasoning end-to-end | L | deletes `ThinkingForwarder` (C7); feeds C3 pricing |
|
||||
| V3 | 03 | invoke_stream + sub-agent propagation + tool lifecycle | L | C4 progress_tracing deletion; `tool_progress.rs` delete |
|
||||
| V4 | 04 §1,4,5 | Arc'd messages (trait break), parallel tools, translation memo | M | coordinated gitlink bump with adapter changes |
|
||||
| V5 | 05 | Native Anthropic + cache_control shaping | L→XL | needs V2 for thinking; enables 02.2/`reliable.rs` re-eval |
|
||||
| V6 | 06 | Upstream extraction batch (dialects, multimodal, artifacts, TaskStore, hooks, ranker, triage node) | rolling | each unlocks a ledger delete; TaskStore item unlocks C6.2 |
|
||||
|
||||
## Suggested sequence
|
||||
|
||||
```
|
||||
now ──► V1 (quick, non-breaking)
|
||||
├─► V2 reasoning ──► V5 Anthropic (thinking replay) ─► 02.2 re-eval
|
||||
├─► V3 streaming ──► C4 progress deletion
|
||||
├─► V4 perf-breaking (single coordinated bump after V2/V3 merge)
|
||||
└─► V6 extractions (continuous, 1–2 in flight, ≤2 unmerged submodule branches)
|
||||
old plan in parallel: wave 3 (C1 read flip, C3 budget flip) — independent
|
||||
```
|
||||
|
||||
Rationale:
|
||||
|
||||
- V1 first: zero-risk, benefits every live turn, no API changes.
|
||||
- V2 before V5: thinking blocks must exist in `ContentBlock` before the
|
||||
Anthropic adapter can replay them; V5 Step 1 (text/tools) may start early.
|
||||
- V2 and V3 both touch `invoke_model_streaming_once` — land V2 first
|
||||
(smaller diff), rebase V3.
|
||||
- V4's `ChatModel` trait break rides one coordinated gitlink bump so
|
||||
OpenHuman's adapter updates land atomically with it.
|
||||
- C1 (sessions read flip) is **independent** of all V-work — don't serialize
|
||||
behind it; but the dialect extraction (V6.1) should merge before C1
|
||||
step 4's dispatcher/parse/pformat delete.
|
||||
|
||||
## Gates (each item ships only when its gate is green)
|
||||
|
||||
| Item | Gate |
|
||||
| ---- | ---- |
|
||||
| V2 merge | crate reasoning e2e tests + OpenHuman turn with thinking model shows deltas + persisted blocks + nonzero reasoning_tokens |
|
||||
| ThinkingForwarder delete | V2 gitlink bump live in one release, no `[thinking]` divergence logs |
|
||||
| V3 merge | nested sub-agent stream test green; `AgentProgress` projection parity vs current UI events (fixture diff) |
|
||||
| progress_tracing delete | C4 journal parity AND V3 projection parity |
|
||||
| V4 merge | crate bench: ≥50% allocation reduction on 500-msg synthetic; OpenHuman conformance suite green single-threaded |
|
||||
| Parallel tools ON (cap>1) | approval-gate batch-parking reviewed; read-only tool families only |
|
||||
| V5 first traffic | flag-gated non-turn traffic; cost/cache telemetry matches `inference/` within tolerance |
|
||||
| Each V6 extraction | crate tests + local residue shrink + ledger tick + upstream PR opened |
|
||||
|
||||
## Expected reclaim (local, incl. tests; adds to old plan §5's ~29k)
|
||||
|
||||
| Source | Lines |
|
||||
| ------ | ----- |
|
||||
| `ThinkingForwarder` + `tool_progress.rs` + progress projection residue (V2+V3) | ~700 |
|
||||
| progress_tracing stack (C4, unblocked by V3) | ~2,000 (already counted in old plan) |
|
||||
| Dialect layer local delete (V6.1 + C1.4) | ~1,900+tests (counted in old plan C1) |
|
||||
| `running_subagents.rs` squeeze via durable TaskStore (V6.4 + C6) | ~1,600 (counted in old plan C6) |
|
||||
| Task-local context carriers via crate RunContext extensions (V6.9) | ~500 |
|
||||
|
||||
Net-new local reclaim beyond the old plan: **~1–2k lines** — the headline
|
||||
value of V-work is not deletion but **capability and cost**: correct thought
|
||||
tokens, live sub-agent streaming, prompt-cache dollars, and a faster loop on
|
||||
every turn. The old plan's ~29k reclaim proceeds in parallel and several of
|
||||
its gates (C4, C6.2, C1.4, C3 pricing, C7 ThinkingForwarder) become
|
||||
*reachable* only because of V2/V3/V5/V6.
|
||||
|
||||
## Tracking
|
||||
|
||||
- Deletions continue to tick `../tinyagents-full-migration-plan/99-deletion-ledger.md`.
|
||||
- Submodule branch queue + upstream PR status tracked in this folder's
|
||||
README as extractions ship (keep ≤2 unmerged).
|
||||
- Re-audit `docs/tinyagents-sdk-gaps.md` after V2/V3/V5 land — they close
|
||||
gaps §3 (reasoning stream), part of §6 (event fidelity), §7 pricing
|
||||
inputs, and the provider half of §8.
|
||||
@@ -0,0 +1,79 @@
|
||||
# TinyAgents Vendor Improvement & Harness Migration Plan
|
||||
|
||||
Status: new plan (2026-07-04). Branch: `docs/tinyagents-vendor-audit-plan`.
|
||||
|
||||
This folder is the successor planning surface to
|
||||
[`../tinyagents-full-migration-plan/`](../tinyagents-full-migration-plan/)
|
||||
(whose `CONTINUATION-2026-07.md` C0–C7 workstreams remain the authoritative
|
||||
*migration* backlog). What changed and why this folder exists:
|
||||
|
||||
1. **TinyAgents is now a vendored, editable submodule** (`vendor/tinyagents`,
|
||||
pinned at `v1.5.0` + `ac73382` MicrocompactMiddleware, patched into both
|
||||
Cargo worlds via `[patch.crates-io]`). Every "file an upstream issue and
|
||||
wait" item in the old plan is now **direct work we can do in-tree**, test
|
||||
against OpenHuman immediately, and PR upstream from the submodule.
|
||||
2. A fresh ground-truth audit (2026-07-04, three parallel sweeps: harness
|
||||
inventory, crate internals, docs status) surfaced **crate-side defects and
|
||||
gaps that no prior doc captured** — dropped reasoning tokens, a
|
||||
preset-only "Anthropic" provider, observability-only prompt caching, and
|
||||
hot-loop cloning. Fixing these in the vendor crate multiplies the value of
|
||||
every migration workstream.
|
||||
|
||||
## Folder map
|
||||
|
||||
| Doc | Theme |
|
||||
| --- | ----- |
|
||||
| [`00-audit-harness.md`](00-audit-harness.md) | What's left in `src/openhuman/agent/` (+ orchestration/inference), classified: adapter / migratable / product |
|
||||
| [`01-audit-tinyagents.md`](01-audit-tinyagents.md) | Crate capability map + concrete defects found (file:line) |
|
||||
| [`02-reasoning-thought-tokens.md`](02-reasoning-thought-tokens.md) | End-to-end thought-token plan: `ContentBlock::Thinking`, signatures, deltas, usage/cost |
|
||||
| [`03-streaming.md`](03-streaming.md) | Stream-returning harness entry point, sub-agent delta propagation, tool-call lifecycle events |
|
||||
| [`04-performance.md`](04-performance.md) | Clone elimination, schema caching, cache-key hashing, parallel tool execution |
|
||||
| [`05-anthropic-prompt-caching.md`](05-anthropic-prompt-caching.md) | Native Anthropic Messages provider + `cache_control` request shaping |
|
||||
| [`06-upstream-extraction.md`](06-upstream-extraction.md) | OpenHuman logic to move INTO tinyagents (extends old-plan C5) + deletions unlocked |
|
||||
| [`07-execution-order.md`](07-execution-order.md) | Sequencing, gates, dependency graph vs C0–C7, reclaim estimates |
|
||||
|
||||
## Headline findings (TL;DR)
|
||||
|
||||
**Migration headroom.** `src/openhuman/agent/` is ~57k LOC (+~26k in
|
||||
`agent_orchestration/`, ~12k adapter in `src/openhuman/tinyagents/`). The turn
|
||||
loop already runs on tinyagents (`run_turn_via_tinyagents_shared`); the
|
||||
remaining deletable/migratable surface is ~29k LOC, dominated by the session/
|
||||
transcript shell (~13k), sub-agent runner (~6k), orchestration lifecycle glue
|
||||
(~3.7k), progress/tracing projection (~2.9k), and tool-call dialects (~1.9k).
|
||||
See `00-audit-harness.md`.
|
||||
|
||||
**Crate defects that block deletions today** (all now fixable in-vendor):
|
||||
|
||||
- **Reasoning is dropped end-to-end**: the OpenAI SSE path hardcodes
|
||||
`reasoning: ""`, `StreamAccumulator::finish` discards accumulated
|
||||
reasoning, the loop's middleware `ModelDelta` omits it, and
|
||||
`Usage.reasoning_tokens` is never populated. This is why OpenHuman's
|
||||
`ThinkingForwarder` shim still exists.
|
||||
- **"Anthropic" is an OpenAI-compat preset**, not a Messages-API adapter: no
|
||||
`cache_control` prompt caching, no thinking blocks, no
|
||||
`cache_creation_input_tokens` accounting.
|
||||
- **Prompt-cache shaping is observability-only**: `cache_segments` /
|
||||
`cacheable_prefix_ids()` never reach any wire field.
|
||||
- **Hot-loop cloning**: full message history cloned per turn *and* per
|
||||
retry/fallback attempt; tool schemas rebuilt per iteration; response-cache
|
||||
key re-serializes the whole transcript per call.
|
||||
- **No caller-facing stream**: `invoke_streaming` returns a completed run;
|
||||
sub-agents never stream to parents; no tool-call start/complete events.
|
||||
- **Tools execute strictly serially** despite a `parallel_tool_calls`
|
||||
capability flag.
|
||||
|
||||
**Sequencing in one line**: fix reasoning + streaming + perf in the vendor
|
||||
crate first (02/03/04 — they unblock `ThinkingForwarder` deletion and improve
|
||||
every live turn immediately), land the Anthropic provider (05) next, and run
|
||||
upstream extraction (06) in parallel with the old plan's C1 sessions cutover.
|
||||
|
||||
## Rules (inherited, unchanged)
|
||||
|
||||
- Approval/security/sandbox/workspace/credential boundaries are inviolate.
|
||||
- JSON-RPC contracts stay stable unless a migration note lands with the change.
|
||||
- Adapter first → proven parity → delete; deletions are mandatory and named.
|
||||
- Vendor-crate changes: commit in `vendor/tinyagents` on a feature branch,
|
||||
bump the gitlink in OpenHuman, PR the submodule diff upstream
|
||||
(precedent: `NoProgressTracker` #7, MicrocompactMiddleware `ac73382`).
|
||||
- Stage files explicitly (`git add <paths>`); verify branch before commit.
|
||||
- Keep every doc in this folder ≤500 lines.
|
||||
@@ -1,151 +0,0 @@
|
||||
# Phase 1.5 Implementation Plan — `automate(app, goal)`
|
||||
|
||||
**Parent tracker:** [`voice-system-actions.md`](voice-system-actions.md) (Change 1.14 / Phase 1.5)
|
||||
**Decided approach:** Rust inner loop + fast model (chat LLM out of the click loop)
|
||||
**First proof target:** Music — "play `<song>`" end-to-end
|
||||
**Status:** Plan — awaiting approval before code
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Turn a single high-level intent ("play Numb by Linkin Park") into a multi-step UI
|
||||
automation that completes in **one tool call from the orchestrator**, runs fast,
|
||||
and self-corrects — instead of N separate chat-LLM turns over the raw
|
||||
`ax_interact` primitives (today's flow; see tracker §1.10–1.13 for why that's
|
||||
slow and fragile).
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
```text
|
||||
orchestrator (chat LLM)
|
||||
│ one call: automate{ app, goal }
|
||||
▼
|
||||
AutomateTool (tools/impl/computer/automate.rs)
|
||||
│ delegates to
|
||||
▼
|
||||
accessibility::automate::run(app, goal) ← the inner loop (Rust)
|
||||
│
|
||||
├─ fast-path dispatch ── app_fastpaths/{music,spotify,slack}.rs
|
||||
│ (deterministic; skip the loop entirely when available)
|
||||
│
|
||||
└─ general loop ──► perceive → decide → act → settle → verify ──┐
|
||||
▲ │
|
||||
└────────────── repeat until done / fail / budget ───────┘
|
||||
perceive: ax_list_elements_filtered (existing)
|
||||
decide: create_chat_provider("automation", cfg) → JSON action
|
||||
act: ax_press_element / ax_set_field_value / launch_app (existing)
|
||||
settle: helper "ax_wait_settled" (new) — AXObserver, not sleep
|
||||
verify: re-read state; confirm the action took effect
|
||||
```
|
||||
|
||||
The **chat model is invoked once** (to pick `automate` and its `goal`). The
|
||||
**fast model** runs the inner loop with a tiny context (goal + current filtered
|
||||
snapshot + last result), so each step is ~0.5–1s and cheap.
|
||||
|
||||
## 3. Inner-loop algorithm
|
||||
|
||||
State carried across iterations: `goal`, `app`, `history: Vec<Step>`, `budget`.
|
||||
|
||||
Each iteration:
|
||||
1. **Perceive** — `ax_list_elements_filtered(app, last_filter_or_"")`, capped/filtered
|
||||
exactly as the `ax_interact` tool does today (≤60 elements, never a raw dump).
|
||||
2. **Decide** — call the fast model with a strict system prompt + the JSON action
|
||||
schema (below). Parse one action.
|
||||
3. **Act** — execute via existing helpers. `launch` → `launch_app`; `press` →
|
||||
`ax_press_element`; `set_value` → `ax_set_field_value`; `list` → just re-perceive
|
||||
with a new filter.
|
||||
4. **Settle** — `ax_wait_settled(app, timeout)` (new helper): block until the AX
|
||||
tree stops changing (debounced AXObserver notifications) or timeout. Removes the
|
||||
timing-race class deterministically.
|
||||
5. **Verify** — re-read; confirm the expected post-condition (e.g. a new control
|
||||
appeared, focus changed, a value was set). Record success/failure in `history`.
|
||||
6. **Loop** until the model emits `done`/`fail`, or the step budget (e.g. 12) is hit.
|
||||
|
||||
### Action schema (fast model output — strict JSON)
|
||||
```jsonc
|
||||
{
|
||||
"thought": "short reasoning",
|
||||
"action": "launch | list | press | set_value | done | fail",
|
||||
"app": "Music", // optional override; defaults to the task app
|
||||
"filter": "Highway", // for list
|
||||
"label": "Play", // for press / set_value
|
||||
"value": "Highway to Hell", // for set_value
|
||||
"summary": "what happened / why done" // for done|fail
|
||||
}
|
||||
```
|
||||
Invalid JSON or unknown action → one repair retry, then `fail` with the raw text
|
||||
logged (never act on a guess — this is the §1.13 hallucination lesson).
|
||||
|
||||
## 4. New files & changes (grounded in current layout)
|
||||
|
||||
**New**
|
||||
- `src/openhuman/accessibility/automate.rs` — `run(app, goal, opts) -> Result<AutomateOutcome, String>`; the loop, action schema (serde), fast-model call, step budget, structured `history`.
|
||||
- `src/openhuman/accessibility/app_fastpaths/mod.rs` + `music.rs` (Spotify/Slack land later) — `try_fastpath(app, goal) -> Option<Result<…>>`.
|
||||
- `src/openhuman/tools/impl/computer/automate.rs` — `AutomateTool { allow_mutations }`; reuses the `ax_interact` gating posture (mutations opt-in, `SENSITIVE_APPS` denylist, `permission_level_with_args` = Dangerous, `external_effect_with_args` = true).
|
||||
- `src/openhuman/accessibility/automate_tests.rs` — unit tests for the loop (mock perceive/act/decide), schema parse/repair, budget, fast-path dispatch.
|
||||
|
||||
**Changed**
|
||||
- `accessibility/helper.rs` (macOS Swift) — add `ax_wait_settled` (AXObserver on `kAXValueChanged`/`kAXFocusedUIElementChanged`/`kAXCreated`, debounce ~150ms, bounded ~3s) and return richer element fields (enabled / on-screen / supported actions) from `ax_list`.
|
||||
- `accessibility/ax_interact.rs` — surface a `ax_wait_settled` Rust wrapper; extend `AXElement` with the new optional fields (back-compat: `#[serde(default)]`).
|
||||
- `accessibility/mod.rs` — declare `automate`, `app_fastpaths`.
|
||||
- `inference/provider/factory.rs` — add an `"automation"` role (falls back to the fast/summarization tier) so the loop's model is independently configurable.
|
||||
- `tools/ops.rs` (`all_tools_with_runtime`), `tools/user_filter.rs` (new `"automate"` family), `agent_registry/agents/orchestrator/agent.toml` (`named` list), `app/src/utils/toolDefinitions.ts` (Settings → Agent Access toggle).
|
||||
- Tracker: flip Change 1.14 / Phase 1.5 rows from ⏳ Planned → in progress as milestones land.
|
||||
|
||||
## 5. Fast-model call
|
||||
|
||||
`create_chat_provider("automation", &cfg)` → `(provider, model)`; build a
|
||||
`ChatRequest { messages, tools: None, stream: None }` with a system prompt that
|
||||
pins the JSON schema and a user message carrying `{goal, snapshot, history_tail}`.
|
||||
No tools array — we want a single JSON object back, parsed by us, executed by us.
|
||||
Temperature low. Token budget small (snapshot is already ≤60 elements).
|
||||
|
||||
## 6. Music proof (first target)
|
||||
|
||||
`app_fastpaths/music.rs` encodes the §1.11 proven sequence behind one entry:
|
||||
1. `launch_app("Music")`
|
||||
2. open `music://music.apple.com/search?term=<query>` (URL scheme)
|
||||
3. `ax_wait_settled`
|
||||
4. `ax_list_elements_filtered("Music", <query>)` → find the song row
|
||||
5. `ax_press_element` the row (navigate into detail)
|
||||
6. `ax_wait_settled` → `ax_list` the detail page → `ax_press_element("Play")`
|
||||
7. verify `osascript … get player state == playing` (best-effort, logged)
|
||||
|
||||
If the fast-path can't find the row (timing/locale), fall through to the **general
|
||||
loop**, which is what proves the architecture is app-agnostic.
|
||||
|
||||
## 7. Progress streaming
|
||||
|
||||
Emit a `DomainEvent` per step (`AutomateProgress { app, step, action, ok }`) on the
|
||||
event bus; a subscriber bridges to the existing notch/voice status surface
|
||||
(PR #3166) so the user sees "Opening Music → searching → playing" live. Reuses the
|
||||
`ApprovalSurfaceSubscriber` bridging pattern.
|
||||
|
||||
## 8. Testing
|
||||
|
||||
- **Unit** (`automate_tests.rs`, CI-safe): action JSON parse + repair; budget exhaustion → `fail`; fast-path dispatch chosen over loop; verify-failure triggers retry/alternate. Perceive/act/decide are trait-injected so tests need no mic/AX/LLM.
|
||||
- **Integration** (`#[ignore]`, run on a real Mac): the Music flow end-to-end (mirrors `ax_interact_tests::test_full_flow_search_and_play_acdc`); tool-level success hard-asserted, playback best-effort.
|
||||
- **Agent-in-the-loop**: ask the running app "play `<song>`", confirm it picks `automate` and the song plays; watch `[automate]` logs.
|
||||
|
||||
## 9. Milestones (sequenced)
|
||||
|
||||
1. **M1** — `automate.rs` loop skeleton + action schema + fast-model call + `AutomateTool` (gated, registered). Loop runs against existing (non-settled) `ax_interact` helpers. Unit tests. *Compiles + agent can call it.*
|
||||
2. **M2** — `ax_wait_settled` (helper + wrapper) + verify step wired into the loop. Kills the timing-race class.
|
||||
3. **M3** — Music fast-path; prove the flow end-to-end on a Mac.
|
||||
4. **M4** — progress streaming to the notch surface.
|
||||
5. **M5** — richer element model (enabled/onscreen/actions) for better matching.
|
||||
6. *(later)* Spotify + Slack fast-paths; vision fallback for Electron; Windows UIA settle parity.
|
||||
|
||||
## 10. Risks / open questions
|
||||
|
||||
- **Fast model availability** — if no fast tier is configured, fall back to the
|
||||
chat model for the loop (still one tool call; just slower). The `"automation"`
|
||||
role makes this a config decision, not a hard dependency.
|
||||
- **AXObserver from the Swift helper** — needs a short run-loop pump; if flaky,
|
||||
fall back to a polling settle (count-stable-for-150ms) behind the same wrapper.
|
||||
- **macOS-only first** — Windows UIA settle/verify parity is M6, gated like the
|
||||
existing cfg-dispatch; non-mac/non-win returns the existing clean runtime error.
|
||||
- **Safety** — `automate` is a mutating tool: same opt-in + `SENSITIVE_APPS`
|
||||
denylist + ApprovalGate routing as `ax_interact`; the inner loop may not target a
|
||||
denylisted app even if the model asks.
|
||||
@@ -1,788 +0,0 @@
|
||||
# Voice → System Action Feature Tracker
|
||||
|
||||
**GitHub Issue:** [#3148](https://github.com/tinyhumansai/openhuman/issues/3148)
|
||||
**Branch:** `feat/voice-always-on-all` (cumulative) — **all merged to `main`.** Phase 1 landed via [#3168](https://github.com/tinyhumansai/openhuman/pull/3168); the full feature was split from the mega-PR [#3307](https://github.com/tinyhumansai/openhuman/pull/3307) (now **closed** in favour of the stack) into an 8-PR stack — **all 8 merged 2026-06-04**: [#3340](https://github.com/tinyhumansai/openhuman/pull/3340) (main-thread input + CEF fix), [#3341](https://github.com/tinyhumansai/openhuman/pull/3341) (AX/UIA perception + automate engine), [#3342](https://github.com/tinyhumansai/openhuman/pull/3342) (wire automate/ax_interact tools), [#3343](https://github.com/tinyhumansai/openhuman/pull/3343) (Phase 2 always-on engine + RPC), [#3344](https://github.com/tinyhumansai/openhuman/pull/3344) (always-on Settings toggle + i18n), [#3345](https://github.com/tinyhumansai/openhuman/pull/3345) (notch status pill — supersedes the closed [#3166](https://github.com/tinyhumansai/openhuman/pull/3166)), [#3346](https://github.com/tinyhumansai/openhuman/pull/3346) (Phase 3 fast command router), [#3362](https://github.com/tinyhumansai/openhuman/pull/3362) (Phase 1.5 vision-click fallback)
|
||||
**Started:** 2026-06-02
|
||||
**Last updated:** 2026-06-09 — added **Change 1.17** (PR [#3558](https://github.com/tinyhumansai/openhuman/pull/3558)): browser & app (Spotify/Apple Music/Slack) keyboard-shortcut fast-paths, cross-platform shortcut tables, the `hotkey` verb, artist-aware Music verification, and actionable `automate` failure responses — all from live-transcript analysis. Earlier this day: verified all 8 always-on stack PRs merged to `main` (code paths confirmed post-refactor [#3424](https://github.com/tinyhumansai/openhuman/pull/3424)); global push-to-talk ([#3090](https://github.com/tinyhumansai/openhuman/issues/3090) via [#3349](https://github.com/tinyhumansai/openhuman/pull/3349)) landed separately. **Only open item across all phases: the on-device audio wake-word model (Phase 3) — not started; text-based "Hey Tiny" match remains the interim.**
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Enable the app to continuously listen to the user, understand spoken commands, and perform system actions on the laptop — e.g., saying *"open my Music player"* causes the app to open it, without any hotkey press or manual send.
|
||||
|
||||
---
|
||||
|
||||
## Companion Feature (Separate PR)
|
||||
|
||||
**Notch Live Activity Indicator** — [PR #3166](https://github.com/tinyhumansai/openhuman/pull/3166)
|
||||
A transparent NSPanel pill at the top of the primary screen (the macOS notch area) that shows live voice/agent status. Built alongside this feature; will light up automatically once always-on listening is implemented.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Quick Wins ✅ Complete
|
||||
|
||||
> Low-effort changes that make the existing hotkey-triggered dictation flow work end-to-end without manual sends or approval prompts.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.1 — Auto-send after transcription
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Commit:** `7269f4373`
|
||||
|
||||
**Problem:** After speaking via the dictation hotkey, the transcript appeared in the chat composer but the user had to press Enter manually to send it.
|
||||
|
||||
**Fix:**
|
||||
- `app/src/hooks/useDictationHotkey.ts` — added `autoSend: true` to the `dictation://insert-text` event detail
|
||||
- `app/src/pages/Conversations.tsx` — `onDictationInsert` now checks the flag; when set, calls `handleSendMessage(text)` directly instead of inserting into the textarea. Added `handleSendMessageRef` (updated every render) so the mount-time effect can access the latest send function
|
||||
|
||||
**Result:** Press hotkey → speak → message auto-sends to agent. No Enter key needed.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.2 — Shell allowlist for app-launching — ⚠️ REVERTED / SUPERSEDED
|
||||
|
||||
**Status:** ❌ Reverted — superseded by Change 1.4 (`launch_app`) and the security review.
|
||||
|
||||
**What was tried (commit `7269f4373`):** added `"open"` / `"xdg-open"` to `READ_ONLY_BASES` so `open -a Music` would run without an approval prompt.
|
||||
|
||||
**Why reverted:** base-command classification can't see args, and `open`/`xdg-open`/`start` can open arbitrary `https://` URLs and custom URI handlers — too broad for the `Read` (no-approval) path (maintainer security review). They were **removed** from `READ_ONLY_BASES`; the current code (`policy_command.rs:514-520`) deliberately keeps them out, with a comment. App launching now goes through the dedicated, gated `launch_app` tool (Change 1.4), which is scoped to named applications only.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.3 — Shell tool description fix
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Commit:** `ec8f5be2e`
|
||||
|
||||
**Problem:** Shell tool description said *"Execute a shell command in the workspace directory"* — the LLM was reasoning that it could only run workspace commands, not launch apps.
|
||||
|
||||
**Fix:**
|
||||
- `src/openhuman/tools/impl/system/shell.rs` — updated description to explicitly mention system actions and app launching examples
|
||||
|
||||
**Result:** Agent now understands the shell tool can perform system actions, not just workspace file operations.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.4 — Dedicated `launch_app` tool
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Commit:** `802fbca76`
|
||||
|
||||
**Problem:** Using the `shell` tool for app launching requires loosening `workspace_only` and expanding `allowed_commands` — a security regression. The `shell` tool also couldn't be used because the orchestrator's strict `named` tool list excluded it.
|
||||
|
||||
**Fix (production approach):**
|
||||
- `src/openhuman/tools/impl/system/launch_app.rs` — **new tool** with `PermissionLevel::ReadOnly` (never triggers approval gate)
|
||||
- macOS: `open -a "<app_name>"` via `tokio::process::Command`
|
||||
- Linux: `gtk-launch`, fallback `xdg-open`
|
||||
- Windows: `Start-Process` via PowerShell
|
||||
- Input validation: rejects paths, metacharacters, empty names
|
||||
- Unit tests: name, permission, schema, validation, error cases
|
||||
- `src/openhuman/tools/impl/system/mod.rs` — registered module + pub use
|
||||
- `src/openhuman/tools/ops.rs` — added `LaunchAppTool` to `all_tools_with_runtime`
|
||||
- `src/openhuman/tools/user_filter.rs` — added `"launch_app"` family, `default_enabled = true`
|
||||
- `app/src/utils/toolDefinitions.ts` — added to frontend tool catalog (Settings → Agent Access toggle)
|
||||
|
||||
**Result:** Agent has a purpose-built, always-allow tool for launching apps. No shell exposure, no path security concerns.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.5 — Orchestrator agent tool scope
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Commit:** `7d04fc4bc`
|
||||
|
||||
**Problem:** Even though `launch_app` was registered, it was invisible to the agent. The orchestrator (`src/openhuman/agent_registry/agents/orchestrator/agent.toml`) has a strict `named = [...]` allowlist. `launch_app` was not in it, so it was filtered out. Confirmed via logs: `visible=24, names=[...no launch_app...]`.
|
||||
|
||||
**Fix:**
|
||||
- `src/openhuman/agent_registry/agents/orchestrator/agent.toml` — added `"launch_app"` to the `[tools] named` list, alongside `"current_time"` (same pattern: direct answer without delegation)
|
||||
|
||||
**Confirmed working via logs:**
|
||||
```text
|
||||
visible=25, names=[..., launch_app, ...]
|
||||
[launch_app] ▶ execute called app_name="Music"
|
||||
[launch_app] macOS: running `open -a "Music"`
|
||||
[launch_app] macOS: `open -a` exit=exit status: 0 stderr=
|
||||
[launch_app] ✓ launch succeeded msg="Opened 'Music'."
|
||||
```
|
||||
|
||||
**Result:** Saying "open my Music app" now opens Music directly. No approval prompt, no delegation, no refusal.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.6 — SOUL.md capability hint
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Commit:** `cdd3bb4a4`
|
||||
|
||||
**Problem:** Even with the tool available, the agent was refusing ("I can't open apps on your device") because its training overrides the function-calling schema.
|
||||
|
||||
**Fix:**
|
||||
- `src/openhuman/agent/prompts/SOUL.md` — added explicit *"What you can do on the user's machine"* section listing `launch_app`, `shell`, `file_read`/`file_write` with the instruction: *"Never say 'I can't open apps' when you have a tool to do it. Use the tool."*
|
||||
|
||||
**Result:** Agent now knows it has these capabilities and is instructed to use them.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.7 — Diagnostic logging
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Commit:** `cdd3bb4a4`
|
||||
|
||||
**Added logging to:**
|
||||
- `src/openhuman/tools/impl/system/launch_app.rs` — logs every step: `▶ execute`, validation pass/fail, platform dispatch, `open -a` exit code + stderr, fallback result
|
||||
- `src/openhuman/agent/harness/session/builder.rs` — logs the **full list** of visible tool names at session build time (previously only logged count)
|
||||
|
||||
**Result:** Can now confirm at a glance whether `launch_app` is in the tool list and trace every step of its execution.
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
### Change 1.8 — Computer control (mouse + keyboard) — ⚠️ REVERTED
|
||||
|
||||
**Status:** ❌ Reverted (commits `50ca434b7` add, `bi0rd96sa` revert)
|
||||
|
||||
**Problem:** Agent could open apps but couldn't interact with their UI.
|
||||
|
||||
**What was tried:** Enabled the existing `mouse` + `keyboard` tools (enigo / `CGEventPost`), wired into the orchestrator, user filter, and frontend catalog.
|
||||
|
||||
**Why reverted:** `CGEventPost` injects synthetic events to the **currently focused window**. When the focused window was OpenHuman's own CEF renderer (the chat UI), a Space keypress crashed the app — `EXC_BREAKPOINT / SIGTRAP` in `CFRelease → NSKeyValueWillChangeWithPerThreadPendingNotifications → -[NSApplication stop:]`. CEF can't handle arbitrary key injection. Confirmed via crash report `OpenHuman-2026-06-02-035139.ips`.
|
||||
|
||||
**Replaced by:** Change 1.9 (`ax_interact`) — AXUIElement targets elements directly by label with no synthetic events and no CEF crash risk.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.9 — AXUIElement app UI interaction (`ax_interact`)
|
||||
|
||||
**Status:** ✅ Done
|
||||
**Commits:** `4f9ca1cad` (feature), `2c32b59c9` (exact-match fix), `betuerj11`/test commits
|
||||
|
||||
**Problem:** Need to interact with desktop app UIs reliably, without the CEF crash from synthetic events.
|
||||
|
||||
**Fix — uses the macOS Accessibility API (AXUIElement) instead of CGEventPost:**
|
||||
- `src/openhuman/accessibility/helper.rs` — extended the unified Swift helper with three commands:
|
||||
- `ax_list` → walk the AX tree, return interactive elements (buttons, fields, cells)
|
||||
- `ax_press` → `AXUIElementPerformAction(kAXPressAction)` by label, **exact match preferred over contains** (so "Play" beats "Playlist")
|
||||
- `ax_set_value` → `AXUIElementSetAttributeValue(kAXValueAttribute)` by label
|
||||
- `src/openhuman/accessibility/ax_interact.rs` (new) — Rust wrappers `ax_list_elements`, `ax_press_element`, `ax_set_field_value`
|
||||
- `src/openhuman/tools/impl/computer/ax_interact.rs` (new) — `AxInteractTool` with actions `list` / `press` / `set_value`, `PermissionLevel::ReadOnly`
|
||||
- `src/openhuman/accessibility/ax_interact_tests.rs` (new) — integration tests (open Music → search AC/DC → find row → press)
|
||||
- Wired into `tools/ops.rs`, `tools/user_filter.rs`, `toolDefinitions.ts` (App UI Control), `orchestrator/agent.toml`, `SOUL.md`
|
||||
|
||||
**Why it's better than mouse/keyboard:**
|
||||
|
||||
| | mouse/keyboard (reverted) | ax_interact |
|
||||
|---|---|---|
|
||||
| Mechanism | `CGEventPost` synthetic events | `AXUIElementPerformAction` direct API |
|
||||
| CEF crash risk | Yes | None |
|
||||
| Coordinates | Required (needs screenshot) | None — finds by label |
|
||||
| Works when app unfocused | No | Yes |
|
||||
|
||||
**Verified working:** Direct AX test against Music listed 256 elements including `Bollywood Hits`, `Play`, etc.; pressing `Bollywood Hits` then `Play` both returned `exact=true` and acted correctly.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.10 — Multi-step UI workflow guidance
|
||||
|
||||
**Status:** ✅ Done
|
||||
|
||||
**Problem:** When asked to "play Highway to Hell by AC/DC", the agent ran: launch → list → press Library → press Songs → press "Show Filter Field" → set_value "Highway to Hell" → **press "Play"**. The final press hit the **global playback bar Play button** (plays last queue item), not the specific song row. Result: app navigated correctly but the wrong/no track played.
|
||||
|
||||
**Fix:**
|
||||
- `src/openhuman/agent/prompts/SOUL.md` — added explicit multi-step workflow:
|
||||
1. `list` → discover elements
|
||||
2. `set_value` → type in filter/search
|
||||
3. `list` **again** → see filtered results
|
||||
4. `press` the **specific item** (song row), not the generic Play button
|
||||
- Added Apple Music guidance: use `shell` to open `music://music.apple.com/search?term=...`, then `ax_interact list` to see song rows as AXCells, then press the specific row. More reliable than the Library filter field.
|
||||
|
||||
**Result:** Agent is directed to select the specific item before pressing playback, instead of pressing the global Play button after filtering.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.11 — Apple Music two-step play (navigate then play)
|
||||
|
||||
**Status:** ✅ Done
|
||||
|
||||
**Problem:** When asked to "play Highway to Hell by AC/DC", the agent navigated to the right screen but **nothing played**. Pressing a search-result row in Apple Music only *selects/navigates* — it does not start playback. The agent then pressed the global transport Play button, but nothing was queued.
|
||||
|
||||
**Investigation (empirical AX probing against live Music):**
|
||||
- Every "Highway to Hell" element (AXCell, AXGroup, AXButton) exposes only the `AXPress` action — which selects/navigates, never plays.
|
||||
- Double `AXPress`, a real CGEvent double-click on the Top-Results card, and AX-select + Return key **all left player state `stopped`**.
|
||||
- **Working sequence found:** AXPress the search-result card to **navigate into the song's detail page**, then AXPress the **Play button on that detail page** → `player state: playing` ✅
|
||||
|
||||
**Fix:**
|
||||
- `src/openhuman/agent/prompts/SOUL.md` — replaced the Apple Music guidance with the exact 5-step sequence: URL-scheme search → list → press song row (navigates in) → list detail page → press detail-page Play. Explicitly warns that pressing a search result only navigates, and the second Play press is mandatory.
|
||||
- `src/openhuman/accessibility/ax_interact_tests.rs` — `test_full_flow_search_and_play_acdc` exercises the full navigate-then-play flow and **logs** the `osascript ... get player state` outcome. Playback is **best-effort, not hard-asserted** (Apple Music's UI is nondeterministic — see change 1.13); the test hard-asserts only the tool-level press/list successes.
|
||||
|
||||
**Verified:**
|
||||
```text
|
||||
[step 4] navigate into song: Ok("Pressed 'Highway to Hell' in 'Music'.")
|
||||
[step 5] press detail Play: Ok("Pressed 'Play' in 'Music'.")
|
||||
[step 6] player state: playing
|
||||
test ... ok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Change 1.12 — One-shot `play_music` tool (root-cause fix)
|
||||
|
||||
**Status:** ✅ Done
|
||||
|
||||
**Problem:** Even after change 1.11, the agent still used the broken filter-field approach and didn't play. Transcript analysis (`~/.openhuman/users/<id>/workspace/session_raw/*.jsonl`) revealed two real root causes:
|
||||
|
||||
1. **The orchestrator has no `shell` tool.** Change 1.11 put the play guidance in `SOUL.md` — but the orchestrator runs with `omit_identity = true` and **never sees SOUL.md**. Change 1.11b moved it to the `ax_interact` description, which told the agent to "use the shell tool to open `music://...`" — but the orchestrator can't run shell (it delegates). The agent wrapped the command in a `prompt` arg to a delegation tool; it never executed, and it fell back to the filter approach.
|
||||
2. **Cross-chat memory contamination.** The user message was prefixed with `[Cross-chat context — historical]` containing prior filter-approach "Progress Checkpoint" steps, biasing the agent back to the wrong method.
|
||||
|
||||
**Fix — stop relying on the LLM to orchestrate a fragile multi-step flow with a tool it lacks. Encapsulate the whole proven sequence in native Rust:**
|
||||
- `src/openhuman/accessibility/ax_interact.rs` — `play_apple_music(query)`: open search URL → AX-find + press song cell (navigate) → press detail-page Play → verify `player state == playing`
|
||||
- `src/openhuman/tools/impl/computer/play_music.rs` (new) — `PlayMusicTool`, single call `play_music{query}`, `PermissionLevel::ReadOnly`, runs the blocking flow via `spawn_blocking`
|
||||
- Registered in `ops.rs`, `user_filter.rs`, `orchestrator/agent.toml`, `toolDefinitions.ts`
|
||||
|
||||
**Result:** Agent calls `play_music{query:'Highway to Hell AC/DC'}` **once**; Rust does search→navigate→play. No shell dependency, no multi-step LLM orchestration, no filter-field fallback. Unit tests pass; the underlying flow is exercised by `test_full_flow_search_and_play_acdc` (tool-level success hard-asserted, playback best-effort). **Note:** `play_music` was later removed in change 1.13 in favour of the generic `ax_interact` tool — this entry documents the investigation that led there.
|
||||
|
||||
**Key learning:** The orchestrator (chat agent) only reads **tool descriptions + agent.toml** — NOT SOUL.md (omit_identity=true). Behavior guidance for the chat agent must live in tool descriptions or be encapsulated in the tool itself.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.13 — Generic any-app tool + filtered list (remove play_music)
|
||||
|
||||
**Status:** ✅ Done
|
||||
|
||||
**Problem:** "Play Numb by Linkin Park" still failed, and the agent **hallucinated**. Transcript (`session_raw/*.jsonl`) showed:
|
||||
1. `play_music` hit a 4s timing race — results hadn't rendered, so it returned "No matching song found. Top result cells: [empty]".
|
||||
2. The agent fell back to `ax_interact list`, which dumped **273 elements**. The tool result was **truncated mid-list**, so the model reasoned over a partial view and hallucinated a wrong result ("Numb - Single by Marshmello").
|
||||
|
||||
**Feedback:** A music-specific tool is the wrong abstraction. Build a generic tool that interacts with **any** app.
|
||||
|
||||
**Fix:**
|
||||
- **Removed** `play_music` tool + `play_apple_music` helper and all registrations.
|
||||
- **`ax_interact` is now a robust generic any-app tool:**
|
||||
- `ax_list_elements_filtered(app, filter)` — Rust-side label filter so `list` returns only relevant elements (fixes the truncation→hallucination root cause).
|
||||
- `list` action takes a new `filter` param; output capped at 60 elements with a "narrow your filter" hint; empty-match returns a "UI may still be loading" hint instead of failing hard.
|
||||
- Description rewritten to be app-agnostic and document the general **navigate-then-activate** pattern (pressing a list row/search result selects/opens it; press the action button afterward) — no hardcoded Apple Music steps.
|
||||
|
||||
**Key learning:** Dumping a full AX tree (hundreds of elements) overflows the tool-result budget; the truncated view makes the model hallucinate. Always filter list results to keep them small and accurate.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.14 — `automate(app, goal)`: Rust-driven multi-step automation ✅ M1–M5 done
|
||||
|
||||
**Status:** ✅ **M1 + M2 + M3 + M4 + M5 shipped; M3 proven live on macOS.** The only remaining `automate` work is the **vision fallback** for Electron/partial-AX apps — tracked under **Phase 1.5** below and [`voice-automate-plan.md`](voice-automate-plan.md). (Per-app native fast-paths beyond the shipped Music one are **descoped**.)
|
||||
|
||||
**Agent-in-the-loop fixes (2026-06-03, from two live chat sessions):**
|
||||
- **Mutations were off** — the agent correctly called `automate` but it (and `ax_interact`) refused because `computer_control.ax_interact_mutations=false`. Enabled it; also rewrote both refusal messages to point at **Settings → Agent Access** instead of a config key (the agent had relayed "controls are locked down").
|
||||
- **Query mis-parse** — orchestrator goal `…search for "Highway to Hell" by AC/DC, and play it` made the after-"play" parser extract `"it"`. `extract_play_query` now prefers a **quoted title + `by <artist>`** and rejects bare pronouns. (Unit-tested with the exact failing goal.)
|
||||
- **General loop spun** — pressed "Search" 11× to budget exhaustion. Added a **no-progress guard**: 3 identical actions in a row → abort.
|
||||
- **Search-results timing** — the fast-path's retry burned out before catalog results rendered (`settle` reports count-stable while the network fetch is pending). Added a real, mockable `wait` between attempts (6 × ~800ms).
|
||||
|
||||
**M5 finding — AXEnabled is unreliable:** plumbed an `enabled` field end-to-end (Swift `axEnabled` → `AXElement.enabled` → Windows stub), but Apple Music reports its **pressable** search-result rows as `enabled=Some(false)`. Gating `pick_row` on it broke playback. So `enabled` is kept **informational only** (documented on the struct); matchers never skip on it. The better future actionability signal is AXPress-action support, not AXEnabled.
|
||||
|
||||
**M4 — live progress in the notch (2026-06-03):** the notch indicator (originally PR #3166) was cherry-picked onto this branch (`feat(notch)` + fmt commits → `notch_window.rs` NSPanel + `notch/NotchApp.tsx`, auto-shown on startup, transparent when idle). The `automate` loop and Music fast-path now call `overlay::publish_attention(...)` at each step (`Opening …`, `Searching Music for …`, `Pressing …`, `Typing …`, `Playing …`, plus done/fail), which the existing Socket.IO bridge emits as `overlay:attention` and the notch renders as a pill — so the user sees the automation happening live. Verified: app boots with `[notch-window] panel shown at top-center`; Tauri shell + frontend compile; 31 automate unit tests green.
|
||||
|
||||
**M3 live proof (2026-06-03):** `music_fastpath_live` drives real Apple Music end-to-end and **hard-asserts `player state == playing`** — confirmed: pre-state `paused` → post-state `playing`. Three bugs the live runs surfaced, all fixed + tested:
|
||||
1. **Perceive filter was the whole multi-word query** — a substring filter can't match a full title → now filters by the first strong token and `pick_row` does the token match.
|
||||
2. **Search results render late (§1.13 race)** — retry perceive across up to 4 settles; `AC/DC` now percent-encodes correctly.
|
||||
3. **False success: pressed the toolbar Play, not the song's** — the first run reported success but *nothing played*. AX probing showed the search screen has only the toolbar transport Play (empty queue → silence); pressing the song row navigates to a detail page where a **second** Play appears (23→24 controls). Fix: capture the baseline Play count, **wait for the detail Play to render**, press it, then **verify real playback** via `osascript … player state` and retry (≤3×). Added `verify_playing()` to `AutomateBackend` (macOS osascript; `None` elsewhere = best-effort). `automate` now only reports a play success when audio is actually playing — the false-success class (§1.11) is closed.
|
||||
|
||||
**M3 — shipped (Music fast-path):**
|
||||
- `src/openhuman/accessibility/app_fastpaths/{mod.rs,music.rs}` (new) — deterministic accelerators consulted by `run()` **before** the general loop. Music encodes the §1.11 proven sequence: launch → open `music://…/search?term=…` → settle → press the song row (navigate) → settle → press the detail-page **Play**. Pure helpers `matches` / `extract_play_query` (handles "play X by Y", "launch Music and play …", "play X in Apple Music").
|
||||
- **Structurally different from the removed `play_music` tool (§1.13):** this is *internal* to `automate`, not a tool the LLM selects, and on any failure/`None` the loop **falls through** to the general model-driven path — so it can only help. Added `open_url` to the `AutomateBackend` trait (cross-platform opener; fast-path only).
|
||||
- **Tests:** 9 unit (parser cases, full scripted sequence, no-row fallthrough, dispatch) + 1 `#[ignore]` macOS live test. **Live proof on a Mac:** `cargo test --lib music_fastpath_live -- --ignored --nocapture` (needs Music + Accessibility permission).
|
||||
|
||||
**M2 — shipped (real settle):**
|
||||
- `src/openhuman/accessibility/ax_interact.rs` — new `ax_wait_settled(app, stable_ms, timeout_ms)`: polls the app's interactive-element count and returns once it holds steady for `stable_ms` (or `timeout_ms` elapses). Portable — rides on `ax_list_elements`, which already cfg-dispatches (macOS AX / Windows UIA). Pure decision core `counts_settled(history, n)` extracted and unit-tested (5 non-OS-gated tests).
|
||||
- `automate.rs` — `RealBackend::settle` now calls `ax_wait_settled` (240ms stable / 2s cap) via `spawn_blocking` instead of the M1 blind 450ms wait. This is the piece that removes the timing-race failure class (§1.11/§1.13): the next perceive always sees a settled tree. An AXObserver-driven settle can later sit behind the same signature.
|
||||
|
||||
**M1 — shipped:**
|
||||
- `src/openhuman/accessibility/automate.rs` (new) — the perceive→decide→act→settle loop, generic over an injectable `AutomateBackend` (so the model + AX + launcher are all mockable). Strict JSON action schema (`launch`/`list`/`press`/`set_value`/`done`/`fail`) with a one-shot repair retry on unparseable output (never acts on a hallucinated guess), a step budget (default 12), and a snapshot cap (40 elements) mirroring `ax_interact`'s anti-truncation guard. `RealBackend` calls the existing AX primitives + `launch_platform`, and routes decisions through the **fast tier** (`create_chat_provider("memory", …)` for now; a dedicated `automation_provider` knob is a follow-up). Settle is a short fixed wait in M1 (M2 makes it AXObserver-driven).
|
||||
- `src/openhuman/tools/impl/computer/automate.rs` (new) — `AutomateTool { app, goal }`. Always `Dangerous` + `external_effect` (routes through the ApprovalGate); reuses `ax_interact`'s mutations opt-in (`computer_control.ax_interact_mutations`) and the shared `is_sensitive_app` denylist.
|
||||
- Registered everywhere: `tools/ops.rs`, `tools/user_filter.rs` (`automate` family), `orchestrator/agent.toml` (`named`), `app/src/utils/toolDefinitions.ts` (Settings → "App Automation").
|
||||
- **Tests:** 18 passing — loop happy path, navigate-then-activate, app override, budget exhaustion, repair retry (1 ok / 2 fail), explicit fail, non-fatal press failure, JSON parse (plain/fenced/garbage), snapshot cap/empty-hint; tool gating (missing args, mutations-off, sensitive-app refusal, schema).
|
||||
|
||||
**Problem (the real-time bar):** The user's target is *"whatever I say happens, live, in front of me"* — e.g. *"Launch Music and play Numb by Linkin Park"* or *"open Slack and message Steven 'hi'"*. Today every UI step (`list` → `set_value` → `list` → `press` …) is a **separate chat-LLM turn**. A Slack message is ~7 turns; at 1–3 s each that's 15–25 s, and each turn is a fresh chance to hit a timing race (1.13) or hallucinate. The heavy chat model is sitting *inside* the click loop — the wrong place for it.
|
||||
|
||||
**Root causes (all four documented earlier in Phase 1):**
|
||||
1. **Timing races** — `list`/`press` do a single AX walk with no settle/wait; the UI hasn't rendered yet (1.11/1.13).
|
||||
2. **Navigate-then-activate is re-reasoned every call** — pressing a row selects; you must then press the action control. That logic lives in prose, so it's re-derived (often wrongly) each turn (1.10/1.11).
|
||||
3. **Round-trip explosion** — N full chat turns per task = latency + cost + N chances to fail.
|
||||
4. **Weak element model + no verification** — `list` returns flat `[role, label]`; `press` reports success on `AXAction == .success` even when nothing changed.
|
||||
|
||||
**Design — take the chat model out of the click loop:**
|
||||
- **New tool `automate { app, goal }`** — one call from the orchestrator. Rust then runs a tight **perceive → act → verify** loop internally: read a *filtered* AX snapshot → pick the next action → act → **wait for the UI to settle (AXObserver, not fixed sleeps)** → verify it took effect → repeat until the goal is met or a step budget is hit.
|
||||
- **A fast model drives the inner loop** (Haiku-class) with a *tiny* context: just the goal, the current small AX snapshot, and the last result — not the whole conversation. Each inner step is ~0.5–1 s and self-corrects, instead of one 3 s chat turn that falsely reports success.
|
||||
- **Settle + verify in Rust** between steps — deterministic, kills the timing-race class in one place.
|
||||
- **Native fast-paths for high-value apps** (skip the UI entirely where possible):
|
||||
- **Music** — `music://` search URL → AX play (already explored in 1.11), or AppleScript for library. **(Shipped — M3.)**
|
||||
- ~~**Spotify** — Web API search → `spotify:track:…` URI + AppleScript `play`.~~ **Descoped** — the general loop covers it; not worth the per-app maintenance now.
|
||||
- ~~**Slack** — deep link `slack://channel?…` to open the DM, then AX to type + send.~~ **Descoped** — handled by the general loop / vision fallback.
|
||||
The general AX loop is the fallback for everything else.
|
||||
- **Vision fallback for Electron/Chromium apps** (Slack, Discord, VS Code, Spotify-desktop) whose AX/UIA tree is partial (documented limitation). Slack needs accessibility enabled (`defaults write com.tinyspeck.slackmacgap AccessibilityEnabled -bool true`, relaunch). Where AX returns empty, fall back to **screenshot → vision-locate → guarded click**. This is the reverted CGEventPost path (1.8) — but it crashed only when events hit *OpenHuman's own focused CEF window*; a guarded click into a *different, foregrounded* app does not have that failure mode.
|
||||
- **Stream progress events** to the UI / notch pill (PR #3166) so the user sees each step happen live.
|
||||
|
||||
**Why a generic `automate`, not per-app tools:** Change 1.13 already established that app-specific tools (`play_music`) are the wrong abstraction. The abstraction that *is* generic is the **navigate-then-activate sequence itself** — `automate(app, goal)` encapsulates it once, in Rust, for every app, instead of asking the chat model to re-orchestrate fragile primitives every time.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1.5 — Reliable, real-time multi-step automation ✅ Complete
|
||||
|
||||
> The bridge between today's `ax_interact` primitives and the always-on voice work. **Prerequisite for Phase 3** — fast voice routing into a slow/fragile action loop still feels slow. This is where "whatever I say happens, live" actually gets delivered.
|
||||
>
|
||||
> **Status:** the Rust inner loop (M1), poll-until-stable settle (M2), Music fast-path (M3, proven live), notch progress streaming (M4), the richer element model (M5), the **vision fallback for Electron/partial-AX apps** (Change 1.16), and **Change 1.17** (browser & app fast-paths + keyboard-shortcut driving + artist-aware Music + actionable responses) are all shipped. Per-app fast-paths now cover **Music, browsers, Spotify, Apple Music, and Slack** (Spotify/Slack — previously descoped — are now handled by the `app_shortcuts.rs` keyboard-shortcut path).
|
||||
|
||||
**Detailed implementation plan:** [`voice-automate-plan.md`](voice-automate-plan.md) — decided approach: **Rust inner loop + fast model**, first proof target **Music**.
|
||||
|
||||
**Planned files:**
|
||||
- `src/openhuman/accessibility/automate.rs` (new) — the perceive→act→verify loop + settle/verify primitives, reusing `ax_interact` helpers.
|
||||
- `src/openhuman/accessibility/app_fastpaths/` (new) — per-app deterministic paths behind a generic dispatch. **Shipped:** `music.rs` (M3). Further per-app fast-paths (Spotify/Slack) are **descoped** — the general loop handles them.
|
||||
- `src/openhuman/tools/impl/computer/automate.rs` (new) — `AutomateTool { app, goal }`, gated like `ax_interact` (mutations opt-in, sensitive-app denylist reused).
|
||||
- macOS helper (`accessibility/helper.rs`) — AXObserver-based settle (`ax_wait_settled`) + post-action verify; richer element model (enabled/onscreen/actions).
|
||||
- Vision fallback — screenshot via `accessibility/capture.rs` → locate → guarded click (only when AX tree is empty, target app foregrounded, never OpenHuman's own window).
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] One `automate{app, goal}` call performs a multi-step flow end-to-end (no per-step chat turns)
|
||||
- [x] Settle/verify removes the timing-race + false-success failure classes (1.11/1.13 do not recur)
|
||||
- [x] Music flow ("play <song>") works end-to-end via the inner loop (proven live, hard-asserts `player state == playing`)
|
||||
- [x] Electron/partial-AX apps fall back to vision+guarded-click without the CEF crash (Change 1.16)
|
||||
- [x] Step-by-step progress streamed to the UI / notch indicator (M4)
|
||||
|
||||
---
|
||||
|
||||
### Change 1.15 — Full computer control (mouse/keyboard/screenshot) ✅ Crash fixed (main-thread dispatch)
|
||||
|
||||
**Status:** ✅ Keyboard/mouse now run on the app main thread → no CEF crash. Screenshot downscales for inline view. Live: `[computer] registered main-thread synthetic-input executor` on boot.
|
||||
|
||||
**The fix:** the crash was enigo's `TSMGetInputSourceProperty` running on a tokio worker (`_dispatch_assert_queue_fail`/SIGTRAP). macOS TSM must run on the main thread. New `tools/impl/computer/main_thread.rs` (`MainThreadInputOp` + `run_input_on_main`) dispatches each enigo op over the native registry to a handler the Tauri shell registers at startup, which runs it via `AppHandle::run_on_main_thread`. Keyboard + mouse tools no longer `spawn_blocking` enigo on a worker. Headless/CLI (no executor) returns a clear error instead of crashing. 66 keyboard/mouse tests green.
|
||||
|
||||
**Goal:** make the agent fully autonomous — when the accessibility tree is empty (Electron apps: Slack/Discord/VS Code), fall back to vision + synthetic input. Enabled `computer_control.enabled`, added `mouse`/`keyboard`/`screenshot` to the orchestrator `named` list + `autonomy.auto_approve`, and taught `prompt.md` a keyboard-first ladder (foreground via `launch_app` → `keyboard type` + Enter; Slack `Cmd+K` recipe).
|
||||
|
||||
**Foreground-first:** `automate::run` now `open -a`s the target app at the very start, always, so AX/input hit the right window.
|
||||
|
||||
**Screenshot fix:** oversized Retina captures were returned as "too large to base64-encode inline" (the model was blind). Now downscaled to a viewable JPEG (`downscale_to_jpeg`) with reported dimensions.
|
||||
|
||||
**Root cause (now fixed) — `OpenHuman-2026-06-03-170058.ips`:** `EXC_BREAKPOINT/SIGTRAP` on a **`tokio-rt-worker`** thread:
|
||||
```text
|
||||
enigo::macos::get_layoutdependent_keycode → TSMGetInputSourceProperty
|
||||
→ dispatch_assert_queue → _dispatch_assert_queue_fail → SIGTRAP
|
||||
```
|
||||
enigo's keyboard-layout lookup (`TSMGetInputSourceProperty`) **must run on the app's main thread**; the keyboard tool ran on a tokio worker → macOS trapped. **Not** a focus issue (same §1.8 root cause); a frontmost-app guard would not have fixed it.
|
||||
|
||||
**Fix applied:** all enigo operations now run on the Tauri **main thread** via `run_input_on_main` (new `main_thread.rs` module + a native-registry handler the shell registers, dispatched through `AppHandle::run_on_main_thread`) and wrapped in `catch_unwind`. Keyboard/mouse tools **and** `vision_click` now execute without TSM traps.
|
||||
|
||||
**Tests:** voice-actions + autonomy suite is exhaustive — 220 feature unit tests + a JSON-RPC E2E (`json_rpc_voice_server_settings_roundtrip_always_on_and_wake_word`). The E2E caught + fixed real gaps (`wake_word` missing from the get output and the update RPC path). Screenshot downscale unit-tested.
|
||||
|
||||
---
|
||||
|
||||
### Change 1.16 — `vision_click`: vision fallback for Electron/partial-AX apps ✅ Done
|
||||
|
||||
**Status:** ✅ Shipped — closes the last open Phase 1.5 item. Electron/Chromium apps (Slack, Discord, VS Code) expose little or no AX/UIA tree, so the `automate` perceive→press loop had nothing to act on. The loop can now *see* the screen and click a described element.
|
||||
|
||||
**Problem:** when `perceive` returns an empty (or target-missing) element list, the loop was stuck — there was no label to press. The planned answer (tracker §1.5) was *screenshot → vision-locate → guarded click*, but two things blocked it: (1) the fast inner-loop model is text-only, and (2) the deferred **F2** coordinate-mapping gap — the screenshot is windowed + downscaled and Retina is 2×, while `mouse` expects absolute screen points, so a vision-returned coordinate would click the wrong spot.
|
||||
|
||||
**Fix — a new model-chosen `vision_click { description }` action in the `automate` loop:**
|
||||
- **Reuses the existing multimodal path** — the OpenAI-compatible provider promotes an embedded `[IMAGE:<data-uri>]` marker into a real `image_url` part (`inference/provider/compatible_types.rs`, #3205), so vision-locate rides the existing `chat_with_system` call with **no new inference API**. The locate call uses the **main `chat`/vision provider** (`create_chat_provider("chat", …)`) — reliable UI grounding, and the fallback only fires when AX is empty (rare).
|
||||
- **Folds in the F2 coordinate transform** — `src/openhuman/accessibility/vision_click.rs` (new): `CaptureGeometry` (window screen-rect in points + image pixel size) + a **pure, unit-tested** `image_to_screen(geom, px, py)`. The px→pt ratio absorbs both the downscale and the Retina backing scale, so no explicit scale factor is needed; the result is clamped strictly inside the target window.
|
||||
- **§1.8 safety guard** — a `vision_click` only fires when the target app is **frontmost** (`focus::foreground_context`); on positive evidence a different app is focused it re-foregrounds once and otherwise **refuses**, so synthetic input never lands on OpenHuman's own CEF window. `None` (can't tell) = best-effort, since the loop already foregrounded the app at start.
|
||||
- **Main-thread click** — the guarded left-click runs via `run_input_on_main` (Change 1.15), so off-thread enigo never traps TSM.
|
||||
- Wired into the `automate` loop: new `Action.description`, a `vision_click` system-prompt verb ("use when the element list is EMPTY — Electron apps"), the no-progress signature extended with `description`, and `RealBackend::{screenshot,locate,frontmost_app,click}`. Inherits `automate`'s existing gating (Dangerous + mutations opt-in + sensitive-app denylist) — no new tool, no new approval surface.
|
||||
|
||||
**Tests:** pure `image_to_screen` (downscale / Retina 2× / origin offset / out-of-range + negative clamp / zero-dim safety), `parse_locate_response` (found / not-found / fenced / prose / garbage), `build_locate_user` (marker), `image_dims_from_data_uri` (real PNG round-trip); loop integration via the scripted backend (locate-and-click when frontmost, proceed when frontmost unknown, **refuse when another app is frontmost**, no-click on not-found, empty-description skip). A macOS live run against a real Electron app is the remaining manual validation (vision is nondeterministic — assert tool-level success, treat the visual outcome as best-effort, same caveat as Music).
|
||||
|
||||
---
|
||||
|
||||
### Change 1.17 — browser & app fast-paths, keyboard-shortcut driving, artist-aware Music, actionable responses ✅ Done
|
||||
|
||||
**Status:** ✅ Shipped (PR [#3558](https://github.com/tinyhumansai/openhuman/pull/3558)). One body of work driven by **live-transcript analysis of the `automate` agent** across browsers, Apple Music, Spotify, and Slack. The transcripts surfaced four recurring failures the agent kept hitting:
|
||||
|
||||
1. **Browsers** — for *"open my Brave browser, go to youtube.com and play a music video"* the `desktop_control_agent` was re-delegated **6×** (each sub re-launched Brave + re-navigated), ran `ax_interact` on the **wrong app names** (`Google Chrome`/`Safari`, never the real `Brave Browser`, because Chromium exposes no AX tree), then "played" via a **blind hardcoded mouse click at (400,350)** with no verification.
|
||||
2. **Wrong track** — *"play Numb by Linkin Park"* played a **same-titled song by the wrong artist** ("Numb" by Marshmello & Khalid) and the agent **claimed success**, because the AX row label is title-only and verification only checked *that* something played.
|
||||
3. **Inert responses** — the `automate` tool's replies were *honest but gave no next move*, so the agent **re-ran the same failing search 6×** instead of changing tactics.
|
||||
4. **No keyboard-shortcut lever** — the loop hunted AX labels even where an app's own shortcut would be instant and reliable.
|
||||
|
||||
The six parts below address all four. All fast-paths are consulted by `automate::run` **before** the model loop (`try_fastpath` order: `music` → `browser` → `app_shortcuts`); on any failure they fall through, so they can only *help*.
|
||||
|
||||
**(a) Browser fast-path** — `app_fastpaths/browser.rs` (new). Pure parsers + `run(app, goal, backend)`:
|
||||
- `resolve_browser(app, goal)` — aliases → macOS display names (`brave`→`Brave Browser`, `chrome`→`Google Chrome`, `edge`→`Microsoft Edge`, `safari`/`firefox`/`arc`). Checks the `app` arg first, then the goal; a generic "browser" with no named product → **`None`** so it never guesses the wrong app (the original bug).
|
||||
- `extract_destination(goal)` — YouTube → `youtube.com/results?search_query=…` (percent-encoded); Google/"search for X" → `google.com/search?q=…`; a bare domain/URL → normalized to `https://…`.
|
||||
- **Navigation is one deterministic step:** `backend.open_url_in_app(browser, url)` → `open -a "<browser>" "<url>"` launches/foregrounds + navigates — no address-bar typing, no AX. Pure-navigation goals **complete here**; **play goals** navigate to the results URL then return non-success so the loop does the single "click first result" via `vision_click` (no native shortcut *selects* a result — we don't fake it). In-page media control ("pause/next the video") sends the YouTube shortcut directly.
|
||||
|
||||
**(b) Cross-platform browser shortcut table** — `app_fastpaths/browser_shortcuts.rs` (new). `shortcut(intent, browser, os)` resolves browser-level intents (address bar, new/close/reopen tab, tab N, next/prev tab, find, reload/hard-reload, back/forward, history, downloads, zoom, private window, fullscreen) per **Chrome/Firefox/Safari/Edge × macOS/Windows/Linux**. Nearly uniform (only ⌘↔Ctrl differs); encodes the real exceptions (Firefox-Linux `Alt+1‑8`, Firefox private window `⌘/Ctrl+Shift+P`, per-browser History/Downloads, Mac `⌘[`/`]` back-forward, `F11` vs `⌃⌘F` fullscreen). Sourced from the official Chrome/Firefox/Edge/Safari docs. `browser.rs` parses commands ("open a new tab", "go back", "reload", "switch to tab 3") and dispatches the resolved chord. (Brave/Arc/Chromium → Chrome family.)
|
||||
|
||||
**(c) App-shortcut fast-path** — `app_fastpaths/app_shortcuts.rs` (new). Same model for **Spotify, Apple Music, Slack**: `shortcut(intent, app, os)` + goal parser. Drives transport/navigation by each app's own global shortcut (faster + more reliable than AX). Media intents (play/pause, next, prev, volume ±, mute, shuffle, repeat, search) for Spotify + Apple Music; Slack nav (quick-switcher, search, new message/compose DM, next/prev unread, next/prev channel, threads, all-unread, mark read). Encodes the quirks: **Spotify uses ↓/↑ for next/prev** (not ←/→); **Apple Music prefixes everything with Ctrl on Windows** (Ctrl+Space vs bare Space on Mac); Slack channel-nav uses **Option/Alt+arrows**. Intents with no simple chord (Spotify volume, Apple Music mute / Windows search access-key) return `None` → fall through. Complementary to `music.rs`, which still owns Apple Music *song search/play*.
|
||||
|
||||
**(d) Keyboard-shortcut plumbing** —
|
||||
- Extracted shared `pub(crate)` helpers `run_hotkey`/`run_key`/`run_type_text` from `tools/impl/computer/keyboard.rs` (one place for validation + main-thread enigo dispatch, Change 1.15) — no behavior change to the `keyboard` tool.
|
||||
- New `AutomateBackend` methods (non-breaking defaults): `open_url_in_app`, `key`, `type_text`, `now_playing`. `RealBackend` impls `open -a` per-app, delegates keystrokes to the keyboard helpers, and reads Music state via osascript.
|
||||
- New **`hotkey` verb** in the general loop + system prompt (`{"action":"hotkey","keys":["Cmd","L"]}`) so the model can use any app's shortcuts — folded into the no-progress signature.
|
||||
|
||||
**(e) Artist-aware Music verification** — `music.rs` + `now_playing`. Apple Music's AX row label is title-only ("Numb - Single"), so a title match lands on the wrong artist. `music.rs` now parses the requested artist (`extract_artist`, "…by Linkin Park") and after Play reads the **now-playing name + artist** via AppleScript (`now_playing` → `osascript … get {name, artist} of current track`). On a match the summary names the verified track + artist; on a **mismatch it says so honestly** ("Now playing 'Numb' by 'Tom Odell'…") instead of claiming the requested track. Closes the wrong-artist false-success. **Follow-up:** *correcting* to the right artist is limited by AX `press`-by-label hitting only the first same-titled row — a library AppleScript `play (track whose name … and artist …)` fallback is the next step.
|
||||
|
||||
**(f) Actionable `automate` responses** — the tool's failures are now agent-actionable, not inert. Four fixes: **(1)** terminal failures (`music.rs` no-match/mismatch, `automate.rs` budget-exhausted/no-progress) carry **next-step guidance + a "don't repeat this" steer**; **(2)** they **list the actual candidates / on-screen labels** (`candidate_labels`, `screen_hint`) instead of a bare negative; **(3)** **verified-only success** — never "Playing 'X'" unless `now_playing` confirms it, and an artist/album press is reported as "navigated, no track started"; **(4)** `pick_row` now **prefers real song rows** (`AXCell`/`AXRow`) over artist/album buttons (the live bug pressed the "LINKIN PARK" artist row) and the step log names the element *type* pressed.
|
||||
|
||||
**Also:** the always-on toggle in `VoicePanel` is now shown in dev/debug builds (`IS_DEV_LIKE`) and hidden in production.
|
||||
|
||||
**Follow-up fix — Full OS access now enables app control.** `ax_interact` (press/set_value) and `automate` gated **only** on `computer_control.ax_interact_mutations`, which is independent of the autonomy level. A user who granted **Full** access in Settings → Agent Access still got *"App control isn't enabled yet…"* because that flag was untouched (confirmed live: `SecurityPolicy autonomy=Full` while the tool refused). Fixed with a shared `app_control_enabled(explicit_opt_in)` gate (`tools/impl/computer/ax_interact.rs`) that returns true when the explicit flag is set **or** the **live** policy (`security::live_policy::current()`) is `AutonomyLevel::Full` — so granting Full access takes effect immediately, no session restart or separate flag flip. Both refusal messages now point at Settings → Agent Access ("Grant Full access (or turn on App UI Control / App Automation)"). Covered by `app_control_enabled_combines_optin_and_full_access` + the two pinned-policy refusal tests.
|
||||
|
||||
**Follow-up fix — Windows browser nav + the re-delegation loop.** Live Windows transcript of *"open my browser, go to youtube.com and play a video"*: the orchestrator delegated to `desktop_control_agent` **7×**, every sub re-`launch_app`-ed Chrome (→ ~10 windows), and the AX path dead-ended — Chromium exposes no page content to UIA, `screenshot`/`vision_click` is unimplemented on Windows, and `ax_interact set_value` on the address bar **can't be submitted** (UIA Invoke on an Edit fails, no Enter). Two fixes:
|
||||
> - **`open_url_in_app` now works on Windows** (`accessibility/automate.rs`) — was macOS-only (fell back to the *default* browser). Maps the resolved browser display name → the shell `start` App-Paths token (`windows_browser_launch_token`: Chrome→`chrome`, Edge→`msedge`, Brave→`brave`, Firefox→`firefox`; Safari/Arc→`None`→default handler) and runs `cmd /C start "" <token> "<url>"`. Launches/foregrounds the named browser and navigates in **one** step; when the browser is already open this lands in a **new tab of the existing window**, so the fast-path no longer piles up windows. The browser fast-path's deterministic nav is now cross-platform (mac `open -a`, win `start`). Tested by `browser_display_names_map_to_start_tokens` (`#[cfg(target_os="windows")]`).
|
||||
> - **`desktop_control_agent` prompt steered to `automate` for browsers** (`agent_registry/agents/desktop_control_agent/prompt.md`) — it had `automate` but its only examples were Music/Slack, so it AX-fumbled the browser. Now: web browsers → use `automate{app, goal}` (deterministic URL nav), never type a URL into the address bar via `ax_interact`; **foreground each app at most once per task** (repeated `launch_app` piles up windows); and **stop after two failed attempts at a step** instead of re-launching and looping. (Still a Windows gap: `vision_click` for the final "click first result" — `screenshot` is unimplemented on Windows, so play-from-search-results can't complete deterministically yet; navigation/search now do.)
|
||||
|
||||
**Net for the browser example:** `open -a "Brave Browser" "https://www.youtube.com/results?search_query=music%20video"` + one `vision_click` — instead of 6 delegations and 5 failed AX calls.
|
||||
|
||||
**Tests (all green):** `app_fastpaths` **58**, `accessibility::automate` **21**, `computer::keyboard` **26**. Coverage: browser/app resolution + destination/intent parsers (incl. Spotify ↓/↑, Apple Music Win-Ctrl, Slack Alt-nav, generic-browser→None, percent-encoding); scripted-backend sequences (model-free nav, play→fall-through, media-control hotkey, dispatch order with music-wins-play); artist match→names-track / mismatch→honest; response shapes (candidate listing, wrong-artist steer, non-song honesty, song-row preference, budget/no-progress on-screen hints); a `hotkey`-verb loop test; `#[ignore]` macOS live tests. **Follow-ups:** wire the shortcut tables into the Phase 3 voice `command_router` (voice "pause"/"next" → frontmost-app shortcut); AppleScript library play-by-name+artist correction; expose `vision_click` as a standalone tool.
|
||||
|
||||
---
|
||||
|
||||
## Windows port — app interaction 🪟 ✅ Implemented
|
||||
|
||||
Phase 1's app-interaction layer is now ported to Windows. The macOS path uses the
|
||||
Accessibility API via a Swift helper; the Windows path uses **Microsoft UI
|
||||
Automation (UIA)** called directly from Rust (no helper process). The
|
||||
agent-facing tool is a single `ax_interact` tool on both platforms — only the
|
||||
backend differs, via cfg-dispatch. The sections below were the design plan; see
|
||||
**"Windows port — implementation status"** at the end of this part for what
|
||||
shipped and the test evidence.
|
||||
|
||||
### What already works cross-platform
|
||||
|
||||
| Capability | macOS (done) | Windows status |
|
||||
|---|---|---|
|
||||
| Auto-send dictation transcript | TS (`useDictationHotkey`/`Conversations`) | ✅ Already cross-platform (frontend) |
|
||||
| App launching | `launch_app` / `policy_command.rs` | ✅ Launchers (`open`/`xdg-open`/`start`) stay OUT of `READ_ONLY_BASES` (can open arbitrary URLs/handlers); app launching goes through the gated `launch_app` tool. |
|
||||
| `launch_app` | `open -a` | ⚠️ Already has a Windows branch (`Start-Process`) — verify it resolves app display names |
|
||||
| `ax_interact` (list/press/set_value) | AXUIElement Swift helper | ❌ Needs a UI Automation (UIA) backend |
|
||||
|
||||
### 1. Launching apps on Windows (`launch_app`)
|
||||
|
||||
`launch_app.rs` already has a `#[cfg(target_os = "windows")]` branch using PowerShell `Start-Process "<app_name>"`. Caveats to verify on the Windows machine:
|
||||
- `Start-Process "Spotify"` works for apps on PATH or registered App Paths, but **Store/UWP apps** (e.g. the Windows "Media Player", "Spotify" from the Store) need their AUMID: `Start-Process "shell:AppsFolder\<AUMID>"`. Consider enumerating Store apps via `Get-StartApps` (returns Name + AppID) and matching by display name.
|
||||
- For URIs (e.g. `spotify:`, `mailto:`), `Start-Process "<uri>"` works the same as macOS `open`.
|
||||
|
||||
### 2. App UI interaction (`ax_interact` → UI Automation)
|
||||
|
||||
**The Windows analog of macOS AXUIElement is Microsoft UI Automation (UIA)** — the OS-level accessibility tree. It exposes the same concepts:
|
||||
|
||||
| macOS AX concept | Windows UIA equivalent |
|
||||
|---|---|
|
||||
| `AXUIElement` | `IUIAutomationElement` |
|
||||
| `kAXRoleAttribute` (AXButton, AXCell…) | `ControlType` (Button, ListItem, Edit, Text…) |
|
||||
| `kAXTitleAttribute` / `kAXDescriptionAttribute` | `Name` property (+ `AutomationId`, `HelpText`) |
|
||||
| `AXUIElementPerformAction(kAXPressAction)` | `InvokePattern.Invoke()` (buttons) / `SelectionItemPattern.Select()` (list rows) |
|
||||
| `AXUIElementSetAttributeValue(kAXValueAttribute)` | `ValuePattern.SetValue()` (text fields) |
|
||||
| `AXUIElementCopyAttributeValue(kAXChildrenAttribute)` | `TreeWalker` / `FindAll(TreeScope.Descendants, …)` |
|
||||
| Walk tree from app PID | `IUIAutomation.ElementFromHandle(hwnd)` or root + `ProcessId` condition |
|
||||
|
||||
**Recommended implementation path (Rust-native, no helper process needed):**
|
||||
- Use the [`uiautomation`](https://crates.io/crates/uiautomation) crate (safe Rust bindings over the UIA COM API). This is cleaner than macOS, where we had to shell out to a Swift helper — on Windows the COM API is callable directly from Rust.
|
||||
- Mirror the existing `accessibility::ax_interact` surface so the **tool stays identical**:
|
||||
- `list(app, filter)` → `CreateTreeWalker` over the app's window, collect elements whose `Name` matches `filter`, return `[{control_type, name}]`.
|
||||
- `press(app, label)` → find element by `Name` (exact-first), then call `InvokePattern` if supported, else `SelectionItemPattern.Select()`, else `LegacyIAccessiblePattern.DoDefaultAction()`.
|
||||
- `set_value(app, label, value)` → find `Edit`/`ComboBox`, call `ValuePattern.SetValue()`.
|
||||
- **Key win over macOS:** UIA Invoke is generally a real "activate" (it triggers the control's default action), so the navigate-then-activate two-step that plagued Apple Music is less likely. A list-item Invoke on most Windows media apps plays directly. Still expect per-app quirks.
|
||||
|
||||
**Suggested module layout (parallel to macOS):**
|
||||
```text
|
||||
src/openhuman/accessibility/
|
||||
ax_interact.rs # macOS (existing)
|
||||
uia_interact.rs # NEW — Windows UIA backend, same fn signatures
|
||||
mod.rs # cfg-dispatch: pub use the right backend per-OS
|
||||
```
|
||||
Then `tools/impl/computer/ax_interact.rs` calls a thin cfg-gated facade so the **agent-facing tool is one tool on both platforms** (same name `ax_interact`, same actions). Update its description to be OS-neutral ("uses the platform accessibility API").
|
||||
|
||||
### 3. Permissions
|
||||
|
||||
- macOS needs the Accessibility permission. **Windows UIA needs no special permission** for same-session, same-integrity-level apps — a big simplification. Caveat: a non-elevated process can't drive an **elevated** app's UI (UIPI). If the agent must control an elevated app, OpenHuman would need to run elevated too (avoid unless necessary).
|
||||
|
||||
### 4. Diagnostics
|
||||
|
||||
Keep the same `[ax_interact]`/`[uia_interact]` log prefixes and the verbose step logging (`▶ action`, found-count, press result) — they were essential for diagnosing the macOS flow and will be just as useful on Windows.
|
||||
|
||||
### 5. Testing
|
||||
|
||||
Port the integration tests using a built-in Windows app that's always present and UIA-friendly:
|
||||
- **Calculator** (`calc.exe`) — press digit/operator buttons by Name, read the result `Text` element. Deterministic, no network, ideal smoke test.
|
||||
- **Notepad** — `set_value` into the `Edit` control, verify via `ValuePattern.Value`.
|
||||
- Media: **Windows Media Player** or the Store **Media Player** for a play test, but expect the same nondeterminism caveat as Apple Music — assert tool-level success, log playback as best-effort.
|
||||
|
||||
### 6. Known-different behaviors to watch for
|
||||
|
||||
- **Element naming:** Windows apps often populate `AutomationId` (stable) where macOS only had a visible title. Prefer matching `Name`, fall back to `AutomationId`.
|
||||
- **Chromium/Electron apps** (Slack, Discord, VS Code, Spotify desktop): on Windows these expose a partial UIA tree by default; some require the app to have accessibility enabled. Same class of limitation as the macOS `chromiumAppPatterns` special-casing already in `helper.rs`.
|
||||
- **Focus/foreground:** UIA generally doesn't require foregrounding to read/invoke, like macOS AX. No CGEventPost-style CEF crash risk because UIA Invoke is not synthetic input injection.
|
||||
|
||||
### Quick start for the Windows machine
|
||||
|
||||
1. `launch_app` should already work — test `"open notepad"` / `"open calculator"` first.
|
||||
2. Do NOT add launchers to `READ_ONLY_BASES` — `launch_app` (gated, URI-rejecting) is the Windows app-launch path. `Start-Process` lives inside that tool, not the shell allowlist.
|
||||
3. Build `uia_interact.rs` against the `uiautomation` crate, mirroring the three `ax_interact` fns.
|
||||
4. cfg-dispatch in `accessibility/mod.rs` so `ax_interact` the tool resolves to UIA on Windows.
|
||||
5. Smoke-test with Calculator (deterministic), then a media app (best-effort).
|
||||
|
||||
### Cross-platform compatibility audit (current state)
|
||||
|
||||
Every Phase 1 change was written to **compile on all platforms** — nothing here breaks the Windows build. macOS-specific native code is `#[cfg(target_os = "macos")]`-gated and the non-macOS branches return a clean `"…macOS-only"` error at runtime rather than failing to build.
|
||||
|
||||
| Change | Cross-platform status | Notes for Windows |
|
||||
|---|---|---|
|
||||
| Auto-send dictation transcript (TS) | ✅ Fully portable | Pure frontend; no OS code. Works as-is. |
|
||||
| `launch_app` | ✅ macOS / Linux / Windows branches | Windows branch now uses `Start-Process` with a **Store/UWP (`Get-StartApps` AUMID) fallback** and injection-safe env passing (§1). |
|
||||
| `ax_interact` **tool** (`tools/impl/computer/ax_interact.rs`) | ✅ Functional on Windows | Delegates to `accessibility::ax_interact` fns, which now cfg-dispatch to the UIA backend on Windows. Description made OS-neutral. |
|
||||
| `accessibility::ax_interact` helpers | ✅ cfg-dispatched | macOS → Swift helper; Windows → `uia_interact.rs`; other → clean runtime error. |
|
||||
| `accessibility::uia_interact` (NEW) | ✅ Windows backend | UIA `list`/`press`/`set_value` via the `uiautomation` crate; same fn signatures as the macOS path. |
|
||||
| Swift unified helper (`accessibility/helper.rs`) | ⚠️ macOS-only by design | Windows needs no helper process — UIA COM API is called directly from Rust. |
|
||||
| App launching | ✅ Done | Launchers stay out of `READ_ONLY_BASES`; `launch_app` (gated) handles Windows `Start-Process`. |
|
||||
| Notch indicator (separate PR #3166) | ⚠️ macOS NSPanel | A Windows equivalent would be a borderless always-on-top WebView2 window or a tray flyout — out of scope for this branch. |
|
||||
|
||||
**Before merging the Windows port, confirm the whole branch still builds and runs on macOS too** (`cargo check` on both `Cargo.toml` and `app/src-tauri/Cargo.toml`) so the cfg-dispatch doesn't regress the working macOS path.
|
||||
|
||||
### ⚠️ Mandatory: extensive E2E testing on Windows
|
||||
|
||||
The macOS path was hardened only through repeated real-app runs (each bug — CEF crash, select-vs-play, list truncation/hallucination — surfaced only by actually driving live apps, not by unit tests). **Do the same on Windows before considering it done.** Treat the following as the required E2E matrix:
|
||||
|
||||
1. **App launch** — `launch_app` for: a Win32 app (Notepad), a Store/UWP app (Media Player / Spotify from Store), and a URI (`spotify:`). Confirm each actually opens.
|
||||
2. **Deterministic UI control** — Calculator: `list filter='5'` → `press '5'`, `press '+'`, `press '='`, then read the result element. Assert the computed value. This is the Windows analog of the AC/DC test and should be a **hard-asserted** automated test (Calculator is deterministic).
|
||||
3. **Text entry** — Notepad: `set_value` into the Edit control, verify via `ValuePattern.Value`.
|
||||
4. **Filtered list correctness** — confirm `list` with a `filter` returns a small, accurate set (the truncation→hallucination bug must not recur; verify the 60-element cap + filter behaves on a busy app like Settings or a browser).
|
||||
5. **Real-world app** — drive a media app end-to-end (open → search → play). Assert tool-level success; treat playback state as **best-effort** (same nondeterminism caveat as Apple Music).
|
||||
6. **Chromium/Electron app** — Slack/Discord/VS Code: confirm whether their UIA tree is exposed; document any app that needs accessibility explicitly enabled.
|
||||
7. **Permissions/elevation** — verify behavior against a normal app vs an elevated one (UIPI boundary); document what fails and why.
|
||||
8. **Agent-in-the-loop run** — the real test: ask the running agent (chat) to perform each action and confirm it picks `launch_app` / `ax_interact` and the action lands. Watch `[ax_interact]`/`[launch_app]` logs exactly as we did on macOS.
|
||||
9. **Regression** — re-run the macOS E2E suite after the Windows changes land to prove cfg-dispatch didn't break the Mac path.
|
||||
|
||||
Add the deterministic ones (Calculator, Notepad, launch) as `#[cfg(target_os = "windows")]` `#[ignore]` integration tests mirroring `ax_interact_tests.rs`, runnable with `cargo test ... -- --include-ignored` on the Windows machine.
|
||||
|
||||
### Windows port — implementation status ✅
|
||||
|
||||
Shipped on the Windows machine (2026-06-02):
|
||||
|
||||
**Code**
|
||||
- `Cargo.toml` — `uiautomation = "0.25"` under `[target.'cfg(windows)'.dependencies]`; `Win32_System_Com` feature added to `windows-sys` for COM init.
|
||||
- `src/openhuman/accessibility/uia_interact.rs` (**new**) — UIA backend. `list` / `press` / `set_value` over the UIA COM tree via the `uiautomation` crate, mirroring the macOS `ax_interact` fn signatures. `press` activates via UIA patterns in order — `Invoke` → `SelectionItem.Select` → `LegacyIAccessible.DoDefaultAction` — never injecting synthetic input. `set_value` finds an editable field, preferring `Edit`, then `ComboBox`, then `Document` (so the Win11 RichEdit Notepad, whose editor is a `Document`, works). Exact-label match preferred over substring. Per-thread COM init via `CoInitializeEx(MTA)`.
|
||||
- `src/openhuman/accessibility/ax_interact.rs` — the three public helpers now cfg-dispatch: macOS → Swift helper, Windows → `uia_interact`, else → clean runtime error. Module + tool docs made OS-neutral.
|
||||
- `src/openhuman/accessibility/mod.rs` — declares `uia_interact` (cfg-gated to Windows).
|
||||
- `src/openhuman/tools/impl/computer/ax_interact.rs` — description rewritten to be platform-neutral ("platform accessibility API (macOS AXUIElement / Windows UI Automation)").
|
||||
- `src/openhuman/tools/impl/system/launch_app.rs` — Windows launcher hardened: app name passed via env var (no string interpolation → no injection), `Start-Process` first, then Store/UWP fallback by display name via `Get-StartApps` → AUMID (`shell:AppsFolder\<AppID>`); stderr surfaced on failure.
|
||||
- `src/openhuman/security/policy_command.rs` — launchers (`open`/`xdg-open`/`start`) deliberately kept OUT of `READ_ONLY_BASES`; `launch_app` is the gated launch path.
|
||||
- `src/openhuman/accessibility/uia_interact_tests.rs` (**new**) — `#[cfg(all(test, target_os = "windows"))]` integration tests, wired into `ax_interact.rs`.
|
||||
|
||||
**Test evidence (real apps on Windows 11)**
|
||||
- `test_uia_calculator_five_plus_five` ✅ — drove the live Calculator entirely by element label: `list` → 41 interactive elements; pressed `Five`/`Plus`/`Five`/`Equals`; **hard-asserted** the readout `[Text] "Display is 10"` (5 + 5 = 10). Deterministic — the Windows analogue of the macOS AC/DC test.
|
||||
- `test_uia_notepad_set_value` ✅ — `set_value` wrote into the live Win11 Notepad's `Document` "Text editor" (`Ok("Set 'Text editor' in 'Notepad' to the provided value.")`). The `Document` fallback is what makes the redesigned Notepad work.
|
||||
- `test_uia_list_nonexistent_app` ✅ (non-ignored) — exercises COM init + window walk + error path deterministically.
|
||||
- `launch_app` (×8) and `ax_interact` tool (×4) unit tests ✅.
|
||||
- Full `cargo test --lib --no-run` compiles clean on Windows (warnings only, all pre-existing).
|
||||
|
||||
**Environment gotcha (this machine):** Norton real-time protection blocks `link.exe` from writing the freshly-linked ~150 MB test `.exe` (LNK1104, "Access denied" creating the file). Fix: exclude the repo's `target` dir under Norton's **Auto-Protect / SONAR / Download Intelligence** exclusion list (not the separate "Scans" list), and restore the file from Quarantine if already flagged.
|
||||
|
||||
**Follow-ups / not done here**
|
||||
- **macOS regression check** — the cfg-dispatch is additive (the `#[cfg(target_os="macos")]` arms are untouched; only the non-macOS catch-all message changed), but per the branch note, re-run `cargo check` + the macOS E2E suite on a Mac before merge to prove the Mac path didn't regress (can't be done from the Windows machine).
|
||||
- **Agent-in-the-loop run** (E2E item 8) — the full Tauri desktop app was built and run on Windows (`pnpm dev:app:win`) and the in-process core booted fine (verbose `[launch_app]`/`[ax_interact]`/`[uia_interact]` logging wired via `RUST_LOG`). The first chat attempt couldn't complete because the configured **local AI model was still downloading** (`kind="empty_provider_response"` — the agent returned an empty response, so it never reached a tool call). **Still pending:** a working model (finish the local download or select a configured cloud model), then ask the agent "open Calculator" / "press 5 in Calculator" and confirm it picks `launch_app`/`ax_interact` and the action lands.
|
||||
- Chromium/Electron UIA coverage, elevation/UIPI behavior, and a busy-app filtered-list check (E2E items 4/6/7) remain to be spot-checked manually.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Always-On Listening ✅ Implemented
|
||||
|
||||
> Continuous microphone listening without requiring a hotkey press.
|
||||
|
||||
**Shipped:**
|
||||
- `src/openhuman/voice/always_on.rs` — pure `VadSegmenter` (onset / silence-hangover / min-speech / max-utterance, 7 unit tests) **plus** the continuous capture loop: a dedicated cpal thread streams 16 kHz mono frames → segmenter → each utterance is encoded (`encode_wav_16k`) → `voice_transcribe_bytes` → `publish_transcription` (so it reaches the agent's auto-send and the notch, exactly like hotkey dictation). Started at boot in `credentials::ops`.
|
||||
- `src/openhuman/config/schema/voice_server.rs` — `always_on_enabled` flag + VAD tuning (`vad_onset_threshold`, `vad_hangover_ms`, `vad_min_speech_ms`, `vad_max_utterance_secs`), opt-in/off by default.
|
||||
- **Settings toggle** — "Always-on listening" in the Voice debug panel, wired through `get/update_voice_server_settings` (RPC patch → apply → snapshot); i18n in en + all 13 locales.
|
||||
- **Privacy hook** — `spawn_lock_watcher` pauses capture + resets the segmenter while the screen is locked (macOS via `CGSessionCopyCurrentDictionary`, null/type-safe FFI; other platforms never pause yet).
|
||||
- Reused `audio_capture` helpers (`to_mono`/`resample`/`chunk_rms` made `pub(crate)` + new `encode_wav_16k`).
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [x] User can speak without pressing any hotkey
|
||||
- [x] VAD detects end of utterance and sends to agent
|
||||
- [x] Toggle in Settings → Voice
|
||||
|
||||
**Wake word "Hey Tiny" (live-fix, 2026-06-03):** always-on now only delivers an utterance to the agent when its transcript contains the wake word (`config.voice_server.wake_word`, default "Hey Tiny"); the phrase is stripped and the remainder is sent. Tolerant match (case/punctuation/leading-filler), empty wake word = deliver everything. This is a **text-based** wake word (transcribe-then-gate) — a first cut of Phase 3's trigger phrase; it fixes the "sends every utterance" spam but still runs STT on all speech (an on-device audio wake-word model for efficiency is the Phase 3 follow-up).
|
||||
|
||||
**Live-fixes found by running it:**
|
||||
- **Toggle did nothing** — `always_on_enabled` wasn't in the `update_voice_server_settings` RPC *param schema*, so validation rejected it before the handler. Added it; the config RPC now also calls `always_on::start_if_enabled` so the toggle starts/idles capture **live** (runtime `ENABLED` gate, no restart).
|
||||
- **`transcription failed: local ai is disabled`** — always-on used `voice_transcribe_bytes` (local whisper only). Now routes through `effective_stt_provider` + `create_stt_provider` (same factory dispatch as `voice.stt_dispatch`), honoring cloud STT.
|
||||
- Toggle surfaced in the reachable **VoicePanel** (Settings → Advanced → AI → Voice), not the hidden debug panel.
|
||||
|
||||
**Pending live validation (mic-dependent, can't be CI-tested):** say "Hey Tiny, <command>" and confirm the command reaches the agent; tune `vad_onset_threshold`/`vad_hangover_ms` to the user's mic + room. Windows/Linux screen-lock pause is a follow-up (no signal wired).
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Wake-Word + Fast Routing 🔨 Fast routing shipped; audio wake-word model pending
|
||||
|
||||
> Fast command routing is **merged** ([#3346](https://github.com/tinyhumansai/openhuman/pull/3346)). The single remaining Phase 3 item is the on-device **audio** wake-word model — **not started**; the text-based "Hey Tiny" transcribe-then-gate match from Phase 2 is the interim.
|
||||
|
||||
**Fast routing — DONE & WIRED:** `src/openhuman/voice/command_router.rs` (new) — pure `route(transcript) -> VoiceIntent` classifier (Play/Pause/Resume/Next/Previous/OpenApp/SetVolume/VolumeUp·Down/Mute/Unmute, else `Unknown`). High-confidence intents execute directly (launch_app / Music fast-path / osascript volume) without a full chat-LLM turn — the ≤500 ms path; `Unknown` defers to the agent so routing only ever shortcuts. Filler-tolerant ("please open up slack"). 5 unit tests.
|
||||
|
||||
**Router wired into delivery:** `always_on::deliver_command` now routes the extracted command through `command_router::route` first. Recognized intents run locally via `execute_intent` (Play → `automate` Music fast-path; OpenApp → `launch_platform`; Pause/Resume/Next/Previous/volume/mute → `osa` osascript); `VoiceIntent::Unknown` (and any local-exec failure) falls back to the full agent turn. Verified live: clean transport/open/volume commands execute on the fast path; complex/garbled `play` queries (e.g. mangled STT like "bts s latest song wind blowing") fall through to the agent as designed. Committed on `feat/voice-phase3-fast-routing`.
|
||||
|
||||
**Remaining Phase 3:** on-device audio wake-word model (Porcupine / ONNX) in `inference/voice/wake_word.rs` to gate STT *before* transcription — the text-based "Hey Tiny" wake match from Phase 2 (fuzzy anchor on the "tiny" token, Levenshtein ≤1) is the interim. This is the only open Phase 3 item; not yet started.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Wake-Word + Fast Routing (original plan)
|
||||
|
||||
> Activate only on a trigger phrase; route simple commands locally without a full LLM turn.
|
||||
|
||||
**Planned files:**
|
||||
- `src/openhuman/inference/voice/wake_word.rs` (new) — lightweight always-on model (Porcupine or custom ONNX) — ⏳ **not started** (interim: text-based "Hey Tiny" match in Phase 2)
|
||||
- `src/openhuman/voice/command_router.rs` (new) — intent→tool mapping for high-confidence commands, LLM fallback for ambiguous input — ✅ **shipped & wired** (see Phase 3 status above)
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Wake-word detection runs fully on-device (still pending — audio wake-word model not yet built)
|
||||
- [x] Latency from end-of-utterance to action start ≤ 500ms for local-routed commands (command router executes recognized intents on the local fast path)
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Polish ⏳ Not Started
|
||||
|
||||
> Voice confirmation loop, UI indicator, computer control onboarding.
|
||||
|
||||
**Planned:**
|
||||
- TTS confirmation before executing sensitive actions ("Opening Music — confirm?")
|
||||
- Always-on status indicator (notch pill from PR #3166 will handle this automatically)
|
||||
- Computer control (`mouse`/`keyboard` tools) toggle in Settings onboarding
|
||||
|
||||
---
|
||||
|
||||
## Fine-tuning backlog ⏳ (deferred until all phases complete)
|
||||
|
||||
From live agent-in-the-loop testing on 2026-06-03 (grounded in `~/.openhuman/logs/openhuman.2026-06-03.log`, `session_raw/*.jsonl`, and the dev run: **keyboard=69 / mouse=0 / screenshot=10** tool calls; **26 wake matches vs 93 misses**; emit=true utterances ranged 0.7s–28s). The feature works but needs tuning. **Do not implement until Phases 3–4 land.**
|
||||
|
||||
### F1 — Listening window too short for long commands
|
||||
- **Observed:** `vad_hangover_ms = 800` closes an utterance on any pause > 0.8s, so multi-clause commands ("Hey Tiny, open Slack and message the team channel saying …") split across utterances — the tail lacks the wake word and is dropped. Compounded by the notch "Listening" pill TTL (2500ms) expiring mid-speech, so it *looks* like it stopped listening.
|
||||
- **Resolve:** (a) raise `vad_hangover_ms` to ~1500ms; (b) **two-stage capture** — once the wake word is detected, open a dedicated longer command window (until a longer silence / N-second cap) instead of relying on a single VAD utterance; (c) keep the "Listening" pill alive for the whole utterance (extend/re-emit on each voiced frame, clear on `SpeechEnd`) so the notch reflects real mic state.
|
||||
|
||||
### F2 — Agent uses keyboard only, never the mouse
|
||||
- **Observed:** keyboard=69, mouse=0. Two causes: the orchestrator prompt is deliberately *keyboard-first*, **and** the downscaled screenshot's coordinates don't map to screen pixels — the capture is shrunk to ≤1568px while `mouse` expects absolute screen pixels (and Retina is 2× points), so any coordinate read from the image clicks the wrong spot. Vision-driven clicking is therefore currently unsafe and the agent (correctly) avoids it.
|
||||
- **Resolve:** (a) make `screenshot` emit a coordinate transform (shown WxH + real screen WxH + backing scale) **or** have `mouse` accept image-relative coordinates and convert internally; (b) once coordinates are trustworthy, soften the prompt so the agent uses screenshot→mouse to click specific elements, not just keyboard.
|
||||
|
||||
### F3 — No periodic screenshot/verify + foreground re-check
|
||||
- **Observed:** the agent screenshots ad-hoc (0 in the last session); `automate` only foregrounds at the start.
|
||||
- **Resolve:** in the `automate` loop **and** the orchestrator prompt — screenshot + verify at **start, after every ~3 actions, and at the end**; before each action confirm the frontmost app is the target and re-`launch_app` (foreground) it if not, then proceed. Fold the actual-vs-expected check into the loop's `verify` step.
|
||||
|
||||
---
|
||||
|
||||
## Permissions matrix — what the desktop agent needs to run
|
||||
|
||||
> Grounded in `src/openhuman/accessibility/permissions.rs` (detection),
|
||||
> `src/openhuman/accessibility/types.rs` (`PermissionKind`),
|
||||
> `app/src-tauri/Info.plist` (usage strings), and
|
||||
> `app/src-tauri/entitlements.sidecar.plist` (Hardened-Runtime entitlements).
|
||||
> Use this as the pre-test checklist when building for macOS (Apple Silicon +
|
||||
> Intel) and Windows.
|
||||
|
||||
### The four OS permissions the app tracks
|
||||
|
||||
`detect_permissions()` returns a `PermissionStatus` over these four
|
||||
`PermissionKind`s:
|
||||
|
||||
| Permission | Detected via (macOS) | Agent capability that needs it |
|
||||
|---|---|---|
|
||||
| **Microphone** | CPAL input-device probe (`detect_microphone_permission`) | All voice capture — dictation hotkey **and** always-on listening (`voice/always_on.rs` mic stream). **Required for "voice on" to capture anything.** |
|
||||
| **Accessibility** | `AXIsProcessTrusted()` | The whole app-control surface: `ax_interact` (AX tree read/press), `automate`, autocomplete focus query, **and synthetic keyboard/mouse injection** (enigo / CGEvent). |
|
||||
| **Screen Recording** | `CGPreflightScreenCaptureAccess()` | `screenshot` → `vision_click` (vision fallback for Electron/partial-AX apps) and screen-intelligence capture. |
|
||||
| **Input Monitoring** | `IOHIDCheckAccess(LISTEN_EVENT)` | Global hotkey **listening** (dictation hotkey / rdev / Globe-key listener — `accessibility/keys.rs`, `globe.rs`). Not needed for always-on (no hotkey). |
|
||||
|
||||
### Supporting macOS entitlements + usage strings (Hardened Runtime)
|
||||
|
||||
These are baked into the **signed build**, not runtime toggles. If missing, the
|
||||
OS blocks the call *before* any consent dialog renders — so they must be correct
|
||||
in the signed `.app`/DMG, and **cannot be validated with `tauri dev` alone**.
|
||||
|
||||
- **`Info.plist` usage strings:** `NSMicrophoneUsageDescription`,
|
||||
`NSCameraUsageDescription`, `NSAppleEventsUsageDescription` (+ Bluetooth /
|
||||
Location / Contacts / Calendar / folder strings — those are mostly for the
|
||||
**embedded provider webviews**, not the agent itself).
|
||||
- **`entitlements.sidecar.plist`:** `device.audio-input`, `device.camera`,
|
||||
`automation.apple-events` (drives `osascript` / System Events — Music
|
||||
transport, volume, foreground detection), `network.client` / `network.server`.
|
||||
|
||||
### Capability → permission map (agent surfaces)
|
||||
|
||||
| Capability | Microphone | Accessibility | Screen Recording | Apple Events / Automation | Input Monitoring |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|
|
||||
| Always-on capture (mic → VAD → STT) | ✅ | — | — | — | — |
|
||||
| Dictation hotkey | ✅ | — | — | — | ✅ (listen) |
|
||||
| `launch_app` / `OpenApp` intent | — | — | — | — | — |
|
||||
| `Pause`/`Next`/volume intents (osascript) | — | — | — | ✅ | — |
|
||||
| `Play` (Music fast-path) | — | ✅ | — | ✅ (verify `player state`) | — |
|
||||
| `ax_interact` (list/press/set_value) | — | ✅ | — | — | — |
|
||||
| `automate` general loop | — | ✅ | ✅ (only for `vision_click`) | ✅ (where it uses osascript) | — |
|
||||
| `vision_click` (Electron fallback) | — | ✅ | ✅ | — | — |
|
||||
| `keyboard` / `mouse` (synthetic input) | — | ✅ | — | — | — |
|
||||
|
||||
**Minimal "voice on" path:** Microphone to capture; then per intent —
|
||||
`OpenApp` needs nothing extra, transport/volume need Automation, `Play` needs
|
||||
Accessibility + Automation.
|
||||
|
||||
### Per-OS differences (the three test machines)
|
||||
|
||||
| | macOS (M-chip & Intel — identical) | Windows | Linux |
|
||||
|---|---|---|---|
|
||||
| **Microphone** | Real TCC prompt; **does** prompt | Real privacy gate — Settings → Privacy → Microphone | Ungated on standard desktops; Flatpak needs an XDG portal |
|
||||
| **Accessibility** | Manual grant in System Settings → Privacy & Security; **does not reliably auto-prompt**; usually needs an app **restart** after granting | No analog — UIA needs **no permission** for same-integrity apps; **cannot drive elevated apps** (UIPI) | App-interaction effectively **unsupported** (backend is macOS Swift helper / Windows UIA only; Linux returns a clean runtime error) |
|
||||
| **Screen Recording** | Manual grant; no reliable auto-prompt | `screenshot`/`vision_click` is **currently unimplemented on Windows** — the vision fallback can't run there yet | Unsupported |
|
||||
| **Input Monitoring** | Manual grant for hotkey listening | Not required | Not required |
|
||||
| **Entitlements** | Must be present in the signed build | n/a | n/a |
|
||||
|
||||
### Gotchas to flag before testing
|
||||
|
||||
1. **Detection ≠ entitlement.** On macOS the runtime checks report
|
||||
`Denied`/`Unknown` if the *entitlement* is missing from the signed build,
|
||||
independent of what the user clicked — so test on a **properly signed build**,
|
||||
not `tauri dev`.
|
||||
2. **Microphone detection is a proxy.** `detect_microphone_permission` infers
|
||||
from CPAL device enumeration, so "no device connected" and "permission denied"
|
||||
both surface as `Unknown` on macOS/Windows (`permissions.rs`). The always-on
|
||||
capture logging (`[voice::always_on] microphone permission: …` + device name +
|
||||
first-chunk confirmation) disambiguates this live.
|
||||
3. **Restart after granting** Accessibility / Screen Recording / Input
|
||||
Monitoring on macOS — TCC changes are not always picked up by a running
|
||||
process.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Phase | Item | Status |
|
||||
|---|---|---|
|
||||
| 1 | Auto-send after transcription | ✅ Done |
|
||||
| 1 | Shell allowlist for `open`/`xdg-open` | ✅ Done |
|
||||
| 1 | Shell tool description clarification | ✅ Done |
|
||||
| 1 | Dedicated `launch_app` tool | ✅ Done |
|
||||
| 1 | Orchestrator tool scope | ✅ Done |
|
||||
| 1 | SOUL.md capability hint | ✅ Done |
|
||||
| 1 | Diagnostic logging | ✅ Done |
|
||||
| 1 | Computer control (mouse/keyboard) | ❌ Reverted (CEF crash) |
|
||||
| 1 | AXUIElement app UI interaction (`ax_interact`) | ✅ Done |
|
||||
| 1 | Multi-step UI workflow guidance | ✅ Done |
|
||||
| 1 | Apple Music two-step play (navigate→play) | ✅ Done (playback best-effort) |
|
||||
| 1 | Full computer control (mouse/keyboard/screenshot) — CEF crash (Change 1.15) | ✅ Fixed (main-thread synthetic-input dispatch; screenshot downscale) |
|
||||
| 1 | Windows port (`launch_app` + UIA `ax_interact`) | ✅ Done (Calculator + Notepad hard-asserted on Win11) |
|
||||
| 1 | `automate(app, goal)` Rust-driven loop (Change 1.14) | ✅ M1–M5 done (proven live on macOS) |
|
||||
| 1.5 | M1: automate loop skeleton + tool | ✅ Done |
|
||||
| 1.5 | M2: poll-until-stable settle | ✅ Done |
|
||||
| 1.5 | M3: Music fast-path | ✅ Done (proven live on macOS) |
|
||||
| 1.5 | Robustness: quoted-query parse + no-progress guard | ✅ Done (from live agent failures) |
|
||||
| 1.5 | M4: progress streaming to notch | ✅ Done — notch cherry-picked in; automate streams live steps |
|
||||
| 1.5 | M5: richer element model (`enabled`) | ✅ Plumbed; AXEnabled found unreliable → informational only |
|
||||
| 1.5 | Native fast-paths beyond Music (Spotify/Slack) | ➖ Descoped (general loop covers them; Music shipped in M3) |
|
||||
| 1.5 | Vision fallback for Electron apps (Change 1.16) | ✅ Done (`vision_click`: screenshot → vision-locate → guarded click; frontmost guard; F2 coord-transform folded in) |
|
||||
| 1.5 | Browser fast-path (Change 1.17a) | ✅ Done (`browser.rs`: deterministic `open -a "<browser>" url` nav; play→fall-through to vision_click) |
|
||||
| 1.5 | Cross-platform browser shortcut table (Change 1.17b) | ✅ Done (`browser_shortcuts.rs`: Chrome/Firefox/Safari/Edge × macOS/Win/Linux) |
|
||||
| 1.5 | App-shortcut fast-path: Spotify/Apple Music/Slack (Change 1.17c) | ✅ Done (`app_shortcuts.rs`: transport + nav via each app's shortcuts) |
|
||||
| 1.5 | Keyboard-shortcut plumbing: `hotkey` verb + backend methods (Change 1.17d) | ✅ Done (`key`/`type_text`/`open_url_in_app`/`now_playing`; shared keyboard helpers) |
|
||||
| 1.5 | Artist-aware Music verification (Change 1.17e) | ✅ Done (`now_playing` osascript; honest wrong-artist report, no false success) |
|
||||
| 1.5 | Actionable `automate` responses (Change 1.17f) | ✅ Done (next-step guidance, candidate/label lists, verified-only success, song-row preference) |
|
||||
| 2 | Always-on microphone loop | ✅ Done (cpal → VAD → STT → agent) |
|
||||
| 2 | `always_on_enabled` config flag + Settings toggle | ✅ Done (RPC + UI + i18n) |
|
||||
| 2 | Privacy hook (screen lock pause) | ✅ Done (macOS; other OSes follow-up) |
|
||||
| 2 | Text-based "Hey Tiny" wake word | ✅ Done (interim; gates delivery, strips phrase) |
|
||||
| 3 | Local command router (intent classifier) | ✅ Done & wired (recognized intents run on the ≤500ms local path; Unknown defers to agent) |
|
||||
| 3 | On-device audio wake-word model | ⏳ Not started (text-based match is the interim) |
|
||||
| 4 | Voice confirmation loop | ⏳ Not started |
|
||||
| 4 | Computer-control onboarding toggle | ⏳ Not started |
|
||||
| 4 | Always-on UI indicator | ✅ Done (notch PR #3166) |
|
||||
@@ -1,35 +0,0 @@
|
||||
# Add WeChat message scraping into context and memory
|
||||
|
||||
## Summary
|
||||
|
||||
Ingest messages from the embedded WeChat webview into OpenHuman so the agent can use recent WeChat conversations as context and persist useful history into memory.
|
||||
|
||||
## Problem
|
||||
|
||||
We can embed WeChat Web in the Tauri shell for sign-in and manual use, but OpenHuman does not yet extract message data from that surface. That leaves a gap for users who coordinate through WeChat, especially for China-based communities and teams. The work needs to respect platform limits in WeChat Web, avoid broad DOM scraping that breaks easily, and keep private message content scoped to explicit product surfaces and consented storage paths.
|
||||
|
||||
## Solution (optional)
|
||||
|
||||
Add a dedicated WeChat webview ingestion path, following the existing Franz-style account model but with provider-specific extraction for chats, messages, unread state, and stable account/chat identifiers. Scope should cover:
|
||||
|
||||
- Tauri/app shell:
|
||||
Add a WeChat-specific extractor path for the embedded webview, with clear load-state handling and bounded retries when login/session state changes.
|
||||
- Core:
|
||||
Define a WeChat webview ingest contract that can write normalized message snapshots into context/memory namespaces, similar to other webview-account providers.
|
||||
- Product safeguards:
|
||||
Gate persistence behind existing memory/write surfaces, document what is stored, and make it easy to disable or purge.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] **Embedded WeChat extraction** — OpenHuman can detect the active WeChat chat surface and extract normalized message data from the embedded webview.
|
||||
- [ ] **Context + memory ingestion** — extracted WeChat messages can be written into the existing context/memory pipeline with provider/account provenance.
|
||||
- [ ] **Unread/respond surfaces** — unread or actionable WeChat activity appears in the same provider-surface/respond-queue flows used by other messaging integrations where applicable.
|
||||
- [ ] **Privacy + control** — users can understand, disable, and purge stored WeChat-derived data using existing account/memory controls or clearly-scoped additions.
|
||||
- [ ] **Tests** — add focused Vitest and Rust coverage for the WeChat provider path, including at least one failure/edge case.
|
||||
- [ ] **Diff coverage ≥ 80%** — the implementing PR meets the changed-lines coverage gate (Vitest + cargo-llvm-cov, enforced by [`.github/workflows/pr-ci.yml`](../.github/workflows/pr-ci.yml)).
|
||||
|
||||
- Evaluate whether WeChat needs a native scanner path, injected bridge, or hybrid approach after validating what the embedded web client exposes at runtime.
|
||||
|
||||
## Related
|
||||
|
||||
- Follows the initial Tauri WeChat webview support added in `feat/wechat-webview-support`.
|
||||
@@ -1,78 +0,0 @@
|
||||
# WhatsApp data flow — scanner, store, agent
|
||||
|
||||
**Issue:** [#1341](https://github.com/tinyhumansai/openhuman/issues/1341)
|
||||
|
||||
This document describes how WhatsApp Web data captured by the desktop scanner becomes available to the agent. It exists to clear up the most common confusion: there are **two** local storage paths and they are intentional, not duplicates — each backs a different agent capability.
|
||||
|
||||
## Pipeline at a glance
|
||||
|
||||
```text
|
||||
┌────────────────────────┐
|
||||
│ WhatsApp Web (CEF view)│
|
||||
└────────────┬───────────┘
|
||||
│ CDP scan tick
|
||||
▼
|
||||
┌────────────────────────────────────┐
|
||||
│ app/src-tauri/src/whatsapp_scanner │
|
||||
│ (DOM + IndexedDB merge) │
|
||||
└─────┬───────────────────────┬──────┘
|
||||
│ exact rows │ canonicalised transcript
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌──────────────────────────┐
|
||||
│ openhuman.whatsapp_ │ │ openhuman.memory_doc_ │
|
||||
│ data_ingest │ │ ingest │
|
||||
│ (internal-only RPC) │ │ (internal-only RPC) │
|
||||
└──────────┬───────────┘ └─────────────┬────────────┘
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌──────────────────────────┐
|
||||
│ whatsapp_data.db │ │ memory tree │
|
||||
│ (SQLite, per-account)│ │ (per-source summaries + │
|
||||
│ - wa_chats │ │ embeddings) │
|
||||
│ - wa_messages │ │ │
|
||||
└──────────┬───────────┘ └─────────────┬────────────┘
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌──────────────────────────┐
|
||||
│ Agent tools │ │ Agent tools │
|
||||
│ whatsapp_data_* │ │ memory_tree_* │
|
||||
│ (exact lookup) │ │ (semantic / cross-src) │
|
||||
└──────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
Both ingest endpoints fire on every scan tick; both are `tokio::spawn` fire-and-forget so the scanner never blocks on either HTTP call.
|
||||
|
||||
## Why two paths
|
||||
|
||||
| Path | Backing store | Strength | Use it for |
|
||||
| --------------- | ----------------------------------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------- |
|
||||
| **Direct** | `whatsapp_data.db` (SQLite) | Exact, structured, paginated | "List my WhatsApp chats", "show the last 50 messages with Alice", "search for `invoice` across WhatsApp" |
|
||||
| **Memory tree** | Per-source memory tree + embeddings | Semantic, cross-source | "Summarise this week of WhatsApp", "find action items across email and WhatsApp", "what did the team agree on?" |
|
||||
|
||||
The same scan tick populates both stores. Idempotency keys make the dual-write safe to retry:
|
||||
|
||||
- `whatsapp_data_ingest` keys on `(account_id, chat_id, message_id)` — UPSERT.
|
||||
- `memory_doc_ingest` keys on `(namespace, key)` where namespace is `whatsapp-web:<account_id>` and key is `<chat_id>:<day>` — also UPSERT.
|
||||
|
||||
If one path fails (network blip, store init race), the other still progresses. The next scan tick converges both stores.
|
||||
|
||||
## Read-only boundary
|
||||
|
||||
The scanner write-path RPCs are registered as **internal-only** in [`src/core/all.rs`](../src/core/all.rs) under `build_internal_only_controllers`. They are reachable over JSON-RPC but invisible to the agent's tool catalog and to schema discovery (`all_controller_schemas`). The agent has **no** way to call `whatsapp_data_ingest` or `memory_doc_ingest` — accidentally or otherwise.
|
||||
|
||||
The agent surfaces are exclusively read-only:
|
||||
|
||||
- [`src/openhuman/whatsapp_data/tools/`](../src/openhuman/whatsapp_data/tools/) — `whatsapp_data_list_chats`, `whatsapp_data_list_messages`, `whatsapp_data_search_messages`. All three wrap their RPC counterparts and emit a `"provider": "whatsapp"` tag in the response so the agent can cite WhatsApp as the source.
|
||||
- [`src/openhuman/memory/query/`](../src/openhuman/memory/query/) — generic `memory_tree_*` tools. Filter by `source_kind: "chat"` or query directly; WhatsApp chat-day transcripts are tagged `whatsapp` so they surface in cross-source flows.
|
||||
|
||||
## Why the orchestrator only lists three of these
|
||||
|
||||
The orchestrator's `agent.toml` exposes the three direct WhatsApp tools alongside the generic `memory_tree_*` family. That choice is deliberate — adding more provider-specific tools would compete with the memory-tree tools for the same intents and fragment routing. The combination satisfies the three real shapes of WhatsApp request:
|
||||
|
||||
1. **Exact lookup** ("what was my last message with Bob") → `whatsapp_data_list_messages` after `whatsapp_data_list_chats`.
|
||||
2. **Keyword search** ("did anyone mention `Q3` on WhatsApp") → `whatsapp_data_search_messages`.
|
||||
3. **Summarisation / action items / cross-source** ("what came up across WhatsApp and email this week") → `memory_tree_query_source { source_kind: "chat" }` or `memory_tree_query_global`.
|
||||
|
||||
If a future intent doesn't fit any of these, the right move is usually a new memory-tree retrieval primitive, not a new provider-specific tool.
|
||||
|
||||
## What this fix changed (#1341)
|
||||
|
||||
Prior to #1341 the read-only RPC controllers existed and were callable over JSON-RPC, but no `Tool` impl wrapped them and the orchestrator didn't list them — so the agent could see WhatsApp data only through the memory tree. That worked for summaries but failed on exact-lookup intents because the memory tree's per-day transcript granularity loses the structure the user asks about (sender JID, exact `chat_id`, per-message timestamp). Adding the three direct tools closed that gap without adding any new ingest path.
|
||||
Reference in New Issue
Block a user