mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
ea088d9f15
* feat(prompt): unified delegation guide + dynamic per-toolkit tool registration (#447) Cuts main agent prompt token cost ~98% on accounts with connected integrations and replaces the generic Composio dispatcher with native per-action tool calling inside skills_agent. ## Why Issue #447: prompt payloads were ballooning because main/orchestrator re-shipped a full prose enumeration of every connected toolkit's actions on every turn (~13.7k tokens for one gmail integration alone), and skills_agent had no way to scope tools to a single toolkit so it inherited a flat dispatcher (composio_execute) that the LLM couldn't reliably call without first round-tripping composio_list_tools. ## What ### main / orchestrator prompt - Replace `ConnectedIntegrationsSection`'s per-action prose dump with a unified Delegation Guide: one bullet per integration (toolkit + one-line description + spawn snippet). Tokens added per integration: ~30. Savings vs old per-action listing: ~5k+ tokens per connected toolkit. - Drop every "Composio" reference from delegating-agent prose so the surface stays provider-neutral. ### skills_agent - Add a `toolkit` argument to `spawn_subagent` with three pre-flight validation modes resolved against the parent's connected integrations list before any LLM call: * missing toolkit -> ToolResult::error (model retries) * unknown toolkit -> ToolResult::error (model retries) * in allowlist, not connected -> ToolResult::success (model surfaces "authorize in Settings -> Integrations" to user; not flagged as a tool failure in the agent loop or web channel) - New `ComposioActionTool` (`src/openhuman/composio/action_tool.rs`) implements the `Tool` trait per Composio action. The sub-agent runner constructs ~N of these at spawn time from the cached integration overview and injects them via a new `extra_tools` parameter through `run_typed_mode` -> `run_inner_loop`. - `filter_tool_indices` drops every skill-category parent tool when `is_skills_agent_with_toolkit` is true so the only skill-category entries the sub-agent sees are the freshly-built per-action tools. This eliminates apify_*, composio_list_*, composio_authorize, and composio_execute from the toolkit-scoped surface. - Sub-agent renderer takes the parent's actual `tool_call_format` instead of hardcoding PFormat. Native dispatchers no longer carry a prose `## Tools` section at all (schemas already travel through the request body's `tools` field) — eliminates a ~30k-token duplication that was blowing past the model's context window. ### integration overview fetch - `fetch_connected_integrations_uncached` now merges Composio's toolkit allowlist with the user's active connections and returns one `ConnectedIntegration` per allowlisted toolkit with a `connected: bool` flag. Unconnected entries carry no schemas, just the toolkit name + description, so the orchestrator can mention them without trying to invoke them. - `ConnectedIntegrationTool` preserves the action's full JSON parameter schema so `ComposioActionTool` can advertise it through native function-calling. ### plumbing - `ParentExecutionContext` carries a `ComposioClient` and the parent's resolved `tool_call_format`, populated alongside `connected_integrations` in `Session::fetch_connected_integrations`. Triage and test paths pass `None` / `PFormat` defaults. - `dispatch_subagent` (the `SkillDelegationTool` path) plumbs its pre-bound skill_id through `toolkit_override` instead of the broken `skill_filter_override` (which used `{skill}__` prefix matching that never matched Composio's `TOOLKIT_*` naming). - `web::run_chat_task` failures now log the underlying error at WARN so debugging an in-flight failure no longer requires turning on TRACE for socket events. - `scripts/stage-core-sidecar.mjs` queries `cargo metadata` for the real target directory instead of assuming `<repo>/target` so the staging step works under workspace-level `target-dir` overrides (the vezures-workspace shared `.cargo-target` setup). ## Verified end-to-end Two RPC tests against a freshly-rebuilt sidecar (gmail connected, notion / slack / etc. allowlisted but not connected): | Test | Behavior | Tokens | Iterations | |---|---|---|---| | Connected (gmail, "fetch 5 unread emails") | spawn_subagent -> 62 dynamic gmail tools registered -> 1 GMAIL_FETCH_EMAILS call with smart args -> markdown table response | 42,911 input | 1 | | Not-connected (notion, "create a page") | main answers directly without spawning, tells user to authorize in Settings -> Integrations | 3,319 input | 1 | Both flows complete cleanly. The connected path proves dynamic per-action tool registration is working with full schema validation; the unconnected path proves the unified delegation guide + ToolResult::success return for not-connected toolkits keeps the model on a graceful path without polluting the chat with error styling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(prompt): update test callers + assertions for new renderer signature Follow-up to #447. The main patch changed three signatures that test callers hadn't been updated for, and flipped one assertion that was validating the now-removed prose schema duplication. - `SubagentRunOptions` test constructors in subagent_runner.rs (2 sites) now pass `toolkit_override: None`. - `ConnectedIntegration` test constructor in orchestrator_tools.rs now passes `connected: true` (the default for test integrations — they're treated as authorized so delegation logic still runs). - 12 `render_subagent_system_prompt` test callers in prompt.rs now pass `&[]` for the new `extra_tools` slice and `ToolCallFormat::PFormat` for the new `tool_call_format` argument. - `render_subagent_system_prompt_honors_identity_safety_and_skills_flags` used to assert `rendered.contains("Parameters:")` on the Json dispatcher branch — that was valid in the old world where the prose `## Tools` section dumped full JSON schemas for Json/Native formats. The main patch deliberately removes that dump (it was the ~30k-token duplication of the native `tools` field), so the test now asserts the opposite: no `## Tools` header and no `Parameters:` line are emitted for Native/Json dispatchers. The schemas still travel through the provider request's `tools` field. Also picks up `rustfmt` rewraps in action_tool.rs and ops.rs from a background linter run — pure whitespace, no semantic change. Verified green against the full `cargo test --lib` suite for every test touched by this PR: - openhuman::context::prompt::tests (26 passed) - openhuman::agent::harness::subagent_runner::tests (19 passed) - openhuman::composio::ops::tests (2 passed) - openhuman::tools::impl::agent::tests (0 scoped) The 7 remaining failures in `cargo test --lib` are pre-existing Windows-path/filesystem flakes in subsystems this PR doesn't touch (self_healing polyfill path separator, cron scheduler shell spawning, local_ai::paths absolute-path detection, security::policy sandbox path handling, composio::trigger_history jsonl archive, and a real pre-existing `Option::unwrap()` panic in browser::screenshot). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(harness): add composio_client + tool_call_format to integration test stub Follow-up to #447. The `tests/agent_harness_public.rs` integration test constructs a `ParentExecutionContext` in `stub_parent_context` that was missing the two fields added in the main patch (`composio_client` and `tool_call_format`). `cargo test --lib` doesn't compile integration tests, which is why this slipped past local verification — it only surfaced on Linux CI. - `composio_client: None` — the stub parent has no composio client because these tests don't exercise the integration-overview path. - `tool_call_format: ToolCallFormat::PFormat` — default legacy format; none of the tests in this file exercise the sub-agent renderer's format branching, so PFormat is the safe pick. Verified locally that `cargo test --no-run` compiles all integration test targets including `agent_harness_public`. (The `agent_memory_loader_public` link error in the local run is a pre-existing Windows `libucrt`/`fgets` C-runtime issue unrelated to this PR — won't reproduce on Linux CI.) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(prompt,composio): address CodeRabbit review on #570 (#447) Four fixes from the CodeRabbit review of PR #570, ranked by impact: ## `context/prompt.rs` — Json dispatcher needs prose tool catalog The initial patch gated the prose `## Tools` section behind `matches!(tool_call_format, ToolCallFormat::PFormat)`, which conflated Json with Native. Only `ToolCallFormat::Native` uses the provider's native function-calling channel where schemas travel via the request body's `tools` field. `ToolCallFormat::Json` is a prompt-driven format (the model wraps JSON tool calls in `<tool_call>` tags) and relies on the prose catalog the same way PFormat does — without it the model sees protocol instructions but no visible tool names or schemas. Inverted the guard to `!matches!(tool_call_format, Native)` so PFormat and Json both render the `## Tools` section with their respective per-entry shapes (compact `Call as:` signature for PFormat, inline `Parameters:` JSON schema for Json). Updated the `render_subagent_system_prompt_honors_identity_safety_and_skills_flags` test: the Json assertion now expects `Parameters:` to be present again (reverting commit 2's flip), and a new Native-branch assertion guards against the ~54k-token schema duplication the original PR fixed. ## `composio/ops.rs` — don't cache degraded snapshots All three backend calls in `fetch_connected_integrations_uncached` (`list_toolkits`, `list_connections`, `list_tools`) previously returned `Some(Vec::new())` or fell through to an empty inner vec on transient errors. The outer `fetch_connected_integrations` caches whatever the uncached path returns, so a single transient 5xx would silently hide every integration, mark connected toolkits as disconnected, or register dynamic Composio tools with zero callable actions — until the cache is invalidated or the process restarts. Changed all three branches to return `None`, which signals the caller to NOT cache the result and retry on the next call. ## `composio/ops.rs` — prefix match needs a delimiter `starts_with(&slug.to_uppercase())` false-matches when two toolkit slugs share a text prefix (e.g. `git` vs `github`). The current allowlist doesn't trigger this, but adding a new toolkit could silently leak actions between integrations. Anchored the prefix with an underscore so `GMAIL_SEND_EMAIL` matches `gmail_`, not just `gmail`. ## `scripts/stage-core-sidecar.mjs` — resolve CARGO_TARGET_DIR vs repo root `resolve(process.env.CARGO_TARGET_DIR)` uses the process's current working directory, but the `cargo build` spawn below runs with `cwd: root`. For an absolute env var this is identical, but a relative `CARGO_TARGET_DIR` and a cwd outside the repo would make the two paths disagree and the binary lookup would miss. Defensive fix: `resolve(root, process.env.CARGO_TARGET_DIR)` so both paths anchor to the same base. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OpenHuman
The age of super intelligence is here. OpenHuman is your Personal AI super intelligence. Private, Simple and extremely powerful.
Discord • Reddit • X/Twitter • Docs
"The Tet. What a brilliant machine" — Morgan Freeman as he reminisces about alien superintelligence in the movie Oblivion
Early Beta — Under active development. Expect rough edges.
To install or get started, either download from the website over at tinyhumans.ai/openhuman or run
# For MacOS/Linux
curl -fsSL https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.sh | bash
# For Windows
irm https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.ps1 | iex
What is OpenHuman?
OpenHuman is an open-source agentic assistant that is designed to integrate with you in your daily life. Here's what makes OpenHuman special:
-
Simple, UI-first — A clean desktop experience and short onboarding paths so you can go from install to a working agent in a few clicks, without a config-first setup. You don't need a terminal to run OpenHuman.
-
One subscription, many providers — You only need one account to get access to many agentic APIs (AI Models, Search, Webhooks/Tunnels and other 3rd party APIs etc..), simplifying the experience to get a powerful agent going.
-
Rich Skills — Plug into Gmail, Slack, Notion, and the rest of your stack via rich, feature-backed skills. Connections are typically one click through setup wizards instead of wiring APIs by hand. Workflow data is kept on device, encrypted locally, and treated as yours: encryption and sensitive context stay on your machine. Webhooks give instant feedback into the agent when external systems or skills emit events, so the loop stays tight without constant polling.
-
Local knowledge base — Built from your data and your activity. How you work across tools, sessions, and connected services—so the agent gets rich, workflow-aware context, not a one-off chat transcript. Everything is stored on your machine and compounding over time without becoming a cloud dossier. Channels, skills and ongoing conversations feed the same loop so day-to-day context does not reset every session.
-
Local AI model — The Rust core exposes local AI paths (and the desktop bundle can ship local/bundled runners where applicable) for the workloads above—vision snippets, speech helpers, summarization, tooling—so sensitive steps can stay off the cloud when you choose.
-
Deep desktop integrations — OpenHuman is a native desktop assistant, not a web-only chat: memory-aware keyboard autocomplete, voice (STT listening and TTS replies), screen intelligence that understands what is on screen and feeds your local context, plus windowing and OS-level permissions—so the agent meets you on the machine, not trapped in a browser tab.
Architecture: docs/ARCHITECTURE.md. Contributor orientation: CONTRIBUTING.md.
OpenHuman vs other agents
High-level comparison (products evolve—verify against each vendor). OpenHuman is built to minimize vendor sprawl, keep workflow knowledge on-device, and ship deep desktop features—not only chat.
| Claude Code/Cowork | OpenClaw | Hermes Agent | OpenHuman | |
|---|---|---|---|---|
| Open-source: Is the codebase open to review? | 🚫 Proprietary client | ✅ MIT License | ✅ MIT License | ✅ GNU License |
| Simple: Is it simple to get started? | ✅ Simple Desktop App + CLI | ⚠️ Terminal first and often complex | ⚠️ Terminal first and often complex | ✅ Simple, Clean UI/UX. Get started within minutes |
| Cost: How expensive is to run? | ⚠️ Subscription + add-on tool/API costs | ⚠️ Tied to models & hosting you choose | ⚠️ Tied to models & hosting you choose | ✅ Cost optimized with the option to run many things locally for free |
| Memory & Knowledge Base (KB): Does the agent know you and your world? | ✅ Built-in memory; mostly chat/session scoped | ⚠️ Has a local memory but often needs plugins for richer behavior | ✅ Self-learning / task loops (typical) | 🚀 Local KB + Self-learning from your activity & data (GMail, Notion etc... via skills) & prompts |
| API spagetti: How complex is it to hook mulitple features together? | 🚫 Claude bill + often extra keys for MCP/tools | 🚫 BYOK / multi-vendor common | 🚫 Multiple providers common | ✅ One account get access to many bundled platform APIs |
| Extensibility: Can you add rich features into it? | ✅ MCP (different model than sandboxed skills) | ✅ Plugin Architecture (SKILL.md) | ✅ Plugin Architecture (SKILL.md) | 🚀 Rich Skills with ability to have realtime updates, local DB & more |
| Desktop integrations: Can it integrate into your desktop completely? | ⚠️ Desktop app & access to folders | ⚠️ Often lighter native surface | ⚠️ Often lighter native surface | ✅ STT, TTS, screen intelligence, memory-aware autocomplete and a whole lot more |
Contributors Hall of Fame
Show some love and end up in the hall of fame