mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
8d4934e634
* feat(agent): summarizer sub-agent compresses oversized tool results Issue #574. Before this change, tool results larger than a few KB landed verbatim in orchestrator history, burning context budget on raw JSON/HTML/file dumps. The only guardrail was tool_result_budget_bytes which hard-truncated mid-payload, dropping everything past the cut. This PR introduces a dedicated `summarizer` sub-agent (model.hint = "summarization") that the runtime automatically dispatches whenever a tool result exceeds summarizer_payload_threshold_bytes (default 100 KB). The summarizer compresses the payload per an extraction contract that preserves identifiers and key facts, and the compressed summary replaces the raw payload before it enters agent history. Payloads above summarizer_max_payload_bytes (default 5 MB) skip summarization and fall through to the existing truncation path — paying for an LLM call on a 5 MB blob is counterproductive. Scoped to the orchestrator session only. Welcome, skills_agent, researcher, planner, and every other typed sub-agent get None and their tool results are untouched. Gated in build_session_agent_inner by checking agent_id == "orchestrator". Instrumented at both tool-loop paths: - run_tool_call_loop in tool_loop.rs (event-bus/channels path) - Agent::execute_tool_call in session/turn.rs (web channel path via run_single) Both paths call into the same `PayloadSummarizer::maybe_summarize` trait so the threshold check, circuit breaker (3 consecutive failures disables for the session), and sub-agent dispatch policy live in exactly one place (agent/harness/payload_summarizer.rs). The summarizer sub-agent is runtime-dispatched only — it is NOT exposed as a delegation tool to the orchestrator's LLM. It is listed in the orchestrator's `subagents = [...]` for explicit registration, but `collect_orchestrator_tools` filters out the `summarizer` id and never synthesises a `delegate_summarizer` tool. Files: * NEW: src/openhuman/agent/agents/summarizer/{agent.toml,prompt.md} * NEW: src/openhuman/agent/harness/payload_summarizer.rs (trait + SubagentPayloadSummarizer impl + circuit breaker + tests) * src/openhuman/agent/agents/mod.rs — register summarizer in BUILTINS * src/openhuman/agent/agents/orchestrator/agent.toml — list summarizer in subagents (runtime-only, no LLM delegation tool) * src/openhuman/agent/harness/mod.rs — declare module * src/openhuman/agent/harness/builtin_definitions.rs — expect summarizer in `expected_builtin_ids_are_present` * src/openhuman/agent/harness/tool_loop.rs — thread Option<&dyn PayloadSummarizer> into run_tool_call_loop + agent_turn, intercept at the tool-execution success site, plus a new integration test using a MockSummarizer * src/openhuman/agent/harness/tests.rs — thread None through the existing run_tool_call_loop callsites * src/openhuman/agent/harness/session/turn.rs — same interception in Agent::execute_tool_call * src/openhuman/agent/harness/session/types.rs — payload_summarizer field on Agent and AgentBuilder * src/openhuman/agent/harness/session/builder.rs — .payload_summarizer() setter + orchestrator-only construction in build_session_agent_inner * src/openhuman/agent/bus.rs — pass None for the bus path * src/openhuman/config/schema/context.rs — summarizer_payload_threshold_bytes + summarizer_max_payload_bytes on ContextConfig * src/openhuman/tools/orchestrator_tools.rs — filter out the summarizer id from delegation-tool synthesis Tests: unit tests in payload_summarizer pin the pass-through rules (below threshold, above max cap, breaker tripped) and the prompt construction. Integration test in tool_loop::tests uses a MockSummarizer to verify the interception wires through end-to-end. Closes tinyhumansai/openhuman#574. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tools): csv_export tool + skills_agent oversized output handling When a Composio tool returns a large payload inside skills_agent, the agent now has two prompt-driven paths: Path A — user wants an answer derived from the data: skills_agent extracts the answer in its next iteration and returns a targeted response (no file I/O needed). Path B — user wants the actual raw dataset: skills_agent calls csv_export (tabular data) or file_write (non-tabular) to persist the output to workspace/exports/, then returns a summary + file path. Changes: * NEW: src/openhuman/tools/impl/filesystem/csv_export.rs — CsvExportTool: parses JSON array, formats as CSV, writes to workspace/exports/{filename}. Handles missing keys (empty cells), nested values (JSON-serialised), and optional column ordering. Sandboxed via SecurityPolicy. * src/openhuman/tools/impl/filesystem/mod.rs — wire module * src/openhuman/tools/ops.rs — register CsvExportTool * src/openhuman/agent/harness/definition.rs — new extra_tools field on AgentDefinition, allowing named system tools to bypass category_filter * src/openhuman/agent/harness/subagent_runner.rs — inject extra_tools into allowed_indices after category filtering * src/openhuman/agent/agents/skills_agent/agent.toml — add file_write + csv_export via extra_tools * src/openhuman/agent/agents/skills_agent/prompt.md — new "Handling Oversized Tool Results" section with Path A (extract answer) vs Path B (export file) decision tree, intent-detection heuristics, and examples Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(agent): progressive-disclosure handoff for skills_agent + token-based summarizer threshold (#574) Two layered fixes for #574 plus supporting changes. ## Summarizer thresholds in tokens, not bytes Renamed `ContextConfig::summarizer_payload_threshold_bytes` -> `summarizer_payload_threshold_tokens` (default 500 000, was 100 KB in bytes) and `summarizer_max_payload_bytes` -> `summarizer_max_payload_tokens` (default 2 000 000). Token count is estimated at ~4 chars/token via a local helper that mirrors `tree_summarizer::types::estimate_tokens`. Serde aliases on both fields keep existing config.toml files parsing. `payload_summarizer.rs` now compares token estimates for the threshold and cap, with both `tokens=` and `bytes=` surfaced in log events. ## Progressive-disclosure handoff for skills_agent `skills_agent` typed subagents (when spawned with a Composio toolkit) now receive a per-spawn `ResultHandoffCache` and a new built-in `extract_from_result` tool. When a per-action tool returns more than `HANDOFF_OVERSIZE_THRESHOLD_TOKENS` (20 000 tokens), the raw payload is stashed in the cache and a compact placeholder replaces it in history: [oversized tool output: 187432 bytes - stashed as result_id="res_1"] Preview (first 1500 chars): ... If the preview does not answer your task, call: extract_from_result(result_id="res_1", query="<specific question>") The subagent can answer from the preview, call `extract_from_result` with a targeted query, or re-call the original tool with tighter filters. `extract_from_result` dispatches the `summarizer` subagent against the cached payload with the query as the extraction contract. For payloads over `SUMMARIZER_CHUNK_CHAR_BUDGET` (60 000 chars) the extractor runs a chunked map-reduce: - split at blank-line / newline boundaries - map each chunk to a per-chunk partial via bounded concurrency (`buffer_unordered(3)`) to avoid storming the provider gateway - reduce partials in one more summarizer turn, or fall back to the concatenation if the reduce stage fails Falls back gracefully when the summarizer definition is missing from the registry (placeholder + preview still work; just no extractor). ## Text-mode dispatcher for skills_agent + tighter tool filter Provider-side grammar decoders (Fireworks etc.) cap tool-schema rules at 65 535 (uint16_t). Large Composio toolkits - Gmail, Notion, GitHub, Salesforce, HubSpot, Google Workspace - ship per-action JSON schemas dense enough that even a handful of them exceed the cap. - `subagent_runner::run_inner_loop` now detects `skills_agent` and omits `tools: [...]` from the API request, instead appending an `<tool_call>{...}</tool_call>` XML protocol block (with compact per-tool parameter summaries) into the system prompt. Tool calls are parsed out of the model's free-form response via the shared `parse_tool_calls` helper. Mirrors XmlToolDispatcher behaviour. - `top_k_for_toolkit()` picks 12 for HEAVY_SCHEMA_TOOLKITS (the six listed above plus googledocs, googlesheets, microsoftteams) vs the default 25 for lighter toolkits. ## Skills_agent agent.toml / prompt changes - Dropped `csv_export` from `skills_agent.extra_tools` (kept `file_write`) - the export tool was triggering superfluous file writes after extraction had already answered the query. - `builtin_definitions::skills_agent_has_extra_tools_for_export` test inverted to assert `csv_export` is NOT present. - `prompt.md` re-aligned: Path B now references only `file_write`. ## Wiring - `session/builder.rs`: threshold rename wiring (`_tokens`). - `subagent_runner.rs::run_inner_loop`: new `handoff_cache` parameter (threaded through both call sites; fork-mode passes `None`). ## Tests All green: - `payload_summarizer::tests`: 6/6 (thresholds in tokens) - `tool_loop::tests`: 10/10 (MockSummarizer integration) - `subagent_runner::tests`: 19/19 - `builtin_definitions::tests`: 5/5 (csv_export removal) - `csv_export::tests`: 7/7 (implementation kept, wiring removed) Closes #574. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(agent): drop reduce LLM call and pre-clean tool outputs in chunked extraction Two optimisations to the oversize-handoff / extract_from_result pipeline: 1. Concat chunk summaries in order; drop reduce stage. The reduce-stage summarizer call was the slowest single turn of the pipeline and a single point of failure when the upstream provider hung — a hung reduce burned minutes before timing out. Map summaries are now sorted by original chunk index and concatenated with a plain-separator join. For listing/extraction queries this is equivalent; for top-N / global-ordering queries the caller can post-process. Observed: 8m24s → ~70s on a Notion SEARCH run. 2. Generic cleaner before the oversize check. Strips <script>/<style>/<svg> blocks, HTML comments, data:...;base64,... inline URIs, remaining HTML tags, and collapses whitespace. Applied on every tool output so results that are mostly markup drop below the 20k-token threshold and skip the extract pipeline entirely; the ones that remain oversized get chunked on clean bytes. Debug log emits before/after bytes + saved_pct per call. No changes to the extract_from_result tool surface or placeholder format. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(chat): rearm 2-min silence timer on inference progress events The client-side safety timer in Conversations.tsx was a flat 120s deadline from the send, not a silence window. Long agent turns (chained tool calls, chunked extraction fan-outs, subagent spawns) would trip the timeout even though the backend was actively making progress — showing "No response from the assistant after 2 minutes" while the server was still working. The effect now watches inferenceStatusByThread and rearms the timer on every status change for the sending thread. Any tool_call / tool_result / iteration_start / subagent_{spawned,done} event resets the 120s window. When status is cleared (chat_done / chat_error), the timer is dropped. A truly silent server still fails after 120s as before. Tracks the sending thread id in a ref (sendingThreadIdRef) so switching threads mid-turn doesn't move the timer's reference point. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OpenHuman
The age of super intelligence is here. OpenHuman is your Personal AI super intelligence. Private, Simple and extremely powerful.
Discord • Reddit • X/Twitter • Docs
"The Tet. What a brilliant machine" — Morgan Freeman as he reminisces about alien superintelligence in the movie Oblivion
Early Beta — Under active development. Expect rough edges.
To install or get started, either download from the website over at tinyhumans.ai/openhuman or run
# For MacOS/Linux
curl -fsSL https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.sh | bash
# For Windows
irm https://raw.githubusercontent.com/tinyhumansai/openhuman/main/scripts/install.ps1 | iex
What is OpenHuman?
OpenHuman is an open-source agentic assistant that is designed to integrate with you in your daily life. Here's what makes OpenHuman special:
-
Simple, UI-first — A clean desktop experience and short onboarding paths so you can go from install to a working agent in a few clicks, without a config-first setup. You don't need a terminal to run OpenHuman.
-
One subscription, many providers — You only need one account to get access to many agentic APIs (AI Models, Search, Webhooks/Tunnels and other 3rd party APIs etc..), simplifying the experience to get a powerful agent going.
-
Rich Skills — Plug into Gmail, Slack, Notion, and the rest of your stack via rich, feature-backed skills. Connections are typically one click through setup wizards instead of wiring APIs by hand. Workflow data is kept on device, encrypted locally, and treated as yours: encryption and sensitive context stay on your machine. Webhooks give instant feedback into the agent when external systems or skills emit events, so the loop stays tight without constant polling.
-
Local knowledge base — Built from your data and your activity. How you work across tools, sessions, and connected services—so the agent gets rich, workflow-aware context, not a one-off chat transcript. Everything is stored on your machine and compounding over time without becoming a cloud dossier. Channels, skills and ongoing conversations feed the same loop so day-to-day context does not reset every session.
-
Local AI model — The Rust core exposes local AI paths (and the desktop bundle can ship local/bundled runners where applicable) for the workloads above—vision snippets, speech helpers, summarization, tooling—so sensitive steps can stay off the cloud when you choose.
-
Deep desktop integrations — OpenHuman is a native desktop assistant, not a web-only chat: memory-aware keyboard autocomplete, voice (STT listening and TTS replies), screen intelligence that understands what is on screen and feeds your local context, plus windowing and OS-level permissions—so the agent meets you on the machine, not trapped in a browser tab.
Architecture: docs/ARCHITECTURE.md. Contributor orientation: CONTRIBUTING.md.
OpenHuman vs other agents
High-level comparison (products evolve—verify against each vendor). OpenHuman is built to minimize vendor sprawl, keep workflow knowledge on-device, and ship deep desktop features—not only chat.
| Claude Code/Cowork | OpenClaw | Hermes Agent | OpenHuman | |
|---|---|---|---|---|
| Open-source: Is the codebase open to review? | 🚫 Proprietary client | ✅ MIT License | ✅ MIT License | ✅ GNU License |
| Simple: Is it simple to get started? | ✅ Simple Desktop App + CLI | ⚠️ Terminal first and often complex | ⚠️ Terminal first and often complex | ✅ Simple, Clean UI/UX. Get started within minutes |
| Cost: How expensive is to run? | ⚠️ Subscription + add-on tool/API costs | ⚠️ Tied to models & hosting you choose | ⚠️ Tied to models & hosting you choose | ✅ Cost optimized with the option to run many things locally for free |
| Memory & Knowledge Base (KB): Does the agent know you and your world? | ✅ Built-in memory; mostly chat/session scoped | ⚠️ Has a local memory but often needs plugins for richer behavior | ✅ Self-learning / task loops (typical) | 🚀 Local KB + Self-learning from your activity & data (GMail, Notion etc... via skills) & prompts |
| API spagetti: How complex is it to hook mulitple features together? | 🚫 Claude bill + often extra keys for MCP/tools | 🚫 BYOK / multi-vendor common | 🚫 Multiple providers common | ✅ One account get access to many bundled platform APIs |
| Extensibility: Can you add rich features into it? | ✅ MCP (different model than sandboxed skills) | ✅ Plugin Architecture (SKILL.md) | ✅ Plugin Architecture (SKILL.md) | 🚀 Rich Skills with ability to have realtime updates, local DB & more |
| Desktop integrations: Can it integrate into your desktop completely? | ⚠️ Desktop app & access to folders | ⚠️ Often lighter native surface | ⚠️ Often lighter native surface | ✅ STT, TTS, screen intelligence, memory-aware autocomplete and a whole lot more |
Contributors Hall of Fame
Show some love and end up in the hall of fame