mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
Feat/docs v6 (#1387)
This commit is contained in:
Steven Enamakel
@@ -63,6 +63,9 @@ OpenHuman is the first agent harness that gets to know in minutes. Insipired by
|
||||
<img src="./gitbooks/.gitbook/assets/image.png" alt="The Tet" />
|
||||
</p>
|
||||
|
||||
> OpenHuman summarizes and compress all your documents, emails & chats; and creates a memory graph that lets your agent remember everything about you.
|
||||
|
||||
|
||||
Most agents start cold. Hermes learns by watching you work; OpenClaw waits for plugins to ferry context in. Either way, you spend days or weeks before the agent knows enough about your stack to be genuinely useful.
|
||||
|
||||
OpenHuman skips the wait. Connect your accounts, let [auto-fetch](https://tinyhumans.gitbook.io/openhuman/features/integrations/auto-fetch) pull from every integration on a 20-minute loop, and the [Memory Tree](https://tinyhumans.gitbook.io/openhuman/features/memory-tree) canonicalizes everything into ≤3k-token Markdown chunks scored and folded into hierarchical summary trees: a Karpathy-style [Obsidian wiki](https://tinyhumans.gitbook.io/openhuman/features/obsidian-wiki) on disk, queryable SQLite under the hood. One sync pass and the agent has read your inbox, your calendar, your repos, your docs. No training period. No "give it a few weeks."
|
||||
|
||||
+7
-3
@@ -19,11 +19,15 @@ OpenHuman solves this with a stack that's calmly, deliberately different:
|
||||
|
||||
**An** [**Obsidian-style wiki**](features/obsidian-wiki/) **on top of it.** The same chunks the agent reasons over land as `.md` files in a vault you can open in [Obsidian](https://obsidian.md), browse, edit, and link by hand. Inspired by [Karpathy's obsidian-wiki workflow](https://x.com/karpathy/status/2039805659525644595). You can't trust a memory you can't read.
|
||||
|
||||
[**118+ third-party integrations**](features/integrations.md)**, with** [**auto-fetch**](features/obsidian-wiki/auto-fetch.md)**.** One-click OAuth into Gmail, GitHub, Slack, Notion, Stripe, Calendar, Drive, Linear, Jira and more. Every twenty minutes, OpenHuman pulls fresh data from every active connection and folds it into the Memory Tree without you asking.
|
||||
[**118+ third-party integrations**](features/integrations.md)**.** One-click OAuth into Gmail, GitHub, Slack, Notion, Stripe, Calendar, Drive, Linear, Jira and more — no API keys to wire by hand, no plugin marketplace to navigate.
|
||||
|
||||
**An agent built for big data.** [Smart token compression (TokenJuice)](features/token-compression.md) compacts verbose tool output before it ever enters the model's context, so sweeping through your last six months of email costs single-digit dollars. [Automatic model routing](features/model-routing/) sends each task to the right model, `hint:reasoning` to a frontier model, `hint:fast` to a cheap one, vision to vision, all under one subscription. Built-in [native tools](features/native-tools.md) cover web search, scraping, and a full filesystem/git/lint/test/grep coder loop.
|
||||
[**Auto-fetch**](features/obsidian-wiki/auto-fetch.md)**.** Every twenty minutes, OpenHuman pulls fresh data from every active connection and folds it into the Memory Tree without you asking, so the agent already has tomorrow's context this morning.
|
||||
|
||||
[**Native voice (ElevenLabs)**](features/voice.md)**.** STT in, ElevenLabs TTS out, mascot lip-sync, and a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call.
|
||||
**An agent built for big data.** [Smart token compression (TokenJuice)](features/token-compression.md) compacts verbose tool output before it ever enters the model's context, so sweeping through your last six months of email costs single-digit dollars. [Automatic model routing](features/model-routing/) sends each task to the right model — `hint:reasoning` to a frontier model, `hint:fast` to a cheap one, vision to vision — all under one subscription. Optional [local AI via Ollama](features/model-routing/local-ai.md) keeps embeddings and summarization on-device.
|
||||
|
||||
[**Batteries included**](features/native-tools/README.md)**.** A complete agent toolbelt is wired in by default: [web search](features/native-tools/web-search.md), a [web-fetch scraper](features/native-tools/web-scraper.md), a full [coder toolset](features/native-tools/coder.md) (filesystem, git, lint, test, grep), [browser & computer control](features/native-tools/browser-and-computer.md), [cron & scheduling](features/native-tools/cron.md), [memory tools](features/native-tools/memory-tools.md), [agent coordination](features/native-tools/agent-coordination.md) for spawning sub-agents, and [native voice](features/native-tools/voice.md) — STT in, TTS out, mascot lip-sync, and a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call. No "install a plugin to read files" friction.
|
||||
|
||||
**Simple, UI-first.** A clean desktop experience and short onboarding paths take you from install to a working agent in a few clicks — no config-first setup, no terminal required. The agent has [a face](features/mascot.md): a desktop mascot that speaks, reacts to its surroundings, joins your Google Meets as a real participant, remembers you across weeks, and keeps thinking in the background even when you've stopped typing.
|
||||
|
||||
Together, these turn OpenHuman into something fundamentally different from a chatbot. It is an AI agent that consumes large amounts of personal data at low cost, maintains a persistent and evolving understanding of your world, and takes proactive actions on your behalf.
|
||||
|
||||
|
||||
+13
-3
@@ -10,14 +10,23 @@
|
||||
|
||||
* [Obsidian-Style Memory](features/obsidian-wiki/README.md)
|
||||
* [Memory Tree](features/obsidian-wiki/memory-tree.md)
|
||||
* [Memory Tree Pipeline](features/obsidian-wiki/memory-tree-pipeline.md)
|
||||
* [Auto-fetch from Integrations](features/obsidian-wiki/auto-fetch.md)
|
||||
* [Third-party Integrations (118+)](features/integrations.md)
|
||||
* [Smart Token Compression](features/token-compression.md)
|
||||
* [Automatic Model Routing](features/model-routing/README.md)
|
||||
* [Local AI (optional)](features/model-routing/local-ai.md)
|
||||
* [Native Voice (ElevenLabs)](features/voice.md)
|
||||
* [Native Tools (search, scraper, coder)](features/native-tools.md)
|
||||
* [Native Tools](features/native-tools/README.md)
|
||||
* [Web Search](features/native-tools/web-search.md)
|
||||
* [Web Scraper](features/native-tools/web-scraper.md)
|
||||
* [Coder](features/native-tools/coder.md)
|
||||
* [Browser & Computer Control](features/native-tools/browser-and-computer.md)
|
||||
* [Cron & Scheduling](features/native-tools/cron.md)
|
||||
* [Voice](features/native-tools/voice.md)
|
||||
* [Memory Tools](features/native-tools/memory-tools.md)
|
||||
* [Third-party Integrations](features/native-tools/integrations.md)
|
||||
* [Agent Coordination](features/native-tools/agent-coordination.md)
|
||||
* [System & Utilities](features/native-tools/system-and-utilities.md)
|
||||
* [The Mascot](features/mascot.md)
|
||||
* [Subconscious Loop](features/subconscious.md)
|
||||
* [Privacy & Security](features/privacy-and-security.md)
|
||||
* [Platform & Availability](features/platform.md)
|
||||
@@ -36,6 +45,7 @@
|
||||
* [Architecture](developing/architecture/README.md)
|
||||
* [Frontend (app/src/)](developing/architecture/frontend.md)
|
||||
* [Tauri Shell (app/src-tauri/)](developing/architecture/tauri-shell.md)
|
||||
* [Agent Harness](developing/architecture/agent-harness.md)
|
||||
|
||||
## Legal
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ OpenHuman is a **React + Tauri v2 desktop app** with a **Rust core** that does t
|
||||
│ • Provider router (model routing) │
|
||||
│ • TokenJuice compression │
|
||||
│ • Native tools (search, fetch, fs, git, …) │
|
||||
│ • Voice (STT, ElevenLabs TTS, Meet agent) │
|
||||
│ • Voice (STT in, TTS out, Meet agent) │
|
||||
└──────────────────────────────────────────────────┘
|
||||
│
|
||||
┌──────────────────────────────────────────────────┐
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
---
|
||||
description: >-
|
||||
How an agent turn actually runs — the tool-call loop, sub-agent dispatch,
|
||||
archetypes, triage, hooks, and the cost/budget machinery around them.
|
||||
icon: layer-group
|
||||
---
|
||||
|
||||
# Agent Harness
|
||||
|
||||
The agent harness is the runtime that turns a user message (or a webhook fire, or a cron tick) into a complete, tool-using LLM interaction. It owns the tool-call loop, sub-agent dispatch, the trigger-triage pipeline, and the hook surface around them. It does **not** own provider HTTP transport, tool implementations, prompt-section assembly, or memory storage — those are separate domains the harness composes.
|
||||
|
||||
This page walks through what happens in one turn, then zooms in on each of the moving parts.
|
||||
|
||||
## The shape of a turn
|
||||
|
||||
Every turn — whether the user just typed a message, a Telegram webhook just fired, or a 9am cron just ticked — flows through the same lifecycle:
|
||||
|
||||
```
|
||||
┌─ inbound ─────────────────────────────────────────────────────────┐
|
||||
│ user message · channel inbound · webhook · cron · composio event │
|
||||
└──────────────────────────┬────────────────────────────────────────┘
|
||||
│
|
||||
▼ (external triggers only)
|
||||
┌──────────────────────┐
|
||||
│ trigger triage │ classify → drop / notify /
|
||||
│ (small local LLM) │ spawn reactor / spawn orchestrator
|
||||
└──────────┬───────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────┐
|
||||
│ Agent::turn() │
|
||||
│ 1. resume transcript │
|
||||
│ 2. build system prompt* │
|
||||
│ 3. inject memory context │
|
||||
│ 4. enter tool-call loop ────┼──► provider call
|
||||
│ 5. dispatch tool calls ────┼──► tool exec / sub-agent spawn
|
||||
│ 6. context guard / compact │
|
||||
│ 7. stop-hook check │
|
||||
│ 8. final assistant text │
|
||||
└──────────┬───────────────────┘
|
||||
│ async, after the user sees the reply
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ post-turn │ archivist · learning · cost log ·
|
||||
│ hooks │ episodic memory indexing
|
||||
└─────────────────┘
|
||||
|
||||
* system prompt is built only on the first turn — subsequent
|
||||
turns reuse the rendered prompt verbatim so the inference
|
||||
backend's KV-cache prefix stays valid.
|
||||
```
|
||||
|
||||
The rest of this page is the same diagram, expanded.
|
||||
|
||||
## Sessions and `Agent::turn`
|
||||
|
||||
A **session** is the live conversation an `Agent` instance is running. The `Agent` struct owns:
|
||||
|
||||
* The conversation history (system + user + assistant + tool messages).
|
||||
* The provider client to call (model resolved by the [model router](../../features/model-routing/)).
|
||||
* The tool registry visible to the model.
|
||||
* A memory loader that hydrates relevant memories before each user message.
|
||||
* Per-turn budgets — max tool iterations, max payload size, max USD cost.
|
||||
|
||||
`Agent::turn(user_message)` is the hot path. In one turn it:
|
||||
|
||||
1. **Resumes the session transcript** if this is a fresh process — re-loading the exact provider messages from disk so the inference backend's KV-cache prefix still hits.
|
||||
2. **Builds the system prompt** (only on the first turn). This pulls in identity, soul, profile, memory, connected integrations, available tools, safety preamble — assembled by the prompt section builder.
|
||||
3. **Injects memory context** for the new user message via the memory loader: relevant chunks from the [Memory Tree](../../features/obsidian-wiki/memory-tree.md), with citations attached so the UI can show provenance.
|
||||
4. **Enters the tool-call loop** (next section).
|
||||
5. **Spawns post-turn hooks** in the background — the user gets their answer before archivist / learning / cost logging finishes.
|
||||
|
||||
The system prompt is **not** rebuilt on subsequent turns. Even cosmetic byte changes invalidate the KV-cache prefix and force a full re-prefill, so dynamic per-turn context (memory recall, freshly-learned snippets) is appended as user-visible message content rather than spliced into the system prompt.
|
||||
|
||||
## The tool-call loop
|
||||
|
||||
Inside `Agent::turn`, the tool-call loop is the inner engine. It runs up to `max_tool_iterations` rounds (default 10):
|
||||
|
||||
```
|
||||
loop {
|
||||
1. context guard — if history is too big, microcompact / autocompact
|
||||
2. stop-hook check — budget caps, max-iterations, custom kill switches
|
||||
3. provider call — send messages + tool specs, stream the response
|
||||
4. parse response — split assistant text from tool calls
|
||||
5. if no tool calls — return final text
|
||||
6. execute tool calls — dispatch each one (next section)
|
||||
7. summarize oversize — route huge tool outputs through the summarizer agent
|
||||
8. append results — push tool results into history, loop again
|
||||
}
|
||||
```
|
||||
|
||||
Every iteration emits a real-time `AgentProgress` event so the UI can render token-by-token streaming, "calling tool X" status, and per-iteration cost updates.
|
||||
|
||||
### Tool dispatch and tool-call dialects
|
||||
|
||||
Different LLMs speak different tool-calling dialects. The harness abstracts that with a `ToolDispatcher` trait, which has three concrete implementations:
|
||||
|
||||
* **Native** — providers with first-class tool-calling APIs (Anthropic, OpenAI). Tool calls come back as structured fields, not in the text body.
|
||||
* **XML** — fallback for models that aren't natively trained for tool-calling but can follow instructions. Tools are wrapped in `<tool_call>{...}</tool_call>` tags in the assistant text.
|
||||
* **P-Format** — a compact text format used by some smaller models.
|
||||
|
||||
The dispatcher is selected per provider, which keeps the loop itself dialect-agnostic. The same loop code drives Claude, GPT, Gemini, and a local Ollama model.
|
||||
|
||||
### Context management mid-loop
|
||||
|
||||
Long tool-calling chains can blow past the context window. Two layers handle that:
|
||||
|
||||
* **Tool-result budget** — every tool result is checked against a per-call byte budget. Anything over is hard-truncated with an explanatory marker so the model knows it didn't see the full output.
|
||||
* **Microcompact / autocompact** — when total history is creeping toward the context window, the harness compacts older turns into summaries before the next provider call. The compacted history keeps the system prompt and the most recent turns intact (KV-cache stability) and rewrites the middle.
|
||||
|
||||
### Oversized tool results — the summarizer detour
|
||||
|
||||
Some tool calls return enormous payloads — a Composio action dumping 200 KB of JSON, a web scrape returning 50 KB of markdown, a `file_read` over a multi-thousand-line log. Hard-truncating mid-payload drops whatever happens to land past the cut.
|
||||
|
||||
When a tool result exceeds the summarizer's threshold, it gets routed through a dedicated `summarizer` sub-agent before entering the parent's history. The summarizer compresses the payload per an extraction contract that preserves identifiers and key facts, and the parent agent only sees the compressed summary. Hard truncation remains the backstop downstream when summarization fails or the payload is so absurdly large that paying for an LLM call on it makes no economic sense.
|
||||
|
||||
### Self-healing for missing commands
|
||||
|
||||
When the code-executor sub-agent runs a shell command and the runtime answers "command not found", a self-healing interceptor catches the error, spawns a `ToolMaker` sub-agent to write a polyfill script for the missing command, and retries the original call. There's a per-command attempt cap so a genuinely impossible command can't loop forever.
|
||||
|
||||
## Sub-agents — the orchestrator pattern
|
||||
|
||||
OpenHuman is **multi-agent**. The agent the user is chatting with is the **Orchestrator** — a senior, strategy-level agent that decides when to answer directly, when to use a direct tool, and when to spawn a specialist sub-agent.
|
||||
|
||||
### Why multi-agent
|
||||
|
||||
A single agent that knows everything also has a system prompt the size of a small book. Splitting work across specialists means:
|
||||
|
||||
* Each sub-agent gets a **narrow system prompt** with only the sections it needs (identity / memory / safety preamble can be stripped).
|
||||
* Each sub-agent gets a **filtered tool registry** — the integrations agent doesn't need filesystem tools, the coder doesn't need the Composio catalog.
|
||||
* Sub-agent histories never leak back to the parent — the parent sees one compact tool result, not the inner conversation.
|
||||
* Cheaper models can do the leaf work. The orchestrator is on a strong reasoning model; a research sub-agent might be on a faster, cheaper one.
|
||||
|
||||
### The built-in archetypes
|
||||
|
||||
Each archetype lives under `agents/<name>/` with an `agent.toml` (metadata, tool scope, model hint) and a prompt:
|
||||
|
||||
| Archetype | When the orchestrator picks it |
|
||||
| ------------------- | --------------------------------------------------------------------------------------- |
|
||||
| `orchestrator` | The top-level agent. Never spawned by another orchestrator. |
|
||||
| `planner` | Multi-step decomposition — break a complex request into ordered sub-tasks. |
|
||||
| `researcher` | Web/doc lookups, citation hunting. |
|
||||
| `code_executor` | Writing, running, and debugging code in the workspace. |
|
||||
| `critic` | Code review, quality checks on another agent's output. |
|
||||
| `summarizer` | Compressing oversized tool results (called by the harness, not usually the model). |
|
||||
| `archivist` | Memory distillation — what to persist, what to forget. |
|
||||
| `tool_maker` | Self-healing — writes polyfills for missing shell commands. |
|
||||
| `tools_agent` | Generic specialist for arbitrary tool-bound tasks. |
|
||||
| `integrations_agent`| Bound to a specific Composio toolkit (Gmail, GitHub, Slack…) for that toolkit's actions.|
|
||||
| `trigger_triage` | Classifies incoming external events into drop / notify / spawn-reactor / spawn-agent. |
|
||||
| `trigger_reactor` | Lightweight reaction to a triaged trigger that doesn't need a full orchestrator turn. |
|
||||
| `morning_briefing` | Curated daily digest run by cron. |
|
||||
| `welcome` / `help` | Onboarding flows. |
|
||||
|
||||
Custom archetypes ship as TOML files under `$OPENHUMAN_WORKSPACE/agents/*.toml` (or `~/.openhuman/agents/*.toml` for user-global specialists). Custom definitions override built-ins on id collision.
|
||||
|
||||
### Running a sub-agent
|
||||
|
||||
When the orchestrator calls `spawn_subagent` (or one of the `delegate_*` convenience tools), the runner:
|
||||
|
||||
1. Reads the parent's execution context from a task-local — the parent's provider, sandbox mode, cancellation fence, transcript root.
|
||||
2. Resolves the sub-agent's model — inherit from parent, follow a hint (`fast` / `reasoning` / `summarization`), or pin an exact model.
|
||||
3. Filters the parent's tool registry per the definition's `tools`, `disallowed_tools`, and `skill_filter`. In `fork` mode, the parent's full registry is inherited verbatim.
|
||||
4. Builds a narrow system prompt, omitting the sections the definition asks to strip.
|
||||
5. Runs an inner tool-call loop using the same machinery as the parent.
|
||||
6. Returns one compact text result. The intra-sub-agent history is never spliced back into the parent — the orchestrator sees a single tool result and moves on.
|
||||
|
||||
For tasks that don't need to block the orchestrator's turn, `spawn_worker_thread` runs the sub-agent in the background and the orchestrator continues immediately.
|
||||
|
||||
### Toolkit-specific specialists
|
||||
|
||||
For Composio toolkits with hundreds of actions (GitHub alone has 500+), loading every action into the sub-agent's tool set balloons prompt size. The harness ranks the toolkit's actions against the parent-refined task prompt with a cheap CPU-only filter (verb detection, token overlap, verb-alignment boost) and only loads the top-ranked subset into the sub-agent. No model call, pure heuristic — fast and explainable.
|
||||
|
||||
## Triage — handling external triggers
|
||||
|
||||
When a webhook fires, a cron ticks, or a Composio event arrives, the system can't just hand it straight to the orchestrator. Most triggers are noise; some warrant a notification; only a few deserve a full agent turn. The **trigger-triage pipeline** is the gate.
|
||||
|
||||
```
|
||||
TriggerEnvelope ──► run_triage ──► TriageDecision ──► apply_decision
|
||||
│ │
|
||||
│ ├─► drop (noise)
|
||||
│ ├─► notify only
|
||||
│ ├─► spawn trigger_reactor
|
||||
│ └─► spawn orchestrator
|
||||
│
|
||||
└── small local LLM (with cloud-LLM retry fallback)
|
||||
```
|
||||
|
||||
The evaluator is intentionally cheap — a small local model where available, falling back to a remote model on retry. The decision is cached so identical triggers don't re-classify. Only triggers that escalate to "spawn orchestrator" go through the full `Agent::turn` machinery.
|
||||
|
||||
## Hooks — observability and policy levers
|
||||
|
||||
Two hook surfaces wrap the loop, on opposite ends:
|
||||
|
||||
### Stop hooks (mid-turn)
|
||||
|
||||
Stop hooks fire **between** iterations of the tool-call loop. They're the policy lever for budget caps, rate limits, and custom kill switches. Built-in hooks:
|
||||
|
||||
* **Budget stop hook** — caps cumulative turn cost in USD using the per-iteration cost accumulator.
|
||||
* **Max-iterations stop hook** — caps iteration count from outside the agent's persistent config.
|
||||
|
||||
A hook returning `Stop` aborts the loop with a clear reason the caller can surface to the user. Stop hooks are distinct from interrupts (next section): they're policy-driven, not user-driven.
|
||||
|
||||
### Post-turn hooks
|
||||
|
||||
Post-turn hooks fire **after** the turn completes, in the background. They get a `TurnContext` snapshot — user message, assistant response, every tool call with arguments and outcome, total wall-clock, iteration count, session ID. Built-in consumers:
|
||||
|
||||
* **Archivist** — distills which facts from the turn are worth persisting to long-term memory.
|
||||
* **Learning** — feeds reflection, tool-tracker, and user-profile updates.
|
||||
* **Cost log** — final per-turn cost line.
|
||||
* **Episodic memory indexing** — writes the turn into the [Memory Tree](../../features/obsidian-wiki/memory-tree.md) as a chunk for future recall.
|
||||
|
||||
Hooks run via `tokio::spawn`, so the user gets their answer before any of them finish.
|
||||
|
||||
## Interrupts — graceful cancellation
|
||||
|
||||
An `InterruptFence` is checked at fixed safe points in the loop — before each tool execution, before each sub-agent spawn, before each provider call. When the user hits Ctrl+C or sends `/stop`:
|
||||
|
||||
* The fence flips.
|
||||
* Every running sub-agent sees the same flag (it's shared via `Arc`) and bails at its next checkpoint.
|
||||
* In-flight provider streams are dropped.
|
||||
* The archivist still fires with whatever partial context exists, so the conversation isn't lost.
|
||||
|
||||
Interrupts are user-driven; stop hooks are policy-driven. They share the underlying "halt the loop cleanly" plumbing but enter from different sides.
|
||||
|
||||
## Cost accounting
|
||||
|
||||
Every provider response carries a `UsageInfo` block — input tokens, output tokens, cached input tokens, and an authoritative `charged_amount_usd` populated by the OpenHuman backend. `TurnCost` sums those across every provider call inside one turn so the harness can:
|
||||
|
||||
* Emit per-iteration cost telemetry over the progress channel.
|
||||
* Feed the budget stop hook so a runaway turn cuts itself off mid-loop.
|
||||
* Log accurate end-of-turn cost lines.
|
||||
|
||||
When the backend doesn't surface a charged amount (older builds, providers that don't bill through it), a small per-tier rate table provides a token-rate floor estimate. Direct cost from the backend always wins when available.
|
||||
|
||||
## Fork context — KV-cache reuse across the harness
|
||||
|
||||
The harness uses a task-local `ParentExecutionContext` to thread parent state into sub-agents without exploding every function signature. The same pattern carries the current sandbox mode, the interrupt fence, and the stop-hook list. Sub-agents that inherit the parent's provider, model, and prompt prefix get to **share the parent's KV-cache prefix** on the inference backend — measurably cheaper than re-prefilling from scratch.
|
||||
|
||||
## Self-healing recap
|
||||
|
||||
A few small adaptive systems sit on top of the main loop:
|
||||
|
||||
* **Self-healing for missing commands** — `ToolMaker` polyfills, capped retry attempts.
|
||||
* **Payload summarizer circuit-breaker** — three consecutive sub-agent failures in a session disable summarization, falling back to truncation.
|
||||
* **Triage local-vs-remote retry** — local LLM first; remote fallback on parse failure.
|
||||
|
||||
None of these change the loop's shape — they just make the common failure modes recoverable without the user having to intervene.
|
||||
|
||||
## Where to look in the code
|
||||
|
||||
The harness lives entirely under `src/openhuman/agent/`. The README in that directory enumerates the public surface; the most load-bearing files are:
|
||||
|
||||
| File / dir | What lives there |
|
||||
| ----------------------------- | ----------------------------------------------------------------- |
|
||||
| `harness/session/turn.rs` | `Agent::turn` — the lifecycle described above. |
|
||||
| `harness/tool_loop.rs` | The inner tool-call loop. |
|
||||
| `harness/subagent_runner/` | `run_subagent`, fork-mode, oversized-result handoff. |
|
||||
| `harness/definition.rs` | `AgentDefinition` — what an archetype declares. |
|
||||
| `harness/tool_filter.rs` | Toolkit-action ranking for integrations sub-agents. |
|
||||
| `harness/payload_summarizer.rs` | Oversized-tool-result detour. |
|
||||
| `harness/self_healing.rs` | Missing-command interceptor. |
|
||||
| `harness/interrupt.rs` | The cancellation fence. |
|
||||
| `dispatcher.rs` | Tool-call dialect abstraction. |
|
||||
| `triage/` | External-trigger classification + escalation. |
|
||||
| `agents/` | Built-in archetypes — one subdirectory per agent. |
|
||||
| `hooks.rs` / `stop_hooks.rs` | Post-turn and mid-turn hook surfaces. |
|
||||
| `cost.rs` | Per-turn USD/token accounting. |
|
||||
| `progress.rs` | Real-time progress events to the UI. |
|
||||
| `memory_loader.rs` | Memory-Tree context injection per user message. |
|
||||
|
||||
## See also
|
||||
|
||||
* [Architecture overview](README.md) — where the harness sits in the bigger picture.
|
||||
* [Memory Tree](../../features/obsidian-wiki/memory-tree.md) — what the memory loader reads from and post-turn hooks write to.
|
||||
* [Automatic Model Routing](../../features/model-routing/) — how `model: "hint:reasoning"` resolves to a concrete provider+model.
|
||||
* [Native Tools — Agent Coordination](../../features/native-tools/agent-coordination.md) — the user-facing surface for `spawn_subagent`, `delegate_*`, `todo_write`.
|
||||
@@ -111,7 +111,7 @@ The CDP surface is general-purpose. Today it powers memory ingest from a fixed l
|
||||
|
||||
### Browser automation as a first-class agent tool
|
||||
|
||||
Today the agent has [native tools](../features/native-tools.md) for filesystem, git, web search, and web fetch. The next obvious tool is **"drive a real browser session"**: log into a SaaS the user is already authed in, fill a form, scrape a paginated table, download an export.
|
||||
Today the agent has [native tools](../features/native-tools/README.md) for filesystem, git, web search, and web fetch. The next obvious tool is **"drive a real browser session"**: log into a SaaS the user is already authed in, fill a form, scrape a paginated table, download an export.
|
||||
|
||||
The plumbing is already there. A `@openhuman/browser_task` skill could spin up a dedicated CEF webview, drive it via CDP from the core, and surface the result as a tool call. The user's existing per-account profiles mean no re-auth.
|
||||
|
||||
|
||||
@@ -66,8 +66,8 @@ Beyond third-party services, OpenHuman has **skills**, small sandboxed modules t
|
||||
|
||||
Two capabilities ship native rather than as integrations because they're load-bearing for the desktop experience:
|
||||
|
||||
* [**Voice**](voice.md). STT in, ElevenLabs TTS out, plus a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call.
|
||||
* [**Native tools**](native-tools.md). built-in web search, web-fetch scraper, and a full filesystem/git/lint/test/grep coder toolset that the agent uses out of the box.
|
||||
* [**Voice**](native-tools/voice.md). STT in, TTS out, plus a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call.
|
||||
* [**Native tools**](native-tools/README.md). built-in web search, web-fetch scraper, and a full filesystem/git/lint/test/grep coder toolset that the agent uses out of the box.
|
||||
|
||||
## Privacy boundary
|
||||
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
description: >-
|
||||
The on-screen face of OpenHuman — a desktop mascot that speaks, reacts,
|
||||
joins your meetings, and thinks in the background even when you aren't
|
||||
looking at it.
|
||||
icon: face-smile
|
||||
---
|
||||
|
||||
# The Mascot
|
||||
|
||||
OpenHuman has a face. The mascot is an animated character that lives on your desktop and acts as the visible surface of the agent — what it's saying, what it's thinking about, when it's idle, when it's busy, when it has something to tell you.
|
||||
|
||||
It is not a chrome ornament. The mascot is wired into the same pieces as the rest of the agent: voice, memory, the [subconscious loop](subconscious.md), and the [Google Meet integration](native-tools/voice.md). When the agent talks, the mascot is the one talking; when the agent is thinking, the mascot is the one thinking.
|
||||
|
||||
## What it does
|
||||
|
||||
### It speaks, and lip-syncs to its own voice
|
||||
|
||||
When the agent replies, the audio is generated through a hosted TTS model and streamed to your speakers. At the same time, the mascot drives a viseme map against the audio so its mouth shapes match the words coming out. There's no separate "talking head" video — the same audio stream that you hear is the one driving the animation.
|
||||
|
||||
See [Native Voice](native-tools/voice.md) for the speech-to-text, text-to-speech, and meeting plumbing the mascot rides on top of.
|
||||
|
||||
### It joins your Google Meet calls — as a real participant
|
||||
|
||||
The mascot is OpenHuman's flagship voice integration. When you bring it into a Google Meet:
|
||||
|
||||
* It joins as a real participant via the embedded webview.
|
||||
* It hears everyone in the call through streaming speech-to-text and writes structured notes into the [Memory Tree](obsidian-wiki/memory-tree.md) as the meeting progresses.
|
||||
* When you ask it to speak — or it decides it has something useful to add — it generates the reply through the TTS model and **plays it back into the meeting as an outbound mic stream**, so other participants actually hear it.
|
||||
* The mascot's face is **piped into the meeting as the camera feed** via a fake-video pipeline, so the other participants see the mascot moving and lip-syncing in their grid, not a black tile.
|
||||
|
||||
### It moves and reacts to its surroundings
|
||||
|
||||
The mascot has mood states — idle, thinking, listening, talking, surprised, dreaming — and it transitions between them based on what the agent is doing. When you start typing it shifts into a listening pose. When the model is reasoning, it shows that. When a tool call returns something noteworthy, it reacts. When you stop interacting for a while, it drifts into idle.
|
||||
|
||||
It is meant to feel alive, not animated-on-rails.
|
||||
|
||||
### It remembers you
|
||||
|
||||
The mascot is the visible part of an agent that has the [Memory Tree](obsidian-wiki/memory-tree.md) underneath it. It remembers what you've talked about, who the people in your life are, what's open on your plate, what's been decided, and what's outstanding — across every source you've connected. When it greets you in the morning, it isn't starting from zero.
|
||||
|
||||
That memory is what makes the personality consistent over weeks and months. The mascot you talk to today knows what the mascot you talked to last Tuesday knows.
|
||||
|
||||
### It thinks in the background — the subconscious
|
||||
|
||||
Even when you've stopped typing, the mascot keeps thinking. The [Subconscious Loop](subconscious.md) is a background tick that:
|
||||
|
||||
* Loads your standing tasks and ambient goals.
|
||||
* Reads the current state of your workspace and recent memory.
|
||||
* Decides what to do about each one — execute autonomously, hold, or escalate to you for approval.
|
||||
* Writes the outcome back to an activity log you can audit.
|
||||
|
||||
So when you come back to the desk, the mascot may have already drafted the email, refreshed the dashboard, or queued the question it needs to ask you. The face on the screen is the one that did the work.
|
||||
|
||||
### It dreams
|
||||
|
||||
When you're away long enough, the mascot enters a dreaming state. Dreaming is the agent's offline consolidation pass — distilling the day's chunks into longer-horizon summaries, refreshing topic trees for the entities that have heated up, surfacing patterns that didn't fit any single source. The mascot animates differently while dreaming so you can tell at a glance: it's not idle, it's processing.
|
||||
|
||||
When you come back, the dreams have already been folded into the Memory Tree. The mascot wakes up smarter than it went to sleep.
|
||||
|
||||
## Why a mascot at all
|
||||
|
||||
Most assistants are a blinking text input. That's fine for a tool. It's not fine for something that's meant to be alongside you all day, with persistent memory of your life, taking actions on your behalf.
|
||||
|
||||
The mascot exists because:
|
||||
|
||||
* **Presence beats panels.** A face you can glance at tells you, in one frame, whether the agent is busy, idle, dreaming, or trying to get your attention.
|
||||
* **It makes voice calls feel like a conversation.** A camera feed of an animated character lip-syncing to its own speech is a different experience than a robotic voice with a black tile.
|
||||
* **Personality is a UX surface.** A consistent character on screen is easier to trust, talk to, and forgive when it makes a mistake than a faceless API.
|
||||
|
||||
## See also
|
||||
|
||||
* [Native Voice](native-tools/voice.md) — the STT / TTS / Meet plumbing the mascot rides on.
|
||||
* [Memory Tree](obsidian-wiki/memory-tree.md) — what the mascot remembers, and how.
|
||||
* [Subconscious Loop](subconscious.md) — what it thinks about while you're away.
|
||||
* [Chromium Embedded Framework](../developing/cef.md) — the camera-into-Meet pipeline (developer reference).
|
||||
@@ -59,5 +59,5 @@ Routing isn't a UI dropdown. The agent loop itself emits hints based on what it'
|
||||
## See also
|
||||
|
||||
- [Smart Token Compression](../token-compression.md). what makes large reasoning calls affordable.
|
||||
- [Native Tools](../native-tools.md). different tool calls hint at different routes.
|
||||
- [Native Tools](../native-tools/README.md). different tool calls hint at different routes.
|
||||
- [Local AI (optional)](local-ai.md). lightweight chat hints can run on-device.
|
||||
|
||||
@@ -30,7 +30,7 @@ Each of these is a **per-feature opt-in flag**. Turning on local AI does not sil
|
||||
| **Chat (default)** | Frontier reasoning quality. Routed via the [model router](README.md) under one subscription. |
|
||||
| **Vision** | Same. |
|
||||
| **STT** | Backend-proxied transcription (`src/openhuman/voice/cloud_transcribe.rs`). |
|
||||
| **TTS** | [ElevenLabs](../voice.md) under the hood (`reply_speech.rs`). |
|
||||
| **TTS** | Hosted [text-to-speech](../native-tools/voice.md) under the hood (`reply_speech.rs`). |
|
||||
| **Web search** | Backend proxy (no API key on your machine). |
|
||||
|
||||
For **lightweight or medium chat hints** (`hint:reaction`, `hint:classify`, `hint:format`, `hint:sentiment`, `hint:summarize`, `hint:medium`, `hint:tool_lite`), the [router](README.md) will prefer the local provider when local AI is enabled and Ollama is reachable. Heavy hints (`hint:reasoning`, `hint:agentic`, `hint:coding`) stay cloud.
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
description: >-
|
||||
Built-in web search, web-fetch scraper, and a full filesystem / git / lint /
|
||||
test / grep coder toolset. No "install a plugin" friction.
|
||||
icon: toolbox
|
||||
---
|
||||
|
||||
# Native Tools (search, scraper, coder)
|
||||
|
||||
OpenHuman's agent has the tools you'd expect from a serious coding-and-research assistant out of the box. None of these are MCP servers you have to wire up; they ship inside the core.
|
||||
|
||||
## Native web search
|
||||
|
||||
`src/openhuman/tools/impl/network/web_search.rs`
|
||||
|
||||
A search tool the agent can call directly. Backed by a server-side proxy (Parallel) so you don't carry a search API key. Returns titles, snippets and URLs ready for the agent to follow up on.
|
||||
|
||||
Use it for: research, "what's the latest on X", citation hunting.
|
||||
|
||||
## Native web scraper
|
||||
|
||||
`src/openhuman/tools/impl/network/web_fetch.rs`
|
||||
|
||||
A purpose-built "GET-and-read" tool, separate from generic `http_request` / `curl`:
|
||||
|
||||
* Caps response at 1 MB.
|
||||
* 20-second timeout.
|
||||
* Strips boilerplate; returns clean text the agent can reason over.
|
||||
|
||||
Use it for: reading articles, blog posts, docs pages, GitHub READMEs without the noise.
|
||||
|
||||
## Native coder
|
||||
|
||||
`src/openhuman/tools/impl/filesystem/`
|
||||
|
||||
A complete toolset for working on real codebases:
|
||||
|
||||
| Tool | What it does |
|
||||
| ---------------- | ----------------------------------------------------------------- |
|
||||
| `file_read` | Read a file (with line numbers, like `cat -n`). |
|
||||
| `file_write` | Write a new file. |
|
||||
| `edit_file` | Targeted edits — match-and-replace with strict uniqueness checks. |
|
||||
| `apply_patch` | Apply a unified diff. |
|
||||
| `glob_search` | Find files by glob pattern. |
|
||||
| `grep` | Ripgrep-style search across the tree. |
|
||||
| `git_operations` | Status, diff, log, blame, branch, commit. |
|
||||
| `run_linter` | Run the project's linter. |
|
||||
| `run_tests` | Run the project's test command. |
|
||||
|
||||
Together this is what makes OpenHuman a viable coding partner instead of a chat window that _pretends_ to know the codebase.
|
||||
|
||||
## Why ship them natively
|
||||
|
||||
A plugin-only model means tools live in different processes, behind RPC, with their own auth and packaging stories. That's fine for open-ended extensibility, but for the **core** tools every agent needs (read a file, search the web, edit code), shipping them in-process means:
|
||||
|
||||
* Consistent error handling.
|
||||
* Zero install friction.
|
||||
* All output passes through [TokenJuice](token-compression.md) for free.
|
||||
* Predictable security boundary (filesystem tools respect workspace scoping).
|
||||
|
||||
## See also
|
||||
|
||||
* [Smart Token Compression](token-compression.md). what keeps tool output costs bounded.
|
||||
* [Third-party Integrations](integrations.md). for the long tail of third-party services.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: >-
|
||||
The full toolset OpenHuman's agent has out of the box — research, code,
|
||||
control your machine, schedule jobs, talk back to you, and call into 118+
|
||||
third-party services.
|
||||
icon: toolbox
|
||||
---
|
||||
|
||||
# Native Tools
|
||||
|
||||
OpenHuman's agent doesn't ship empty. Every model behind the agent has a curated set of tools available the moment you install — no plugin marketplace, no API keys to wire up, no MCP servers to register. The whole toolbelt is in the box.
|
||||
|
||||
This page is the index. Each subpage covers one family of tools.
|
||||
|
||||
## Why ship them natively
|
||||
|
||||
A plugin-only model means tools live in different processes, behind RPC, with their own auth and packaging stories. That's fine for open-ended extensibility, but for the **core** tools every agent needs (read a file, search the web, edit code, set a reminder, join a meeting), shipping them in-process means:
|
||||
|
||||
* Consistent error handling.
|
||||
* Zero install friction.
|
||||
* All output passes through [Smart Token Compression](../token-compression.md) for free.
|
||||
* Predictable security boundary — filesystem tools respect workspace scoping, network tools go through the OpenHuman proxy.
|
||||
|
||||
## The toolbelt
|
||||
|
||||
| Family | What it covers |
|
||||
| ------ | -------------- |
|
||||
| [Web Search](web-search.md) | Search the live web without bringing your own API key. |
|
||||
| [Web Scraper](web-scraper.md) | Pull clean text out of any URL — articles, docs, READMEs. |
|
||||
| [Coder](coder.md) | Read/write/edit/patch files, glob, grep, git, lint, test. |
|
||||
| [Browser & Computer Control](browser-and-computer.md) | Open URLs, screenshot, click, type, move the mouse. |
|
||||
| [Cron & Scheduling](cron.md) | Recurring jobs, one-off reminders, scheduled agent runs. |
|
||||
| [Voice](voice.md) | Speech-to-text in, text-to-speech out, live Google Meet agent. |
|
||||
| [Memory Tools](memory-tools.md) | Recall, store, forget, and search the [Memory Tree](../obsidian-wiki/memory-tree.md). |
|
||||
| [Third-party Integrations](integrations.md) | The agent's view of the [118+ connected services](../integrations.md). |
|
||||
| [Agent Coordination](agent-coordination.md) | Spawn subagents, delegate to skills, plan, ask the user. |
|
||||
| [System & Utilities](system-and-utilities.md) | Shell, node, SQL, current time, push notifications, LSP. |
|
||||
|
||||
## See also
|
||||
|
||||
* [Smart Token Compression](../token-compression.md) — what keeps tool output costs bounded.
|
||||
* [Third-party Integrations](../integrations.md) — the user-facing pitch and OAuth flow for the 118+ catalog.
|
||||
* [Privacy & Security](../privacy-and-security.md) — the boundary every tool runs inside.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
description: Tools the agent uses to plan, delegate, and ask for help.
|
||||
icon: sitemap
|
||||
---
|
||||
|
||||
# Agent Coordination
|
||||
|
||||
Beyond doing the work, the agent has tools for *organising* the work — planning multi-step jobs, delegating to specialists, spawning subagents, and pausing to ask the user when something is genuinely ambiguous.
|
||||
|
||||
## Tools in the family
|
||||
|
||||
| Tool | What it does |
|
||||
| ----------------------- | --------------------------------------------------------------------------------------------- |
|
||||
| `todo_write` | Maintain a structured TODO list across a long task. Marked done as work progresses. |
|
||||
| `spawn_subagent` | Spin up a fresh agent with its own context window for a self-contained subtask. |
|
||||
| `spawn_worker_thread` | Background work that doesn't need to block the main conversation. |
|
||||
| `delegate` | Hand a task to a specialist (e.g. an archetype with different prompts/tools/permissions). |
|
||||
| `archetype_delegation` | Route to a named archetype — coder, researcher, planner, etc. |
|
||||
| `skill_delegation` | Hand off to a [skill](../integrations.md#skills) installed in the workspace. |
|
||||
| `ask_clarification` | Pause and ask the user a precise question instead of guessing. |
|
||||
| `plan_exit` | Exit a planning phase and start executing. |
|
||||
| `check_onboarding_status` / `complete_onboarding` | Gate behaviour on whether the user has finished onboarding. |
|
||||
|
||||
## Why these are tools, not implicit behaviour
|
||||
|
||||
Long tasks fall apart when the agent tries to keep everything in one head. Splitting work via TODOs and subagents means:
|
||||
|
||||
* Each subagent gets a clean context — fewer tokens, fewer distractions.
|
||||
* The main thread keeps a high-level view of progress.
|
||||
* Failures in one branch don't poison the rest.
|
||||
|
||||
Asking for clarification is a tool too, on purpose: it makes "I should ask the user" a *visible* decision the agent can be steered toward, not an emergent behaviour.
|
||||
|
||||
## See also
|
||||
|
||||
* [Coder](coder.md) — what a coder-archetype subagent typically uses.
|
||||
* [Subconscious Loop](../subconscious.md) — the always-on background agent thread.
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
description: Open URLs, take screenshots, click, type, and move the mouse — natively.
|
||||
icon: display
|
||||
---
|
||||
|
||||
# Browser & Computer Control
|
||||
|
||||
When the agent needs to *use* your machine the way a person would — open a page, screenshot it, click a button, type a phrase — these tools are how it does it.
|
||||
|
||||
## Browser
|
||||
|
||||
* **Open** a URL in an embedded webview the agent can read back from.
|
||||
* **Screenshot** the current page.
|
||||
* **Inspect** image output and metadata, so the agent can describe what it sees.
|
||||
|
||||
The browser surface runs through CEF (Chromium Embedded Framework) and includes a security layer that scopes what pages can do. See [Chromium Embedded Framework](../../developing/cef.md) for the platform details.
|
||||
|
||||
## Computer (mouse + keyboard)
|
||||
|
||||
* **Mouse** — move, click, drag.
|
||||
* **Keyboard** — type text, send key chords.
|
||||
* **Human path** — moves and clicks follow human-like trajectories rather than teleporting, so they don't trip naive bot detection.
|
||||
|
||||
## What it's good for
|
||||
|
||||
* Driving sites that don't have an API or a [native integration](integrations.md).
|
||||
* Multi-step UI flows where a single screenshot isn't enough.
|
||||
* Automating local apps from inside a chat.
|
||||
|
||||
## See also
|
||||
|
||||
* [Web Scraper](web-scraper.md) — when you only need the article, not the whole page.
|
||||
* [Chromium Embedded Framework](../../developing/cef.md) — the runtime browser layer.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: A complete toolset for working on real codebases — read, write, edit, search, git, lint, test.
|
||||
icon: code
|
||||
---
|
||||
|
||||
# Coder
|
||||
|
||||
The coder family is what makes OpenHuman a viable coding partner instead of a chat window that *pretends* to know the codebase.
|
||||
|
||||
## Tools in the family
|
||||
|
||||
| Tool | What it does |
|
||||
| ---------------- | ----------------------------------------------------------------- |
|
||||
| `file_read` | Read a file (with line numbers, like `cat -n`). |
|
||||
| `file_write` | Write a new file. |
|
||||
| `edit_file` | Targeted edits — match-and-replace with strict uniqueness checks. |
|
||||
| `apply_patch` | Apply a unified diff. |
|
||||
| `glob_search` | Find files by glob pattern. |
|
||||
| `grep` | Ripgrep-style search across the tree. |
|
||||
| `list_files` | Walk a directory tree. |
|
||||
| `read_diff` | Diff between two files or revisions. |
|
||||
| `git_operations` | Status, diff, log, blame, branch, commit. |
|
||||
| `run_linter` | Run the project's linter. |
|
||||
| `run_tests` | Run the project's test command. |
|
||||
| `csv_export` | Export query results as CSV. |
|
||||
|
||||
## Why these are native, not shell-only
|
||||
|
||||
A shell tool plus `cat`/`sed`/`awk` could *technically* do all of this. The native tools exist because:
|
||||
|
||||
* Edits go through a uniqueness check, so the agent can't accidentally clobber the wrong line.
|
||||
* Reads come back with line numbers the agent can refer to in follow-ups.
|
||||
* Git operations parse output into structured data, instead of leaving the agent to scrape porcelain.
|
||||
* Lint and test runs are wired to the project's actual commands, not generic guesses.
|
||||
|
||||
## Workspace scoping
|
||||
|
||||
Filesystem tools respect a workspace boundary — the agent can't read or write outside it without explicit permission. Same boundary the rest of the app uses for `OPENHUMAN_WORKSPACE`.
|
||||
|
||||
## See also
|
||||
|
||||
* [System & Utilities](system-and-utilities.md) — `shell`, `node_exec`, `npm_exec` for the rest of the dev loop.
|
||||
* [Agent Coordination](agent-coordination.md) — `todo_write`, `spawn_subagent` for larger refactors.
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
description: Recurring jobs, one-off reminders, and scheduled agent runs — first-class.
|
||||
icon: clock
|
||||
---
|
||||
|
||||
# Cron & Scheduling
|
||||
|
||||
Scheduling is a first-class capability, not a workaround. The agent can set up recurring jobs ("every weekday at 9am, summarise my inbox"), one-off reminders ("nudge me about this in three hours"), and arbitrary agent runs on a cron schedule.
|
||||
|
||||
## Tools in the family
|
||||
|
||||
| Tool | What it does |
|
||||
| ------------- | ------------------------------------------------------------------ |
|
||||
| `cron_add` | Create a new scheduled job — cron expression + agent prompt. |
|
||||
| `cron_list` | List existing jobs and their next-run times. |
|
||||
| `cron_update` | Edit an existing job — change schedule, prompt, or enabled state. |
|
||||
| `cron_remove` | Delete a job. |
|
||||
| `cron_run` | Run a job once, immediately, regardless of its schedule. |
|
||||
| `cron_runs` | Inspect the recent run history — when, how long, what it produced. |
|
||||
|
||||
There's also a one-shot `schedule` tool in [System & Utilities](system-and-utilities.md) for "do this once at time T" cases that don't need a recurring entry.
|
||||
|
||||
## What it's good for
|
||||
|
||||
* Daily / weekly digests delivered to your messaging channel of choice.
|
||||
* Polling a slow integration that doesn't push events.
|
||||
* Reminders the agent itself owns ("remind me Thursday to follow up with Alice").
|
||||
* Recurring research — "every Monday, check what's new on this topic and write me a brief".
|
||||
|
||||
## How it ties back to the rest
|
||||
|
||||
Every cron run is just a normal agent invocation, so it can use any other tool — search the web, query the [Memory Tree](../obsidian-wiki/memory-tree.md), call a [third-party integration](integrations.md), send a message. Run history is recorded so you can see what each tick produced.
|
||||
|
||||
## See also
|
||||
|
||||
* [System & Utilities](system-and-utilities.md) — the one-shot `schedule` tool.
|
||||
* [Agent Coordination](agent-coordination.md) — for jobs that fan out into subagents.
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
description: The agent's view of the 118+ connected third-party services.
|
||||
icon: plug
|
||||
---
|
||||
|
||||
# Third-party Integrations
|
||||
|
||||
OpenHuman's agent can call into [118+ third-party services](../integrations.md) — Gmail, Notion, GitHub, Slack, Stripe, Calendar, and the long tail — through a single proxied tool surface.
|
||||
|
||||
## How it shows up to the agent
|
||||
|
||||
Once you've connected a service via OAuth, its actions become callable tools. The agent doesn't need to know whether a tool talks to Gmail or to a local file — it just calls the tool, the proxy routes the request through the OpenHuman backend with your token, and the result comes back like any other tool output.
|
||||
|
||||
A few examples of what becomes available:
|
||||
|
||||
* "Send a message to #engineering on Slack."
|
||||
* "Create an issue in the openhuman repo."
|
||||
* "What's on my calendar tomorrow?"
|
||||
* "Pull the last 20 Stripe charges over $1000."
|
||||
|
||||
## Native vs proxied
|
||||
|
||||
Some services have **native providers** — Rust modules that know how to ingest the service into the [Memory Tree](../obsidian-wiki/memory-tree.md) directly (e.g. Gmail's native ingest path). Others are exposed as **proxied tools** only: the agent can call them, but there's no automatic ingest yet. New native providers are added as features land.
|
||||
|
||||
## Privacy boundary
|
||||
|
||||
OpenHuman's core never calls any third-party API directly. All requests go through the OpenHuman backend, which handles OAuth tokens and rate limiting. Your tokens never sit on disk in plaintext on your machine, and the agent only sees the *results* of tool calls, not the credentials.
|
||||
|
||||
## See also
|
||||
|
||||
* [Third-party Integrations (catalog)](../integrations.md) — the user-facing pitch, OAuth flow, and connection management.
|
||||
* [Auto-fetch](../obsidian-wiki/auto-fetch.md) — how connected services flow into the Memory Tree.
|
||||
* [Privacy & Security](../privacy-and-security.md) — the full boundary.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
description: How the agent reads, writes, and searches its own long-term memory.
|
||||
icon: brain
|
||||
---
|
||||
|
||||
# Memory Tools
|
||||
|
||||
The [Memory Tree](../obsidian-wiki/memory-tree.md) is OpenHuman's knowledge base. The memory tools are how the agent talks to it during a conversation.
|
||||
|
||||
## Tools in the family
|
||||
|
||||
| Tool | What it does |
|
||||
| -------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| `recall` | Search the Memory Tree by query — source-scoped, topic-scoped, or global. Returns chunks with provenance. |
|
||||
| `store` | Write a new memory the agent decided is worth keeping (a fact, a preference, a piece of context). |
|
||||
| `forget` | Remove a memory by ID — used when something is wrong, stale, or the user explicitly asks to forget it. |
|
||||
|
||||
There is also a tree-aware retrieval surface (drill down a topic, fetch the global digest for a day) — the agent picks the right one based on the question.
|
||||
|
||||
## Why these are tools, not implicit context
|
||||
|
||||
The Memory Tree is too big to dump into every conversation. The tools let the model *ask* — "what do I know about Alice?", "what happened yesterday?", "remind me of last week's Stripe webhook" — and the retrieval layer returns just the relevant chunks, with provenance back to the source file in your Obsidian vault.
|
||||
|
||||
## See also
|
||||
|
||||
* [Memory Tree](../obsidian-wiki/memory-tree.md) — what these tools read from and write to.
|
||||
* [Auto-fetch](../obsidian-wiki/auto-fetch.md) — how the tree gets populated in the first place.
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
description: Shell, node, SQL, current time, push notifications — the small tools that round out the toolbelt.
|
||||
icon: gear
|
||||
---
|
||||
|
||||
# System & Utilities
|
||||
|
||||
The catch-all family. Small, sharp tools the agent reaches for to round out a task.
|
||||
|
||||
## Tools in the family
|
||||
|
||||
| Tool | What it does |
|
||||
| ------------------- | ----------------------------------------------------------------------------- |
|
||||
| `shell` | Run a shell command. Bounded output, captured exit code. |
|
||||
| `node_exec` | Run a Node.js snippet — useful for one-off scripting. |
|
||||
| `npm_exec` | Run an `npm`/`pnpm`/`yarn` script. |
|
||||
| `current_time` | Get the current time in any timezone, with formatting options. |
|
||||
| `schedule` | One-shot "do this once at time T" — for recurring jobs see [Cron](cron.md). |
|
||||
| `pushover` | Send a push notification to your devices. |
|
||||
| `insert_sql_record` | Append a row to the agent's structured workspace SQL store. |
|
||||
| `lsp` | Query a language server (definitions, references, diagnostics). |
|
||||
| `workspace_state` | Inspect the current workspace — open files, recent edits, environment. |
|
||||
| `proxy_config` | Read or change proxy configuration for outbound requests. |
|
||||
| `tool_stats` | Self-reflection — what tools have been used in this session and how often. |
|
||||
|
||||
## What it's good for
|
||||
|
||||
* The bits of a workflow that don't fit a richer tool family.
|
||||
* "Just run this command and tell me what it printed."
|
||||
* Time-aware behaviour ("what time is it for the user right now?") without baking timezone assumptions into prompts.
|
||||
* Letting the agent *notify you* when it's done with a long-running job.
|
||||
|
||||
## See also
|
||||
|
||||
* [Coder](coder.md) — for filesystem-heavy work, prefer the dedicated tools over `shell`.
|
||||
* [Cron & Scheduling](cron.md) — for anything recurring.
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
description: >-
|
||||
Native voice — speech-to-text in, text-to-speech out, mascot lip-sync,
|
||||
and a live Google Meet agent that listens and speaks.
|
||||
icon: microphone
|
||||
---
|
||||
|
||||
# Voice
|
||||
|
||||
OpenHuman is voice-first when you want it to be. STT, TTS, and the live Google Meet agent are part of the core, not a third-party plugin.
|
||||
|
||||
## Speech-to-text
|
||||
|
||||
* **Hotkey** — push-to-talk and toggle modes.
|
||||
* **Audio capture** — cross-platform mic capture with voice-activity detection.
|
||||
* **Streaming transcription** — words appear as you speak.
|
||||
* **Hallucination filter** — strips well-known artefacts ("Thanks for watching", silence-induced phrases).
|
||||
* **Postprocess** — punctuation, capitalisation, dictation cleanup.
|
||||
|
||||
Dictation can replace the active text input on your desktop, or be sent straight into a chat with the agent.
|
||||
|
||||
## Text-to-speech
|
||||
|
||||
Reply speech routes through a hosted TTS model. The agent's responses can be spoken back in a voice you pick, with natural timing and prosody. Voice selection is configurable per user, and the mascot avatar lip-syncs to the audio stream via a viseme map.
|
||||
|
||||
## Live Google Meet agent
|
||||
|
||||
OpenHuman's flagship voice integration:
|
||||
|
||||
* Joins a Google Meet via the embedded webview.
|
||||
* Streams audio out to STT in real time, transcribes everyone in the call, and writes structured notes into the [Memory Tree](../obsidian-wiki/memory-tree.md) as the meeting progresses.
|
||||
* When you ask it to speak (or it decides it has something useful to add), it generates audio through the TTS model and **plays it back into the meeting as an outbound camera/mic stream**, so other participants actually hear it.
|
||||
|
||||
## Privacy
|
||||
|
||||
* Audio capture is local. Streaming STT goes through the OpenHuman backend; no recording is retained beyond the live transcript.
|
||||
* TTS audio is streamed and discarded — nothing stored.
|
||||
* Meeting transcripts land in your local memory tree, like any other source.
|
||||
|
||||
## See also
|
||||
|
||||
* [Memory Tree](../obsidian-wiki/memory-tree.md) — where Meet transcripts and notes live.
|
||||
* [Automatic Model Routing](../model-routing/) — Meet's brain uses `hint:fast` for low-latency conversational turns.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
description: A purpose-built "GET-and-read" tool that returns clean text, not raw HTML.
|
||||
icon: globe
|
||||
---
|
||||
|
||||
# Web Scraper
|
||||
|
||||
A purpose-built fetch tool, separate from generic `http_request` / `curl`. It exists because the agent doesn't want raw HTML — it wants the *article*.
|
||||
|
||||
## What it does
|
||||
|
||||
* Fetches a URL.
|
||||
* Strips boilerplate (nav, ads, footer, scripts).
|
||||
* Returns clean text the agent can reason over.
|
||||
|
||||
## Guardrails
|
||||
|
||||
* Caps response at 1 MB — large pages get truncated, not silently dropped.
|
||||
* 20-second timeout — slow servers don't stall the conversation.
|
||||
* Subject to the same proxy and URL-guard rules as other network tools.
|
||||
|
||||
## What it's good for
|
||||
|
||||
* Reading articles, blog posts, docs pages, GitHub READMEs without the noise.
|
||||
* Following up on a [Web Search](web-search.md) result.
|
||||
* Summarising a single page on demand.
|
||||
|
||||
## See also
|
||||
|
||||
* [Web Search](web-search.md) — find URLs to feed into the scraper.
|
||||
* [Smart Token Compression](../token-compression.md) — what trims long pages before they hit the model.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
description: A native search tool the agent can call directly — no API key required.
|
||||
icon: magnifying-glass
|
||||
---
|
||||
|
||||
# Web Search
|
||||
|
||||
The agent can search the live web on its own. Backed by a server-side proxy (Parallel) so you don't carry a search API key, the tool returns titles, snippets, and URLs ready to follow up on.
|
||||
|
||||
## What it's good for
|
||||
|
||||
* Research — "what's the latest on X".
|
||||
* Citation hunting — "find me three sources for Y".
|
||||
* Fact-checking before answering — the agent runs a quick search if it isn't confident.
|
||||
|
||||
## How it differs from generic HTTP
|
||||
|
||||
A pure `http_request` tool can fetch a URL but can't *find* one. Web Search is the discovery layer: it picks the right URLs for the agent, which then hands them off to the [Web Scraper](web-scraper.md) for the actual reading.
|
||||
|
||||
## See also
|
||||
|
||||
* [Web Scraper](web-scraper.md) — fetch and clean a specific URL.
|
||||
* [Smart Token Compression](../token-compression.md) — search snippets are compressed before they hit the model.
|
||||
@@ -1,154 +0,0 @@
|
||||
---
|
||||
description: >-
|
||||
Deep dive into the Memory Tree's async pipeline — how leaves get from "user
|
||||
sent a message" to "global digest summarized for the day".
|
||||
icon: diagram-project
|
||||
---
|
||||
|
||||
# Memory Tree Pipeline
|
||||
|
||||
The user-facing pitch of the [Memory Tree](memory-tree.md) is simple: connect a source, the agent gets persistent memory of it. The pipeline that delivers on that pitch is **not** simple, it spans an HTTP-triggered ingest path, a job queue inside SQLite, a pool of background workers, three independent summary trees, and a daily UTC scheduler. This page walks through the whole thing.
|
||||
|
||||
The diagram below is the source of truth. Every box maps to code under `src/openhuman/memory/tree/`.
|
||||
|
||||
{% file src="../../.gitbook/assets/memory-tree-pipeline (1).excalidraw" %}
|
||||
Memory Tree Async Pipeline, leaf ingestion → jobs queue → workers → source / topic / global tree building.
|
||||
{% endfile %}
|
||||
|
||||
## The six lanes
|
||||
|
||||
The pipeline has six conceptual lanes. Read left to right (1 → 4) for the hot path; bottom row (5 → 6) is independent background flow and the leaf state machine.
|
||||
|
||||
### 1. Ingest
|
||||
|
||||
Entry point: a JSON-RPC call (or in-process equivalent) carrying chat / email / document content.
|
||||
|
||||
```
|
||||
canonicalize
|
||||
→ chunk_markdown
|
||||
→ score_chunks_fast
|
||||
→ upsert_chunks_tx
|
||||
→ lifecycle_status = pending_extraction
|
||||
→ persist fast score rows
|
||||
→ enqueue extract_chunk per chunk
|
||||
→ wake_workers()
|
||||
```
|
||||
|
||||
Hot-path requirements:
|
||||
|
||||
* **Deterministic.** The `chunk_id` is a hash of `(source_kind, source_id, position, body_hash)`. Re-running ingest on identical input never produces duplicates.
|
||||
* **Fast.** No LLM calls in this lane. `score_chunks_fast` uses cheap heuristics; deeper scoring runs out of the worker pool.
|
||||
* **Bounded write.** Everything happens inside one SQLite transaction so a partial ingest can't leave dangling rows.
|
||||
|
||||
Code: `src/openhuman/memory/tree/ingest.rs`, `chunker.rs`, `score/fast.rs`.
|
||||
|
||||
### 2. Queue
|
||||
|
||||
Storage: SQLite at `<workspace>/memory_tree/chunks.db`. Tables:
|
||||
|
||||
| Table | What's there |
|
||||
| ----------------------- | ---------------------------------------------------------------- |
|
||||
| `mem_tree_chunks` | The chunks themselves — body hash, provenance, lifecycle status. |
|
||||
| `mem_tree_score` | Per-chunk score rows (fast + deep). |
|
||||
| `mem_tree_entity_index` | Entity → chunk lookup for topic-tree hotness. |
|
||||
| `mem_tree_jobs` | The job queue (see below). |
|
||||
| `mem_tree_trees` | Per-scope tree metadata (source / topic / global). |
|
||||
| `mem_tree_buffers` | L0 buffers (unsealed leaves) per tree. |
|
||||
| `mem_tree_summaries` | Sealed summaries (L1/L2/...) per tree. |
|
||||
|
||||
`mem_tree_jobs` columns that matter:
|
||||
|
||||
| Column | Purpose |
|
||||
| ------------------------------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| `kind` | One of `extract_chunk`, `append_buffer`, `seal`, `topic_route`, `digest_daily`, `flush_stale`. |
|
||||
| `payload_json` | Job-specific args. |
|
||||
| `dedupe_key` | Coalesces duplicate enqueues — re-running an idempotent job costs one row, not N. |
|
||||
| `status` | `pending` / `running` / `done` / `failed`. |
|
||||
| `attempts` / `last_error` | Retry bookkeeping. |
|
||||
| `available_at_ms` / `locked_until_ms` | Scheduling and worker leasing. |
|
||||
|
||||
Code: `src/openhuman/memory/tree/store.rs`, `jobs/queue.rs`.
|
||||
|
||||
### 3. Workers
|
||||
|
||||
Bootstrap: `jobs::start(workspace_dir)` is called once at process startup. It:
|
||||
|
||||
* Calls `recover_stale_locks()`, any job whose `locked_until_ms` is in the past becomes `pending` again. Crashes don't strand work.
|
||||
* Spawns **3 worker tasks** (configurable, but 3 is the default and what production runs).
|
||||
* Wires a `tokio::sync::Notify` so the ingest path can wake workers immediately, with a **5-second polling fallback** so a missed notify doesn't strand work.
|
||||
* Holds a shared `Semaphore(3)` for LLM-bound steps so concurrent embedding / summarization calls can't blow past the configured budget.
|
||||
|
||||
Each worker pulls a job, runs the right handler, and updates the row. Handlers:
|
||||
|
||||
| Handler | Job kind | What it does |
|
||||
| --------------- | --------------- | -------------------------------------------------------------------------------------------------------- |
|
||||
| `extract_chunk` | `extract_chunk` | Deep score + entity extraction. Decides `admitted` vs `dropped` based on the score. |
|
||||
| `append_buffer` | `append_buffer` | Adds an admitted leaf to the source (or topic) tree's L0 buffer. May trigger a seal. |
|
||||
| `seal` | `seal` | Compresses L0 buffer into an L1 summary; cascades up through L2/L3/... if the parent buffer is now full. |
|
||||
| `topic_route` | `topic_route` | Routes a leaf into per-entity topic trees, gated by the curator hotness check. |
|
||||
| `digest_daily` | `digest_daily` | Builds the global daily digest node. |
|
||||
| `flush_stale` | `flush_stale` | Force-seals buffers that have been sitting too long. |
|
||||
|
||||
Code: `src/openhuman/memory/tree/jobs/{worker.rs, handlers/}`.
|
||||
|
||||
### 4. Tree state
|
||||
|
||||
Three independent trees are built from the same leaf stream.
|
||||
|
||||
**Source tree**, one per source. New leaves land in the L0 buffer. When the buffer fills (or `flush_stale` fires), `seal` writes an L1 summary; if the L1 buffer fills, the cascade continues up.
|
||||
|
||||
**Topic tree**, one per high-hotness entity. The `topic_route` handler runs a curator check (is this entity hot enough to deserve its own tree?) and, if it passes, calls `append_buffer` against the topic's tree.
|
||||
|
||||
**Global tree**, one tree, growing one node per UTC day. The `digest_daily` handler builds yesterday's daily node and `append_daily_and_cascade` walks it up the global hierarchy.
|
||||
|
||||
Code: `src/openhuman/memory/tree/{tree_source, tree_topic, tree_global}/`.
|
||||
|
||||
### 5. Scheduler / background
|
||||
|
||||
A separate scheduler loop runs independently of the ingest path:
|
||||
|
||||
* **UTC daily tick.** At 00:00 UTC each day, enqueue `digest_daily(yesterday)` and `flush_stale(today)`. Both go through the same `mem_tree_jobs` pipeline workers consume.
|
||||
* **`flush_stale`** scans every tree's buffers for ones older than the configured TTL and enqueues force-seal jobs.
|
||||
|
||||
The scheduler **does not** run summarizers itself. Everything goes through the queue, so retries, dedupe, and stale-lock recovery all stay centralized.
|
||||
|
||||
Code: `src/openhuman/memory/tree/jobs/scheduler.rs`.
|
||||
|
||||
### 6. Leaf lifecycle
|
||||
|
||||
Each chunk moves through a small state machine:
|
||||
|
||||
```
|
||||
pending_extraction ──► admitted ──► buffered ──► sealed
|
||||
╲
|
||||
──► dropped
|
||||
```
|
||||
|
||||
* `extract_chunk` decides `admitted` vs `dropped` based on the deep score.
|
||||
* `append_buffer` moves admitted leaves into a buffer, `buffered`.
|
||||
* `seal` writes the buffer's contents into a summary and marks each leaf `sealed`.
|
||||
* `dropped` leaves stop here. Their chunk row stays for provenance, but no buffer / summary references them.
|
||||
|
||||
This is why retrieval can show provenance without re-running the pipeline: the chunk row plus its terminal lifecycle status is enough.
|
||||
|
||||
## Why a queue instead of in-process futures
|
||||
|
||||
Three reasons:
|
||||
|
||||
1. **Crash safety.** A worker panic, a process kill, a power loss, none of them lose admitted-but-not-yet-sealed work. The job row is durable in SQLite; the next start picks it up.
|
||||
2. **Retries with backoff.** `attempts` + `available_at_ms` + `last_error` give us per-job retry without ad-hoc retry loops in business logic.
|
||||
3. **One throttle for LLM cost.** All summarization paths share a single semaphore, so a burst of new sources can't accidentally fan out to 50 concurrent embeddings calls.
|
||||
|
||||
## Touching this code
|
||||
|
||||
A few rules to keep the pipeline coherent:
|
||||
|
||||
* **All paths go through `mem_tree_jobs`.** Don't add a feature that does its own background scheduling. If you need a periodic step, enqueue a job; if you need a one-shot async step, enqueue a job.
|
||||
* **Idempotent handlers.** Every handler must be safe to run twice on the same `(kind, payload, dedupe_key)`. Workers retry on transient errors; you can't assume "called once".
|
||||
* **No LLM calls in the ingest hot path.** Anything that needs a model goes into the queue.
|
||||
* **Workspace-scoped.** Tests reset state by creating a fresh `OPENHUMAN_WORKSPACE`. Don't reach outside the workspace dir.
|
||||
|
||||
## See also
|
||||
|
||||
* [Memory Tree (user-facing)](memory-tree.md). the product surface this pipeline powers.
|
||||
* [Local AI](../model-routing/local-ai.md). opt-in path for running embeddings + summarization on-device.
|
||||
@@ -7,7 +7,7 @@ icon: tree
|
||||
|
||||
# Memory Tree
|
||||
|
||||
The Memory Tree is OpenHuman's knowledge base. It is not a vector database with a thin "memory" wrapper. It is a deterministic, bucket-sealed pipeline that turns the messy stream of your day, chats, emails, documents, integration sync results, into structured, queryable, summary-backed Markdown that lives on your machine.
|
||||
The Memory Tree is OpenHuman's knowledge base. It is not a vector database with a thin "memory" wrapper. It is a deterministic, bucket-sealed pipeline that turns the messy stream of your day — chats, emails, documents, integration sync results — into structured, queryable, summary-backed Markdown that lives on your machine.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -26,27 +26,27 @@ chunker ── deterministic IDs, ≤3k-token bounded segments
|
||||
content_store ── atomic .md files on disk (body + tags)
|
||||
│
|
||||
▼
|
||||
store ── SQLite persistence (chunks, scores, summaries, jobs)
|
||||
store ── persistence (chunks, scores, summaries, jobs)
|
||||
│
|
||||
▼
|
||||
score ── signals + embeddings + entity extraction
|
||||
│
|
||||
▼
|
||||
tree_source / tree_topic / tree_global ── per-scope summary trees
|
||||
source / topic / global trees ── per-scope summary trees
|
||||
│
|
||||
▼
|
||||
retrieval ── search · drill_down · topic · global · fetch
|
||||
```
|
||||
|
||||
The hot path (`canonicalize → chunk → stage → fast-score → persist → enqueue extract jobs`) is fast. Heavy work, embeddings, entity extraction, sealing summary buckets, daily digests, runs in background workers out of the `jobs/` queue so the UI never blocks.
|
||||
The hot path (canonicalize → chunk → fast-score → persist → enqueue follow-up work) is fast. Heavy work — embeddings, entity extraction, sealing summary buckets, daily digests — runs in background workers so the UI never blocks.
|
||||
|
||||
Embeddings and summary-tree building can run **on-device via Ollama** if you turn on [Local AI](../model-routing/local-ai.md); otherwise they go through the OpenHuman backend like any other model call.
|
||||
|
||||
## Three trees, three scopes
|
||||
|
||||
* **Source trees** (`tree_source/`), per-source rolling buffer (L0) that seals into L1 → L2 → … as it fills. One per Gmail label, one per Slack channel, one per uploaded document, etc.
|
||||
* **Topic trees** (`tree_topic/`), per-entity summaries materialized lazily by _hotness_. The more an entity (person, project, ticker, repo) shows up, the more aggressively its topic tree is built and refreshed.
|
||||
* **Global tree** (`tree_global/`), daily global digest across everything ingested that day.
|
||||
* **Source trees**, per-source rolling buffer (L0) that seals into L1 → L2 → … as it fills. One per Gmail label, one per Slack channel, one per uploaded document, etc.
|
||||
* **Topic trees**, per-entity summaries materialized lazily by _hotness_. The more an entity (person, project, ticker, repo) shows up, the more aggressively its topic tree is built and refreshed.
|
||||
* **Global tree**, one daily global digest across everything ingested that day.
|
||||
|
||||
Retrieval can target any scope: search a single source, drill down a topic, or pull the global digest.
|
||||
|
||||
@@ -56,7 +56,7 @@ Inside your workspace (default `~/.openhuman`, or whatever `OPENHUMAN_WORKSPACE`
|
||||
|
||||
| Path | What's there |
|
||||
| ----------------------- | --------------------------------------------------------------- |
|
||||
| `memory_tree/chunks.db` | SQLite — chunks, scores, summaries, entity index, jobs, hotness |
|
||||
| `memory_tree/chunks.db` | Chunks, scores, summaries, entity index, jobs, hotness |
|
||||
| `wiki/` | The Markdown vault — see [Obsidian Wiki](./) |
|
||||
|
||||
Everything is local. Nothing about your raw data leaves your machine unless you explicitly send a chat message that includes it.
|
||||
@@ -69,13 +69,87 @@ Vector stores answer "what is similar to this query?" Memory needs to answer mor
|
||||
* **What's the latest on this person?** (topic tree, hotness-driven)
|
||||
* **What did the Stripe webhook say last Tuesday at 3pm?** (source tree + provenance)
|
||||
|
||||
Trees give you compression _and_ navigation. Embeddings still live inside (in `score/`) so semantic search keeps working, but the structure on top is what makes the memory feel like a brain instead of a bag of fragments.
|
||||
Trees give you compression _and_ navigation. Embeddings still live inside so semantic search keeps working, but the structure on top is what makes the memory feel like a brain instead of a bag of fragments.
|
||||
|
||||
## How the pipeline works
|
||||
|
||||
The user-facing pitch is simple: connect a source, the agent gets persistent memory of it. The pipeline that delivers on that pitch spans an HTTP-triggered ingest path, a durable job queue, a pool of background workers, three independent summary trees, and a daily UTC scheduler.
|
||||
|
||||
The diagram below is the source of truth.
|
||||
|
||||
{% file src="../../.gitbook/assets/memory-tree-pipeline (1).excalidraw" %}
|
||||
Memory Tree Async Pipeline — leaf ingestion → jobs queue → workers → source / topic / global tree building.
|
||||
{% endfile %}
|
||||
|
||||
### 1. Ingest
|
||||
|
||||
A new chat / email / document arrives. The hot path canonicalizes it into Markdown, splits it into bounded chunks with deterministic IDs, runs a cheap fast-score, persists everything in a single transaction, marks each chunk as `pending_extraction`, and enqueues follow-up work for the workers.
|
||||
|
||||
Three properties matter here:
|
||||
|
||||
* **Deterministic.** Chunk IDs are content-addressed, so re-running ingest on identical input never produces duplicates.
|
||||
* **Fast.** No LLM calls in this lane — only cheap heuristics.
|
||||
* **Bounded write.** Everything happens in one transaction so a partial ingest can't leave dangling rows.
|
||||
|
||||
### 2. Queue
|
||||
|
||||
Follow-up work lands in a durable job queue (in the same on-disk store as the chunks). Each job carries a kind, a payload, a dedupe key, retry bookkeeping, and a scheduling window. The kinds:
|
||||
|
||||
| Kind | What it does |
|
||||
| ---------------- | --------------------------------------------------------------------------------------------- |
|
||||
| `extract_chunk` | Deep score + entity extraction. Decides `admitted` vs `dropped`. |
|
||||
| `append_buffer` | Adds an admitted leaf to the source (or topic) tree's L0 buffer. May trigger a seal. |
|
||||
| `seal` | Compresses an L0 buffer into an L1 summary; cascades up if the parent buffer is now full. |
|
||||
| `topic_route` | Routes a leaf into per-entity topic trees, gated by a hotness check. |
|
||||
| `digest_daily` | Builds the global daily digest node. |
|
||||
| `flush_stale` | Force-seals buffers that have been sitting too long. |
|
||||
|
||||
### 3. Workers
|
||||
|
||||
A small pool of background workers (3 by default) picks jobs off the queue and runs them. The pool is woken immediately by the ingest path, with a short polling fallback so a missed wake-up doesn't strand work. A shared semaphore caps concurrent LLM-bound calls so a burst of new sources can't accidentally fan out to dozens of concurrent embeddings.
|
||||
|
||||
On startup, any job whose worker lease has expired (because of a crash or kill) is returned to the queue. Crashes don't lose admitted-but-not-yet-sealed work.
|
||||
|
||||
### 4. Tree state
|
||||
|
||||
Three independent trees are built from the same leaf stream.
|
||||
|
||||
* **Source tree** — one per source. New leaves land in the L0 buffer; when the buffer fills (or a stale-flush fires), a `seal` writes an L1 summary, and the cascade continues up.
|
||||
* **Topic tree** — one per high-hotness entity. The router checks whether an entity is hot enough to deserve its own tree and, if so, appends to its buffer.
|
||||
* **Global tree** — one tree, growing one node per UTC day, walked up the hierarchy as days accumulate.
|
||||
|
||||
### 5. Scheduler
|
||||
|
||||
A scheduler loop runs independently of the ingest path. At 00:00 UTC each day it enqueues a global daily digest for yesterday and a stale-flush for today. The scheduler **does not** run summarizers itself — everything goes through the queue, so retries, dedupe, and stale-lock recovery stay centralized.
|
||||
|
||||
### 6. Leaf lifecycle
|
||||
|
||||
Each chunk moves through a small state machine:
|
||||
|
||||
```
|
||||
pending_extraction ──► admitted ──► buffered ──► sealed
|
||||
╲
|
||||
──► dropped
|
||||
```
|
||||
|
||||
* Extraction decides `admitted` vs `dropped` based on the deep score.
|
||||
* Admitted leaves move into a buffer (`buffered`).
|
||||
* When the buffer seals, every leaf inside is marked `sealed`.
|
||||
* `dropped` leaves stop here. Their chunk row stays for provenance, but no buffer or summary references them.
|
||||
|
||||
This is why retrieval can show provenance without re-running the pipeline: the chunk row plus its terminal lifecycle status is enough.
|
||||
|
||||
### Why a queue instead of in-process futures
|
||||
|
||||
* **Crash safety.** A worker panic, a process kill, a power loss — none of them lose admitted-but-not-yet-sealed work. The next start picks up where the last one left off.
|
||||
* **Retries with backoff.** Per-job retries with attempt counts and scheduled re-runs, no ad-hoc retry loops in business logic.
|
||||
* **One throttle for LLM cost.** All summarization paths share a single semaphore.
|
||||
|
||||
## Triggering ingest
|
||||
|
||||
* **Automatic**. every active integration is auto-fetched every twenty minutes; see [Auto-fetch](auto-fetch.md).
|
||||
* **Manual**. the Memory tab in the desktop app exposes a "Run ingest" trigger per source.
|
||||
* **RPC**. `openhuman.memory_tree_ingest` for advanced workflows.
|
||||
* **Automatic** — every active integration is auto-fetched every twenty minutes; see [Auto-fetch](auto-fetch.md).
|
||||
* **Manual** — the Memory tab in the desktop app exposes a "Run ingest" trigger per source.
|
||||
* **RPC** — `openhuman.memory_tree_ingest` for advanced workflows.
|
||||
|
||||
## In the desktop app — the Intelligence tab
|
||||
|
||||
@@ -93,7 +167,7 @@ Open it from the bottom navigation bar.
|
||||
| **Topics** | Number of topic trees materialized so far (per-entity summaries built from "hot" entities). |
|
||||
| **First / latest memory** | Timestamps of the oldest and newest chunks. |
|
||||
|
||||
**Memory graph.** A force-directed visualization of entities and their relationships, drawn from the entity index. The graph grows as auto-fetch pulls more data, sparse early on, denser within a few days.
|
||||
**Memory graph.** A force-directed visualization of entities and their relationships, drawn from the entity index. The graph grows as auto-fetch pulls more data — sparse early on, denser within a few days.
|
||||
|
||||
**Obsidian vault.** A **View vault in Obsidian** button opens `<workspace>/wiki/` directly via an `obsidian://open?path=...` deep link. You can also open the folder in any file browser.
|
||||
|
||||
@@ -101,12 +175,11 @@ Open it from the bottom navigation bar.
|
||||
|
||||
**Search & retrieval.** A search bar over the Memory Tree. Source-scoped, topic-scoped or global queries are all supported, and any result links back to the underlying chunk file in your Obsidian vault for full provenance.
|
||||
|
||||
**Routing.** The Intelligence tab also surfaces which model the agent is using per task, see [Automatic Model Routing](../model-routing/).
|
||||
**Routing.** The Intelligence tab also surfaces which model the agent is using per task — see [Automatic Model Routing](../model-routing/).
|
||||
|
||||
## See also
|
||||
|
||||
* [Obsidian Wiki](./). open the vault in Obsidian and edit it directly.
|
||||
* [Auto-fetch from Integrations](auto-fetch.md). how the tree stays fresh.
|
||||
* [Smart Token Compression](../token-compression.md). what makes ingesting "everything" cheap.
|
||||
* [Local AI (optional)](../model-routing/local-ai.md). opt in to keep embeddings and summary-tree building on-device.
|
||||
* [Memory Tree Pipeline](memory-tree-pipeline.md). contributor-facing deep dive on the async queue, workers and tree-state machine.
|
||||
* [Obsidian Wiki](./) — open the vault in Obsidian and edit it directly.
|
||||
* [Auto-fetch from Integrations](auto-fetch.md) — how the tree stays fresh.
|
||||
* [Smart Token Compression](../token-compression.md) — what makes ingesting "everything" cheap.
|
||||
* [Local AI (optional)](../model-routing/local-ai.md) — opt in to keep embeddings and summary-tree building on-device.
|
||||
|
||||
@@ -41,7 +41,7 @@ OpenHuman is built as a native application rather than a web wrapper for three r
|
||||
│ Rust core (`openhuman` sidecar) │
|
||||
│ • Memory Tree, integrations, auto-fetch │
|
||||
│ • Model router, TokenJuice, native tools │
|
||||
│ • Voice (STT, ElevenLabs TTS, Meet agent) │
|
||||
│ • Voice (STT in, TTS out, Meet agent) │
|
||||
└──────────────────────────────────────────────────┘
|
||||
│
|
||||
┌──────────────────────────────────────────────────┐
|
||||
|
||||
@@ -36,9 +36,9 @@ OpenHuman is designed so that the **memory of your life lives on your machine**.
|
||||
| | |
|
||||
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **LLM calls** | Proxied through the backend under one subscription, then forwarded to the underlying provider (Anthropic / OpenAI / Google / etc.) per the [model router](model-routing/). |
|
||||
| **Web search proxy** | The native [web search tool](native-tools.md) calls a backend proxy so you don't carry a search API key. |
|
||||
| **Web search proxy** | The native [web search tool](native-tools/web-search.md) calls a backend proxy so you don't carry a search API key. |
|
||||
| **Integration OAuth & tool proxy** | Token storage and rate-limited request brokering for [118+ integrations](integrations.md). |
|
||||
| **TTS streaming** | [ElevenLabs](voice.md) audio streams. Audio is generated and discarded — not retained. |
|
||||
| **TTS streaming** | Hosted [text-to-speech](native-tools/voice.md) audio streams. Audio is generated and discarded — not retained. |
|
||||
|
||||
***
|
||||
|
||||
@@ -72,7 +72,7 @@ Compression and locality together become the privacy architecture.
|
||||
|
||||
**Sandboxed skills.** Each skill runs in its own isolated execution environment with enforced memory and resource limits. Skills cannot access each other's data, the host system's file system, or your credentials.
|
||||
|
||||
**Workspace-scoped tools.** The native [filesystem tools](native-tools.md) operate within the workspace the user opens; they do not have ambient access to the rest of the disk.
|
||||
**Workspace-scoped tools.** The native [filesystem tools](native-tools/coder.md) operate within the workspace the user opens; they do not have ambient access to the rest of the disk.
|
||||
|
||||
**Short-lived tokens.** Authentication tokens between the app and the backend are time-limited.
|
||||
|
||||
|
||||
@@ -47,5 +47,5 @@ Implementation: `src/openhuman/tokenjuice/` (`classify.rs`, `reduce.rs`, `rules/
|
||||
|
||||
## See also
|
||||
|
||||
* [Native Tools](native-tools.md). most heavy tool output flows through TokenJuice.
|
||||
* [Native Tools](native-tools/README.md). most heavy tool output flows through TokenJuice.
|
||||
* [Memory Tree](obsidian-wiki/memory-tree.md). the downstream consumer of compressed output.
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
---
|
||||
description: >-
|
||||
Native voice — speech-to-text in, ElevenLabs text-to-speech out, mascot
|
||||
lip-sync, and a live Google Meet agent that listens and speaks.
|
||||
icon: microphone
|
||||
---
|
||||
|
||||
# Native Voice (ElevenLabs)
|
||||
|
||||
OpenHuman is voice-first when you want it to be. STT, TTS and the live Google Meet agent are part of the core, not a third-party plugin.
|
||||
|
||||
## Speech-to-text
|
||||
|
||||
Voice input runs through `src/openhuman/voice/`:
|
||||
|
||||
* **Hotkey** (`hotkey.rs`), push-to-talk and toggle modes.
|
||||
* **Audio capture** (`audio_capture.rs`), cross-platform mic capture with VAD.
|
||||
* **Streaming transcription** (`streaming.rs`, `cloud_transcribe.rs`), words appear as you speak.
|
||||
* **Hallucination filter** (`hallucination.rs`), strips the well-known artefacts ("Thanks for watching", silence-induced phrases).
|
||||
* **Postprocess** (`postprocess.rs`), punctuation, capitalization, dictation cleanup.
|
||||
|
||||
Dictation can replace the active text input on your desktop, or be sent straight into a chat with the agent.
|
||||
|
||||
## Text-to-speech via ElevenLabs
|
||||
|
||||
Reply speech (`reply_speech.rs`) routes through **ElevenLabs**. The agent's responses can be spoken back in a voice you pick, with the timing and prosody you'd expect from ElevenLabs' models.
|
||||
|
||||
Key bits:
|
||||
|
||||
* Voice selection is configurable per user.
|
||||
* The TTS client lives in the desktop app at `app/src/features/human/voice/ttsClient.ts`.
|
||||
* The mascot avatar uses a viseme map (`visemeMap.ts`) to lip-sync to the audio stream.
|
||||
|
||||
## Live Google Meet agent
|
||||
|
||||
The Meet agent (`src/openhuman/meet_agent/brain.rs`) is OpenHuman's flagship voice integration:
|
||||
|
||||
* Joins a Google Meet via the embedded webview.
|
||||
* Streams audio out to STT in real time, transcribes everyone in the call, and writes structured notes into the [Memory Tree](obsidian-wiki/memory-tree.md) as the meeting progresses.
|
||||
* When you ask it to speak (or it decides it has something useful to add), it generates audio with ElevenLabs and **plays it back into the meeting as an outbound camera/mic stream**, so other participants actually hear it.
|
||||
|
||||
This is real, not a demo: see commits `0bc74575` (live note-taking), `f1203479` (real LLM turns + tuned TTS), `b6d05cb4` (mascot canvas as outbound camera).
|
||||
|
||||
## Privacy
|
||||
|
||||
* Audio capture is local. Streaming STT goes through the OpenHuman backend; no recording is retained beyond the live transcript.
|
||||
* TTS audio is streamed and discarded, nothing is stored.
|
||||
* Meeting transcripts land in your local memory tree, like any other source.
|
||||
|
||||
## See also
|
||||
|
||||
* [Memory Tree](obsidian-wiki/memory-tree.md). where Meet transcripts and notes live.
|
||||
* [Automatic Model Routing](model-routing/README.md). Meet's brain uses `hint:fast` for low-latency conversational turns.
|
||||
@@ -35,9 +35,9 @@ Once connected, the [auto-fetch scheduler](../features/obsidian-wiki/auto-fetch.
|
||||
When you talk to the agent, four things happen behind the scenes:
|
||||
|
||||
* **Model routing**. The model parameter can be a hint (`hint:reasoning`, `hint:fast`, `hint:vision`). The [router](../features/model-routing/) resolves the hint to the right provider+model. One subscription, many models.
|
||||
* **Native tools**. A built-in [web search, web-fetch scraper, and full filesystem/git/lint/test/grep coder toolset](../features/native-tools.md) are wired in by default. No "install a plugin to read files" friction.
|
||||
* **Native tools**. A built-in [web search, web-fetch scraper, and full filesystem/git/lint/test/grep coder toolset](../features/native-tools/README.md) are wired in by default. No "install a plugin to read files" friction.
|
||||
* **TokenJuice compression**. Verbose tool output (git logs, large emails, build output) is compacted by a [rule overlay](../features/token-compression.md) before it ever enters the model's context. Sweeping through your last six months of email costs single-digit dollars instead of hundreds.
|
||||
* **Voice, when you want it**. STT in, [ElevenLabs TTS](../features/voice.md) out, with a live Google Meet agent that can listen, take notes, and speak back into the call.
|
||||
* **Voice, when you want it**. STT in, [TTS](../features/native-tools/voice.md) out, with a live Google Meet agent that can listen, take notes, and speak back into the call.
|
||||
|
||||
## How they connect
|
||||
|
||||
@@ -61,7 +61,7 @@ When you talk to the agent, four things happen behind the scenes:
|
||||
│ Agent (model router) │
|
||||
│ + native tools │
|
||||
│ + TokenJuice compression │
|
||||
│ + voice in/out (ElevenLabs) │
|
||||
│ + voice in/out (STT + TTS) │
|
||||
└────────────────────────────────┘
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user