diff --git a/README.md b/README.md index ed9cdae58..754b388fd 100644 --- a/README.md +++ b/README.md @@ -57,18 +57,19 @@ For contributors: [Architecture](https://tinyhumans.gitbook.io/openhuman/develop ## Context in minutes, not weeks -OpenHuman is the first agent harness that gets to know in minutes. Insipired by [Karpathy's LLM Knowledgebase](https://x.com/karpathy/status/2039805659525644595). Connect → sync → ingest → done. +OpenHuman is the first agent harness that gets to know you in minutes. Inspired by [Karpathy's LLM Knowledgebase](https://x.com/karpathy/status/2039805659525644595). Connect → sync → ingest → done.

The Tet

-> OpenHuman summarizes and compress all your documents, emails & chats; and creates a memory graph that lets your agent remember everything about you. - +> OpenHuman summarizes and compresses all your documents, emails & chats; and creates a memory graph that lets your agent remember everything about you. Most agents start cold. Hermes learns by watching you work; OpenClaw waits for plugins to ferry context in. Either way, you spend days or weeks before the agent knows enough about your stack to be genuinely useful. -OpenHuman skips the wait. Connect your accounts, let [auto-fetch](https://tinyhumans.gitbook.io/openhuman/features/integrations/auto-fetch) pull from every integration on a 20-minute loop, and the [Memory Tree](https://tinyhumans.gitbook.io/openhuman/features/memory-tree) canonicalizes everything into ≤3k-token Markdown chunks scored and folded into hierarchical summary trees: a Karpathy-style [Obsidian wiki](https://tinyhumans.gitbook.io/openhuman/features/obsidian-wiki) on disk, queryable SQLite under the hood. One sync pass and the agent has read your inbox, your calendar, your repos, your docs. No training period. No "give it a few weeks." +OpenHuman skips the wait. Connect your accounts, let [auto-fetch](https://tinyhumans.gitbook.io/openhuman/features/integrations/auto-fetch) pull context on a 20-minute loop, and the [Memory Tree](https://tinyhumans.gitbook.io/openhuman/features/memory-tree) compresses everything into Markdown chunks stored in a Karpathy-style [Obsidian wiki](https://tinyhumans.gitbook.io/openhuman/features/obsidian-wiki) on disk. + +One sync pass and the agent has understood your inbox, your calendar, your repos, your docs, your messages. All local & privately. No training period. No "give it a few weeks.". It becomes you, controlled by you. ## OpenHuman vs Other Agent Harnesses @@ -81,7 +82,7 @@ High-level comparison (products evolve, so verify against each vendor). OpenHuma | **Cost** | ⚠️ Sub + add-ons | ⚠️ BYO models | ⚠️ BYO models | ✅ One sub + TokenJuice | | **Memory** | ✅ Chat-scoped | ⚠️ Plugin-reliant | ✅ Self-learning | 🚀 Memory Tree + Obsidian vault | | **Integrations** | ⚠️ Few connectors | ⚠️ BYO | ⚠️ BYO | 🚀 118+ via OAuth | -| **Auto-fetch** | 🚫 None | 🚫 None | 🚫 None | ✅ 20-min sync into memory | +| **Auto-fetch** | 🚫 None | 🚫 None | 🚫 None | ✅ 20-min sync into memory | | **API sprawl** | 🚫 Extra keys | 🚫 BYOK | 🚫 Multi-vendor | ✅ One account | | **Model routing** | 🚫 Single model | ⚠️ Manual | ⚠️ Manual | ✅ Built-in | | **Native tools** | ✅ Code-only | ✅ Code-only | ✅ Code-only | ✅ Code + search + scraper + voice | diff --git a/gitbooks/README.md b/gitbooks/README.md index 51fbde424..113a75e38 100644 --- a/gitbooks/README.md +++ b/gitbooks/README.md @@ -18,11 +18,11 @@ OpenHuman solves this with a stack that's calmly, deliberately different: * **A local-first** [**Memory Tree**](features/obsidian-wiki/memory-tree.md)**.** Every source you connect. Gmail, Slack, GitHub, Notion, your own notes, flows through a deterministic pipeline: canonical Markdown, ≤3k-token chunks, scored, folded into per-source / per-topic / per-day summary trees. Stored in SQLite on your machine. No vector-soup black box. * **An** [**Obsidian-style wiki**](features/obsidian-wiki/) **on top of it.** The same chunks the agent reasons over land as `.md` files in a vault you can open in [Obsidian](https://obsidian.md), browse, edit, and link by hand. Inspired by [Karpathy's obsidian-wiki workflow](https://x.com/karpathy/status/2039805659525644595). You can't trust a memory you can't read. -* [**118+ third-party integrations**](features/integrations.md)**.** One-click OAuth into Gmail, GitHub, Slack, Notion, Stripe, Calendar, Drive, Linear, Jira and more — no API keys to wire by hand, no plugin marketplace to navigate. +* [**118+ third-party integrations**](features/integrations/README.md)**.** One-click OAuth into Gmail, GitHub, Slack, Notion, Stripe, Calendar, Drive, Linear, Jira and more - no API keys to wire by hand, no plugin marketplace to navigate. * [**Auto-fetch**](features/obsidian-wiki/auto-fetch.md)**.** Every twenty minutes, OpenHuman pulls fresh data from every active connection and folds it into the Memory Tree without you asking, so the agent already has tomorrow's context this morning. -* **An agent built for big data.** [Smart token compression (TokenJuice)](features/token-compression.md) compacts verbose tool output before it ever enters the model's context, so sweeping through your last six months of email costs single-digit dollars. [Automatic model routing](features/model-routing/) sends each task to the right model — `hint:reasoning` to a frontier model, `hint:fast` to a cheap one, vision to vision — all under one subscription. Optional [local AI via Ollama](features/model-routing/local-ai.md) keeps embeddings and summarization on-device. -* [**Batteries included**](features/native-tools/)**.** A complete agent toolbelt is wired in by default: [web search](features/native-tools/web-search.md), a [web-fetch scraper](features/native-tools/web-scraper.md), a full [coder toolset](features/native-tools/coder.md) (filesystem, git, lint, test, grep), [browser & computer control](features/native-tools/browser-and-computer.md), [cron & scheduling](features/native-tools/cron.md), [memory tools](features/native-tools/memory-tools.md), [agent coordination](features/native-tools/agent-coordination.md) for spawning sub-agents, and [native voice](features/native-tools/voice.md) — STT in, TTS out, mascot lip-sync, and a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call. No "install a plugin to read files" friction. -* **Simple, UI-first.** A clean desktop experience and short onboarding paths take you from install to a working agent in a few clicks — no config-first setup, no terminal required. The agent has [a face](features/mascot.md): a desktop mascot that speaks, reacts to its surroundings, joins your Google Meets as a real participant, remembers you across weeks, and keeps thinking in the background even when you've stopped typing. +* **An agent built for big data.** [Smart token compression (TokenJuice)](features/token-compression.md) compacts verbose tool output before it ever enters the model's context, so sweeping through your last six months of email costs single-digit dollars. [Automatic model routing](features/model-routing/) sends each task to the right model - `hint:reasoning` to a frontier model, `hint:fast` to a cheap one, vision to vision - all under one subscription. Optional [local AI via Ollama](features/model-routing/local-ai.md) keeps embeddings and summarization on-device. +* [**Batteries included**](features/native-tools/)**.** A complete agent toolbelt is wired in by default: [web search](features/native-tools/web-search.md), a [web-fetch scraper](features/native-tools/web-scraper.md), a full [coder toolset](features/native-tools/coder.md) (filesystem, git, lint, test, grep), [browser & computer control](features/native-tools/browser-and-computer.md), [cron & scheduling](features/native-tools/cron.md), [memory tools](features/native-tools/memory-tools.md), [agent coordination](features/native-tools/agent-coordination.md) for spawning sub-agents, and [native voice](features/native-tools/voice.md) - STT in, TTS out, mascot lip-sync, and a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call. No "install a plugin to read files" friction. +* **Simple, UI-first.** A clean desktop experience and short onboarding paths take you from install to a working agent in a few clicks - no config-first setup, no terminal required. The agent has [a face](features/mascot.md): a desktop mascot that speaks, reacts to its surroundings, joins your Google Meets as a real participant, remembers you across weeks, and keeps thinking in the background even when you've stopped typing. Together, these turn OpenHuman into something fundamentally different from a chatbot. It is an AI agent that consumes large amounts of personal data at low cost, maintains a persistent and evolving understanding of your world, and takes proactive actions on your behalf. diff --git a/gitbooks/SUMMARY.md b/gitbooks/SUMMARY.md index e616ebe3b..8b3f105dd 100644 --- a/gitbooks/SUMMARY.md +++ b/gitbooks/SUMMARY.md @@ -7,11 +7,13 @@ ## Features -* [Realtime Mascot](features/mascot.md) +* [Realtime Mascot](features/mascot/README.md) + * [Meeting Agents](features/mascot/meeting-agents.md) * [Obsidian-Style Memory](features/obsidian-wiki/README.md) * [Memory Tree](features/obsidian-wiki/memory-tree.md) * [Auto-fetch from Integrations](features/obsidian-wiki/auto-fetch.md) -* [Third-party Integrations (118+)](features/integrations.md) +* [Third-party Integrations (118+)](features/integrations/README.md) + * [Triggers](features/integrations/triggers.md) * [Smart Token Compression](features/token-compression.md) * [Automatic Model Routing](features/model-routing/README.md) * [Local AI (optional)](features/model-routing/local-ai.md) diff --git a/gitbooks/developing/README.md b/gitbooks/developing/README.md index 6c90a05cc..4f1faa465 100644 --- a/gitbooks/developing/README.md +++ b/gitbooks/developing/README.md @@ -28,7 +28,7 @@ If you just want to use the app, head to [Getting Started](../overview/getting-s If it's your first time pulling the repo: -1. [**Getting Set Up**](getting-set-up.md). Toolchain, dependencies, the vendored Tauri CLI, sidecar staging — everything `pnpm dev` needs to actually start. +1. [**Getting Set Up**](getting-set-up.md). Toolchain, dependencies, the vendored Tauri CLI, sidecar staging - everything `pnpm dev` needs to actually start. 2. [**Architecture**](architecture.md). How the desktop app, the Rust core sidecar, the JSON-RPC bridge, and the dual sockets fit together. Read this before you make non-trivial changes. 3. [**Frontend**](architecture/frontend.md) and [**Tauri Shell**](architecture/tauri-shell.md). The React app and the desktop host that wraps it. @@ -69,4 +69,4 @@ For features still being built, the [Subconscious Loop](../features/subconscious * Follow [`CONTRIBUTING.md`](../../CONTRIBUTING.md) and the issue/PR templates. * Keep changes focused. A bug fix doesn't need surrounding cleanup; a one-shot operation doesn't need a helper. -Help building toward AGI doesn't have to mean shipping a kernel — bugfixes, docs, integrations, and tests all move the bar. +Help building toward AGI doesn't have to mean shipping a kernel - bugfixes, docs, integrations, and tests all move the bar. diff --git a/gitbooks/developing/architecture.md b/gitbooks/developing/architecture.md index f98f3f831..f30aa77f0 100644 --- a/gitbooks/developing/architecture.md +++ b/gitbooks/developing/architecture.md @@ -1,5 +1,5 @@ --- -description: Deep architecture reference for the OpenHuman codebase — repo layout, runtime scope, dual-socket sync, RPC flow. +description: Deep architecture reference for the OpenHuman codebase - repo layout, runtime scope, dual-socket sync, RPC flow. icon: code-branch --- @@ -7,7 +7,7 @@ icon: code-branch **AI-powered super assistant for crypto communities, built on Rust.** -OpenHuman is a cross-platform communication and automation platform purpose-built for the cryptocurrency ecosystem. A single React + Rust (Tauri) codebase can target multiple platforms; **what we document and ship for users today is desktop only** — **Windows, macOS, and Linux**. Android, iOS, and web are **not** supported in current docs or releases. The stack includes a sandboxed JavaScript skills engine, persistent Rust-native WebSocket infrastructure, and an AI tool protocol that lets language models invoke any connected service in real time. +OpenHuman is a cross-platform communication and automation platform purpose-built for the cryptocurrency ecosystem. A single React + Rust (Tauri) codebase can target multiple platforms; **what we document and ship for users today is desktop only** - **Windows, macOS, and Linux**. Android, iOS, and web are **not** supported in current docs or releases. The stack includes a sandboxed JavaScript skills engine, persistent Rust-native WebSocket infrastructure, and an AI tool protocol that lets language models invoke any connected service in real time. --- @@ -16,7 +16,7 @@ OpenHuman is a cross-platform communication and automation platform purpose-buil | Path | Contents | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **`app/`** | Yarn workspace **`openhuman-app`**: Vite/React UI (`app/src/`), Tauri shell (`app/src-tauri/`), Vitest tests | -| **Repo root `src/`** | Rust **`openhuman_core`** library + **`openhuman`** CLI binary — `core_server`, JSON-RPC, QuickJS skills runtime (`src/openhuman/skills/`), channels, memory, etc. | +| **Repo root `src/`** | Rust **`openhuman_core`** library + **`openhuman`** CLI binary - `core_server`, JSON-RPC, QuickJS skills runtime (`src/openhuman/skills/`), channels, memory, etc. | | **`Cargo.toml`** (root) | Builds `openhuman` binary (`cargo build --bin openhuman`) staged into `app/src-tauri/binaries/` for the desktop bundle | | **`skills/`** | Skill packages consumed by the runtime | | **`docs/`** | This book + per-tree guides (`docs/src/`, `docs/src-tauri/`) | diff --git a/gitbooks/developing/architecture/README.md b/gitbooks/developing/architecture/README.md index 206e24709..dcfdb1130 100644 --- a/gitbooks/developing/architecture/README.md +++ b/gitbooks/developing/architecture/README.md @@ -1,4 +1,7 @@ --- +description: >- + High-level shape of the OpenHuman system (desktop shell, Rust core, Memory + Tree, agent loop). Pointer to the deep developer architecture in the repo. icon: code-branch --- @@ -31,7 +34,7 @@ OpenHuman is a **React + Tauri v2 desktop app** with a **Rust core** that does t │ React frontend (app/src/) │ │ • Screens, navigation │ │ • Talks to core over `coreRpcClient` │ -│ • No business logic — presentation only │ +│ • No business logic - presentation only │ └──────────────────────────────────────────────────┘ ``` @@ -43,7 +46,7 @@ OpenHuman is a **React + Tauri v2 desktop app** with a **Rust core** that does t ## Data flow -1. **Connect**. OAuth into a [integration](../../features/integrations.md). Backend stores the token; core never sees it in plaintext. +1. **Connect**. OAuth into a [integration](../../features/integrations/README.md). Backend stores the token; core never sees it in plaintext. 2. **Auto-fetch**. Every twenty minutes the [scheduler](../../features/obsidian-wiki/auto-fetch.md) walks every active connection and asks each native provider to sync. 3. **Canonicalize**. Provider output (an email page, a GitHub diff, a Slack channel dump) is normalized into provenance-tagged Markdown. 4. **Chunk**. Markdown is split into ≤3k-token deterministic chunks. diff --git a/gitbooks/developing/architecture/agent-harness.md b/gitbooks/developing/architecture/agent-harness.md index 87ee373c7..cf3567db2 100644 --- a/gitbooks/developing/architecture/agent-harness.md +++ b/gitbooks/developing/architecture/agent-harness.md @@ -1,19 +1,19 @@ --- description: >- - How an agent turn actually runs — the tool-call loop, sub-agent dispatch, + How an agent turn actually runs - the tool-call loop, sub-agent dispatch, archetypes, triage, hooks, and the cost/budget machinery around them. icon: layer-group --- # Agent Harness -The agent harness is the runtime that turns a user message (or a webhook fire, or a cron tick) into a complete, tool-using LLM interaction. It owns the tool-call loop, sub-agent dispatch, the trigger-triage pipeline, and the hook surface around them. It does **not** own provider HTTP transport, tool implementations, prompt-section assembly, or memory storage — those are separate domains the harness composes. +The agent harness is the runtime that turns a user message (or a webhook fire, or a cron tick) into a complete, tool-using LLM interaction. It owns the tool-call loop, sub-agent dispatch, the trigger-triage pipeline, and the hook surface around them. It does **not** own provider HTTP transport, tool implementations, prompt-section assembly, or memory storage - those are separate domains the harness composes. This page walks through what happens in one turn, then zooms in on each of the moving parts. ## The shape of a turn -Every turn — whether the user just typed a message, a Telegram webhook just fired, or a 9am cron just ticked — flows through the same lifecycle: +Every turn - whether the user just typed a message, a Telegram webhook just fired, or a 9am cron just ticked - flows through the same lifecycle: ``` ┌─ inbound ─────────────────────────────────────────────────────────┐ @@ -45,7 +45,7 @@ Every turn — whether the user just typed a message, a Telegram webhook just fi │ hooks │ episodic memory indexing └─────────────────┘ -* system prompt is built only on the first turn — subsequent +* system prompt is built only on the first turn - subsequent turns reuse the rendered prompt verbatim so the inference backend's KV-cache prefix stays valid. ``` @@ -60,15 +60,15 @@ A **session** is the live conversation an `Agent` instance is running. The `Agen * The provider client to call (model resolved by the [model router](../../features/model-routing/)). * The tool registry visible to the model. * A memory loader that hydrates relevant memories before each user message. -* Per-turn budgets — max tool iterations, max payload size, max USD cost. +* Per-turn budgets - max tool iterations, max payload size, max USD cost. `Agent::turn(user_message)` is the hot path. In one turn it: -1. **Resumes the session transcript** if this is a fresh process — re-loading the exact provider messages from disk so the inference backend's KV-cache prefix still hits. -2. **Builds the system prompt** (only on the first turn). This pulls in identity, soul, profile, memory, connected integrations, available tools, safety preamble — assembled by the prompt section builder. +1. **Resumes the session transcript** if this is a fresh process - re-loading the exact provider messages from disk so the inference backend's KV-cache prefix still hits. +2. **Builds the system prompt** (only on the first turn). This pulls in identity, soul, profile, memory, connected integrations, available tools, safety preamble - assembled by the prompt section builder. 3. **Injects memory context** for the new user message via the memory loader: relevant chunks from the [Memory Tree](../../features/obsidian-wiki/memory-tree.md), with citations attached so the UI can show provenance. 4. **Enters the tool-call loop** (next section). -5. **Spawns post-turn hooks** in the background — the user gets their answer before archivist / learning / cost logging finishes. +5. **Spawns post-turn hooks** in the background - the user gets their answer before archivist / learning / cost logging finishes. The system prompt is **not** rebuilt on subsequent turns. Even cosmetic byte changes invalidate the KV-cache prefix and force a full re-prefill, so dynamic per-turn context (memory recall, freshly-learned snippets) is appended as user-visible message content rather than spliced into the system prompt. @@ -78,14 +78,14 @@ Inside `Agent::turn`, the tool-call loop is the inner engine. It runs up to `max ``` loop { - 1. context guard — if history is too big, microcompact / autocompact - 2. stop-hook check — budget caps, max-iterations, custom kill switches - 3. provider call — send messages + tool specs, stream the response - 4. parse response — split assistant text from tool calls - 5. if no tool calls — return final text - 6. execute tool calls — dispatch each one (next section) - 7. summarize oversize — route huge tool outputs through the summarizer agent - 8. append results — push tool results into history, loop again + 1. context guard - if history is too big, microcompact / autocompact + 2. stop-hook check - budget caps, max-iterations, custom kill switches + 3. provider call - send messages + tool specs, stream the response + 4. parse response - split assistant text from tool calls + 5. if no tool calls - return final text + 6. execute tool calls - dispatch each one (next section) + 7. summarize oversize - route huge tool outputs through the summarizer agent + 8. append results - push tool results into history, loop again } ``` @@ -95,9 +95,9 @@ Every iteration emits a real-time `AgentProgress` event so the UI can render tok Different LLMs speak different tool-calling dialects. The harness abstracts that with a `ToolDispatcher` trait, which has three concrete implementations: -* **Native** — providers with first-class tool-calling APIs (Anthropic, OpenAI). Tool calls come back as structured fields, not in the text body. -* **XML** — fallback for models that aren't natively trained for tool-calling but can follow instructions. Tools are wrapped in `{...}` tags in the assistant text. -* **P-Format** — a compact text format used by some smaller models. +* **Native** - providers with first-class tool-calling APIs (Anthropic, OpenAI). Tool calls come back as structured fields, not in the text body. +* **XML** - fallback for models that aren't natively trained for tool-calling but can follow instructions. Tools are wrapped in `{...}` tags in the assistant text. +* **P-Format** - a compact text format used by some smaller models. The dispatcher is selected per provider, which keeps the loop itself dialect-agnostic. The same loop code drives Claude, GPT, Gemini, and a local Ollama model. @@ -105,12 +105,12 @@ The dispatcher is selected per provider, which keeps the loop itself dialect-agn Long tool-calling chains can blow past the context window. Two layers handle that: -* **Tool-result budget** — every tool result is checked against a per-call byte budget. Anything over is hard-truncated with an explanatory marker so the model knows it didn't see the full output. -* **Microcompact / autocompact** — when total history is creeping toward the context window, the harness compacts older turns into summaries before the next provider call. The compacted history keeps the system prompt and the most recent turns intact (KV-cache stability) and rewrites the middle. +* **Tool-result budget** - every tool result is checked against a per-call byte budget. Anything over is hard-truncated with an explanatory marker so the model knows it didn't see the full output. +* **Microcompact / autocompact** - when total history is creeping toward the context window, the harness compacts older turns into summaries before the next provider call. The compacted history keeps the system prompt and the most recent turns intact (KV-cache stability) and rewrites the middle. -### Oversized tool results — the summarizer detour +### Oversized tool results - the summarizer detour -Some tool calls return enormous payloads — a Composio action dumping 200 KB of JSON, a web scrape returning 50 KB of markdown, a `file_read` over a multi-thousand-line log. Hard-truncating mid-payload drops whatever happens to land past the cut. +Some tool calls return enormous payloads - a Composio action dumping 200 KB of JSON, a web scrape returning 50 KB of markdown, a `file_read` over a multi-thousand-line log. Hard-truncating mid-payload drops whatever happens to land past the cut. When a tool result exceeds the summarizer's threshold, it gets routed through a dedicated `summarizer` sub-agent before entering the parent's history. The summarizer compresses the payload per an extraction contract that preserves identifiers and key facts, and the parent agent only sees the compressed summary. Hard truncation remains the backstop downstream when summarization fails or the payload is so absurdly large that paying for an LLM call on it makes no economic sense. @@ -118,17 +118,17 @@ When a tool result exceeds the summarizer's threshold, it gets routed through a When the code-executor sub-agent runs a shell command and the runtime answers "command not found", a self-healing interceptor catches the error, spawns a `ToolMaker` sub-agent to write a polyfill script for the missing command, and retries the original call. There's a per-command attempt cap so a genuinely impossible command can't loop forever. -## Sub-agents — the orchestrator pattern +## Sub-agents - the orchestrator pattern -OpenHuman is **multi-agent**. The agent the user is chatting with is the **Orchestrator** — a senior, strategy-level agent that decides when to answer directly, when to use a direct tool, and when to spawn a specialist sub-agent. +OpenHuman is **multi-agent**. The agent the user is chatting with is the **Orchestrator** - a senior, strategy-level agent that decides when to answer directly, when to use a direct tool, and when to spawn a specialist sub-agent. ### Why multi-agent A single agent that knows everything also has a system prompt the size of a small book. Splitting work across specialists means: * Each sub-agent gets a **narrow system prompt** with only the sections it needs (identity / memory / safety preamble can be stripped). -* Each sub-agent gets a **filtered tool registry** — the integrations agent doesn't need filesystem tools, the coder doesn't need the Composio catalog. -* Sub-agent histories never leak back to the parent — the parent sees one compact tool result, not the inner conversation. +* Each sub-agent gets a **filtered tool registry** - the integrations agent doesn't need filesystem tools, the coder doesn't need the Composio catalog. +* Sub-agent histories never leak back to the parent - the parent sees one compact tool result, not the inner conversation. * Cheaper models can do the leaf work. The orchestrator is on a strong reasoning model; a research sub-agent might be on a faster, cheaper one. ### The built-in archetypes @@ -138,13 +138,13 @@ Each archetype lives under `agents//` with an `agent.toml` (metadata, tool | Archetype | When the orchestrator picks it | | ------------------- | --------------------------------------------------------------------------------------- | | `orchestrator` | The top-level agent. Never spawned by another orchestrator. | -| `planner` | Multi-step decomposition — break a complex request into ordered sub-tasks. | +| `planner` | Multi-step decomposition - break a complex request into ordered sub-tasks. | | `researcher` | Web/doc lookups, citation hunting. | | `code_executor` | Writing, running, and debugging code in the workspace. | | `critic` | Code review, quality checks on another agent's output. | | `summarizer` | Compressing oversized tool results (called by the harness, not usually the model). | -| `archivist` | Memory distillation — what to persist, what to forget. | -| `tool_maker` | Self-healing — writes polyfills for missing shell commands. | +| `archivist` | Memory distillation - what to persist, what to forget. | +| `tool_maker` | Self-healing - writes polyfills for missing shell commands. | | `tools_agent` | Generic specialist for arbitrary tool-bound tasks. | | `integrations_agent`| Bound to a specific Composio toolkit (Gmail, GitHub, Slack…) for that toolkit's actions.| | `trigger_triage` | Classifies incoming external events into drop / notify / spawn-reactor / spawn-agent. | @@ -158,20 +158,20 @@ Custom archetypes ship as TOML files under `$OPENHUMAN_WORKSPACE/agents/*.toml` When the orchestrator calls `spawn_subagent` (or one of the `delegate_*` convenience tools), the runner: -1. Reads the parent's execution context from a task-local — the parent's provider, sandbox mode, cancellation fence, transcript root. -2. Resolves the sub-agent's model — inherit from parent, follow a hint (`fast` / `reasoning` / `summarization`), or pin an exact model. +1. Reads the parent's execution context from a task-local - the parent's provider, sandbox mode, cancellation fence, transcript root. +2. Resolves the sub-agent's model - inherit from parent, follow a hint (`fast` / `reasoning` / `summarization`), or pin an exact model. 3. Filters the parent's tool registry per the definition's `tools`, `disallowed_tools`, and `skill_filter`. In `fork` mode, the parent's full registry is inherited verbatim. 4. Builds a narrow system prompt, omitting the sections the definition asks to strip. 5. Runs an inner tool-call loop using the same machinery as the parent. -6. Returns one compact text result. The intra-sub-agent history is never spliced back into the parent — the orchestrator sees a single tool result and moves on. +6. Returns one compact text result. The intra-sub-agent history is never spliced back into the parent - the orchestrator sees a single tool result and moves on. For tasks that don't need to block the orchestrator's turn, `spawn_worker_thread` runs the sub-agent in the background and the orchestrator continues immediately. ### Toolkit-specific specialists -For Composio toolkits with hundreds of actions (GitHub alone has 500+), loading every action into the sub-agent's tool set balloons prompt size. The harness ranks the toolkit's actions against the parent-refined task prompt with a cheap CPU-only filter (verb detection, token overlap, verb-alignment boost) and only loads the top-ranked subset into the sub-agent. No model call, pure heuristic — fast and explainable. +For Composio toolkits with hundreds of actions (GitHub alone has 500+), loading every action into the sub-agent's tool set balloons prompt size. The harness ranks the toolkit's actions against the parent-refined task prompt with a cheap CPU-only filter (verb detection, token overlap, verb-alignment boost) and only loads the top-ranked subset into the sub-agent. No model call, pure heuristic - fast and explainable. -## Triage — handling external triggers +## Triage - handling external triggers When a webhook fires, a cron ticks, or a Composio event arrives, the system can't just hand it straight to the orchestrator. Most triggers are noise; some warrant a notification; only a few deserve a full agent turn. The **trigger-triage pipeline** is the gate. @@ -186,9 +186,9 @@ TriggerEnvelope ──► run_triage ──► TriageDecision ──► apply_de └── small local LLM (with cloud-LLM retry fallback) ``` -The evaluator is intentionally cheap — a small local model where available, falling back to a remote model on retry. The decision is cached so identical triggers don't re-classify. Only triggers that escalate to "spawn orchestrator" go through the full `Agent::turn` machinery. +The evaluator is intentionally cheap - a small local model where available, falling back to a remote model on retry. The decision is cached so identical triggers don't re-classify. Only triggers that escalate to "spawn orchestrator" go through the full `Agent::turn` machinery. -## Hooks — observability and policy levers +## Hooks - observability and policy levers Two hook surfaces wrap the loop, on opposite ends: @@ -196,25 +196,25 @@ Two hook surfaces wrap the loop, on opposite ends: Stop hooks fire **between** iterations of the tool-call loop. They're the policy lever for budget caps, rate limits, and custom kill switches. Built-in hooks: -* **Budget stop hook** — caps cumulative turn cost in USD using the per-iteration cost accumulator. -* **Max-iterations stop hook** — caps iteration count from outside the agent's persistent config. +* **Budget stop hook** - caps cumulative turn cost in USD using the per-iteration cost accumulator. +* **Max-iterations stop hook** - caps iteration count from outside the agent's persistent config. A hook returning `Stop` aborts the loop with a clear reason the caller can surface to the user. Stop hooks are distinct from interrupts (next section): they're policy-driven, not user-driven. ### Post-turn hooks -Post-turn hooks fire **after** the turn completes, in the background. They get a `TurnContext` snapshot — user message, assistant response, every tool call with arguments and outcome, total wall-clock, iteration count, session ID. Built-in consumers: +Post-turn hooks fire **after** the turn completes, in the background. They get a `TurnContext` snapshot - user message, assistant response, every tool call with arguments and outcome, total wall-clock, iteration count, session ID. Built-in consumers: -* **Archivist** — distills which facts from the turn are worth persisting to long-term memory. -* **Learning** — feeds reflection, tool-tracker, and user-profile updates. -* **Cost log** — final per-turn cost line. -* **Episodic memory indexing** — writes the turn into the [Memory Tree](../../features/obsidian-wiki/memory-tree.md) as a chunk for future recall. +* **Archivist** - distills which facts from the turn are worth persisting to long-term memory. +* **Learning** - feeds reflection, tool-tracker, and user-profile updates. +* **Cost log** - final per-turn cost line. +* **Episodic memory indexing** - writes the turn into the [Memory Tree](../../features/obsidian-wiki/memory-tree.md) as a chunk for future recall. Hooks run via `tokio::spawn`, so the user gets their answer before any of them finish. -## Interrupts — graceful cancellation +## Interrupts - graceful cancellation -An `InterruptFence` is checked at fixed safe points in the loop — before each tool execution, before each sub-agent spawn, before each provider call. When the user hits Ctrl+C or sends `/stop`: +An `InterruptFence` is checked at fixed safe points in the loop - before each tool execution, before each sub-agent spawn, before each provider call. When the user hits Ctrl+C or sends `/stop`: * The fence flips. * Every running sub-agent sees the same flag (it's shared via `Arc`) and bails at its next checkpoint. @@ -225,7 +225,7 @@ Interrupts are user-driven; stop hooks are policy-driven. They share the underly ## Cost accounting -Every provider response carries a `UsageInfo` block — input tokens, output tokens, cached input tokens, and an authoritative `charged_amount_usd` populated by the OpenHuman backend. `TurnCost` sums those across every provider call inside one turn so the harness can: +Every provider response carries a `UsageInfo` block - input tokens, output tokens, cached input tokens, and an authoritative `charged_amount_usd` populated by the OpenHuman backend. `TurnCost` sums those across every provider call inside one turn so the harness can: * Emit per-iteration cost telemetry over the progress channel. * Feed the budget stop hook so a runaway turn cuts itself off mid-loop. @@ -233,19 +233,19 @@ Every provider response carries a `UsageInfo` block — input tokens, output tok When the backend doesn't surface a charged amount (older builds, providers that don't bill through it), a small per-tier rate table provides a token-rate floor estimate. Direct cost from the backend always wins when available. -## Fork context — KV-cache reuse across the harness +## Fork context - KV-cache reuse across the harness -The harness uses a task-local `ParentExecutionContext` to thread parent state into sub-agents without exploding every function signature. The same pattern carries the current sandbox mode, the interrupt fence, and the stop-hook list. Sub-agents that inherit the parent's provider, model, and prompt prefix get to **share the parent's KV-cache prefix** on the inference backend — measurably cheaper than re-prefilling from scratch. +The harness uses a task-local `ParentExecutionContext` to thread parent state into sub-agents without exploding every function signature. The same pattern carries the current sandbox mode, the interrupt fence, and the stop-hook list. Sub-agents that inherit the parent's provider, model, and prompt prefix get to **share the parent's KV-cache prefix** on the inference backend - measurably cheaper than re-prefilling from scratch. ## Self-healing recap A few small adaptive systems sit on top of the main loop: -* **Self-healing for missing commands** — `ToolMaker` polyfills, capped retry attempts. -* **Payload summarizer circuit-breaker** — three consecutive sub-agent failures in a session disable summarization, falling back to truncation. -* **Triage local-vs-remote retry** — local LLM first; remote fallback on parse failure. +* **Self-healing for missing commands** - `ToolMaker` polyfills, capped retry attempts. +* **Payload summarizer circuit-breaker** - three consecutive sub-agent failures in a session disable summarization, falling back to truncation. +* **Triage local-vs-remote retry** - local LLM first; remote fallback on parse failure. -None of these change the loop's shape — they just make the common failure modes recoverable without the user having to intervene. +None of these change the loop's shape - they just make the common failure modes recoverable without the user having to intervene. ## Where to look in the code @@ -253,17 +253,17 @@ The harness lives entirely under `src/openhuman/agent/`. The README in that dire | File / dir | What lives there | | ----------------------------- | ----------------------------------------------------------------- | -| `harness/session/turn.rs` | `Agent::turn` — the lifecycle described above. | +| `harness/session/turn.rs` | `Agent::turn` - the lifecycle described above. | | `harness/tool_loop.rs` | The inner tool-call loop. | | `harness/subagent_runner/` | `run_subagent`, fork-mode, oversized-result handoff. | -| `harness/definition.rs` | `AgentDefinition` — what an archetype declares. | +| `harness/definition.rs` | `AgentDefinition` - what an archetype declares. | | `harness/tool_filter.rs` | Toolkit-action ranking for integrations sub-agents. | | `harness/payload_summarizer.rs` | Oversized-tool-result detour. | | `harness/self_healing.rs` | Missing-command interceptor. | | `harness/interrupt.rs` | The cancellation fence. | | `dispatcher.rs` | Tool-call dialect abstraction. | | `triage/` | External-trigger classification + escalation. | -| `agents/` | Built-in archetypes — one subdirectory per agent. | +| `agents/` | Built-in archetypes - one subdirectory per agent. | | `hooks.rs` / `stop_hooks.rs` | Post-turn and mid-turn hook surfaces. | | `cost.rs` | Per-turn USD/token accounting. | | `progress.rs` | Real-time progress events to the UI. | @@ -271,7 +271,7 @@ The harness lives entirely under `src/openhuman/agent/`. The README in that dire ## See also -* [Architecture overview](README.md) — where the harness sits in the bigger picture. -* [Memory Tree](../../features/obsidian-wiki/memory-tree.md) — what the memory loader reads from and post-turn hooks write to. -* [Automatic Model Routing](../../features/model-routing/) — how `model: "hint:reasoning"` resolves to a concrete provider+model. -* [Native Tools — Agent Coordination](../../features/native-tools/agent-coordination.md) — the user-facing surface for `spawn_subagent`, `delegate_*`, `todo_write`. +* [Architecture overview](README.md) - where the harness sits in the bigger picture. +* [Memory Tree](../../features/obsidian-wiki/memory-tree.md) - what the memory loader reads from and post-turn hooks write to. +* [Automatic Model Routing](../../features/model-routing/) - how `model: "hint:reasoning"` resolves to a concrete provider+model. +* [Native Tools - Agent Coordination](../../features/native-tools/agent-coordination.md) - the user-facing surface for `spawn_subagent`, `delegate_*`, `todo_write`. diff --git a/gitbooks/developing/architecture/frontend.md b/gitbooks/developing/architecture/frontend.md index 73261eb5c..606ce3351 100644 --- a/gitbooks/developing/architecture/frontend.md +++ b/gitbooks/developing/architecture/frontend.md @@ -1,6 +1,6 @@ --- description: >- - The React + Vite frontend (`app/src/`) — architecture, state, services, + The React + Vite frontend (`app/src/`) - architecture, state, services, providers, routing, components, hooks. icon: browsers --- @@ -94,14 +94,14 @@ Redux Provider ``` App.tsx ├─ Redux store + persistor - ├─ UserProvider — user profile / workspace context - ├─ SocketProvider — connects socketService when token present - ├─ AIProvider — AI session / memory client coordination - ├─ SkillProvider — skills catalog and sync + ├─ UserProvider - user profile / workspace context + ├─ SocketProvider - connects socketService when token present + ├─ AIProvider - AI session / memory client coordination + ├─ SkillProvider - skills catalog and sync └─ AppRoutes - ├─ PublicRoute — e.g. Welcome on `/` - ├─ ProtectedRoute — onboarding, home, skills, settings, … - └─ DefaultRedirect — unauthenticated users + ├─ PublicRoute - e.g. Welcome on `/` + ├─ ProtectedRoute - onboarding, home, skills, settings, … + └─ DefaultRedirect - unauthenticated users ``` ### Services layer (conceptual) @@ -403,7 +403,7 @@ app/src/services/ │ └─ Tauri: coordinates with Rust-side socket via utils/tauriSocket.ts ├─ coreRpcClient.ts │ └─ invoke('core_rpc_relay', …) → local openhuman core (JSON-RPC) - └─ services/api/* — domain REST modules (auth, user, teams, …) + └─ services/api/* - domain REST modules (auth, user, teams, …) ``` ### API Client (`services/apiClient.ts`) diff --git a/gitbooks/developing/architecture/tauri-shell.md b/gitbooks/developing/architecture/tauri-shell.md index f9e3734b8..48429dffd 100644 --- a/gitbooks/developing/architecture/tauri-shell.md +++ b/gitbooks/developing/architecture/tauri-shell.md @@ -1,5 +1,5 @@ --- -description: The desktop host (`app/src-tauri/`) — Tauri v2 + WebView, IPC, sidecar lifecycle, core bridge. +description: The desktop host (`app/src-tauri/`) - Tauri v2 + WebView, IPC, sidecar lifecycle, core bridge. icon: desktop --- diff --git a/gitbooks/developing/cef.md b/gitbooks/developing/cef.md index a98130aad..5b3d00373 100644 --- a/gitbooks/developing/cef.md +++ b/gitbooks/developing/cef.md @@ -53,7 +53,7 @@ Each provider has a **scanner module** in [`app/src-tauri/src/`](https://github. | ------------------ | ------------------------------- | -------------------------------------------------------------------- | | `whatsapp_scanner` | 2s DOM tick + 30s full IDB walk | Reads message stores, pulls media metadata | | `telegram_scanner` | Same | Plus QR-login hand-off to native Telegram Desktop | -| `slack_scanner` | 30s IDB walk | Pure IDB — no DOM scrape needed | +| `slack_scanner` | 30s IDB walk | Pure IDB - no DOM scrape needed | | `discord_scanner` | Periodic | Channel + DM state via CDP | | `meet_scanner` | Periodic | Live captions + participant state during calls | | `imessage_scanner` | Periodic | **No webview.** Reads `~/Library/Messages/chat.db` directly on macOS | diff --git a/gitbooks/developing/getting-set-up.md b/gitbooks/developing/getting-set-up.md index 665a81a17..8fdd576b2 100644 --- a/gitbooks/developing/getting-set-up.md +++ b/gitbooks/developing/getting-set-up.md @@ -1,5 +1,5 @@ --- -description: How to build OpenHuman from source — toolchain, vendored Tauri CLI, sidecar staging. +description: How to build OpenHuman from source - toolchain, vendored Tauri CLI, sidecar staging. icon: wrench --- @@ -105,7 +105,7 @@ pnpm tauri build --target aarch64-unknown-linux-gnu The binary requires the CEF library path to be set: -### Option 1 — Direct invocation +### Option 1 - Direct invocation ```bash REL_DIR=app/src-tauri/target/aarch64-unknown-linux-gnu/release @@ -114,7 +114,7 @@ export LD_LIBRARY_PATH="$CEF_DIR:$REL_DIR/deps:$REL_DIR${LD_LIBRARY_PATH:+:$LD_L "$REL_DIR/OpenHuman" --no-sandbox ``` -### Option 2 — Wrapper script (recommended) +### Option 2 - Wrapper script (recommended) Save to `~/bin/openhuman` and make it executable (`chmod +x ~/bin/openhuman`): diff --git a/gitbooks/developing/release-policy.md b/gitbooks/developing/release-policy.md index 592cac6a1..4b0d99803 100644 --- a/gitbooks/developing/release-policy.md +++ b/gitbooks/developing/release-policy.md @@ -19,7 +19,7 @@ Production web builds embed a **minimum supported app semver** at **build time** | Variable | Purpose | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | -| `VITE_MINIMUM_SUPPORTED_APP_VERSION` | e.g. `0.51.0` — desktop app must be **≥** this to finish `openhuman://oauth/success`. | +| `VITE_MINIMUM_SUPPORTED_APP_VERSION` | e.g. `0.51.0` - desktop app must be **≥** this to finish `openhuman://oauth/success`. | | `VITE_LATEST_APP_DOWNLOAD_URL` | Optional; defaults to `https://github.com/tinyhumansai/openhuman/releases/latest`. Opened when the gate blocks OAuth. | Configure these as **GitHub Actions variables**. They must be present on **both** the standalone **`pnpm build`** step and the **`tauri-apps/tauri-action`** step env in `.github/workflows/build-desktop.yml` (the reusable matrix invoked by `release-production.yml` / `release-staging.yml`) and `build-windows.yml` so the Vite bundle embedded in shipped installers includes the gate. Leave `VITE_MINIMUM_SUPPORTED_APP_VERSION` **unset** for local dev (gate disabled). diff --git a/gitbooks/developing/testing-strategy.md b/gitbooks/developing/testing-strategy.md index 11f745cdc..9daaf7bbb 100644 --- a/gitbooks/developing/testing-strategy.md +++ b/gitbooks/developing/testing-strategy.md @@ -1,5 +1,5 @@ --- -description: How OpenHuman tests its product — Vitest, cargo test, WDIO E2E. Where each test goes. +description: How OpenHuman tests its product - Vitest, cargo test, WDIO E2E. Where each test goes. icon: vial --- @@ -21,14 +21,14 @@ How OpenHuman tests its product. Source of truth for "where does my test go?". C --- -## Decision tree — where does my test go? +## Decision tree - where does my test go? ```text Is the change behind the JSON-RPC boundary (in `src/`)? -├─ YES — does it cross domains or talk to external services? +├─ YES - does it cross domains or talk to external services? │ ├─ YES → Rust integration (tests/*.rs) │ └─ NO → Rust unit (next to source) -└─ NO — change is in `app/` +└─ NO - change is in `app/` ├─ Is it a pure function, hook, slice, or component in isolation? │ └─ YES → Vitest unit (*.test.tsx co-located) └─ Is it user-visible AND it crosses UI ⇄ Tauri ⇄ sidecar ⇄ JSON-RPC? @@ -116,14 +116,14 @@ pnpm test:unit # Rust integration with mock backend pnpm test:rust -# E2E (slow — run when behaviour changes user-visibly) +# E2E (slow - run when behaviour changes user-visibly) pnpm test:e2e:build bash app/scripts/e2e-run-spec.sh test/e2e/specs/.spec.ts ``` --- -## Not driver-automatable — manual smoke required +## Not driver-automatable - manual smoke required Some surfaces cannot be driven by WDIO / Appium because they cross OS-level trust boundaries or hardware paths. The complete checklist + sign-off block lives in [`docs/RELEASE-MANUAL-SMOKE.md`](../../docs/RELEASE-MANUAL-SMOKE.md), that file is the source of truth for what must be verified per release. Examples of what it covers: diff --git a/gitbooks/features/cloud-deploy.md b/gitbooks/features/cloud-deploy.md index fcc39ddb1..20a248297 100644 --- a/gitbooks/features/cloud-deploy.md +++ b/gitbooks/features/cloud-deploy.md @@ -1,5 +1,5 @@ --- -description: Hosting the headless openhuman-core in the cloud — DigitalOcean App Platform or Docker Compose on any VPS. +description: Hosting the headless openhuman-core in the cloud - DigitalOcean App Platform or Docker Compose on any VPS. icon: cloud --- @@ -87,7 +87,7 @@ If you'd rather not click through the UI: # One-time: install doctl and authenticate. doctl auth init -# Edit .do/app.yaml — set OPENHUMAN_CORE_TOKEN to a real value (or pass it in +# Edit .do/app.yaml - set OPENHUMAN_CORE_TOKEN to a real value (or pass it in # at create time via --spec with envsubst). Then: doctl apps create --spec .do/app.yaml @@ -116,7 +116,7 @@ cd openhuman # Configure secrets: cp .env.example .env -# Edit .env — at minimum: +# Edit .env - at minimum: # BACKEND_URL=https://api.tinyhumans.ai # OPENHUMAN_CORE_TOKEN= # OPENHUMAN_APP_ENV=production diff --git a/gitbooks/features/integrations.md b/gitbooks/features/integrations/README.md similarity index 75% rename from gitbooks/features/integrations.md rename to gitbooks/features/integrations/README.md index 7826a2e0f..b53f62b50 100644 --- a/gitbooks/features/integrations.md +++ b/gitbooks/features/integrations/README.md @@ -1,7 +1,7 @@ --- description: >- - 118+ third-party integrations — Gmail, Notion, GitHub, Slack, Stripe, Calendar - and more — with one-click OAuth and zero API keys. + 118+ third-party integrations - Gmail, Notion, GitHub, Slack, Stripe, Calendar + and more - with one-click OAuth and zero API keys. icon: plug --- @@ -11,11 +11,12 @@ OpenHuman ships with backend-proxied access to **118+ third-party services**. Co (Under the hood, the connector layer is powered by [Composio](https://composio.dev). You will not need to think about it.) -Once a service is connected, it shows up in three places at once: +Once a service is connected, it shows up in four places at once: 1. As an **agent tool**, the model can call it directly. -2. As a **memory source**, [auto-fetch](obsidian-wiki/auto-fetch.md) syncs it into the [Memory Tree](obsidian-wiki/memory-tree.md) every twenty minutes. +2. As a **memory source**, [auto-fetch](../obsidian-wiki/auto-fetch.md) syncs it into the [Memory Tree](../obsidian-wiki/memory-tree.md) every twenty minutes. 3. As a **profile signal**, your activity across services feeds your personalization. +4. As a **trigger source**, live events (a new email, a new charge, an inbound DM) flow into the [Triggers](triggers.md) pipeline and can fire off agent actions automatically. ## Some of what's in the catalog @@ -38,7 +39,7 @@ Some services have **native providers**. Rust modules that know how to ingest th ## How connections work -Click **Connect** on any integration. A browser window opens for OAuth. Once you sign in, the connection becomes active and OpenHuman starts syncing it through [auto-fetch](obsidian-wiki/auto-fetch.md) on the next 20-minute tick. +Click **Connect** on any integration. A browser window opens for OAuth. Once you sign in, the connection becomes active and OpenHuman starts syncing it through [auto-fetch](../obsidian-wiki/auto-fetch.md) on the next 20-minute tick. Each integration shows its current status: @@ -66,16 +67,17 @@ Beyond third-party services, OpenHuman has **skills**, small sandboxed modules t Two capabilities ship native rather than as integrations because they're load-bearing for the desktop experience: -* [**Voice**](native-tools/voice.md). STT in, TTS out, plus a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call. -* [**Native tools**](native-tools/README.md). built-in web search, web-fetch scraper, and a full filesystem/git/lint/test/grep coder toolset that the agent uses out of the box. +* [**Voice**](../native-tools/voice.md). STT in, TTS out, plus a live Google Meet agent that joins meetings, transcribes them into your Memory Tree, and can speak back into the call. +* [**Native tools**](../native-tools/README.md). built-in web search, web-fetch scraper, and a full filesystem/git/lint/test/grep coder toolset that the agent uses out of the box. ## Privacy boundary OpenHuman's core never calls any third-party API directly. All requests go through the OpenHuman backend, which handles OAuth tokens and rate limiting. Your tokens never sit on disk in plaintext on your machine, and the agent only sees the _results_ of tool calls, not the credentials. -See [Privacy & Security](privacy-and-security.md) for the full boundary. +See [Privacy & Security](../privacy-and-security.md) for the full boundary. ## See also -* [Auto-fetch from Integrations](obsidian-wiki/auto-fetch.md) -* [Memory Tree](obsidian-wiki/memory-tree.md) +* [Triggers](triggers.md), live events from connected integrations and how they fire agent actions. +* [Auto-fetch from Integrations](../obsidian-wiki/auto-fetch.md) +* [Memory Tree](../obsidian-wiki/memory-tree.md) diff --git a/gitbooks/features/integrations/triggers.md b/gitbooks/features/integrations/triggers.md new file mode 100644 index 000000000..f6dc22b33 --- /dev/null +++ b/gitbooks/features/integrations/triggers.md @@ -0,0 +1,134 @@ +--- +description: >- + External events from connected integrations (a new Gmail message, a Notion + edit, a Stripe charge, an inbound Slack DM) flow into OpenHuman as + triggers, get classified by a triage agent, and can fire off agent + actions automatically. +icon: bolt +--- + +# Triggers + +A connected integration is not just a place the agent can read from on demand. It is also a **source of live events**. When someone sends you an email, edits a Notion page, opens a GitHub issue on one of your repos, charges a card on Stripe, or DMs you on Slack, OpenHuman receives that event in near-real-time and can decide whether to do something about it. + +This page is about that pipeline: how triggers arrive, how they get classified, and how a trigger can turn into a full agent action without you typing a thing. + +## What a trigger is + +A trigger is an external event published by an integration you've connected. Common shapes: + +| Integration | Example trigger | +| --- | --- | +| **Gmail** | `GMAIL_NEW_GMAIL_MESSAGE`, new mail in inbox | +| **Slack** | `SLACK_NEW_MESSAGE`, channel/DM message you were mentioned in | +| **Notion** | `NOTION_PAGE_UPDATED`, a tracked page changed | +| **GitHub** | `GITHUB_ISSUE_OPENED`, `GITHUB_PULL_REQUEST_OPENED` on your repos | +| **Stripe** | `STRIPE_CHARGE_SUCCEEDED`, a successful charge on your account | +| **Calendar** | `GOOGLE_CALENDAR_EVENT_CREATED`, a new event on your calendar | + +The full set comes from the [Composio](https://composio.dev) connector layer that powers [third-party integrations](README.md). When a connection is active, the relevant trigger subscriptions are wired up automatically. + +## Where triggers come from, end to end + +``` +┌────────────────────┐ +│ third-party API │ Gmail / Slack / Notion / GitHub / ... +└─────────┬──────────┘ + │ webhook + ▼ +┌────────────────────┐ +│ OpenHuman backend │ HMAC-verifies the webhook, normalises the payload +└─────────┬──────────┘ + │ Socket.IO event ("composio:trigger") + ▼ +┌────────────────────┐ +│ Rust core │ publishes DomainEvent::ComposioTriggerReceived +│ (your laptop) │ on the in-process event bus +└─────────┬──────────┘ + │ + ▼ +┌────────────────────┐ +│ Trigger Triage │ classifies: drop / acknowledge / react / escalate +└─────────┬──────────┘ + │ + ▼ +┌────────────────────┐ +│ One of: │ +│ - nothing │ ← drop +│ - memory note │ ← acknowledge +│ - Trigger Reactor │ ← react (1-2 tool calls) +│ - Orchestrator │ ← escalate (full multi-step planning) +└────────────────────┘ +``` + +The webhook never reaches your machine raw. The backend is what holds the OAuth token and what receives the webhook directly from the third-party. It does HMAC verification, normalises the payload, and forwards it to your Rust core over the existing authenticated socket. Your laptop sees a clean, validated `ComposioTriggerReceived` event on the bus, nothing else. + +## The triage step + +Before any action runs, every trigger goes through the [`trigger_triage`](https://github.com/tinyhumansai/openhuman/tree/main/src/openhuman/agent/agents/trigger_triage) agent. Its only job is to decide what the rest of the system should do. + +It picks exactly one of four actions: + +| Action | What happens | When to use | +| --- | --- | --- | +| **`drop`** | Nothing. Trigger is silently logged and discarded. | Spam, duplicates, irrelevant noise. The default for things you don't care about. | +| **`acknowledge`** | A short memory note is persisted, no agent runs. | Passive notifications worth remembering ("a new page was created in archive"). | +| **`react`** | The [`trigger_reactor`](https://github.com/tinyhumansai/openhuman/tree/main/src/openhuman/agent/agents/trigger_reactor) agent runs with one or two tool calls. | A small, single-step side effect: store a memory entry, post a quick acknowledgement, mark a thread read. | +| **`escalate`** | The full **orchestrator** agent takes over with planning capability. | Anything that needs reasoning, multiple steps, or multiple skills: drafting a reply, updating several Notion pages, deciding how to triage an inbound issue. | + +The triage agent has the same memory and workspace context the rest of the agent has. It can tell whether a trigger is relevant to something you're currently working on, who the people involved are, and whether it's the kind of thing you've asked OpenHuman to act on before. + +## When a trigger turns into an agent action + +This is the part that distinguishes "OpenHuman has a Gmail integration" from "OpenHuman is on call for your inbox": + +- **`react`** is the cheap path. The Trigger Reactor is a narrow specialist with a hard budget of a couple of tool calls. It's perfect for: writing a one-line memory note that says "saw a new charge from Stripe for $84, customer X, merchant Y", silently marking a Slack message as handled because it's the same automated alert you've already triaged twice this week, or storing a structured record of an event the user might want to look up later. + +- **`escalate`** is the heavy path. When the Triage agent decides the trigger needs real work, it hands off to the Orchestrator with a self-contained task description. The orchestrator has access to your full skill surface, tools, memory, and the [Subconscious Loop](../subconscious.md) outputs. From there it might: + - Draft a reply to an important email and queue it for your approval. + - Pull up the relevant Notion / Linear / Drive context for an inbound issue and write a structured comment. + - Update three connected systems based on a single inbound event ("this customer's plan changed in Stripe, update HubSpot, post in #revenue, and add a note to their Notion file"). + - Decide the trigger means a meeting just got scheduled and pre-load the [Meeting Agent](../mascot/meeting-agents.md) for that call. + +In both cases the action runs on your machine, against your local Memory Tree, with the same model-routing and tool surface the rest of the agent uses. + +## Why a triage step at all + +It's tempting to skip the classifier and just pipe every trigger straight into the orchestrator. That's a bad idea for two reasons: + +1. **Most triggers are noise.** A connected Gmail account fires dozens of triggers an hour, the vast majority of which the user doesn't care about. Running the orchestrator on each would burn budget and produce a constant stream of background activity. +2. **Different triggers deserve different ceilings.** An automated Stripe receipt and a personal Slack DM should not cost the same number of tokens to handle. Triage lets the cheap path be cheap and reserves the orchestrator for things that earn it. + +Triage runs on the fast model tier (see [Automatic Model Routing](../model-routing/README.md)) so the classification itself is sub-second. + +## Configuration and opt-out + +- **On by default.** Once an integration is connected, its triggers feed into the pipeline automatically. +- **Opt-out.** The triage path is gated on the `OPENHUMAN_TRIGGER_TRIAGE_DISABLED` environment variable. Setting it to `1` / `true` / `yes` turns off agent classification and falls back to passive logging only. The integration itself stays connected; only the auto-action behaviour is suppressed. +- **Per-trigger settings.** Trigger settings (which integrations and event types should be evaluated) are managed under **Settings**; the underlying RPC methods are `update_composio_trigger_settings` / `get_composio_trigger_settings`. +- **Audit log.** Every trigger, regardless of decision, is written to the trigger history so you can see what arrived, what the classifier decided, and what (if anything) ran. Decisions and escalations are also published as `TriggerEvaluated` / `TriggerEscalated` events on the in-process bus, which means anything inside the core can subscribe to them. + +## Privacy boundary + +Triggers follow the same boundary as the rest of the product (see [Privacy & Security](../privacy-and-security.md)): + +- The third-party token lives on the backend, never on your laptop. +- The webhook is HMAC-verified by the backend before it reaches your machine. +- The trigger payload is processed by your local core; classification and any reaction run on your machine, against your local Memory Tree. +- Memory notes written by `acknowledge` / `react` / `escalate` paths are stored in your local SQLite memory tree and Markdown vault, the same as any other source. + +## Implementation pointers (for developers) + +- Triage agent: `src/openhuman/agent/agents/trigger_triage/` +- Reactor agent: `src/openhuman/agent/agents/trigger_reactor/` +- Composio bus subscriber: `src/openhuman/composio/bus.rs` (`ComposioTriggerSubscriber`) +- Trigger history persistence: `src/openhuman/composio/trigger_history.rs` +- Domain events: `DomainEvent::ComposioTriggerReceived`, `DomainEvent::TriggerEscalated` in `src/core/event_bus/events.rs` +- Trigger settings RPC: `update_composio_trigger_settings` / `get_composio_trigger_settings` in `src/openhuman/config/` + +## See also + +* [Third-party Integrations](README.md), the catalog of services triggers come from. +* [Auto-fetch from Integrations](../obsidian-wiki/auto-fetch.md), the polling counterpart, periodic ingest of source data into the Memory Tree. +* [Subconscious Loop](../subconscious.md), the background loop that uses trigger context and memory to plan ahead. +* [Meeting Agents](../mascot/meeting-agents.md), one place an escalated trigger can land (a calendar event with a Meet link). diff --git a/gitbooks/features/mascot.md b/gitbooks/features/mascot.md deleted file mode 100644 index b8ddfe816..000000000 --- a/gitbooks/features/mascot.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -description: >- - The on-screen face of OpenHuman — a desktop mascot that speaks, reacts, joins - your meetings, and thinks in the background even when you aren't looking at - it. -icon: face-smile ---- - -# The Mascot - -OpenHuman has a face. The mascot is an animated character that lives on your desktop and acts as the visible surface of the agent — what it's saying, what it's thinking about, when it's idle, when it's busy, when it has something to tell you. - -It is not a chrome ornament. The mascot is wired into the same pieces as the rest of the agent: voice, memory, the [subconscious loop](subconscious.md), and the [Google Meet integration](native-tools/voice.md). When the agent talks, the mascot is the one talking; when the agent is thinking, the mascot is the one thinking. - -### It speaks, and lip-syncs to its own voice - -When the agent replies, the audio is generated through a hosted TTS model and streamed to your speakers. At the same time, the mascot drives a viseme map against the audio so its mouth shapes match the words coming out. There's no separate "talking head" video — the same audio stream that you hear is the one driving the animation. - -See [Native Voice](native-tools/voice.md) for the speech-to-text, text-to-speech, and meeting plumbing the mascot rides on top of. - -### It joins your Google Meet calls — as a real participant - -The mascot is OpenHuman's flagship voice integration. When you bring it into a Google Meet: - -* It joins as a real participant via the embedded webview. -* It hears everyone in the call through streaming speech-to-text and writes structured notes into the [Memory Tree](obsidian-wiki/memory-tree.md) as the meeting progresses. -* When you ask it to speak — or it decides it has something useful to add — it generates the reply through the TTS model and **plays it back into the meeting as an outbound mic stream**, so other participants actually hear it. -* The mascot's face is **piped into the meeting as the camera feed** via a fake-video pipeline, so the other participants see the mascot moving and lip-syncing in their grid, not a black tile. - -### It moves and reacts to its surroundings - -The mascot has mood states — idle, thinking, listening, talking, surprised, dreaming — and it transitions between them based on what the agent is doing. When you start typing it shifts into a listening pose. When the model is reasoning, it shows that. When a tool call returns something noteworthy, it reacts. When you stop interacting for a while, it drifts into idle. - -It is meant to feel alive, not animated-on-rails. - -### It remembers you - -The mascot is the visible part of an agent that has the [Memory Tree](obsidian-wiki/memory-tree.md) underneath it. It remembers what you've talked about, who the people in your life are, what's open on your plate, what's been decided, and what's outstanding — across every source you've connected. When it greets you in the morning, it isn't starting from zero. - -That memory is what makes the personality consistent over weeks and months. The mascot you talk to today knows what the mascot you talked to last Tuesday knows. - -### It thinks in the background — the subconscious - -Even when you've stopped typing, the mascot keeps thinking. The [Subconscious Loop](subconscious.md) is a background tick that: - -* Loads your standing tasks and ambient goals. -* Reads the current state of your workspace and recent memory. -* Decides what to do about each one — execute autonomously, hold, or escalate to you for approval. -* Writes the outcome back to an activity log you can audit. - -So when you come back to the desk, the mascot may have already drafted the email, refreshed the dashboard, or queued the question it needs to ask you. The face on the screen is the one that did the work. - -### It dreams - -When you're away long enough, the mascot enters a dreaming state. Dreaming is the agent's offline consolidation pass — distilling the day's chunks into longer-horizon summaries, refreshing topic trees for the entities that have heated up, surfacing patterns that didn't fit any single source. The mascot animates differently while dreaming so you can tell at a glance: it's not idle, it's processing. - -When you come back, the dreams have already been folded into the Memory Tree. The mascot wakes up smarter than it went to sleep. - -### Why have a mascot at all? - -Most assistants are a blinking text input. That's fine for a tool. It's not fine for something that's meant to be alongside you all day, with persistent memory of your life, taking actions on your behalf. - -The mascot exists because: - -* **Presence beats panels.** A face you can glance at tells you, in one frame, whether the agent is busy, idle, dreaming, or trying to get your attention. -* **It makes voice calls feel like a conversation.** A camera feed of an animated character lip-syncing to its own speech is a different experience than a robotic voice with a black tile. -* **Personality is a UX surface.** A consistent character on screen is easier to trust, talk to, and forgive when it makes a mistake than a faceless API. diff --git a/gitbooks/features/mascot/README.md b/gitbooks/features/mascot/README.md new file mode 100644 index 000000000..6a86be17d --- /dev/null +++ b/gitbooks/features/mascot/README.md @@ -0,0 +1,74 @@ +--- +description: >- + The on-screen face of OpenHuman, a desktop mascot that speaks, reacts, joins + your meetings, and thinks in the background even when you aren't looking at + it. +icon: face-smile +--- + +# The Mascot + +OpenHuman has a face. The mascot is an animated character that lives on your desktop and acts as the visible surface of the agent, what it's saying, what it's thinking about, when it's idle, when it's busy, when it has something to tell you. + +It is not a chrome ornament. The mascot is wired into the same pieces as the rest of the agent: voice, memory, the [subconscious loop](../subconscious.md), and the [Google Meet integration](../native-tools/voice.md). When the agent talks, the mascot is the one talking; when the agent is thinking, the mascot is the one thinking. + +## What it does + +### It speaks, and lip-syncs to its own voice + +When the agent replies, the audio is generated through a hosted TTS model and streamed to your speakers. At the same time, the mascot drives a viseme map against the audio so its mouth shapes match the words coming out. There's no separate "talking head" video, the same audio stream that you hear is the one driving the animation. + +See [Native Voice](../native-tools/voice.md) for the speech-to-text, text-to-speech, and meeting plumbing the mascot rides on top of. + +### It joins your meetings, as a real participant + +The mascot is OpenHuman's flagship voice integration. It can join a Google Meet call as a real participant: it hears everyone, takes notes into your [Memory Tree](../obsidian-wiki/memory-tree.md), speaks back into the call when it has something to say, and pipes its own animated face into the meeting as the camera feed. + +This is the headline use case and has its own page, see [Meeting Agents](meeting-agents.md). + +### It moves and reacts to its surroundings + +The mascot has mood states (idle, thinking, listening, talking, surprised, dreaming) and it transitions between them based on what the agent is doing. When you start typing it shifts into a listening pose. When the model is reasoning, it shows that. When a tool call returns something noteworthy, it reacts. When you stop interacting for a while, it drifts into idle. + +It is meant to feel alive, not animated-on-rails. + +### It remembers you + +The mascot is the visible part of an agent that has the [Memory Tree](../obsidian-wiki/memory-tree.md) underneath it. It remembers what you've talked about, who the people in your life are, what's open on your plate, what's been decided, and what's outstanding, across every source you've connected. When it greets you in the morning, it isn't starting from zero. + +That memory is what makes the personality consistent over weeks and months. The mascot you talk to today knows what the mascot you talked to last Tuesday knows. + +### It thinks in the background, the subconscious + +Even when you've stopped typing, the mascot keeps thinking. The [Subconscious Loop](../subconscious.md) is a background tick that: + +* Loads your standing tasks and ambient goals. +* Reads the current state of your workspace and recent memory. +* Decides what to do about each one (execute autonomously, hold, or escalate to you for approval). +* Writes the outcome back to an activity log you can audit. + +So when you come back to the desk, the mascot may have already drafted the email, refreshed the dashboard, or queued the question it needs to ask you. The face on the screen is the one that did the work. + +### It dreams + +When you're away long enough, the mascot enters a dreaming state. Dreaming is the agent's offline consolidation pass, distilling the day's chunks into longer-horizon summaries, refreshing topic trees for the entities that have heated up, surfacing patterns that didn't fit any single source. The mascot animates differently while dreaming so you can tell at a glance: it's not idle, it's processing. + +When you come back, the dreams have already been folded into the Memory Tree. The mascot wakes up smarter than it went to sleep. + +## Why have a mascot at all? + +Most assistants are a blinking text input. That's fine for a tool. It's not fine for something that's meant to be alongside you all day, with persistent memory of your life, taking actions on your behalf. + +The mascot exists because: + +* **Presence beats panels.** A face you can glance at tells you, in one frame, whether the agent is busy, idle, dreaming, or trying to get your attention. +* **It makes voice calls feel like a conversation.** A camera feed of an animated character lip-syncing to its own speech is a different experience than a robotic voice with a black tile. +* **Personality is a UX surface.** A consistent character on screen is easier to trust, talk to, and forgive when it makes a mistake than a faceless API. + +## See also + +* [Meeting Agents](meeting-agents.md), the mascot in Google Meet: listening, speaking, animating, using tools. +* [Native Voice](../native-tools/voice.md), the STT / TTS plumbing the mascot rides on. +* [Memory Tree](../obsidian-wiki/memory-tree.md), what the mascot remembers, and how. +* [Subconscious Loop](../subconscious.md), what it thinks about while you're away. +* [Chromium Embedded Framework](../../developing/cef.md), the camera-into-Meet pipeline (developer reference). diff --git a/gitbooks/features/mascot/meeting-agents.md b/gitbooks/features/mascot/meeting-agents.md new file mode 100644 index 000000000..1c6333e98 --- /dev/null +++ b/gitbooks/features/mascot/meeting-agents.md @@ -0,0 +1,95 @@ +--- +description: >- + The mascot can join a Google Meet as a real participant - it listens, + takes notes, speaks back into the call, animates its face into the camera + grid, and uses tools mid-meeting. Not a notetaker. A participant. +icon: video +--- + +# Meeting Agents + +The mascot's flagship integration is the **Meeting Agent**: the same character you talk to on your desktop can join a Google Meet on your behalf, sit in the participant grid as an animated face, hear everyone in the room, talk back into the call with its own voice, and reach for tools while the meeting is happening. + +It is not a notetaker. A notetaker sits silently and produces a transcript. A meeting agent participates - it answers questions, looks things up live, remembers prior meetings with the same people, and contributes when you (or it) decide it has something useful to add. + +## What it actually does in a call + +### 1. It joins as a real participant + +The mascot joins the meeting through an embedded webview, the same way a person joins from their browser. There is a name, a face, and a tile in the grid. Other participants see and hear it the way they'd see and hear any other attendee - no calendar bot, no dial-in number, no "this meeting is being recorded by …" banner. + +Under the hood the meeting brain lives in `src/openhuman/meet_agent/brain.rs`, and the webview side is the same CEF child window OpenHuman uses for other embedded providers. + +### 2. It listens to everyone in the room + +Inbound audio from the meeting is captured and pushed through streaming speech-to-text in real time. The transcript is diarized per speaker, cleaned up by the same hallucination filter and postprocessor used for desktop dictation, and folded into the [Memory Tree](../obsidian-wiki/memory-tree.md) as the meeting unfolds - under the right people, the right topics, the right project, with backlinks the mascot can use later. + +Because the transcript is being structured live, the mascot can answer questions about *this* meeting (or any prior meeting with the same people) while it is still happening. + +### 3. It interacts - it answers, it asks, it follows up + +The agent is not muted. When you address it directly ("Ghosty, can you pull up the numbers from last quarter?"), or when it decides it has something useful to add, it generates a reply on the fly using the project's normal LLM stack and speaks it into the meeting. + +Conversational turns are routed through the fast model tier (see [Automatic Model Routing](../model-routing/README.md)) so the latency feels like talking to a person who's listening, not waiting on a chatbot. + +### 4. It speaks - its own TTS audio plays back into the call + +Replies are generated by the project's TTS stack and streamed straight into the meeting as an outbound microphone feed. It is not played through your local speakers and re-captured by your mic - it is injected directly as the agent's audio, so it lands clean for everyone else and doesn't echo through your room. + +### 5. It animates - the mascot's face IS the camera feed + +The mascot's canvas is piped into the Meet call as the outbound camera stream (the work introduced in commit `b6d05cb4`, with the mascot frame pipeline polished further in `f5dce783`). When the agent is talking, the mascot is talking on the camera tile - mouth shapes lip-sync to the same TTS audio everyone else is hearing. When it is listening, it shows the listening pose. When it is reasoning before it speaks, you see the thinking pose. + +Other participants don't see a black tile or a static avatar. They see an animated character reacting in time with what's being said, which is what makes the call feel like a conversation with something alive rather than a voice coming out of nowhere. + +### 6. It uses tools mid-meeting - this is the part a notetaker can't do + +This is the difference between a transcription bot and a meeting *agent*. + +While the call is happening, the mascot has access to the same tool surface it has on your desktop: + +* **[Memory Tree](../obsidian-wiki/memory-tree.md)** - recall prior meetings, decisions, open threads, who said what last time, what's been promised. +* **[Auto-fetch from integrations](../obsidian-wiki/auto-fetch.md)** and **[third-party integrations](../integrations/README.md)** - pull a thread from Slack, an email, a Linear ticket, a Notion doc, a calendar entry, a file from Drive. +* **[Native tools](../native-tools/README.md)** - search the web, scrape a page, run a quick code/data lookup, all without leaving the call. +* **[Subconscious Loop](../subconscious.md)** outputs - anything it has been working on in the background is already on hand. + +So when someone in the call asks "wait, didn't we decide to drop the Q3 launch last month?", the mascot doesn't just transcribe the question. It answers it - with the actual decision, the meeting it was made in, and who agreed. + +That moves it from *notetaker* to *the most informed participant in the room*. + +## Why it feels alive + +A meeting agent that only transcribes is a tool. A meeting agent that participates is a presence. The Meet integration is deliberately built to make the mascot feel like a real attendee, not a recording device: + +* It has a **face on the camera grid** that lip-syncs and reacts, not a black square or a logo. +* It has its **own voice** that plays into the call, not into your speakers. +* It has **persistent memory** of the people in the room, the project, the prior decisions - so it can be addressed by name and answer in context. +* It has **tools** so it can act on what's said, not just record it. +* It runs the **subconscious loop** between meetings - so when it joins your next call, it has already done the homework on what was promised in the last one. + +The result, in practice, is that participants stop treating it like a bot and start treating it like a colleague who happens to be very fast at looking things up. + +## Setup, controls, privacy + +* **Joining a call.** You can hand the mascot a Google Meet link from the desktop app; it will open the embedded Meet webview, join with the configured display name, and switch its camera tile to the mascot canvas. +* **Mic and camera control.** The agent's mic is the TTS injection stream, not your real microphone. The agent's camera is the mascot frame producer, not your real webcam. You can mute the agent's mic from the app at any time, the same way you'd mute yourself in Meet. +* **Transcripts and memory.** Live transcripts land in the [Memory Tree](../obsidian-wiki/memory-tree.md) the same way any other source does - under the people in the call, the project, and the topics that came up. They are local-first and follow the project's [Privacy & Security](../privacy-and-security.md) rules. +* **No covert recording.** The agent appears as a normal participant in the grid; everyone in the call can see it and see when it's speaking. + +## Implementation pointers (for developers) + +Curious how this is wired up: + +* Brain - `src/openhuman/meet_agent/brain.rs` (LLM turns, speak/no-speak decisions, tool calls). +* Voice plumbing - `src/openhuman/voice/` (STT in, TTS out, hallucination filter, postprocess). See [Native Voice](../native-tools/voice.md). +* Mascot canvas as outbound camera - `app/src/features/meet/MascotFrameProducer.tsx` and the Tauri-side `mascot_native_window.rs` window. +* Embedded Meet webview - see [Chromium Embedded Framework](../../developing/cef.md). The Meet child webview ships with **zero injected JavaScript**; everything host-side runs natively via CDP. +* Notable commits to read for context - `0bc74575` (live note-taking), `f1203479` (real LLM turns + tuned TTS), `b6d05cb4` (mascot canvas as outbound camera), `f5dce783` (mascot frame pipeline + off-screen meet window). + +## See also + +* [The Mascot](README.md) - the on-screen character itself, outside of meetings. +* [Native Voice](../native-tools/voice.md) - STT / TTS that the meeting agent rides on. +* [Memory Tree](../obsidian-wiki/memory-tree.md) - where transcripts and decisions land. +* [Native Tools](../native-tools/README.md) - what the mascot can reach for mid-call. +* [Automatic Model Routing](../model-routing/README.md) - why conversational turns feel low-latency. diff --git a/gitbooks/features/model-routing/README.md b/gitbooks/features/model-routing/README.md index 2bb18db2a..0c7e77691 100644 --- a/gitbooks/features/model-routing/README.md +++ b/gitbooks/features/model-routing/README.md @@ -1,6 +1,6 @@ --- description: >- - One subscription, many models. Tasks pick their model via hint prefixes — + One subscription, many models. Tasks pick their model via hint prefixes: reasoning goes to a strong model, fast paths go to a fast one, vision to vision. icon: route --- diff --git a/gitbooks/features/model-routing/local-ai.md b/gitbooks/features/model-routing/local-ai.md index 2435d5f61..3c722f582 100644 --- a/gitbooks/features/model-routing/local-ai.md +++ b/gitbooks/features/model-routing/local-ai.md @@ -15,11 +15,11 @@ This is a deliberate scoping. The previous design tried to put chat, vision, STT | Workload | Default model | Implementation | | ------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -| **Memory embeddings** | `all-minilm:latest` | `src/openhuman/embeddings/ollama.rs` — used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search. | -| **Summary-tree building** | `gemma3:1b-it-qat` (configurable) | `src/openhuman/tree_summarizer/ops.rs` — source / topic / global summary builders for the Memory Tree. | -| **Heartbeat loop** | small chat model | `src/openhuman/heartbeat/` — periodic background reflection. | -| **Learning / reflection** | small chat model | `src/openhuman/learning/reflection.rs` — passes that consolidate what was learned. | -| **Subconscious** | small chat model | `src/openhuman/subconscious/executor.rs` — background evaluation loop. | +| **Memory embeddings** | `all-minilm:latest` | `src/openhuman/embeddings/ollama.rs` - used by the [Memory Tree](../obsidian-wiki/memory-tree.md) for vector search. | +| **Summary-tree building** | `gemma3:1b-it-qat` (configurable) | `src/openhuman/tree_summarizer/ops.rs` - source / topic / global summary builders for the Memory Tree. | +| **Heartbeat loop** | small chat model | `src/openhuman/heartbeat/` - periodic background reflection. | +| **Learning / reflection** | small chat model | `src/openhuman/learning/reflection.rs` - passes that consolidate what was learned. | +| **Subconscious** | small chat model | `src/openhuman/subconscious/executor.rs` - background evaluation loop. | Each of these is a **per-feature opt-in flag**. Turning on local AI does not silently route everything through it, you choose the workloads. diff --git a/gitbooks/features/native-tools/README.md b/gitbooks/features/native-tools/README.md index 1919a84b2..a905f171c 100644 --- a/gitbooks/features/native-tools/README.md +++ b/gitbooks/features/native-tools/README.md @@ -1,6 +1,6 @@ --- description: >- - The full toolset OpenHuman's agent has out of the box — research, code, + The full toolset OpenHuman's agent has out of the box - research, code, control your machine, schedule jobs, talk back to you, and call into 118+ third-party services. icon: toolbox @@ -8,7 +8,7 @@ icon: toolbox # Native Tools -OpenHuman's agent doesn't ship empty. Every model behind the agent has a curated set of tools available the moment you install — no plugin marketplace, no API keys to wire up, no MCP servers to register. The whole toolbelt is in the box. +OpenHuman's agent doesn't ship empty. Every model behind the agent has a curated set of tools available the moment you install - no plugin marketplace, no API keys to wire up, no MCP servers to register. The whole toolbelt is in the box. This page is the index. Each subpage covers one family of tools. @@ -19,25 +19,25 @@ A plugin-only model means tools live in different processes, behind RPC, with th * Consistent error handling. * Zero install friction. * All output passes through [Smart Token Compression](../token-compression.md) for free. -* Predictable security boundary — filesystem tools respect workspace scoping, network tools go through the OpenHuman proxy. +* Predictable security boundary - filesystem tools respect workspace scoping, network tools go through the OpenHuman proxy. ## The toolbelt | Family | What it covers | | ------ | -------------- | | [Web Search](web-search.md) | Search the live web without bringing your own API key. | -| [Web Scraper](web-scraper.md) | Pull clean text out of any URL — articles, docs, READMEs. | +| [Web Scraper](web-scraper.md) | Pull clean text out of any URL - articles, docs, READMEs. | | [Coder](coder.md) | Read/write/edit/patch files, glob, grep, git, lint, test. | | [Browser & Computer Control](browser-and-computer.md) | Open URLs, screenshot, click, type, move the mouse. | | [Cron & Scheduling](cron.md) | Recurring jobs, one-off reminders, scheduled agent runs. | | [Voice](voice.md) | Speech-to-text in, text-to-speech out, live Google Meet agent. | | [Memory Tools](memory-tools.md) | Recall, store, forget, and search the [Memory Tree](../obsidian-wiki/memory-tree.md). | -| [Third-party Integrations](integrations.md) | The agent's view of the [118+ connected services](../integrations.md). | +| [Third-party Integrations](../integrations/README.md) | The agent's view of the [118+ connected services](../integrations/README.md). | | [Agent Coordination](agent-coordination.md) | Spawn subagents, delegate to skills, plan, ask the user. | | [System & Utilities](system-and-utilities.md) | Shell, node, SQL, current time, push notifications, LSP. | ## See also -* [Smart Token Compression](../token-compression.md) — what keeps tool output costs bounded. -* [Third-party Integrations](../integrations.md) — the user-facing pitch and OAuth flow for the 118+ catalog. -* [Privacy & Security](../privacy-and-security.md) — the boundary every tool runs inside. +* [Smart Token Compression](../token-compression.md) - what keeps tool output costs bounded. +* [Third-party Integrations](../integrations/README.md) - the user-facing pitch and OAuth flow for the 118+ catalog. +* [Privacy & Security](../privacy-and-security.md) - the boundary every tool runs inside. diff --git a/gitbooks/features/native-tools/agent-coordination.md b/gitbooks/features/native-tools/agent-coordination.md index a2d126fc1..fb0a1a683 100644 --- a/gitbooks/features/native-tools/agent-coordination.md +++ b/gitbooks/features/native-tools/agent-coordination.md @@ -5,7 +5,7 @@ icon: sitemap # Agent Coordination -Beyond doing the work, the agent has tools for *organising* the work — planning multi-step jobs, delegating to specialists, spawning subagents, and pausing to ask the user when something is genuinely ambiguous. +Beyond doing the work, the agent has tools for *organising* the work - planning multi-step jobs, delegating to specialists, spawning subagents, and pausing to ask the user when something is genuinely ambiguous. ## Tools in the family @@ -15,8 +15,8 @@ Beyond doing the work, the agent has tools for *organising* the work — plannin | `spawn_subagent` | Spin up a fresh agent with its own context window for a self-contained subtask. | | `spawn_worker_thread` | Background work that doesn't need to block the main conversation. | | `delegate` | Hand a task to a specialist (e.g. an archetype with different prompts/tools/permissions). | -| `archetype_delegation` | Route to a named archetype — coder, researcher, planner, etc. | -| `skill_delegation` | Hand off to a [skill](../integrations.md#skills) installed in the workspace. | +| `archetype_delegation` | Route to a named archetype - coder, researcher, planner, etc. | +| `skill_delegation` | Hand off to a [skill](../integrations/README.md#skills) installed in the workspace. | | `ask_clarification` | Pause and ask the user a precise question instead of guessing. | | `plan_exit` | Exit a planning phase and start executing. | | `check_onboarding_status` / `complete_onboarding` | Gate behaviour on whether the user has finished onboarding. | @@ -25,7 +25,7 @@ Beyond doing the work, the agent has tools for *organising* the work — plannin Long tasks fall apart when the agent tries to keep everything in one head. Splitting work via TODOs and subagents means: -* Each subagent gets a clean context — fewer tokens, fewer distractions. +* Each subagent gets a clean context - fewer tokens, fewer distractions. * The main thread keeps a high-level view of progress. * Failures in one branch don't poison the rest. @@ -33,5 +33,5 @@ Asking for clarification is a tool too, on purpose: it makes "I should ask the u ## See also -* [Coder](coder.md) — what a coder-archetype subagent typically uses. -* [Subconscious Loop](../subconscious.md) — the always-on background agent thread. +* [Coder](coder.md) - what a coder-archetype subagent typically uses. +* [Subconscious Loop](../subconscious.md) - the always-on background agent thread. diff --git a/gitbooks/features/native-tools/browser-and-computer.md b/gitbooks/features/native-tools/browser-and-computer.md index 0cc6ff945..153a12526 100644 --- a/gitbooks/features/native-tools/browser-and-computer.md +++ b/gitbooks/features/native-tools/browser-and-computer.md @@ -1,11 +1,11 @@ --- -description: Open URLs, take screenshots, click, type, and move the mouse — natively. +description: Open URLs, take screenshots, click, type, and move the mouse - natively. icon: display --- # Browser & Computer Control -When the agent needs to *use* your machine the way a person would — open a page, screenshot it, click a button, type a phrase — these tools are how it does it. +When the agent needs to *use* your machine the way a person would - open a page, screenshot it, click a button, type a phrase - these tools are how it does it. ## Browser @@ -17,17 +17,17 @@ The browser surface runs through CEF (Chromium Embedded Framework) and includes ## Computer (mouse + keyboard) -* **Mouse** — move, click, drag. -* **Keyboard** — type text, send key chords. -* **Human path** — moves and clicks follow human-like trajectories rather than teleporting, so they don't trip naive bot detection. +* **Mouse** - move, click, drag. +* **Keyboard** - type text, send key chords. +* **Human path** - moves and clicks follow human-like trajectories rather than teleporting, so they don't trip naive bot detection. ## What it's good for -* Driving sites that don't have an API or a [native integration](integrations.md). +* Driving sites that don't have an API or a [native integration](../integrations/README.md). * Multi-step UI flows where a single screenshot isn't enough. * Automating local apps from inside a chat. ## See also -* [Web Scraper](web-scraper.md) — when you only need the article, not the whole page. -* [Chromium Embedded Framework](../../developing/cef.md) — the runtime browser layer. +* [Web Scraper](web-scraper.md) - when you only need the article, not the whole page. +* [Chromium Embedded Framework](../../developing/cef.md) - the runtime browser layer. diff --git a/gitbooks/features/native-tools/coder.md b/gitbooks/features/native-tools/coder.md index 752a20e87..25038449c 100644 --- a/gitbooks/features/native-tools/coder.md +++ b/gitbooks/features/native-tools/coder.md @@ -1,5 +1,5 @@ --- -description: A complete toolset for working on real codebases — read, write, edit, search, git, lint, test. +description: A complete toolset for working on real codebases - read, write, edit, search, git, lint, test. icon: code --- @@ -13,7 +13,7 @@ The coder family is what makes OpenHuman a viable coding partner instead of a ch | ---------------- | ----------------------------------------------------------------- | | `file_read` | Read a file (with line numbers, like `cat -n`). | | `file_write` | Write a new file. | -| `edit_file` | Targeted edits — match-and-replace with strict uniqueness checks. | +| `edit_file` | Targeted edits - match-and-replace with strict uniqueness checks. | | `apply_patch` | Apply a unified diff. | | `glob_search` | Find files by glob pattern. | | `grep` | Ripgrep-style search across the tree. | @@ -35,9 +35,9 @@ A shell tool plus `cat`/`sed`/`awk` could *technically* do all of this. The nati ## Workspace scoping -Filesystem tools respect a workspace boundary — the agent can't read or write outside it without explicit permission. Same boundary the rest of the app uses for `OPENHUMAN_WORKSPACE`. +Filesystem tools respect a workspace boundary - the agent can't read or write outside it without explicit permission. Same boundary the rest of the app uses for `OPENHUMAN_WORKSPACE`. ## See also -* [System & Utilities](system-and-utilities.md) — `shell`, `node_exec`, `npm_exec` for the rest of the dev loop. -* [Agent Coordination](agent-coordination.md) — `todo_write`, `spawn_subagent` for larger refactors. +* [System & Utilities](system-and-utilities.md) - `shell`, `node_exec`, `npm_exec` for the rest of the dev loop. +* [Agent Coordination](agent-coordination.md) - `todo_write`, `spawn_subagent` for larger refactors. diff --git a/gitbooks/features/native-tools/cron.md b/gitbooks/features/native-tools/cron.md index a07964527..1ff5008d7 100644 --- a/gitbooks/features/native-tools/cron.md +++ b/gitbooks/features/native-tools/cron.md @@ -1,5 +1,5 @@ --- -description: Recurring jobs, one-off reminders, and scheduled agent runs — first-class. +description: Recurring jobs, one-off reminders, and scheduled agent runs - first-class. icon: clock --- @@ -11,12 +11,12 @@ Scheduling is a first-class capability, not a workaround. The agent can set up r | Tool | What it does | | ------------- | ------------------------------------------------------------------ | -| `cron_add` | Create a new scheduled job — cron expression + agent prompt. | +| `cron_add` | Create a new scheduled job - cron expression + agent prompt. | | `cron_list` | List existing jobs and their next-run times. | -| `cron_update` | Edit an existing job — change schedule, prompt, or enabled state. | +| `cron_update` | Edit an existing job - change schedule, prompt, or enabled state. | | `cron_remove` | Delete a job. | | `cron_run` | Run a job once, immediately, regardless of its schedule. | -| `cron_runs` | Inspect the recent run history — when, how long, what it produced. | +| `cron_runs` | Inspect the recent run history - when, how long, what it produced. | There's also a one-shot `schedule` tool in [System & Utilities](system-and-utilities.md) for "do this once at time T" cases that don't need a recurring entry. @@ -25,13 +25,13 @@ There's also a one-shot `schedule` tool in [System & Utilities](system-and-utili * Daily / weekly digests delivered to your messaging channel of choice. * Polling a slow integration that doesn't push events. * Reminders the agent itself owns ("remind me Thursday to follow up with Alice"). -* Recurring research — "every Monday, check what's new on this topic and write me a brief". +* Recurring research - "every Monday, check what's new on this topic and write me a brief". ## How it ties back to the rest -Every cron run is just a normal agent invocation, so it can use any other tool — search the web, query the [Memory Tree](../obsidian-wiki/memory-tree.md), call a [third-party integration](integrations.md), send a message. Run history is recorded so you can see what each tick produced. +Every cron run is just a normal agent invocation, so it can use any other tool - search the web, query the [Memory Tree](../obsidian-wiki/memory-tree.md), call a [third-party integration](../integrations/README.md), send a message. Run history is recorded so you can see what each tick produced. ## See also -* [System & Utilities](system-and-utilities.md) — the one-shot `schedule` tool. -* [Agent Coordination](agent-coordination.md) — for jobs that fan out into subagents. +* [System & Utilities](system-and-utilities.md) - the one-shot `schedule` tool. +* [Agent Coordination](agent-coordination.md) - for jobs that fan out into subagents. diff --git a/gitbooks/features/native-tools/integrations.md b/gitbooks/features/native-tools/integrations.md index 8095ab3c7..d463b5d17 100644 --- a/gitbooks/features/native-tools/integrations.md +++ b/gitbooks/features/native-tools/integrations.md @@ -5,11 +5,11 @@ icon: plug # Third-party Integrations -OpenHuman's agent can call into [118+ third-party services](../integrations.md) — Gmail, Notion, GitHub, Slack, Stripe, Calendar, and the long tail — through a single proxied tool surface. +OpenHuman's agent can call into [118+ third-party services](../integrations/README.md) - Gmail, Notion, GitHub, Slack, Stripe, Calendar, and the long tail - through a single proxied tool surface. ## How it shows up to the agent -Once you've connected a service via OAuth, its actions become callable tools. The agent doesn't need to know whether a tool talks to Gmail or to a local file — it just calls the tool, the proxy routes the request through the OpenHuman backend with your token, and the result comes back like any other tool output. +Once you've connected a service via OAuth, its actions become callable tools. The agent doesn't need to know whether a tool talks to Gmail or to a local file - it just calls the tool, the proxy routes the request through the OpenHuman backend with your token, and the result comes back like any other tool output. A few examples of what becomes available: @@ -20,7 +20,7 @@ A few examples of what becomes available: ## Native vs proxied -Some services have **native providers** — Rust modules that know how to ingest the service into the [Memory Tree](../obsidian-wiki/memory-tree.md) directly (e.g. Gmail's native ingest path). Others are exposed as **proxied tools** only: the agent can call them, but there's no automatic ingest yet. New native providers are added as features land. +Some services have **native providers** - Rust modules that know how to ingest the service into the [Memory Tree](../obsidian-wiki/memory-tree.md) directly (e.g. Gmail's native ingest path). Others are exposed as **proxied tools** only: the agent can call them, but there's no automatic ingest yet. New native providers are added as features land. ## Privacy boundary @@ -28,6 +28,6 @@ OpenHuman's core never calls any third-party API directly. All requests go throu ## See also -* [Third-party Integrations (catalog)](../integrations.md) — the user-facing pitch, OAuth flow, and connection management. -* [Auto-fetch](../obsidian-wiki/auto-fetch.md) — how connected services flow into the Memory Tree. -* [Privacy & Security](../privacy-and-security.md) — the full boundary. +* [Third-party Integrations (catalog)](../integrations/README.md) - the user-facing pitch, OAuth flow, and connection management. +* [Auto-fetch](../obsidian-wiki/auto-fetch.md) - how connected services flow into the Memory Tree. +* [Privacy & Security](../privacy-and-security.md) - the full boundary. diff --git a/gitbooks/features/native-tools/memory-tools.md b/gitbooks/features/native-tools/memory-tools.md index 58b6db7c9..02b7ab892 100644 --- a/gitbooks/features/native-tools/memory-tools.md +++ b/gitbooks/features/native-tools/memory-tools.md @@ -11,17 +11,17 @@ The [Memory Tree](../obsidian-wiki/memory-tree.md) is OpenHuman's knowledge base | Tool | What it does | | -------- | ----------------------------------------------------------------------------------------------------------- | -| `recall` | Search the Memory Tree by query — source-scoped, topic-scoped, or global. Returns chunks with provenance. | +| `recall` | Search the Memory Tree by query - source-scoped, topic-scoped, or global. Returns chunks with provenance. | | `store` | Write a new memory the agent decided is worth keeping (a fact, a preference, a piece of context). | -| `forget` | Remove a memory by ID — used when something is wrong, stale, or the user explicitly asks to forget it. | +| `forget` | Remove a memory by ID - used when something is wrong, stale, or the user explicitly asks to forget it. | -There is also a tree-aware retrieval surface (drill down a topic, fetch the global digest for a day) — the agent picks the right one based on the question. +There is also a tree-aware retrieval surface (drill down a topic, fetch the global digest for a day) - the agent picks the right one based on the question. ## Why these are tools, not implicit context -The Memory Tree is too big to dump into every conversation. The tools let the model *ask* — "what do I know about Alice?", "what happened yesterday?", "remind me of last week's Stripe webhook" — and the retrieval layer returns just the relevant chunks, with provenance back to the source file in your Obsidian vault. +The Memory Tree is too big to dump into every conversation. The tools let the model *ask* - "what do I know about Alice?", "what happened yesterday?", "remind me of last week's Stripe webhook" - and the retrieval layer returns just the relevant chunks, with provenance back to the source file in your Obsidian vault. ## See also -* [Memory Tree](../obsidian-wiki/memory-tree.md) — what these tools read from and write to. -* [Auto-fetch](../obsidian-wiki/auto-fetch.md) — how the tree gets populated in the first place. +* [Memory Tree](../obsidian-wiki/memory-tree.md) - what these tools read from and write to. +* [Auto-fetch](../obsidian-wiki/auto-fetch.md) - how the tree gets populated in the first place. diff --git a/gitbooks/features/native-tools/system-and-utilities.md b/gitbooks/features/native-tools/system-and-utilities.md index ef5e4736c..41211e08f 100644 --- a/gitbooks/features/native-tools/system-and-utilities.md +++ b/gitbooks/features/native-tools/system-and-utilities.md @@ -1,5 +1,5 @@ --- -description: Shell, node, SQL, current time, push notifications — the small tools that round out the toolbelt. +description: Shell, node, SQL, current time, push notifications - the small tools that round out the toolbelt. icon: gear --- @@ -12,16 +12,16 @@ The catch-all family. Small, sharp tools the agent reaches for to round out a ta | Tool | What it does | | ------------------- | ----------------------------------------------------------------------------- | | `shell` | Run a shell command. Bounded output, captured exit code. | -| `node_exec` | Run a Node.js snippet — useful for one-off scripting. | +| `node_exec` | Run a Node.js snippet - useful for one-off scripting. | | `npm_exec` | Run an `npm`/`pnpm`/`yarn` script. | | `current_time` | Get the current time in any timezone, with formatting options. | -| `schedule` | One-shot "do this once at time T" — for recurring jobs see [Cron](cron.md). | +| `schedule` | One-shot "do this once at time T" - for recurring jobs see [Cron](cron.md). | | `pushover` | Send a push notification to your devices. | | `insert_sql_record` | Append a row to the agent's structured workspace SQL store. | | `lsp` | Query a language server (definitions, references, diagnostics). | -| `workspace_state` | Inspect the current workspace — open files, recent edits, environment. | +| `workspace_state` | Inspect the current workspace - open files, recent edits, environment. | | `proxy_config` | Read or change proxy configuration for outbound requests. | -| `tool_stats` | Self-reflection — what tools have been used in this session and how often. | +| `tool_stats` | Self-reflection - what tools have been used in this session and how often. | ## What it's good for @@ -32,5 +32,5 @@ The catch-all family. Small, sharp tools the agent reaches for to round out a ta ## See also -* [Coder](coder.md) — for filesystem-heavy work, prefer the dedicated tools over `shell`. -* [Cron & Scheduling](cron.md) — for anything recurring. +* [Coder](coder.md) - for filesystem-heavy work, prefer the dedicated tools over `shell`. +* [Cron & Scheduling](cron.md) - for anything recurring. diff --git a/gitbooks/features/native-tools/voice.md b/gitbooks/features/native-tools/voice.md index e41ee07ec..81417ceaf 100644 --- a/gitbooks/features/native-tools/voice.md +++ b/gitbooks/features/native-tools/voice.md @@ -1,6 +1,6 @@ --- description: >- - Native voice — speech-to-text in, text-to-speech out, mascot lip-sync, + Native voice - speech-to-text in, text-to-speech out, mascot lip-sync, and a live Google Meet agent that listens and speaks. icon: microphone --- @@ -11,11 +11,11 @@ OpenHuman is voice-first when you want it to be. STT, TTS, and the live Google M ## Speech-to-text -* **Hotkey** — push-to-talk and toggle modes. -* **Audio capture** — cross-platform mic capture with voice-activity detection. -* **Streaming transcription** — words appear as you speak. -* **Hallucination filter** — strips well-known artefacts ("Thanks for watching", silence-induced phrases). -* **Postprocess** — punctuation, capitalisation, dictation cleanup. +* **Hotkey** - push-to-talk and toggle modes. +* **Audio capture** - cross-platform mic capture with voice-activity detection. +* **Streaming transcription** - words appear as you speak. +* **Hallucination filter** - strips well-known artefacts ("Thanks for watching", silence-induced phrases). +* **Postprocess** - punctuation, capitalisation, dictation cleanup. Dictation can replace the active text input on your desktop, or be sent straight into a chat with the agent. @@ -34,10 +34,10 @@ OpenHuman's flagship voice integration: ## Privacy * Audio capture is local. Streaming STT goes through the OpenHuman backend; no recording is retained beyond the live transcript. -* TTS audio is streamed and discarded — nothing stored. +* TTS audio is streamed and discarded - nothing stored. * Meeting transcripts land in your local memory tree, like any other source. ## See also -* [Memory Tree](../obsidian-wiki/memory-tree.md) — where Meet transcripts and notes live. -* [Automatic Model Routing](../model-routing/) — Meet's brain uses `hint:fast` for low-latency conversational turns. +* [Memory Tree](../obsidian-wiki/memory-tree.md) - where Meet transcripts and notes live. +* [Automatic Model Routing](../model-routing/) - Meet's brain uses `hint:fast` for low-latency conversational turns. diff --git a/gitbooks/features/native-tools/web-scraper.md b/gitbooks/features/native-tools/web-scraper.md index f14b091ee..2433d57af 100644 --- a/gitbooks/features/native-tools/web-scraper.md +++ b/gitbooks/features/native-tools/web-scraper.md @@ -5,7 +5,7 @@ icon: globe # Web Scraper -A purpose-built fetch tool, separate from generic `http_request` / `curl`. It exists because the agent doesn't want raw HTML — it wants the *article*. +A purpose-built fetch tool, separate from generic `http_request` / `curl`. It exists because the agent doesn't want raw HTML - it wants the *article*. ## What it does @@ -15,8 +15,8 @@ A purpose-built fetch tool, separate from generic `http_request` / `curl`. It ex ## Guardrails -* Caps response at 1 MB — large pages get truncated, not silently dropped. -* 20-second timeout — slow servers don't stall the conversation. +* Caps response at 1 MB - large pages get truncated, not silently dropped. +* 20-second timeout - slow servers don't stall the conversation. * Subject to the same proxy and URL-guard rules as other network tools. ## What it's good for @@ -27,5 +27,5 @@ A purpose-built fetch tool, separate from generic `http_request` / `curl`. It ex ## See also -* [Web Search](web-search.md) — find URLs to feed into the scraper. -* [Smart Token Compression](../token-compression.md) — what trims long pages before they hit the model. +* [Web Search](web-search.md) - find URLs to feed into the scraper. +* [Smart Token Compression](../token-compression.md) - what trims long pages before they hit the model. diff --git a/gitbooks/features/native-tools/web-search.md b/gitbooks/features/native-tools/web-search.md index 0cb4ba2b9..41487935d 100644 --- a/gitbooks/features/native-tools/web-search.md +++ b/gitbooks/features/native-tools/web-search.md @@ -1,5 +1,5 @@ --- -description: A native search tool the agent can call directly — no API key required. +description: A native search tool the agent can call directly - no API key required. icon: magnifying-glass --- @@ -9,9 +9,9 @@ The agent can search the live web on its own. Backed by a server-side proxy (Par ## What it's good for -* Research — "what's the latest on X". -* Citation hunting — "find me three sources for Y". -* Fact-checking before answering — the agent runs a quick search if it isn't confident. +* Research - "what's the latest on X". +* Citation hunting - "find me three sources for Y". +* Fact-checking before answering - the agent runs a quick search if it isn't confident. ## How it differs from generic HTTP @@ -19,5 +19,5 @@ A pure `http_request` tool can fetch a URL but can't *find* one. Web Search is t ## See also -* [Web Scraper](web-scraper.md) — fetch and clean a specific URL. -* [Smart Token Compression](../token-compression.md) — search snippets are compressed before they hit the model. +* [Web Scraper](web-scraper.md) - fetch and clean a specific URL. +* [Smart Token Compression](../token-compression.md) - search snippets are compressed before they hit the model. diff --git a/gitbooks/features/obsidian-wiki/auto-fetch.md b/gitbooks/features/obsidian-wiki/auto-fetch.md index 84243ab27..232f9783b 100644 --- a/gitbooks/features/obsidian-wiki/auto-fetch.md +++ b/gitbooks/features/obsidian-wiki/auto-fetch.md @@ -11,25 +11,16 @@ Most "AI assistants" are reactive: you ask, they think, they answer. OpenHuman i ## How it works -A single periodic scheduler ticks every twenty minutes. On each tick it walks every active [integration](../integrations.md), looks up the matching native provider, and, if enough time has elapsed since that connection's last sync, calls `provider.sync(ctx, SyncReason::Periodic)`. +A single periodic scheduler ticks every twenty minutes. On each tick it walks every active [integration](../integrations/README.md), looks up the matching native provider, and, if enough time has elapsed since that connection's last sync, calls `provider.sync(ctx, SyncReason::Periodic)`. -``` -┌──────────────────────────────────────────────────────────┐ -│ every 20 min │ -│ │ │ -│ ▼ │ -│ for each active connection (Gmail, Notion, GitHub, …): │ -│ │ │ -│ ├─ check sync_state (toolkit, connection_id) │ -│ │ • last sync timestamp │ -│ │ • daily budget │ -│ │ • dedup set │ -│ │ • cursor │ -│ │ │ -│ ├─ if interval elapsed → provider.sync() │ -│ │ │ -│ └─ on success → record_sync_success(ts) │ -└──────────────────────────────────────────────────────────┘ +```mermaid +flowchart TD + Tick["every 20 min"] --> Loop["for each active connection
(Gmail, Notion, GitHub, …)"] + Loop --> State["check sync_state (toolkit, connection_id)
• last sync timestamp
• daily budget
• dedup set
• cursor"] + State --> Gate{"interval elapsed?"} + Gate -- yes --> Sync["provider.sync()"] + Gate -- no --> Skip["skip this connection"] + Sync --> Record["record_sync_success(ts)"] ``` A few things matter here: @@ -57,6 +48,6 @@ The original design ran at 60 seconds. With several connected providers, that me ## See also -* [Third-party Integrations](../integrations.md). the connector layer auto-fetch runs on top of. +* [Third-party Integrations](../integrations/README.md). the connector layer auto-fetch runs on top of. * [Memory Tree](memory-tree.md). where everything ends up. * [Smart Token Compression](../token-compression.md). what keeps "fetch everything" cheap. diff --git a/gitbooks/features/obsidian-wiki/memory-tree.md b/gitbooks/features/obsidian-wiki/memory-tree.md index 9efb3b789..0d5efc51e 100644 --- a/gitbooks/features/obsidian-wiki/memory-tree.md +++ b/gitbooks/features/obsidian-wiki/memory-tree.md @@ -7,38 +7,27 @@ icon: tree # Memory Tree -The Memory Tree is OpenHuman's knowledge base. It is not a vector database with a thin "memory" wrapper. It is a deterministic, bucket-sealed pipeline that turns the messy stream of your day — chats, emails, documents, integration sync results — into structured, queryable, summary-backed Markdown that lives on your machine. +The Memory Tree is OpenHuman's knowledge base. It is not a vector database with a thin "memory" wrapper. It is a deterministic, bucket-sealed pipeline that turns the messy stream of your day - chats, emails, documents, integration sync results - into structured, queryable, summary-backed Markdown that lives on your machine. ## What it does Every source you connect feeds the same pipeline: -``` -source adapters (chat / email / document) - │ - ▼ -canonicalize ── normalised Markdown + provenance metadata - │ - ▼ -chunker ── deterministic IDs, ≤3k-token bounded segments - │ - ▼ -content_store ── atomic .md files on disk (body + tags) - │ - ▼ -store ── persistence (chunks, scores, summaries, jobs) - │ - ▼ -score ── signals + embeddings + entity extraction - │ - ▼ -source / topic / global trees ── per-scope summary trees - │ - ▼ -retrieval ── search · drill_down · topic · global · fetch +```mermaid +flowchart TD + Adapters["source adapters
(chat / email / document)"] + Canon["canonicalize
normalised Markdown + provenance metadata"] + Chunker["chunker
deterministic IDs, ≤3k-token bounded segments"] + ContentStore["content_store
atomic .md files on disk (body + tags)"] + Store["store
persistence (chunks, scores, summaries, jobs)"] + Score["score
signals + embeddings + entity extraction"] + Trees["source / topic / global trees
per-scope summary trees"] + Retrieval["retrieval
search · drill_down · topic · global · fetch"] + + Adapters --> Canon --> Chunker --> ContentStore --> Store --> Score --> Trees --> Retrieval ``` -The hot path (canonicalize → chunk → fast-score → persist → enqueue follow-up work) is fast. Heavy work — embeddings, entity extraction, sealing summary buckets, daily digests — runs in background workers so the UI never blocks. +The hot path (canonicalize → chunk → fast-score → persist → enqueue follow-up work) is fast. Heavy work - embeddings, entity extraction, sealing summary buckets, daily digests - runs in background workers so the UI never blocks. Embeddings and summary-tree building can run **on-device via Ollama** if you turn on [Local AI](../model-routing/local-ai.md); otherwise they go through the OpenHuman backend like any other model call. @@ -57,7 +46,7 @@ Inside your workspace (default `~/.openhuman`, or whatever `OPENHUMAN_WORKSPACE` | Path | What's there | | ----------------------- | --------------------------------------------------------------- | | `memory_tree/chunks.db` | Chunks, scores, summaries, entity index, jobs, hotness | -| `wiki/` | The Markdown vault — see [Obsidian Wiki](./) | +| `wiki/` | The Markdown vault - see [Obsidian Wiki](./) | Everything is local. Nothing about your raw data leaves your machine unless you explicitly send a chat message that includes it. @@ -78,7 +67,7 @@ The user-facing pitch is simple: connect a source, the agent gets persistent mem The diagram below is the source of truth. {% file src="../../.gitbook/assets/memory-tree-pipeline (1).excalidraw" %} -Memory Tree Async Pipeline — leaf ingestion → jobs queue → workers → source / topic / global tree building. +Memory Tree Async Pipeline - leaf ingestion → jobs queue → workers → source / topic / global tree building. {% endfile %} ### 1. Ingest @@ -88,7 +77,7 @@ A new chat / email / document arrives. The hot path canonicalizes it into Markdo Three properties matter here: * **Deterministic.** Chunk IDs are content-addressed, so re-running ingest on identical input never produces duplicates. -* **Fast.** No LLM calls in this lane — only cheap heuristics. +* **Fast.** No LLM calls in this lane - only cheap heuristics. * **Bounded write.** Everything happens in one transaction so a partial ingest can't leave dangling rows. ### 2. Queue @@ -114,22 +103,23 @@ On startup, any job whose worker lease has expired (because of a crash or kill) Three independent trees are built from the same leaf stream. -* **Source tree** — one per source. New leaves land in the L0 buffer; when the buffer fills (or a stale-flush fires), a `seal` writes an L1 summary, and the cascade continues up. -* **Topic tree** — one per high-hotness entity. The router checks whether an entity is hot enough to deserve its own tree and, if so, appends to its buffer. -* **Global tree** — one tree, growing one node per UTC day, walked up the hierarchy as days accumulate. +* **Source tree** - one per source. New leaves land in the L0 buffer; when the buffer fills (or a stale-flush fires), a `seal` writes an L1 summary, and the cascade continues up. +* **Topic tree** - one per high-hotness entity. The router checks whether an entity is hot enough to deserve its own tree and, if so, appends to its buffer. +* **Global tree** - one tree, growing one node per UTC day, walked up the hierarchy as days accumulate. ### 5. Scheduler -A scheduler loop runs independently of the ingest path. At 00:00 UTC each day it enqueues a global daily digest for yesterday and a stale-flush for today. The scheduler **does not** run summarizers itself — everything goes through the queue, so retries, dedupe, and stale-lock recovery stay centralized. +A scheduler loop runs independently of the ingest path. At 00:00 UTC each day it enqueues a global daily digest for yesterday and a stale-flush for today. The scheduler **does not** run summarizers itself - everything goes through the queue, so retries, dedupe, and stale-lock recovery stay centralized. ### 6. Leaf lifecycle Each chunk moves through a small state machine: -``` -pending_extraction ──► admitted ──► buffered ──► sealed - ╲ - ──► dropped +```mermaid +flowchart LR + Pending["pending_extraction"] --> Admitted["admitted"] + Pending --> Dropped["dropped"] + Admitted --> Buffered["buffered"] --> Sealed["sealed"] ``` * Extraction decides `admitted` vs `dropped` based on the deep score. @@ -141,17 +131,17 @@ This is why retrieval can show provenance without re-running the pipeline: the c ### Why a queue instead of in-process futures -* **Crash safety.** A worker panic, a process kill, a power loss — none of them lose admitted-but-not-yet-sealed work. The next start picks up where the last one left off. +* **Crash safety.** A worker panic, a process kill, a power loss - none of them lose admitted-but-not-yet-sealed work. The next start picks up where the last one left off. * **Retries with backoff.** Per-job retries with attempt counts and scheduled re-runs, no ad-hoc retry loops in business logic. * **One throttle for LLM cost.** All summarization paths share a single semaphore. ## Triggering ingest -* **Automatic** — every active integration is auto-fetched every twenty minutes; see [Auto-fetch](auto-fetch.md). -* **Manual** — the Memory tab in the desktop app exposes a "Run ingest" trigger per source. -* **RPC** — `openhuman.memory_tree_ingest` for advanced workflows. +* **Automatic** - every active integration is auto-fetched every twenty minutes; see [Auto-fetch](auto-fetch.md). +* **Manual** - the Memory tab in the desktop app exposes a "Run ingest" trigger per source. +* **RPC** - `openhuman.memory_tree_ingest` for advanced workflows. -## In the desktop app — the Intelligence tab +## In the desktop app - the Intelligence tab Open it from the bottom navigation bar. @@ -167,7 +157,7 @@ Open it from the bottom navigation bar. | **Topics** | Number of topic trees materialized so far (per-entity summaries built from "hot" entities). | | **First / latest memory** | Timestamps of the oldest and newest chunks. | -**Memory graph.** A force-directed visualization of entities and their relationships, drawn from the entity index. The graph grows as auto-fetch pulls more data — sparse early on, denser within a few days. +**Memory graph.** A force-directed visualization of entities and their relationships, drawn from the entity index. The graph grows as auto-fetch pulls more data - sparse early on, denser within a few days. **Obsidian vault.** A **View vault in Obsidian** button opens `/wiki/` directly via an `obsidian://open?path=...` deep link. You can also open the folder in any file browser. @@ -175,11 +165,11 @@ Open it from the bottom navigation bar. **Search & retrieval.** A search bar over the Memory Tree. Source-scoped, topic-scoped or global queries are all supported, and any result links back to the underlying chunk file in your Obsidian vault for full provenance. -**Routing.** The Intelligence tab also surfaces which model the agent is using per task — see [Automatic Model Routing](../model-routing/). +**Routing.** The Intelligence tab also surfaces which model the agent is using per task - see [Automatic Model Routing](../model-routing/). ## See also -* [Obsidian Wiki](./) — open the vault in Obsidian and edit it directly. -* [Auto-fetch from Integrations](auto-fetch.md) — how the tree stays fresh. -* [Smart Token Compression](../token-compression.md) — what makes ingesting "everything" cheap. -* [Local AI (optional)](../model-routing/local-ai.md) — opt in to keep embeddings and summary-tree building on-device. +* [Obsidian Wiki](./) - open the vault in Obsidian and edit it directly. +* [Auto-fetch from Integrations](auto-fetch.md) - how the tree stays fresh. +* [Smart Token Compression](../token-compression.md) - what makes ingesting "everything" cheap. +* [Local AI (optional)](../model-routing/local-ai.md) - opt in to keep embeddings and summary-tree building on-device. diff --git a/gitbooks/features/platform.md b/gitbooks/features/platform.md index b4bf2b555..32e79e7a8 100644 --- a/gitbooks/features/platform.md +++ b/gitbooks/features/platform.md @@ -1,4 +1,7 @@ --- +description: >- + What OpenHuman ships as (native React + Tauri v2 desktop app with a Rust + core), supported platforms, and what's in scope today. icon: layer-plus --- @@ -34,7 +37,7 @@ OpenHuman is built as a native application rather than a web wrapper for three r ``` ┌──────────────────────────────────────────────────┐ -│ Tauri shell — windowing, OS integration │ +│ Tauri shell - windowing, OS integration │ └──────────────────────────────────────────────────┘ │ JSON-RPC ↕ ┌──────────────────────────────────────────────────┐ @@ -45,7 +48,7 @@ OpenHuman is built as a native application rather than a web wrapper for three r └──────────────────────────────────────────────────┘ │ ┌──────────────────────────────────────────────────┐ -│ React frontend — screens, navigation │ +│ React frontend - screens, navigation │ └──────────────────────────────────────────────────┘ ``` diff --git a/gitbooks/features/privacy-and-security.md b/gitbooks/features/privacy-and-security.md index 3dbf09c5b..dc611ec95 100644 --- a/gitbooks/features/privacy-and-security.md +++ b/gitbooks/features/privacy-and-security.md @@ -26,8 +26,8 @@ OpenHuman is designed so that the **memory of your life lives on your machine**. | | | | ------------------------------- | --------------------------------------------------------------- | -| **Memory Tree SQLite database** | Local — `/memory_tree/chunks.db`. | -| **Obsidian Markdown vault** | Local — `/wiki/`. Yours to read, edit, copy, delete. | +| **Memory Tree SQLite database** | Local - `/memory_tree/chunks.db`. | +| **Obsidian Markdown vault** | Local - `/wiki/`. Yours to read, edit, copy, delete. | | **Audio capture buffers** | Local. Discarded after STT. | | **Local model state** | Local. | @@ -37,8 +37,8 @@ OpenHuman is designed so that the **memory of your life lives on your machine**. | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **LLM calls** | Proxied through the backend under one subscription, then forwarded to the underlying provider (Anthropic / OpenAI / Google / etc.) per the [model router](model-routing/). | | **Web search proxy** | The native [web search tool](native-tools/web-search.md) calls a backend proxy so you don't carry a search API key. | -| **Integration OAuth & tool proxy** | Token storage and rate-limited request brokering for [118+ integrations](integrations.md). | -| **TTS streaming** | Hosted [text-to-speech](native-tools/voice.md) audio streams. Audio is generated and discarded — not retained. | +| **Integration OAuth & tool proxy** | Token storage and rate-limited request brokering for [118+ integrations](integrations/README.md). | +| **TTS streaming** | Hosted [text-to-speech](native-tools/voice.md) audio streams. Audio is generated and discarded - not retained. | *** diff --git a/gitbooks/features/token-compression.md b/gitbooks/features/token-compression.md index ff754d1fd..f9f142218 100644 --- a/gitbooks/features/token-compression.md +++ b/gitbooks/features/token-compression.md @@ -1,6 +1,6 @@ --- description: >- - TokenJuice — a rule overlay that compacts verbose tool output before it ever + TokenJuice - a rule overlay that compacts verbose tool output before it ever enters LLM context. Sweeping through thousands of emails stays cheap. icon: file-zipper --- diff --git a/gitbooks/legal/privacy-policy.md b/gitbooks/legal/privacy-policy.md index a723c7601..9e2f3f119 100644 --- a/gitbooks/legal/privacy-policy.md +++ b/gitbooks/legal/privacy-policy.md @@ -1,4 +1,7 @@ --- +description: >- + How OpenHuman collects, uses, processes, stores, and protects information + when you use the service. icon: key --- @@ -66,7 +69,7 @@ Information may be processed in locations outside your country of residence. Whe ## Your Rights -Depending on your location, you may have rights to access, correct, delete, restrict processing of, or withdraw consent for your personal data, as well as request data portability. Requests can be done on the dashboard at any time or may be submitted to privacy@\[yourdomain].com. +Depending on your location, you may have rights to access, correct, delete, restrict processing of, or withdraw consent for your personal data, as well as request data portability. Requests can be done on the dashboard at any time or may be submitted to privacy@tinyhumans.ai. ## Changes to This Policy diff --git a/gitbooks/legal/terms-of-use.md b/gitbooks/legal/terms-of-use.md index 526ba128b..2e21d4568 100644 --- a/gitbooks/legal/terms-of-use.md +++ b/gitbooks/legal/terms-of-use.md @@ -1,4 +1,5 @@ --- +description: Terms and conditions governing use of the OpenHuman service. icon: file-contract --- diff --git a/gitbooks/overview/getting-started.md b/gitbooks/overview/getting-started.md index e832f9b15..0f0150c85 100644 --- a/gitbooks/overview/getting-started.md +++ b/gitbooks/overview/getting-started.md @@ -1,10 +1,14 @@ --- +description: >- + Install OpenHuman, walk through the in-app onboarding (sign in, connect + Gmail, choose how AI runs), and run your first request against your own + Memory Tree. icon: play --- # Getting Started -This page walks you through setting up OpenHuman and running your first request. +This page walks you through installing OpenHuman, going through the in-app onboarding, and running your first request. OpenHuman is open source under the GNU GPL3 license. The codebase is at [github.com/tinyhumansai/openhuman](https://github.com/tinyhumansai/openhuman). @@ -12,85 +16,96 @@ OpenHuman is open source under the GNU GPL3 license. The codebase is at [github. #### System requirements -OpenHuman runs on **macOS, Windows and Linux** desktops. 8 GB+ RAM is recommended; 16 GB+ if you intend to ingest very large mailboxes or repos. +OpenHuman runs on **macOS, Windows and Linux** desktops. 8 GB+ RAM is recommended; 16 GB+ if you intend to ingest very large mailboxes or repos, or run a [local model](../features/model-routing/local-ai.md) on the same machine. #### Permissions -The first time you launch OpenHuman, the OS will prompt for the permissions the app needs (Accessibility on macOS, Input Monitoring for voice hotkey, etc.). You can review and adjust these any time under **Settings → Automation & Channels**. +The first time you launch OpenHuman, the OS will prompt for the permissions the app needs (Accessibility on macOS, Input Monitoring for the voice hotkey, Camera/Microphone if you plan to use the [Meeting Agent](../features/mascot/meeting-agents.md)). You can review and adjust these any time under **Settings → Automation & Channels**. *** -## Download & install +## 1. Download and install -Get the OpenHuman desktop app from [openhuman.ai](https://openhuman.ai) or via your platform's package manager. +Get the OpenHuman desktop app from [openhuman.ai](https://openhuman.ai) or via your platform's package manager. Open the app once it's installed. *** -## Create your account +## 2. Sign in -When you first open OpenHuman, you'll be asked to sign in. Multiple sign-in options are available, including social login. +The first screen is **"Sign in! Let's Cook"**. Multiple sign-in options are available, including social login. There's also an **Advanced** panel for pointing the app at a custom core RPC URL if you're running your own backend; most users can ignore it. {% hint style="info" %} -**No permanent lock-in.** Creating an account does not grant OpenHuman ongoing access to anything. All third-party access requires explicit OAuth approval per integration later. +**No permanent lock-in.** Signing in does not grant OpenHuman ongoing access to anything. All third-party access requires explicit OAuth approval per integration in the steps below. {% endhint %} *** -## Connect your first source +## 3. Onboarding flow -OpenHuman works by connecting to your existing tools through [third-party integrations](../features/integrations.md). Each connection expands your [Memory Tree](../features/obsidian-wiki/memory-tree.md). You choose what to connect, and you can revoke access at any time. +After sign-in, OpenHuman runs you through a short routed onboarding under `/onboarding/*`. The flow is intentionally tiny: five short steps and you're in. -The 118+ catalog spans Gmail, Notion, GitHub, Slack, Stripe, Calendar, Drive, Linear, Jira, Outlook, Dropbox, Airtable, Salesforce, HubSpot, Figma, Asana, Trello, Telegram, WhatsApp, Discord, Microsoft Teams, Twitter / X, Reddit, Spotify, YouTube, Facebook, Instagram and more. +### Step 1 - Welcome ("Hi. I'm OpenHuman.") -Recommended starting points: +The mascot introduces itself. This is the same character that lives on your desktop and can [join your meetings](../features/mascot/meeting-agents.md) once you're set up. -* **Gmail**. high-signal, has a native ingest path into the Memory Tree. -* **Slack**. picks up workplace chat context fast. -* **Notion**. for structured docs and exports. -* **GitHub**. if you write code. +### Step 2 - Connect your Gmail -Click **Connect** on any integration, complete the OAuth flow, and the next [auto-fetch](../features/obsidian-wiki/auto-fetch.md) tick will start syncing it within twenty minutes. +OpenHuman asks you to connect **Gmail** as the first source. Gmail is high-signal, has a native ingest path into the [Memory Tree](../features/obsidian-wiki/memory-tree.md), and is enough on its own to give the agent a real picture of you to work with. + +You connect via OAuth. Nothing is read until the next [auto-fetch](../features/obsidian-wiki/auto-fetch.md) tick. + +You can connect more sources later - the full [118+ catalog](../features/integrations/README.md) is available from **Settings**. + +### Step 3 - Building your profile ("Almost there!") + +OpenHuman runs a short context-gathering pass on what you just connected and builds an initial profile of you (recurring people, ongoing projects, timezone, working hours). This is the same memory pipeline that runs on every auto-fetch tick afterwards - it's just doing the first pass with the lights on. + +### Step 4 - Choose how AI runs + +Pick **Cloud Mode** (managed model routing - the fast default) or **Local AI** (run models on your own machine for full offline + privacy). You can switch any time later from **Settings → AI & Skills**. See [Local AI (optional)](../features/model-routing/local-ai.md) and [Automatic Model Routing](../features/model-routing/README.md) for what's behind each option. + +### Step 5 - Referral code (optional) + +If someone gave you a referral code, paste it here. Otherwise skip. + +When the flow finishes, OpenHuman flips your `onboarding_completed` flag and lands you on **/home**. *** -## Run your first request +## 4. Run your first request -Once a source is connected and auto-fetch has run a tick, try prompts like: +Once Gmail has been ingested (the first auto-fetch tick happens within twenty minutes), try prompts like: -**Briefings:** +**Briefings** * "What do I need to know from the last 12 hours?" * "What's waiting on me?" -**Messaging queries:** +**Cross-source queries** -* "Summarize what I missed today across my channels." +* "Summarize what I missed today." * "What are the key decisions from this week?" * "Extract action items from my recent conversations." - -**Cross-source queries:** - -* "Connect what my team discussed in Slack with what I was reviewing in Notion." * "What did Sarah say about the project across email and chat?" -OpenHuman picks the right model for each task automatically, see [Automatic Model Routing](../features/model-routing/). +OpenHuman picks the right model for each task automatically. See [Automatic Model Routing](../features/model-routing/README.md). *** -## Open the Obsidian vault +## 5. Open the Obsidian vault -The Memory tab has a **View vault in Obsidian** button. Click it to open `/wiki/` in [Obsidian](https://obsidian.md). You can browse summaries, drop in your own notes, and even build manual links, the agent will see your edits next ingest. See [Obsidian Wiki](../features/obsidian-wiki/). +The Memory tab has a **View vault in Obsidian** button. Click it to open `/wiki/` in [Obsidian](https://obsidian.md). You can browse the agent's summaries, drop in your own notes, and even build manual links - the agent will pick up your edits on the next ingest. See [Obsidian-Style Memory](../features/obsidian-wiki/). *** -## Explore Skills & Integrations +## 6. Let the mascot do more -After your first request, explore what else OpenHuman can do: +Now that the agent has memory and a model, the rest of the product is about giving it more surfaces: -* **Skills** extend the assistant's capabilities, fetching data, running scheduled tasks, processing information. -* **Integrations** let you push structured results to Notion, Google Sheets, and other connected tools. - -Learn more in [Skills & Integrations](../features/integrations.md). +* **[Meeting Agents](../features/mascot/meeting-agents.md)** - drop a Google Meet link in and the mascot joins as a real participant: it listens, takes notes into the Memory Tree, speaks back into the call, and uses tools live. +* **[Auto-fetch from Integrations](../features/obsidian-wiki/auto-fetch.md)** - connect more sources from **Settings**; every twenty minutes the scheduler pulls fresh data into your tree. +* **[Native Voice](../features/native-tools/voice.md)** - push-to-talk dictation and TTS replies so you can talk to OpenHuman instead of typing. +* **[Subconscious Loop](../features/subconscious.md)** - let the mascot keep working on standing tasks while you're away. ***