openhuman/.claude/rules/17-skills-memory-inference-flow.md

Skills → Memory Layer → Agent Inference: Full Flow

Overview

This document traces the complete data flow from skill discovery and OAuth/sync events, through the TinyHumans Neocortex memory layer (tinyhumansai SDK), and into the Rust-side agentic inference loop — showing exactly how skill data is written to and read from memory and how it reaches the LLM context at inference time.

---

1. Skill Discovery & Lifecycle (SkillProvider.tsx)

File: src/providers/SkillProvider.tsx

On app mount (when a JWT token is present) SkillProvider calls invoke('runtime_discover_skills') → the Rust runtime scans skills/skills/{skill-id}/manifest.json from the git submodule and returns a manifest list.

Token present
  → discoverSkills()
    → invoke('runtime_discover_skills')         // Rust: qjs_engine.rs:184
      → reads skills/skills/*/manifest.json
      → filters: is_javascript() && supports_current_platform()
      → returns manifests[]
  → skillManager.registerSkill(manifest)        // in-memory registry
  → for each manifest with setupComplete:
      skillManager.startSkill(manifest)          // starts V8/QuickJS instance

Two Tauri event listeners run continuously:

Event	What it does
skill-state-changed	Dispatches setSkillState into Redux skillsSlice.skillStates[skillId]
runtime:skill-status-changed	Updates skillsSlice.skills[skillId].status; surfaces errors

---

2. Memory Client Initialisation

File: src-tauri/src/memory/mod.rs Tauri command: init_memory_client (called by frontend after auth)

The MemoryClient wraps the TinyHumansMemoryClient from the tinyhumansai Rust crate. It is constructed with the user's JWT (authSlice.token) and stored as Arc<MemoryClient> in MemoryState (a Mutex<Option<MemoryClientRef>>).

Base URL resolution (in priority order):

1. OPENHUMAN_BASE_URL env var
2. TINYHUMANS_BASE_URL env var
3. SDK default

---

3. Skill → Memory Sync: Two Write Paths

Both trigger inside src-tauri/src/runtime/qjs_skill_instance.rs.

3a. OAuth Completion (skill/oauth-complete)

When a user connects a skill via OAuth, the JS runtime calls back with skill/oauth-complete. After the OAuth flow completes the skill's ops state (data published via state.set()) is snapshotted and stored to memory:

skill/oauth-complete handler
  → handle_js_call(rt, ctx, "onOAuthComplete", params)   // runs skill JS
  → ops_state.read().data.clone()                        // snapshot skill state
  → tokio::spawn (fire-and-forget):
      MemoryClient::store_skill_sync(
          skill_id       = e.g. "gmail"
          integration_id = params["integrationId"]       // e.g. user email
          title          = "{skill} OAuth sync — {integrationId}"
          content        = JSON snapshot of ops state
          namespace      = "skill:{skill_id}:{integration_id}"
      )
      → tinyhumansai::TinyHumansMemoryClient::insert_memory(InsertMemoryParams { ... })
      → HTTP POST to TinyHumans Neocortex API

3b. Periodic Sync (skill/sync)

The runtime fires skill/sync events on a cron schedule. The flow is identical to OAuth completion but uses integration_id = "default" and title "{skill} periodic sync":

skill/sync handler
  → handle_js_call(rt, ctx, "onSync", "{}")
  → ops_state.read().data.clone()
  → tokio::spawn:
      MemoryClient::store_skill_sync(
          skill_id       = e.g. "notion"
          integration_id = "default"
          namespace      = "skill:notion:default"
          content        = JSON snapshot
      )

Namespace Pattern

All skill memories are stored under skill:{skill_id}:{integration_id}. Examples: skill:gmail:user@example.com, skill:notion:default.

---

4. Memory Operations Reference

File: src-tauri/src/memory/mod.rs

Method	SDK call	When used
store_skill_sync(...)	insert_memory	OAuth complete, periodic sync
query_skill_context(...)	query_memory	RAG query — fetch relevant chunks for a user question
recall_skill_context(...)	recall_memory	Recall synthesised summary from Master node
clear_skill_memory(...)	delete_memory	OAuth revoke / disconnect

---

5. Conversation → Inference: Two Code Paths

File: src/pages/Conversations.tsx

useRustChat() returns true when running in Tauri (desktop). This selects between two paths:

handleSendMessage(text)
  ├─ rustChat == true  → Rust path  (invoke chat_send)
  └─ rustChat == false → Web path   (handleSendMessageWeb — TypeScript loop)

5a. Rust Path (Desktop)

chatSend({ threadId, message, model, authToken, backendUrl, messages, notionContext })
  → invoke('chat_send')               // src-tauri/src/commands/chat.rs:359
    → spawns background task
    → chat_send_inner(...)

Completion events flow back over Tauri events:

Event	Frontend handler
chat:tool_call	shows active tool indicator
chat:tool_result	clears tool indicator
chat:done	dispatch(addInferenceResponse(...))
chat:error	shows error, clears loading state

5b. Web/Fallback Path

Used when not running in Tauri (browser). Runs the agentic loop entirely in TypeScript:

• Calls inferenceApi.createChatCompletion(request) directly
• Executes tools via skillManager.callTool(skillId, toolName, args)
• Both paths share the same 5-round MAX_TOOL_ROUNDS limit and {skillId}__{toolName} naming convention

---

6. Rust Agentic Loop in Detail

File: src-tauri/src/commands/chat.rs — chat_send_inner()

Step 1: Load OpenClaw context
  → load_openclaw_context(app)
    → reads ai/SOUL.md, IDENTITY.md, AGENTS.md, USER.md, BOOTSTRAP.md, MEMORY.md, TOOLS.md
    → cached in static AI_CONFIG_CACHE (cleared on restart)
    → truncated to MAX_CONTEXT_CHARS (20,000 chars)

Step 2: Recall memory context
  → MemoryClient::recall_skill_context("conversations", thread_id, 10)
    → tinyhumansai recall_memory(namespace="skill:conversations:{thread_id}")
    → returns synthesised summary string or None

Step 3: Build processed user message
  processed = user_message
  if openclaw_context  → prepend as "## Project Context\n...\n\nUser message: {processed}"
  if memory_context    → prepend as "[MEMORY_CONTEXT]\n{mem}\n[/MEMORY_CONTEXT]\n\n{processed}"
  if notion_context    → prepend as "{notionContext}\n\n{processed}"

Step 4: Build messages array
  → history (ChatMessagePayload[]) + processed user message

Step 5: Discover tools
  → engine.all_tools()
    → returns all tools from running skills
    → namespaced: "{skill_id}__{tool_name}"
    → formatted as OpenAI function-calling schema

Step 6: Agentic loop (max 5 rounds)
  for round in 0..MAX_TOOL_ROUNDS:
    POST {backend_url}/openai/v1/chat/completions
      body: { model, messages, tools, tool_choice: "auto" }
      timeout: 120s
      auth: Bearer {auth_token}

    if finish_reason == "tool_calls":
      emit chat:tool_call
      engine.call_tool(skill_id, tool_name, args)   // 60s timeout
        → QuickJS/V8 runtime executes skill JS tool handler
      emit chat:tool_result
      append tool result to messages
      continue loop

    else (finish_reason == "stop"):
      emit chat:done { full_response, rounds_used, token_counts }
      return Ok(())

---

7. Tool Execution: Rust → Skill JS

File: src-tauri/src/runtime/qjs_engine.rs — call_tool(skill_id, tool_name, args)

The Rust runtime routes call_tool into the running QuickJS/V8 skill instance:

• Serialises args as JSON
• Calls the JS tool handler registered by the skill
• Returns ToolCallResult { content: Vec<ToolContent>, is_error: bool }
• Text content is extracted and appended as the tool role message in the loop

---

8. End-to-End Flow Summary

User types message (Conversations.tsx)
  │
  ├─ [Web path only] invoke('recall_memory')      → TinyHumans API (recall)
  │                                                  returns synthesised context
  ├─ buildNotionContext()                          → from Redux skillStates.notion
  │
  ▼
invoke('chat_send')                                [Rust/desktop path]
  │
  ▼
Rust: chat_send_inner()
  ├─ load_openclaw_context()                       → ai/*.md files (cached)
  ├─ recall_skill_context("conversations", tid)    → TinyHumans API (recall)
  ├─ Build prompt: [OpenClaw] + [MEMORY] + [Notion] + user_message
  ├─ discover_tools()                              → all running skill tools
  │
  └─ Agentic loop (≤5 rounds):
       POST /openai/v1/chat/completions             → Backend LLM (neocortex-mk1)
         │
         ├─ tool_calls → engine.call_tool()         → QuickJS/V8 skill instance
         │                  └─ JS tool handler
         │                  └─ returns result string
         │                  └─ appended as tool message
         │
         └─ stop → emit chat:done
                     └─ Frontend: dispatch(addInferenceResponse(...))

Separately (async, fire-and-forget):
  Skill onSync() / onOAuthComplete()
    → MemoryClient::store_skill_sync()
      → TinyHumans insert_memory (namespace: skill:{id}:{integrationId})

---

Key Files Reference

File	Role
src/providers/SkillProvider.tsx	Discovery, lifecycle, Redux state sync
src/pages/Conversations.tsx	Message send, both code paths, Notion context
src/services/chatService.ts	chatSend(), chatCancel(), useRustChat()
src-tauri/src/commands/chat.rs	Rust agentic loop, context assembly, tool dispatch
src-tauri/src/commands/memory.rs	recall_memory Tauri command
src-tauri/src/memory/mod.rs	MemoryClient wrapping tinyhumansai SDK
src-tauri/src/runtime/qjs_skill_instance.rs	Skill sync → memory write triggers
src-tauri/src/runtime/qjs_engine.rs	discover_skills(), call_tool(), all_tools()
skills/skills/*/manifest.json	Skill metadata (git submodule)