mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-28 13:32:23 +00:00
153 lines
11 KiB
Markdown
153 lines
11 KiB
Markdown
---
|
|
description: >-
|
|
TokenJuice - a multi-stage compression router that compacts verbose tool
|
|
output before it ever enters LLM context.
|
|
icon: file-zipper
|
|
---
|
|
|
|
# Smart Token Compression
|
|
|
|
LLM tokens are expensive, and verbose tool output is where most of them go to die. A `git status` in a busy repo, a `cargo build` log, a 600-message email thread, a `docker ps -a` against a real cluster — each can balloon a context window for almost no information gain.
|
|
|
|
OpenHuman ships with **TokenJuice**, a compression router wired directly into the agent's tool-execution path. Before any tool result reaches a model, TokenJuice classifies it, routes it to a specialized compressor, optionally offloads the full original to a recoverable cache, and records how many tokens (and dollars) it saved.
|
|
|
|
It began as a port of [vincentkoc/tokenjuice](https://github.com/vincentkoc/tokenjuice) — that JSON rule overlay is still in here as the log/command compressor — but it has since grown into a multi-stage, content-aware pipeline.
|
|
|
|
***
|
|
|
|
## The pipeline, step by step
|
|
|
|
Every blob that flows through the policy-aware TokenJuice tool-output adapters
|
|
takes the same path (`src/openhuman/tokenjuice/compress.rs`):
|
|
|
|
```text
|
|
raw tool result
|
|
│
|
|
▼
|
|
1. Size gate router enabled? input ≥ min_bytes_to_compress (2 KB)?
|
|
│ yes
|
|
▼
|
|
2. Detect kind Json · Diff · Html · Search · Code · Log · PlainText
|
|
│
|
|
▼
|
|
3. Select compressor one specialized compressor per kind (+ per-kind toggles)
|
|
│
|
|
▼
|
|
4. Compress run it; if it declines or grows the output, fall back / pass through
|
|
│
|
|
▼
|
|
5. CCR eligibility lossy AND ≥ ccr_min_tokens (≈500)? → offload original to cache
|
|
│
|
|
▼
|
|
6. Append marker ⟦tj:<hash>⟧ footer so the agent can retrieve the full original
|
|
│
|
|
▼
|
|
7. Record savings tokens + cost saved, by model and by compressor
|
|
│
|
|
▼
|
|
compact text → LLM context
|
|
```
|
|
|
|
1. **Size gate.** If the router is disabled or the input is below `min_bytes_to_compress` (default **2048 bytes**), it passes through untouched — tiny outputs aren't worth compressing.
|
|
2. **Content detection** (`detect/kind.rs`). The blob is classified into one of seven `ContentKind`s. Precedence: an explicit hint → MIME/extension tag → a per-tool prior (e.g. `grep` → Search, `git_operations` → Diff, `run_tests` → Log) → cheap structural heuristics (JSON → Diff → HTML → Search → Code → Log → PlainText). No regex on the hot path.
|
|
3. **Compressor selection.** Each kind routes to a dedicated compressor, honoring per-kind toggles (`search_enabled`, `code_enabled`, `html_enabled`, `ml_compression_enabled`).
|
|
4. **Compression.** The compressor runs. If it declines or its output is no smaller than the input, TokenJuice falls back to the generic compressor or passes the original through — it never makes things bigger.
|
|
5. **CCR offload.** For **lossy** compressions where the original is large enough (`ccr_min_tokens`, default ~500 tokens), the full original is stowed in the **Compress-Cache-Retrieve** store so nothing is permanently lost.
|
|
6. **Recovery marker.** A footer carrying the canonical marker `⟦tj:<hash>⟧` is appended, telling the agent it's looking at a partial view and how to fetch the rest.
|
|
7. **Savings accounting.** Tokens saved and estimated cost saved are recorded, attributed by model and by compressor.
|
|
|
|
***
|
|
|
|
## The compressors
|
|
|
|
Each content kind has a purpose-built compressor (`src/openhuman/tokenjuice/compressors/`):
|
|
|
|
| Compressor | Kind | What it does |
|
|
| ---------------- | ----------- | -------------------------------------------------------------------------------------------------------- |
|
|
| **SmartCrusher** | JSON | Re-renders arrays of objects as a compact table; past ~40 rows keeps head + tail + error rows + numeric outliers. |
|
|
| **Code** | Code | Keeps signatures and imports, collapses deep function bodies to `{ … N lines … }` (tree-sitter when available, brace-depth heuristic otherwise). Preserves `TODO`/`FIXME`/`error`/`panic`/`unsafe` markers. |
|
|
| **Log** | Log | For **command output**, delegates to the JSON rule engine (below). For other logs, keeps errors / warnings / stack traces / summaries and drops the noise. |
|
|
| **Search** | Search | Groups grep/ripgrep `path:line:body` hits by file, ranks by query-term density, keeps the top matches per file, and tallies `[+N more]`. |
|
|
| **Diff** | Diff | Keeps changed lines and hunk headers, collapses long unchanged runs to an anchor; lockfile hunks shrink to a one-line `+A/-B` summary. |
|
|
| **Html** | HTML | Strips markup to readable text with sensible block-boundary newlines and entity decoding — allocation-light, no DOM. |
|
|
| **MlText** | PlainText | Opt-in ML salience compression (see below). |
|
|
| **Generic** | fallback | Head/tail summariser for command output that no specific rule matched; declines on structured blobs so they're preserved. |
|
|
|
|
Multi-byte text (CJK, emoji, combining marks) is handled grapheme-by-grapheme throughout — never split mid-character.
|
|
|
|
***
|
|
|
|
## ML compression (opt-in)
|
|
|
|
Beyond the deterministic compressors, TokenJuice can route plain text through a **ModernBERT** token-salience model that scores and drops low-information spans (`src/openhuman/tokenjuice/ml/`).
|
|
|
|
* **Off by default.** Enable with `ml_compression_enabled = true` in `[tokenjuice]`.
|
|
* **Runs locally** as the `kompress` backend of the shared Python runtime sidecar — no data leaves your machine.
|
|
* **Tunable:** `ml_model_id` (default `answerdotai/ModernBERT-base`), `ml_target_ratio` (default `0.5`), `ml_max_input_chars` (default `200000`), `ml_device` (`cpu`/`auto`), `ml_sidecar_idle_timeout_secs`.
|
|
* **Graceful:** if the sidecar is unavailable or an input exceeds the char cap, it degrades to the native compressors without ever failing the agent loop.
|
|
|
|
***
|
|
|
|
## Nothing is lost: CCR cache & retrieval
|
|
|
|
Lossy compression would normally mean throwing data away. TokenJuice instead **offloads** the full original into the **Compress-Cache-Retrieve (CCR)** store and leaves a breadcrumb (`src/openhuman/tokenjuice/cache/`).
|
|
|
|
* **In-memory tier** (always on): a process-global store keyed by SHA-256 hash, bounded by entry count (`max_cache_entries`, default 256) and total bytes (`max_cache_bytes`, default 64 MiB), FIFO eviction.
|
|
* **On-disk tier** (optional): `<workspace>/.tokenjuice/ccr/`, enabled with `ccr_disk_enabled`, survives memory eviction. Optional TTL via `ccr_ttl_secs`.
|
|
* **The marker:** compacted output ends with a footer like `[compacted tool output — PARTIAL view; full original available via tokenjuice_retrieve with token "…"]` carrying the `⟦tj:<hash>⟧` token.
|
|
* **Retrieval tool:** the agent calls the read-only **`tokenjuice_retrieve`** tool with that token (optionally a byte/line `range`) to pull back the full original or a slice. The token is an unguessable SHA-256 digest.
|
|
|
|
So the agent gets the cheap compacted view by default, and can transparently "zoom in" on the full text only when it actually needs it.
|
|
|
|
***
|
|
|
|
## Savings tracking
|
|
|
|
Every compression is metered (`src/openhuman/tokenjuice/savings.rs`). TokenJuice tracks events, original vs. compacted tokens, tokens saved, and **estimated cost saved in USD** (using per-model input pricing) — aggregated `total`, `by_model`, and `by_compressor`. Stats persist to `<workspace>/state/tokenjuice_savings.json` and survive restarts.
|
|
|
|
Read them over RPC with `openhuman.tokenjuice_savings_stats`; clear them with `openhuman.tokenjuice_savings_reset`.
|
|
|
|
***
|
|
|
|
## The rule overlay (command & log output)
|
|
|
|
The original three-layer JSON rule overlay still powers the Log/command compressor. Rules merge in order, later layers overriding earlier ones:
|
|
|
|
| Layer | Path | Purpose |
|
|
| ------------ | ----------------------------- | ------------------------------------------------------------- |
|
|
| **Builtin** | shipped with the binary | ~96 vendored rules for git, npm, cargo, docker, kubectl, ls… |
|
|
| **User** | `~/.config/tokenjuice/rules/` | personal overrides, apply everywhere |
|
|
| **Project** | `.tokenjuice/rules/` | repo-specific overrides, checked in and shared with the team |
|
|
|
|
Each rule names a command/tool pattern and a reduction strategy (skip/keep filters, transforms like strip-ANSI and dedupe, head/tail summarize, named counters, canned messages). Rules are JSON — add one and it applies with no recompile.
|
|
|
|
***
|
|
|
|
## Configuration, RPC & tools
|
|
|
|
Everything lives under the `[tokenjuice]` config block (`src/openhuman/config/schema/tokenjuice.rs`) and can be changed live.
|
|
|
|
* **Master switch:** `router_enabled` (default `true`).
|
|
* **Thresholds:** `min_bytes_to_compress`, `ccr_min_tokens`.
|
|
* **CCR:** `ccr_enabled`, `ccr_disk_enabled`, `max_cache_entries`, `max_cache_bytes`, `ccr_ttl_secs`.
|
|
* **Per-kind:** `search_enabled`, `code_enabled`, `html_enabled`, plus the `ml_*` keys.
|
|
* **RPC** (`openhuman.tokenjuice_*`): `detect`, `compress` (dry-run the pipeline), `settings_get` / `settings_update` (live partial patch), `cache_stats`, `retrieve`, `savings_stats`, `savings_reset`.
|
|
* **Agent tool:** `tokenjuice_retrieve` (read-only) recovers offloaded originals.
|
|
* **Debugging:** start the core with `RUST_LOG=openhuman_core::openhuman::tokenjuice=debug` to watch detection, matching, and how much each blob is trimmed.
|
|
|
|
***
|
|
|
|
## Why this matters
|
|
|
|
Agents live or die by their context budget. A single working session can fan out across dozens of tool calls — greps, builds, test runs, `git` output, and large [web-fetch / scrape](native-tools/web-scraper.md) results the agent pulls down. TokenJuice sits on that tool-execution path and compacts each result before it lands in context, so an agent can sweep a noisy repo or a long web page without each step ballooning the window. The savings compound across a session and are metered in real dollars (see [Billing, Cost & Usage](billing-and-usage.md)).
|
|
|
|
> **Scope note.** TokenJuice runs on the agent's **tool results**, not on the background [auto-fetch](obsidian-wiki/auto-fetch.md) ingestion pipeline. The 20-minute sync that builds the [Memory Tree](obsidian-wiki/memory-tree.md) has its own canonicalization and chunking and does not route payloads through TokenJuice today.
|
|
|
|
***
|
|
|
|
## See also
|
|
|
|
* [Available Tools](native-tools/README.md) — most heavy tool output flows through TokenJuice.
|
|
* [Memory Tree](obsidian-wiki/memory-tree.md) — the downstream consumer of compressed output.
|
|
* [Billing, Cost & Usage](billing-and-usage.md) — where token savings show up as real money.
|