openhuman/gitbooks/features/token-compression.md

description, icon

description	icon
TokenJuice - a multi-stage compression router that compacts verbose tool output and ingested data before it ever enters LLM context.	file-zipper

Smart Token Compression

LLM tokens are expensive, and verbose tool output is where most of them go to die. A git status in a busy repo, a cargo build log, a 600-message email thread, a docker ps -a against a real cluster — each can balloon a context window for almost no information gain.

OpenHuman ships with TokenJuice, a compression router wired directly into the tool-execution and memory-ingestion paths. Before any tool result or ingested payload reaches a model, TokenJuice classifies it, routes it to a specialized compressor, optionally offloads the full original to a recoverable cache, and records how many tokens (and dollars) it saved.

It began as a port of vincentkoc/tokenjuice — that JSON rule overlay is still in here as the log/command compressor — but it has since grown into a multi-stage, content-aware pipeline.

---

The pipeline, step by step

Every blob that flows through compact_tool_output(...) takes the same path (src/openhuman/tokenjuice/compress.rs):

raw tool result / ingested payload
        │
        ▼
1. Size gate          router enabled? input ≥ min_bytes_to_compress (2 KB)?
        │  yes
        ▼
2. Detect kind        Json · Diff · Html · Search · Code · Log · PlainText
        │
        ▼
3. Select compressor  one specialized compressor per kind (+ per-kind toggles)
        │
        ▼
4. Compress           run it; if it declines or grows the output, fall back / pass through
        │
        ▼
5. CCR eligibility    lossy AND ≥ ccr_min_tokens (≈500)? → offload original to cache
        │
        ▼
6. Append marker      ⟦tj:<hash>⟧ footer so the agent can retrieve the full original
        │
        ▼
7. Record savings     tokens + cost saved, by model and by compressor
        │
        ▼
   compact text → LLM context

1. Size gate. If the router is disabled or the input is below min_bytes_to_compress (default 2048 bytes), it passes through untouched — tiny outputs aren't worth compressing.
2. Content detection (detect/kind.rs). The blob is classified into one of seven ContentKinds. Precedence: an explicit hint → MIME/extension tag → a per-tool prior (e.g. grep → Search, git_operations → Diff, run_tests → Log) → cheap structural heuristics (JSON → Diff → HTML → Search → Code → Log → PlainText). No regex on the hot path.
3. Compressor selection. Each kind routes to a dedicated compressor, honoring per-kind toggles (search_enabled, code_enabled, html_enabled, ml_compression_enabled).
4. Compression. The compressor runs. If it declines or its output is no smaller than the input, TokenJuice falls back to the generic compressor or passes the original through — it never makes things bigger.
5. CCR offload. For lossy compressions where the original is large enough (ccr_min_tokens, default ~500 tokens), the full original is stowed in the Compress-Cache-Retrieve store so nothing is permanently lost.
6. Recovery marker. A footer carrying the canonical marker ⟦tj:<hash>⟧ is appended, telling the agent it's looking at a partial view and how to fetch the rest.
7. Savings accounting. Tokens saved and estimated cost saved are recorded, attributed by model and by compressor.

---

The compressors

Each content kind has a purpose-built compressor (src/openhuman/tokenjuice/compressors/):

Compressor	Kind	What it does
SmartCrusher	JSON	Re-renders arrays of objects as a compact table; past ~40 rows keeps head + tail + error rows + numeric outliers.
Code	Code	Keeps signatures and imports, collapses deep function bodies to { … N lines … } (tree-sitter when available, brace-depth heuristic otherwise). Preserves TODO/FIXME/error/panic/unsafe markers.
Log	Log	For command output, delegates to the JSON rule engine (below). For other logs, keeps errors / warnings / stack traces / summaries and drops the noise.
Search	Search	Groups grep/ripgrep path:line:body hits by file, ranks by query-term density, keeps the top matches per file, and tallies [+N more].
Diff	Diff	Keeps changed lines and hunk headers, collapses long unchanged runs to an anchor; lockfile hunks shrink to a one-line +A/-B summary.
Html	HTML	Strips markup to readable text with sensible block-boundary newlines and entity decoding — allocation-light, no DOM.
MlText	PlainText	Opt-in ML salience compression (see below).
Generic	fallback	Head/tail summariser for command output that no specific rule matched; declines on structured blobs so they're preserved.

Multi-byte text (CJK, emoji, combining marks) is handled grapheme-by-grapheme throughout — never split mid-character.

---

ML compression (opt-in)

Beyond the deterministic compressors, TokenJuice can route plain text through a ModernBERT token-salience model that scores and drops low-information spans (src/openhuman/tokenjuice/ml/).

• Off by default. Enable with ml_compression_enabled = true in [tokenjuice].
• Runs locally as the kompress backend of the shared Python runtime sidecar — no data leaves your machine.
• Tunable: ml_model_id (default answerdotai/ModernBERT-base), ml_target_ratio (default 0.5), ml_max_input_chars (default 200000), ml_device (cpu/auto), ml_sidecar_idle_timeout_secs.
• Graceful: if the sidecar is unavailable or an input exceeds the char cap, it degrades to the native compressors without ever failing the agent loop.

---

Nothing is lost: CCR cache & retrieval

Lossy compression would normally mean throwing data away. TokenJuice instead offloads the full original into the Compress-Cache-Retrieve (CCR) store and leaves a breadcrumb (src/openhuman/tokenjuice/cache/).

• In-memory tier (always on): a process-global store keyed by SHA-256 hash, bounded by entry count (max_cache_entries, default 256) and total bytes (max_cache_bytes, default 64 MiB), FIFO eviction.
• On-disk tier (optional): <workspace>/.tokenjuice/ccr/, enabled with ccr_disk_enabled, survives memory eviction. Optional TTL via ccr_ttl_secs.
• The marker: compacted output ends with a footer like [compacted tool output — PARTIAL view; full original available via tokenjuice_retrieve with token "…"] carrying the ⟦tj:<hash>⟧ token.
• Retrieval tool: the agent calls the read-only tokenjuice_retrieve tool with that token (optionally a byte/line range) to pull back the full original or a slice. The token is an unguessable SHA-256 digest.

So the agent gets the cheap compacted view by default, and can transparently "zoom in" on the full text only when it actually needs it.

---

Savings tracking

Every compression is metered (src/openhuman/tokenjuice/savings.rs). TokenJuice tracks events, original vs. compacted tokens, tokens saved, and estimated cost saved in USD (using per-model input pricing) — aggregated total, by_model, and by_compressor. Stats persist to <workspace>/state/tokenjuice_savings.json and survive restarts.

Read them over RPC with tokenjuice.savings_stats; clear them with tokenjuice.savings_reset.

---

The rule overlay (command & log output)

The original three-layer JSON rule overlay still powers the Log/command compressor. Rules merge in order, later layers overriding earlier ones:

Layer	Path	Purpose
Builtin	shipped with the binary	~96 vendored rules for git, npm, cargo, docker, kubectl, ls…
User	~/.config/tokenjuice/rules/	personal overrides, apply everywhere
Project	.tokenjuice/rules/	repo-specific overrides, checked in and shared with the team

Each rule names a command/tool pattern and a reduction strategy (skip/keep filters, transforms like strip-ANSI and dedupe, head/tail summarize, named counters, canned messages). Rules are JSON — add one and it applies with no recompile.

---

Configuration, RPC & tools

Everything lives under the [tokenjuice] config block (src/openhuman/config/schema/tokenjuice.rs) and can be changed live.

• Master switch: router_enabled (default true).
• Thresholds: min_bytes_to_compress, ccr_min_tokens.
• CCR: ccr_enabled, ccr_disk_enabled, max_cache_entries, max_cache_bytes, ccr_ttl_secs.
• Per-kind: search_enabled, code_enabled, html_enabled, plus the ml_* keys.
• RPC (tokenjuice.*): detect, compress (dry-run the pipeline), settings_get / settings_update (live partial patch), cache_stats, retrieve, savings_stats, savings_reset.
• Agent tool: tokenjuice_retrieve (read-only) recovers offloaded originals.
• Debugging: start the core with RUST_LOG=openhuman_core::openhuman::tokenjuice=debug to watch detection, matching, and how much each blob is trimmed.

---

Why this matters for memory

TokenJuice is what makes auto-fetch economically viable. When the Gmail provider syncs a page of 200 messages, TokenJuice compacts each canonicalized email before it enters the model that builds Memory Tree summaries. The same applies to GitHub diffs, Slack dumps, and any other firehose source. Concretely: ingesting six months of email through a frontier model costs single-digit dollars instead of hundreds.

---

See also

• Available Tools — most heavy tool output flows through TokenJuice.
• Memory Tree — the downstream consumer of compressed output.
• Billing, Cost & Usage — where token savings show up as real money.