admin/openhuman
1
0
Fork 0
mirror of https://github.com/tinyhumansai/openhuman.git synced 2026-07-27 21:08:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

[codex] Port TokenJuice engine to vendored TinyJuice (#4523)

Browse Source
This commit is contained in:
Steven Enamakel
2026-07-04 21:02:48 -07:00
committed by GitHub
parent 14ce6361c0
commit 1d7f0cc62d
156 changed files with 133 additions and 14391 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gitbooks/developing/architecture/agent-harness.md
+1 -1
View File
@@ -171,7 +171,7 @@ When a tool result exceeds the summarizer's threshold, it gets routed through a
### TokenJuice - content-aware tool-output compaction (Stage 1a)
Before a fresh tool result enters history (and ahead of the byte-budget backstop), it passes through the **TokenJuice content router** (`src/openhuman/tokenjuice/`). Inspired by Headroom, the router *detects the content kind* (JSON, code, log, search, diff, HTML, plain text) from the bytes and/or a hint derived from the tool name and arguments, then dispatches to a specialised compressor:
Before a fresh tool result enters history (and ahead of the byte-budget backstop), it passes through the **TokenJuice content router** in the vendored TinyJuice crate (`vendor/tinyjuice`), with OpenHuman adapters in `src/openhuman/tokenjuice/`. Inspired by Headroom, the router *detects the content kind* (JSON, code, log, search, diff, HTML, plain text) from the bytes and/or a hint derived from the tool name and arguments, then dispatches to a specialised compressor:
* **JSON** → SmartCrusher: array-of-objects → table (each key once), preserving rows that carry errors or numeric outliers.
* **Code** → tree-sitter (Rust/TS/JS/Python) signature keeper that collapses function bodies; brace-depth heuristic fallback.
gitbooks/features/token-compression.md
+5 -5
View File
@@ -18,7 +18,7 @@ It began as a port of [vincentkoc/tokenjuice](https://github.com/vincentkoc/toke
## The pipeline, step by step
Every blob that flows through the policy-aware TokenJuice tool-output adapters
takes the same path (`src/openhuman/tokenjuice/compress.rs`):
takes the same path through the vendored TinyJuice router (`vendor/tinyjuice/src/compress.rs`):
```text
raw tool result
@@ -60,7 +60,7 @@ raw tool result
## The compressors
Each content kind has a purpose-built compressor (`src/openhuman/tokenjuice/compressors/`):
Each content kind has a purpose-built compressor (`vendor/tinyjuice/src/compressors/`):
| Compressor | Kind | What it does |
| ---------------- | ----------- | -------------------------------------------------------------------------------------------------------- |
@@ -79,7 +79,7 @@ Multi-byte text (CJK, emoji, combining marks) is handled grapheme-by-grapheme th
## ML compression (opt-in)
Beyond the deterministic compressors, TokenJuice can route plain text through a **ModernBERT** token-salience model that scores and drops low-information spans (`src/openhuman/tokenjuice/ml/`).
Beyond the deterministic compressors, TokenJuice can route plain text through a **ModernBERT** token-salience model that scores and drops low-information spans. The TinyJuice compressor exposes the optional ML slot, and OpenHuman bridges it to Kompress in `src/openhuman/tokenjuice/ml/`.
* **Off by default.** Enable with `ml_compression_enabled = true` in `[tokenjuice]`.
* **Runs locally** as the `kompress` backend of the shared Python runtime sidecar. No data leaves your machine.
@@ -90,7 +90,7 @@ Beyond the deterministic compressors, TokenJuice can route plain text through a
## Nothing is lost: CCR cache & retrieval
Lossy compression would normally mean throwing data away. TokenJuice instead **offloads** the full original into the **Compress-Cache-Retrieve (CCR)** store and leaves a breadcrumb (`src/openhuman/tokenjuice/cache/`).
Lossy compression would normally mean throwing data away. TokenJuice instead **offloads** the full original into the **Compress-Cache-Retrieve (CCR)** store and leaves a breadcrumb (`vendor/tinyjuice/src/cache/`).
* **In-memory tier** (always on): a process-global store keyed by SHA-256 hash, bounded by entry count (`max_cache_entries`, default 256) and total bytes (`max_cache_bytes`, default 64 MiB), FIFO eviction.
* **On-disk tier** (optional): `<workspace>/.tokenjuice/ccr/`, enabled with `ccr_disk_enabled`, survives memory eviction. Optional TTL via `ccr_ttl_secs`.
@@ -103,7 +103,7 @@ So the agent gets the cheap compacted view by default, and can transparently "zo
## Savings tracking
Every compression is metered (`src/openhuman/tokenjuice/savings.rs`). TokenJuice tracks events, original vs. compacted tokens, tokens saved, and **estimated cost saved in USD** (using per-model input pricing), aggregated as `total`, `by_model`, and `by_compressor`. Stats persist to `<workspace>/state/tokenjuice_savings.json` and survive restarts.
Every compression is metered by an OpenHuman savings callback (`src/openhuman/tokenjuice/savings.rs`). TokenJuice reports events and token deltas; OpenHuman applies per-model input pricing, aggregates `total`, `by_model`, and `by_compressor`, and persists stats to `<workspace>/state/tokenjuice_savings.json`.
Read them over RPC with `openhuman.tokenjuice_savings_stats`; clear them with `openhuman.tokenjuice_savings_reset`.