mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-30 15:03:57 +00:00
feat(transcript): derived transcript view — append-only session log as source of truth (#5086)
This commit is contained in:
Steven Enamakel
@@ -0,0 +1,106 @@
|
||||
# Transcript-Derived View — Raw Session Files as Source of Truth
|
||||
|
||||
Status: **draft — approved direction, phased implementation**
|
||||
Branch: `feat/transcript-derived-view` (stacked on `fix/transcript-restore-fidelity`)
|
||||
Companion: [`conversations-timeline-refactor.md`](conversations-timeline-refactor.md) (Phases 1–2, 4–5 landed; this plan supersedes its Phase 5 hydration story for settled turns)
|
||||
|
||||
## Goal
|
||||
|
||||
Stop maintaining chat state that must be *synced* with what is live. Derive the settled
|
||||
transcript from the raw session files (`session_raw/*.jsonl`), and demote every other
|
||||
store to a cache over that file. Live token streaming is untouched: in-flight turns
|
||||
render from ephemeral socket-fed state exactly as today; the file is authoritative only
|
||||
for **settled** turns.
|
||||
|
||||
This is the Codex rollout model (one JSONL replayed into both model context and UI,
|
||||
with an explicit persistence policy) adapted to our layout, plus hermes-agent's
|
||||
soft-compaction lesson (history is never destroyed, only superseded).
|
||||
|
||||
## Why derivation is unsafe today (must fix first)
|
||||
|
||||
1. **Destructive compaction.** `agent/harness/session/transcript.rs::write_transcript`
|
||||
full-rewrites the file so context reduction deletes earlier turns from disk.
|
||||
2. **Display data missing from the file**: interrupted partial answers, `request_id`
|
||||
turn boundaries, narration items. They exist only in `turn_state` snapshots.
|
||||
3. **Internal scaffolding in message content**: channel-context prefixes on user
|
||||
messages, tool-policy preamble in system content — must be sanitized (or tagged) at
|
||||
projection, never shown raw.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
live turn (unchanged)
|
||||
socket events ──► chatRuntimeSlice (ephemeral) ──► renderer
|
||||
│ chat_done
|
||||
▼ invalidate
|
||||
settled turns:
|
||||
session_raw/{root}.jsonl ─┐
|
||||
session_raw/{root}__sub-*.jsonl ─┴─► core projection (threads.transcript_get)
|
||||
│ mtime-keyed cache
|
||||
▼
|
||||
typed display items ──► renderer (same components)
|
||||
```
|
||||
|
||||
- **Model context** keeps reading the same file via the existing loader, now replaying
|
||||
compaction records instead of trusting a rewritten file.
|
||||
- **`turn_state`** shrinks to live-turn crash recovery (interrupted `streamingText`
|
||||
until the interrupted line is appended to the file, then only the in-flight turn).
|
||||
- The 20-turn retention cap stops being user-visible loss: history comes from the file.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase A — append-only transcript (Rust, prerequisite)
|
||||
`transcript.rs` + harness call sites:
|
||||
- Replace full-rewrite with **append-only** line writes. Context reduction appends a
|
||||
`compaction` record `{ kind: "compaction", replacement_ids | replacement_history }`;
|
||||
the model-context loader (`read_transcript` path) replays records to reconstruct the
|
||||
post-compaction context; a new display reader returns *all* records.
|
||||
- Stamp `request_id` on every line of a turn (turn boundary markers); keep `iteration`,
|
||||
`ts`, `seq` alignment with the progress-bridge envelope.
|
||||
- On turn abort/interrupt, append the partial assistant line flagged
|
||||
`{ interrupted: true }` so the partial answer is in the file, not only in turn_state.
|
||||
- Migration: existing files are valid append-only files with zero compaction records —
|
||||
no migration needed. Old cores reading new files must skip unknown `kind` lines
|
||||
(verify the `_extra` flatten tolerates this; add a version field to `_meta`).
|
||||
- Tests: compaction round-trip (model context reduced, display history complete),
|
||||
interrupted-partial append, request_id stamping, legacy-file read.
|
||||
|
||||
### Phase B — projection RPC (Rust)
|
||||
New `threads.transcript_get(thread_id, {cursor?, limit?})` in the threads domain
|
||||
(canonical module shape: ops/schemas):
|
||||
- Resolve root transcript via `find_root_transcript_for_thread`; discover
|
||||
`__sub-*.jsonl` children; project into typed display items:
|
||||
`user_message | assistant_message | reasoning | tool_call {args, result, status} |
|
||||
subagent {id, items} | turn_boundary {request_id} | interrupted_partial`.
|
||||
- Sanitize scaffolding (channel-context prefix, tool-policy preamble) at projection;
|
||||
tag rather than mutate where ambiguity exists.
|
||||
- Cache: per-thread projection keyed on (file paths, mtimes, lengths); invalidated
|
||||
implicitly by key change. No cache writes to disk — pure memory cache.
|
||||
- Pagination newest-first with cursor; default window sized for one screen.
|
||||
- Tests: JSON-RPC E2E (`tests/json_rpc_e2e.rs`) — write file, call RPC, assert items;
|
||||
subagent merge; sanitization; cache-key invalidation.
|
||||
|
||||
### Phase C — frontend switch (TS)
|
||||
- Thread-open restore path: replace `turn_state_history` hydration for settled turns
|
||||
with `transcript_get`; map items onto the existing renderers
|
||||
(`PastTurnInsights`/`ToolTimelineBlock`/`ProcessingTranscriptView`, bubbles).
|
||||
- Live turn: untouched (socket → chatRuntimeSlice). On `chat_done`, drop the live
|
||||
turn's ephemeral state and refetch/append the settled projection.
|
||||
- Keep the turn_state hydration path as fallback behind a flag for one release.
|
||||
- Tests: restore renders identical item sequence to live for a scripted turn;
|
||||
interrupted partial visible; legacy thread (no request_id) fallback.
|
||||
|
||||
### Phase D — cleanup
|
||||
- Demote/remove `turn_state` ring retention (keep live-turn snapshot only), drop the
|
||||
`.md` mirror or mark derived-only, delete the fallback flag.
|
||||
|
||||
## Non-goals
|
||||
- No change to the socket streaming protocol or delta handling.
|
||||
- No SQLite consolidation (sessions.db stays an index; separate discussion).
|
||||
- Phase 3 of the timeline refactor (dedup consolidation) remains its own track.
|
||||
|
||||
## Risks
|
||||
- **Old-core / new-file compat** — unknown `kind` lines must be skipped, not fatal.
|
||||
- **File growth** — append-only grows; mitigate later with Codex-style zstd of old
|
||||
sessions; out of scope here.
|
||||
- **Sanitization false positives** — prefer tagging + frontend hiding over deletion.
|
||||
Reference in New Issue
Block a user