Transcript-Derived View — Raw Session Files as Source of Truth

Status: draft — approved direction, phased implementation Branch: feat/transcript-derived-view (stacked on fix/transcript-restore-fidelity) Companion: conversations-timeline-refactor.md (Phases 1–2, 4–5 landed; this plan supersedes its Phase 5 hydration story for settled turns)

Goal

Stop maintaining chat state that must be synced with what is live. Derive the settled transcript from the raw session files (session_raw/*.jsonl), and demote every other store to a cache over that file. Live token streaming is untouched: in-flight turns render from ephemeral socket-fed state exactly as today; the file is authoritative only for settled turns.

This is the Codex rollout model (one JSONL replayed into both model context and UI, with an explicit persistence policy) adapted to our layout, plus hermes-agent's soft-compaction lesson (history is never destroyed, only superseded).

Why derivation is unsafe today (must fix first)

Destructive compaction. agent/harness/session/transcript.rs::write_transcript full-rewrites the file so context reduction deletes earlier turns from disk.
Display data missing from the file: interrupted partial answers, request_id turn boundaries, narration items. They exist only in turn_state snapshots.
Internal scaffolding in message content: channel-context prefixes on user messages, tool-policy preamble in system content — must be sanitized (or tagged) at projection, never shown raw.

Architecture

                     live turn (unchanged)
socket events ──► chatRuntimeSlice (ephemeral) ──► renderer
                                  │ chat_done
                                  ▼ invalidate
settled turns:
session_raw/{root}.jsonl ─┐
session_raw/{root}__sub-*.jsonl ─┴─► core projection (threads.transcript_get)
                                        │  mtime-keyed cache
                                        ▼
                              typed display items ──► renderer (same components)

Model context keeps reading the same file via the existing loader, now replaying compaction records instead of trusting a rewritten file.
turn_state shrinks to live-turn crash recovery (interrupted streamingText until the interrupted line is appended to the file, then only the in-flight turn).
The 20-turn retention cap stops being user-visible loss: history comes from the file.

Phases

Phase A — append-only transcript (Rust, prerequisite)

transcript.rs + harness call sites:

Replace full-rewrite with append-only line writes. Context reduction appends a compaction record { kind: "compaction", replacement_ids | replacement_history }; the model-context loader (read_transcript path) replays records to reconstruct the post-compaction context; a new display reader returns all records.
Stamp request_id on every line of a turn (turn boundary markers); keep iteration, ts, seq alignment with the progress-bridge envelope.
On turn abort/interrupt, append the partial assistant line flagged { interrupted: true } so the partial answer is in the file, not only in turn_state.
Migration: existing files are valid append-only files with zero compaction records — no migration needed. Old cores reading new files must skip unknown kind lines (verify the _extra flatten tolerates this; add a version field to _meta).
Tests: compaction round-trip (model context reduced, display history complete), interrupted-partial append, request_id stamping, legacy-file read.

Phase B — projection RPC (Rust)

New threads.transcript_get(thread_id, {cursor?, limit?}) in the threads domain (canonical module shape: ops/schemas):

Resolve root transcript via find_root_transcript_for_thread; discover __sub-*.jsonl children; project into typed display items: user_message | assistant_message | reasoning | tool_call {args, result, status} | subagent {id, items} | turn_boundary {request_id} | interrupted_partial.
Sanitize scaffolding (channel-context prefix, tool-policy preamble) at projection; tag rather than mutate where ambiguity exists.
Cache: per-thread projection keyed on (file paths, mtimes, lengths); invalidated implicitly by key change. No cache writes to disk — pure memory cache.
Pagination newest-first with cursor; default window sized for one screen.
Tests: JSON-RPC E2E (tests/json_rpc_e2e.rs) — write file, call RPC, assert items; subagent merge; sanitization; cache-key invalidation.

Phase C — frontend switch (TS)

Thread-open restore path: replace turn_state_history hydration for settled turns with transcript_get; map items onto the existing renderers (PastTurnInsights/ToolTimelineBlock/ProcessingTranscriptView, bubbles).
Live turn: untouched (socket → chatRuntimeSlice). On chat_done, drop the live turn's ephemeral state and refetch/append the settled projection.
Keep the turn_state hydration path as fallback behind a flag for one release.
Tests: restore renders identical item sequence to live for a scripted turn; interrupted partial visible; legacy thread (no request_id) fallback.

Phase D — cleanup

Demote/remove turn_state ring retention (keep live-turn snapshot only), drop the .md mirror or mark derived-only, delete the fallback flag.

Non-goals

No change to the socket streaming protocol or delta handling.
No SQLite consolidation (sessions.db stays an index; separate discussion).
Phase 3 of the timeline refactor (dedup consolidation) remains its own track.

Risks

Old-core / new-file compat — unknown kind lines must be skipped, not fatal.
File growth — append-only grows; mitigate later with Codex-style zstd of old sessions; out of scope here.
Sanitization false positives — prefer tagging + frontend hiding over deletion.

5.9 KiB Raw Permalink Blame History Unescape Escape