openhuman/docs/tinycortex-parity-checklist.md

# TinyCortex Data-Format Parity Checklist (Phase 0.3)

**Purpose.** Existing user workspaces must open **unchanged** after every cutover flip.
This checklist enumerates every on-disk format shared between the host engine and TinyCortex,
records the audit result, and specifies the **golden-workspace parity harness** that gates W3, W5, W6.

**Hard rule (plan §0.3/§6):** any mismatch is fixed **upstream in tinycortex**, never papered over
with a host shim.

Anchors: host `7850cf363` · tinycortex `d1a8c7be` (v0.1.1).

## Ownership tiers in the shared workspace (key finding)

A user workspace's `chunks.db` (and content vault) holds **two tiers**:

1. **Crate-owned substrate** — moves to TinyCortex, schema must match byte-for-byte:
   `vectors`, `kv_global`, `kv_namespace`, `store_meta`, `legacy_marker`, `mcp_writes`,
   `mem_tree_chunks`, `mem_tree_chunk_embeddings`, `mem_tree_chunk_reembed_skipped`,
   `mem_tree_summaries`, `mem_tree_summary_embeddings`, `mem_tree_summary_reembed_skipped`,
   `mem_tree_buffers`, `mem_tree_trees`, `mem_tree_score`, `mem_tree_entity_index`,
   `mem_tree_entity_edges`, `mem_tree_entity_hotness`, `mem_tree_ingested_sources`,
   `mem_tree_jobs` **(20 tables — exact name parity confirmed)**.

2. **Host-retained `UnifiedMemory` namespace-document tier** — stays host (the "namespace
   document/graph store" plan §1 keeps host), coexisting in the **same** DB file:
   `memory_docs`, `graph_global`, `graph_namespace`, `episodic_log` (+ `episodic_fts` virtual +
   `episodic_ai/ad/au` triggers), `event_log` (+ `event_fts` + `event_embeddings` +
   `event_ai/ad/au` triggers), `conversation_segments`, `segment_embeddings`, `vector_chunks`,
   `user_profile` **(10 tables + FTS + triggers)**.

   These live in `src/openhuman/memory_store/unified/{init,fts5,events,segments,profile}.rs`.
   **W3 must keep the host creating/reading these in the same DB the crate now manages** —
   the crate's `chunks::with_connection` opens the shared handle; host `UnifiedMemory` schema
   init runs alongside the crate's. Parity requirement: crate schema init and host `unified`
   init must **compose without collision** on both fresh and existing DBs.

## Parity results by format dimension

| # | Format | Result | Evidence |
| --- | --- | --- | --- |
| P1 | **Deterministic chunk ID** | ✅ **IDENTICAL** | `chunk_id()` = SHA-256 over `source_kind.as_str()` `\0` `source_id` `\0` `seq_in_source.to_be_bytes()` `\0` `content`, first 32 hex chars. Host `memory_store/chunks/types.rs:269` vs crate `chunks/types.rs:282` — byte-for-byte identical. |
| P2 | **Vector encoding (packed f32)** | ✅ **IDENTICAL** | `vec_to_bytes`: little-endian `f32::to_le_bytes`, 4 bytes/elem, no header. Host `vectors/store.rs:467` vs crate `store/vectors/store.rs:397` — identical. |
| P3 | **`vectors` table schema** | ✅ **IDENTICAL** | `(id TEXT, namespace TEXT, text TEXT, embedding BLOB, metadata TEXT DEFAULT '{}', created_at REAL, updated_at REAL, PRIMARY KEY(namespace,id))` + `idx_vectors_ns`. Identical. |
| P4 | **`mem_tree_jobs` (queue) columns** | ✅ **MATCH** | Both persist `(id, kind, payload_json, dedupe_key, status, attempts, max_attempts, available_at_ms, locked_until_ms, last_error, created_at_ms, started_at_ms, completed_at_ms)` (identical INSERT column list). Job **payload_json** shapes must also match — verify per `JobKind` in harness (P9). |
| P5 | **`mem_tree_chunks` base columns** | ✅ **MATCH (base)** ⚠️ **new-DB divergence** | Base 15 columns identical (`id…chunk_id`). **Divergence:** crate's `CREATE` inlines 3 legacy embedding columns (`model_signature TEXT, vector BLOB, dim INTEGER`) that the host **dropped** after migrating inline embeddings to the `mem_tree_chunk_embeddings` sidecar (host `chunks/store.rs:110` comment, #1574). **Existing DBs: compatible** — both run `CREATE TABLE IF NOT EXISTS` (no-op on existing) + `migrate_legacy_embeddings_to_sidecar` (crate `chunks/migrations.rs:23`). **Fresh DBs: crate adds 3 unused columns.** Risk: positional `INSERT`/`SELECT *`. **Action: harness asserts fresh-DB schema equality; if the 3 cols matter, drop them upstream.** |
| P6 | **Content vault paths** | ✅ **IDENTICAL sig** ⚠️ verify `sanitize_filename` | `chunk_rel_path(source_kind, source_id, chunk_id)` and `summary_rel_path(tree_kind, scope_slug, level, summary_id)` — identical signatures, same per-`source_kind` branching (`email` special-case). Host sanitizes chunk-id colons → `-` (`content/paths.rs:284`); crate uses `sanitize_filename` (`paths.rs:179`). **Action: harness asserts identical relative paths for a corpus of colon/unicode/long chunk-ids** (Windows-illegal chars are the risk). |
| P7 | **YAML frontmatter** | ✅ **MATCH** (verify key order) | Summary markdown frontmatter delimited by `---\n … ---\n` (crate `content/compose/summary.rs:76,127`; `split_front_matter` on `rposition(line=="---")`). **Action: harness asserts identical frontmatter key set + order + serialization** for chunk & summary markdown (byte-compare composed files). |
| P8 | **Entity markdown** | ⏳ harness | `entities/` registry markdown. Host `memory_entities` (0 external refs) ↔ crate `entities/`. Low risk (full port, no drift). Harness byte-compares entity files. |
| P9 | **Git diff-ledger layout** | ✅ git-backed both | Host migrated to git-backed ledger 06-25 (`040e6e20d`), captured in port (crate `diff/ledger.rs`, `diff.rs:98`). Verify `.git` repo layout + snapshot markdown + read-marker storage identical. Harness opens an existing diff repo with both. |
| P10 | **`store_meta` / embedding signature** | ✅ format | `format_embedding_signature` = `"provider={name};model={model};dims={dims}"` (crate `store/vectors/embedding.rs`). Host must produce the identical signature string from `Config` (W1 `embeddings.rs` seam) or re-embed churn triggers. **Action: seam test asserts signature string equality.** |
| P11 | **Remaining `mem_tree_*` column parity** | ⏳ harness | `mem_tree_summaries`, `mem_tree_buffers`, `mem_tree_trees`, `mem_tree_score`, `mem_tree_entity_index`, `mem_tree_entity_edges`, `mem_tree_entity_hotness`, `mem_tree_ingested_sources`, `mem_tree_chunk_embeddings`, reembed-skipped tables, `kv_*`, `mcp_writes`, `legacy_marker`. Spot-checks clean; full column+index+PK diff is automated in the harness. |
| P12 | **Host-retained tier coexistence** | ⏳ W3 gate | Crate schema init + host `unified` init must both run on the shared DB without `CREATE`/index collisions. Harness opens a real workspace, runs crate init then host init (and vice-versa), asserts full `sqlite_master` superset is preserved and no data dropped. |

Legend: ✅ audited-identical · ⚠️ divergence flagged · ⏳ deferred to harness (automated per-flip).

---

## The golden-workspace parity harness (design)

The built-in parity harness is the read-side comparator that gates each risky flip. It exists at
two layers.

### Layer 1 — schema/format asserters (host-side unit tests, cheap, run every PR)

A `tests/tinycortex_parity/` module with pure-function comparators (no disk):

- **`chunk_id_parity`** — table of `(source_kind, source_id, seq, content)` → assert host `chunk_id`
  == `tinycortex::memory::chunks::chunk_id` (covers P1). *(After W3 both resolve to the crate; keep
  the vector as a regression pin.)*
- **`vector_roundtrip_parity`** — random `Vec<f32>` → `vec_to_bytes` → `bytes_to_vec` byte-equal
  across both (P2).
- **`content_path_parity`** — corpus of adversarial ids (colons, unicode, >255 chars, `email`
  source) → assert identical `chunk_rel_path`/`summary_rel_path` (P6).
- **`frontmatter_parity`** — compose a fixed chunk+summary → byte-compare markdown incl. frontmatter
  key order (P7).
- **`embedding_signature_parity`** — assert the W1 seam's `signature()` string == the format the
  store persisted into `store_meta` (P10).

### Layer 2 — golden-workspace differential harness (the flip gate)

The core mechanism from plan §0.3: **one on-disk workspace, opened by both engines, outputs compared.**

**Fixture.** Check in a small, deterministic `tests/fixtures/golden-workspace/` produced by the
*pre-migration* build: a real `chunks.db` + content vault + diff `.git`, seeded via a fixed script
(`scripts/gen-golden-workspace.sh`) with: a handful of chat + document + email sources across ≥2
namespaces, ingested + scored + sealed to ≥2 tree levels, some entities/edges, a few queue jobs in
mixed states, and both tiers populated (episodic/event/segment rows present). Store the **generator
script** and a **manifest** (expected chunk-ids, summary paths, recall snapshots) so the fixture is
regenerable and reviewable, not an opaque blob.

**Comparators (read-only, both engines open the SAME copied workspace):**

1. **Schema snapshot** — dump `sqlite_master` (tables, indexes, triggers, `sql` text normalized) from
   the DB after each engine's `open`/init; assert the crate-owned 20-table set is identical and the
   host-retained 10-table tier is untouched (P3–P5, P11, P12).
2. **Recall/retrieval snapshot** — run a fixed query battery through the stable public surface
   (`openhuman::memory::` recall + `read_rpc` retrieval primitives) on pre- and post-migration builds;
   assert identical ordered hit ids + scores (within f64 epsilon) + `supporting_relations` (guards G2)
   + taint (guards the security seam).
3. **Tree read snapshot** — `read_tree` / `drill_down` / `cover_window` over the sealed tree; assert
   identical node structure + summary content.
4. **Byte-compare vault** — after a read-only open, assert **no content files changed** (a flip must
   not rewrite the vault) and, for a controlled re-ingest of one source, assert composed markdown is
   byte-identical.
5. **Idempotent re-open** — open → close → open with the post-migration build; assert no migration
   churn (no re-embed storm, no schema rewrite) on an already-current DB.

**Wiring.** Runs under `pnpm test:rust` (host-side, counts toward coverage) and as a dedicated
`tests/memory_golden_parity_e2e.rs`. The existing crate-level integration tests
(`tests/memory_roundtrip_e2e.rs`, `memory_tree_sync_deep_raw_coverage_e2e.rs`) act as the
"public-surface still green" guard (plan §5.1); this harness adds the **differential** guard that a
flip preserves *existing* data, not just that the API still functions.

**Gate mapping:** Layer-1 asserters run every PR. Layer-2 golden harness is **green-before-merge** on
**W3** (store+chunks), **W5** (tree+retrieval+score), **W6** (ingest). W4 (queue) additionally asserts
job payload_json parity (P4/P9). Any red = upstream fix in tinycortex, re-bump submodule, re-run.

## Open divergences to resolve upstream before their flip

| Item | Flip gated | Resolution |
| --- | --- | --- |
| P5 `mem_tree_chunks` 3 legacy inline columns (fresh-DB) | W3 | Confirm no positional `INSERT`/`SELECT *`; if the columns are dead, drop them in a tinycortex PR so fresh DBs match. |
| P6 `sanitize_filename` vs host colon→`-` | W3 | Prove identical output on adversarial id corpus; align upstream if any diverge (Windows-illegal chars). |
| P7 frontmatter key **order** | W5/W6 | Byte-compare composed markdown; align serializer order upstream if diff. |
| G2 `supporting_relations` (graph_* host-retained vs crate derive-on-read) | W6/W7 | Recall snapshot (comparator 2) must match; else upstream relation persist. |

All other dimensions (P1–P4, P8–P12) audited compatible or covered by the automated harness.