diff --git a/docs/TEST-COVERAGE-MATRIX.md b/docs/TEST-COVERAGE-MATRIX.md index 4c14aa15f..e910e2558 100644 --- a/docs/TEST-COVERAGE-MATRIX.md +++ b/docs/TEST-COVERAGE-MATRIX.md @@ -293,14 +293,10 @@ Canonical mapping of every product feature to its test source(s). Drives gap-fil | ID | Feature | Layer | Test path(s) | Status | Notes | | ----- | ---------------------------------------- | ----- | ---------------------------------------------------------------------------------- | ------ | ----- | -| 8.3.1 | Cross-Chat Recall | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_cross_chat_recall` | ✅ | Synthetic fixture; verifies relevant source retrieval across chat scopes | | 8.3.2 | Cross-Chat Entity Discoverability | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_cross_chat_entity_discoverable` | ✅ | Verifies entity canonicalisation across multiple chats | | 8.3.3 | Citation Bundle Provenance | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_citation_bundle_provenance` | ✅ | Verifies source_ref and tree_scope are populated in retrieval hits | | 8.3.4 | Citation Fetch Leaves Hydration | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_citation_fetch_leaves_hydrates` | ✅ | Verifies fetch_leaves returns content for exact chunk IDs | -| 8.3.5 | Stale Preference Newer Supersedes | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_stale_preference_newer_supersedes` | ✅ | Verifies newer explicit correction appears alongside older preference | -| 8.3.6 | Contradiction Surfaces Both with Provenance | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_contradiction_surfaces_both_with_provenance` | ✅ | Verifies disagreeing sources surface with provenance labels | | 8.3.7 | Long-Source Exact Leaf Retrieval | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_long_source_retrieves_exact_leaf` | 🟡 | Embedder required for seal + chunking; test runs in inert mode but assertions are conditional | -| 8.3.8 | Drill-Down Isolates Children | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_drill_down_isolates_children` | ✅ | Verifies query_topic does not cross scope boundaries | | 8.3.9 | Scale Ingest 20 Sources No Real Data | RU | `src/openhuman/memory/tree/retrieval/benchmarks.rs::bench_scale_ingest_20_sources_no_real_data` | ✅ | Verifies retrieval correctness at scale with synthetic data | ### 8.4 Explicit User Preferences (Two-Lane) diff --git a/gitbooks/developing/architecture/memory-tree.md b/gitbooks/developing/architecture/memory-tree.md index bfc6ae34c..6d6801fed 100644 --- a/gitbooks/developing/architecture/memory-tree.md +++ b/gitbooks/developing/architecture/memory-tree.md @@ -2,15 +2,15 @@ description: >- The generic summary-tree engine under the Memory Tree feature - bucket-seal cascades, scoring, embedding, entity extraction, retrieval, summarisation. - Kind-agnostic mechanics that Source / Global / Topic trees all share. + Kind-agnostic mechanics for the Source trees (the only kind now built). icon: diagram-project --- # Memory Tree (`src/openhuman/memory_tree/`) -`src/openhuman/memory_tree/` is the **generic tree engine** sitting under the user-facing [Memory Tree feature](../../features/obsidian-wiki/memory-tree.md). It owns the kind-agnostic mechanics — appending leaves, cascading bucket seals, summarising one level to the next, scoring and embedding, retrieving for agents — that every concrete tree flavour (`Source`, `Global`, `Topic`) shares. It is deliberately **unaware** of which flavour a tree belongs to. +`src/openhuman/memory_tree/` is the **generic tree engine** sitting under the user-facing [Memory Tree feature](../../features/obsidian-wiki/memory-tree.md). It owns the kind-agnostic mechanics — appending leaves, cascading bucket seals, summarising one level to the next, scoring and embedding, retrieving for agents — that a concrete tree uses. It is deliberately **unaware** of which flavour a tree belongs to. -Kind-specific policy — when to spawn a topic tree, what scope a global tree covers, how digests are written — lives in `src/openhuman/memory/tree_global` and `src/openhuman/memory/tree_topic`. Persistence (the single `Tree` table and its schema) lives one layer down in `memory_store::trees`. This module sits between them. +> **Removed: Global & Topic trees.** Earlier revisions also built a singleton **Global** (cross-source, time-axis: day → week → month → year) digest tree and per-entity **Topic** (subject-axis) trees. Both were derived projections over the Source trees — no original content lived only in them — and were removed in favour of "walk the Source trees + the entity index." Source-tree policy lives in `src/openhuman/memory/tree_source`; persistence (the single `Tree` table) lives one layer down in `memory_store::trees`. The `TreeKind::Global`/`Topic` enum variants survive only as inert serialization plumbing so the one-shot purge migration can read and delete legacy rows. ```text memory (orchestrator) ──┐ @@ -36,7 +36,7 @@ memory_store::trees (persistence: one Tree table, one schema) | `io.rs` | Canonical contract types: `TreeWriteRequest` / `TreeWriteOutcome`, `TreeReadRequest` / `TreeReadHit` / `TreeReadResult`, `TreeLeafPayload`, `TreeLabelStrategy`. Pure types, no IO. | | `tree/` | `bucket_seal` (append leaf + cascade seal), `flush` (time-based partial seal), `registry` (kind-parameterized `get_or_create_tree` with UNIQUE-race recovery), `mod.rs` (re-exports + `memory_store::trees` shims for legacy paths). | | `summarise.rs` | One function: produce the next-level summary text for a bucket. Wraps the chat model with a fixed prompt and token budget. | -| `retrieval/` | Agent-facing tools. Read: `walk` (agentic), `drill_down`, `fetch_leaves`, `query_{source,global,topic}`, `search_entities`. Write: `ingest_document` (orchestrator-facing). | +| `retrieval/` | Agent-facing tools. Read: `walk` (agentic), `drill_down`, `fetch_leaves`, `query_source`, `search_entities`. Write: `ingest_document` (orchestrator-facing). (`query_global`/`query_topic` were removed with those trees.) | | `score/` | Scoring signals, embedding (cloud / Ollama / inert), entity extraction (regex / LLM), canonical resolver, entity index store. | | `tools.rs` | Re-exports from `memory::query` for backward compatibility. | | `tree_runtime/` | Tree-summarizer controller registry — exposed through `all_tree_summarizer_controller_schemas` / `all_tree_summarizer_registered_controllers` re-exports in `mod.rs`. | @@ -65,7 +65,7 @@ Agents reach this module through the tools in `retrieval/`: - `walk` — agentic exploration; the agent picks summary nodes to drill into. - `drill_down` — deterministic traversal from a known starting summary. - `fetch_leaves` — pull raw leaves for a sealed bucket. -- `query_{source,global,topic}` — kind-scoped retrieval; the orchestrator's tree-kind policy decides which one the agent sees. +- `query_source` — source-scoped retrieval (the only kind-scoped query left; `query_global`/`query_topic` were removed). - `search_entities` — entity-index lookup backed by `score/`. All retrieval handlers consult `memory_store::trees::hotness` so warm content surfaces first. diff --git a/src/openhuman/agent/harness/session/turn_tests.rs b/src/openhuman/agent/harness/session/turn_tests.rs index 2fd066aeb..fe740cc5a 100644 --- a/src/openhuman/agent/harness/session/turn_tests.rs +++ b/src/openhuman/agent/harness/session/turn_tests.rs @@ -889,6 +889,12 @@ async fn turn_runs_full_tool_cycle_with_context_and_hooks() { }, crate::openhuman::config::ContextConfig::default(), ); + // Suppress the memory-tree eager prefetch — it reads the real workspace + // via `load_config_with_timeout`, not the injected loader, so leaving it + // on would make this test depend on whatever is in `~/.openhuman`. This + // test exercises the injected memory context + tool cycle, not the + // prefetch; marking it already-prefetched skips that path deterministically. + agent.last_tree_prefetch_at = Some(std::time::Instant::now()); let response = agent .turn("hello world") @@ -966,6 +972,10 @@ async fn turn_uses_cached_transcript_prefix_on_first_iteration() { ChatMessage::system("cached-system"), ChatMessage::assistant("cached-assistant"), ]); + // Skip the memory-tree eager prefetch (reads the real workspace, not the + // injected loader) so the user message stays exactly "fresh" regardless + // of local `~/.openhuman` content. + agent.last_tree_prefetch_at = Some(std::time::Instant::now()); let response = agent.turn("fresh").await.expect("turn should succeed"); assert_eq!(response, "cached-final"); diff --git a/src/openhuman/agent/tree_loader.rs b/src/openhuman/agent/tree_loader.rs index 2ec5a87cc..e775f1ec8 100644 --- a/src/openhuman/agent/tree_loader.rs +++ b/src/openhuman/agent/tree_loader.rs @@ -1,19 +1,21 @@ -//! Eager prefetch of the cross-source memory-tree digest into the -//! orchestrator's session context (Phase 4 follow-on, #710 wiring). +//! Eager prefetch of recent memory-tree activity into the orchestrator's +//! session context (Phase 4 follow-on, #710 wiring). //! //! The orchestrator answers "what happened this week?" / "what's been going //! on with X?" style questions out of the user's own ingested memory. We -//! pre-load a 7-day global digest on the session's first turn AND -//! periodically thereafter (every [`REFRESH_INTERVAL`]) so long-running -//! conversations stay current with newly-ingested memory without needing -//! the LLM to round-trip a tool call. The injection rides on the user -//! message (NOT the system prompt) to keep the KV-cache prefix stable. +//! pre-load a 7-day recap on the session's first turn AND periodically +//! thereafter (every [`REFRESH_INTERVAL`]) so long-running conversations +//! stay current with newly-ingested memory without needing the LLM to +//! round-trip a tool call. The injection rides on the user message (NOT the +//! system prompt) to keep the KV-cache prefix stable. //! -//! When the workspace has no global summaries yet (early-life workspaces -//! or no ingest configured), [`TreeContextLoader::load`] returns an empty -//! string and the caller silently no-ops. The session-side timestamp is -//! still bumped on those empty results so an empty workspace doesn't get -//! re-queried every turn. +//! The recap is assembled by walking the **per-source** trees across the +//! window (the global digest tree was removed — source trees plus the +//! entity index are the substrate). When the workspace has no source +//! summaries yet (early-life workspaces or no ingest configured), +//! [`TreeContextLoader::load`] returns an empty string and the caller +//! silently no-ops. The session-side timestamp is still bumped on those +//! empty results so an empty workspace doesn't get re-queried every turn. //! //! Failure is non-fatal by design — the orchestrator must still be able to //! reply when the memory tree is unavailable, mis-configured, or empty. We @@ -21,7 +23,7 @@ //! concatenate without branching. use crate::openhuman::config::Config; -use crate::openhuman::memory_tree::retrieval::query_global; +use crate::openhuman::memory_tree::retrieval::query_source; /// Default lookback window for the eager digest. Mirrors the language in /// the orchestrator prompt ("7-day digest pre-loaded into session context"). @@ -39,9 +41,9 @@ pub const REFRESH_INTERVAL: std::time::Duration = std::time::Duration::from_secs /// would otherwise dominate the prompt budget. const MAX_CONTENT_CHARS: usize = 500; -/// Number of hits to surface from the digest. The recap typically returns -/// one hit per fold (day/week/month) — three is enough headroom for a -/// 7-day window without flooding the system prompt. +/// Number of hits to surface from the recap. Source-tree summaries across +/// the window are newest-first — three is enough headroom for a 7-day +/// window without flooding the system prompt. const MAX_HITS: usize = 3; const HEADER: &str = "[Memory tree — last 7 days]\n"; @@ -67,19 +69,30 @@ impl TreeContextLoader { /// Build the eager-prefetch context block for the current workspace. /// /// Returns: - /// - `Ok("")` when the workspace has no global digest yet, or when - /// `query_global` returns an error (logged at warn level). + /// - `Ok("")` when the workspace has no source summaries yet, or when + /// `query_source` returns an error (logged at warn level). /// - `Ok(rendered)` with the formatted block when there are hits. pub async fn load(config: &Config) -> anyhow::Result { log::debug!( "[memory_tree] tree_loader.load window_days={}", DEFAULT_WINDOW_DAYS ); - let resp = match query_global(config, DEFAULT_WINDOW_DAYS).await { + // Walk all source trees across the window (no source filter, no + // semantic query — just the most recent summaries everywhere). + let resp = match query_source( + config, + None, + None, + Some(DEFAULT_WINDOW_DAYS), + None, + MAX_HITS, + ) + .await + { Ok(r) => r, Err(e) => { log::warn!( - "[memory_tree] tree_loader.load: query_global failed — returning empty: {e}" + "[memory_tree] tree_loader.load: query_source failed — returning empty: {e}" ); return Ok(String::new()); } @@ -130,12 +143,12 @@ mod tests { } #[tokio::test] - async fn load_returns_empty_when_no_global_digest() { + async fn load_returns_empty_when_no_source_summaries() { let (_tmp, cfg) = empty_config(); let s = TreeContextLoader::load(&cfg).await.unwrap(); assert!( s.is_empty(), - "fresh workspace has no global digest — expected empty string, got: {s}" + "fresh workspace has no source summaries — expected empty string, got: {s}" ); } diff --git a/src/openhuman/memory/mod.rs b/src/openhuman/memory/mod.rs index 25915d511..c36ecf5ff 100644 --- a/src/openhuman/memory/mod.rs +++ b/src/openhuman/memory/mod.rs @@ -30,10 +30,10 @@ pub mod tools; pub mod util; // Tree instances — policy and orchestration over the generic memory_tree engine. -pub mod tree_global; +// The global (time-axis) and topic (subject-axis) trees were removed; source +// trees plus the entity index are the substrate. pub mod tree_policy; pub mod tree_source; -pub mod tree_topic; #[cfg(test)] mod sync_pipeline_e2e_tests; diff --git a/src/openhuman/memory/query/backend.rs b/src/openhuman/memory/query/backend.rs index 71c98d95d..60457973a 100644 --- a/src/openhuman/memory/query/backend.rs +++ b/src/openhuman/memory/query/backend.rs @@ -10,37 +10,26 @@ use anyhow::Result; use crate::openhuman::config::Config; use crate::openhuman::memory_store::chunks::types::SourceKind; use crate::openhuman::memory_tree::retrieval::{self, QueryResponse, RetrievalHit}; -use crate::openhuman::memory_tree::tree::TreeProfile; -pub async fn query_profile( +/// Query the per-source summary trees. The global (time-axis) and topic +/// (subject-axis) trees were removed; source trees plus the entity index are +/// the substrate, so this is the only remaining tree-query backend. +pub async fn query_source_scope( config: &Config, - profile: TreeProfile, scope: Option<&str>, time_window_days: Option, query: Option<&str>, limit: usize, ) -> Result { - match profile { - TreeProfile::Source => { - retrieval::source::query_source( - config, - scope, - None::, - time_window_days, - query, - limit, - ) - .await - } - TreeProfile::Topic => { - let entity_id = - scope.ok_or_else(|| anyhow::anyhow!("topic query requires scope/entity_id"))?; - retrieval::topic::query_topic(config, entity_id, time_window_days, query, limit).await - } - TreeProfile::Global => { - retrieval::global::query_global(config, time_window_days.unwrap_or(7)).await - } - } + retrieval::source::query_source( + config, + scope, + None::, + time_window_days, + query, + limit, + ) + .await } pub async fn query_source_kind( diff --git a/src/openhuman/memory/query/ingest_document.rs b/src/openhuman/memory/query/ingest_document.rs index 719167767..04157b9d3 100644 --- a/src/openhuman/memory/query/ingest_document.rs +++ b/src/openhuman/memory/query/ingest_document.rs @@ -20,7 +20,7 @@ impl Tool for MemoryTreeIngestDocumentTool { This is the write path into the knowledge index — use it after \ fetching web content, extracting facts, or collecting data from \ external sources. The ingested document will be chunked, embedded, \ - and available via query_global, query_source, and search_entities." + and available via query_source and search_entities." } fn parameters_schema(&self) -> serde_json::Value { diff --git a/src/openhuman/memory/query/mod.rs b/src/openhuman/memory/query/mod.rs index d8e7e8077..bf27c3e7b 100644 --- a/src/openhuman/memory/query/mod.rs +++ b/src/openhuman/memory/query/mod.rs @@ -10,9 +10,7 @@ mod backend; mod drill_down; mod fetch_leaves; mod ingest_document; -mod query_global; mod query_source; -mod query_topic; mod search_entities; pub mod walk; @@ -21,9 +19,7 @@ pub mod walk; pub use drill_down::MemoryTreeDrillDownTool; pub use fetch_leaves::MemoryTreeFetchLeavesTool; pub use ingest_document::MemoryTreeIngestDocumentTool; -pub use query_global::MemoryTreeQueryGlobalTool; pub use query_source::MemoryTreeQuerySourceTool; -pub use query_topic::MemoryTreeQueryTopicTool; pub use search_entities::MemoryTreeSearchEntitiesTool; pub use walk::MemoryTreeWalkTool as MemoryQueryWalkTool; pub use walk::{run_walk, MemoryTreeWalkTool, WalkOptions, WalkOutcome, WalkStep, WalkStopReason}; @@ -48,9 +44,7 @@ impl Tool for MemoryTreeTool { "Query the user's ingested email/chat/document memory tree. \ Set `mode` to one of: `search_entities` (resolve a name to a \ canonical id — call first when the user mentions someone by name), \ - `query_topic` (all cross-source mentions of an entity), \ `query_source` (filter by source type + time window), \ - `query_global` (cross-source daily digest), \ `drill_down` (expand a coarse summary one level), \ `fetch_leaves` (pull raw chunks for citation), `ingest_document` (write a document into the tree for future retrieval), \ `walk` (agentic multi-turn walk — LLM navigates summaries and returns a synthesized answer for a natural-language query)." @@ -62,25 +56,20 @@ impl Tool for MemoryTreeTool { "properties": { "mode": { "type": "string", - "enum": ["search_entities", "query_topic", "query_source", - "query_global", "drill_down", "fetch_leaves", "ingest_document", "walk"], + "enum": ["search_entities", "query_source", + "drill_down", "fetch_leaves", "ingest_document", "walk"], "description": "Which operation to run (retrieval or write)." }, // search_entities params "query": { "type": "string", - "description": "search_entities: substring to match. query_topic/query_source: semantic rerank query (optional). walk: natural-language question to answer by walking the memory tree." + "description": "search_entities: substring to match. query_source: semantic rerank query (optional). walk: natural-language question to answer by walking the memory tree." }, "kinds": { "type": "array", "items": {"type": "string"}, "description": "search_entities: optional entity kind filter (email, url, handle, person, ...)." }, - // query_topic params - "entity_id": { - "type": "string", - "description": "query_topic: canonical entity id returned by search_entities." - }, // query_source params "source_kind": { "type": "string", @@ -88,11 +77,7 @@ impl Tool for MemoryTreeTool { }, "time_window_days": { "type": "integer", - "description": "query_source/query_topic: look-back window in days. query_global also accepts this as a compatibility alias." - }, - "window_days": { - "type": "integer", - "description": "query_global: look-back window in days." + "description": "query_source: look-back window in days." }, // drill_down params "node_id": { @@ -148,13 +133,7 @@ impl Tool for MemoryTreeTool { log::debug!("[tool][memory_tree] mode={mode}"); match mode { "search_entities" => MemoryTreeSearchEntitiesTool.execute(args).await, - "query_topic" => MemoryTreeQueryTopicTool.execute(args).await, "query_source" => MemoryTreeQuerySourceTool.execute(args).await, - "query_global" => { - MemoryTreeQueryGlobalTool - .execute(translate_query_global_args(args)) - .await - } "drill_down" => MemoryTreeDrillDownTool.execute(args).await, "fetch_leaves" => MemoryTreeFetchLeavesTool.execute(args).await, "ingest_document" => MemoryTreeIngestDocumentTool.execute(args).await, @@ -162,36 +141,13 @@ impl Tool for MemoryTreeTool { other => { log::debug!("[tool][memory_tree] unknown_mode mode={other}"); Err(anyhow::anyhow!( - "memory_tree: unknown mode `{other}`. Valid: search_entities, query_topic, query_source, query_global, drill_down, fetch_leaves, ingest_document, walk" + "memory_tree: unknown mode `{other}`. Valid: search_entities, query_source, drill_down, fetch_leaves, ingest_document, walk" )) } } } } -/// Rename the consolidated tool's `time_window_days` field to `window_days` -/// before dispatching to [`MemoryTreeQueryGlobalTool`]. -/// -/// The consolidated `parameters_schema()` exposes one shared -/// `time_window_days` field for both `query_source` and `query_global` (the -/// `query_source` backend uses that exact name). `QueryGlobalRequest` in -/// `memory_tree/retrieval/rpc.rs` uses `time_window_days` as its primary -/// field name and accepts `window_days` via `#[serde(alias = "window_days")]`. -/// The translation here keeps the LLM-facing consolidated schema stable -/// (callers always send `time_window_days`) while routing through the alias -/// path, which is equivalent. An explicit `window_days` in the payload -/// always wins — the translator is a no-op for callers that already use it. -fn translate_query_global_args(mut args: serde_json::Value) -> serde_json::Value { - if let Some(obj) = args.as_object_mut() { - if !obj.contains_key("window_days") { - if let Some(value) = obj.remove("time_window_days") { - obj.insert("window_days".to_string(), value); - } - } - } - args -} - #[cfg(test)] mod memory_tree_dispatcher_tests { use super::*; @@ -226,23 +182,23 @@ mod memory_tree_dispatcher_tests { .filter_map(|v| v.as_str()) .collect(); assert!(modes.contains(&"search_entities")); - assert!(modes.contains(&"query_topic")); assert!(modes.contains(&"query_source")); - assert!(modes.contains(&"query_global")); assert!(modes.contains(&"drill_down")); assert!(modes.contains(&"fetch_leaves")); assert!(modes.contains(&"ingest_document")); assert!(modes.contains(&"walk")); + // Removed with the global/topic trees. + assert!(!modes.contains(&"query_topic")); + assert!(!modes.contains(&"query_global")); } #[test] - fn memory_tree_schema_exposes_global_window_days() { + fn memory_tree_schema_exposes_source_window_days() { let schema = MemoryTreeTool.parameters_schema(); let properties = schema .get("properties") .and_then(|p| p.as_object()) .unwrap(); - assert!(properties.contains_key("window_days")); assert!(properties.contains_key("time_window_days")); } @@ -265,30 +221,6 @@ mod memory_tree_dispatcher_tests { assert!(result.is_err()); } - #[tokio::test] - async fn memory_tree_query_global_mode_dispatches_successfully() { - // Hold TEST_ENV_LOCK for the duration of the test so that concurrent - // tests using WorkspaceEnvGuard (which sets OPENHUMAN_WORKSPACE to a - // temp path and then deletes it) cannot race with the Config::load_or_init - // call inside MemoryTreeTool.execute — the race caused "Failed to read - // config file" when the temp dir was deleted between exists() and read(). - let _env_guard = crate::openhuman::config::TEST_ENV_LOCK - .lock() - .unwrap_or_else(|e| e.into_inner()); - let result = MemoryTreeTool - .execute(json!({ - "mode": "query_global", - "time_window_days": 7 - })) - .await - .expect("query_global mode should dispatch successfully"); - assert!(!result.is_error); - let parsed: serde_json::Value = - serde_json::from_str(&result.text()).expect("result should be valid json"); - assert!(parsed.get("hits").is_some()); - assert!(parsed.get("total").is_some()); - } - #[tokio::test] async fn memory_tree_fetch_leaves_mode_dispatches_successfully() { let result = MemoryTreeTool @@ -303,65 +235,4 @@ mod memory_tree_dispatcher_tests { serde_json::from_str(&result.text()).expect("result should be valid json"); assert!(parsed.is_array()); } - - #[test] - fn translate_query_global_args_renames_time_window_days_to_window_days() { - // Per-issue #2252: the consolidated schema advertises `time_window_days` - // but `QueryGlobalRequest` deserializes from `window_days`. The - // translation closes that gap inside the dispatch. - let translated = translate_query_global_args(json!({ - "mode": "query_global", - "time_window_days": 7, - })); - assert_eq!( - translated, - json!({ - "mode": "query_global", - "window_days": 7, - }), - ); - } - - #[test] - fn translate_query_global_args_passes_through_window_days_when_already_set() { - // The standalone `MemoryTreeQueryGlobalTool` schema advertises - // `window_days` natively — callers using that path should reach the - // backend unchanged. - let translated = translate_query_global_args(json!({ - "mode": "query_global", - "window_days": 30, - })); - assert_eq!(translated["window_days"], 30); - assert!(translated.get("time_window_days").is_none()); - } - - #[test] - fn translate_query_global_args_prefers_explicit_window_days_over_time_window_days() { - // If a caller somehow supplies both, the underlying contract wins - // (`window_days` is what the deserializer reads). Without this, - // a future caller migrating to `window_days` while leaving a stale - // `time_window_days` in the payload would silently lose their - // explicit choice to the legacy alias. - let translated = translate_query_global_args(json!({ - "mode": "query_global", - "window_days": 30, - "time_window_days": 7, - })); - assert_eq!(translated["window_days"], 30); - } - - #[test] - fn translate_query_global_args_leaves_payload_untouched_when_neither_field_present() { - // The underlying tool surfaces its own missing-field error in this - // case; the translator should not invent a value. - let translated = translate_query_global_args(json!({ - "mode": "query_global", - })); - assert_eq!( - translated, - json!({ - "mode": "query_global", - }), - ); - } } diff --git a/src/openhuman/memory/query/query_global.rs b/src/openhuman/memory/query/query_global.rs deleted file mode 100644 index c40c3dbef..000000000 --- a/src/openhuman/memory/query/query_global.rs +++ /dev/null @@ -1,186 +0,0 @@ -use crate::openhuman::config::rpc as config_rpc; -use crate::openhuman::memory::query::backend; -use crate::openhuman::memory_tree::retrieval::rpc::QueryGlobalRequest; -use crate::openhuman::memory_tree::tree::TreeProfile; -use crate::openhuman::tools::traits::{Tool, ToolResult}; -use async_trait::async_trait; -use serde_json::json; - -pub struct MemoryTreeQueryGlobalTool; - -#[async_trait] -impl Tool for MemoryTreeQueryGlobalTool { - fn name(&self) -> &str { - "memory_tree_query_global" - } - - fn description(&self) -> &str { - "Return the cross-source global digest for the last `time_window_days`. \ - The 7-day digest is also pre-loaded into the session context at \ - start, so only call this for a different window (e.g. 30 days, \ - 1 day) or to refresh after new ingest." - } - - fn parameters_schema(&self) -> serde_json::Value { - json!({ - "type": "object", - "properties": { - "time_window_days": { - "type": "integer", - "minimum": 1, - "description": "Lookback window in days (e.g. 7 for weekly recap)." - } - }, - "required": ["time_window_days"] - }) - } - - async fn execute(&self, args: serde_json::Value) -> anyhow::Result { - log::debug!("[tool][memory_tree] query_global invoked"); - let req: QueryGlobalRequest = serde_json::from_value(args) - .map_err(|e| anyhow::anyhow!("invalid arguments for memory_tree_query_global: {e}"))?; - let cfg = config_rpc::load_config_with_timeout() - .await - .map_err(|e| anyhow::anyhow!("memory_tree_query_global: load config failed: {e}"))?; - let resp = backend::query_profile( - &cfg, - TreeProfile::Global, - None, - Some(req.time_window_days), - None, - 10, - ) - .await?; - log::debug!( - "[tool][memory_tree] query_global returning hits={} total={}", - resp.hits.len(), - resp.total - ); - let json = serde_json::to_string(&resp)?; - Ok(ToolResult::success(json)) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use std::ffi::OsString; - - use tempfile::TempDir; - - use crate::openhuman::config::{Config, TEST_ENV_LOCK}; - use crate::openhuman::tools::traits::Tool; - use serde_json::json; - - struct WorkspaceEnvGuard { - _lock: std::sync::MutexGuard<'static, ()>, - previous: Option, - } - - impl WorkspaceEnvGuard { - fn set(path: &std::path::Path) -> Self { - let lock = TEST_ENV_LOCK.lock().unwrap_or_else(|err| err.into_inner()); - let previous = std::env::var_os("OPENHUMAN_WORKSPACE"); - std::env::set_var("OPENHUMAN_WORKSPACE", path); - Self { - _lock: lock, - previous, - } - } - } - - impl Drop for WorkspaceEnvGuard { - fn drop(&mut self) { - if let Some(previous) = self.previous.as_ref() { - std::env::set_var("OPENHUMAN_WORKSPACE", previous); - } else { - std::env::remove_var("OPENHUMAN_WORKSPACE"); - } - } - } - - async fn isolated_config(tmp: &TempDir) -> (WorkspaceEnvGuard, Config) { - let guard = WorkspaceEnvGuard::set(tmp.path()); - let config = Config::load_or_init().await.expect("load config"); - (guard, config) - } - - #[test] - fn parameters_schema_requires_time_window_days() { - let tool = MemoryTreeQueryGlobalTool; - let schema = tool.parameters_schema(); - assert_eq!(schema["required"], json!(["time_window_days"])); - assert_eq!(schema["properties"]["time_window_days"]["minimum"], 1); - } - - #[tokio::test] - async fn execute_rejects_missing_time_window_days() { - let tool = MemoryTreeQueryGlobalTool; - let err = tool - .execute(json!({})) - .await - .expect_err("missing time_window_days should fail"); - assert!(err - .to_string() - .contains("invalid arguments for memory_tree_query_global")); - } - - #[tokio::test] - async fn execute_rejects_wrong_type_for_time_window_days() { - let tool = MemoryTreeQueryGlobalTool; - let err = tool - .execute(json!({"time_window_days": "seven"})) - .await - .expect_err("wrong type should fail"); - assert!(err - .to_string() - .contains("invalid arguments for memory_tree_query_global")); - } - - #[tokio::test] - async fn execute_accepts_window_days_alias() { - let tmp = TempDir::new().expect("tempdir"); - let (_workspace, _cfg) = isolated_config(&tmp).await; - let tool = MemoryTreeQueryGlobalTool; - let req: QueryGlobalRequest = - serde_json::from_value(json!({"window_days": 7})).expect("alias should deserialize"); - assert_eq!(req.time_window_days, 7); - - let result = tool - .execute(json!({"window_days": 7})) - .await - .expect("window_days alias should succeed"); - assert!(!result.is_error); - } - - #[tokio::test] - async fn execute_success_path_returns_empty_payload_for_isolated_workspace() { - let tmp = TempDir::new().expect("tempdir"); - let (_workspace, cfg) = isolated_config(&tmp).await; - let tool = MemoryTreeQueryGlobalTool; - let result = tool - .execute(json!({"time_window_days": 7})) - .await - .expect("valid query_global should succeed in isolated workspace"); - assert!(!result.is_error); - let payload = result.text(); - let parsed: serde_json::Value = - serde_json::from_str(&payload).expect("result should be valid json"); - assert!( - parsed.get("hits").is_some(), - "payload should include hits array" - ); - assert!( - parsed.get("total").is_some(), - "payload should include total count" - ); - assert_eq!(parsed["total"], json!(0)); - assert_eq!(parsed["hits"], json!([])); - - let direct = crate::openhuman::memory_tree::retrieval::global::query_global(&cfg, 7) - .await - .expect("direct query_global on empty workspace"); - assert_eq!(direct.total, 0); - assert!(direct.hits.is_empty()); - } -} diff --git a/src/openhuman/memory/query/query_source.rs b/src/openhuman/memory/query/query_source.rs index e18279f70..0a56e4d2b 100644 --- a/src/openhuman/memory/query/query_source.rs +++ b/src/openhuman/memory/query/query_source.rs @@ -69,9 +69,8 @@ impl Tool for MemoryTreeQuerySourceTool { }; let resp = match req.source_id.as_deref() { Some(source_id) => { - backend::query_profile( + backend::query_source_scope( &cfg, - crate::openhuman::memory_tree::tree::TreeProfile::Source, Some(source_id), req.time_window_days, req.query.as_deref(), diff --git a/src/openhuman/memory/query/query_topic.rs b/src/openhuman/memory/query/query_topic.rs deleted file mode 100644 index 37be3b2c2..000000000 --- a/src/openhuman/memory/query/query_topic.rs +++ /dev/null @@ -1,205 +0,0 @@ -use crate::openhuman::config::rpc as config_rpc; -use crate::openhuman::memory::query::backend; -use crate::openhuman::memory_tree::retrieval::rpc::QueryTopicRequest; -use crate::openhuman::memory_tree::tree::TreeProfile; -use crate::openhuman::tools::traits::{Tool, ToolResult}; -use async_trait::async_trait; -use serde_json::json; - -pub struct MemoryTreeQueryTopicTool; - -#[async_trait] -impl Tool for MemoryTreeQueryTopicTool { - fn name(&self) -> &str { - "memory_tree_query_topic" - } - - fn description(&self) -> &str { - "Return summaries / chunks linked to a canonical entity id (e.g. \ - `email:alice@example.com`, `topic:phoenix`) across every memory \ - tree. Sorted by score then recency, or by cosine similarity if \ - `query` is provided. Use this after `memory_tree_search_entities` \ - resolves a name to a canonical id." - } - - fn parameters_schema(&self) -> serde_json::Value { - json!({ - "type": "object", - "properties": { - "entity_id": { - "type": "string", - "description": "Canonical entity id (e.g. `email:alice@example.com`, `topic:phoenix`)." - }, - "time_window_days": { - "type": "integer", - "minimum": 0, - "description": "Only return hits whose time range overlaps the last N days." - }, - "query": { - "type": "string", - "description": "Optional natural-language query for cosine-similarity rerank." - }, - "limit": { - "type": "integer", - "minimum": 0, - "description": "Max hits to return (default 10)." - } - }, - "required": ["entity_id"] - }) - } - - async fn execute(&self, args: serde_json::Value) -> anyhow::Result { - log::debug!("[tool][memory_tree] query_topic invoked"); - let req: QueryTopicRequest = serde_json::from_value(args) - .map_err(|e| anyhow::anyhow!("invalid arguments for memory_tree_query_topic: {e}"))?; - let cfg = config_rpc::load_config_with_timeout() - .await - .map_err(|e| anyhow::anyhow!("memory_tree_query_topic: load config failed: {e}"))?; - let resp = backend::query_profile( - &cfg, - TreeProfile::Topic, - Some(req.entity_id.as_str()), - req.time_window_days, - req.query.as_deref(), - req.limit.unwrap_or(10), - ) - .await?; - log::debug!( - "[tool][memory_tree] query_topic returning hits={} total={}", - resp.hits.len(), - resp.total - ); - let json = serde_json::to_string(&resp)?; - Ok(ToolResult::success(json)) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use std::ffi::OsString; - - use tempfile::TempDir; - - use crate::openhuman::config::{Config, TEST_ENV_LOCK}; - use crate::openhuman::tools::traits::Tool; - use serde_json::json; - - struct WorkspaceEnvGuard { - _lock: std::sync::MutexGuard<'static, ()>, - previous: Option, - } - - impl WorkspaceEnvGuard { - fn set(path: &std::path::Path) -> Self { - let lock = TEST_ENV_LOCK.lock().unwrap_or_else(|err| err.into_inner()); - let previous = std::env::var_os("OPENHUMAN_WORKSPACE"); - std::env::set_var("OPENHUMAN_WORKSPACE", path); - Self { - _lock: lock, - previous, - } - } - } - - impl Drop for WorkspaceEnvGuard { - fn drop(&mut self) { - if let Some(previous) = self.previous.as_ref() { - std::env::set_var("OPENHUMAN_WORKSPACE", previous); - } else { - std::env::remove_var("OPENHUMAN_WORKSPACE"); - } - } - } - - async fn isolated_config(tmp: &TempDir) -> (WorkspaceEnvGuard, Config) { - let guard = WorkspaceEnvGuard::set(tmp.path()); - let config = Config::load_or_init().await.expect("load config"); - (guard, config) - } - - #[test] - fn parameters_schema_requires_entity_id() { - let tool = MemoryTreeQueryTopicTool; - let schema = tool.parameters_schema(); - assert_eq!(schema["required"], json!(["entity_id"])); - assert_eq!(schema["properties"]["time_window_days"]["minimum"], 0); - } - - #[tokio::test] - async fn execute_rejects_missing_entity_id() { - let tool = MemoryTreeQueryTopicTool; - let err = tool - .execute(json!({})) - .await - .expect_err("missing entity_id should fail"); - assert!(err - .to_string() - .contains("invalid arguments for memory_tree_query_topic")); - } - - #[tokio::test] - async fn execute_rejects_wrong_type_for_entity_id() { - let tool = MemoryTreeQueryTopicTool; - let err = tool - .execute(json!({"entity_id": 42})) - .await - .expect_err("wrong type should fail"); - assert!(err - .to_string() - .contains("invalid arguments for memory_tree_query_topic")); - } - - #[tokio::test] - async fn execute_success_path_returns_empty_payload_for_isolated_workspace() { - let tmp = TempDir::new().expect("tempdir"); - let (_workspace, cfg) = isolated_config(&tmp).await; - let tool = MemoryTreeQueryTopicTool; - let result = tool - .execute(json!({ - "entity_id": "topic:phoenix", - "limit": 2 - })) - .await - .expect("valid query_topic should succeed in isolated workspace"); - assert!(!result.is_error); - let payload = result.text(); - let parsed: serde_json::Value = - serde_json::from_str(&payload).expect("result should be valid json"); - assert!(parsed.get("hits").is_some(), "payload should include hits"); - assert!( - parsed.get("total").is_some(), - "payload should include total" - ); - assert_eq!(parsed["hits"], json!([])); - assert_eq!(parsed["total"], json!(0)); - - let direct = crate::openhuman::memory_tree::retrieval::topic::query_topic( - &cfg, - "topic:phoenix", - None, - None, - 2, - ) - .await - .expect("direct query_topic on empty workspace"); - assert!(direct.hits.is_empty()); - assert_eq!(direct.total, 0); - } - - #[tokio::test] - async fn execute_accepts_time_window_without_query() { - let tmp = TempDir::new().expect("tempdir"); - let (_workspace, _cfg) = isolated_config(&tmp).await; - let tool = MemoryTreeQueryTopicTool; - let result = tool - .execute(json!({ - "entity_id": "email:alice@example.com", - "time_window_days": 7 - })) - .await - .expect("time-window-only topic query should succeed"); - assert!(!result.is_error); - } -} diff --git a/src/openhuman/memory/schema.rs b/src/openhuman/memory/schema.rs index d8e2e4b52..56b3dfa9e 100644 --- a/src/openhuman/memory/schema.rs +++ b/src/openhuman/memory/schema.rs @@ -1,7 +1,7 @@ //! Controller schemas for the memory tree. //! //! Registered JSON-RPC methods include the original Phase 1 surface -//! (`ingest`, `list_chunks`, `get_chunk`, `trigger_digest`) plus the new +//! (`ingest`, `list_chunks`, `get_chunk`) plus the new //! Memory-tab read RPCs added by the cloud-default backend refactor: //! `list_sources`, `search`, `recall`, `entity_index_for`, //! `top_entities`, `chunk_score`, `delete_chunk`, and destructive @@ -29,7 +29,6 @@ pub fn all_controller_schemas() -> Vec { schemas("ingest"), schemas("list_chunks"), schemas("get_chunk"), - schemas("trigger_digest"), schemas("memory_backfill_status"), schemas("list_sources"), schemas("search"), @@ -65,10 +64,6 @@ pub fn all_registered_controllers() -> Vec { schema: schemas("get_chunk"), handler: handle_get_chunk, }, - RegisteredController { - schema: schemas("trigger_digest"), - handler: handle_trigger_digest, - }, RegisteredController { schema: schemas("memory_backfill_status"), handler: handle_memory_backfill_status, @@ -661,44 +656,6 @@ pub fn schemas(function: &str) -> ControllerSchema { }, ], }, - "trigger_digest" => ControllerSchema { - namespace: NAMESPACE, - function: "trigger_digest", - description: "Manually enqueue a daily-digest job for the global \ - tree. Idempotent — re-running for a day that already has a \ - digest is a no-op (the handler skips). When no date is \ - supplied, defaults to yesterday in UTC, matching the \ - scheduler's autonomous behavior.", - inputs: vec![FieldSchema { - name: "date_iso", - ty: TypeSchema::Option(Box::new(TypeSchema::String)), - comment: "UTC calendar date as `YYYY-MM-DD`. Optional; \ - defaults to yesterday when omitted.", - required: false, - }], - outputs: vec![ - FieldSchema { - name: "enqueued", - ty: TypeSchema::Bool, - comment: "True when a fresh job row was inserted; false \ - when an active job for the same date suppressed it.", - required: true, - }, - FieldSchema { - name: "job_id", - ty: TypeSchema::Option(Box::new(TypeSchema::String)), - comment: "ID of the newly enqueued job row, when enqueued.", - required: false, - }, - FieldSchema { - name: "date_iso", - ty: TypeSchema::String, - comment: "The date the digest will cover, echoed back \ - as `YYYY-MM-DD`.", - required: true, - }, - ], - }, "pipeline_status" => ControllerSchema { namespace: NAMESPACE, function: "pipeline_status", @@ -867,14 +824,6 @@ fn handle_get_chunk(params: Map) -> ControllerFuture { }) } -fn handle_trigger_digest(params: Map) -> ControllerFuture { - Box::pin(async move { - let config = config_rpc::load_config_with_timeout().await?; - let req = parse_value::(Value::Object(params))?; - to_json(rpc::trigger_digest_rpc(&config, req).await?) - }) -} - fn handle_memory_backfill_status(_params: Map) -> ControllerFuture { Box::pin(async move { let config = config_rpc::load_config_with_timeout().await?; diff --git a/src/openhuman/memory/sync_pipeline_e2e_tests.rs b/src/openhuman/memory/sync_pipeline_e2e_tests.rs index b80c286bc..763474b40 100644 --- a/src/openhuman/memory/sync_pipeline_e2e_tests.rs +++ b/src/openhuman/memory/sync_pipeline_e2e_tests.rs @@ -24,21 +24,15 @@ use crate::core::event_bus::{ init_global as init_event_bus, subscribe_global, DomainEvent, EventHandler, SubscriptionHandle, }; use crate::openhuman::config::Config; -use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; use crate::openhuman::memory::ingest_pipeline::ingest_chat; use crate::openhuman::memory::sync::{emit_sync_stage, MemorySyncStage, MemorySyncTrigger}; -use crate::openhuman::memory::tree_global::digest::{end_of_day_digest, DigestOutcome}; -use crate::openhuman::memory::tree_topic::curator::{force_recompute, SpawnOutcome}; use crate::openhuman::memory_queue::{self, count_total, drain_until_idle, JobStatus}; use crate::openhuman::memory_store::chunks::store::{ count_chunks, count_chunks_by_lifecycle_status, CHUNK_STATUS_BUFFERED, }; -use crate::openhuman::memory_store::trees::{ - hotness, store as tree_store, - types::{HotnessCounters, TreeKind}, -}; +use crate::openhuman::memory_store::trees::{store as tree_store, types::TreeKind}; use crate::openhuman::memory_sync::canonicalize::chat::{ChatBatch, ChatMessage}; -use crate::openhuman::memory_tree::retrieval::{query_source, query_topic, search_entities}; +use crate::openhuman::memory_tree::retrieval::{query_source, search_entities}; use crate::openhuman::memory_tree::score::store::lookup_entity; // ── helpers ───────────────────────────────────────────────────────────── @@ -299,67 +293,8 @@ async fn multi_batch_volume_builds_full_tree() { .iter() .any(|m| m.canonical_id == "email:alice@example.com")); - // ── Global digest ────────────────────────────────────────────────── - - let provider: Arc = Arc::new(StaticChatProvider::new("daily digest summary")); - - let digest_day = Utc.timestamp_millis_opt(base_ts).unwrap().date_naive(); - - let digest_outcome = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, digest_day).await.unwrap() - }) - .await; - - match &digest_outcome { - DigestOutcome::Emitted { - source_count, - daily_id, - .. - } => { - assert!(*source_count >= 1); - assert!(!daily_id.is_empty()); - } - DigestOutcome::EmptyDay => { - // Acceptable — timestamp alignment may miss the exact day. - } - DigestOutcome::Skipped { .. } => { - panic!("first run should not skip"); - } - } - - // ── Topic tree ───────────────────────────────────────────────────── - - let mut counters = HotnessCounters::fresh("email:alice@example.com", 0); - counters.mention_count_30d = 500; - counters.distinct_sources = 2; - counters.last_seen_ms = Some(Utc::now().timestamp_millis()); - counters.query_hits_30d = 5; - hotness::upsert(&cfg, &counters).unwrap(); - - let spawn = test_override::with_provider(Arc::clone(&provider), async { - force_recompute(&cfg, "email:alice@example.com") - .await - .unwrap() - }) - .await; - - match &spawn { - SpawnOutcome::Spawned { tree_id, .. } | SpawnOutcome::TreeExists { tree_id, .. } => { - assert!(tree_id.starts_with("topic:")); - } - other => panic!("expected Spawned or TreeExists, got {other:?}"), - } - - // Drain backfill follow-up jobs. - drain_until_idle(&cfg).await.unwrap(); - - let topic_trees = tree_store::list_trees_by_kind(&cfg, TreeKind::Topic).unwrap(); - assert!(!topic_trees.is_empty()); - - let topic_resp = query_topic(&cfg, "email:alice@example.com", None, None, 10) - .await - .unwrap(); - assert!(!topic_resp.hits.is_empty()); + // (The global-digest and topic-spawn steps were removed with those + // trees — source trees plus the entity index are the substrate.) // Verify event stream. tokio::task::yield_now().await; diff --git a/src/openhuman/memory/tree_e2e_tests.rs b/src/openhuman/memory/tree_e2e_tests.rs index 5352b8cf6..3e728fc4c 100644 --- a/src/openhuman/memory/tree_e2e_tests.rs +++ b/src/openhuman/memory/tree_e2e_tests.rs @@ -19,15 +19,9 @@ use tempfile::TempDir; use crate::openhuman::config::Config; use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; use crate::openhuman::memory::ingest_pipeline::ingest_chat; -use crate::openhuman::memory::tree_global::digest::{end_of_day_digest, DigestOutcome}; -use crate::openhuman::memory::tree_topic::curator::{force_recompute, SpawnOutcome}; use crate::openhuman::memory_queue::drain_until_idle; -use crate::openhuman::memory_store::trees::hotness; -use crate::openhuman::memory_store::trees::types::HotnessCounters; use crate::openhuman::memory_sync::canonicalize::chat::{ChatBatch, ChatMessage}; -use crate::openhuman::memory_tree::retrieval::{ - query_global, query_source, query_topic, search_entities, -}; +use crate::openhuman::memory_tree::retrieval::{query_source, search_entities}; use crate::openhuman::memory_tree::score::embed::build_embedder_from_config; fn test_config() -> (TempDir, Config) { @@ -75,31 +69,16 @@ fn heavy_batch(platform: &str, channel: &str, seq_offset: u32) -> ChatBatch { } } -/// Seed hotness counters so `force_recompute` crosses the creation threshold -/// even if the extract jobs haven't yet produced enough cross-source index -/// rows on their own. -fn seed_hotness(cfg: &Config, entity_id: &str, distinct_sources: u32) { - let mut counters = HotnessCounters::fresh(entity_id, 0); - counters.mention_count_30d = 500; - counters.distinct_sources = distinct_sources; - counters.last_seen_ms = Some(Utc::now().timestamp_millis()); - counters.query_hits_30d = 5; - hotness::upsert(cfg, &counters).unwrap(); -} - -/// Full pipeline: ingest → seal → global digest → topic spawn → retrieval. +/// Full pipeline: ingest → seal → source retrieval → entity search. /// /// Steps: /// 1. Ingest heavy batches from two distinct sources so the source trees' /// L0 buffers cross the token-budget seal threshold. /// 2. Drain the async job queue so extract / admit / buffer / seal jobs run. /// 3. Verify source-tree retrieval returns sealed summaries. -/// 4. Run the end-of-day digest and assert it emits a global node. -/// 5. Verify global retrieval returns at least one hit. -/// 6. Pre-seed hotness counters and call `force_recompute` to spawn the -/// topic tree for `email:alice@example.com`. -/// 7. Verify topic-tree retrieval returns results. -/// 8. Verify `search_entities("alice")` resolves to alice's canonical id. +/// 4. Verify `search_entities("alice")` resolves to alice's canonical id. +/// +/// (The global-digest and topic-spawn steps were removed with those trees.) #[tokio::test] async fn full_pipeline_ingest_to_retrieval() { let (_tmp, cfg) = test_config(); @@ -187,147 +166,7 @@ async fn full_pipeline_ingest_to_retrieval() { "query_source total must be >= hits.len()" ); - // ── Step 4: global digest ───────────────────────────────────────── - // The digest picks up source summaries from sealed trees and folds - // them into one cross-source daily node. - let today = Utc::now().date_naive(); - let digest_outcome = end_of_day_digest(&cfg, today) - .await - .expect("end_of_day_digest must succeed"); - - log::debug!( - "[tree_e2e_test] digest outcome: {:?}", - digest_outcome - ); - - // Two possible happy outcomes: a fresh emission, or a skip if a - // prior test in the same workspace already ran today's digest. - // EmptyDay is acceptable when the seal budget wasn't crossed (e.g. - // small CI runners where chunker produces fewer tokens). - match &digest_outcome { - DigestOutcome::Emitted { - source_count, - daily_id, - sealed_ids, - } => { - log::debug!( - "[tree_e2e_test] digest emitted: daily_id={daily_id} source_count={source_count} cascade_seals={}", - sealed_ids.len() - ); - assert!(*source_count >= 1, "emitted digest must reference at least one source"); - } - DigestOutcome::Skipped { existing_id } => { - log::debug!("[tree_e2e_test] digest skipped (already exists): {existing_id}"); - } - DigestOutcome::EmptyDay => { - log::debug!( - "[tree_e2e_test] digest returned EmptyDay — \ - source trees may not have sealed yet (insufficient tokens). \ - Continuing; global retrieval will return empty." - ); - } - } - - // ── Step 5: verify global retrieval ────────────────────────────── - let global_resp = query_global(&cfg, 7) - .await - .expect("query_global must succeed"); - - log::debug!( - "[tree_e2e_test] query_global: total={} hits={}", - global_resp.total, - global_resp.hits.len() - ); - - // When a digest was emitted the global tree has at least one node. - // When the day was empty (no seals) we accept an empty response. - if matches!(digest_outcome, DigestOutcome::Emitted { .. }) { - assert!( - global_resp.total >= 1, - "global tree should have at least one node after a successful digest" - ); - } - // Structural invariant always holds. - assert!( - global_resp.total >= global_resp.hits.len(), - "query_global total must be >= hits.len()" - ); - - // ── Step 6: spawn topic tree ────────────────────────────────────── - // Pre-seed hotness counters above the creation threshold so - // `force_recompute` materialises the topic tree immediately - // rather than waiting for organic ingest cycles. - seed_hotness(&cfg, "email:alice@example.com", 2); - - let spawn_outcome = force_recompute(&cfg, "email:alice@example.com") - .await - .expect("force_recompute must succeed"); - - log::debug!( - "[tree_e2e_test] topic spawn outcome: {:?}", - spawn_outcome - ); - - // Topic must have spawned or already existed from a re-run. - match &spawn_outcome { - SpawnOutcome::Spawned { - hotness, - tree_id, - backfilled, - } => { - log::debug!( - "[tree_e2e_test] topic spawned: tree_id={tree_id} hotness={hotness:.3} backfilled={backfilled}" - ); - assert!(tree_id.starts_with("topic:"), "topic tree id must carry the 'topic:' prefix"); - } - SpawnOutcome::TreeExists { hotness, tree_id } => { - log::debug!( - "[tree_e2e_test] topic tree already exists: tree_id={tree_id} hotness={hotness:.3}" - ); - } - SpawnOutcome::BelowThreshold { hotness } => { - // With seeded counters (mention_count_30d=500, 2 distinct - // sources, query_hits=5) the hotness formula must fire. - // A BelowThreshold here is a test bug — panic with details. - panic!( - "[tree_e2e_test] expected Spawned/TreeExists after hotness seeding, \ - got BelowThreshold (hotness={hotness:.3}). Check seed_hotness values \ - against TreePolicy::topic_creation_threshold()." - ); - } - SpawnOutcome::CountersBumped => { - panic!( - "[tree_e2e_test] force_recompute should never return CountersBumped — \ - it bypasses the cadence gate." - ); - } - } - - // ── Step 7: verify topic retrieval ─────────────────────────────── - let topic_resp = query_topic(&cfg, "email:alice@example.com", None, None, 20) - .await - .expect("query_topic must succeed"); - - log::debug!( - "[tree_e2e_test] query_topic: total={} hits={}", - topic_resp.total, - topic_resp.hits.len() - ); - - // After ingesting messages that mention alice@example.com and - // running extract jobs, the entity index must have at least one - // association. - assert!( - topic_resp.total >= 1, - "query_topic for alice@example.com must return at least one hit \ - after ingest + queue drain" - ); - assert!( - topic_resp.total >= topic_resp.hits.len(), - "query_topic total must be >= hits.len()" - ); - - // ── Step 8: search_entities cross-check ────────────────────────── + // ── Step 4: search_entities cross-check ────────────────────────── let entity_matches = search_entities(&cfg, "alice", None, 10) .await .expect("search_entities must succeed"); @@ -356,10 +195,8 @@ async fn full_pipeline_ingest_to_retrieval() { log::info!( "[tree_e2e_test] full_pipeline_ingest_to_retrieval PASSED \ - source_hits={} global_hits={} topic_hits={} entity_matches={}", + source_hits={} entity_matches={}", source_resp.hits.len(), - global_resp.hits.len(), - topic_resp.hits.len(), entity_matches.len() ); }) @@ -466,17 +303,6 @@ async fn pipeline_works_with_embeddings_disabled() { "query_source total must be >= hits.len()" ); - // ── Global digest should also succeed ─────────────────────────── - let today = Utc::now().date_naive(); - let digest_outcome = end_of_day_digest(&cfg, today) - .await - .expect("end_of_day_digest must succeed with embeddings disabled"); - - log::debug!( - "[tree_e2e_test::embeddings_disabled] digest outcome: {:?}", - digest_outcome - ); - // ── Entity search (keyword-based, no embeddings needed) ───────── let entity_matches = search_entities(&cfg, "bob", None, 10) .await diff --git a/src/openhuman/memory/tree_global/digest.rs b/src/openhuman/memory/tree_global/digest.rs deleted file mode 100644 index de2fe5731..000000000 --- a/src/openhuman/memory/tree_global/digest.rs +++ /dev/null @@ -1,673 +0,0 @@ -//! End-of-day digest builder for the global activity tree (#709 Phase 3b). -//! -//! Once per calendar day we walk every active source tree, collect the -//! summary material that covers that day, fold it into one cross-source -//! recap, and persist it as an L0 node in the singleton global tree. A -//! cascade then checks whether enough daily nodes have accumulated to seal -//! the weekly/monthly/yearly levels. -//! -//! Design: -//! - Populated day → exactly one L0 (daily) node emitted + cascade. -//! - Empty day (no source tree touched today) → no-op, logs the skip. -//! - The digest picks the best "representative" input from each source -//! tree in priority order: (a) the latest L1+ summary whose time range -//! intersects the target day, else (b) the most recent chunk that day's -//! L0 buffer still holds, else (c) skip that tree. This keeps the digest -//! accurate for both high-volume sources (where material has already -//! sealed into an L1) and low-volume sources (where the day's activity -//! is still in the L0 buffer). -//! - Idempotency: if an L0 daily node already exists for the target day, -//! return `DigestOutcome::Skipped` rather than emitting a duplicate. - -use std::collections::BTreeSet; - -use anyhow::{Context, Result}; -use chrono::{DateTime, Duration, NaiveDate, TimeZone, Utc}; -use rusqlite::OptionalExtension; - -use crate::openhuman::config::Config; -use crate::openhuman::memory::tree_global::seal::append_daily_and_cascade; -use crate::openhuman::memory_store::chunks::store::with_connection; -use crate::openhuman::memory_store::content::{ - atomic::stage_summary, paths::slugify_source_id, read as content_read, SummaryComposeInput, - SummaryTreeKind, -}; -use crate::openhuman::memory_store::trees::types::{SummaryNode, Tree, TreeKind}; -use crate::openhuman::memory_tree::score::embed::build_embedder_from_config; -use crate::openhuman::memory_tree::summarise::{summarise, SummaryContext, SummaryInput}; -use crate::openhuman::memory_tree::tree::registry::new_summary_id; -use crate::openhuman::memory_tree::tree::store; - -/// Outcome of a single `end_of_day_digest` call — lets the caller decide -/// whether to log skip details or propagate seal counts to telemetry. -#[derive(Debug, Clone)] -pub enum DigestOutcome { - /// Emitted one L0 daily node covering `date`, and possibly cascaded - /// into higher-level seals. `sealed_ids` lists any L1/L2/L3 nodes that - /// sealed during the cascade (empty when the weekly threshold wasn't - /// crossed). - Emitted { - daily_id: String, - source_count: usize, - sealed_ids: Vec, - }, - /// No source tree had material to contribute for `date` — nothing was - /// written. - EmptyDay, - /// An L0 node already exists for `date` (e.g. this is a re-run of the - /// same day's digest). Nothing was written. - Skipped { existing_id: String }, -} - -/// Run an end-of-day digest for `day`, appending one L0 node to the global -/// tree and cascade-sealing upward if thresholds are crossed. The -/// `summarise` function is called once to fold the per-source material into -/// a single cross-source recap; on failure it falls back to a deterministic -/// concat-and-truncate so the digest never aborts due to an LLM error. -/// -/// `day` is the calendar date in UTC the digest should cover. Callers that -/// simply want "yesterday" can pass `Utc::now().date_naive() - Duration::days(1)`. -pub async fn end_of_day_digest(config: &Config, day: NaiveDate) -> Result { - let (day_start, day_end) = day_bounds_utc(day)?; - log::info!( - "[tree_global::digest] end_of_day_digest day={} window=[{}, {})", - day, - day_start, - day_end - ); - - let global = crate::openhuman::memory::tree_global::factory().get_or_create(config)?; - - // Idempotency: check for an existing L0 daily node whose time range - // matches this day. - if let Some(existing) = find_existing_daily(config, &global.id, day_start, day_end)? { - log::info!( - "[tree_global::digest] daily already exists for {day} id={} — skipping", - existing.id - ); - return Ok(DigestOutcome::Skipped { - existing_id: existing.id, - }); - } - - // Gather one contribution per active source tree. - let source_trees = store::list_trees_by_kind(config, TreeKind::Source)?; - log::debug!( - "[tree_global::digest] scanning {} source trees", - source_trees.len() - ); - let mut inputs: Vec = Vec::with_capacity(source_trees.len()); - for source_tree in &source_trees { - match pick_source_contribution(config, source_tree, day_start, day_end)? { - Some(inp) => { - log::debug!( - "[tree_global::digest] source={} contributed id={} tokens={}", - source_tree.scope, - inp.id, - inp.token_count - ); - inputs.push(inp); - } - None => { - log::debug!( - "[tree_global::digest] source={} had no material for {day}", - source_tree.scope - ); - } - } - } - - if inputs.is_empty() { - log::info!( - "[tree_global::digest] empty day — no source trees contributed material for {day}" - ); - return Ok(DigestOutcome::EmptyDay); - } - - // Fold cross-source material into one daily recap. - let ctx = SummaryContext { - tree_id: &global.id, - tree_kind: TreeKind::Global, - target_level: 0, // daily node lives at L0 on the global tree - token_budget: crate::openhuman::memory::tree_global::GLOBAL_TOKEN_BUDGET, - }; - let output = match summarise(config, &inputs, &ctx).await { - Ok(o) => o, - Err(e) => { - log::warn!( - "[tree_global::digest] summarise failed for day={day}: {e:#} — using fallback" - ); - crate::openhuman::memory_tree::summarise::fallback_summary(&inputs, ctx.token_budget) - } - }; - - // Envelope: time range is the day's bounds, score carries the max - // contribution score so recall still has a ranking signal. - let score = inputs - .iter() - .map(|i| i.score) - .fold(f32::NEG_INFINITY, f32::max) - .max(0.0); - - // Phase 4 (#710): embed before opening the write tx so an embedder - // error aborts the digest without leaving a half-committed row. - let embedder = - build_embedder_from_config(config).context("build embedder during end_of_day_digest")?; - let embedding = embedder - .embed(&output.content) - .await - .context("embed daily summary during end_of_day_digest")?; - - // L0 daily node inherits entities/topics by union of contributing - // source-tree summaries. Each input was already labeled at source-tree - // seal time, so emergent themes don't need another extractor pass - // here — global is a sink; union preserves "days that mentioned X" - // retrieval without an extra LLM call. See LabelStrategy in - // tree::bucket_seal for the full design. - let mut entities_set: BTreeSet = BTreeSet::new(); - let mut topics_set: BTreeSet = BTreeSet::new(); - for inp in &inputs { - for e in &inp.entities { - entities_set.insert(e.clone()); - } - for t in &inp.topics { - topics_set.insert(t.clone()); - } - } - let daily_entities: Vec = entities_set.into_iter().collect(); - let daily_topics: Vec = topics_set.into_iter().collect(); - - let now = Utc::now(); - let daily_id = new_summary_id(0); - let daily = SummaryNode { - id: daily_id.clone(), - tree_id: global.id.clone(), - tree_kind: TreeKind::Global, - level: 0, - parent_id: None, - child_ids: inputs.iter().map(|i| i.id.clone()).collect(), - content: output.content, - token_count: output.token_count, - entities: daily_entities, - topics: daily_topics, - time_range_start: day_start, - time_range_end: day_end, - score, - sealed_at: now, - deleted: false, - embedding: Some(embedding), - }; - - // Phase MD-content: stage the L0 daily .md file before the write tx. - // `date_for_global` = day_start (the calendar day this digest covers). - let daily_compose_input = SummaryComposeInput { - summary_id: &daily.id, - tree_kind: SummaryTreeKind::Global, - tree_id: &daily.tree_id, - tree_scope: &global.scope, - level: daily.level, - child_ids: &daily.child_ids, - child_basenames: None, - child_count: daily.child_ids.len(), - time_range_start: daily.time_range_start, - time_range_end: daily.time_range_end, - sealed_at: daily.sealed_at, - body: &daily.content, - }; - // Stage the summary .md file — abort the digest on failure so the database - // never commits a row with content_path = NULL. The digest job is retried - // via the normal job-retry path. - let content_root_daily = config.memory_tree_content_root(); - let global_scope_slug = slugify_source_id(&global.scope); - let staged_daily = stage_summary( - &content_root_daily, - &daily_compose_input, - &global_scope_slug, - Some(day_start), - ) - .with_context(|| { - format!( - "stage_summary failed for daily {}; digest aborted for retry", - daily.id - ) - })?; - log::debug!( - "[tree_global::digest] staged daily summary {} → {}", - daily.id, - staged_daily.content_path - ); - - // Persist the daily node. Note: we do NOT backlink parent_id on the - // child summaries here — their parents are their own source trees, not - // the global tree. The global-tree child_ids are cross-source - // *references*, not ownership. - let daily_clone = daily.clone(); - let tree_id_clone = global.id.clone(); - with_connection(config, move |conn| { - let tx = conn.unchecked_transaction()?; - store::insert_summary_tx( - &tx, - &daily_clone, - Some(&staged_daily), - &crate::openhuman::memory_store::chunks::store::tree_active_signature(config), - )?; - // Index any entities the summariser emitted (no-op under inert). - crate::openhuman::memory_tree::score::store::index_summary_entity_ids_tx( - &tx, - &daily_clone.entities, - &daily_clone.id, - daily_clone.score, - now.timestamp_millis(), - Some(&tree_id_clone), - )?; - tx.commit()?; - Ok(()) - })?; - - log::info!( - "[tree_global::digest] emitted daily id={} sources={} tokens={}", - daily.id, - inputs.len(), - daily.token_count - ); - - // Append into L0 buffer + cascade-seal if thresholds crossed. - let sealed_ids = append_daily_and_cascade(config, &global, &daily).await?; - - Ok(DigestOutcome::Emitted { - daily_id: daily.id, - source_count: inputs.len(), - sealed_ids, - }) -} - -/// Compute [00:00, 24:00) UTC bounds for a calendar day. -fn day_bounds_utc(day: NaiveDate) -> Result<(DateTime, DateTime)> { - let start_naive = day - .and_hms_opt(0, 0, 0) - .ok_or_else(|| anyhow::anyhow!("invalid day {day} — failed to build 00:00 timestamp"))?; - let start = Utc - .from_local_datetime(&start_naive) - .single() - .ok_or_else(|| anyhow::anyhow!("non-unique UTC time for day {day}"))?; - Ok((start, start + Duration::days(1))) -} - -/// Look for an already-emitted L0 daily node for this day. Matches on -/// `tree_kind='global' AND level=0 AND time_range_start=day_start AND deleted=0`. -fn find_existing_daily( - config: &Config, - global_tree_id: &str, - day_start: DateTime, - _day_end: DateTime, -) -> Result> { - let start_ms = day_start.timestamp_millis(); - let opt_id: Option = with_connection(config, |conn| { - let id: Option = conn - .query_row( - "SELECT id FROM mem_tree_summaries - WHERE tree_id = ?1 - AND level = 0 - AND time_range_start_ms = ?2 - AND deleted = 0 - LIMIT 1", - rusqlite::params![global_tree_id, start_ms], - |r| r.get::<_, String>(0), - ) - .optional() - .context("query for existing daily node")?; - Ok(id) - })?; - match opt_id { - Some(id) => store::get_summary(config, &id), - None => Ok(None), - } -} - -/// Pick the single best contribution from one source tree for the target -/// day. Priority: -/// 1. The latest L1+ summary whose time range intersects the day. -/// 2. The tree's current root summary (any level), as a fallback when no -/// summary intersects the exact day window. -/// -/// Returns `None` when the tree has no sealed summaries at all — a -/// brand-new tree whose L0 buffer has not yet crossed the token budget. -/// Phase 3b intentionally skips such trees rather than plumbing the raw -/// L0 buffer into the digest; low-volume sources become visible once -/// either the token or time-based flush lands them in a summary. -fn pick_source_contribution( - config: &Config, - source_tree: &Tree, - day_start: DateTime, - day_end: DateTime, -) -> Result> { - let start_ms = day_start.timestamp_millis(); - let end_ms = day_end.timestamp_millis(); - let intersecting_id: Option = with_connection(config, |conn| { - let mut stmt = conn.prepare( - "SELECT id FROM mem_tree_summaries - WHERE tree_id = ?1 - AND deleted = 0 - AND time_range_start_ms < ?3 - AND time_range_end_ms >= ?2 - ORDER BY level DESC, sealed_at_ms DESC - LIMIT 1", - )?; - let row = stmt - .query_row(rusqlite::params![&source_tree.id, start_ms, end_ms], |r| { - r.get::<_, String>(0) - }) - .optional() - .context("query intersecting source summary")?; - Ok(row) - })?; - - let chosen_id = match intersecting_id { - Some(id) => Some(id), - None => source_tree.root_id.clone(), - }; - - let Some(id) = chosen_id else { - return Ok(None); - }; - - let node = match store::get_summary(config, &id)? { - Some(n) => n, - None => { - log::warn!( - "[tree_global::digest] picked id={id} for tree={} but row missing — skipping", - source_tree.scope - ); - return Ok(None); - } - }; - - // Read the full body from disk — `node.content` is a ≤500-char preview - // after the MD-on-disk migration. The digest summariser must receive the - // complete summary text so the daily recap is not assembled from previews. - let body = match content_read::read_summary_body(config, &node.id) { - Ok(b) => b, - Err(e) => { - log::warn!( - "[tree_global::digest] read_summary_body failed for {} — using preview: {e:#}", - node.id - ); - // Non-fatal: fall back to preview for pre-MD-migration rows. - node.content.clone() - } - }; - Ok(Some(SummaryInput { - id: node.id, - content: format!("[{}]\n{}", source_tree.scope, body), - token_count: node.token_count, - entities: node.entities, - topics: node.topics, - time_range_start: node.time_range_start, - time_range_end: node.time_range_end, - score: node.score, - })) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - use crate::openhuman::memory::tree_source::registry::get_or_create_source_tree; - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_store::content as content_store; - use crate::openhuman::memory_tree::tree::bucket_seal::{append_leaf, LabelStrategy, LeafRef}; - use std::sync::Arc; - use tempfile::TempDir; - - // --------------------------------------------------------------------------- - // Helpers - // --------------------------------------------------------------------------- - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - cfg.memory_tree.embedding_endpoint = None; - cfg.memory_tree.embedding_model = None; - cfg.memory_tree.embedding_strict = false; - (tmp, cfg) - } - - /// Stage chunk content files on disk so `read_summary_body` can find them - /// during `pick_source_contribution`. - fn stage_test_chunks(cfg: &Config, chunks: &[Chunk]) { - let content_root = cfg.memory_tree_content_root(); - std::fs::create_dir_all(&content_root).expect("create content_root for test"); - let staged = content_store::stage_chunks(&content_root, chunks) - .expect("stage_chunks for test chunks"); - crate::openhuman::memory_store::chunks::store::with_connection(cfg, |conn| { - let tx = conn.unchecked_transaction()?; - crate::openhuman::memory_store::chunks::store::upsert_staged_chunks_tx(&tx, &staged)?; - tx.commit()?; - Ok(()) - }) - .expect("persist staged chunk pointers"); - } - - /// Create a source tree for `scope`, upsert two 30 k-token chunks so that - /// `append_leaf` triggers an L0→L1 seal (the 50 k-token threshold), and - /// stage the chunk content files on disk. The resulting source tree has - /// at least one sealed summary that `pick_source_contribution` can return. - async fn seed_source_l1(cfg: &Config, scope: &str, ts: DateTime) { - let tree = get_or_create_source_tree(cfg, scope).unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - - let c1 = Chunk { - id: chunk_id(SourceKind::Chat, scope, 0, "test-content"), - content: format!("c1-{scope}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: scope.into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new("slack://x")), - }, - token_count: 30_000, - seq_in_source: 0, - created_at: ts, - partial_message: false, - }; - let c2 = Chunk { - id: chunk_id(SourceKind::Chat, scope, 1, "test-content"), - content: format!("c2-{scope}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: scope.into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new("slack://y")), - }, - token_count: 30_000, - seq_in_source: 1, - created_at: ts, - partial_message: false, - }; - - upsert_chunks(cfg, &[c1.clone(), c2.clone()]).unwrap(); - stage_test_chunks(cfg, &[c1.clone(), c2.clone()]); - - let leaf1 = LeafRef { - chunk_id: c1.id.clone(), - token_count: 30_000, - timestamp: ts, - content: c1.content.clone(), - entities: vec![], - topics: vec![], - score: 0.5, - }; - let leaf2 = LeafRef { - chunk_id: c2.id.clone(), - token_count: 30_000, - timestamp: ts, - content: c2.content.clone(), - entities: vec![], - topics: vec![], - score: 0.5, - }; - - test_override::with_provider(Arc::clone(&provider), async { - append_leaf(cfg, &tree, &leaf1, &LabelStrategy::Empty) - .await - .unwrap(); - append_leaf(cfg, &tree, &leaf2, &LabelStrategy::Empty) - .await - .unwrap(); - }) - .await; - } - - // --------------------------------------------------------------------------- - // Tests - // --------------------------------------------------------------------------- - - /// When there are no source trees at all, `end_of_day_digest` must return - /// `EmptyDay` without writing any rows. - #[tokio::test] - async fn empty_day_with_no_source_trees() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let day = Utc::now().date_naive(); - - let outcome = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - - assert!( - matches!(outcome, DigestOutcome::EmptyDay), - "expected EmptyDay when no source trees exist, got {outcome:?}" - ); - } - - /// A source tree exists but has no sealed summaries covering the target - /// day. A freshly-created tree whose L0 buffer has never crossed the - /// token threshold has `root_id = None`, so `pick_source_contribution` - /// returns `None` and the digest should be `EmptyDay`. - #[tokio::test] - async fn empty_day_no_contributions() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - - // Create the source tree but add no leaves — root_id stays None. - get_or_create_source_tree(&cfg, "slack:#empty").unwrap(); - - let day = Utc::now().date_naive(); - let outcome = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - - assert!( - matches!(outcome, DigestOutcome::EmptyDay), - "expected EmptyDay when source tree has no sealed summaries, got {outcome:?}" - ); - } - - /// One source tree with sealed L1 material covering today should yield - /// `DigestOutcome::Emitted` with `source_count == 1`. - #[tokio::test] - async fn emits_daily_node_from_single_source() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let day = Utc::now().date_naive(); - // Use a timestamp inside today so the intersecting-summary query hits. - let ts = day.and_hms_opt(12, 0, 0).unwrap().and_utc(); - - seed_source_l1(&cfg, "slack:#general", ts).await; - - let outcome = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - - match outcome { - DigestOutcome::Emitted { - source_count, - daily_id, - .. - } => { - assert_eq!(source_count, 1, "expected exactly one contributing source"); - assert!(!daily_id.is_empty(), "daily_id must be non-empty"); - } - other => panic!("expected Emitted, got {other:?}"), - } - } - - /// Calling `end_of_day_digest` twice for the same calendar day must return - /// `Skipped` on the second call and must NOT insert a second L0 node. - #[tokio::test] - async fn idempotent_skip_on_rerun() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let day = Utc::now().date_naive(); - let ts = day.and_hms_opt(9, 0, 0).unwrap().and_utc(); - - seed_source_l1(&cfg, "slack:#idempotent", ts).await; - - // First call — should emit. - let first = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - assert!( - matches!(first, DigestOutcome::Emitted { .. }), - "first call must emit, got {first:?}" - ); - - // Second call — must skip. - let second = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - assert!( - matches!(second, DigestOutcome::Skipped { .. }), - "second call must be Skipped, got {second:?}" - ); - } - - /// Two source trees with sealed L1 material covering today → `Emitted` - /// with `source_count == 2`. - #[tokio::test] - async fn multiple_sources_contribute() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let day = Utc::now().date_naive(); - let ts = day.and_hms_opt(11, 0, 0).unwrap().and_utc(); - - seed_source_l1(&cfg, "slack:#alpha", ts).await; - seed_source_l1(&cfg, "slack:#beta", ts).await; - - let outcome = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - - match outcome { - DigestOutcome::Emitted { source_count, .. } => { - assert_eq!( - source_count, 2, - "both source trees must contribute; got source_count={source_count}" - ); - } - other => panic!("expected Emitted, got {other:?}"), - } - } -} diff --git a/src/openhuman/memory/tree_global/mod.rs b/src/openhuman/memory/tree_global/mod.rs deleted file mode 100644 index fb08455a9..000000000 --- a/src/openhuman/memory/tree_global/mod.rs +++ /dev/null @@ -1,31 +0,0 @@ -//! Global tree instance — policy and orchestration for the singleton -//! cross-source digest tree. -//! -//! The generic tree engine lives in [`memory_tree`]; this module owns -//! the global-specific algorithms: end-of-day digest, window-scoped -//! recap, and count-based cascade-seal thresholds. - -pub mod digest; -pub mod recap; -pub mod seal; - -pub use crate::openhuman::memory_store::trees::get_or_create_global_tree; -pub use crate::openhuman::memory_store::trees::registry; -pub use crate::openhuman::memory_tree::tree::factory::GLOBAL_SCOPE; -use crate::openhuman::memory_tree::tree::TreeFactory; -pub use digest::{end_of_day_digest, DigestOutcome}; -pub use recap::{recap, RecapOutput}; - -/// Number of L0 (daily) nodes that seal into one L1 (weekly) node. -pub const WEEKLY_SEAL_THRESHOLD: usize = 7; -/// Number of L1 (weekly) nodes that seal into one L2 (monthly) node. -pub const MONTHLY_SEAL_THRESHOLD: usize = 4; -/// Number of L2 (monthly) nodes that seal into one L3 (yearly) node. -pub const YEARLY_SEAL_THRESHOLD: usize = 12; -/// Token budget passed into the summariser for global-tree seals. -pub const GLOBAL_TOKEN_BUDGET: u32 = 4_000; - -/// Canonical factory for the singleton global tree. -pub fn factory() -> TreeFactory<'static> { - TreeFactory::global() -} diff --git a/src/openhuman/memory/tree_global/recap.rs b/src/openhuman/memory/tree_global/recap.rs deleted file mode 100644 index 1e1b6dbb0..000000000 --- a/src/openhuman/memory/tree_global/recap.rs +++ /dev/null @@ -1,368 +0,0 @@ -//! Window-scoped recap retrieval for the global activity tree (#709 Phase 3b). -//! -//! Given a duration (e.g. `Duration::days(7)`), pick the tree level that -//! naturally matches the time axis and return the latest summary at that -//! level. This is the read half of the global digest: the digest builder -//! plants daily/weekly/monthly/yearly nodes, and `recap` retrieves the one -//! best suited for the caller's question. -//! -//! Level selection (width thresholds chosen to cover expected call sites): -//! - `< 2 days` → latest L0 (today's digest) -//! - `< 14 days` → latest L1 (weekly) -//! - `< 60 days` → latest L2 (monthly) -//! - `≥ 60 days` → latest L3 (yearly), padded with the covering L2s when no L3 has sealed yet. -//! -//! When no summary exists at the chosen level, the function falls back -//! downward (to the latest lower-level node) and reports the actual level -//! used in the `level_used` field of the result so callers can surface -//! "best available" to users. Returns `None` only when the global tree has -//! no sealed summaries at all. - -use anyhow::Result; -use chrono::{DateTime, Duration, Utc}; - -use crate::openhuman::config::Config; -use crate::openhuman::memory_store::trees::types::SummaryNode; -use crate::openhuman::memory_tree::tree::store; - -/// Aggregated recap returned to the caller. -#[derive(Debug, Clone)] -pub struct RecapOutput { - /// The rolled-up content for the chosen window. - pub content: String, - /// The time span actually covered by the returned content. Start is the - /// earliest `time_range_start` across included summaries, end is the - /// latest `time_range_end`. - pub time_range: (DateTime, DateTime), - /// The level actually used to build the recap. May be lower than the - /// requested level when the higher level has no sealed nodes yet. - pub level_used: u32, - /// One entry per summary folded into the content, in the order they - /// were concatenated. Lets callers surface provenance ("this recap - /// covers weekly summaries W, W-1, W-2"). - pub summary_ids: Vec, -} - -/// Return a recap for the given window, or `None` if no global summaries -/// have sealed yet. -pub async fn recap(config: &Config, window: Duration) -> Result> { - let target_level = pick_level(window); - log::info!( - "[tree_global::recap] recap window={:?} target_level={}", - window, - target_level - ); - - let global = crate::openhuman::memory::tree_global::factory().get_or_create(config)?; - let now = Utc::now(); - let window_start = now - window; - - // Walk down from `target_level` to 0 looking for material. - for level in (0..=target_level).rev() { - let all_at_level = store::list_summaries_at_level(config, &global.id, level)?; - if all_at_level.is_empty() { - continue; - } - let covering = pick_covering(&all_at_level, window_start, now); - if covering.is_empty() { - continue; - } - log::debug!( - "[tree_global::recap] using level={} summaries={}", - level, - covering.len() - ); - return Ok(Some(assemble_recap(&covering, level))); - } - - log::info!("[tree_global::recap] no global summaries yet — nothing to recap"); - Ok(None) -} - -/// Select every summary at the given level whose time range overlaps the -/// [window_start, now] window, ordered oldest → newest. When none overlap -/// (a long quiet stretch ending before the window) we fall back to the -/// single latest summary so callers still get *something* useful. -fn pick_covering( - summaries: &[SummaryNode], - window_start: DateTime, - now: DateTime, -) -> Vec<&SummaryNode> { - let mut overlapping: Vec<&SummaryNode> = summaries - .iter() - .filter(|s| s.time_range_end >= window_start && s.time_range_start <= now) - .collect(); - overlapping.sort_by_key(|s| s.time_range_start); - - if overlapping.is_empty() { - if let Some(latest) = summaries.iter().max_by_key(|s| s.sealed_at) { - return vec![latest]; - } - } - overlapping -} - -/// Concatenate the selected summaries with provenance markers and compute -/// the time envelope. -fn assemble_recap(covering: &[&SummaryNode], level: u32) -> RecapOutput { - let mut parts: Vec = Vec::with_capacity(covering.len()); - let mut summary_ids: Vec = Vec::with_capacity(covering.len()); - for s in covering { - parts.push(format!( - "[{} → {}]\n{}", - s.time_range_start.to_rfc3339(), - s.time_range_end.to_rfc3339(), - s.content - )); - summary_ids.push(s.id.clone()); - } - let content = parts.join("\n\n"); - - let time_start = covering - .iter() - .map(|s| s.time_range_start) - .min() - .unwrap_or_else(Utc::now); - let time_end = covering - .iter() - .map(|s| s.time_range_end) - .max() - .unwrap_or_else(Utc::now); - - RecapOutput { - content, - time_range: (time_start, time_end), - level_used: level, - summary_ids, - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - use crate::openhuman::memory::tree_global::digest::{end_of_day_digest, DigestOutcome}; - use crate::openhuman::memory::tree_source::registry::get_or_create_source_tree; - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_store::content as content_store; - use crate::openhuman::memory_tree::tree::bucket_seal::{append_leaf, LabelStrategy, LeafRef}; - use std::sync::Arc; - use tempfile::TempDir; - - fn stage_test_chunks(cfg: &Config, chunks: &[Chunk]) { - let content_root = cfg.memory_tree_content_root(); - std::fs::create_dir_all(&content_root).expect("create content_root for test"); - let staged = content_store::stage_chunks(&content_root, chunks) - .expect("stage_chunks for test chunks"); - crate::openhuman::memory_store::chunks::store::with_connection(cfg, |conn| { - let tx = conn.unchecked_transaction()?; - crate::openhuman::memory_store::chunks::store::upsert_staged_chunks_tx(&tx, &staged)?; - tx.commit()?; - Ok(()) - }) - .expect("persist staged chunk pointers"); - } - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - // Phase 4 (#710): recap exercises the digest path which embeds. - cfg.memory_tree.embedding_endpoint = None; - cfg.memory_tree.embedding_model = None; - cfg.memory_tree.embedding_strict = false; - (tmp, cfg) - } - - #[test] - fn pick_level_matches_window_thresholds() { - assert_eq!(pick_level(Duration::hours(1)), 0); - assert_eq!(pick_level(Duration::days(1)), 0); - assert_eq!(pick_level(Duration::days(2)), 1); - assert_eq!(pick_level(Duration::days(7)), 1); - assert_eq!(pick_level(Duration::days(13)), 1); - assert_eq!(pick_level(Duration::days(14)), 2); - assert_eq!(pick_level(Duration::days(30)), 2); - assert_eq!(pick_level(Duration::days(59)), 2); - assert_eq!(pick_level(Duration::days(60)), 3); - assert_eq!(pick_level(Duration::days(365)), 3); - } - - #[tokio::test] - async fn recap_on_empty_tree_returns_none() { - let (_tmp, cfg) = test_config(); - let out = recap(&cfg, Duration::days(7)).await.unwrap(); - assert!(out.is_none()); - } - - async fn seed_source_l1(cfg: &Config, scope: &str, ts: DateTime) { - let tree = get_or_create_source_tree(cfg, scope).unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let c1 = Chunk { - id: chunk_id(SourceKind::Chat, scope, 0, "test-content"), - content: format!("c1-{scope}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: scope.into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new("slack://x")), - }, - token_count: 30_000, - seq_in_source: 0, - created_at: ts, - partial_message: false, - }; - let c2 = Chunk { - id: chunk_id(SourceKind::Chat, scope, 1, "test-content"), - content: format!("c2-{scope}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: scope.into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new("slack://y")), - }, - token_count: 30_000, - seq_in_source: 1, - created_at: ts, - partial_message: false, - }; - upsert_chunks(cfg, &[c1.clone(), c2.clone()]).unwrap(); - stage_test_chunks(cfg, &[c1.clone(), c2.clone()]); - let leaf1 = LeafRef { - chunk_id: c1.id.clone(), - token_count: 30_000, - timestamp: ts, - content: c1.content.clone(), - entities: vec![], - topics: vec![], - score: 0.5, - }; - let leaf2 = LeafRef { - chunk_id: c2.id.clone(), - token_count: 30_000, - timestamp: ts, - content: c2.content.clone(), - entities: vec![], - topics: vec![], - score: 0.5, - }; - test_override::with_provider(Arc::clone(&provider), async { - append_leaf(cfg, &tree, &leaf1, &LabelStrategy::Empty) - .await - .unwrap(); - append_leaf(cfg, &tree, &leaf2, &LabelStrategy::Empty) - .await - .unwrap(); - }) - .await; - } - - #[tokio::test] - async fn recap_one_day_window_returns_latest_l0() { - // One daily digest → recap(1 day) should return the L0 at the - // correct level. - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - // Use "today" so the digest's time range covers now. - let day = Utc::now().date_naive(); - let ts = day.and_hms_opt(12, 0, 0).unwrap().and_utc(); - seed_source_l1(&cfg, "slack:#eng", ts).await; - let outcome = test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - assert!(matches!(outcome, DigestOutcome::Emitted { .. })); - - let r = recap(&cfg, Duration::hours(24)) - .await - .unwrap() - .expect("expected a recap with one daily node emitted"); - assert_eq!(r.level_used, 0); - assert_eq!(r.summary_ids.len(), 1); - assert!(!r.content.is_empty()); - } - - #[tokio::test] - async fn recap_weekly_window_falls_back_to_l0_when_no_l1() { - // With only 3 daily nodes (< 7) no L1 has sealed. A 7-day recap - // should fall back from level 1 to level 0 and return whatever - // daily nodes exist. - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let today = Utc::now().date_naive(); - let base = today - Duration::days(2); - for i in 0..3 { - let day = base + Duration::days(i); - let ts = day.and_hms_opt(10, 0, 0).unwrap().and_utc(); - seed_source_l1(&cfg, &format!("slack:#d{i}"), ts).await; - test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - } - let r = recap(&cfg, Duration::days(7)) - .await - .unwrap() - .expect("expected fallback recap"); - assert_eq!( - r.level_used, 0, - "no L1 available yet → recap falls back to L0" - ); - assert_eq!(r.summary_ids.len(), 3, "all three daily nodes folded in"); - } - - #[tokio::test] - async fn recap_weekly_window_uses_l1_when_sealed() { - // After 7 daily digests a weekly L1 exists. A 7-day recap should - // return that L1 at level 1. - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let today = Utc::now().date_naive(); - let base = today - Duration::days(6); - for i in 0..7 { - let day = base + Duration::days(i); - let ts = day.and_hms_opt(10, 0, 0).unwrap().and_utc(); - seed_source_l1(&cfg, &format!("slack:#w{i}"), ts).await; - test_override::with_provider(Arc::clone(&provider), async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - } - let r = recap(&cfg, Duration::days(7)) - .await - .unwrap() - .expect("expected recap with weekly seal"); - assert_eq!(r.level_used, 1); - assert_eq!( - r.summary_ids.len(), - 1, - "one weekly L1 node covers the window" - ); - } -} -/// Map a window duration to the level whose node-granularity best matches -/// the window. See module-level doc for the thresholds. -pub fn pick_level(window: Duration) -> u32 { - if window < Duration::days(2) { - 0 - } else if window < Duration::days(14) { - 1 - } else if window < Duration::days(60) { - 2 - } else { - 3 - } -} diff --git a/src/openhuman/memory/tree_global/seal.rs b/src/openhuman/memory/tree_global/seal.rs deleted file mode 100644 index ed08bf9f6..000000000 --- a/src/openhuman/memory/tree_global/seal.rs +++ /dev/null @@ -1,567 +0,0 @@ -//! Count-based cascade-seal for the global activity digest tree (#709 Phase 3b). -//! -//! The global tree's trigger is **time/count-based**, not token-based: seal -//! L0 → L1 when 7 daily nodes accumulate, L1 → L2 when 4 weekly nodes -//! accumulate, L2 → L3 when 12 monthly nodes accumulate. This keeps the -//! tree aligned to the time axis (day / week / month / year) so -//! window-scoped recap queries can map a duration to a level deterministically. -//! -//! Reuses Phase 3a storage primitives from `tree::store` without -//! their token-budget cascade logic — all global seals route through -//! `mem_tree_summaries` on both sides (children and output), since even L0 -//! is a sealed summary node rather than a raw chunk. - -use std::collections::BTreeSet; - -use anyhow::{Context, Result}; -use chrono::{DateTime, Utc}; - -use crate::openhuman::config::Config; -use crate::openhuman::memory::tree_global::{ - GLOBAL_TOKEN_BUDGET, MONTHLY_SEAL_THRESHOLD, WEEKLY_SEAL_THRESHOLD, YEARLY_SEAL_THRESHOLD, -}; -use crate::openhuman::memory_store::chunks::store::with_connection; -use crate::openhuman::memory_store::content::{ - atomic::stage_summary, SummaryComposeInput, SummaryTreeKind, -}; -use crate::openhuman::memory_store::trees::types::{Buffer, SummaryNode, Tree, TreeKind}; -use crate::openhuman::memory_tree::score::embed::build_embedder_from_config; -use crate::openhuman::memory_tree::summarise::{ - fallback_summary, summarise, SummaryContext, SummaryInput, -}; -use crate::openhuman::memory_tree::tree::registry::new_summary_id; -use crate::openhuman::memory_tree::tree::store; - -/// Hard cap on cascade depth — mirrors the source-tree constant. L0→L1→L2→L3 -/// is only 3 hops so we have ample slack. -const MAX_CASCADE_DEPTH: u32 = 32; - -/// Idempotently append one level-0 (daily) summary id to the global tree's -/// L0 buffer, then cascade-seal upward if count thresholds are crossed. -/// -/// The caller (`digest::end_of_day_digest`) has already inserted the L0 -/// node into `mem_tree_summaries`; this function only handles the buffer -/// accounting + cascade. -pub async fn append_daily_and_cascade( - config: &Config, - tree: &Tree, - daily_summary: &SummaryNode, -) -> Result> { - log::debug!( - "[tree_global::seal] append_daily tree_id={} daily_id={} tokens={}", - tree.id, - daily_summary.id, - daily_summary.token_count - ); - - append_to_buffer( - config, - &tree.id, - 0, - &daily_summary.id, - daily_summary.token_count as i64, - daily_summary.time_range_start, - )?; - - cascade_seals(config, tree).await -} - -/// Transactionally append a single summary id to the buffer at -/// `(tree_id, level)`. Idempotent on the `(tree_id, level, item_id)` tuple -/// so retries of a partially-applied digest don't double-count. -fn append_to_buffer( - config: &Config, - tree_id: &str, - level: u32, - item_id: &str, - token_delta: i64, - item_ts: DateTime, -) -> Result<()> { - with_connection(config, |conn| { - let tx = conn.unchecked_transaction()?; - let mut buf = store::get_buffer_conn(&tx, tree_id, level)?; - if buf.item_ids.iter().any(|existing| existing == item_id) { - log::debug!( - "[tree_global::seal] append_to_buffer: {item_id} already in buffer \ - tree_id={tree_id} level={level} — no-op" - ); - return Ok(()); - } - buf.item_ids.push(item_id.to_string()); - buf.token_sum = buf.token_sum.saturating_add(token_delta); - buf.oldest_at = match buf.oldest_at { - Some(existing) => Some(existing.min(item_ts)), - None => Some(item_ts), - }; - store::upsert_buffer_tx(&tx, &buf)?; - tx.commit()?; - Ok(()) - }) -} - -async fn cascade_seals(config: &Config, tree: &Tree) -> Result> { - let mut sealed_ids: Vec = Vec::new(); - // `level` is independent of the iteration counter — it only bumps when a - // seal fires, and the loop can break early if `should_seal` returns - // false. Clippy's loop-counter suggestion would merge them incorrectly. - #[allow(clippy::explicit_counter_loop)] - { - let mut level: u32 = 0; - for _ in 0..MAX_CASCADE_DEPTH { - let buf = store::get_buffer(config, &tree.id, level)?; - if !should_seal(&buf, level) { - log::debug!( - "[tree_global::seal] cascade done tree_id={} stop_level={} count={}", - tree.id, - level, - buf.item_ids.len() - ); - break; - } - - let summary_id = seal_one_level(config, tree, &buf).await?; - sealed_ids.push(summary_id); - level += 1; - } - } - - Ok(sealed_ids) -} - -/// Count-based threshold per level. L0→L1 needs 7 daily nodes, L1→L2 needs -/// 4 weekly nodes, L2→L3 needs 12 monthly nodes. Levels ≥ 3 never seal in -/// this phase — a yearly node is the top of the global tree. -fn should_seal(buf: &Buffer, level: u32) -> bool { - let threshold = match level { - 0 => WEEKLY_SEAL_THRESHOLD, - 1 => MONTHLY_SEAL_THRESHOLD, - 2 => YEARLY_SEAL_THRESHOLD, - _ => return false, - }; - !buf.is_empty() && buf.item_ids.len() >= threshold -} - -async fn seal_one_level(config: &Config, tree: &Tree, buf: &Buffer) -> Result { - let level = buf.level; - let target_level = level + 1; - - let inputs = hydrate_summary_inputs(config, &buf.item_ids)?; - if inputs.is_empty() { - anyhow::bail!( - "[tree_global::seal] refused to seal empty buffer tree_id={} level={}", - tree.id, - level - ); - } - - let time_range_start = inputs - .iter() - .map(|i| i.time_range_start) - .min() - .unwrap_or_else(Utc::now); - let time_range_end = inputs - .iter() - .map(|i| i.time_range_end) - .max() - .unwrap_or_else(Utc::now); - let score = inputs - .iter() - .map(|i| i.score) - .fold(f32::NEG_INFINITY, f32::max) - .max(0.0); - - let ctx = SummaryContext { - tree_id: &tree.id, - tree_kind: TreeKind::Global, - target_level, - token_budget: GLOBAL_TOKEN_BUDGET, - }; - let output = match summarise(config, &inputs, &ctx).await { - Ok(o) => o, - Err(e) => { - log::warn!( - "[tree_global::seal] summarise failed tree_id={} level={}: {e:#} — using fallback", - tree.id, - level - ); - fallback_summary(&inputs, ctx.token_budget) - } - }; - - // Global-tree summaries inherit their entity/topic labels via union - // from their already-labeled inputs (source-tree summaries carry - // labels from the source-tree seal extractor; global L1+ inputs - // carry labels from this same union path one level down). We - // deliberately do NOT run an extractor on the daily/weekly/monthly - // synthesis: the inputs already cover what the summary represents, - // and global is a sink — no second-pass labeling earns its keep. - let mut entities_set: BTreeSet = BTreeSet::new(); - let mut topics_set: BTreeSet = BTreeSet::new(); - for inp in &inputs { - for e in &inp.entities { - entities_set.insert(e.clone()); - } - for t in &inp.topics { - topics_set.insert(t.clone()); - } - } - let node_entities: Vec = entities_set.into_iter().collect(); - let node_topics: Vec = topics_set.into_iter().collect(); - - // Phase 4 (#710): embed BEFORE opening the write tx so an embedder - // error aborts the cascade without half-committing the summary. - let embedder = - build_embedder_from_config(config).context("build embedder during global seal")?; - let embedding = embedder.embed(&output.content).await.with_context(|| { - format!( - "embed global summary during seal tree_id={} level={}", - tree.id, level - ) - })?; - - let now = Utc::now(); - let summary_id = new_summary_id(target_level); - let node = SummaryNode { - id: summary_id.clone(), - tree_id: tree.id.clone(), - tree_kind: TreeKind::Global, - level: target_level, - parent_id: None, - child_ids: buf.item_ids.clone(), - content: output.content, - token_count: output.token_count, - entities: node_entities, - topics: node_topics, - time_range_start, - time_range_end, - score, - sealed_at: now, - deleted: false, - embedding: Some(embedding), - }; - - // Phase MD-content: stage the global summary .md file before opening the - // write tx. date_for_global = time_range_start date (daily for L0, or - // the start of the range for higher levels). - let global_date = Some(time_range_start); - let compose_input_global = SummaryComposeInput { - summary_id: &node.id, - tree_kind: SummaryTreeKind::Global, - tree_id: &node.tree_id, - tree_scope: &tree.scope, - level: node.level, - child_ids: &node.child_ids, - child_basenames: None, - child_count: node.child_ids.len(), - time_range_start: node.time_range_start, - time_range_end: node.time_range_end, - sealed_at: node.sealed_at, - body: &node.content, - }; - // Stage the summary .md file — abort the seal on failure so the database - // never commits a row with content_path = NULL. The job-retry path will - // re-attempt the file write on next execution. - let content_root_global = config.memory_tree_content_root(); - // Global tree scope is typically the literal "global" string. - // Use it as-is for the path (slugify passes through short ascii strings unchanged). - let global_scope_slug = - crate::openhuman::memory_store::content::paths::slugify_source_id(&tree.scope); - let staged_global = stage_summary( - &content_root_global, - &compose_input_global, - &global_scope_slug, - global_date, - ) - .with_context(|| { - format!( - "stage_summary failed for {}; global-tree seal aborted for retry", - node.id - ) - })?; - log::debug!( - "[tree_global::seal] staged summary {} → {}", - node.id, - staged_global.content_path - ); - - // Single write transaction: insert the new summary, clear this level's - // buffer, append the new id to the parent buffer, and bump the tree's - // max_level/root_id if we just climbed. Re-read `max_level` inside the - // tx so cascading seals within one call see the bump from earlier - // iterations. - let summary_id_for_closure = summary_id.clone(); - let target_level_for_closure = target_level; - let tree_id = tree.id.clone(); - with_connection(config, move |conn| { - let tx = conn.unchecked_transaction()?; - - let current_max: u32 = tx - .query_row( - "SELECT max_level FROM mem_tree_trees WHERE id = ?1", - rusqlite::params![&tree_id], - |r| r.get::<_, i64>(0), - ) - .map(|n| n.max(0) as u32) - .context("Failed to read current max_level for global tree")?; - - store::insert_summary_tx( - &tx, - &node, - Some(&staged_global), - &crate::openhuman::memory_store::chunks::store::tree_active_signature(config), - )?; - // Index any entities the summariser emitted. No-op under - // InertSummariser (entities stays empty by design — see - // summariser/inert.rs). Becomes active when the Ollama summariser - // lands and emits curated canonical ids. - crate::openhuman::memory_tree::score::store::index_summary_entity_ids_tx( - &tx, - &node.entities, - &node.id, - node.score, - now.timestamp_millis(), - Some(&tree_id), - )?; - // Backlink children → new parent. In the global tree every level is - // already a summary, so the backlink always targets - // `mem_tree_summaries`. - for child_id in &node.child_ids { - tx.execute( - "UPDATE mem_tree_summaries - SET parent_id = ?1 - WHERE id = ?2 AND parent_id IS NULL", - rusqlite::params![&summary_id_for_closure, child_id], - ) - .context("Failed to backlink global summary to parent summary")?; - } - store::clear_buffer_tx(&tx, &tree_id, level)?; - - // Append to parent buffer. - let mut parent = store::get_buffer_conn(&tx, &tree_id, target_level_for_closure)?; - parent.item_ids.push(summary_id_for_closure.clone()); - parent.token_sum = parent.token_sum.saturating_add(node.token_count as i64); - parent.oldest_at = match parent.oldest_at { - Some(existing) => Some(existing.min(time_range_start)), - None => Some(time_range_start), - }; - store::upsert_buffer_tx(&tx, &parent)?; - - // Update tree root / max_level if we just climbed. - if target_level_for_closure > current_max { - store::update_tree_after_seal_tx( - &tx, - &tree_id, - &summary_id_for_closure, - target_level_for_closure, - now, - )?; - } else { - // Same max level — refresh last_sealed_at only. - tx.execute( - "UPDATE mem_tree_trees SET last_sealed_at_ms = ?1 WHERE id = ?2", - rusqlite::params![now.timestamp_millis(), &tree_id], - ) - .context("Failed to refresh last_sealed_at for global tree")?; - } - - tx.commit()?; - Ok(()) - })?; - - log::info!( - "[tree_global::seal] sealed tree_id={} level={}→{} summary_id={} children={}", - tree.id, - level, - target_level, - summary_id, - buf.item_ids.len() - ); - - Ok(summary_id) -} - -/// Hydrate summary rows for the ids in a buffer. Global-tree buffers at -/// every level reference summary nodes (not chunks), so we always pull from -/// `mem_tree_summaries`. -fn hydrate_summary_inputs(config: &Config, summary_ids: &[String]) -> Result> { - let mut out: Vec = Vec::with_capacity(summary_ids.len()); - for id in summary_ids { - let node = match store::get_summary(config, id)? { - Some(n) => n, - None => { - log::warn!( - "[tree_global::seal] hydrate_summary_inputs: missing summary {id} — skipping" - ); - continue; - } - }; - out.push(SummaryInput { - id: node.id.clone(), - content: node.content.clone(), - token_count: node.token_count, - entities: node.entities.clone(), - topics: node.topics.clone(), - time_range_start: node.time_range_start, - time_range_end: node.time_range_end, - score: node.score, - }); - } - Ok(out) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - use crate::openhuman::memory_store::trees::registry::get_or_create_global_tree; - use chrono::TimeZone; - use std::sync::Arc; - use tempfile::TempDir; - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - // Phase 4 (#710): tests exercise the seal cascade which embeds - // output; force the inert path so no Ollama server is required. - cfg.memory_tree.embedding_endpoint = None; - cfg.memory_tree.embedding_model = None; - cfg.memory_tree.embedding_strict = false; - (tmp, cfg) - } - - fn mk_daily(id: &str, tree_id: &str, day_ms: i64) -> SummaryNode { - let ts = Utc.timestamp_millis_opt(day_ms).single().unwrap(); - SummaryNode { - id: id.to_string(), - tree_id: tree_id.to_string(), - tree_kind: TreeKind::Global, - level: 0, - parent_id: None, - child_ids: vec![], // not used by seal hydrator - content: format!("daily digest {id}"), - token_count: 200, - entities: vec![], - topics: vec![], - time_range_start: ts, - time_range_end: ts, - score: 0.5, - sealed_at: ts, - deleted: false, - embedding: None, - } - } - - fn insert_daily(cfg: &Config, node: &SummaryNode) { - with_connection(cfg, |conn| { - let tx = conn.unchecked_transaction()?; - store::insert_summary_tx( - &tx, - node, - None, - &crate::openhuman::memory_store::chunks::store::tree_active_signature(cfg), - )?; - tx.commit()?; - Ok(()) - }) - .unwrap(); - } - - #[tokio::test] - async fn below_threshold_does_not_seal() { - let (_tmp, cfg) = test_config(); - let tree = get_or_create_global_tree(&cfg).unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - - // Append 3 daily nodes — well below the 7-day weekly threshold. - for i in 0..3 { - let node = mk_daily( - &format!("summary:L0:day{i}"), - &tree.id, - 1_700_000_000_000 + i, - ); - insert_daily(&cfg, &node); - let sealed = test_override::with_provider(Arc::clone(&provider), async { - append_daily_and_cascade(&cfg, &tree, &node).await.unwrap() - }) - .await; - assert!(sealed.is_empty(), "no cascade expected below threshold"); - } - - let buf = store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert_eq!(buf.item_ids.len(), 3); - } - - #[tokio::test] - async fn crossing_weekly_threshold_seals_l1() { - let (_tmp, cfg) = test_config(); - let tree = get_or_create_global_tree(&cfg).unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - - // Append exactly 7 daily nodes — should trigger one L0→L1 seal. - for i in 0..WEEKLY_SEAL_THRESHOLD { - let node = mk_daily( - &format!("summary:L0:day{i}"), - &tree.id, - 1_700_000_000_000 + i as i64, - ); - insert_daily(&cfg, &node); - let sealed = test_override::with_provider(Arc::clone(&provider), async { - append_daily_and_cascade(&cfg, &tree, &node).await.unwrap() - }) - .await; - if i + 1 < WEEKLY_SEAL_THRESHOLD { - assert!(sealed.is_empty(), "no seal before threshold (i={i})"); - } else { - assert_eq!(sealed.len(), 1, "expected one weekly seal on 7th append"); - } - } - - // L0 buffer cleared; L1 buffer holds the new weekly summary. - let l0 = store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert!(l0.is_empty()); - let l1 = store::get_buffer(&cfg, &tree.id, 1).unwrap(); - assert_eq!(l1.item_ids.len(), 1); - - // Tree metadata reflects the climb to level 1. - let t = store::get_tree(&cfg, &tree.id).unwrap().unwrap(); - assert_eq!(t.max_level, 1); - assert_eq!(t.root_id.as_deref(), Some(l1.item_ids[0].as_str())); - assert!(t.last_sealed_at.is_some()); - - // Weekly summary row carries children = the 7 daily ids. - let weekly = store::get_summary(&cfg, &l1.item_ids[0]).unwrap().unwrap(); - assert_eq!(weekly.level, 1); - assert_eq!(weekly.tree_kind, TreeKind::Global); - assert_eq!(weekly.child_ids.len(), WEEKLY_SEAL_THRESHOLD); - } - - #[tokio::test] - async fn append_is_idempotent_on_retry() { - let (_tmp, cfg) = test_config(); - let tree = get_or_create_global_tree(&cfg).unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - - let node = mk_daily("summary:L0:dayA", &tree.id, 1_700_000_000_000); - insert_daily(&cfg, &node); - test_override::with_provider(Arc::clone(&provider), async { - append_daily_and_cascade(&cfg, &tree, &node).await.unwrap() - }) - .await; - test_override::with_provider(Arc::clone(&provider), async { - append_daily_and_cascade(&cfg, &tree, &node).await.unwrap() - }) - .await; - - let buf = store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert_eq!( - buf.item_ids.len(), - 1, - "retry must not double-insert the same daily id" - ); - assert_eq!(buf.token_sum, 200); - } -} diff --git a/src/openhuman/memory/tree_topic/backfill.rs b/src/openhuman/memory/tree_topic/backfill.rs deleted file mode 100644 index c588a9123..000000000 --- a/src/openhuman/memory/tree_topic/backfill.rs +++ /dev/null @@ -1,419 +0,0 @@ -//! Topic-tree backfill — hydrate a freshly-materialised topic tree with -//! recent leaves mentioning the entity (#709 Phase 3c). -//! -//! When the curator decides an entity has crossed the hotness threshold -//! for the first time, we create a fresh topic tree AND walk the -//! `mem_tree_entity_index` inverted index to append matching leaves into -//! its L0 buffer. Reusing `bucket_seal::append_leaf` means the cascade -//! fires automatically. -//! -//! ## Why bounded by hotness window -//! -//! Hotness uses a 30-day recency decay (see `tree_topic::hotness`). Leaves -//! older than 30 days contribute zero to current hotness, so by definition -//! they cannot be the reason a tree is spawning *now*. Including them -//! bloats the spawn latency, wastes summariser LLM calls, and amplifies -//! ancient signal that has already decayed away. We cap the backfill -//! window at [`BACKFILL_WINDOW_DAYS`] to align with the hotness math. -//! -//! Older content is still queryable through source-tree retrieval and the -//! entity index — it just doesn't get its own slot in the topic tree. -//! -//! Backfill is intentionally best-effort: missing chunks are skipped with -//! a warn log rather than failing the whole spawn, because Phase 3c is -//! additive — a partial topic tree is still useful. - -use anyhow::{Context, Result}; -use chrono::Utc; - -use crate::openhuman::config::Config; -use crate::openhuman::memory::util::redact::redact; -use crate::openhuman::memory_store::chunks::store::get_chunk; -use crate::openhuman::memory_store::trees::types::Tree; -use crate::openhuman::memory_tree::score::store::lookup_entity; -use crate::openhuman::memory_tree::tree::bucket_seal::{append_leaf, LabelStrategy, LeafRef}; - -/// Max leaves to pull from the entity index during backfill. A hard cap -/// keeps initial spawn latency bounded even for very active entities. -const BACKFILL_LIMIT: usize = 500; - -/// Backfill window in days — matches `tree_topic::hotness::recency_decay`'s -/// hard cliff. Leaves older than this contribute zero to current hotness -/// so they cannot have driven the spawn decision. -pub const BACKFILL_WINDOW_DAYS: i64 = 30; - -const DAY_MS: i64 = 24 * 60 * 60 * 1_000; - -/// Walk the entity index for `entity_id` and append every discovered leaf -/// to `tree`. Returns the number of leaves appended (NOT the number of -/// summaries sealed). Idempotent: `append_leaf` itself is a no-op when a -/// leaf is already in the buffer, so re-running backfill is safe. -pub async fn backfill_topic_tree(config: &Config, tree: &Tree, entity_id: &str) -> Result { - backfill_topic_tree_at(config, tree, entity_id, Utc::now().timestamp_millis()).await -} - -/// Deterministic variant — backfill against a caller-supplied `now_ms` -/// for the recency window. Used by tests so the 30-day cutoff doesn't -/// depend on the wall clock. -pub async fn backfill_topic_tree_at( - config: &Config, - tree: &Tree, - entity_id: &str, - now_ms: i64, -) -> Result { - let cutoff_ms = now_ms.saturating_sub(BACKFILL_WINDOW_DAYS.saturating_mul(DAY_MS)); - log::info!( - "[tree_topic::backfill] start entity_id_hash={} tree_id={} window_days={} cutoff_ms={}", - redact(entity_id), - tree.id, - BACKFILL_WINDOW_DAYS, - cutoff_ms - ); - - let hits = lookup_entity(config, entity_id, Some(BACKFILL_LIMIT)) - .with_context(|| format!("failed to lookup entity {}", redact(entity_id)))?; - - if hits.is_empty() { - log::debug!( - "[tree_topic::backfill] no entity-index hits for entity_id_hash={} — empty backfill", - redact(entity_id) - ); - return Ok(0); - } - - // Drop hits older than the hotness recency window — see module docs. - let total_hits = hits.len(); - let mut hits: Vec<_> = hits - .into_iter() - .filter(|h| h.timestamp_ms >= cutoff_ms) - .collect(); - let dropped = total_hits - hits.len(); - if dropped > 0 { - log::debug!( - "[tree_topic::backfill] dropped {dropped} hits older than {BACKFILL_WINDOW_DAYS}d \ - for entity_id_hash={}", - redact(entity_id) - ); - } - if hits.is_empty() { - log::debug!( - "[tree_topic::backfill] all entity-index hits fell outside the {BACKFILL_WINDOW_DAYS}d \ - window for entity_id_hash={} — empty backfill", - redact(entity_id) - ); - return Ok(0); - } - - // Sort by timestamp ASC so the buffer's `oldest_at` and the sealed - // summary's `time_range_start` reflect the true historical order, not - // the DESC ordering `lookup_entity` returns. - hits.sort_by_key(|h| h.timestamp_ms); - - let mut appended = 0usize; - for hit in hits { - // Skip summary-node hits — Phase 3c backfill only routes raw leaves - // into the topic tree. Including summary nodes would fold - // summaries-of-summaries across unrelated sources, which defeats - // the point. - if hit.node_kind != "leaf" { - log::debug!( - "[tree_topic::backfill] skipping non-leaf hit node_id={} kind={}", - hit.node_id, - hit.node_kind - ); - continue; - } - - let chunk = match get_chunk(config, &hit.node_id)? { - Some(c) => c, - None => { - log::warn!( - "[tree_topic::backfill] missing chunk {} for entity_id_hash={} — skipping", - hit.node_id, - redact(entity_id) - ); - continue; - } - }; - - let leaf = LeafRef { - chunk_id: chunk.id.clone(), - token_count: chunk.token_count, - timestamp: chunk.metadata.timestamp, - content: chunk.content.clone(), - entities: vec![entity_id.to_string()], - topics: chunk.metadata.tags.clone(), - score: hit.score, - }; - - // Topic-tree backfill: empty labels for sealed summaries — the - // tree's scope already pins the canonical id, so cross-pollinating - // descendants' entities would noise the index. See LabelStrategy. - append_leaf(config, tree, &leaf, &LabelStrategy::Empty) - .await - .with_context(|| { - format!( - "backfill append_leaf failed entity_id_hash={} tree_id={} chunk_id={}", - redact(entity_id), - tree.id, - chunk.id - ) - })?; - appended += 1; - } - - log::info!( - "[tree_topic::backfill] done entity_id_hash={} tree_id={} appended={}", - redact(entity_id), - tree.id, - appended - ); - - Ok(appended) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_store::trees::registry::get_or_create_topic_tree; - use crate::openhuman::memory_tree::score::extract::EntityKind; - use crate::openhuman::memory_tree::score::resolver::CanonicalEntity; - use crate::openhuman::memory_tree::score::store::index_entity; - use crate::openhuman::memory_tree::tree::store as src_store; - use chrono::{TimeZone, Utc}; - use std::sync::Arc; - use tempfile::TempDir; - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - // Phase 4 (#710): backfill may trigger seal cascades. - cfg.memory_tree.embedding_endpoint = None; - cfg.memory_tree.embedding_model = None; - cfg.memory_tree.embedding_strict = false; - (tmp, cfg) - } - - fn mk_chunk(source_id: &str, seq: u32, ts_ms: i64, tokens: u32) -> Chunk { - let ts = Utc.timestamp_millis_opt(ts_ms).unwrap(); - Chunk { - id: chunk_id(SourceKind::Chat, source_id, seq, "test-content"), - content: format!("substantive chunk mentioning alice {source_id}#{seq}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: source_id.to_string(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec!["eng".into()], - source_ref: Some(SourceRef::new(format!("{source_id}://{seq}"))), - }, - token_count: tokens, - seq_in_source: seq, - created_at: ts, - partial_message: false, - } - } - - fn sample_entity(canonical: &str, surface: &str) -> CanonicalEntity { - CanonicalEntity { - canonical_id: canonical.to_string(), - kind: EntityKind::Email, - surface: surface.to_string(), - span_start: 0, - span_end: surface.len() as u32, - score: 1.0, - } - } - - /// Deterministic "now" used by the windowed-backfill tests: 1 hour - /// after the latest seeded leaf so all three sit inside the 30-day - /// cutoff. Lets us keep the legacy 2023-era timestamps unchanged. - const TEST_NOW_MS: i64 = 1_700_000_020_000 + 3_600_000; - - #[tokio::test] - async fn backfill_appends_all_entity_leaves() { - let (_tmp, cfg) = test_config(); - // Persist 3 chunks across 2 sources. - let c1 = mk_chunk("slack:#eng", 0, 1_700_000_000_000, 100); - let c2 = mk_chunk("gmail:alice", 0, 1_700_000_010_000, 100); - let c3 = mk_chunk("slack:#eng", 1, 1_700_000_020_000, 100); - upsert_chunks(&cfg, &[c1.clone(), c2.clone(), c3.clone()]).unwrap(); - - let e = sample_entity("email:alice@example.com", "alice@example.com"); - index_entity( - &cfg, - &e, - &c1.id, - "leaf", - 1_700_000_000_000, - Some("source:slack"), - ) - .unwrap(); - index_entity( - &cfg, - &e, - &c2.id, - "leaf", - 1_700_000_010_000, - Some("source:gmail"), - ) - .unwrap(); - index_entity( - &cfg, - &e, - &c3.id, - "leaf", - 1_700_000_020_000, - Some("source:slack"), - ) - .unwrap(); - - let tree = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let n = test_override::with_provider(provider, async { - backfill_topic_tree_at(&cfg, &tree, "email:alice@example.com", TEST_NOW_MS) - .await - .unwrap() - }) - .await; - assert_eq!(n, 3); - - // L0 buffer should hold all three leaves (combined tokens well - // under the 10k seal budget). - let buf = src_store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert_eq!(buf.item_ids.len(), 3); - assert_eq!(buf.token_sum, 300); - // Oldest item is c1. - assert_eq!(buf.oldest_at.unwrap().timestamp_millis(), 1_700_000_000_000); - } - - #[tokio::test] - async fn backfill_drops_leaves_older_than_window() { - let (_tmp, cfg) = test_config(); - // c_old is 60d before TEST_NOW_MS — outside the 30d cutoff. - // c_new is 5d before TEST_NOW_MS — inside the window. - let old_ts = TEST_NOW_MS - 60 * DAY_MS; - let new_ts = TEST_NOW_MS - 5 * DAY_MS; - let c_old = mk_chunk("slack:#eng", 0, old_ts, 100); - let c_new = mk_chunk("slack:#eng", 1, new_ts, 100); - upsert_chunks(&cfg, &[c_old.clone(), c_new.clone()]).unwrap(); - - let e = sample_entity("email:alice@example.com", "alice@example.com"); - index_entity(&cfg, &e, &c_old.id, "leaf", old_ts, Some("source:slack")).unwrap(); - index_entity(&cfg, &e, &c_new.id, "leaf", new_ts, Some("source:slack")).unwrap(); - - let tree = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let n = test_override::with_provider(provider, async { - backfill_topic_tree_at(&cfg, &tree, "email:alice@example.com", TEST_NOW_MS) - .await - .unwrap() - }) - .await; - assert_eq!(n, 1, "only the in-window leaf should be appended"); - let buf = src_store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert_eq!(buf.item_ids.len(), 1); - assert_eq!(buf.item_ids[0], c_new.id); - } - - #[tokio::test] - async fn backfill_skips_missing_chunks_without_failing() { - let (_tmp, cfg) = test_config(); - let e = sample_entity("email:alice@example.com", "alice@example.com"); - // Index a chunk that was never persisted. - index_entity(&cfg, &e, "chunk:missing", "leaf", 1_700_000_000_000, None).unwrap(); - // And one that was. - let c = mk_chunk("slack:#eng", 0, 1_700_000_010_000, 100); - upsert_chunks(&cfg, &[c.clone()]).unwrap(); - index_entity( - &cfg, - &e, - &c.id, - "leaf", - 1_700_000_010_000, - Some("source:slack"), - ) - .unwrap(); - - let tree = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let n = test_override::with_provider(provider, async { - backfill_topic_tree_at(&cfg, &tree, "email:alice@example.com", TEST_NOW_MS) - .await - .unwrap() - }) - .await; - assert_eq!(n, 1, "only the existing chunk should be appended"); - let buf = src_store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert_eq!(buf.item_ids.len(), 1); - } - - #[tokio::test] - async fn backfill_is_idempotent() { - let (_tmp, cfg) = test_config(); - let c = mk_chunk("slack:#eng", 0, 1_700_000_000_000, 50); - upsert_chunks(&cfg, &[c.clone()]).unwrap(); - let e = sample_entity("email:alice@example.com", "alice@example.com"); - index_entity( - &cfg, - &e, - &c.id, - "leaf", - 1_700_000_000_000, - Some("source:slack"), - ) - .unwrap(); - - let tree = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - test_override::with_provider(provider, async { - backfill_topic_tree_at(&cfg, &tree, "email:alice@example.com", TEST_NOW_MS) - .await - .unwrap(); - backfill_topic_tree_at(&cfg, &tree, "email:alice@example.com", TEST_NOW_MS) - .await - .unwrap(); - }) - .await; - // append_leaf is idempotent so the buffer still has exactly one row. - let buf = src_store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert_eq!(buf.item_ids.len(), 1); - } - - #[tokio::test] - async fn backfill_skips_summary_nodes() { - let (_tmp, cfg) = test_config(); - let e = sample_entity("email:alice@example.com", "alice@example.com"); - // A summary-node hit in the entity index — should be skipped. - index_entity( - &cfg, - &e, - "summary:L1:abc", - "summary", - 1_700_000_000_000, - Some("source:slack"), - ) - .unwrap(); - let tree = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let n = test_override::with_provider(provider, async { - backfill_topic_tree_at(&cfg, &tree, "email:alice@example.com", TEST_NOW_MS) - .await - .unwrap() - }) - .await; - assert_eq!(n, 0); - } -} diff --git a/src/openhuman/memory/tree_topic/curator.rs b/src/openhuman/memory/tree_topic/curator.rs deleted file mode 100644 index 9059389b4..000000000 --- a/src/openhuman/memory/tree_topic/curator.rs +++ /dev/null @@ -1,385 +0,0 @@ -//! Topic-tree curator — the hotness gate (#709 Phase 3c). -//! -//! On every ingest that touches an entity we bump cheap counters -//! (`mention_count_30d`, `last_seen_ms`, `ingests_since_check`). Every -//! [`TOPIC_RECHECK_EVERY`] bumps we run the full hotness recompute: -//! -//! 1. Refresh `distinct_sources` from `mem_tree_entity_index`. -//! 2. Compute [`hotness`](super::hotness::hotness). -//! 3. If hotness ≥ [`TOPIC_CREATION_THRESHOLD`] and no topic tree exists -//! yet → create one and kick off [`backfill_topic_tree`]. -//! 4. Reset `ingests_since_check` to 0. -//! -//! The function is idempotent: if a topic tree already exists for the -//! entity it's a no-op at the creation step. Spawning is single-shot — -//! re-crossing the threshold after an archive would require explicit -//! unarchival (not Phase 3c). - -use anyhow::Result; -use chrono::Utc; - -use crate::openhuman::config::Config; -use crate::openhuman::memory::tree_policy::TreePolicy; -use crate::openhuman::memory::tree_topic::backfill::backfill_topic_tree; -use crate::openhuman::memory_store::trees::hotness::{distinct_sources_for, get_or_fresh, upsert}; -use crate::openhuman::memory_store::trees::types::HotnessCounters; -use crate::openhuman::memory_store::trees::types::{Tree, TreeKind}; -use crate::openhuman::memory_tree::tree::store as src_store; - -/// Outcome of one curator invocation. Surfaced so the caller (typically -/// the routing layer) can log / emit metrics. -#[derive(Clone, Debug, PartialEq)] -pub enum SpawnOutcome { - /// Counters bumped; hotness not yet recomputed this round. - CountersBumped, - /// Full recompute ran; hotness below threshold, no tree spawned. - BelowThreshold { hotness: f32 }, - /// Tree already existed — just bumped counters and refreshed hotness. - TreeExists { hotness: f32, tree_id: String }, - /// Brand new topic tree materialised. - Spawned { - hotness: f32, - tree_id: String, - backfilled: usize, - }, -} - -/// Record an ingest touching `entity_id` and, when the recheck cadence -/// fires, consider spawning a topic tree. -pub async fn maybe_spawn_topic_tree(config: &Config, entity_id: &str) -> Result { - let now_ms = Utc::now().timestamp_millis(); - - // 1. Read existing counters (fresh row if first sighting). - let mut counters = get_or_fresh(config, entity_id)?; - - // 2. Cheap per-ingest bumps. - counters.mention_count_30d = counters.mention_count_30d.saturating_add(1); - counters.last_seen_ms = Some(now_ms); - counters.ingests_since_check = counters.ingests_since_check.saturating_add(1); - counters.last_updated_ms = now_ms; - - // 3. Decide whether to run the full recompute. - if counters.ingests_since_check < TreePolicy::topic().topic_recheck_every() { - upsert(config, &counters)?; - log::debug!( - "[tree_topic::curator] bumped counters entity={} mentions={} ingests_since_check={}", - entity_id, - counters.mention_count_30d, - counters.ingests_since_check - ); - return Ok(SpawnOutcome::CountersBumped); - } - - // 4. Full recompute. - run_full_recompute(config, entity_id, &mut counters, now_ms).await -} - -/// Admin path: force a recompute + spawn-if-hot regardless of the -/// [`TOPIC_RECHECK_EVERY`] cadence. Used by (future) RPCs that want to -/// prod the curator without waiting for the next bump cycle. -pub async fn force_recompute(config: &Config, entity_id: &str) -> Result { - let now_ms = Utc::now().timestamp_millis(); - let mut counters = get_or_fresh(config, entity_id)?; - counters.last_updated_ms = now_ms; - run_full_recompute(config, entity_id, &mut counters, now_ms).await -} - -async fn run_full_recompute( - config: &Config, - entity_id: &str, - counters: &mut HotnessCounters, - now_ms: i64, -) -> Result { - // Refresh distinct_sources from the entity index — the authoritative - // source of cross-tree coverage. - let distinct = distinct_sources_for(config, entity_id)?; - counters.distinct_sources = distinct; - - // Compute hotness against the refreshed stats. - let stats = counters.stats(); - let h = TreePolicy::topic().topic_hotness(entity_id, &stats, now_ms); - - counters.last_hotness = Some(h); - counters.ingests_since_check = 0; - - let outcome = if h < TreePolicy::topic().topic_creation_threshold() { - log::debug!( - "[tree_topic::curator] below threshold entity={} hotness={:.3} threshold={}", - entity_id, - h, - TreePolicy::topic().topic_creation_threshold() - ); - SpawnOutcome::BelowThreshold { hotness: h } - } else if let Some(existing) = existing_topic_tree(config, entity_id)? { - log::debug!( - "[tree_topic::curator] tree already exists entity={} tree_id={} hotness={:.3}", - entity_id, - existing.id, - h - ); - SpawnOutcome::TreeExists { - hotness: h, - tree_id: existing.id, - } - } else { - // Crossed threshold for the first time — materialise. - log::info!( - "[tree_topic::curator] spawning topic tree entity={} hotness={:.3}", - entity_id, - h - ); - let tree = - crate::openhuman::memory::tree_topic::factory(entity_id).get_or_create(config)?; - let backfilled = backfill_topic_tree(config, &tree, entity_id).await?; - SpawnOutcome::Spawned { - hotness: h, - tree_id: tree.id, - backfilled, - } - }; - - // Persist the refreshed counters regardless of outcome. - upsert(config, counters)?; - Ok(outcome) -} - -fn existing_topic_tree(config: &Config, entity_id: &str) -> Result> { - src_store::get_tree_by_scope(config, TreeKind::Topic, entity_id) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_store::trees::hotness::get; - use crate::openhuman::memory_store::trees::{TOPIC_CREATION_THRESHOLD, TOPIC_RECHECK_EVERY}; - use crate::openhuman::memory_tree::score::extract::EntityKind; - use crate::openhuman::memory_tree::score::resolver::CanonicalEntity; - use crate::openhuman::memory_tree::score::store::index_entity; - use chrono::{TimeZone, Utc}; - use std::sync::Arc; - use tempfile::TempDir; - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - (tmp, cfg) - } - - fn seed_leaf_for_entity(cfg: &Config, entity_id: &str, source_tree: &str, seq: u32) { - // Use a "now-anchored" timestamp so backfill's 30-day window - // (see tree_topic::backfill::BACKFILL_WINDOW_DAYS) always - // includes these seeded leaves. Spread by seq to keep ordering - // deterministic. - let ts_ms = Utc::now().timestamp_millis() - (seq as i64) * 1_000; - let ts = Utc.timestamp_millis_opt(ts_ms).unwrap(); - let c = Chunk { - id: chunk_id(SourceKind::Chat, source_tree, seq, "test-content"), - content: format!("mentioning entity in {source_tree}#{seq}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: source_tree.to_string(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new(format!("{source_tree}://{seq}"))), - }, - token_count: 50, - seq_in_source: seq, - created_at: ts, - partial_message: false, - }; - upsert_chunks(cfg, &[c.clone()]).unwrap(); - let e = CanonicalEntity { - canonical_id: entity_id.to_string(), - kind: EntityKind::Email, - surface: entity_id.to_string(), - span_start: 0, - span_end: entity_id.len() as u32, - score: 1.0, - }; - index_entity(cfg, &e, &c.id, "leaf", ts_ms, Some(source_tree)).unwrap(); - } - - #[tokio::test] - async fn first_ingest_just_bumps_counters() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let out = test_override::with_provider(provider, async { - maybe_spawn_topic_tree(&cfg, "email:alice@example.com") - .await - .unwrap() - }) - .await; - assert_eq!(out, SpawnOutcome::CountersBumped); - let c = get(&cfg, "email:alice@example.com").unwrap().unwrap(); - assert_eq!(c.mention_count_30d, 1); - assert_eq!(c.ingests_since_check, 1); - assert!(c.last_hotness.is_none(), "no recompute yet"); - } - - #[tokio::test] - async fn no_spawn_below_threshold_on_recompute() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - // Force a recompute on the very first call — but with no index data - // the hotness comes out well below threshold. - let out = test_override::with_provider(provider, async { - force_recompute(&cfg, "email:alice@example.com") - .await - .unwrap() - }) - .await; - match out { - SpawnOutcome::BelowThreshold { hotness } => { - assert!(hotness < TOPIC_CREATION_THRESHOLD); - } - other => panic!("expected BelowThreshold, got {other:?}"), - } - // No topic tree created. - let t = existing_topic_tree(&cfg, "email:alice@example.com").unwrap(); - assert!(t.is_none()); - } - - #[tokio::test] - async fn spawn_fires_exactly_once_when_threshold_crossed() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - // Seed substantial activity across several sources so hotness is - // well above threshold. - let mut counters = HotnessCounters::fresh("email:alice@example.com", 0); - counters.mention_count_30d = 500; - counters.distinct_sources = 4; - counters.last_seen_ms = Some(Utc::now().timestamp_millis()); - counters.query_hits_30d = 5; - upsert(&cfg, &counters).unwrap(); - // Seed leaves in the entity index so backfill has something to do. - for i in 0..3 { - seed_leaf_for_entity(&cfg, "email:alice@example.com", "slack:#eng", i); - } - for i in 0..2 { - seed_leaf_for_entity(&cfg, "email:alice@example.com", "gmail:alice", i); - } - - let (out, out2) = test_override::with_provider(provider, async { - let o1 = force_recompute(&cfg, "email:alice@example.com") - .await - .unwrap(); - // Re-running should report TreeExists, NOT a second spawn. - let o2 = force_recompute(&cfg, "email:alice@example.com") - .await - .unwrap(); - (o1, o2) - }) - .await; - - match out { - SpawnOutcome::Spawned { - hotness, - tree_id, - backfilled, - } => { - assert!(hotness >= TOPIC_CREATION_THRESHOLD); - assert!(tree_id.starts_with("topic:")); - assert_eq!(backfilled, 5); - } - other => panic!("expected Spawned, got {other:?}"), - } - match out2 { - SpawnOutcome::TreeExists { tree_id, .. } => { - assert!(tree_id.starts_with("topic:")); - } - other => panic!("expected TreeExists on retry, got {other:?}"), - } - } - - #[tokio::test] - async fn recompute_refreshes_distinct_sources_from_entity_index() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - // Counter says 0 distinct sources but the index has 3. - let mut counters = HotnessCounters::fresh("email:alice@example.com", 0); - counters.mention_count_30d = 1; - counters.distinct_sources = 0; - upsert(&cfg, &counters).unwrap(); - seed_leaf_for_entity(&cfg, "email:alice@example.com", "slack:#eng", 0); - seed_leaf_for_entity(&cfg, "email:alice@example.com", "gmail:alice", 0); - seed_leaf_for_entity(&cfg, "email:alice@example.com", "notion:abc", 0); - - test_override::with_provider(provider, async { - force_recompute(&cfg, "email:alice@example.com") - .await - .unwrap(); - }) - .await; - let c = get(&cfg, "email:alice@example.com").unwrap().unwrap(); - assert_eq!(c.distinct_sources, 3); - // ingests_since_check should also reset. - assert_eq!(c.ingests_since_check, 0); - } - - #[tokio::test] - async fn cadence_only_recomputes_every_n_ingests() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - // Pre-seed entity index with enough cross-source signal that the - // recompute (which refreshes `distinct_sources` from the index) will - // still produce a hotness above threshold. - for i in 0..4 { - seed_leaf_for_entity( - &cfg, - "email:alice@example.com", - &format!("slack:#eng-{i}"), - i, - ); - } - - let mut counters = HotnessCounters::fresh("email:alice@example.com", 0); - counters.mention_count_30d = 500; - counters.distinct_sources = 4; - counters.last_seen_ms = Some(Utc::now().timestamp_millis()); - // Boost query_hits so hotness stays comfortably above threshold - // after the distinct_sources refresh. - counters.query_hits_30d = 5; - // ingests_since_check just below the cadence: next call should - // NOT yet recompute. - counters.ingests_since_check = TOPIC_RECHECK_EVERY - 2; - upsert(&cfg, &counters).unwrap(); - - let out2 = test_override::with_provider(provider, async { - let o1 = maybe_spawn_topic_tree(&cfg, "email:alice@example.com") - .await - .unwrap(); - assert_eq!(o1, SpawnOutcome::CountersBumped); - // No tree yet — cadence not crossed after first call. - assert!( - existing_topic_tree(&cfg, "email:alice@example.com") - .unwrap() - .is_none(), - "no topic tree should exist before cadence is crossed" - ); - // One more bump — now ingests_since_check == TOPIC_RECHECK_EVERY - // and the recompute fires. - maybe_spawn_topic_tree(&cfg, "email:alice@example.com") - .await - .unwrap() - }) - .await; - - match out2 { - SpawnOutcome::Spawned { .. } | SpawnOutcome::TreeExists { .. } => {} - other => panic!("expected Spawn/TreeExists after cadence, got {other:?}"), - } - } -} diff --git a/src/openhuman/memory/tree_topic/hotness.rs b/src/openhuman/memory/tree_topic/hotness.rs deleted file mode 100644 index 236e2bc70..000000000 --- a/src/openhuman/memory/tree_topic/hotness.rs +++ /dev/null @@ -1,167 +0,0 @@ -//! Pure hotness math for Phase 3c (#709). -//! -//! The formula intentionally folds a handful of pre-existing signals into -//! one arithmetic score. No LLM, no learned weights — the goal is -//! deterministic, greppable, testable behaviour: -//! -//! ```text -//! hotness = ln(mentions + 1) // dampened high-volume bias -//! + 0.5 * distinct_sources // cross-source is valuable -//! + recency_decay(last_seen) // prefer active entities -//! + graph_centrality // Phase 4+ (None → 0.0) -//! + 2.0 * query_hits // retrieval feedback (Phase 4+) -//! ``` -//! -//! Recency decay is a piecewise linear taper: -//! - age ≤ 1 day → 1.0 -//! - age 1…7 days → 1.0 → 0.5 -//! - age 7…30 days → 0.5 → 0.0 -//! - age > 30 days → 0.0 -//! -//! The unit tests lock in the coarse behaviour (zero-mention, spike, -//! old-but-widely-cited) so tuning the constants later stays honest. - -use chrono::Utc; - -use crate::openhuman::memory::tree_policy::TreePolicy; -use crate::openhuman::memory_store::trees::types::EntityIndexStats; - -/// Pure hotness function — no I/O, no clocks unless the caller passes one. -/// -/// `entity_id` is taken for diagnostic logging only and has no effect on -/// the numeric result. -pub fn hotness(entity_id: &str, idx: &EntityIndexStats) -> f32 { - let now_ms = Utc::now().timestamp_millis(); - hotness_at(entity_id, idx, now_ms) -} - -/// Deterministic variant — computes hotness as if the current wall clock -/// were `now_ms`. Useful in tests so the recency term doesn't drift. -pub fn hotness_at(entity_id: &str, idx: &EntityIndexStats, now_ms: i64) -> f32 { - TreePolicy::topic().topic_hotness(entity_id, idx, now_ms) -} - -/// Recency decay helper. Operates on absolute epoch-millis so tests can -/// pin the clock. Returns 0.0 when `last_seen_ms` is `None`. -pub fn recency_decay(last_seen_ms: Option, now_ms: i64) -> f32 { - TreePolicy::topic().topic_recency_decay(last_seen_ms, now_ms) -} - -#[cfg(test)] -mod tests { - use super::*; - - const DAY_MS: i64 = 24 * 60 * 60 * 1_000; - - fn stats(mentions: u32, sources: u32, last_seen: Option) -> EntityIndexStats { - EntityIndexStats { - mention_count_30d: mentions, - distinct_sources: sources, - last_seen_ms: last_seen, - query_hits_30d: 0, - graph_centrality: None, - } - } - - #[test] - fn zero_signal_entity_is_zero() { - let now_ms = 1_700_000_000_000; - let s = stats(0, 0, None); - let h = hotness_at("e:none", &s, now_ms); - // ln(0+1) + 0 + 0 + 0 + 0 = 0 - assert!(h.abs() < 1e-6); - } - - #[test] - fn spike_of_mentions_pushes_over_creation_threshold() { - use crate::openhuman::memory_store::trees::types::TOPIC_CREATION_THRESHOLD; - let now_ms = 1_700_000_000_000; - // 100 mentions across 5 sources, 3 recent query hits, seen today. - let s = EntityIndexStats { - mention_count_30d: 100, - distinct_sources: 5, - last_seen_ms: Some(now_ms - DAY_MS / 2), - query_hits_30d: 3, - graph_centrality: None, - }; - let h = hotness_at("e:hot", &s, now_ms); - assert!( - h > TOPIC_CREATION_THRESHOLD, - "expected hot entity > {TOPIC_CREATION_THRESHOLD}, got {h}" - ); - } - - #[test] - fn old_but_widely_cited_still_has_some_heat() { - // 50 mentions, 8 sources, last seen 20 days ago, no queries. - let now_ms = 1_700_000_000_000; - let s = EntityIndexStats { - mention_count_30d: 50, - distinct_sources: 8, - last_seen_ms: Some(now_ms - 20 * DAY_MS), - query_hits_30d: 0, - graph_centrality: None, - }; - let h = hotness_at("e:old-wide", &s, now_ms); - // mention_weight = ln(51) ≈ 3.93, source_weight = 4.0, - // recency at day 20 ≈ 0.5 * (30-20)/23 ≈ 0.217 → total ≈ 8.1 - assert!(h > 5.0, "widely-cited entity should retain signal: {h}"); - } - - #[test] - fn ancient_single_mention_decays_toward_zero() { - let now_ms = 1_700_000_000_000; - let s = stats(1, 1, Some(now_ms - 60 * DAY_MS)); - let h = hotness_at("e:ancient", &s, now_ms); - // ln(2) + 0.5 + 0 = ~1.19 — well below creation threshold - assert!(h < 2.0, "ancient entity should decay: {h}"); - } - - #[test] - fn recency_decay_today_is_one() { - let now_ms = 1_700_000_000_000; - let r = recency_decay(Some(now_ms), now_ms); - assert!((r - 1.0).abs() < 1e-6); - } - - #[test] - fn recency_decay_week_old_is_half() { - let now_ms = 1_700_000_000_000; - let r = recency_decay(Some(now_ms - 7 * DAY_MS), now_ms); - assert!((r - 0.5).abs() < 1e-3, "expected 0.5 at 7d, got {r}"); - } - - #[test] - fn recency_decay_month_old_is_zero() { - let now_ms = 1_700_000_000_000; - let r = recency_decay(Some(now_ms - 30 * DAY_MS), now_ms); - assert!(r.abs() < 1e-3, "expected ~0 at 30d, got {r}"); - } - - #[test] - fn recency_decay_none_last_seen_is_zero() { - assert_eq!(recency_decay(None, 1_700_000_000_000), 0.0); - } - - #[test] - fn query_hits_boost_hotness_aggressively() { - let now_ms = 1_700_000_000_000; - let base = stats(5, 1, Some(now_ms)); - let boosted = EntityIndexStats { - query_hits_30d: 10, - ..base.clone() - }; - let h_base = hotness_at("e", &base, now_ms); - let h_boosted = hotness_at("e", &boosted, now_ms); - // 10 query hits * 2.0 = +20 - assert!(h_boosted - h_base > 19.0); - } - - #[test] - fn future_last_seen_is_treated_as_now() { - // Clock drift could produce negative ages — we clamp at 0. - let now_ms = 1_700_000_000_000; - let r = recency_decay(Some(now_ms + DAY_MS), now_ms); - assert!((r - 1.0).abs() < 1e-6); - } -} diff --git a/src/openhuman/memory/tree_topic/mod.rs b/src/openhuman/memory/tree_topic/mod.rs deleted file mode 100644 index 38825f319..000000000 --- a/src/openhuman/memory/tree_topic/mod.rs +++ /dev/null @@ -1,32 +0,0 @@ -//! Topic tree instance — policy and orchestration for per-entity -//! topic trees. -//! -//! The generic tree engine lives in [`memory_tree`]; this module owns -//! the topic-specific algorithms: hotness scoring, curator (spawn -//! gate), per-leaf routing, and historical backfill. - -pub mod backfill; -pub mod curator; -pub mod hotness; -pub mod routing; - -use crate::openhuman::memory_tree::tree::TreeFactory; - -pub use crate::openhuman::memory_store::trees::hotness as store; -pub use crate::openhuman::memory_store::trees::registry; -pub use crate::openhuman::memory_store::trees::types; -pub use crate::openhuman::memory_store::trees::{ - archive_topic_tree, force_create_topic_tree, get_or_create_topic_tree, list_topic_trees, -}; -pub use crate::openhuman::memory_store::trees::{ - EntityIndexStats, HotnessCounters, TOPIC_ARCHIVE_THRESHOLD, TOPIC_CREATION_THRESHOLD, - TOPIC_RECHECK_EVERY, -}; -pub use curator::{maybe_spawn_topic_tree, SpawnOutcome}; -pub use hotness::{hotness, recency_decay}; -pub use routing::route_leaf_to_topic_trees; - -/// Canonical factory for one topic tree scope / entity id. -pub fn factory(scope: &str) -> TreeFactory<'_> { - TreeFactory::topic(scope) -} diff --git a/src/openhuman/memory/tree_topic/routing.rs b/src/openhuman/memory/tree_topic/routing.rs deleted file mode 100644 index 2360a23d9..000000000 --- a/src/openhuman/memory/tree_topic/routing.rs +++ /dev/null @@ -1,354 +0,0 @@ -//! Per-leaf routing into topic trees (#709 Phase 3c). -//! -//! This is the hook point the ingest path calls after it has finished -//! appending a leaf to its source tree. For each canonical entity on the -//! chunk we: -//! -//! 1. Append the leaf to that entity's topic tree *if* one already exists -//! (active status only — archived topic trees don't receive new -//! leaves). -//! 2. Notify the curator that this entity was just mentioned, which may -//! cross the hotness threshold and spawn a new topic tree. -//! -//! Steps 1 and 2 are independent — if an entity's topic tree already -//! exists, step 2 just bumps counters; if it doesn't, step 1 is skipped -//! and step 2 may materialise it on this ingest. -//! -//! Failures are logged at warn level but never bubble up: Phase 3c is -//! additive and must not poison the ingest path. The source-tree append -//! has already succeeded by the time we get here. - -use anyhow::Result; - -use crate::openhuman::config::Config; -use crate::openhuman::memory::tree_topic::curator::maybe_spawn_topic_tree; -use crate::openhuman::memory_store::trees::types::{TreeKind, TreeStatus}; -use crate::openhuman::memory_tree::tree::bucket_seal::{append_leaf, LabelStrategy, LeafRef}; -use crate::openhuman::memory_tree::tree::store as src_store; - -/// Route `leaf` into every active topic tree matching one of -/// `canonical_entities`. Also ticks the curator for each entity so the -/// next cadence-aligned ingest may spawn a new tree. -/// -/// Returns `Ok(())` even if individual entities fail — per-entity errors -/// are logged. A hard DB failure early in the process is surfaced so the -/// caller can decide how loud to be in logs. -pub async fn route_leaf_to_topic_trees( - config: &Config, - leaf: &LeafRef, - canonical_entities: &[String], -) -> Result<()> { - if canonical_entities.is_empty() { - return Ok(()); - } - - log::debug!( - "[tree_topic::routing] leaf={} entities={}", - leaf.chunk_id, - canonical_entities.len() - ); - - for entity_id in canonical_entities { - if let Err(e) = route_one_entity(config, leaf, entity_id).await { - let entity_kind = entity_id - .split_once(':') - .map(|(k, _)| k) - .unwrap_or("unknown"); - log::warn!( - "[tree_topic::routing] failed routing leaf={} entity_kind={} err={:#}", - leaf.chunk_id, - entity_kind, - e - ); - } - } - Ok(()) -} - -async fn route_one_entity(config: &Config, leaf: &LeafRef, entity_id: &str) -> Result<()> { - // Step 1: if a topic tree already exists and is active, append the leaf. - // We intentionally do this BEFORE asking the curator to spawn — a - // same-call spawn would also include this leaf via backfill - // (`lookup_entity` was just updated by the ingest's score persist) but - // keeping the existing-tree fast path separate keeps the common case - // (hot entity already has a tree) clean. - if let Some(tree) = src_store::get_tree_by_scope(config, TreeKind::Topic, entity_id)? { - if tree.status == TreeStatus::Active { - log::debug!( - "[tree_topic::routing] appending leaf={} → topic_tree={}", - leaf.chunk_id, - tree.id - ); - // Rebuild the leaf with this entity-id stamped on so the seal - // path sees the topic membership. The source-tree append used - // the full entity list; here we scope to just this entity so - // the curated summariser (future) can prompt accordingly. - let topic_leaf = LeafRef { - entities: vec![entity_id.to_string()], - ..leaf.clone() - }; - // Topic-tree seals leave entities/topics empty: the tree's - // scope already pins the canonical id this tree represents. - append_leaf(config, &tree, &topic_leaf, &LabelStrategy::Empty).await?; - } else { - let entity_kind = entity_id - .split_once(':') - .map(|(k, _)| k) - .unwrap_or("unknown"); - log::debug!( - "[tree_topic::routing] skip archived topic tree id={} entity_kind={}", - tree.id, - entity_kind - ); - } - } - - // Step 2: curator tick — may spawn a new tree on cadence. - maybe_spawn_topic_tree(config, entity_id).await?; - - Ok(()) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_store::trees::hotness::get as get_hotness; - use crate::openhuman::memory_store::trees::registry::{ - archive_topic_tree, get_or_create_topic_tree, - }; - use crate::openhuman::memory_tree::score::extract::EntityKind; - use crate::openhuman::memory_tree::score::resolver::CanonicalEntity; - use crate::openhuman::memory_tree::score::store::index_entity; - use chrono::{TimeZone, Utc}; - use std::sync::Arc; - use tempfile::TempDir; - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - // Phase 4 (#710): routing may trigger seals which embed. - cfg.memory_tree.embedding_endpoint = None; - cfg.memory_tree.embedding_model = None; - cfg.memory_tree.embedding_strict = false; - (tmp, cfg) - } - - fn mk_leaf(chunk_id_s: &str, tokens: u32, ts_ms: i64) -> LeafRef { - LeafRef { - chunk_id: chunk_id_s.to_string(), - token_count: tokens, - timestamp: Utc.timestamp_millis_opt(ts_ms).unwrap(), - content: format!("content for {chunk_id_s}"), - entities: vec!["email:alice@example.com".into()], - topics: vec![], - score: 0.5, - } - } - - fn persist_chunk(cfg: &Config, source_id: &str, seq: u32, ts_ms: i64, tokens: u32) -> String { - let ts = Utc.timestamp_millis_opt(ts_ms).unwrap(); - let c = Chunk { - id: chunk_id(SourceKind::Chat, source_id, seq, "test-content"), - content: format!("chunk content {source_id} {seq}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: source_id.to_string(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new(format!("{source_id}://{seq}"))), - }, - token_count: tokens, - seq_in_source: seq, - created_at: ts, - partial_message: false, - }; - let id = c.id.clone(); - upsert_chunks(cfg, &[c]).unwrap(); - id - } - - #[tokio::test] - async fn empty_entities_is_noop() { - let (_tmp, cfg) = test_config(); - let leaf = mk_leaf("c1", 10, 1_700_000_000_000); - route_leaf_to_topic_trees(&cfg, &leaf, &[]).await.unwrap(); - // No hotness rows were created. - assert_eq!( - crate::openhuman::memory_store::trees::hotness::count(&cfg).unwrap(), - 0 - ); - } - - #[tokio::test] - async fn appends_to_existing_topic_tree() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - // Pre-create the topic tree so the hot-path append fires. - let tree = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - // Persist the backing chunk so hydrate can read it on seal. - let chunk_id_s = persist_chunk(&cfg, "slack:#eng", 0, 1_700_000_000_000, 100); - let leaf = mk_leaf(&chunk_id_s, 100, 1_700_000_000_000); - - test_override::with_provider(provider, async { - route_leaf_to_topic_trees(&cfg, &leaf, &["email:alice@example.com".to_string()]) - .await - .unwrap() - }) - .await; - - let buf = src_store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert_eq!(buf.item_ids.len(), 1); - assert_eq!(buf.item_ids[0], chunk_id_s); - // Counter should also be bumped. - let c = get_hotness(&cfg, "email:alice@example.com") - .unwrap() - .unwrap(); - assert_eq!(c.mention_count_30d, 1); - } - - #[tokio::test] - async fn archived_topic_tree_does_not_receive_new_leaves() { - let (_tmp, cfg) = test_config(); - let tree = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - archive_topic_tree(&cfg, &tree.id).unwrap(); - - let chunk_id_s = persist_chunk(&cfg, "slack:#eng", 0, 1_700_000_000_000, 100); - let leaf = mk_leaf(&chunk_id_s, 100, 1_700_000_000_000); - route_leaf_to_topic_trees(&cfg, &leaf, &["email:alice@example.com".to_string()]) - .await - .unwrap(); - - let buf = src_store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert!( - buf.is_empty(), - "archived topic tree should not receive new leaves" - ); - // Counter should still be bumped — archiving doesn't freeze hotness. - let c = get_hotness(&cfg, "email:alice@example.com") - .unwrap() - .unwrap(); - assert_eq!(c.mention_count_30d, 1); - } - - #[tokio::test] - async fn one_leaf_multiple_entities_fans_out() { - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let t1 = get_or_create_topic_tree(&cfg, "email:alice@example.com").unwrap(); - let t2 = get_or_create_topic_tree(&cfg, "hashtag:launch").unwrap(); - - let chunk_id_s = persist_chunk(&cfg, "slack:#eng", 0, 1_700_000_000_000, 100); - let leaf = mk_leaf(&chunk_id_s, 100, 1_700_000_000_000); - test_override::with_provider(provider, async { - route_leaf_to_topic_trees( - &cfg, - &leaf, - &[ - "email:alice@example.com".to_string(), - "hashtag:launch".to_string(), - ], - ) - .await - .unwrap() - }) - .await; - - // Both topic trees' L0 buffers hold the leaf. - let b1 = src_store::get_buffer(&cfg, &t1.id, 0).unwrap(); - let b2 = src_store::get_buffer(&cfg, &t2.id, 0).unwrap(); - assert_eq!(b1.item_ids.len(), 1); - assert_eq!(b2.item_ids.len(), 1); - } - - #[tokio::test] - async fn integration_two_sources_mentioning_alice_materialise_topic_tree() { - // Phase 3c acceptance scenario: ingest across 2 sources mentioning - // Alice → hotness crosses threshold → topic tree materialised → - // new Alice-mentioning leaf routes into both the source tree AND - // the topic tree. - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let entity_id = "email:alice@example.com"; - - // Pre-seed counters / index so the next call crosses threshold. - // Note: the curator refreshes `distinct_sources` from the entity - // index during recompute, so we also need enough `query_hits_30d` - // to keep hotness above `TOPIC_CREATION_THRESHOLD` once the index - // is queried (two indexed sources below → distinct_sources → 2). - let mut counters = - crate::openhuman::memory_store::trees::types::HotnessCounters::fresh(entity_id, 0); - counters.mention_count_30d = 1_000; - counters.distinct_sources = 2; - counters.last_seen_ms = Some(Utc::now().timestamp_millis()); - counters.query_hits_30d = 5; - counters.ingests_since_check = - crate::openhuman::memory_store::trees::types::TOPIC_RECHECK_EVERY - 1; - crate::openhuman::memory_store::trees::hotness::upsert(&cfg, &counters).unwrap(); - - // Seed leaves in slack and gmail referencing Alice. Anchor the - // timestamps to "now" so the 30-day backfill window - // (tree_topic::backfill::BACKFILL_WINDOW_DAYS) covers them. - let now_ms = Utc::now().timestamp_millis(); - let ts_c1 = now_ms - 20_000; - let ts_c2 = now_ms - 10_000; - let ts_c3 = now_ms; - let c1 = persist_chunk(&cfg, "slack:#eng", 0, ts_c1, 100); - let c2 = persist_chunk(&cfg, "gmail:alice", 0, ts_c2, 100); - let e = CanonicalEntity { - canonical_id: entity_id.into(), - kind: EntityKind::Email, - surface: entity_id.into(), - span_start: 0, - span_end: entity_id.len() as u32, - score: 1.0, - }; - index_entity(&cfg, &e, &c1, "leaf", ts_c1, Some("slack:#eng")).unwrap(); - index_entity(&cfg, &e, &c2, "leaf", ts_c2, Some("gmail:alice")).unwrap(); - - // A third leaf arrives — should both fan out to (future) topic tree - // and push the curator over the recheck cadence, materialising it. - let c3 = persist_chunk(&cfg, "slack:#eng", 1, ts_c3, 100); - let leaf = LeafRef { - chunk_id: c3.clone(), - token_count: 100, - timestamp: Utc.timestamp_millis_opt(ts_c3).unwrap(), - content: "new mention".into(), - entities: vec![entity_id.into()], - topics: vec![], - score: 0.5, - }; - - test_override::with_provider(provider, async { - route_leaf_to_topic_trees(&cfg, &leaf, &[entity_id.to_string()]) - .await - .unwrap() - }) - .await; - - // Topic tree now exists. - let tree = src_store::get_tree_by_scope(&cfg, TreeKind::Topic, entity_id) - .unwrap() - .expect("topic tree should be materialised"); - assert_eq!(tree.kind, TreeKind::Topic); - assert_eq!(tree.scope, entity_id); - // Backfill pulled c1 + c2 into the buffer. (c3 didn't get into the - // entity index during this test since we didn't run the full ingest - // path — we're exercising routing in isolation.) - let buf = src_store::get_buffer(&cfg, &tree.id, 0).unwrap(); - assert!( - buf.item_ids.len() >= 2, - "backfill should pull historic leaves" - ); - } -} diff --git a/src/openhuman/memory_queue/handlers/mod.rs b/src/openhuman/memory_queue/handlers/mod.rs index 21c2bad5f..d042d5007 100644 --- a/src/openhuman/memory_queue/handlers/mod.rs +++ b/src/openhuman/memory_queue/handlers/mod.rs @@ -12,14 +12,11 @@ use anyhow::{Context, Result}; use crate::openhuman::config::Config; -use crate::openhuman::memory::tree_global::digest::{self, DigestOutcome}; use crate::openhuman::memory::tree_source::get_or_create_source_tree; -use crate::openhuman::memory::tree_topic::curator; use crate::openhuman::memory_queue::store; use crate::openhuman::memory_queue::types::{ - AppendBufferPayload, AppendTarget, DigestDailyPayload, ExtractChunkPayload, FlushStalePayload, - Job, JobKind, JobOutcome, NewJob, NodeRef, ReembedBackfillPayload, SealPayload, - TopicRoutePayload, + AppendBufferPayload, AppendTarget, ExtractChunkPayload, FlushStalePayload, Job, JobKind, + JobOutcome, NewJob, NodeRef, ReembedBackfillPayload, SealPayload, }; use crate::openhuman::memory_store::chunks::store as chunk_store; use crate::openhuman::memory_store::content::{ @@ -46,8 +43,6 @@ pub async fn handle_job(config: &Config, job: &Job) -> Result { JobKind::ExtractChunk => handle_extract(config, job).await, JobKind::AppendBuffer => handle_append_buffer(config, job).await, JobKind::Seal => handle_seal(config, job).await, - JobKind::TopicRoute => handle_topic_route(config, job).await, - JobKind::DigestDaily => handle_digest_daily(config, job).await, JobKind::FlushStale => handle_flush_stale(config, job).await, JobKind::ReembedBackfill => handle_reembed_backfill(config, job).await, } @@ -115,20 +110,11 @@ async fn handle_extract(config: &Config, job: &Job) -> Result { } else { None }; - let route_job = if result.kept { - Some(NewJob::topic_route(&TopicRoutePayload { - node: NodeRef::Leaf { - chunk_id: chunk.id.clone(), - }, - })?) - } else { - None - }; // #1574: resolve the active embedding signature once (probe-stable, // config-derived) so the sidecar write below is keyed correctly. let active_sig = chunk_store::tree_active_signature(config); - let (did_enqueue_source, did_enqueue_route) = chunk_store::with_connection(config, |conn| { + let did_enqueue_source = chunk_store::with_connection(config, |conn| { let tx = conn.unchecked_transaction()?; score::persist_score_tx( &tx, @@ -167,19 +153,15 @@ async fn handle_extract(config: &Config, job: &Job) -> Result { )?; } - // Enqueue follow-up jobs inside the SAME transaction so they are - // atomically visible with the lifecycle update. + // Enqueue the source append-buffer follow-up inside the SAME + // transaction so it is atomically visible with the lifecycle update. let mut eq_src = false; - let mut eq_route = false; if let Some(ref j) = source_job { eq_src = store::enqueue_tx(&tx, j)?.is_some(); } - if let Some(ref j) = route_job { - eq_route = store::enqueue_tx(&tx, j)?.is_some(); - } tx.commit()?; - Ok((eq_src, eq_route)) + Ok(eq_src) })?; // Phase MD-content: rewrite the `tags:` block in the on-disk chunk file @@ -229,9 +211,6 @@ async fn handle_extract(config: &Config, job: &Job) -> Result { if did_enqueue_source { super::worker::wake_workers(); } - if did_enqueue_route { - super::worker::wake_workers(); - } Ok(JobOutcome::Done) } @@ -438,76 +417,6 @@ async fn handle_seal(config: &Config, job: &Job) -> Result { Ok(JobOutcome::Done) } -async fn handle_topic_route(config: &Config, job: &Job) -> Result { - let payload: TopicRoutePayload = - serde_json::from_str(&job.payload_json).context("parse TopicRoute payload")?; - - // Resolve the source node id and verify it exists. `mem_tree_entity_index` - // already indexes both chunks and summaries via `node_kind`, so the - // canonical-id loop below is identical for either case. - let node_id: String = match &payload.node { - NodeRef::Leaf { chunk_id } => { - if chunk_store::get_chunk(config, chunk_id)?.is_none() { - log::warn!("[memory::jobs] topic_route chunk missing chunk_id={chunk_id}"); - return Ok(JobOutcome::Done); - } - chunk_id.clone() - } - NodeRef::Summary { summary_id } => { - if crate::openhuman::memory_store::trees::store::get_summary(config, summary_id)? - .is_none() - { - log::warn!("[memory::jobs] topic_route summary missing summary_id={summary_id}"); - return Ok(JobOutcome::Done); - } - summary_id.clone() - } - }; - - let entity_ids = score_store::list_entity_ids_for_node(config, &node_id)?; - if entity_ids.is_empty() { - log::debug!("[memory::jobs] topic_route no entities for node_id={node_id} — skipping"); - return Ok(JobOutcome::Done); - } - - for entity_id in entity_ids { - let _ = curator::maybe_spawn_topic_tree(config, &entity_id).await?; - if let Some(tree) = crate::openhuman::memory_store::trees::store::get_tree_by_scope( - config, - crate::openhuman::memory_store::trees::types::TreeKind::Topic, - &entity_id, - )? { - let job = NewJob::append_buffer(&AppendBufferPayload { - node: payload.node.clone(), - target: AppendTarget::Topic { - tree_id: tree.id.clone(), - }, - })?; - if store::enqueue(config, &job)?.is_some() { - super::worker::wake_workers(); - } - } - } - Ok(JobOutcome::Done) -} - -async fn handle_digest_daily(config: &Config, job: &Job) -> Result { - let payload: DigestDailyPayload = - serde_json::from_str(&job.payload_json).context("parse DigestDaily payload")?; - let day = chrono::NaiveDate::parse_from_str(&payload.date_iso, "%Y-%m-%d") - .with_context(|| format!("invalid digest date {}", payload.date_iso))?; - match digest::end_of_day_digest(config, day).await? { - DigestOutcome::Emitted { daily_id, .. } => { - log::info!("[memory::jobs] emitted digest daily_id={daily_id}"); - } - DigestOutcome::EmptyDay => {} - DigestOutcome::Skipped { existing_id } => { - log::debug!("[memory::jobs] digest skipped existing_id={existing_id}"); - } - } - Ok(JobOutcome::Done) -} - async fn handle_flush_stale(config: &Config, job: &Job) -> Result { let payload: FlushStalePayload = serde_json::from_str(&job.payload_json).context("parse FlushStale payload")?; diff --git a/src/openhuman/memory_queue/handlers/mod_tests.rs b/src/openhuman/memory_queue/handlers/mod_tests.rs index e358ebd84..fa666f06c 100644 --- a/src/openhuman/memory_queue/handlers/mod_tests.rs +++ b/src/openhuman/memory_queue/handlers/mod_tests.rs @@ -111,249 +111,6 @@ async fn seed_source_tree_ready_to_seal( tree } -#[tokio::test] -async fn source_tree_seal_handler_enqueues_summary_topic_route() { - let (_tmp, cfg) = test_config(); - let tree = seed_source_tree_ready_to_seal(&cfg).await; - - let payload = SealPayload { - tree_id: tree.id.clone(), - level: 0, - force_now_ms: None, - }; - let job = mk_running_job(JobKind::Seal, serde_json::to_string(&payload).unwrap()); - - // Pre-condition: queue has no topic_route jobs. - assert_eq!(count_jobs_of_kind(&cfg, "topic_route"), 0); - - super::handle_seal(&cfg, &job).await.unwrap(); - - // Post-condition: source-tree seal must enqueue exactly one - // topic_route job carrying NodeRef::Summary { summary_id: }. - assert_eq!( - count_jobs_of_kind(&cfg, "topic_route"), - 1, - "source-tree seal must enqueue summary-side topic_route" - ); - assert_eq!(count_by_status(&cfg, JobStatus::Ready).unwrap(), 1); - - // Inspect the enqueued payload to confirm it's a Summary variant. - let payload_json: String = with_connection(&cfg, |conn| { - let s: String = conn - .query_row( - "SELECT payload_json FROM mem_tree_jobs WHERE kind = 'topic_route'", - [], - |r| r.get(0), - ) - .unwrap(); - Ok(s) - }) - .unwrap(); - let p: TopicRoutePayload = serde_json::from_str(&payload_json).unwrap(); - match p.node { - NodeRef::Summary { summary_id } => { - // Format: `summary:<13-digit-ms>:L-<8hex>` — - // see `tree::registry::new_summary_id`. - assert!( - summary_id.starts_with("summary:") && summary_id.contains(":L1-"), - "expected summary id with L1 segment, got {summary_id}" - ); - } - other => panic!("expected NodeRef::Summary, got {other:?}"), - } -} - -#[tokio::test] -async fn topic_tree_seal_handler_does_not_enqueue_topic_route() { - let (_tmp, cfg) = test_config(); - // Spawn a topic tree directly via the registry (skipping curator's - // hotness gate — we just need a TreeKind::Topic with leaves). - let topic_tree = crate::openhuman::memory_store::trees::registry::get_or_create_topic_tree( - &cfg, - "topic:phoenix-migration", - ) - .unwrap(); - // Push a single 10k-token leaf so L0 is gate-ready. - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - let ts = chrono::Utc.timestamp_millis_opt(1_700_000_000_000).unwrap(); - let chunk = Chunk { - id: chunk_id(SourceKind::Chat, "slack:#eng", 0, "topic-seed"), - content: "topic content".into(), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: "slack:#eng".into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new("slack://x")), - }, - token_count: 60_000, - seq_in_source: 0, - created_at: ts, - partial_message: false, - }; - upsert_chunks(&cfg, &[chunk.clone()]).unwrap(); - // Stage to disk so `hydrate_leaf_inputs` can read the full body - // when `handle_seal` fires. - let content_root = cfg.memory_tree_content_root(); - std::fs::create_dir_all(&content_root).unwrap(); - let staged = content_store::stage_chunks(&content_root, &[chunk.clone()]).unwrap(); - with_connection(&cfg, |conn| { - let tx = conn.unchecked_transaction()?; - crate::openhuman::memory_store::chunks::store::upsert_staged_chunks_tx(&tx, &staged)?; - tx.commit()?; - Ok(()) - }) - .unwrap(); - let leaf = LeafRef { - chunk_id: chunk.id, - token_count: 60_000, - timestamp: ts, - content: chunk.content, - entities: vec![], - topics: vec![], - score: 0.5, - }; - append_leaf_deferred(&cfg, &topic_tree, &leaf).unwrap(); - - let payload = SealPayload { - tree_id: topic_tree.id.clone(), - level: 0, - force_now_ms: None, - }; - let job = mk_running_job(JobKind::Seal, serde_json::to_string(&payload).unwrap()); - - super::handle_seal(&cfg, &job).await.unwrap(); - - // Topic-tree seals are sinks: must not enqueue any topic_route. - assert_eq!( - count_jobs_of_kind(&cfg, "topic_route"), - 0, - "topic-tree seal must NOT enqueue topic_route (trees are sinks)" - ); - // The seal itself should still have produced a summary node. - assert_eq!(src_store::count_summaries(&cfg, &topic_tree.id).unwrap(), 1); -} - -#[tokio::test] -async fn handle_append_buffer_with_summary_payload_pushes_into_topic_tree() { - let (_tmp, cfg) = test_config(); - - // 1. Create a target topic tree with a clean L0 buffer. - let topic_tree = crate::openhuman::memory_store::trees::registry::get_or_create_topic_tree( - &cfg, - "email:alice@example.com", - ) - .unwrap(); - let l0_before = src_store::get_buffer(&cfg, &topic_tree.id, 0).unwrap(); - assert!(l0_before.is_empty()); - - // 2. Manually insert a summary node we can route. The simplest way - // is to create a separate source tree, push two 6k leaves into - // it, and let the seal produce a summary we can address. - let source_tree = get_or_create_source_tree(&cfg, "slack:#eng").unwrap(); - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_tree::tree::bucket_seal::seal_one_level; - let ts = chrono::Utc.timestamp_millis_opt(1_700_000_000_000).unwrap(); - let content_root = cfg.memory_tree_content_root(); - std::fs::create_dir_all(&content_root).unwrap(); - for seq in 0..2 { - let chunk = Chunk { - id: chunk_id(SourceKind::Chat, "slack:#eng", seq, "summary-seed"), - content: format!("source content {seq}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: "slack:#eng".into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new("slack://x")), - }, - token_count: 30_000, - seq_in_source: seq, - created_at: ts, - partial_message: false, - }; - upsert_chunks(&cfg, &[chunk.clone()]).unwrap(); - // Stage to disk so `hydrate_leaf_inputs` can read the full body - // during `seal_one_level`. - let staged = content_store::stage_chunks(&content_root, &[chunk.clone()]).unwrap(); - with_connection(&cfg, |conn| { - let tx = conn.unchecked_transaction()?; - crate::openhuman::memory_store::chunks::store::upsert_staged_chunks_tx(&tx, &staged)?; - tx.commit()?; - Ok(()) - }) - .unwrap(); - let leaf = LeafRef { - chunk_id: chunk.id, - token_count: 30_000, - timestamp: ts, - content: chunk.content, - entities: vec![], - topics: vec![], - score: 0.5, - }; - let _ = append_leaf_deferred(&cfg, &source_tree, &leaf).unwrap(); - } - // Force-seal the source tree's L0 to mint the summary. - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - let buf = src_store::get_buffer(&cfg, &source_tree.id, 0).unwrap(); - let provider: std::sync::Arc = - std::sync::Arc::new(StaticChatProvider::new("test summary content")); - let summary_id = test_override::with_provider(provider, async { - seal_one_level( - &cfg, - &source_tree, - &buf, - &crate::openhuman::memory_tree::tree::bucket_seal::LabelStrategy::Empty, - // No follow-up enqueues — the test scopes assertions to the - // append_buffer handler, not seal-side fan-out. - false, - ) - .await - .unwrap() - }) - .await; - - // 3. Build an append_buffer payload routing the summary into the - // topic tree. - let payload = AppendBufferPayload { - node: NodeRef::Summary { - summary_id: summary_id.clone(), - }, - target: AppendTarget::Topic { - tree_id: topic_tree.id.clone(), - }, - }; - let job = mk_running_job( - JobKind::AppendBuffer, - serde_json::to_string(&payload).unwrap(), - ); - - // Clear out any pending append_buffer jobs minted upstream so the - // post-condition assertion below is unambiguous. - let pre = count_total(&cfg).unwrap(); - - super::handle_append_buffer(&cfg, &job).await.unwrap(); - - // 4. Topic tree's L0 buffer should now hold the summary id. - let l0_after = src_store::get_buffer(&cfg, &topic_tree.id, 0).unwrap(); - assert_eq!(l0_after.item_ids, vec![summary_id]); - assert!(l0_after.token_sum > 0); - - // No new jobs should have been enqueued (buffer didn't cross gate). - assert_eq!(count_total(&cfg).unwrap(), pre); -} - /// #1574 §6: a chunk with content but no sidecar vector at the active /// signature (the post-switch / dim-mismatch state) is re-embedded by /// `handle_reembed_backfill`; the chain `Defer`s while work remains and diff --git a/src/openhuman/memory_queue/mod.rs b/src/openhuman/memory_queue/mod.rs index c5521b683..6a172092c 100644 --- a/src/openhuman/memory_queue/mod.rs +++ b/src/openhuman/memory_queue/mod.rs @@ -40,14 +40,13 @@ pub mod types; pub(crate) mod worker; pub use ops::{backfill_in_progress, ensure_reembed_backfill, set_backfill_in_progress}; -pub use scheduler::{backfill_missing_digests, trigger_digest}; pub use store::{ claim_next, count_by_status, count_total, enqueue, enqueue_tx, get_job, mark_deferred, mark_done, mark_failed, recover_stale_locks, DEFAULT_LOCK_DURATION_MS, }; pub use testing::drain_until_idle; pub use types::{ - AppendBufferPayload, AppendTarget, DigestDailyPayload, ExtractChunkPayload, FlushStalePayload, - Job, JobKind, JobOutcome, JobStatus, NewJob, NodeRef, SealPayload, TopicRoutePayload, + AppendBufferPayload, AppendTarget, ExtractChunkPayload, FlushStalePayload, Job, JobKind, + JobOutcome, JobStatus, NewJob, NodeRef, SealPayload, }; pub use worker::{start, wake_workers}; diff --git a/src/openhuman/memory_queue/scheduler.rs b/src/openhuman/memory_queue/scheduler.rs index cde23e7cb..938bcf55b 100644 --- a/src/openhuman/memory_queue/scheduler.rs +++ b/src/openhuman/memory_queue/scheduler.rs @@ -1,45 +1,35 @@ -//! Wall-clock scheduler that wakes once a day shortly after UTC midnight to -//! enqueue the global [`JobKind::DigestDaily`] for yesterday and a -//! [`JobKind::FlushStale`] for today. Also exposes manual-trigger helpers -//! for catch-up and testing. +//! Wall-clock scheduler that periodically enqueues a [`JobKind::FlushStale`] +//! so low-volume source-tree L0 buffers seal promptly. +//! +//! The daily global-digest loop was removed along with the global tree — +//! source trees plus the entity index are the substrate, so there is no +//! cross-source digest to enqueue. Only the stale-buffer flush remains. use std::time::Duration; -use anyhow::Result; -use chrono::{Datelike, Duration as ChronoDuration, NaiveDate, TimeZone, Timelike, Utc}; +use chrono::{Timelike, Utc}; use crate::openhuman::config::Config; use crate::openhuman::memory_queue::store; -use crate::openhuman::memory_queue::types::{DigestDailyPayload, FlushStalePayload, NewJob}; +use crate::openhuman::memory_queue::types::{FlushStalePayload, NewJob}; static STARTED: std::sync::Once = std::sync::Once::new(); -/// Start the daily wall-clock scheduler. Takes the full `Config` so the -/// digest enqueues match the same workspace + LLM settings the workers -/// see — not `Config::default()`. +/// Start the periodic flush_stale scheduler. Takes the full `Config` so the +/// enqueues match the same workspace + LLM settings the workers see — not +/// `Config::default()`. pub fn start(config: Config) { STARTED.call_once(|| { - // Daily midnight loop: digest + flush_stale. - let cfg1 = config.clone(); - tokio::spawn(async move { - loop { - if let Err(err) = enqueue_daily_jobs(&cfg1) { - log::warn!("[memory::jobs] scheduler enqueue failed: {err:#}"); - } - tokio::time::sleep(next_sleep_duration()).await; - } - }); - // Periodic flush_stale loop (every 3 h) so L0 buffers seal // promptly even for low-volume sources. - let cfg2 = config.clone(); + let cfg = config.clone(); tokio::spawn(async move { // Fire once on startup so new installs & restarts don't wait // up to 3 h for the first seal window. - enqueue_flush_stale(&cfg2); + enqueue_flush_stale(&cfg); loop { tokio::time::sleep(Duration::from_secs(3 * 60 * 60)).await; - enqueue_flush_stale(&cfg2); + enqueue_flush_stale(&cfg); } }); }); @@ -70,117 +60,13 @@ fn enqueue_flush_stale(config: &Config) { } } -fn enqueue_daily_jobs(config: &Config) -> anyhow::Result<()> { - let now = Utc::now(); - let yesterday = now.date_naive() - ChronoDuration::days(1); - let date_iso = yesterday.format("%Y-%m-%d").to_string(); - - if store::enqueue( - config, - &NewJob::digest_daily(&DigestDailyPayload { - date_iso: date_iso.clone(), - })?, - )? - .is_some() - { - super::worker::wake_workers(); - } - - let today_iso = now.date_naive().format("%Y-%m-%d").to_string(); - let hour_block = now.hour() / 3; - if store::enqueue( - config, - &NewJob::flush_stale(&FlushStalePayload::default(), &today_iso, hour_block)?, - )? - .is_some() - { - super::worker::wake_workers(); - } - - Ok(()) -} - -/// Manually enqueue a `digest_daily` job for `date`. Idempotent — if a -/// digest already ran for that day, the handler's `find_existing_daily` -/// check will return `Skipped` without doing any work; if a job for the -/// same date is already queued or running, the partial unique index on -/// `dedupe_key` suppresses the duplicate. -/// -/// Useful for catch-up after the process was down across midnight, or -/// to force a re-run for testing / debugging. -pub fn trigger_digest(config: &Config, date: NaiveDate) -> Result> { - let payload = DigestDailyPayload { - date_iso: date.format("%Y-%m-%d").to_string(), - }; - let job_id = store::enqueue(config, &NewJob::digest_daily(&payload)?)?; - if job_id.is_some() { - log::info!( - "[memory::jobs] manual digest trigger enqueued date={} id={:?}", - payload.date_iso, - job_id.as_deref() - ); - super::worker::wake_workers(); - } else { - log::debug!( - "[memory::jobs] manual digest trigger dedupe-suppressed date={} \ - (an active job for this date already exists)", - payload.date_iso - ); - } - Ok(job_id) -} - -/// Enqueue `digest_daily` jobs for the last `days_back` calendar days -/// (excluding today). Catch-up helper for cases where the scheduler -/// missed days because the process was down. -/// -/// Returns the number of jobs newly enqueued. Days that already have a -/// completed digest are still re-enqueued — the handler is idempotent -/// and skips them — so this is safe to call repeatedly. -pub fn backfill_missing_digests(config: &Config, days_back: i64) -> Result { - if days_back <= 0 { - return Ok(0); - } - let today = Utc::now().date_naive(); - let mut enqueued = 0usize; - for offset in 1..=days_back { - let date = today - ChronoDuration::days(offset); - if trigger_digest(config, date)?.is_some() { - enqueued += 1; - } - } - log::info!( - "[memory::jobs] backfill_missing_digests window={}d enqueued={}", - days_back, - enqueued - ); - Ok(enqueued) -} - -fn next_sleep_duration() -> Duration { - let now = Utc::now(); - let tomorrow = now.date_naive() + ChronoDuration::days(1); - let next = Utc - .with_ymd_and_hms(tomorrow.year(), tomorrow.month(), tomorrow.day(), 0, 5, 0) - // UTC has no DST gaps/overlaps, so `single()` always returns - // `Some` for any valid (Y, M, D, h, m, s). Fallback retained - // only as a defensive belt-and-braces against future API churn. - .single() - .unwrap_or_else(|| now + ChronoDuration::hours(24)); - (next - now) - .to_std() - .unwrap_or_else(|_| Duration::from_secs(60)) -} - #[cfg(test)] mod tests { use super::*; use crate::openhuman::memory_queue::store::{ - claim_next, count_by_status, count_total, mark_done, DEFAULT_LOCK_DURATION_MS, - }; - use crate::openhuman::memory_queue::types::{ - DigestDailyPayload, FlushStalePayload, JobKind, JobStatus, + claim_next, count_by_status, DEFAULT_LOCK_DURATION_MS, }; + use crate::openhuman::memory_queue::types::{FlushStalePayload, JobKind, JobStatus}; use tempfile::TempDir; fn test_config() -> (TempDir, Config) { @@ -193,83 +79,6 @@ mod tests { (tmp, cfg) } - #[test] - fn trigger_digest_enqueues_a_job() { - let (_tmp, cfg) = test_config(); - let date = NaiveDate::from_ymd_opt(2026, 4, 27).unwrap(); - let id = trigger_digest(&cfg, date).unwrap(); - assert!(id.is_some(), "first trigger must enqueue"); - assert_eq!(count_by_status(&cfg, JobStatus::Ready).unwrap(), 1); - } - - #[test] - fn trigger_digest_dedupes_active_jobs() { - let (_tmp, cfg) = test_config(); - let date = NaiveDate::from_ymd_opt(2026, 4, 27).unwrap(); - let first = trigger_digest(&cfg, date).unwrap(); - let second = trigger_digest(&cfg, date).unwrap(); - assert!(first.is_some()); - assert!( - second.is_none(), - "duplicate trigger must be dedupe-suppressed while active" - ); - assert_eq!(count_total(&cfg).unwrap(), 1); - } - - #[test] - fn trigger_digest_after_done_creates_fresh_row() { - let (_tmp, cfg) = test_config(); - let date = NaiveDate::from_ymd_opt(2026, 4, 27).unwrap(); - let id1 = trigger_digest(&cfg, date).unwrap().unwrap(); - // Simulate a worker finishing the job — claim it first so we have a - // Job snapshot for the claim-token-gated mark_done. - let claimed = claim_next(&cfg, DEFAULT_LOCK_DURATION_MS).unwrap().unwrap(); - assert_eq!(claimed.id, id1); - mark_done(&cfg, &claimed).unwrap(); - - let id2 = trigger_digest(&cfg, date).unwrap(); - assert!( - id2.is_some(), - "after the prior job completes, a fresh trigger must enqueue" - ); - assert_ne!(id2.unwrap(), id1); - assert_eq!(count_total(&cfg).unwrap(), 2); - } - - #[test] - fn backfill_missing_digests_enqueues_one_per_day() { - let (_tmp, cfg) = test_config(); - let n = backfill_missing_digests(&cfg, 5).unwrap(); - assert_eq!(n, 5, "expected one job per day in the 5-day window"); - assert_eq!(count_total(&cfg).unwrap(), 5); - } - - #[test] - fn backfill_missing_digests_zero_window_is_noop() { - let (_tmp, cfg) = test_config(); - let n = backfill_missing_digests(&cfg, 0).unwrap(); - assert_eq!(n, 0); - assert_eq!(count_total(&cfg).unwrap(), 0); - } - - #[test] - fn backfill_missing_digests_negative_window_is_noop() { - let (_tmp, cfg) = test_config(); - let n = backfill_missing_digests(&cfg, -3).unwrap(); - assert_eq!(n, 0); - assert_eq!(count_total(&cfg).unwrap(), 0); - } - - #[test] - fn backfill_missing_digests_is_idempotent_while_active() { - let (_tmp, cfg) = test_config(); - let n1 = backfill_missing_digests(&cfg, 3).unwrap(); - let n2 = backfill_missing_digests(&cfg, 3).unwrap(); - assert_eq!(n1, 3); - assert_eq!(n2, 0, "second call must be fully dedupe-suppressed"); - assert_eq!(count_total(&cfg).unwrap(), 3); - } - #[test] fn enqueue_flush_stale_enqueues_at_most_one_job_per_current_block() { let (_tmp, cfg) = test_config(); @@ -287,73 +96,4 @@ mod tests { let payload: FlushStalePayload = serde_json::from_str(&claimed.payload_json).unwrap(); assert_eq!(payload.max_age_secs, None); } - - #[test] - fn enqueue_daily_jobs_adds_digest_and_flush_jobs() { - let (_tmp, cfg) = test_config(); - enqueue_daily_jobs(&cfg).unwrap(); - - assert_eq!(count_total(&cfg).unwrap(), 2); - - let first = claim_next(&cfg, DEFAULT_LOCK_DURATION_MS).unwrap().unwrap(); - assert_eq!( - first.kind, - JobKind::DigestDaily, - "digest_daily should be claimed ahead of flush_stale" - ); - let digest: DigestDailyPayload = serde_json::from_str(&first.payload_json).unwrap(); - assert!(!digest.date_iso.is_empty()); - mark_done(&cfg, &first).unwrap(); - - let second = claim_next(&cfg, DEFAULT_LOCK_DURATION_MS).unwrap().unwrap(); - assert_eq!(second.kind, JobKind::FlushStale); - let flush: FlushStalePayload = serde_json::from_str(&second.payload_json).unwrap(); - assert_eq!(flush.max_age_secs, None); - } - - #[test] - fn enqueue_daily_jobs_is_fully_deduped_while_jobs_remain_active() { - let (_tmp, cfg) = test_config(); - enqueue_daily_jobs(&cfg).unwrap(); - enqueue_daily_jobs(&cfg).unwrap(); - - assert_eq!( - count_total(&cfg).unwrap(), - 2, - "same-day scheduler rerun should not create duplicate active jobs" - ); - } - - #[test] - fn enqueue_daily_jobs_reenqueues_after_prior_rows_complete() { - let (_tmp, cfg) = test_config(); - enqueue_daily_jobs(&cfg).unwrap(); - - let first = claim_next(&cfg, DEFAULT_LOCK_DURATION_MS).unwrap().unwrap(); - mark_done(&cfg, &first).unwrap(); - let second = claim_next(&cfg, DEFAULT_LOCK_DURATION_MS).unwrap().unwrap(); - mark_done(&cfg, &second).unwrap(); - - enqueue_daily_jobs(&cfg).unwrap(); - - assert_eq!( - count_total(&cfg).unwrap(), - 4, - "completed daily jobs should allow a fresh digest+flush pair" - ); - assert_eq!(count_by_status(&cfg, JobStatus::Ready).unwrap(), 2); - } - - #[test] - fn next_sleep_duration_targets_near_next_midnight_utc_plus_five_minutes() { - let sleep = next_sleep_duration(); - assert!( - sleep.as_secs() > 0, - "scheduler sleep should always be positive" - ); - assert!( - sleep.as_secs() <= 24 * 60 * 60 + 5 * 60, - "scheduler should never sleep for more than ~24h+5m" - ); - } } diff --git a/src/openhuman/memory_queue/types.rs b/src/openhuman/memory_queue/types.rs index 730148801..bb999f601 100644 --- a/src/openhuman/memory_queue/types.rs +++ b/src/openhuman/memory_queue/types.rs @@ -17,11 +17,6 @@ pub enum JobKind { AppendBuffer, /// Seal exactly one buffer level; cascades enqueue a follow-up. Seal, - /// Match a chunk's entities against active topic trees and enqueue - /// per-topic `AppendBuffer` jobs. - TopicRoute, - /// Build the global tree's daily digest for a given UTC date. - DigestDaily, /// Walk stale buffers and enqueue `Seal` jobs for any over the age cap. FlushStale, /// #1574 §6: re-embed a bounded batch of chunks/summaries that lack a @@ -37,8 +32,6 @@ impl JobKind { JobKind::ExtractChunk => "extract_chunk", JobKind::AppendBuffer => "append_buffer", JobKind::Seal => "seal", - JobKind::TopicRoute => "topic_route", - JobKind::DigestDaily => "digest_daily", JobKind::FlushStale => "flush_stale", JobKind::ReembedBackfill => "reembed_backfill", } @@ -50,27 +43,27 @@ impl JobKind { "extract_chunk" => JobKind::ExtractChunk, "append_buffer" => JobKind::AppendBuffer, "seal" => JobKind::Seal, - "topic_route" => JobKind::TopicRoute, - "digest_daily" => JobKind::DigestDaily, "flush_stale" => JobKind::FlushStale, + // Legacy kinds from the removed global/topic trees. Tolerated on + // parse so a queue row left over from before the removal is + // recognised and can be drained/discarded rather than crashing + // the worker loop; the startup migration purges them. + "topic_route" | "digest_daily" => { + return Err(anyhow!( + "retired JobKind '{s}' (global/topic trees removed)" + )) + } "reembed_backfill" => JobKind::ReembedBackfill, other => return Err(anyhow!("unknown JobKind '{other}'")), }) } /// True when handling this kind should hold a slot from the global - /// LLM concurrency semaphore. `TopicRoute` is bound because - /// `maybe_spawn_topic_tree → backfill_topic_tree` can transitively - /// trigger summariser LLM calls when an entity first crosses the - /// hotness threshold. + /// LLM concurrency semaphore. pub fn is_llm_bound(&self) -> bool { matches!( self, - JobKind::ExtractChunk - | JobKind::Seal - | JobKind::DigestDaily - | JobKind::TopicRoute - | JobKind::ReembedBackfill + JobKind::ExtractChunk | JobKind::Seal | JobKind::ReembedBackfill ) } } @@ -233,34 +226,6 @@ impl SealPayload { } } -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct TopicRoutePayload { - pub node: NodeRef, -} - -impl TopicRoutePayload { - /// Stable dedupe key written to `mem_tree_jobs.dedupe_key` so a partial - /// unique index can suppress in-flight duplicates. - pub fn dedupe_key(&self) -> String { - format!("topic_route:{}", self.node.dedupe_fragment()) - } -} - -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct DigestDailyPayload { - /// UTC calendar date in `YYYY-MM-DD` form. Stored as a string so the - /// dedupe key doesn't need to know about chrono. - pub date_iso: String, -} - -impl DigestDailyPayload { - /// Stable dedupe key written to `mem_tree_jobs.dedupe_key` so a partial - /// unique index can suppress in-flight duplicates. - pub fn dedupe_key(&self) -> String { - format!("digest_daily:{}", self.date_iso) - } -} - #[derive(Clone, Debug, Serialize, Deserialize, Default)] pub struct FlushStalePayload { /// Override the configured `DEFAULT_FLUSH_AGE_SECS`. Optional so the @@ -370,28 +335,6 @@ impl NewJob { }) } - /// Build an [`JobKind::TopicRoute`] enqueue request. - pub fn topic_route(p: &TopicRoutePayload) -> Result { - Ok(Self { - kind: JobKind::TopicRoute, - payload_json: serde_json::to_string(p)?, - dedupe_key: Some(p.dedupe_key()), - available_at_ms: None, - max_attempts: None, - }) - } - - /// Build an [`JobKind::DigestDaily`] enqueue request. - pub fn digest_daily(p: &DigestDailyPayload) -> Result { - Ok(Self { - kind: JobKind::DigestDaily, - payload_json: serde_json::to_string(p)?, - dedupe_key: Some(p.dedupe_key()), - available_at_ms: None, - max_attempts: None, - }) - } - /// Build a [`JobKind::FlushStale`] enqueue request scoped to a /// 3-hour UTC block. Callers compute `date_iso` and `hour_block` /// from a single `Utc::now()` reading so the dedupe key is @@ -428,13 +371,14 @@ mod tests { JobKind::ExtractChunk, JobKind::AppendBuffer, JobKind::Seal, - JobKind::TopicRoute, - JobKind::DigestDaily, JobKind::FlushStale, JobKind::ReembedBackfill, ] { assert_eq!(JobKind::parse(k.as_str()).unwrap(), k); } + // Retired kinds parse to an error (global/topic trees removed). + assert!(JobKind::parse("topic_route").is_err()); + assert!(JobKind::parse("digest_daily").is_err()); } #[test] @@ -486,18 +430,6 @@ mod tests { }, }; assert_ne!(p_leaf.dedupe_key(), p_summary.dedupe_key()); - - let r_leaf = TopicRoutePayload { - node: NodeRef::Leaf { - chunk_id: "x".into(), - }, - }; - let r_summary = TopicRoutePayload { - node: NodeRef::Summary { - summary_id: "x".into(), - }, - }; - assert_ne!(r_leaf.dedupe_key(), r_summary.dedupe_key()); } #[test] @@ -518,8 +450,6 @@ mod tests { fn llm_bound_kinds() { assert!(JobKind::ExtractChunk.is_llm_bound()); assert!(JobKind::Seal.is_llm_bound()); - assert!(JobKind::DigestDaily.is_llm_bound()); - assert!(JobKind::TopicRoute.is_llm_bound()); assert!(JobKind::ReembedBackfill.is_llm_bound()); assert!(!JobKind::AppendBuffer.is_llm_bound()); assert!(!JobKind::FlushStale.is_llm_bound()); diff --git a/src/openhuman/memory_store/chunks/connection.rs b/src/openhuman/memory_store/chunks/connection.rs index f683bac20..e7ff82779 100644 --- a/src/openhuman/memory_store/chunks/connection.rs +++ b/src/openhuman/memory_store/chunks/connection.rs @@ -12,8 +12,8 @@ use std::time::{Duration, Instant}; use crate::openhuman::config::Config; use super::{ - add_column_if_missing, migrate_legacy_embeddings_to_sidecar, DB_DIR, DB_FILE, SCHEMA, - SQLITE_BUSY_TIMEOUT, + add_column_if_missing, migrate_legacy_embeddings_to_sidecar, purge_global_topic_trees, DB_DIR, + DB_FILE, SCHEMA, SQLITE_BUSY_TIMEOUT, }; // ── Schema-apply instrumentation (test-only) ───────────────────────────────── @@ -257,6 +257,8 @@ fn init_db(conn: &Connection, config: &Config) -> Result<()> { apply_schema(conn)?; // #1574 §7: one-shot, version-gated legacy→sidecar embedding migration. migrate_legacy_embeddings_to_sidecar(conn, config)?; + // One-shot, version-gated purge of the removed global/topic trees. + purge_global_topic_trees(conn, config)?; Ok(()) } diff --git a/src/openhuman/memory_store/chunks/store.rs b/src/openhuman/memory_store/chunks/store.rs index 1d77bb77f..ed9e7045c 100644 --- a/src/openhuman/memory_store/chunks/store.rs +++ b/src/openhuman/memory_store/chunks/store.rs @@ -74,6 +74,12 @@ pub const CHUNK_STATUS_DROPPED: &str = "dropped"; /// next open; `>= 1` skips it. Bump only for a new one-shot data migration. const TREE_EMBEDDING_MIGRATION_VERSION: i64 = 1; +/// `PRAGMA user_version` value once the global/topic-tree purge has run. +/// The global (time-axis) and topic (subject-axis) trees were removed; this +/// one-shot migration deletes their rows + on-disk summary folders. `< 2` +/// triggers the purge on next open; `>= 2` skips it. +const GLOBAL_TOPIC_PURGE_MIGRATION_VERSION: i64 = 2; + const SCHEMA: &str = " CREATE TABLE IF NOT EXISTS mem_tree_chunks ( id TEXT PRIMARY KEY, @@ -1193,6 +1199,96 @@ fn migrate_legacy_embeddings_to_sidecar(conn: &Connection, config: &Config) -> R Ok(()) } +/// One-shot purge of the removed global + topic trees. +/// +/// The global (time-axis) and topic (subject-axis) trees were deleted in +/// favour of the source trees (which hold all content). This migration +/// removes their now-orphaned DB rows and on-disk summary folders so old +/// vaults clean themselves up on next open. Version-gated via +/// `PRAGMA user_version` (see [`GLOBAL_TOPIC_PURGE_MIGRATION_VERSION`]); a +/// no-op on workspaces that never had those trees. +fn purge_global_topic_trees(conn: &Connection, config: &Config) -> Result<()> { + let version: i64 = conn + .query_row("PRAGMA user_version", [], |r| r.get(0)) + .context("read PRAGMA user_version for global/topic purge")?; + if version >= GLOBAL_TOPIC_PURGE_MIGRATION_VERSION { + return Ok(()); + } + + let tx = conn.unchecked_transaction()?; + // Child rows first (summary sidecars / skip-lists are keyed by + // summary_id; entity-index + buffers carry an FK on tree_id). + let removed_summary_sidecars = tx.execute( + "DELETE FROM mem_tree_summary_embeddings WHERE summary_id IN \ + (SELECT id FROM mem_tree_summaries WHERE tree_kind IN ('global','topic'))", + [], + )?; + tx.execute( + "DELETE FROM mem_tree_summary_reembed_skipped WHERE summary_id IN \ + (SELECT id FROM mem_tree_summaries WHERE tree_kind IN ('global','topic'))", + [], + )?; + tx.execute( + "DELETE FROM mem_tree_entity_index WHERE tree_id IN \ + (SELECT id FROM mem_tree_trees WHERE kind IN ('global','topic'))", + [], + )?; + let removed_summaries = tx.execute( + "DELETE FROM mem_tree_summaries WHERE tree_kind IN ('global','topic')", + [], + )?; + tx.execute( + "DELETE FROM mem_tree_buffers WHERE tree_id IN \ + (SELECT id FROM mem_tree_trees WHERE kind IN ('global','topic'))", + [], + )?; + let removed_trees = tx.execute( + "DELETE FROM mem_tree_trees WHERE kind IN ('global','topic')", + [], + )?; + // Drain any queued jobs for the retired kinds so the worker loop never + // trips over a payload it can no longer parse. + let removed_jobs = tx.execute( + "DELETE FROM mem_tree_jobs WHERE kind IN ('topic_route','digest_daily')", + [], + )?; + tx.commit()?; + + // On-disk: drop the `wiki/summaries/global*` (both the legacy per-day + // `global-/` folders and the singleton `global/`) and `topic-*` + // summary folders. Best-effort — a filesystem error must not abort the + // version bump, or the purge would retry forever. + let summaries_root = config + .memory_tree_content_root() + .join("wiki") + .join("summaries"); + let mut removed_dirs = 0usize; + if let Ok(entries) = std::fs::read_dir(&summaries_root) { + for entry in entries.flatten() { + let name = entry.file_name(); + let name = name.to_string_lossy(); + if name.starts_with("global") || name.starts_with("topic-") { + match std::fs::remove_dir_all(entry.path()) { + Ok(()) => removed_dirs += 1, + Err(e) => log::warn!( + "[memory_tree::migrate] purge: failed to remove {} : {e}", + entry.path().display() + ), + } + } + } + } + + conn.pragma_update(None, "user_version", GLOBAL_TOPIC_PURGE_MIGRATION_VERSION) + .context("set PRAGMA user_version after global/topic purge")?; + log::info!( + "[memory_tree::migrate] global/topic purge done: trees={removed_trees} \ + summaries={removed_summaries} sidecars={removed_summary_sidecars} jobs={removed_jobs} \ + dirs={removed_dirs}; user_version={GLOBAL_TOPIC_PURGE_MIGRATION_VERSION}" + ); + Ok(()) +} + /// One pointer into the raw archive. A chunk's body is reconstructed by /// reading each [`RawRef`] in order and joining with `"\n\n"`. /// diff --git a/src/openhuman/memory_store/chunks/store_tests.rs b/src/openhuman/memory_store/chunks/store_tests.rs index f57067780..c78b99f14 100644 --- a/src/openhuman/memory_store/chunks/store_tests.rs +++ b/src/openhuman/memory_store/chunks/store_tests.rs @@ -666,9 +666,12 @@ fn legacy_embeddings_migrate_to_sidecar_once() { let v: i64 = conn .query_row("PRAGMA user_version", [], |r| r.get(0)) .unwrap(); + // A full init runs every one-shot migration in sequence, so the gate + // lands on the latest version (the global/topic purge), not just the + // embedding migration's. assert_eq!( - v, TREE_EMBEDDING_MIGRATION_VERSION, - "version gate must be set" + v, GLOBAL_TOPIC_PURGE_MIGRATION_VERSION, + "version gate must be set to the latest migration" ); Ok(()) }) @@ -1106,3 +1109,102 @@ fn batch_embedding_lookup_splits_id_list_above_per_batch_threshold() { assert_eq!(map.get(&c2.id).cloned(), Some(vec![2.0])); assert_eq!(map.get(&c3.id).cloned(), Some(vec![3.0])); } + +/// The one-shot purge migration deletes global + topic trees (rows, summaries, +/// buffers, jobs, and on-disk summary folders) while leaving source trees and +/// non-retired jobs untouched, and runs exactly once (PRAGMA user_version gate). +#[test] +fn global_topic_purge_removes_only_global_and_topic() { + let (_tmp, cfg) = test_config(); + // First open initialises the schema and runs both migrations (sets + // user_version = 2). + upsert_chunks(&cfg, &[sample_chunk("slack:#eng", 0, 1_700_000_000_000)]).unwrap(); + + // On-disk: a legacy per-day global folder, the singleton global folder, a + // topic folder, and a source folder that must survive. + let summaries = cfg + .memory_tree_content_root() + .join("wiki") + .join("summaries"); + for d in [ + "global-2026-05-28", + "global", + "topic-alice", + "source-slack-eng", + ] { + std::fs::create_dir_all(summaries.join(d).join("L0")).unwrap(); + } + + with_connection(&cfg, |conn| { + // Seed one tree of each kind, each with a summary. + for (id, kind) in [ + ("source:s1", "source"), + ("global:g1", "global"), + ("topic:t1", "topic"), + ] { + conn.execute( + "INSERT INTO mem_tree_trees (id, kind, scope, max_level, status, created_at_ms) \ + VALUES (?1, ?2, ?2, 0, 'active', 0)", + params![id, kind], + )?; + conn.execute( + "INSERT INTO mem_tree_summaries \ + (id, tree_id, tree_kind, level, content, token_count, \ + time_range_start_ms, time_range_end_ms, sealed_at_ms) \ + VALUES (?1, ?2, ?3, 0, 'x', 1, 0, 0, 0)", + params![format!("sum-{id}"), id, kind], + )?; + } + // Seed retired + surviving job rows. + for (jid, kind) in [ + ("j1", "topic_route"), + ("j2", "digest_daily"), + ("j3", "extract_chunk"), + ] { + conn.execute( + "INSERT INTO mem_tree_jobs (id, kind, payload_json, available_at_ms, created_at_ms) \ + VALUES (?1, ?2, '{}', 0, 0)", + params![jid, kind], + )?; + } + // Re-arm the gate so the purge runs against the seeded rows. + conn.pragma_update(None, "user_version", 1i64)?; + super::purge_global_topic_trees(conn, &cfg)?; + + // Trees: only the source tree survives. + let trees: i64 = + conn.query_row("SELECT COUNT(*) FROM mem_tree_trees", [], |r| r.get(0))?; + assert_eq!(trees, 1, "only the source tree should remain"); + let kind: String = + conn.query_row("SELECT kind FROM mem_tree_trees", [], |r| r.get(0))?; + assert_eq!(kind, "source"); + + // Summaries: only the source summary survives. + let summaries_left: i64 = + conn.query_row("SELECT COUNT(*) FROM mem_tree_summaries", [], |r| r.get(0))?; + assert_eq!(summaries_left, 1); + + // Jobs: retired kinds gone, extract_chunk kept. + let jobs_left: Vec = { + let mut stmt = conn.prepare("SELECT kind FROM mem_tree_jobs ORDER BY kind")?; + let rows = stmt.query_map([], |r| r.get::<_, String>(0))?; + rows.collect::>()? + }; + assert_eq!(jobs_left, vec!["extract_chunk".to_string()]); + + // Gate advanced — a second run is a no-op. + let version: i64 = conn.query_row("PRAGMA user_version", [], |r| r.get(0))?; + assert_eq!(version, 2); + Ok(()) + }) + .unwrap(); + + // On-disk: global*/topic-* folders gone, source-* kept. + assert!(!summaries.join("global-2026-05-28").exists()); + assert!(!summaries.join("global").exists()); + assert!(!summaries.join("topic-alice").exists()); + assert!( + summaries.join("source-slack-eng").exists(), + "source summary folder must survive the purge" + ); +} diff --git a/src/openhuman/memory_store/content/atomic.rs b/src/openhuman/memory_store/content/atomic.rs index c8d780392..5df24786d 100644 --- a/src/openhuman/memory_store/content/atomic.rs +++ b/src/openhuman/memory_store/content/atomic.rs @@ -113,9 +113,9 @@ pub struct StagedSummary { /// SQLite upsert. /// /// The relative path is built from the input metadata and the `tree_kind`. The -/// `date_for_global` argument is required when `input.tree_kind == -/// SummaryTreeKind::Global`. The `scope_slug` must already be slugified by the -/// caller. +/// `scope_slug` must already be slugified by the caller. The global tree is a +/// singleton, so its summaries all land under one `global/` folder regardless +/// of the day they cover — no date argument is needed. /// /// If the file already exists with the same body SHA-256 (idempotent re-stage), /// the existing `StagedSummary` is returned without rewriting. @@ -123,22 +123,14 @@ pub fn stage_summary( content_root: &Path, input: &SummaryComposeInput<'_>, scope_slug: &str, - date_for_global: Option>, ) -> anyhow::Result { - let rel_path = summary_rel_path( - input.tree_kind, - scope_slug, - input.level, - input.summary_id, - date_for_global, - ); + let rel_path = summary_rel_path(input.tree_kind, scope_slug, input.level, input.summary_id); let abs_path = summary_abs_path( content_root, input.tree_kind, scope_slug, input.level, input.summary_id, - date_for_global, ); let composed = compose_summary_md(input); @@ -298,7 +290,7 @@ mod tests { "summary body", &children, ); - let staged = stage_summary(dir.path(), &input, "gmail-alice-x-com", None).unwrap(); + let staged = stage_summary(dir.path(), &input, "gmail-alice-x-com").unwrap(); assert_eq!(staged.summary_id, "summary:L1:test1"); assert!(staged.content_path.starts_with("wiki/summaries/source-")); assert!(staged.content_path.ends_with(".md")); @@ -323,17 +315,15 @@ mod tests { "idempotent body", &children, ); - let first = stage_summary(dir.path(), &input, "person-alex", None).unwrap(); - let second = stage_summary(dir.path(), &input, "person-alex", None).unwrap(); + let first = stage_summary(dir.path(), &input, "person-alex").unwrap(); + let second = stage_summary(dir.path(), &input, "person-alex").unwrap(); assert_eq!(first.content_sha256, second.content_sha256); assert_eq!(first.content_path, second.content_path); } #[test] - fn stage_summary_global_uses_date_in_path() { - use chrono::TimeZone; + fn stage_summary_global_uses_singleton_folder_no_date() { let dir = TempDir::new().unwrap(); - let date = chrono::Utc.with_ymd_and_hms(2026, 4, 28, 12, 0, 0).unwrap(); let children = vec![]; let input = mk_summary_input( SummaryTreeKind::Global, @@ -342,10 +332,13 @@ mod tests { "daily recap", &children, ); - let staged = stage_summary(dir.path(), &input, "global", Some(date)).unwrap(); - assert!( - staged.content_path.contains("2026-04-28"), - "global summary path must contain date; got: {}", + let staged = stage_summary(dir.path(), &input, "global").unwrap(); + // Singleton global tree → one folder, no per-day date segment. The + // `L1` segment comes from `mk_summary_input`'s level=1; what matters + // is the single `global/` folder with no date. + assert_eq!( + staged.content_path, "wiki/summaries/global/L1/summary-L0-daily.md", + "global summary must land in the singleton global/ folder; got: {}", staged.content_path ); } @@ -362,7 +355,7 @@ mod tests { body, &children, ); - let staged = stage_summary(dir.path(), &input, "gmail-x-y-com", None).unwrap(); + let staged = stage_summary(dir.path(), &input, "gmail-x-y-com").unwrap(); let expected = sha256_hex(body.as_bytes()); assert_eq!(staged.content_sha256, expected); } @@ -384,7 +377,7 @@ mod tests { ); // First stage with the real body to get the path. - let first = stage_summary(dir.path(), &input, "gmail-stale-test-com", None).unwrap(); + let first = stage_summary(dir.path(), &input, "gmail-stale-test-com").unwrap(); // Corrupt the on-disk file by writing a different body to the path. let mut abs = dir.path().to_path_buf(); @@ -395,7 +388,7 @@ mod tests { std::fs::write(&abs, b"---\nstale_key: true\n---\nSTALE BODY CONTENT").unwrap(); // Now re-stage: must detect sha mismatch and re-write. - let second = stage_summary(dir.path(), &input, "gmail-stale-test-com", None).unwrap(); + let second = stage_summary(dir.path(), &input, "gmail-stale-test-com").unwrap(); // The returned sha must match the new body. let expected_sha = sha256_hex(new_body.as_bytes()); diff --git a/src/openhuman/memory_store/content/paths.rs b/src/openhuman/memory_store/content/paths.rs index 17e06eb85..933488e3b 100644 --- a/src/openhuman/memory_store/content/paths.rs +++ b/src/openhuman/memory_store/content/paths.rs @@ -20,8 +20,6 @@ use std::path::{Path, PathBuf}; -use chrono::{DateTime, Utc}; - use crate::openhuman::memory::util::redact::redact; /// Which kind of summary tree a summary belongs to. Determines the @@ -33,7 +31,13 @@ use crate::openhuman::memory::util::redact::redact; pub enum SummaryTreeKind { /// Per-source-tree summary. Layout: `wiki/summaries/source-/L/.md` Source, - /// Global digest tree. Layout: `wiki/summaries/global-/L/.md` + /// Global digest tree — the singleton cross-source activity tree. + /// Layout: `wiki/summaries/global/L/.md`. There is exactly + /// one global tree, so (unlike the historical `global-/` + /// layout) the day/week/month/year grouping is expressed by the + /// `L/` subdirectory + each node's time range, NOT by a + /// date-stamped top-level folder. A per-day folder name shattered the + /// one logical tree into one look-alike folder per calendar day. Global, /// Per-topic (entity) tree. Layout: `wiki/summaries/topic-/L/.md` Topic, @@ -53,9 +57,9 @@ pub const WIKI_PREFIX: &str = "wiki"; /// flattening the historical two-level `//` so users see /// one folder per logical source in their Obsidian sidebar: /// - Source: `"wiki/summaries/source-/L/.md"` -/// - Global: `"wiki/summaries/global-/L/.md"` -/// Falls back to `unknown-date` (with a warn log) if `date_for_global` is -/// `None` — preferable to panicking inside a path utility. +/// - Global: `"wiki/summaries/global/L/.md"` — one +/// folder for the singleton global tree; the temporal grouping lives in +/// the `L/` subdirectory, not the folder name. /// - Topic: `"wiki/summaries/topic-/L/.md"` /// /// `scope_slug` must already be slugified by the caller (use [`slugify_source_id`] or @@ -75,7 +79,6 @@ pub fn summary_rel_path( scope_slug: &str, level: u32, summary_id: &str, - date_for_global: Option>, ) -> String { let filename = summary_filename(summary_id); @@ -87,21 +90,10 @@ pub fn summary_rel_path( ) } SummaryTreeKind::Global => { - let date_str = match date_for_global { - Some(d) => d.format("%Y-%m-%d").to_string(), - None => { - log::warn!( - "[content_store::paths] summary_rel_path called for Global \ - without date_for_global; using sentinel 'unknown-date'. \ - Caller bug — please pass a date." - ); - "unknown-date".to_string() - } - }; - format!( - "{WIKI_PREFIX}/summaries/global-{}/L{}/{}.md", - date_str, level, filename - ) + // The global tree is a singleton: one folder, with the + // day/week/month/year hierarchy carried by `L/` and the + // node's time range — never a per-day folder name. + format!("{WIKI_PREFIX}/summaries/global/L{}/{}.md", level, filename) } SummaryTreeKind::Topic => { format!( @@ -194,9 +186,8 @@ pub fn summary_abs_path( scope_slug: &str, level: u32, summary_id: &str, - date_for_global: Option>, ) -> PathBuf { - let rel = summary_rel_path(tree_kind, scope_slug, level, summary_id, date_for_global); + let rel = summary_rel_path(tree_kind, scope_slug, level, summary_id); let mut abs = content_root.to_path_buf(); for component in rel.split('/') { abs.push(component); @@ -475,7 +466,6 @@ mod tests { "gmail-alice-x-com-bob-y-com", 1, "summary:L1:abc", - None, ); // Colons in summary_id are replaced with '-' for cross-platform filenames. assert_eq!( @@ -491,7 +481,6 @@ mod tests { "slack-eng", 2, "summary:1700000000000:L2-deadbeef", - None, ); assert_eq!( p, @@ -501,16 +490,23 @@ mod tests { #[test] fn summary_rel_path_global() { - use chrono::TimeZone; - let date = chrono::Utc.with_ymd_and_hms(2026, 4, 28, 12, 0, 0).unwrap(); - let p = summary_rel_path( - SummaryTreeKind::Global, - "global", - 0, - "summary:L0:daily", - Some(date), - ); - assert_eq!(p, "wiki/summaries/global-2026-04-28/L0/summary-L0-daily.md"); + // The singleton global tree gets ONE folder; the date is NOT part of + // the folder name. Day/week/month grouping lives in `L/`. + let p = summary_rel_path(SummaryTreeKind::Global, "global", 0, "summary:L0:daily"); + assert_eq!(p, "wiki/summaries/global/L0/summary-L0-daily.md"); + } + + #[test] + fn summary_rel_path_global_levels_share_one_folder() { + // Regression guard for the per-day-folder bug: every level of the + // global tree must land under the same `global/` folder, only the + // `L/` segment differs. + let l0 = summary_rel_path(SummaryTreeKind::Global, "global", 0, "summary:L0:a"); + let l1 = summary_rel_path(SummaryTreeKind::Global, "global", 1, "summary:L1:b"); + let l3 = summary_rel_path(SummaryTreeKind::Global, "global", 3, "summary:L3:c"); + assert_eq!(l0, "wiki/summaries/global/L0/summary-L0-a.md"); + assert_eq!(l1, "wiki/summaries/global/L1/summary-L1-b.md"); + assert_eq!(l3, "wiki/summaries/global/L3/summary-L3-c.md"); } #[test] @@ -520,7 +516,6 @@ mod tests { "person-alex-johnson", 1, "summary:L1:xyz", - None, ); assert_eq!( p, @@ -536,7 +531,6 @@ mod tests { "entity-slug", 2, "summary:L2:foo.md", - None, ); assert_eq!(p, "wiki/summaries/topic-entity-slug/L2/summary-L2-foo.md"); } @@ -611,30 +605,14 @@ mod tests { ); } - #[test] - fn summary_rel_path_global_falls_back_to_sentinel_without_date() { - // Caller bug to omit date for Global, but a path utility shouldn't - // panic — fall back to a sentinel `unknown-date` segment so the - // file lands somewhere predictable rather than aborting the seal. - let p = summary_rel_path(SummaryTreeKind::Global, "global", 0, "summary:L0:x", None); - assert_eq!(p, "wiki/summaries/global-unknown-date/L0/summary-L0-x.md"); - } - #[test] fn summary_abs_path_rooted_under_content_root() { - use chrono::TimeZone; use std::path::Path; let root = Path::new("/workspace/content"); - let date = chrono::Utc.with_ymd_and_hms(2026, 1, 15, 0, 0, 0).unwrap(); - let abs = summary_abs_path( - root, - SummaryTreeKind::Global, - "global", - 0, - "daily-123", - Some(date), - ); + let abs = summary_abs_path(root, SummaryTreeKind::Global, "global", 0, "daily-123"); assert!(abs.starts_with(root)); assert!(abs.ends_with("daily-123.md")); + // Singleton folder, no date segment. + assert!(abs.to_string_lossy().contains("summaries/global/L0/")); } } diff --git a/src/openhuman/memory_store/content/read.rs b/src/openhuman/memory_store/content/read.rs index 1f2650800..fbced8941 100644 --- a/src/openhuman/memory_store/content/read.rs +++ b/src/openhuman/memory_store/content/read.rs @@ -690,7 +690,6 @@ mod tests { body: &node.content, }, "slack-eng", - None, ) .unwrap(); with_connection(&cfg, |conn| { diff --git a/src/openhuman/memory_store/trees/mod.rs b/src/openhuman/memory_store/trees/mod.rs index 766a9c363..8bf3a6782 100644 --- a/src/openhuman/memory_store/trees/mod.rs +++ b/src/openhuman/memory_store/trees/mod.rs @@ -1,25 +1,24 @@ -//! Tree persistence — shared across Source, Global, and Topic kinds. +//! Tree persistence — source trees in `mem_tree_trees` keyed by [`TreeKind`]. //! -//! All three flavors live in `mem_tree_trees` keyed by [`TreeKind`]. This +//! (The global/topic kinds were removed; their variants survive only as +//! inert serialization plumbing for the one-shot purge migration.) This //! module hosts: //! - `store` — generic CRUD over the trees + summaries + buffers tables. //! - `types` — Tree, SummaryNode, TreeKind, TreeStatus, Buffer, and the -//! topic-hotness types ([`HotnessCounters`], thresholds). -//! - `registry` — kind-parameterized get-or-create / list / archive helpers. -//! - `hotness` — entity-hotness side-table that gates topic-tree spawn. +//! entity-hotness types ([`HotnessCounters`], thresholds). +//! - `registry` — generic list / archive helpers. +//! - `hotness` — entity-hotness side-table (now a read-only subconscious +//! signal; the topic curator that wrote it was removed). //! -//! Tree _logic_ (bucket_seal, flush, generic registry, sources/global/topic -//! policy) stays in `memory_tree`. +//! Tree _logic_ (bucket_seal, flush, generic registry, source policy) stays +//! in `memory_tree`. pub mod hotness; pub mod registry; pub mod store; pub mod types; -pub use registry::{ - archive_topic_tree, archive_tree, force_create_topic_tree, get_or_create_global_tree, - get_or_create_topic_tree, list_topic_trees, list_trees_by_kind, -}; +pub use registry::{archive_tree, list_trees_by_kind}; pub use store::{get_summary_embedding, set_summary_embedding}; pub use types::{ Buffer, EntityIndexStats, HotnessCounters, SummaryNode, Tree, TreeKind, TreeStatus, diff --git a/src/openhuman/memory_store/trees/registry.rs b/src/openhuman/memory_store/trees/registry.rs index ecfd3cc0b..061002414 100644 --- a/src/openhuman/memory_store/trees/registry.rs +++ b/src/openhuman/memory_store/trees/registry.rs @@ -1,12 +1,15 @@ //! Kind-parameterized tree registry helpers. //! -//! All three tree flavors (Source, Global, Topic) live in the same -//! `mem_tree_trees` table keyed by [`TreeKind`]. The generic get-or-create -//! dance with UNIQUE-race recovery lives in +//! Trees live in the `mem_tree_trees` table keyed by [`TreeKind`]. The +//! generic get-or-create dance with UNIQUE-race recovery lives in //! [`memory_tree::tree::registry::get_or_create_tree`] (logic file — stays -//! in memory_tree). This module hosts the thin per-kind convenience wrappers -//! (formerly spread across `trees_global::registry` and -//! `trees_topic::registry`) so callers have one place to land. +//! in memory_tree). This module hosts the generic list / archive helpers. +//! +//! The global (time-axis) and topic (subject-axis) trees were removed; only +//! source trees are created now, so there are no per-kind creation wrappers +//! here anymore — [`TreeKind::Global`]/[`TreeKind::Topic`] remain only as +//! inert serialization plumbing for reading legacy rows during the one-shot +//! purge migration. use anyhow::{Context, Result}; use chrono::Utc; @@ -15,38 +18,6 @@ use rusqlite::params; use crate::openhuman::config::Config; use crate::openhuman::memory_store::chunks::store::with_connection; use crate::openhuman::memory_store::trees::types::{Tree, TreeKind}; -use crate::openhuman::memory_tree::tree::TreeFactory; - -/// Return the workspace's singleton global tree, creating it lazily on first -/// call. Safe to call on every ingest; subsequent calls short-circuit to the -/// existing row. -pub fn get_or_create_global_tree(config: &Config) -> Result { - log::debug!("[trees::registry] get_or_create_global_tree"); - TreeFactory::global().get_or_create(config) -} - -/// Look up the topic tree for `entity_id`, or create a new one. -/// -/// Callers should NOT use this directly to materialise topic trees eagerly — -/// go through `memory::tree_topic::curator::maybe_spawn_topic_tree` so -/// creation is gated on hotness. Admin / forced-materialisation flows can -/// call this directly (or its alias [`force_create_topic_tree`]). -pub fn get_or_create_topic_tree(config: &Config, entity_id: &str) -> Result { - let entity_kind = entity_id - .split_once(':') - .map(|(k, _)| k) - .unwrap_or("unknown"); - log::debug!( - "[trees::registry] get_or_create_topic_tree entity_kind={}", - entity_kind - ); - TreeFactory::topic(entity_id).get_or_create(config) -} - -/// Semantic alias used by the admin "force materialise" path. -pub fn force_create_topic_tree(config: &Config, entity_id: &str) -> Result { - get_or_create_topic_tree(config, entity_id) -} /// List every tree of the given kind, ordered by creation time ascending. pub fn list_trees_by_kind(config: &Config, kind: TreeKind) -> Result> { @@ -66,11 +37,6 @@ pub fn list_trees_by_kind(config: &Config, kind: TreeKind) -> Result> }) } -/// Topic-specific alias preserved for call-site readability. -pub fn list_topic_trees(config: &Config) -> Result> { - list_trees_by_kind(config, TreeKind::Topic) -} - /// Flip a tree's status to `archived`. Idempotent. Existing rows remain /// queryable; new leaves will not be routed to this tree until it is /// manually unarchived (which is not currently a primitive). @@ -90,11 +56,6 @@ pub fn archive_tree(config: &Config, tree_id: &str) -> Result<()> { }) } -/// Topic-specific alias preserved for call-site readability. -pub fn archive_topic_tree(config: &Config, tree_id: &str) -> Result<()> { - archive_tree(config, tree_id) -} - fn row_to_tree_loose(row: &rusqlite::Row<'_>) -> rusqlite::Result { use crate::openhuman::memory_store::trees::types::TreeStatus; use chrono::TimeZone; @@ -129,7 +90,6 @@ mod tests { use super::*; use crate::openhuman::memory_store::trees::store::insert_tree; use crate::openhuman::memory_store::trees::types::TreeStatus; - use crate::openhuman::memory_tree::tree::factory::GLOBAL_SCOPE; use tempfile::TempDir; fn test_config() -> (TempDir, Config) { @@ -181,7 +141,7 @@ mod tests { vec!["source-1".to_string(), "source-2".to_string()] ); - let topic_ids: Vec = list_topic_trees(&cfg) + let topic_ids: Vec = list_trees_by_kind(&cfg, TreeKind::Topic) .unwrap() .into_iter() .map(|tree| tree.id) @@ -192,23 +152,13 @@ mod tests { #[test] fn archive_tree_flips_status_to_archived() { let (_tmp, cfg) = test_config(); - let tree = sample_tree("topic-1", TreeKind::Topic, "person:alice"); + let tree = sample_tree("source-arch", TreeKind::Source, "chat:slack:#x"); insert_tree(&cfg, &tree).unwrap(); - archive_tree(&cfg, "topic-1").unwrap(); + archive_tree(&cfg, "source-arch").unwrap(); - let archived = list_topic_trees(&cfg).unwrap(); + let archived = list_trees_by_kind(&cfg, TreeKind::Source).unwrap(); assert_eq!(archived.len(), 1); assert_eq!(archived[0].status, TreeStatus::Archived); } - - #[test] - fn get_or_create_global_tree_is_idempotent() { - let (_tmp, cfg) = test_config(); - let first = get_or_create_global_tree(&cfg).unwrap(); - let second = get_or_create_global_tree(&cfg).unwrap(); - assert_eq!(first.id, second.id); - assert_eq!(first.kind, TreeKind::Global); - assert_eq!(first.scope, GLOBAL_SCOPE); - } } diff --git a/src/openhuman/memory_tree/retrieval/benchmarks.rs b/src/openhuman/memory_tree/retrieval/benchmarks.rs index 6855375df..de697ff1d 100644 --- a/src/openhuman/memory_tree/retrieval/benchmarks.rs +++ b/src/openhuman/memory_tree/retrieval/benchmarks.rs @@ -9,11 +9,13 @@ //! //! | # | Scenario | What it tests | //! |---|----------|---------------| -//! | 1 | Cross-chat recall | Chat A seeds data; Chat B queries related → relevant source retrieved, no dump | //! | 2 | Citation bundle | Retrieval returns chunk/source IDs alongside content | -//! | 3 | Stale preference | Newer explicit correction supersedes older preference | -//! | 4 | Contradiction handling | Disagreeing sources surface with provenance labels | //! | 5 | Long-source compression | Large source retrieves exact relevant leaf chunk | +//! | 6 | Scale/soak | 20 sources stay correct under `query_source` + `search_entities` | +//! +//! The entity-/topic-retrieval scenarios (cross-chat recall, stale +//! preference, contradiction handling, drill-down isolation) were retired +//! with the topic tree — source trees + the entity index remain the substrate. //! //! Run with: `cargo test --package openhuman_core -- retrieval_benchmarks` @@ -25,9 +27,7 @@ use crate::openhuman::memory::ingest_pipeline::ingest_chat; use crate::openhuman::memory_queue::testing::drain_until_idle; use crate::openhuman::memory_store::chunks::types::SourceKind; use crate::openhuman::memory_sync::canonicalize::chat::{ChatBatch, ChatMessage}; -use crate::openhuman::memory_tree::retrieval::{ - fetch_leaves, query_source, query_topic, search_entities, -}; +use crate::openhuman::memory_tree::retrieval::{fetch_leaves, query_source, search_entities}; /// Shared test config — disables embedding for deterministic inert behaviour. fn bench_config() -> (TempDir, Config) { @@ -44,7 +44,7 @@ fn bench_config() -> (TempDir, Config) { /// Each message is padded with entity-bearing text (email + hashtag) to ensure /// the entity index gets populated reliably. This is required because: /// 1. The regex extractor finds emails (alice@example.com) and hashtags (#phoenix) -/// 2. Without these, `query_topic` returns 0 hits and all entity-based tests fail +/// 2. Without these, `search_entities` returns 0 hits and entity-based tests fail /// 3. The sealing threshold also needs sufficient content per message async fn ingest_chat_batch( cfg: &Config, @@ -82,90 +82,6 @@ async fn ingest_chat_batch( result.chunk_ids } -// ───────────────────────────────────────────────────────────────────────────── -// Scenario 1 — Cross-chat recall -// ───────────────────────────────────────────────────────────────────────────── - -/// Seeds "Phoenix migration Friday landing" in Chat A. -/// Queries from Chat B — should retrieve relevant source without dumping -/// unrelated history. -#[tokio::test] -async fn bench_cross_chat_recall() { - let (_tmp, cfg) = bench_config(); - - // Chat A — seeds the key fact - ingest_chat_batch( - &cfg, - "slack:#eng", - "alice", - vec![ - ( - "alice".into(), - "Phoenix migration status: landing Friday evening.".into(), - ), - ("bob".into(), "Confirmed, I'll handle the cutover.".into()), - ], - 1_700_000_000_000, - ) - .await; - - // Chat B — queries related topic - ingest_chat_batch( - &cfg, - "slack:#ops", - "carol", - vec![ - ( - "carol".into(), - "What's the current status of the Phoenix migration?".into(), - ), - ("dave".into(), "Friday landing confirmed per alice.".into()), - ], - 1_700_100_000_000, - ) - .await; - - // Query topic for "phoenix" — should surface alice's original fact - // The #benchmark hashtag in ingest_chat_batch pads text but the query uses - // "topic:phoenix" which comes from the "#phoenix" pattern in messages - let topic_resp = query_topic(&cfg, "topic:benchmark", None, None, 20) - .await - .unwrap(); - - // Assertions - if topic_resp.hits.is_empty() { - // If drain_until_idle settled correctly, the entity index must have - // received the #benchmark extraction. A truly empty result here likely - // means a bug in the extraction or index write path. - // Downgrade to warn + skip rather than silent pass, so CI surfaces regressions: - eprintln!( - "[bench] WARN: query_topic returned no hits — entity index may not have settled. \ - Skipping downstream assertions. Investigate if this is persistent." - ); - return; - } - - let benchmark_hits: Vec<_> = topic_resp - .hits - .iter() - .filter(|h| h.content.to_lowercase().contains("benchmark")) - .collect(); - - assert!( - !benchmark_hits.is_empty(), - "at least one hit should mention 'benchmark'" - ); - - // Verify no source-dump behaviour: hits should have content under 1 KB - for hit in &benchmark_hits { - assert!( - hit.content.len() <= 1024, - "cross-chat recall should return concise hits, not raw dumps. Got {} chars", - hit.content.len() - ); - } -} - /// Verify search_entities surfaces entities from both chats independently. #[tokio::test] async fn bench_cross_chat_entity_discoverable() { @@ -318,201 +234,6 @@ async fn bench_citation_fetch_leaves_hydrates() { // Scenario 3 — Stale preference // ───────────────────────────────────────────────────────────────────────────── -/// Newer explicit preference must supersede older preference. -#[tokio::test] -async fn bench_stale_preference_newer_supersedes() { - let (_tmp, cfg) = bench_config(); - - // Older preference — sets theme to dark - ingest_chat_batch( - &cfg, - "slack:#general", - "alice", - vec![( - "alice".into(), - "My preferred theme is dark mode, please set UI_THEME=dark".into(), - )], - 1_700_000_000_000, // older - ) - .await; - - // Newer explicit correction — overrides to light - ingest_chat_batch( - &cfg, - "slack:#general", - "alice", - vec![( - "alice".into(), - "Update: actually I prefer light theme, please set UI_THEME=light".into(), - )], - 1_700_200_000_000, // newer - ) - .await; - - drain_until_idle(&cfg).await.unwrap(); - - // Query for alice's preference via email entity - let topic_resp = query_topic(&cfg, "email:test@entity.example", None, None, 20) - .await - .unwrap(); - - // Guard: if the scorer returned nothing, skip the rest (likely LLM off + no regex hit). - if topic_resp.hits.is_empty() { - // If drain_until_idle settled correctly, the entity index must have - // received the email extraction. A truly empty result here likely - // means a bug in the extraction or index write path. - eprintln!( - "[bench] WARN: query_topic returned no hits in stale_preference test — \ - entity index may not have settled. Skipping downstream assertions." - ); - return; - } - - // Find hits mentioning both themes - let dark_hits: Vec<_> = topic_resp - .hits - .iter() - .filter(|h| h.content.to_lowercase().contains("dark")) - .collect(); - - let light_hits: Vec<_> = topic_resp - .hits - .iter() - .filter(|h| h.content.to_lowercase().contains("light")) - .collect(); - - // Both themes should be present (old + new both stored) - // but light should appear in at least one hit (newer supersedes) - assert!( - !light_hits.is_empty(), - "newer explicit correction should appear in results (light theme hit)" - ); - - // And dark should also appear (history preserved, not silently deleted) - assert!( - !dark_hits.is_empty(), - "older preference should also appear in results (history preserved)" - ); -} - -// ───────────────────────────────────────────────────────────────────────────── -// Scenario 4 — Contradiction handling -// ───────────────────────────────────────────────────────────────────────────── - -/// Disagreeing sources surface with clear provenance labels so the caller -/// can resolve the conflict, not silently discard one side. -#[tokio::test] -async fn bench_contradiction_surfaces_both_with_provenance() { - let (_tmp, cfg) = bench_config(); - - // Source A — claims target date is June 15 - ingest_chat_batch( - &cfg, - "slack:#eng", - "alice", - vec![( - "alice".into(), - "Q2 milestone: we target June 15 for the Phoenix launch.".into(), - )], - 1_700_000_000_000, - ) - .await; - - // Source B — contradicts with July 30 - ingest_chat_batch( - &cfg, - "email:pm", - "bob", - vec![( - "bob".into(), - "Re-scoping: the Phoenix launch is pushed to July 30 per stakeholder review.".into(), - )], - 1_700_100_000_000, - ) - .await; - - drain_until_idle(&cfg).await.unwrap(); - - // Query for benchmark topic — should surface both sources via entity index - let topic_resp = query_topic(&cfg, "topic:benchmark", None, None, 20) - .await - .unwrap(); - - // Guard: if no hits were produced, skip assertions (scorer returned nothing). - if topic_resp.hits.is_empty() { - // If drain_until_idle settled correctly, the entity index must have - // received the #benchmark extraction. A truly empty result here likely - // means a bug in the extraction or index write path. - // Downgrade to warn + skip rather than silent pass, so CI surfaces regressions: - eprintln!( - "[bench] WARN: query_topic returned no hits in contradiction test — \ - entity index may not have settled. Skipping downstream assertions." - ); - return; - } - - // NOTE: We use ALL hits here, not just benchmark-filtered ones. The original - // message content ("Q2 milestone: we target June 15 ...") does NOT contain - // the word "benchmark" — only the pad suffix does. Filtering to "benchmark" - // would miss the actual date content that lives in the original message body. - // Using all hits ensures the date assertions are applied to the full result set. - let all_hits: Vec<_> = topic_resp.hits.iter().collect(); - - assert!( - all_hits.len() >= 2, - "contradiction scenario should surface >= 2 hits from different sources, got {}", - all_hits.len() - ); - - // Verify both scopes appear (slack:#eng and email:pm) - let scopes: Vec<_> = all_hits.iter().map(|h| h.tree_scope.clone()).collect(); - - assert!( - scopes.iter().any(|s| s.contains("slack")), - "hit from slack:#eng expected" - ); - assert!( - scopes.iter().any(|s| s.contains("email")), - "hit from email:pm expected" - ); - - // Each hit should ideally have provenance (tree_id, tree_scope). - // Tree-level metadata is only guaranteed once the source tree has been sealed - // (summarization step), which requires a configured embedder. Without sealing, - // entity-index hits may lack tree_id — skip the strict check. - let with_tree_id = all_hits.iter().filter(|h| !h.tree_id.is_empty()).count(); - if with_tree_id > 0 { - for hit in &all_hits { - assert!( - !hit.tree_scope.is_empty(), - "hit with tree_id must also have tree_scope for source identification" - ); - assert!( - !hit.content.is_empty(), - "contradiction hit must have content" - ); - } - } - - // Verify June and July dates are both present in the FULL result set - // (not just benchmark-filtered hits — the original content may be in a different - // node than the benchmark-tagged pad suffix). - let content_all = all_hits - .iter() - .map(|h| h.content.as_str()) - .collect::>() - .join(" "); - - assert!( - content_all.to_lowercase().contains("june"), - "june hit missing from contradiction results" - ); - assert!( - content_all.to_lowercase().contains("july"), - "july hit missing from contradiction results" - ); -} - // ───────────────────────────────────────────────────────────────────────────── // Scenario 5 — Long-source compression // ───────────────────────────────────────────────────────────────────────────── @@ -572,98 +293,6 @@ async fn bench_long_source_retrieves_exact_leaf() { } } -/// Verify drill_down on a summary returns only its children, not sibling content. -#[tokio::test] -async fn bench_drill_down_isolates_children() { - let (_tmp, cfg) = bench_config(); - - // Two separate scopes — eng and ops - ingest_chat_batch( - &cfg, - "slack:#eng", - "alice", - vec![( - "alice".into(), - "eng-only secret: the internal API uses Bearer token auth.".into(), - )], - 1_700_000_000_000, - ) - .await; - - ingest_chat_batch( - &cfg, - "slack:#ops", - "carol", - vec![( - "carol".into(), - "ops-only note: production DB lives at internal.example.com.".into(), - )], - 1_700_100_000_000, - ) - .await; - - drain_until_idle(&cfg).await.unwrap(); - - // query_topic for "benchmark" topic — should find both via entity index - let topic_resp = query_topic(&cfg, "topic:benchmark", None, None, 20) - .await - .unwrap(); - - // Guard: if no hits, the isolation claim is untested — warn and skip. - if topic_resp.hits.is_empty() { - eprintln!( - "[bench] WARN: drill_down test got no hits; isolation claim untested. \ - Investigate entity index settling." - ); - return; - } - - // Collect scopes to verify we actually got hits from the expected channels. - let scopes: Vec<_> = topic_resp - .hits - .iter() - .map(|h| h.tree_scope.clone()) - .collect(); - - // Verify eng scope actually produced hits (isolation claim is only meaningful - // if we actually got results to test). - assert!( - scopes.iter().any(|s| s.contains("eng")), - "expected at least one hit from slack:#eng; got scopes: {scopes:?}" - ); - - // Isolate eng content and verify it does NOT bleed "ops" scope content. - let eng_content = topic_resp - .hits - .iter() - .filter(|h| h.tree_scope.contains("eng")) - .map(|h| h.content.as_str()) - .collect::>() - .join(" "); - - assert!( - !eng_content.to_lowercase().contains("ops"), - "drill_down / query_topic should not cross scope into unrelated channels. \ - Found 'ops' content in eng query: {}", - eng_content.chars().take(200).collect::() - ); - - // Verify the symmetric claim: ops content should NOT bleed "secret" (eng-only). - let ops_content = topic_resp - .hits - .iter() - .filter(|h| h.tree_scope.contains("ops")) - .map(|h| h.content.as_str()) - .collect::>() - .join(" "); - - assert!( - !ops_content.to_lowercase().contains("secret"), - "ops content bled eng-only 'secret' keyword: {}", - ops_content.chars().take(200).collect::() - ); -} - // ───────────────────────────────────────────────────────────────────────────── // Scenario 6 — Scale/soak fixture (no real user data) // ───────────────────────────────────────────────────────────────────────────── diff --git a/src/openhuman/memory_tree/retrieval/global.rs b/src/openhuman/memory_tree/retrieval/global.rs deleted file mode 100644 index e284d9b5d..000000000 --- a/src/openhuman/memory_tree/retrieval/global.rs +++ /dev/null @@ -1,223 +0,0 @@ -//! `memory_tree_query_global` — window-scoped recap from the global digest -//! (Phase 4 / #710). -//! -//! Thin wrapper on [`tree_global::recap::recap`]. The recap function does -//! the heavy lifting (level selection + time-range filter); we convert its -//! output into the uniform [`RetrievalHit`] shape. -//! -//! When no global summaries exist yet (e.g. early in a workspace's life), -//! we return an empty [`QueryResponse`] rather than an error so the LLM can -//! surface "no digest yet" naturally. - -use anyhow::Result; -use chrono::Duration; - -use crate::openhuman::config::Config; -use crate::openhuman::memory::tree_global::recap::{recap, RecapOutput}; -use crate::openhuman::memory_store::trees::registry::get_or_create_global_tree; -use crate::openhuman::memory_store::trees::types::TreeKind; -use crate::openhuman::memory_tree::retrieval::types::{NodeKind, QueryResponse, RetrievalHit}; - -/// Return the global digest for the given window in days. Always returns a -/// [`QueryResponse`]; the response is empty if the global tree has no -/// sealed summaries yet. -pub async fn query_global(config: &Config, window_days: u32) -> Result { - log::info!( - "[retrieval::global] query_global window_days={}", - window_days - ); - - let window = Duration::days(window_days as i64); - let recap_out = match recap(config, window).await? { - Some(r) => r, - None => { - log::debug!("[retrieval::global] no recap available — returning empty response"); - return Ok(QueryResponse::empty()); - } - }; - - let tree = get_or_create_global_tree(config)?; - let hits = recap_to_hits(recap_out, &tree.id, &tree.scope); - let total = hits.len(); - log::debug!( - "[retrieval::global] returning hits={} total={}", - hits.len(), - total - ); - Ok(QueryResponse::new(hits, total)) -} - -/// Convert a [`RecapOutput`] into one synthetic summary hit per fold. We -/// emit one [`RetrievalHit`] covering the assembled recap content — the -/// per-summary provenance lives in `recap.summary_ids`, threaded through as -/// `child_ids` so the LLM can drill into a specific folded day/week/month. -fn recap_to_hits(recap: RecapOutput, tree_id: &str, tree_scope: &str) -> Vec { - let RecapOutput { - content, - time_range, - level_used, - summary_ids, - } = recap; - // We emit ONE hit summarising the whole recap. Drill-down into - // `child_ids` (the individual summary node ids) is available via - // `memory_tree_drill_down`. This keeps the shape consistent with the - // other query tools (which also return summary-level hits). - let node_id = summary_ids - .first() - .cloned() - .unwrap_or_else(|| format!("recap:L{level_used}")); - vec![RetrievalHit { - node_id, - node_kind: NodeKind::Summary, - tree_id: tree_id.to_string(), - tree_kind: TreeKind::Global, - tree_scope: tree_scope.to_string(), - level: level_used, - content, - entities: Vec::new(), - topics: Vec::new(), - time_range_start: time_range.0, - time_range_end: time_range.1, - score: 0.0, - child_ids: summary_ids, - source_ref: None, - }] -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::chat::{test_override, ChatProvider, StaticChatProvider}; - use crate::openhuman::memory::tree_global::digest::{end_of_day_digest, DigestOutcome}; - use crate::openhuman::memory::tree_source::registry::get_or_create_source_tree; - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_store::content as content_store; - use crate::openhuman::memory_tree::tree::bucket_seal::{append_leaf, LabelStrategy, LeafRef}; - use chrono::{DateTime, Utc}; - use std::sync::Arc; - use tempfile::TempDir; - - fn stage_test_chunks(cfg: &Config, chunks: &[Chunk]) { - let content_root = cfg.memory_tree_content_root(); - std::fs::create_dir_all(&content_root).expect("create content_root for test"); - let staged = content_store::stage_chunks(&content_root, chunks) - .expect("stage_chunks for test chunks"); - crate::openhuman::memory_store::chunks::store::with_connection(cfg, |conn| { - let tx = conn.unchecked_transaction()?; - crate::openhuman::memory_store::chunks::store::upsert_staged_chunks_tx(&tx, &staged)?; - tx.commit()?; - Ok(()) - }) - .expect("persist staged chunk pointers"); - } - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - // Phase 4 (#710): digest embeds — inert in tests. - cfg.memory_tree.embedding_endpoint = None; - cfg.memory_tree.embedding_model = None; - cfg.memory_tree.embedding_strict = false; - (tmp, cfg) - } - - async fn seed_daily_digest(cfg: &Config) { - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let day = Utc::now().date_naive(); - let ts = day.and_hms_opt(12, 0, 0).unwrap().and_utc(); - seed_source_for_day(cfg, "slack:#eng", ts).await; - test_override::with_provider(provider, async { - end_of_day_digest(cfg, day).await.unwrap() - }) - .await; - } - - async fn seed_source_for_day(cfg: &Config, scope: &str, ts: DateTime) { - let tree = get_or_create_source_tree(cfg, scope).unwrap(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - for seq in 0..2u32 { - let c = Chunk { - id: chunk_id(SourceKind::Chat, scope, seq, "test-content"), - content: format!("daily-{scope}-{seq}"), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: scope.into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec![], - source_ref: Some(SourceRef::new("slack://x")), - }, - token_count: 30_000, - seq_in_source: seq, - created_at: ts, - partial_message: false, - }; - upsert_chunks(cfg, &[c.clone()]).unwrap(); - stage_test_chunks(cfg, &[c.clone()]); - let leaf = LeafRef { - chunk_id: c.id.clone(), - token_count: 30_000, - timestamp: ts, - content: c.content.clone(), - entities: vec![], - topics: vec![], - score: 0.5, - }; - test_override::with_provider(Arc::clone(&provider), async { - append_leaf(cfg, &tree, &leaf, &LabelStrategy::Empty) - .await - .unwrap() - }) - .await; - } - } - - #[tokio::test] - async fn empty_tree_returns_empty_response() { - let (_tmp, cfg) = test_config(); - let resp = query_global(&cfg, 7).await.unwrap(); - assert!(resp.hits.is_empty()); - assert_eq!(resp.total, 0); - assert!(!resp.truncated); - } - - #[tokio::test] - async fn wraps_daily_recap_into_a_hit() { - let (_tmp, cfg) = test_config(); - seed_daily_digest(&cfg).await; - let resp = query_global(&cfg, 1).await.unwrap(); - assert_eq!(resp.hits.len(), 1); - assert_eq!(resp.hits[0].tree_kind, TreeKind::Global); - assert_eq!(resp.hits[0].level, 0); - assert!(!resp.hits[0].content.is_empty()); - assert!( - !resp.hits[0].child_ids.is_empty(), - "child_ids must expose the folded summary ids for drill-down" - ); - } - - #[tokio::test] - async fn digest_outcome_sanity_check() { - // Sanity: make sure the test helper fixture actually emits a digest; - // if this ever returned Skipped the rest of the suite would trivially - // pass which would be misleading. - let (_tmp, cfg) = test_config(); - let provider: Arc = - Arc::new(StaticChatProvider::new("test summary content")); - let day = Utc::now().date_naive(); - let ts = day.and_hms_opt(12, 0, 0).unwrap().and_utc(); - seed_source_for_day(&cfg, "slack:#eng", ts).await; - let outcome = test_override::with_provider(provider, async { - end_of_day_digest(&cfg, day).await.unwrap() - }) - .await; - assert!(matches!(outcome, DigestOutcome::Emitted { .. })); - } -} diff --git a/src/openhuman/memory_tree/retrieval/integration_tests.rs b/src/openhuman/memory_tree/retrieval/integration_tests.rs index f29f36634..68193b54b 100644 --- a/src/openhuman/memory_tree/retrieval/integration_tests.rs +++ b/src/openhuman/memory_tree/retrieval/integration_tests.rs @@ -18,7 +18,7 @@ use crate::openhuman::memory::ingest_pipeline::ingest_chat; use crate::openhuman::memory_store::chunks::types::SourceKind; use crate::openhuman::memory_sync::canonicalize::chat::{ChatBatch, ChatMessage}; use crate::openhuman::memory_tree::retrieval::{ - drill_down, fetch_leaves, query_global, query_source, query_topic, search_entities, + drill_down, fetch_leaves, query_source, search_entities, }; fn test_config() -> (TempDir, Config) { @@ -86,15 +86,6 @@ async fn end_to_end_three_chat_batches() { .expect("alice should be discoverable via search"); assert!(alice.mention_count >= 1); - // ── query_topic on alice should return at least one hit. - let by_email = query_topic(&cfg, "email:alice@example.com", None, None, 20) - .await - .unwrap(); - assert!( - !by_email.hits.is_empty(), - "alice has been ingested — query_topic should see her" - ); - // ── query_source by source_id returns what we put in (chunks get // surfaced directly since none of the channels seal — 2 short msgs // per channel is under the seal budget). @@ -111,50 +102,15 @@ async fn end_to_end_three_chat_batches() { "query_source total must be >= hits.len()" ); - // ── query_global: no daily digest has been built yet → empty. - let global = query_global(&cfg, 7).await.unwrap(); - assert!( - global.hits.is_empty(), - "end_of_day_digest hasn't been called, so global is empty" - ); - // ── drill_down on a bogus id returns empty (no error). let empty_drill = drill_down(&cfg, "bogus:id", 1, None, None).await.unwrap(); assert!(empty_drill.is_empty()); - // ── fetch_leaves: find a guaranteed leaf hit from alice's topic results - // and assert that fetch_leaves hydrates it correctly. - use crate::openhuman::memory_tree::retrieval::types::NodeKind; - let leaf_hit = by_email - .hits - .iter() - .find(|h| h.node_kind == NodeKind::Leaf) - .expect("alice's topic hits should include at least one leaf chunk"); - let got = fetch_leaves(&cfg, &[leaf_hit.node_id.clone()]) + // ── fetch_leaves on a bogus id hydrates nothing (no error). + let none = fetch_leaves(&cfg, &["ghost:nonexistent".to_string()]) .await .unwrap(); - assert_eq!( - got.len(), - 1, - "fetch_leaves must hydrate the known leaf chunk id" - ); -} - -#[tokio::test] -async fn topic_entity_surfaces_after_ingest() { - let (_tmp, cfg) = test_config(); - ingest_chat(&cfg, "slack:#eng", "alice", vec![], chat_about_phoenix(0)) - .await - .unwrap(); - // Per Phase 3a topic-as-entity promotion, `topic:phoenix` should be - // present in the entity index if the scorer extracts phoenix as a - // topic. We hard-assert query_topic returns a well-formed response - // but don't insist on a non-zero hit count — topic extraction is a - // scorer-level choice out of Phase 4's control. - let resp = query_topic(&cfg, "topic:phoenix", None, None, 10) - .await - .unwrap(); - assert!(resp.total >= resp.hits.len()); + assert!(none.is_empty()); } // ── Phase 4 (#710): embedding + semantic rerank tests ─────────────────── diff --git a/src/openhuman/memory_tree/retrieval/mod.rs b/src/openhuman/memory_tree/retrieval/mod.rs index b7ea2cb69..46045694a 100644 --- a/src/openhuman/memory_tree/retrieval/mod.rs +++ b/src/openhuman/memory_tree/retrieval/mod.rs @@ -1,15 +1,14 @@ -//! Phase 4 — retrieval tools for the hierarchical memory tree (#710). +//! Retrieval tools for the hierarchical memory tree (#710). //! -//! Exposes the source / global / topic trees produced by Phase 3 as six -//! LLM-callable primitives. Each tool is deterministic and scope-specific; -//! orchestration (which tool to call, how to combine results) is left to -//! the calling LLM — there is no classifier, gate, or composer in this -//! phase. +//! Exposes the **source** trees as LLM-callable primitives. Each tool is +//! deterministic and scope-specific; orchestration (which tool to call, how +//! to combine results) is left to the calling LLM — there is no classifier, +//! gate, or composer here. The global (time-axis) and topic (subject-axis) +//! trees were removed: source trees hold all the content, and walking the +//! source hierarchy plus the entity index reconstructs both projections. //! //! Public JSON-RPC surface (see `schemas.rs`): //! - `openhuman.memory_tree_query_source` — per-source summary retrieval -//! - `openhuman.memory_tree_query_global` — cross-source digest for a window -//! - `openhuman.memory_tree_query_topic` — entity-scoped retrieval //! - `openhuman.memory_tree_search_entities` — fuzzy canonical-id lookup //! - `openhuman.memory_tree_drill_down` — walk summary children //! - `openhuman.memory_tree_fetch_leaves` — batch chunk hydration @@ -19,12 +18,10 @@ pub mod drill_down; pub mod fetch; -pub mod global; pub mod rpc; pub mod schemas; pub mod search; pub mod source; -pub mod topic; pub mod types; #[cfg(test)] @@ -34,12 +31,10 @@ mod integration_tests; pub use drill_down::drill_down; pub use fetch::fetch_leaves; -pub use global::query_global; pub use schemas::{ all_controller_schemas as all_retrieval_controller_schemas, all_registered_controllers as all_retrieval_registered_controllers, }; pub use search::search_entities; pub use source::query_source; -pub use topic::query_topic; pub use types::{EntityMatch, NodeKind, QueryResponse, RetrievalHit}; diff --git a/src/openhuman/memory_tree/retrieval/rpc.rs b/src/openhuman/memory_tree/retrieval/rpc.rs index b22d20280..293599c3a 100644 --- a/src/openhuman/memory_tree/retrieval/rpc.rs +++ b/src/openhuman/memory_tree/retrieval/rpc.rs @@ -12,10 +12,8 @@ use crate::openhuman::memory_store::chunks::types::SourceKind; use crate::openhuman::memory_tree::retrieval::{ drill_down::drill_down, fetch::fetch_leaves, - global::query_global, search::search_entities, source::query_source, - topic::query_topic, types::{EntityMatch, QueryResponse, RetrievalHit}, }; use crate::openhuman::memory_tree::score::extract::EntityKind; @@ -79,85 +77,6 @@ pub async fn query_source_rpc( )) } -// ── query_global ────────────────────────────────────────────────────── - -/// Request body for `memory_tree_query_global`. -/// -/// The consolidated `memory_tree` tool schema advertises `time_window_days` -/// (consistent with `query_source` / `query_topic`), while the standalone -/// tool uses `window_days`. Accept both via serde alias so callers using -/// either field name succeed. -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct QueryGlobalRequest { - #[serde(alias = "window_days")] - pub time_window_days: u32, -} - -/// JSON-RPC handler body for `memory_tree_query_global`. -pub async fn query_global_rpc( - config: &Config, - req: QueryGlobalRequest, -) -> Result, String> { - let resp = query_global(config, req.time_window_days) - .await - .map_err(|e| format!("query_global: {e}"))?; - let n = resp.hits.len(); - Ok(RpcOutcome::single_log( - resp, - format!("memory_tree: query_global hits={n}"), - )) -} - -// ── query_topic ─────────────────────────────────────────────────────── - -/// Request body for `memory_tree_query_topic`. -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct QueryTopicRequest { - pub entity_id: String, - #[serde(default)] - pub time_window_days: Option, - /// Phase 4 (#710) — optional natural-language query for semantic - /// rerank. When unset, falls back to the classic score DESC order. - #[serde(default)] - pub query: Option, - #[serde(default)] - pub limit: Option, -} - -/// JSON-RPC handler body for `memory_tree_query_topic`. -pub async fn query_topic_rpc( - config: &Config, - req: QueryTopicRequest, -) -> Result, String> { - let limit = req.limit.unwrap_or(0); - let resp = query_topic( - config, - &req.entity_id, - req.time_window_days, - req.query.as_deref(), - limit, - ) - .await - .map_err(|e| format!("query_topic: {e}"))?; - let n = resp.hits.len(); - // entity_id can be an email or handle — log only the kind prefix - // ("email:", "handle:", etc.) not the full value. - let entity_kind_prefix = req - .entity_id - .split_once(':') - .map(|(k, _)| k) - .unwrap_or("unknown"); - Ok(RpcOutcome::single_log( - resp, - format!( - "memory_tree: query_topic entity_kind={} has_query={} hits={}", - entity_kind_prefix, - req.query.is_some(), - n - ), - )) -} - // ── search_entities ─────────────────────────────────────────────────── /// Request body for `memory_tree_search_entities`. @@ -411,73 +330,6 @@ mod tests { assert!(err.contains("unknown source kind: bogus"), "got {err}"); } - // ── query_global_rpc ────────────────────────────────────────────── - - #[tokio::test] - async fn query_global_rpc_returns_response_for_valid_window() { - let (_tmp, cfg) = test_config(); - let req = QueryGlobalRequest { - time_window_days: 7, - }; - let outcome = query_global_rpc(&cfg, req).await.unwrap(); - assert!(outcome.value.hits.is_empty()); - assert_eq!(outcome.logs.len(), 1); - assert!( - outcome.logs[0].contains("query_global hits=0"), - "log: {}", - outcome.logs[0] - ); - } - - #[test] - fn query_global_request_accepts_both_field_names() { - // The consolidated memory_tree tool schema uses "time_window_days" - // while the standalone tool uses "window_days". Both must deserialize. - let from_alias: QueryGlobalRequest = - serde_json::from_value(serde_json::json!({"window_days": 7})) - .expect("legacy window_days alias should deserialize"); - assert_eq!(from_alias.time_window_days, 7); - - let from_primary: QueryGlobalRequest = - serde_json::from_value(serde_json::json!({"time_window_days": 30})) - .expect("primary time_window_days should deserialize"); - assert_eq!(from_primary.time_window_days, 30); - } - // ── query_topic_rpc ─────────────────────────────────────────────── - - #[tokio::test] - async fn query_topic_rpc_logs_entity_kind_prefix_for_colon_separated_id() { - let (_tmp, cfg) = test_config(); - let req = QueryTopicRequest { - entity_id: "email:alice@example.com".into(), - time_window_days: None, - query: None, - limit: None, - }; - let outcome = query_topic_rpc(&cfg, req).await.unwrap(); - let log = &outcome.logs[0]; - assert!(log.contains("entity_kind=email"), "log: {log}"); - // PII redaction — the raw email must NOT appear anywhere in the log. - assert!(!log.contains("alice@example.com"), "log leaked PII: {log}"); - } - - #[tokio::test] - async fn query_topic_rpc_logs_unknown_when_entity_id_has_no_colon() { - let (_tmp, cfg) = test_config(); - let req = QueryTopicRequest { - entity_id: "nocolonhere".into(), - time_window_days: None, - query: None, - limit: None, - }; - let outcome = query_topic_rpc(&cfg, req).await.unwrap(); - assert!( - outcome.logs[0].contains("entity_kind=unknown"), - "log: {}", - outcome.logs[0] - ); - } - // ── search_entities_rpc ─────────────────────────────────────────── #[tokio::test] diff --git a/src/openhuman/memory_tree/retrieval/schemas.rs b/src/openhuman/memory_tree/retrieval/schemas.rs index 5621be04e..0dc347ca2 100644 --- a/src/openhuman/memory_tree/retrieval/schemas.rs +++ b/src/openhuman/memory_tree/retrieval/schemas.rs @@ -2,8 +2,6 @@ //! //! Registered JSON-RPC methods: //! - `openhuman.memory_tree_query_source` -//! - `openhuman.memory_tree_query_global` -//! - `openhuman.memory_tree_query_topic` //! - `openhuman.memory_tree_search_entities` //! - `openhuman.memory_tree_drill_down` //! - `openhuman.memory_tree_fetch_leaves` @@ -28,8 +26,6 @@ const NAMESPACE: &str = "memory_tree"; pub fn all_controller_schemas() -> Vec { vec![ schemas("query_source"), - schemas("query_global"), - schemas("query_topic"), schemas("search_entities"), schemas("drill_down"), schemas("fetch_leaves"), @@ -44,14 +40,6 @@ pub fn all_registered_controllers() -> Vec { schema: schemas("query_source"), handler: handle_query_source, }, - RegisteredController { - schema: schemas("query_global"), - handler: handle_query_global, - }, - RegisteredController { - schema: schemas("query_topic"), - handler: handle_query_topic, - }, RegisteredController { schema: schemas("search_entities"), handler: handle_search_entities, @@ -145,58 +133,6 @@ pub fn schemas(function: &str) -> ControllerSchema { ], outputs: query_response_outputs(), }, - "query_global" => ControllerSchema { - namespace: NAMESPACE, - function: "query_global", - description: "Return the global digest for the last N days. Wraps \ - `tree_global::recap`; the returned hit carries `child_ids` pointing \ - at the folded per-day summary ids for drill-down.", - inputs: vec![FieldSchema { - name: "time_window_days", - ty: TypeSchema::U64, - comment: "Lookback window in days (e.g. 7 for weekly recap).", - required: true, - }], - outputs: query_response_outputs(), - }, - "query_topic" => ControllerSchema { - namespace: NAMESPACE, - function: "query_topic", - description: "Return summaries / chunks associated with a canonical \ - entity id across every tree (source, topic, global). Also returns \ - the topic tree's root if one has materialised for the entity. \ - Sorted by (score DESC, timestamp DESC), or by cosine similarity \ - if `query` is provided.", - inputs: vec![ - FieldSchema { - name: "entity_id", - ty: TypeSchema::String, - comment: "Canonical id (e.g. `email:alice@example.com`, `topic:phoenix`).", - required: true, - }, - FieldSchema { - name: "time_window_days", - ty: TypeSchema::Option(Box::new(TypeSchema::U64)), - comment: "Only return hits whose time range overlaps the last N days.", - required: false, - }, - FieldSchema { - name: "query", - ty: TypeSchema::Option(Box::new(TypeSchema::String)), - comment: "Optional natural-language query — when present, \ - candidates are reranked by cosine similarity to the query's \ - embedding. Candidates without stored embeddings sort last.", - required: false, - }, - FieldSchema { - name: "limit", - ty: TypeSchema::Option(Box::new(TypeSchema::U64)), - comment: "Max hits (default 10).", - required: false, - }, - ], - outputs: query_response_outputs(), - }, "search_entities" => ControllerSchema { namespace: NAMESPACE, function: "search_entities", @@ -338,22 +274,6 @@ fn handle_query_source(params: Map) -> ControllerFuture { }) } -fn handle_query_global(params: Map) -> ControllerFuture { - Box::pin(async move { - let config = config_rpc::load_config_with_timeout().await?; - let req = parse_value::(Value::Object(params))?; - to_json(retrieval_rpc::query_global_rpc(&config, req).await?) - }) -} - -fn handle_query_topic(params: Map) -> ControllerFuture { - Box::pin(async move { - let config = config_rpc::load_config_with_timeout().await?; - let req = parse_value::(Value::Object(params))?; - to_json(retrieval_rpc::query_topic_rpc(&config, req).await?) - }) -} - fn handle_search_entities(params: Map) -> ControllerFuture { Box::pin(async move { let config = config_rpc::load_config_with_timeout().await?; @@ -398,8 +318,6 @@ mod tests { functions, vec![ "query_source", - "query_global", - "query_topic", "search_entities", "drill_down", "fetch_leaves", @@ -410,7 +328,7 @@ mod tests { #[test] fn registered_controllers_use_memory_tree_namespace() { let controllers = all_registered_controllers(); - assert_eq!(controllers.len(), 6); + assert_eq!(controllers.len(), 4); assert!(controllers.iter().all(|c| c.schema.namespace == NAMESPACE)); } @@ -422,12 +340,4 @@ mod tests { assert_eq!(schema.outputs.len(), 1); assert_eq!(schema.outputs[0].name, "error"); } - - #[test] - fn query_global_schema_requires_time_window_days() { - let schema = schemas("query_global"); - assert_eq!(schema.inputs.len(), 1); - assert_eq!(schema.inputs[0].name, "time_window_days"); - assert!(schema.inputs[0].required); - } } diff --git a/src/openhuman/memory_tree/retrieval/topic.rs b/src/openhuman/memory_tree/retrieval/topic.rs deleted file mode 100644 index 4cac9c3eb..000000000 --- a/src/openhuman/memory_tree/retrieval/topic.rs +++ /dev/null @@ -1,879 +0,0 @@ -//! `memory_tree_query_topic` — entity-scoped retrieval across every tree -//! that has seen the entity (Phase 4 / #710). -//! -//! Two data sources combined: -//! 1. [`score::store::lookup_entity`] returns every `(node_id, tree_id)` -//! association from the `mem_tree_entity_index` — covers leaves AND -//! summaries across all trees regardless of kind. -//! 2. If a per-entity topic tree exists (`(kind=topic, scope=entity_id)`), -//! we also surface its current root so the LLM can ask "summarise -//! everything you know about $entity" in one hop. -//! -//! Hits are filtered by `time_window_days` if given, then sorted -//! `score DESC, timestamp DESC` (strongest signal first, then newest). -//! Truncation to `limit` comes last. - -use anyhow::Result; -use chrono::{Duration, TimeZone, Utc}; - -use crate::openhuman::config::Config; -use crate::openhuman::memory_store::content::read as content_read; -use crate::openhuman::memory_store::trees::types::{Tree, TreeKind}; -use crate::openhuman::memory_tree::retrieval::types::{ - hit_from_summary, QueryResponse, RetrievalHit, -}; -use crate::openhuman::memory_tree::score::embed::{build_embedder_from_config, cosine_similarity}; -use crate::openhuman::memory_tree::score::store::{lookup_entity, EntityHit}; -use crate::openhuman::memory_tree::tree::store; - -const DEFAULT_LIMIT: usize = 10; -/// How many rows we pull from the entity index before filtering. We give -/// ourselves plenty of headroom because time-window + score-based filtering -/// can drop many rows — asking the index for exactly `limit` would bias -/// toward the newest hits at the expense of the strongest-score ones. -const LOOKUP_HEADROOM: usize = 200; - -/// Public entrypoint. `entity_id` should be the canonical id string -/// (e.g. `email:alice@example.com`, `topic:phoenix`). Unknown ids return -/// an empty response — callers that want fuzzy matching should go through -/// `memory_tree_search_entities` first. -/// -/// When `query` is `Some`, hits are reranked by cosine similarity to the -/// query's embedding; candidates without embeddings (legacy rows) fall -/// to the bottom. When `None`, the classic `(score DESC, timestamp DESC)` -/// ordering applies. -pub async fn query_topic( - config: &Config, - entity_id: &str, - time_window_days: Option, - query: Option<&str>, - limit: usize, -) -> Result { - let limit = if limit == 0 { DEFAULT_LIMIT } else { limit }; - // Redact `entity_id` — typically `email:` or `handle:`. - // Log the kind prefix only so operators can still see what kind of - // entity was queried. - let entity_kind_prefix = entity_id - .split_once(':') - .map(|(k, _)| k) - .unwrap_or("unknown"); - log::debug!( - "[retrieval::topic] query_topic entity_kind={} window_days={:?} has_query={} limit={}", - entity_kind_prefix, - time_window_days, - query.is_some(), - limit - ); - - let entity_id_owned = entity_id.to_string(); - let config_owned = config.clone(); - let (index_hits, topic_tree_summary) = - tokio::task::spawn_blocking(move || -> Result<(Vec, Option)> { - let hits = lookup_entity(&config_owned, &entity_id_owned, Some(LOOKUP_HEADROOM))?; - let topic_summary = fetch_topic_tree_root_summary(&config_owned, &entity_id_owned)?; - Ok((hits, topic_summary)) - }) - .await - .map_err(|e| anyhow::anyhow!("query_topic join error: {e}"))??; - - log::debug!( - "[retrieval::topic] index hits={} topic_tree_summary_present={}", - index_hits.len(), - topic_tree_summary.is_some() - ); - - // Deduplicate by node_id: the same node can appear multiple times - // across the entity index (one row per occurrence) and may also - // overlap the topic-tree root summary. Without dedup we inflate - // `total` and waste result slots. For duplicates, keep the higher - // score; if scores tie, prefer the newer `time_range_end`. - // Flagged on PR #831 CodeRabbit review. - // - // `BTreeMap` (not `HashMap`) so the post-dedup iteration order is a - // deterministic function of `node_id`. The downstream sort is - // stable, so when many hits tie on `(score, time_range_end)` — - // which is common with the inert embedder used in tests and with - // freshly-ingested workspaces where score normalisation hasn't run - // — the surviving order falls back to alphabetical `node_id` - // instead of `HashMap`'s randomised SipHash iteration. Without - // this, `tests/agent_retrieval_e2e.rs::orchestrator_query_topic…` - // picked a different "first leaf hit" on each run. - use std::collections::BTreeMap; - let mut by_node: BTreeMap = BTreeMap::new(); - - let merge = |map: &mut BTreeMap, hit: RetrievalHit| { - map.entry(hit.node_id.clone()) - .and_modify(|existing| { - let better = match hit - .score - .partial_cmp(&existing.score) - .unwrap_or(std::cmp::Ordering::Equal) - { - std::cmp::Ordering::Greater => true, - std::cmp::Ordering::Less => false, - std::cmp::Ordering::Equal => hit.time_range_end > existing.time_range_end, - }; - if better { - *existing = hit.clone(); - } - }) - .or_insert(hit); - }; - - if let Some(summary) = topic_tree_summary { - merge(&mut by_node, summary); - } - for h in index_hits { - if let Some(hit) = entity_hit_to_retrieval_hit(config, &h).await? { - merge(&mut by_node, hit); - } - } - - let mut hits: Vec = by_node.into_values().collect(); - if let Some(days) = time_window_days { - hits = filter_by_window(hits, days); - } - - let total = hits.len(); - - let sorted = if let Some(q) = query { - rerank_by_semantic_similarity(config, q, hits).await? - } else { - let mut by_score = hits; - // Sort: score DESC, then newest first on ties, then `node_id` - // ASC as a final tie-break so two hits that match on every - // ranked dimension still produce a deterministic order across - // runs (matters with the inert embedder used in tests, where - // every score lands at 0.0 and only the `node_id` differs). - by_score.sort_by(|a, b| { - b.score - .partial_cmp(&a.score) - .unwrap_or(std::cmp::Ordering::Equal) - .then_with(|| b.time_range_end.cmp(&a.time_range_end)) - .then_with(|| a.node_id.cmp(&b.node_id)) - }); - by_score - }; - let mut sorted = sorted; - sorted.truncate(limit); - - log::debug!( - "[retrieval::topic] returning hits={} total={}", - sorted.len(), - total - ); - Ok(QueryResponse::new(sorted, total)) -} - -/// Rerank hits by cosine similarity to the query embedding. Reads each -/// hit's stored embedding (summary rows from `mem_tree_summaries`, leaf -/// rows from `mem_tree_chunks`) directly via store helpers. Rows with no -/// embedding sort to the bottom, preserving their relative (score, time) -/// order so the unranked tail remains readable. -async fn rerank_by_semantic_similarity( - config: &Config, - query: &str, - hits: Vec, -) -> Result> { - use crate::openhuman::memory_store::chunks::store::get_chunk_embeddings_batch; - use crate::openhuman::memory_store::trees::store::get_summary_embeddings_batch; - use crate::openhuman::memory_tree::retrieval::types::NodeKind; - - let embedder = build_embedder_from_config(config)?; - let query_vec = embedder.embed(query).await?; - log::debug!( - "[retrieval::topic] query embedded provider={} hits_to_rerank={}", - embedder.name(), - hits.len() - ); - - // Partition hit ids by node kind so each table gets a single batched - // `IN (...)` lookup. Summary embeddings live in - // `mem_tree_summary_embeddings`, leaf/chunk embeddings in - // `mem_tree_chunk_embeddings` — two tables, two batched queries. - // - // Why partition + batch instead of one query per hit: - // - // Previously this function looped over `hits` and ran - // `spawn_blocking(get_*_embedding).await` per element. That `.await` - // inside the `for` was *sequential* — N round-trips to SQLite, each - // paying its own prepare + bind + busy-wait. With LOOKUP_HEADROOM=200 - // the rerank path could fire 200 sequential SQL statements on every - // entity-scoped query carrying a `query=` arg, which is the common - // per-turn shape. - // - // The batched helpers (see `chunks::store:: - // get_chunk_embeddings_for_signature_batch` and `trees::store:: - // get_summary_embeddings_for_signature_batch`) collapse all - // same-kind lookups into one `IN (?,?,?,...) AND model_signature = ?` - // query (chunked internally to stay below SQLite's variable cap so - // large future headroom values stay safe). The hit list is decorated - // in its original order from the resulting maps, so sort stability, - // tie-break behaviour, and the existing `(NEG_INFINITY, false)` - // handling for missing embeddings are all preserved bit-for-bit. - let mut summary_ids: Vec = Vec::new(); - let mut chunk_ids: Vec = Vec::new(); - for h in &hits { - match h.node_kind { - NodeKind::Summary => summary_ids.push(h.node_id.clone()), - NodeKind::Leaf => chunk_ids.push(h.node_id.clone()), - } - } - - // Both fetches run under one `spawn_blocking` to keep the event loop - // free while the SQLite reads happen on the blocking pool. - let config_owned = config.clone(); - let (summary_embeddings, chunk_embeddings) = - tokio::task::spawn_blocking(move || -> Result<(_, _)> { - let s = get_summary_embeddings_batch(&config_owned, &summary_ids)?; - let c = get_chunk_embeddings_batch(&config_owned, &chunk_ids)?; - Ok((s, c)) - }) - .await - .map_err(|e| anyhow::anyhow!("embedding batch join error: {e}"))??; - - let mut decorated: Vec<(f32, bool, RetrievalHit)> = Vec::with_capacity(hits.len()); - for h in hits { - // Decorate in the original `hits` iteration order so two hits - // that tie on every ranked dimension still produce the same - // relative ordering as before this refactor. - let emb_lookup = match h.node_kind { - NodeKind::Summary => summary_embeddings.get(&h.node_id), - NodeKind::Leaf => chunk_embeddings.get(&h.node_id), - }; - match emb_lookup { - Some(v) => { - let sim = cosine_similarity(&query_vec, v); - decorated.push((sim, true, h)); - } - None => { - // Identical to the pre-batch path: absent embedding - // sinks the hit to the bottom of the rerank without - // dropping it from the result set. - decorated.push((f32::NEG_INFINITY, false, h)); - } - } - } - - decorated.sort_by(|a, b| match (a.1, b.1) { - (true, false) => std::cmp::Ordering::Less, - (false, true) => std::cmp::Ordering::Greater, - _ => { - b.0.partial_cmp(&a.0) - .unwrap_or(std::cmp::Ordering::Equal) - .then_with(|| { - b.2.score - .partial_cmp(&a.2.score) - .unwrap_or(std::cmp::Ordering::Equal) - }) - .then_with(|| b.2.time_range_end.cmp(&a.2.time_range_end)) - } - }); - - Ok(decorated.into_iter().map(|(_, _, h)| h).collect()) -} - -/// Look up the topic tree for `entity_id` and return its current root as a -/// retrieval hit. Returns `None` if no topic tree exists (per Phase 3c -/// lazy materialisation — topic trees only spawn on hotness) or if the -/// tree has no sealed root yet. -fn fetch_topic_tree_root_summary(config: &Config, entity_id: &str) -> Result> { - let tree = match store::get_tree_by_scope(config, TreeKind::Topic, entity_id)? { - Some(t) => t, - None => return Ok(None), - }; - let root_id = match &tree.root_id { - Some(id) => id.clone(), - None => return Ok(None), - }; - let mut summary = match store::get_summary(config, &root_id)? { - Some(s) => s, - None => { - log::warn!( - "[retrieval::topic] topic tree has root_id set but the summary row is missing" - ); - return Ok(None); - } - }; - // Hydrate the full body from disk — `summary.content` is a ≤500-char - // preview after the MD-on-disk migration. Non-fatal fallback for - // pre-MD-migration rows. - match content_read::read_summary_body(config, &root_id) { - Ok(body) => summary.content = body, - Err(e) => { - log::warn!( - "[retrieval::topic] read_summary_body failed for topic root — serving preview: {e:#}" - ); - } - } - Ok(Some(hit_from_summary(&summary, &tree.scope))) -} - -/// Convert a raw [`EntityHit`] row into a [`RetrievalHit`] by hydrating the -/// backing node. Summary hits fetch from `mem_tree_summaries`; leaf hits -/// fetch from `mem_tree_chunks`. Missing rows are skipped with a warn log -/// — the index row is stale but the retrieval doesn't error out. -async fn entity_hit_to_retrieval_hit( - config: &Config, - hit: &EntityHit, -) -> Result> { - let node_id = hit.node_id.clone(); - let node_kind = hit.node_kind.clone(); - let tree_id_opt = hit.tree_id.clone(); - let score = hit.score; - let timestamp_ms = hit.timestamp_ms; - let config_owned = config.clone(); - - tokio::task::spawn_blocking(move || -> Result> { - if node_kind == "summary" { - let mut summary = match store::get_summary(&config_owned, &node_id)? { - Some(s) => s, - None => { - log::warn!("[retrieval::topic] entity index points at missing summary row"); - return Ok(None); - } - }; - // Hydrate the full body from disk — `summary.content` is a - // ≤500-char preview after the MD-on-disk migration. - match content_read::read_summary_body(&config_owned, &node_id) { - Ok(body) => summary.content = body, - Err(e) => { - log::warn!( - "[retrieval::topic] read_summary_body failed — serving preview: {e:#}" - ); - } - } - // Prefer tree scope from the summary's parent tree if resolvable. - let scope = if let Some(tid) = &tree_id_opt { - store::get_tree(&config_owned, tid)? - .map(|t: Tree| t.scope) - .unwrap_or_default() - } else { - String::new() - }; - let mut h = hit_from_summary(&summary, &scope); - // The index row's own score is a per-(entity, node) signal — - // inherit it so topic ordering uses the association strength - // rather than the summary's overall score. - h.score = score; - return Ok(Some(h)); - } - // Leaf: fetch chunk and hydrate. - use crate::openhuman::memory_store::chunks::store::get_chunk; - use crate::openhuman::memory_tree::retrieval::types::hit_from_chunk; - let mut chunk = match get_chunk(&config_owned, &node_id)? { - Some(c) => c, - None => { - log::warn!("[retrieval::topic] entity index points at missing chunk row"); - return Ok(None); - } - }; - // Hydrate the full body from disk — `chunk.content` is a ≤500-char - // preview after the MD-on-disk migration. - match content_read::read_chunk_body(&config_owned, &node_id) { - Ok(body) => chunk.content = body, - Err(e) => { - log::warn!("[retrieval::topic] read_chunk_body failed — serving preview: {e:#}"); - } - } - let scope = if let Some(tid) = &tree_id_opt { - store::get_tree(&config_owned, tid)? - .map(|t: Tree| t.scope) - .unwrap_or_else(|| chunk.metadata.source_id.clone()) - } else { - chunk.metadata.source_id.clone() - }; - let mut h = hit_from_chunk(&chunk, tree_id_opt.as_deref().unwrap_or(""), &scope, score); - // Stamp the hit's time range end to the index's recorded timestamp - // if our chunk row lacks a meaningful range (e.g. pre-3a leaves). - if h.time_range_end <= chrono::DateTime::::MIN_UTC { - if let chrono::LocalResult::Single(dt) = Utc.timestamp_millis_opt(timestamp_ms) { - h.time_range_end = dt; - h.time_range_start = dt; - } - } - Ok(Some(h)) - }) - .await - .map_err(|e| anyhow::anyhow!("entity_hit conversion join error: {e}"))? -} - -fn filter_by_window(hits: Vec, window_days: u32) -> Vec { - let now = Utc::now(); - let window_start = now - Duration::days(window_days as i64); - hits.into_iter() - .filter(|h| h.time_range_end >= window_start && h.time_range_start <= now) - .collect() -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::openhuman::memory::ingest_pipeline::ingest_chat; - use crate::openhuman::memory_store::chunks::store::upsert_chunks; - use crate::openhuman::memory_store::chunks::types::{ - chunk_id, Chunk, Metadata, SourceKind, SourceRef, - }; - use crate::openhuman::memory_sync::canonicalize::chat::{ChatBatch, ChatMessage}; - use crate::openhuman::memory_tree::score::extract::EntityKind; - use crate::openhuman::memory_tree::score::store::EntityHit; - use chrono::TimeZone; - use tempfile::TempDir; - - fn test_config() -> (TempDir, Config) { - let tmp = TempDir::new().unwrap(); - let mut cfg = Config::default(); - cfg.workspace_dir = tmp.path().to_path_buf(); - // Phase 4 (#710): ingest triggers seals which embed. - cfg.memory_tree.embedding_endpoint = None; - cfg.memory_tree.embedding_model = None; - cfg.memory_tree.embedding_strict = false; - (tmp, cfg) - } - - fn substantive_batch() -> ChatBatch { - ChatBatch { - platform: "slack".into(), - channel_label: "#eng".into(), - messages: vec![ChatMessage { - author: "alice".into(), - timestamp: Utc.timestamp_millis_opt(1_700_000_000_000).unwrap(), - text: "We are planning to ship the Phoenix migration on Friday \ - after reviewing the runbook and staging results. \ - alice@example.com please confirm." - .into(), - source_ref: Some("slack://m1".into()), - }], - } - } - - #[tokio::test] - async fn unknown_entity_returns_empty() { - let (_tmp, cfg) = test_config(); - let resp = query_topic(&cfg, "email:nobody@example.com", None, None, 10) - .await - .unwrap(); - assert!(resp.hits.is_empty()); - assert_eq!(resp.total, 0); - } - - #[tokio::test] - async fn query_email_entity_after_ingest() { - let (_tmp, cfg) = test_config(); - ingest_chat(&cfg, "slack:#eng", "alice", vec![], substantive_batch()) - .await - .unwrap(); - let resp = query_topic(&cfg, "email:alice@example.com", None, None, 10) - .await - .unwrap(); - assert!( - !resp.hits.is_empty(), - "alice's chunk should be surfaced via the entity index" - ); - } - - #[tokio::test] - async fn query_topic_entity_after_ingest() { - // The topic-as-entity promotion from Phase 3a means "phoenix" shows - // up under `topic:phoenix` once the ingest's scorer extracts it. - let (_tmp, cfg) = test_config(); - ingest_chat(&cfg, "slack:#eng", "alice", vec![], substantive_batch()) - .await - .unwrap(); - let resp = query_topic(&cfg, "topic:phoenix", None, None, 10) - .await - .unwrap(); - // Topic extraction may depend on the specific scorer config; at - // minimum the call should succeed and the response is a well-formed - // (possibly empty) `QueryResponse`. We don't hard-assert hits here - // because the scorer extraction rules are out of Phase 4's scope. - assert!(resp.total >= resp.hits.len()); - } - - #[tokio::test] - async fn query_filters_by_time_window() { - let (_tmp, cfg) = test_config(); - // Seed an old chunk via a batch whose timestamp is ancient. - let old_batch = ChatBatch { - platform: "slack".into(), - channel_label: "#eng".into(), - messages: vec![ChatMessage { - author: "alice".into(), - timestamp: Utc.timestamp_millis_opt(1_000_000_000_000).unwrap(), - text: "Ancient plan to ship Phoenix. alice@example.com has been \ - the owner of the runbook for ages." - .into(), - source_ref: Some("slack://ancient".into()), - }], - }; - ingest_chat(&cfg, "slack:#ancient", "alice", vec![], old_batch) - .await - .unwrap(); - - // 7-day window should reject the ancient hit. - let resp = query_topic(&cfg, "email:alice@example.com", Some(7), None, 10) - .await - .unwrap(); - assert!(resp.hits.is_empty(), "ancient mention filtered by window"); - } - - #[tokio::test] - async fn query_truncates_to_limit() { - let (_tmp, cfg) = test_config(); - // Three separate sources all mentioning alice. - for i in 0..3 { - let source = format!("slack:#c{i}"); - let batch = ChatBatch { - platform: "slack".into(), - channel_label: format!("#c{i}"), - messages: vec![ChatMessage { - author: "alice".into(), - timestamp: Utc::now(), - text: format!( - "Meeting {i} about Phoenix migration. alice@example.com owns it. \ - Launch status looks good." - ), - source_ref: None, - }], - }; - ingest_chat(&cfg, &source, "alice", vec![], batch) - .await - .unwrap(); - } - let resp = query_topic(&cfg, "email:alice@example.com", None, None, 2) - .await - .unwrap(); - assert!(resp.hits.len() <= 2); - assert!(resp.total >= resp.hits.len()); - if resp.total > 2 { - assert!(resp.truncated); - } - } - - #[tokio::test] - async fn hits_sorted_by_score_descending() { - let (_tmp, cfg) = test_config(); - ingest_chat(&cfg, "slack:#eng", "alice", vec![], substantive_batch()) - .await - .unwrap(); - let resp = query_topic(&cfg, "email:alice@example.com", None, None, 10) - .await - .unwrap(); - for w in resp.hits.windows(2) { - assert!( - w[0].score >= w[1].score, - "expected score DESC ordering, got {} then {}", - w[0].score, - w[1].score - ); - } - } - - #[tokio::test] - async fn zero_limit_uses_default_limit() { - let (_tmp, cfg) = test_config(); - for i in 0..12 { - let source = format!("slack:#c{i}"); - let batch = ChatBatch { - platform: "slack".into(), - channel_label: format!("#c{i}"), - messages: vec![ChatMessage { - author: "alice".into(), - timestamp: Utc::now(), - text: format!( - "Meeting {i} about Phoenix migration. alice@example.com owns it. \ - Launch status looks good." - ), - source_ref: None, - }], - }; - ingest_chat(&cfg, &source, "alice", vec![], batch) - .await - .unwrap(); - } - - let resp = query_topic(&cfg, "email:alice@example.com", None, None, 0) - .await - .unwrap(); - assert!(resp.hits.len() <= DEFAULT_LIMIT); - if resp.total > DEFAULT_LIMIT { - assert!(resp.truncated); - } - } - - #[tokio::test] - async fn topic_tree_root_missing_summary_row_is_ignored() { - use crate::openhuman::memory_store::chunks::store::with_connection; - use crate::openhuman::memory_store::trees::types::{Tree, TreeKind, TreeStatus}; - use crate::openhuman::memory_tree::tree::store as tree_store; - - let (_tmp, cfg) = test_config(); - let ts = Utc::now(); - let entity_id = "topic:phoenix"; - let tree = Tree { - id: "test:phoenix-missing-root".into(), - kind: TreeKind::Topic, - scope: entity_id.into(), - root_id: Some("summary:missing".into()), - max_level: 1, - status: TreeStatus::Active, - created_at: ts, - last_sealed_at: Some(ts), - }; - - with_connection(&cfg, |conn| { - let tx = conn.unchecked_transaction()?; - tree_store::insert_tree_conn(&tx, &tree)?; - tx.commit()?; - Ok(()) - }) - .unwrap(); - - let resp = query_topic(&cfg, entity_id, None, None, 10).await.unwrap(); - assert!(resp.hits.is_empty()); - assert_eq!(resp.total, 0); - } - - // Regression: the same node_id must only appear once in `hits`, even - // when the topic-tree root overlaps with its own entity-index row. - // Flagged on PR #831 CodeRabbit review — see the HashMap-based merge - // in `query_topic`. Without the dedup, `total` would be 2 and the - // caller would see two rows for the same summary. - #[tokio::test] - async fn duplicate_node_is_deduplicated_across_index_and_topic_tree_root() { - use crate::openhuman::memory_store::chunks::store::with_connection; - use crate::openhuman::memory_store::trees::types::{ - SummaryNode, Tree, TreeKind, TreeStatus, - }; - use crate::openhuman::memory_tree::score::resolver::CanonicalEntity; - use crate::openhuman::memory_tree::score::store as score_store; - use crate::openhuman::memory_tree::tree::store as tree_store; - - let (_tmp, cfg) = test_config(); - let ts = Utc::now(); - let entity_id = "topic:phoenix"; - let summary_id = "summary:L1:phoenix-root"; - - // 1. Create a topic tree whose root points at `summary_id`. - let tree = Tree { - id: "test:phoenix-topic-tree".into(), - kind: TreeKind::Topic, - scope: entity_id.into(), - root_id: Some(summary_id.into()), - max_level: 1, - status: TreeStatus::Active, - created_at: ts, - last_sealed_at: Some(ts), - }; - - // 2. Create the summary row itself. - let summary = SummaryNode { - id: summary_id.into(), - tree_id: tree.id.clone(), - tree_kind: TreeKind::Topic, - level: 1, - parent_id: None, - child_ids: vec![], - content: "Phoenix migration recap".into(), - token_count: 100, - entities: vec![entity_id.into()], - topics: vec![], - time_range_start: ts, - time_range_end: ts, - score: 0.5, - sealed_at: ts, - deleted: false, - embedding: None, - }; - - // 3. Write tree + summary + entity-index row in one tx. The - // entity-index row is what creates the dedup scenario: both - // `lookup_entity` AND `fetch_topic_tree_root_summary` will - // now return the same node. - let entity = CanonicalEntity { - canonical_id: entity_id.into(), - kind: EntityKind::Topic, - surface: "phoenix".into(), - span_start: 0, - span_end: 7, - score: 0.9, - }; - with_connection(&cfg, |conn| { - let tx = conn.unchecked_transaction()?; - tree_store::insert_tree_conn(&tx, &tree)?; - tree_store::insert_summary_tx(&tx, &summary, None, "test")?; - score_store::index_entities_tx( - &tx, - &[entity], - summary_id, - "summary", - ts.timestamp_millis(), - Some(&tree.id), - )?; - tx.commit()?; - Ok(()) - }) - .unwrap(); - - // 4. Query — expect exactly one hit (the summary), not two. - let resp = query_topic(&cfg, entity_id, None, None, 10).await.unwrap(); - let phoenix_hits: Vec<_> = resp - .hits - .iter() - .filter(|h| h.node_id == summary_id) - .collect(); - assert_eq!( - phoenix_hits.len(), - 1, - "summary should appear once, not duplicated between index \ - and topic-tree root; got {} phoenix hits in response of {}", - phoenix_hits.len(), - resp.hits.len() - ); - // `total` also reflects the dedup'd count. - assert_eq!( - resp.total, 1, - "total should count distinct nodes, not raw row occurrences" - ); - } - - #[tokio::test] - async fn stale_leaf_entity_index_row_is_skipped() { - let (_tmp, cfg) = test_config(); - let hit = EntityHit { - entity_id: "email:alice@example.com".into(), - node_id: "chunk:missing".into(), - node_kind: "leaf".into(), - entity_kind: EntityKind::Email, - surface: "alice@example.com".into(), - score: 0.7, - timestamp_ms: Utc::now().timestamp_millis(), - tree_id: Some("tree:missing".into()), - is_user: false, - }; - - let converted = entity_hit_to_retrieval_hit(&cfg, &hit).await.unwrap(); - assert!(converted.is_none()); - } - - #[tokio::test] - async fn leaf_hit_falls_back_to_source_scope_when_tree_lookup_misses() { - let (_tmp, cfg) = test_config(); - let ts = Utc.timestamp_millis_opt(1_700_123_456_789).unwrap(); - let chunk = Chunk { - id: chunk_id( - SourceKind::Chat, - "slack:#eng", - 0, - "Phoenix owner is alice@example.com", - ), - content: "Phoenix owner is alice@example.com".into(), - metadata: Metadata { - source_kind: SourceKind::Chat, - source_id: "slack:#eng".into(), - owner: "alice".into(), - timestamp: ts, - time_range: (ts, ts), - tags: vec!["phoenix".into()], - source_ref: Some(SourceRef::new("slack://eng/1")), - }, - token_count: 8, - seq_in_source: 0, - created_at: ts, - partial_message: false, - }; - upsert_chunks(&cfg, &[chunk.clone()]).unwrap(); - - let hit = EntityHit { - entity_id: "email:alice@example.com".into(), - node_id: chunk.id.clone(), - node_kind: "leaf".into(), - entity_kind: EntityKind::Email, - surface: "alice@example.com".into(), - score: 0.91, - timestamp_ms: ts.timestamp_millis(), - tree_id: Some("tree:missing".into()), - is_user: false, - }; - - let converted = entity_hit_to_retrieval_hit(&cfg, &hit) - .await - .unwrap() - .unwrap(); - assert_eq!(converted.tree_scope, "slack:#eng"); - assert_eq!(converted.tree_id, "tree:missing"); - assert_eq!(converted.score, 0.91); - assert_eq!(converted.source_ref.as_deref(), Some("slack://eng/1")); - assert_eq!(converted.topics, vec!["phoenix"]); - } - - #[tokio::test] - async fn summary_hit_without_tree_id_uses_empty_scope_and_index_score() { - use crate::openhuman::memory_store::chunks::store::with_connection; - use crate::openhuman::memory_store::trees::types::{ - SummaryNode, Tree, TreeKind, TreeStatus, - }; - use crate::openhuman::memory_tree::tree::store as tree_store; - - let (_tmp, cfg) = test_config(); - let ts = Utc::now(); - let tree = Tree { - id: "tree:topic:phoenix".into(), - kind: TreeKind::Topic, - scope: "topic:phoenix".into(), - root_id: Some("summary:l1:phoenix".into()), - max_level: 1, - status: TreeStatus::Active, - created_at: ts, - last_sealed_at: Some(ts), - }; - let summary = SummaryNode { - id: "summary:l1:phoenix".into(), - tree_id: tree.id.clone(), - tree_kind: TreeKind::Topic, - level: 1, - parent_id: None, - child_ids: vec!["chunk:a".into()], - content: "Phoenix recap preview".into(), - token_count: 16, - entities: vec!["topic:phoenix".into()], - topics: vec!["phoenix".into()], - time_range_start: ts, - time_range_end: ts, - score: 0.12, - sealed_at: ts, - deleted: false, - embedding: None, - }; - - with_connection(&cfg, |conn| { - let tx = conn.unchecked_transaction()?; - tree_store::insert_tree_conn(&tx, &tree)?; - tree_store::insert_summary_tx(&tx, &summary, None, "test")?; - tx.commit()?; - Ok(()) - }) - .unwrap(); - - let hit = EntityHit { - entity_id: "topic:phoenix".into(), - node_id: summary.id.clone(), - node_kind: "summary".into(), - entity_kind: EntityKind::Topic, - surface: "phoenix".into(), - score: 0.88, - timestamp_ms: ts.timestamp_millis(), - tree_id: None, - is_user: false, - }; - - let converted = entity_hit_to_retrieval_hit(&cfg, &hit) - .await - .unwrap() - .unwrap(); - assert_eq!(converted.tree_scope, ""); - assert_eq!(converted.tree_id, tree.id); - assert_eq!(converted.score, 0.88); - assert_eq!(converted.content, "Phoenix recap preview"); - } -} diff --git a/src/openhuman/memory_tree/tree/bucket_seal.rs b/src/openhuman/memory_tree/tree/bucket_seal.rs index 422959028..cad339765 100644 --- a/src/openhuman/memory_tree/tree/bucket_seal.rs +++ b/src/openhuman/memory_tree/tree/bucket_seal.rs @@ -576,13 +576,12 @@ pub(crate) async fn seal_one_level( continuing seal without vault defaults" ); } - let staged = - stage_summary(&content_root, &compose_input, &scope_slug, None).with_context(|| { - format!( - "stage_summary failed for {}; seal aborted, buffer stays unsealed for retry", - node.id - ) - })?; + let staged = stage_summary(&content_root, &compose_input, &scope_slug).with_context(|| { + format!( + "stage_summary failed for {}; seal aborted, buffer stays unsealed for retry", + node.id + ) + })?; log::debug!( "[tree::bucket_seal] staged summary {} → {}", node.id, @@ -685,19 +684,9 @@ pub(crate) async fn seal_one_level( }; enqueue_job_tx(&tx, &NewJob::seal(&parent_seal)?)?; } - // Source-tree summary routing: feed the new summary's - // entities back into the topic-tree spawn pipeline. Topic - // and global trees are sinks — no fan-out from their seals. - if matches!(tree_kind, TreeKind::Source) { - use crate::openhuman::memory_queue::store::enqueue_tx as enqueue_job_tx; - use crate::openhuman::memory_queue::types::{NewJob, NodeRef, TopicRoutePayload}; - let route = TopicRoutePayload { - node: NodeRef::Summary { - summary_id: summary_id_for_closure.clone(), - }, - }; - enqueue_job_tx(&tx, &NewJob::topic_route(&route)?)?; - } + // (Topic-tree routing removed: the topic/global trees were + // deleted — source trees plus the entity index are the + // substrate, so a source seal no longer fans out anywhere.) } // Update tree root / max_level if we just climbed. diff --git a/src/openhuman/memory_tree/tree/bucket_seal_tests.rs b/src/openhuman/memory_tree/tree/bucket_seal_tests.rs index 8edd327ee..3db223d69 100644 --- a/src/openhuman/memory_tree/tree/bucket_seal_tests.rs +++ b/src/openhuman/memory_tree/tree/bucket_seal_tests.rs @@ -790,7 +790,6 @@ async fn hydrate_summary_inputs_batch_preserves_order_and_skips_missing_ids() { body: &n.content, }, "slack-eng", - None, ) .unwrap() }; diff --git a/src/openhuman/memory_tree/tree/rpc.rs b/src/openhuman/memory_tree/tree/rpc.rs index bf220acaa..ff2c5e566 100644 --- a/src/openhuman/memory_tree/tree/rpc.rs +++ b/src/openhuman/memory_tree/tree/rpc.rs @@ -185,77 +185,6 @@ pub async fn get_chunk_rpc( )) } -/// Manual-trigger surface for the global tree's daily digest. Default -/// behavior (no `date_iso`) targets yesterday in UTC, matching the -/// scheduler's autonomous behavior. Pass an explicit `YYYY-MM-DD` to -/// re-run a specific date (idempotent — the handler skips if a daily -/// node already exists for that day). -#[derive(Clone, Debug, Default, Serialize, Deserialize)] -pub struct TriggerDigestRequest { - /// UTC calendar date in `YYYY-MM-DD` form. When omitted, defaults to - /// `yesterday` (today minus one day, UTC). - #[serde(default)] - pub date_iso: Option, -} - -/// Response from the `trigger_digest` RPC. -#[derive(Clone, Debug, Serialize, Deserialize)] -pub struct TriggerDigestResponse { - /// True when the job was newly enqueued; false when an active job for - /// the same date was suppressed by the dedupe partial unique index. - pub enqueued: bool, - /// ID of the freshly-inserted job row (None when dedupe-suppressed). - pub job_id: Option, - /// The actual date the digest will run for, echoed back as - /// `YYYY-MM-DD`. Useful when the caller didn't pass `date_iso` and - /// wants to know what default got chosen. - pub date_iso: String, -} - -/// `trigger_digest` RPC handler. Manually enqueues the global tree's daily -/// digest job for `date_iso` (defaults to yesterday in UTC); idempotent via the -/// jobs-queue dedupe index. -pub async fn trigger_digest_rpc( - config: &Config, - req: TriggerDigestRequest, -) -> Result, String> { - use crate::openhuman::memory_queue as jobs; - use chrono::{Duration as ChronoDuration, NaiveDate, Utc}; - - let date = match req - .date_iso - .as_deref() - .map(str::trim) - .filter(|s| !s.is_empty()) - { - Some(s) => NaiveDate::parse_from_str(s, "%Y-%m-%d") - .map_err(|e| format!("invalid date_iso (expected YYYY-MM-DD): {e}"))?, - None => Utc::now().date_naive() - ChronoDuration::days(1), - }; - let date_iso = date.format("%Y-%m-%d").to_string(); - - // Run the synchronous enqueue on a blocking thread — `trigger_digest` - // touches SQLite and we don't want to block the async runtime even - // for the few-microsecond INSERT. - let cfg_clone = config.clone(); - let date_for_blocking = date; - let job_id = - tokio::task::spawn_blocking(move || jobs::trigger_digest(&cfg_clone, date_for_blocking)) - .await - .map_err(|e| format!("trigger_digest join error: {e}"))? - .map_err(|e| format!("trigger_digest: {e}"))?; - - let enqueued = job_id.is_some(); - Ok(RpcOutcome::single_log( - TriggerDigestResponse { - enqueued, - job_id, - date_iso: date_iso.clone(), - }, - format!("memory_tree: trigger_digest date={date_iso} enqueued={enqueued}"), - )) -} - /// Response from the `memory_backfill_status` RPC (#1574 §4b). The frontend /// polls this while the re-embed modal is open to surface progress and to /// dismiss the modal once the new embedding space is fully covered. @@ -828,61 +757,6 @@ mod tests { assert!(outcome.value.chunk.is_none()); } - #[tokio::test] - async fn trigger_digest_with_explicit_date_enqueues() { - let (_tmp, cfg) = test_config(); - let req = TriggerDigestRequest { - date_iso: Some("2026-04-27".into()), - }; - let outcome = trigger_digest_rpc(&cfg, req).await.unwrap(); - let resp = outcome.value; - assert!(resp.enqueued); - assert!(resp.job_id.is_some()); - assert_eq!(resp.date_iso, "2026-04-27"); - assert_eq!(count_total(&cfg).unwrap(), 1); - } - - #[tokio::test] - async fn trigger_digest_with_no_date_defaults_to_yesterday() { - let (_tmp, cfg) = test_config(); - let req = TriggerDigestRequest::default(); - let outcome = trigger_digest_rpc(&cfg, req).await.unwrap(); - let resp = outcome.value; - assert!(resp.enqueued); - let expected = (Utc::now().date_naive() - ChronoDuration::days(1)) - .format("%Y-%m-%d") - .to_string(); - assert_eq!(resp.date_iso, expected); - } - - #[tokio::test] - async fn trigger_digest_rejects_malformed_date() { - let (_tmp, cfg) = test_config(); - let req = TriggerDigestRequest { - date_iso: Some("not-a-date".into()), - }; - let err = trigger_digest_rpc(&cfg, req).await.unwrap_err(); - assert!( - err.contains("invalid date_iso"), - "expected schema-shaped error message, got: {err}" - ); - assert_eq!(count_total(&cfg).unwrap(), 0); - } - - #[tokio::test] - async fn trigger_digest_dedupes_active_jobs() { - let (_tmp, cfg) = test_config(); - let req = TriggerDigestRequest { - date_iso: Some("2026-04-27".into()), - }; - let first = trigger_digest_rpc(&cfg, req.clone()).await.unwrap().value; - let second = trigger_digest_rpc(&cfg, req).await.unwrap().value; - assert!(first.enqueued); - assert!(!second.enqueued, "duplicate must be dedupe-suppressed"); - assert!(second.job_id.is_none()); - assert_eq!(count_total(&cfg).unwrap(), 1); - } - /// #1574 §4b: `backfill_status_rpc` reports 0 pending on an idle space /// and reflects a queued `reembed_backfill` job (forcing `in_progress`). /// `in_progress` for the empty case is intentionally not asserted — the diff --git a/src/openhuman/subconscious/situation_report/digest.rs b/src/openhuman/subconscious/situation_report/digest.rs deleted file mode 100644 index 93b742328..000000000 --- a/src/openhuman/subconscious/situation_report/digest.rs +++ /dev/null @@ -1,108 +0,0 @@ -//! Latest global L0 digest section (#623). -//! -//! The global tree's L0 nodes are daily digests. We fetch the most recent -//! one for the situation report. The body is truncated to keep prompt -//! footprint tight. -//! -//! Cutoff semantics: only the digest sealed *after* `last_tick_at` is -//! emitted. Without this gate the same digest gets re-rendered in every -//! tick's report verbatim, the LLM keeps citing its id, and -//! `persist_and_surface_reflections` (no insert-time dedupe) accumulates -//! near-duplicate reflections about the same digest forever — which is -//! exactly what was happening before this section was gated. - -use crate::openhuman::config::Config; -use crate::openhuman::memory_store::trees::types::TreeKind; - -/// Truncate point for the digest body in the situation report. -const DIGEST_BODY_PREVIEW: usize = 1200; - -pub async fn build_section(config: &Config, last_tick_at: f64) -> String { - let cutoff_ms: i64 = if last_tick_at <= 0.0 { - // Cold start — accept any digest. The summaries / query_window - // sections do the same thing on cold start. - 0 - } else { - (last_tick_at * 1000.0) as i64 - }; - log::debug!( - "[subconscious::situation_report::digest] building section last_tick_at={last_tick_at} cutoff_ms={cutoff_ms}" - ); - - let row = match read_latest_global_l0(config, cutoff_ms) { - Ok(Some(row)) => row, - Ok(None) => { - // Distinguish "no digest exists at all" from "digest exists - // but hasn't advanced since last tick" — both render the - // same to the LLM (no fresh content), but the log is - // useful for diagnosing why it stopped citing the digest. - log::debug!( - "[subconscious::situation_report::digest] no digest sealed since cutoff_ms={cutoff_ms}" - ); - return "## Latest daily digest\n\nNo new global digest sealed since last tick.\n" - .to_string(); - } - Err(e) => { - log::warn!("[subconscious::situation_report::digest] read failed: {e}"); - return "## Latest daily digest\n\nDigest unavailable.\n".to_string(); - } - }; - - let preview = truncate(&row.content, DIGEST_BODY_PREVIEW); - format!( - "## Latest daily digest\n\nSealed at unix-ms {} (id={}):\n\n{}\n", - row.sealed_at_ms, row.id, preview - ) -} - -#[derive(Debug)] -struct DigestRow { - id: String, - content: String, - sealed_at_ms: i64, -} - -fn read_latest_global_l0(config: &Config, cutoff_ms: i64) -> anyhow::Result> { - crate::openhuman::memory_store::chunks::store::with_connection(config, |conn| { - let row = conn - .query_row( - "SELECT s.id, s.content, s.sealed_at_ms - FROM mem_tree_summaries s - JOIN mem_tree_trees t ON t.id = s.tree_id - WHERE t.kind = ?1 AND s.level = 0 AND s.deleted = 0 - AND s.sealed_at_ms > ?2 - ORDER BY s.sealed_at_ms DESC LIMIT 1", - rusqlite::params![tree_kind_global_str(), cutoff_ms], - |row| { - Ok(DigestRow { - id: row.get(0)?, - content: row.get(1)?, - sealed_at_ms: row.get(2)?, - }) - }, - ) - .ok(); - Ok(row) - }) -} - -/// Stable wire string for `TreeKind::Global` as persisted by the -/// memory_tree's `tree_source` writer. Centralised here so a future -/// rename in the source-of-truth lands in one place. -fn tree_kind_global_str() -> &'static str { - // `TreeKind` serialises via serde with rename_all = "snake_case", - // so `Global` -> "global". Keep the constant explicit (rather than - // round-tripping serde at runtime) so the prompt section is cheap. - let _kind_check = TreeKind::Global; - "global" -} - -fn truncate(text: &str, max_chars: usize) -> String { - let trimmed = text.trim(); - if trimmed.chars().count() <= max_chars { - return trimmed.to_string(); - } - let mut out: String = trimmed.chars().take(max_chars).collect(); - out.push('…'); - out -} diff --git a/src/openhuman/subconscious/situation_report/hotness.rs b/src/openhuman/subconscious/situation_report/hotness.rs deleted file mode 100644 index 76a37d7ec..000000000 --- a/src/openhuman/subconscious/situation_report/hotness.rs +++ /dev/null @@ -1,201 +0,0 @@ -//! Hotness deltas section — top-K entities whose `mem_tree_entity_hotness` -//! score moved meaningfully since the last tick (#623). -//! -//! Joins the live hotness table against the `subconscious_hotness_snapshots` -//! table populated at the end of each tick. Returns the top 10 movers by -//! absolute delta. After formatting, refreshes the snapshots so the next -//! tick has a fresh baseline. -//! -//! Failure is non-fatal — any DB error returns a "Hotness deltas -//! unavailable" stub so the rest of the situation report still renders. - -use std::fmt::Write; -use std::path::Path; - -use crate::openhuman::config::Config; -use crate::openhuman::subconscious::reflection_store; -use crate::openhuman::subconscious::store as subconscious_store; - -/// Maximum entries to render in the section. -const MAX_DELTAS: usize = 10; - -pub async fn build_section(config: &Config, workspace_dir: &Path, _last_tick_at: f64) -> String { - log::debug!("[subconscious::situation_report::hotness] building section"); - - // 1. Read current hotness from the memory_tree DB. `is_user` joins - // against the entity index (#1365) so reflection generation can - // tell which movers are the user vs other people. - let current = match read_current_hotness(config) { - Ok(rows) => rows, - Err(e) => { - log::warn!("[subconscious::situation_report::hotness] read failed: {e}"); - return "## Hotness deltas\n\nHotness deltas unavailable.\n".to_string(); - } - }; - - if current.is_empty() { - let _ = update_snapshots(workspace_dir, &[]); - return "## Hotness deltas\n\nNo entity hotness data yet.\n".to_string(); - } - - // 2. Read previous snapshot. - let previous = subconscious_store::with_connection(workspace_dir, |conn| { - reflection_store::load_hotness_snapshots(conn) - }) - .unwrap_or_else(|e| { - log::warn!("[subconscious::situation_report::hotness] snapshot load failed: {e}"); - Vec::new() - }); - let prev_map: std::collections::HashMap = previous.into_iter().collect(); - - // 3. Compute deltas; carry is_user through. - let mut deltas: Vec = current - .iter() - .map(|row| { - let prev = prev_map.get(&row.entity_id).copied().unwrap_or(0.0); - HotnessDelta { - entity_id: row.entity_id.clone(), - score: row.score, - delta: row.score - prev, - is_user: row.is_user, - } - }) - .collect(); - // Highest |delta| first; ties broken by current score. - deltas.sort_by(|a, b| { - b.delta - .abs() - .partial_cmp(&a.delta.abs()) - .unwrap_or(std::cmp::Ordering::Equal) - .then_with(|| { - b.score - .partial_cmp(&a.score) - .unwrap_or(std::cmp::Ordering::Equal) - }) - }); - - // 4. Format top-K. - let top: Vec<&HotnessDelta> = deltas - .iter() - .filter(|d| d.delta.abs() > f64::EPSILON) - .take(MAX_DELTAS) - .collect(); - - let mut section = String::from("## Hotness deltas\n\n"); - if top.is_empty() { - section.push_str("No movement since last tick.\n"); - } else { - let _ = writeln!( - section, - "Top {} entity movers (score = post-delta, Δ = change). \ - Items tagged `(you)` are the user's own identifiers — \ - reflect on these in second person; for everything else, \ - reflect on what *others* are doing or talking about.", - top.len() - ); - section.push('\n'); - for d in &top { - let arrow = if d.delta > 0.0 { "▲" } else { "▼" }; - let self_marker = if d.is_user { " (you)" } else { "" }; - let _ = writeln!( - section, - "- {arrow} {eid}{self_marker} (score={score:.2}, Δ={delta:+.2})", - eid = d.entity_id, - score = d.score, - delta = d.delta - ); - } - } - - // 5. Refresh snapshots. - let now = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_secs_f64()) - .unwrap_or(0.0); - let snapshot_pairs: Vec<(String, f64)> = current - .iter() - .map(|r| (r.entity_id.clone(), r.score)) - .collect(); - if let Err(e) = update_snapshots_with_now(workspace_dir, &snapshot_pairs, now) { - log::warn!("[subconscious::situation_report::hotness] snapshot refresh failed: {e}"); - } - - section -} - -/// One row from `read_current_hotness`. `is_user` is OR'd across all -/// indexed nodes for the entity — true if any mention of this entity in -/// the tree resolved against the Composio identity registry. -struct CurrentHotness { - entity_id: String, - score: f64, - is_user: bool, -} - -/// Internal: a delta row with the carry-through identity flag. -struct HotnessDelta { - entity_id: String, - score: f64, - delta: f64, - is_user: bool, -} - -/// Read `(entity_id, last_hotness, is_user)` rows from the memory_tree -/// DB, filtering nulls. The `is_user` flag is computed via a correlated -/// subquery over `mem_tree_entity_index` (#1365): true iff any indexed -/// row for this entity has `is_user = 1`. -fn read_current_hotness(config: &Config) -> anyhow::Result> { - crate::openhuman::memory_store::chunks::store::with_connection(config, |conn| { - let mut stmt = conn.prepare( - "SELECT h.entity_id, - h.last_hotness, - EXISTS ( - SELECT 1 FROM mem_tree_entity_index i - WHERE i.entity_id = h.entity_id - AND i.is_user = 1 - ) AS is_user - FROM mem_tree_entity_hotness h - WHERE h.last_hotness IS NOT NULL - ORDER BY h.last_hotness DESC", - )?; - let rows = stmt - .query_map([], |row| { - let id: String = row.get(0)?; - let score: f64 = row.get(1)?; - let is_user_int: i64 = row.get(2)?; - Ok(CurrentHotness { - entity_id: id, - score, - is_user: is_user_int != 0, - }) - })? - .collect::, _>>()?; - Ok(rows) - }) -} - -/// Refresh the snapshot table. Wrapper that captures `now` once. -fn update_snapshots(workspace_dir: &Path, snapshots: &[(String, f64)]) -> anyhow::Result<()> { - let now = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_secs_f64()) - .unwrap_or(0.0); - update_snapshots_with_now(workspace_dir, snapshots, now) -} - -fn update_snapshots_with_now( - workspace_dir: &Path, - snapshots: &[(String, f64)], - now: f64, -) -> anyhow::Result<()> { - // The closure-based `with_connection` API does not expose a `&mut Connection` - // — we need one for the transaction in `replace_hotness_snapshots`. - // Open a direct handle just for this write. Schema is a no-op since - // the table already exists; we just need the migration to be applied - // (callers always go through `with_connection` first, so the migration - // ran by the time we get here). - let db_path = workspace_dir.join("subconscious").join("subconscious.db"); - let mut conn = rusqlite::Connection::open(&db_path)?; - reflection_store::replace_hotness_snapshots(&mut conn, snapshots, now)?; - Ok(()) -} diff --git a/src/openhuman/subconscious/situation_report/mod.rs b/src/openhuman/subconscious/situation_report/mod.rs index a367fe892..3b5bf1bba 100644 --- a/src/openhuman/subconscious/situation_report/mod.rs +++ b/src/openhuman/subconscious/situation_report/mod.rs @@ -9,16 +9,17 @@ //! reflection LLM can disambiguate body-text mentions — "Cyrus said X" //! is the user iff `Cyrus` (or the email/handle) appears in this list. //! 3. **Pending Tasks** (kept): subconscious task list from SQLite. -//! 4. **Hotness deltas** (new): top movers in `mem_tree_entity_hotness` -//! since the last tick. Highest signal density. Items tagged `(you)` -//! are the user's own identifiers (#1365). -//! 5. **Recently-sealed summaries** (new): rows from `mem_tree_summaries` +//! 4. **Recently-sealed summaries** (new): rows from `mem_tree_summaries` //! grouped by tree. -//! 6. **Latest global L0 digest** (new): most recent daily digest body. -//! 7. **`query_global` recap window** (new): since `last_tick_at`. -//! 8. **Recent reflections** (new): the last N reflections from the +//! 5. **Source-tree recap window** (new): recent source summaries since +//! `last_tick_at`. +//! 6. **Recent reflections** (new): the last N reflections from the //! subconscious store, used by the LLM as anti-double-emit context. //! +//! The hotness-deltas and global-L0-digest sections were removed with the +//! topic/global trees (the entity-hotness signal was a topic-curator +//! byproduct, and there is no longer a global digest node). +//! //! Sections are appended in priority order; truncation drops the tail //! when `token_budget` is exceeded. The legacy unified-store sections //! (`MemoryClient::list_documents`, `graph_query`) and the local-skills @@ -32,8 +33,6 @@ use crate::openhuman::config::Config; use super::reflection::Reflection; -mod digest; -mod hotness; mod query_window; pub(crate) mod reflections; mod summaries; @@ -74,25 +73,15 @@ pub async fn build_situation_report( let tasks_section = build_tasks_section(workspace_dir); append_section(&mut report, &mut remaining, &tasks_section); - // Section 3: hotness deltas (highest priority memory-tree signal). - let hotness_section = hotness::build_section(config, workspace_dir, last_tick_at).await; - append_section(&mut report, &mut remaining, &hotness_section); - - // Section 4: recently-sealed summaries since last tick. + // Section 4: recently-sealed source summaries since last tick. let summaries_section = summaries::build_section(config, last_tick_at).await; append_section(&mut report, &mut remaining, &summaries_section); - // Section 5: latest global L0 digest body — gated by `last_tick_at` - // so a digest the previous tick already saw doesn't get re-fed and - // re-cited (which was producing duplicate reflections). - let digest_section = digest::build_section(config, last_tick_at).await; - append_section(&mut report, &mut remaining, &digest_section); - - // Section 6: query_global recap window since last tick. + // Section 5: source-tree recap window since last tick. let recap_section = query_window::build_section(config, last_tick_at).await; append_section(&mut report, &mut remaining, &recap_section); - // Section 7: previous reflections (anti-double-emit context). + // Section 6: previous reflections (anti-double-emit context). let reflections_section = reflections::build_section(recent_reflections); append_section(&mut report, &mut remaining, &reflections_section); diff --git a/src/openhuman/subconscious/situation_report/query_window.rs b/src/openhuman/subconscious/situation_report/query_window.rs index d167239a9..ff17f3a9a 100644 --- a/src/openhuman/subconscious/situation_report/query_window.rs +++ b/src/openhuman/subconscious/situation_report/query_window.rs @@ -1,9 +1,11 @@ -//! `query_global` recap window section (#623). +//! Source-tree recap window section (#623). //! -//! Wraps `tree::retrieval::global::query_global` for the window between -//! `last_tick_at` and now. Translates seconds-since-last-tick into a -//! day window (rounded up to ≥ 1 so cold start still produces a useful -//! recap). +//! Walks the per-source summary trees (`retrieval::query_source`) for the +//! window between `last_tick_at` and now. Translates seconds-since-last-tick +//! into a day window (rounded up to ≥ 1 so cold start still produces a +//! useful recap). The global digest tree was removed — source trees plus +//! the entity index are the substrate, so the recap is reconstructed by +//! walking source-tree summaries across the window. //! //! Failures degrade gracefully — the section just reports //! "Recap unavailable" rather than aborting the tick. @@ -11,14 +13,17 @@ use std::fmt::Write; use crate::openhuman::config::Config; -use crate::openhuman::memory_tree::retrieval::global::query_global; +use crate::openhuman::memory_tree::retrieval::query_source; /// Cold-start fallback window when `last_tick_at` is unset. const COLD_START_DAYS: u32 = 7; -/// Minimum window — `query_global` ignores sub-day windows. +/// Minimum window — sub-day windows round up to one day. const MIN_WINDOW_DAYS: u32 = 1; +/// Max source summaries to pull into the recap window. +const MAX_RECAP_HITS: usize = 20; + pub async fn build_section(config: &Config, last_tick_at: f64) -> String { let window_days = compute_window_days(last_tick_at); log::debug!( @@ -26,7 +31,8 @@ pub async fn build_section(config: &Config, last_tick_at: f64) -> String { last_tick_at={last_tick_at}" ); - let resp = match query_global(config, window_days).await { + let resp = match query_source(config, None, None, Some(window_days), None, MAX_RECAP_HITS).await + { Ok(r) => r, Err(e) => { log::warn!("[subconscious::situation_report::query_window] failed: {e}"); @@ -34,11 +40,11 @@ pub async fn build_section(config: &Config, last_tick_at: f64) -> String { } }; - // Post-filter the hits against `last_tick_at`. `query_global` rounds - // up to whole days (`MIN_WINDOW_DAYS=1`), so even a 5-minute gap - // between ticks pulls back the same 24h window of digest summaries - // — those would re-feed the LLM the very content that produced the - // last tick's reflections, and the no-insert-time-dedupe path on + // Post-filter the hits against `last_tick_at`. The window rounds up to + // whole days (`MIN_WINDOW_DAYS=1`), so even a 5-minute gap between ticks + // pulls back the same 24h window of source summaries — those would + // re-feed the LLM the very content that produced the last tick's + // reflections, and the no-insert-time-dedupe path on // `persist_and_surface_reflections` would happily store the // duplicates. Cutoff semantics match `summaries::build_section`: // anything whose `time_range_end` is at or before `last_tick_at` has diff --git a/tests/agent_retrieval_e2e.rs b/tests/agent_retrieval_e2e.rs index 10c81b1c2..6291147b2 100644 --- a/tests/agent_retrieval_e2e.rs +++ b/tests/agent_retrieval_e2e.rs @@ -26,7 +26,7 @@ use openhuman_core::openhuman::memory::jobs::drain_until_idle; use openhuman_core::openhuman::memory_sync::canonicalize::chat::{ChatBatch, ChatMessage}; use openhuman_core::openhuman::memory_sync::canonicalize::email::{EmailMessage, EmailThread}; use openhuman_core::openhuman::tools::{ - MemoryTreeFetchLeavesTool, MemoryTreeQueryTopicTool, MemoryTreeSearchEntitiesTool, Tool, + MemoryTreeFetchLeavesTool, MemoryTreeSearchEntitiesTool, Tool, }; use serde_json::{json, Value}; use tempfile::TempDir; @@ -188,161 +188,6 @@ fn orchestrator_lists_memory_tree_tools() { } } -#[tokio::test] -async fn orchestrator_query_topic_tool_returns_alice_phoenix_hits() { - let (tmp, cfg) = test_config(); - - // ── Ingest the email thread + drain async extract jobs so the entity - // index is fully populated before retrieval. - ingest_email( - &cfg, - "gmail:thread-phoenix-1", - "alice", - vec![], - alice_phoenix_thread(), - ) - .await - .expect("ingest_email should succeed"); - drain_until_idle(&cfg) - .await - .expect("job queue should drain cleanly"); - - // Set workspace dir so config_rpc::load_config_with_timeout() inside the - // tool resolves to the same workspace we just ingested into. The tool - // wrappers always go through that loader (mirrors the production RPC - // handlers in retrieval/schemas.rs). - // - // Pointing OPENHUMAN_WORKSPACE at `tmp` (not `tmp/workspace`) makes - // `resolve_config_dir_for_workspace` derive `tmp/workspace` as the - // resolved workspace_dir — matching what we already passed into - // `ingest_email` via `cfg.workspace_dir`. - let _ws_guard = set_workspace_env(&tmp); - - // ── 1. search_entities resolves "alice" → email:alice@example.com. - // Mirrors the orchestrator prompt's "ALWAYS call this first when - // the user mentions someone by name" flow. - let search = MemoryTreeSearchEntitiesTool; - let search_args = json!({"query": "alice"}); - let search_res = search - .execute(search_args) - .await - .expect("search_entities should not error"); - assert!( - !search_res.is_error, - "search_entities returned an error result: {}", - search_res.output() - ); - let search_json: Value = - serde_json::from_str(&search_res.output()).expect("search output must be valid JSON"); - let matches = search_json - .as_array() - .expect("search_entities returns an array of EntityMatch"); - let alice = matches - .iter() - .find(|m| m.get("canonical_id").and_then(|v| v.as_str()) == Some("email:alice@example.com")) - .unwrap_or_else(|| panic!("search_entities did not return alice; got: {search_json:?}")); - assert!( - alice - .get("mention_count") - .and_then(|v| v.as_u64()) - .unwrap_or(0) - >= 1, - "alice should have at least one mention" - ); - - // ── 2. query_topic on alice's canonical id returns at least one hit - // referencing both her email and the phoenix migration content. - let topic_tool = MemoryTreeQueryTopicTool; - let topic_args = json!({"entity_id": "email:alice@example.com"}); - let topic_res = topic_tool - .execute(topic_args) - .await - .expect("query_topic should not error"); - assert!( - !topic_res.is_error, - "query_topic returned an error result: {}", - topic_res.output() - ); - let topic_json: Value = - serde_json::from_str(&topic_res.output()).expect("topic output must be valid JSON"); - let hits = topic_json - .get("hits") - .and_then(|v| v.as_array()) - .expect("query_topic must include `hits` array"); - assert!( - !hits.is_empty(), - "query_topic returned zero hits — expected at least one for alice" - ); - // Returning ANY hit at all from `query_topic("email:alice@example.com")` - // proves the entity index resolved the canonical id and hydrated nodes - // back. The leaf-level `entities` field on a chunk hit isn't populated - // synchronously by ingest — entity extraction lives in a separate async - // job stage that may not have populated leaf rows. Instead we assert on - // the hydrated content + source_ref so we still catch a regression where - // the chunk lookup returns garbage. - let any_phoenix = hits.iter().any(|h| { - h.get("content") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_lowercase() - .contains("phoenix") - }); - assert!( - any_phoenix, - "expected at least one query_topic hit with phoenix content; got: {topic_json:#}" - ); - let any_source_ref = hits - .iter() - .any(|h| h.get("source_ref").and_then(|v| v.as_str()).is_some()); - assert!( - any_source_ref, - "expected at least one hit to carry a `source_ref` for citation; got: {topic_json:#}" - ); - - // ── 3. fetch_leaves hydrates a leaf chunk — proves the citation path - // (LLM picks an id from a query_* hit, calls fetch_leaves to get - // the verbatim content + source_ref). - let leaf_id = hits - .iter() - .find_map(|h| { - if h.get("node_kind").and_then(|v| v.as_str()) == Some("leaf") { - h.get("node_id") - .and_then(|v| v.as_str()) - .map(str::to_string) - } else { - None - } - }) - .expect("alice's topic hits should include at least one leaf"); - let fetch_tool = MemoryTreeFetchLeavesTool; - let fetch_args = json!({"chunk_ids": [leaf_id.clone()]}); - let fetch_res = fetch_tool - .execute(fetch_args) - .await - .expect("fetch_leaves should not error"); - assert!( - !fetch_res.is_error, - "fetch_leaves returned an error result: {}", - fetch_res.output() - ); - let fetched: Value = - serde_json::from_str(&fetch_res.output()).expect("fetch output must be valid JSON"); - let fetched_arr = fetched.as_array().expect("fetch_leaves returns array"); - assert_eq!( - fetched_arr.len(), - 1, - "fetch_leaves should hydrate exactly the requested chunk" - ); - let content = fetched_arr[0] - .get("content") - .and_then(|v| v.as_str()) - .expect("fetched leaf must carry content"); - assert!( - !content.is_empty(), - "fetched leaf content must not be empty" - ); -} - // ── Cross-chat retrieval: chat A seeds facts; retrieve from chat B ────────── /// Ingests two distinct chat source IDs (simulating two separate chat channels) @@ -509,41 +354,25 @@ async fn fetch_leaves_hydrates_source_ref_for_cited_chunks() { let _ws_guard = set_workspace_env(&tmp); - // query_topic for alice's entity to get chunk hits with their ids. - let topic_tool = MemoryTreeQueryTopicTool; - let topic_res = topic_tool - .execute(json!({"entity_id": "email:alice@example.com"})) - .await - .expect("query_topic must not error"); - assert!( - !topic_res.is_error, - "query_topic error: {}", - topic_res.output() - ); + // List the ingested chunks directly to get leaf chunk ids with their refs. + let chunks = openhuman_core::openhuman::memory_store::chunks::store::list_chunks( + &cfg, + &openhuman_core::openhuman::memory_store::chunks::store::ListChunksQuery::default(), + ) + .expect("list_chunks must not error"); - let topic_json: Value = serde_json::from_str(&topic_res.output()).unwrap(); - let hits = topic_json - .get("hits") - .and_then(|v| v.as_array()) - .expect("query_topic response must have hits array"); + assert!(!chunks.is_empty(), "ingest must produce at least one chunk"); - assert!(!hits.is_empty(), "query_topic must return at least one hit"); - - // Collect the first leaf chunk id. - let leaf_ids: Vec = hits + // Collect the first couple of leaf chunk ids. + let leaf_ids: Vec = chunks .iter() - .filter(|h| h.get("node_kind").and_then(|v| v.as_str()) == Some("leaf")) - .filter_map(|h| { - h.get("node_id") - .and_then(|v| v.as_str()) - .map(str::to_string) - }) + .map(|chunk| chunk.id.clone()) .take(2) .collect(); assert!( !leaf_ids.is_empty(), - "at least one leaf hit required for fetch_leaves provenance test" + "at least one leaf chunk required for fetch_leaves provenance test" ); // fetch_leaves by chunk_ids and assert source_ref is populated. diff --git a/tests/json_rpc_e2e.rs b/tests/json_rpc_e2e.rs index 35189b970..26e6e3c76 100644 --- a/tests/json_rpc_e2e.rs +++ b/tests/json_rpc_e2e.rs @@ -2767,7 +2767,6 @@ async fn json_rpc_memory_tree_end_to_end() { "openhuman.memory_tree_ingest".to_string(), "openhuman.memory_tree_list_chunks".to_string(), "openhuman.memory_tree_get_chunk".to_string(), - "openhuman.memory_tree_trigger_digest".to_string(), ]; assert!( controllers.len() >= expected_methods.len(), diff --git a/tests/memory_artifacts_e2e.rs b/tests/memory_artifacts_e2e.rs index 04607bb20..401d3458c 100644 --- a/tests/memory_artifacts_e2e.rs +++ b/tests/memory_artifacts_e2e.rs @@ -87,7 +87,6 @@ async fn sync_raw_artifacts_and_mocked_summary_match_obsidian_contract() { body: "Phoenix migration launch window confirmed for Friday 22:00 UTC.", }, "slack-conn-slack-1", - None, ) .expect("stage mocked summary"); diff --git a/tests/memory_core_threads_raw_coverage_e2e.rs b/tests/memory_core_threads_raw_coverage_e2e.rs index a7e991479..9701ce6be 100644 --- a/tests/memory_core_threads_raw_coverage_e2e.rs +++ b/tests/memory_core_threads_raw_coverage_e2e.rs @@ -16,7 +16,6 @@ use openhuman_core::openhuman::config::Config; use openhuman_core::openhuman::memory::read_rpc::{ self, ChunkFilter, GraphMode, ResetTreeResponse, }; -use openhuman_core::openhuman::memory::tree_global::{digest, seal}; use openhuman_core::openhuman::memory::tree_source::get_or_create_source_tree; use openhuman_core::openhuman::memory::{ AppendConversationMessageRequest, ConversationMessageRecord, ConversationMessagesRequest, @@ -32,7 +31,6 @@ use openhuman_core::openhuman::memory_store::chunks::types::{ approx_token_count, chunk_id, Chunk, Metadata, SourceKind, SourceRef, }; use openhuman_core::openhuman::memory_store::content; -use openhuman_core::openhuman::memory_store::trees::registry::get_or_create_global_tree; use openhuman_core::openhuman::memory_store::trees::store as tree_store; use openhuman_core::openhuman::memory_store::trees::types::{SummaryNode, TreeKind}; use openhuman_core::openhuman::memory_tree::score::embed::pack_embedding; @@ -402,102 +400,6 @@ async fn memory_read_rpc_filters_graphs_scores_reset_and_wipe_seeded_rows() { assert!(wipe.dirs_removed.iter().any(|dir| dir == "raw")); } -#[tokio::test] -async fn global_digest_and_seal_cover_empty_emit_skip_cascade_and_queue_paths() { - let tmp = TempDir::new().unwrap(); - let cfg = config_in(&tmp); - let cascade_start = Utc.with_ymd_and_hms(2026, 5, 21, 0, 0, 0).unwrap(); - let digest_day = Utc.with_ymd_and_hms(2026, 6, 15, 0, 0, 0).unwrap(); - - assert!(matches!( - digest::end_of_day_digest(&cfg, digest_day.date_naive()) - .await - .unwrap(), - digest::DigestOutcome::EmptyDay - )); - - let global = get_or_create_global_tree(&cfg).unwrap(); - for i in 0..7 { - let node = daily_node( - &format!("summary:L0:round16-day-{i}"), - &global.id, - cascade_start + Duration::days(i), - ); - insert_summary(&cfg, &node, None); - let sealed = seal::append_daily_and_cascade(&cfg, &global, &node) - .await - .unwrap(); - if i < 6 { - assert!(sealed.is_empty()); - } else { - assert_eq!(sealed.len(), 1); - } - } - assert!(tree_store::get_buffer(&cfg, &global.id, 0) - .unwrap() - .is_empty()); - assert_eq!( - tree_store::get_buffer(&cfg, &global.id, 1) - .unwrap() - .item_ids - .len(), - 1 - ); - - let source_tree = get_or_create_source_tree(&cfg, "slack:#round16").unwrap(); - let source_summary = SummaryNode { - id: "summary:L1:source-round16".into(), - tree_id: source_tree.id.clone(), - tree_kind: TreeKind::Source, - level: 1, - parent_id: None, - child_ids: vec!["chunk-a".into()], - content: "Source contribution for the daily digest.".into(), - token_count: 72, - entities: vec!["person:alice".into()], - topics: vec!["digest".into()], - time_range_start: digest_day + Duration::hours(10), - time_range_end: digest_day + Duration::hours(11), - score: 0.88, - sealed_at: digest_day + Duration::hours(12), - deleted: false, - embedding: Some(vec![0.0; 1024]), - }; - insert_summary(&cfg, &source_summary, None); - with_connection(&cfg, |conn| { - conn.execute( - "UPDATE mem_tree_trees SET root_id = ?1, max_level = 1 WHERE id = ?2", - rusqlite::params![source_summary.id, source_tree.id], - )?; - Ok(()) - }) - .unwrap(); - - let emitted = digest::end_of_day_digest(&cfg, digest_day.date_naive()) - .await - .unwrap(); - let daily_id = match emitted { - digest::DigestOutcome::Emitted { - daily_id, - source_count, - sealed_ids, - } => { - assert_eq!(source_count, 1); - assert!(sealed_ids.is_empty()); - daily_id - } - other => panic!("expected emitted digest, got {other:?}"), - }; - assert!(matches!( - digest::end_of_day_digest(&cfg, digest_day.date_naive()) - .await - .unwrap(), - digest::DigestOutcome::Skipped { .. } - )); - - assert!(tree_store::get_summary(&cfg, &daily_id).unwrap().is_some()); -} - #[tokio::test] async fn thread_ops_welcome_migration_and_turn_state_cover_error_and_cleanup_paths() { let tmp = TempDir::new().unwrap(); diff --git a/tests/memory_sync_tree_round21_raw_coverage_e2e.rs b/tests/memory_sync_tree_round21_raw_coverage_e2e.rs index 1baab30c1..b0f0ef36d 100644 --- a/tests/memory_sync_tree_round21_raw_coverage_e2e.rs +++ b/tests/memory_sync_tree_round21_raw_coverage_e2e.rs @@ -547,7 +547,6 @@ fn seed_source_summary( body, }, scope, - None, ) .expect("stage summary body"); let embedding_blob = embedding.as_ref().map(|values| pack_embedding(values)); diff --git a/tests/memory_threads_raw_coverage_e2e.rs b/tests/memory_threads_raw_coverage_e2e.rs index 35c8b6e6e..33e034048 100644 --- a/tests/memory_threads_raw_coverage_e2e.rs +++ b/tests/memory_threads_raw_coverage_e2e.rs @@ -23,8 +23,8 @@ use openhuman_core::openhuman::config::Config; use openhuman_core::openhuman::embeddings::NoopEmbedding; use openhuman_core::openhuman::memory::query::{ MemoryQueryTool, MemoryTreeDrillDownTool, MemoryTreeFetchLeavesTool, - MemoryTreeIngestDocumentTool, MemoryTreeQueryGlobalTool, MemoryTreeQuerySourceTool, - MemoryTreeQueryTopicTool, MemoryTreeSearchEntitiesTool, MemoryTreeWalkTool, + MemoryTreeIngestDocumentTool, MemoryTreeQuerySourceTool, MemoryTreeSearchEntitiesTool, + MemoryTreeWalkTool, }; use openhuman_core::openhuman::memory::tools::{ MemoryForgetTool, MemoryRecallTool, MemoryStoreTool, @@ -55,9 +55,8 @@ use openhuman_core::openhuman::memory::{ }; use openhuman_core::openhuman::memory_queue::types::ReembedBackfillPayload; use openhuman_core::openhuman::memory_queue::{ - self, AppendBufferPayload, AppendTarget, DigestDailyPayload, ExtractChunkPayload, - FlushStalePayload, JobKind, JobStatus, NewJob, NodeRef, SealPayload, TopicRoutePayload, - DEFAULT_LOCK_DURATION_MS, + self, AppendBufferPayload, AppendTarget, ExtractChunkPayload, FlushStalePayload, JobKind, + JobStatus, NewJob, NodeRef, SealPayload, DEFAULT_LOCK_DURATION_MS, }; use openhuman_core::openhuman::memory_sources::readers::reader_for; use openhuman_core::openhuman::memory_sources::registry; @@ -988,13 +987,12 @@ fn memory_schema_registries_and_query_tool_metadata_cover_public_surfaces() { let legacy_tree_schemas = openhuman_core::openhuman::memory::schema::all_controller_schemas(); let legacy_tree_controllers = openhuman_core::openhuman::memory::schema::all_registered_controllers(); - assert_eq!(legacy_tree_schemas.len(), 20); + assert_eq!(legacy_tree_schemas.len(), 19); assert_eq!(legacy_tree_schemas.len(), legacy_tree_controllers.len()); for function in [ "ingest", "list_chunks", "get_chunk", - "trigger_digest", "memory_backfill_status", "list_sources", "search", @@ -1037,9 +1035,7 @@ fn memory_schema_registries_and_query_tool_metadata_cover_public_surfaces() { for tool in [ &MemoryTreeSearchEntitiesTool as &dyn Tool, - &MemoryTreeQueryTopicTool, &MemoryTreeQuerySourceTool, - &MemoryTreeQueryGlobalTool, &MemoryTreeDrillDownTool, &MemoryTreeFetchLeavesTool, &MemoryTreeIngestDocumentTool, @@ -1092,35 +1088,6 @@ fn memory_tree_policy_and_source_registry_write_metadata_mirror() { assert!(body.contains("kind: source")); assert!(body.contains("scope: \"gmail:user@example.com\"")); assert!(body.contains("last_sealed_at: null")); - - let cold = openhuman_core::openhuman::memory::tree_topic::hotness::hotness_at( - "email:cold@example.com", - &openhuman_core::openhuman::memory_store::trees::types::EntityIndexStats { - mention_count_30d: 0, - distinct_sources: 0, - last_seen_ms: None, - query_hits_30d: 0, - graph_centrality: None, - }, - now, - ); - assert_eq!(cold, 0.0); - let warm = openhuman_core::openhuman::memory::tree_topic::hotness::hotness_at( - "email:warm@example.com", - &stats, - now, - ); - assert!(warm > cold); - assert_eq!( - openhuman_core::openhuman::memory::tree_topic::hotness::recency_decay(None, now), - 0.0 - ); - assert!( - openhuman_core::openhuman::memory::tree_topic::hotness::hotness( - "email:live@example.com", - &stats - ) > 0.0 - ); } #[test] @@ -2354,8 +2321,6 @@ async fn memory_queue_and_tool_memory_public_stores_cover_persistence_edges() { JobKind::ExtractChunk, JobKind::AppendBuffer, JobKind::Seal, - JobKind::TopicRoute, - JobKind::DigestDaily, JobKind::FlushStale, JobKind::ReembedBackfill, ] { @@ -2407,17 +2372,6 @@ async fn memory_queue_and_tool_memory_public_stores_cover_persistence_edges() { .dedupe_key(), "seal:tree-1:2" ); - assert_eq!( - TopicRoutePayload { node: leaf.clone() }.dedupe_key(), - "topic_route:leaf:chunk-tool-memory" - ); - assert_eq!( - DigestDailyPayload { - date_iso: "2026-05-29".into() - } - .dedupe_key(), - "digest_daily:2026-05-29" - ); assert_eq!( FlushStalePayload { max_age_secs: Some(60) @@ -4383,7 +4337,7 @@ async fn memory_tree_retrieval_rpc_and_schema_wrappers_cover_empty_and_invalid_p openhuman_core::openhuman::memory_tree::retrieval::schemas::all_controller_schemas(); let controllers = openhuman_core::openhuman::memory_tree::retrieval::schemas::all_registered_controllers(); - assert_eq!(schemas.len(), 6); + assert_eq!(schemas.len(), 4); assert_eq!(schemas.len(), controllers.len()); assert_eq!( openhuman_core::openhuman::memory_tree::retrieval::schemas::schemas("missing").function, @@ -4427,28 +4381,6 @@ async fn memory_tree_retrieval_rpc_and_schema_wrappers_cover_empty_and_invalid_p .contains("unknown source kind") ); - let global = openhuman_core::openhuman::memory_tree::retrieval::rpc::query_global_rpc( - &config, - serde_json::from_value(json!({ "window_days": 3 })).expect("global alias"), - ) - .await - .expect("query global rpc"); - assert_eq!(global.value.total, 0); - - let topic = openhuman_core::openhuman::memory_tree::retrieval::rpc::query_topic_rpc( - &config, - openhuman_core::openhuman::memory_tree::retrieval::rpc::QueryTopicRequest { - entity_id: "email:alice@example.com".into(), - time_window_days: Some(30), - query: None, - limit: Some(5), - }, - ) - .await - .expect("query topic rpc"); - assert!(topic.value.hits.is_empty()); - assert!(topic.logs[0].contains("entity_kind=email")); - let search = openhuman_core::openhuman::memory_tree::retrieval::rpc::search_entities_rpc( &config, openhuman_core::openhuman::memory_tree::retrieval::rpc::SearchEntitiesRequest { @@ -4534,22 +4466,6 @@ async fn memory_query_backend_and_tree_flush_wrappers_cover_public_edges() { assert!(source_response.hits.is_empty()); assert_eq!(source_response.total, 0); - let global_result = MemoryTreeQueryGlobalTool - .execute(json!({ "time_window_days": 1 })) - .await - .expect("global query tool"); - let global_response: retrieval::types::QueryResponse = - serde_json::from_str(&global_result.text()).expect("global response json"); - assert!(global_response.hits.is_empty()); - - let missing_topic = MemoryTreeQueryTopicTool - .execute(json!({})) - .await - .unwrap_err(); - assert!(missing_topic - .to_string() - .contains("missing field `entity_id`")); - let kind_result = MemoryTreeQuerySourceTool .execute(json!({ "source_kind": "chat", "limit": 3 })) .await diff --git a/tests/memory_tree_sync_deep_raw_coverage_e2e.rs b/tests/memory_tree_sync_deep_raw_coverage_e2e.rs index 8e4dd5056..7b3aa47f9 100644 --- a/tests/memory_tree_sync_deep_raw_coverage_e2e.rs +++ b/tests/memory_tree_sync_deep_raw_coverage_e2e.rs @@ -29,7 +29,6 @@ use openhuman_core::openhuman::memory_store::chunks::types::{ chunk_id, Chunk, Metadata, SourceKind, SourceRef, }; use openhuman_core::openhuman::memory_store::trees::types::{SummaryNode, Tree, TreeKind}; -use openhuman_core::openhuman::memory_tree::retrieval::topic::query_topic; use openhuman_core::openhuman::memory_tree::score::embed::EMBEDDING_DIM; use openhuman_core::openhuman::memory_tree::score::extract::{ EntityExtractor, EntityKind, ExtractedEntities, LlmEntityExtractor, LlmExtractorConfig, @@ -38,8 +37,7 @@ use openhuman_core::openhuman::memory_tree::score::resolver::{canonicalise, Cano use openhuman_core::openhuman::memory_tree::score::store::{index_entity, lookup_entity}; use openhuman_core::openhuman::memory_tree::tree::rpc::{ backfill_status_rpc, get_chunk_rpc, ingest_rpc, list_chunks_rpc, pipeline_status_rpc, - set_enabled_rpc, trigger_digest_rpc, GetChunkRequest, IngestRequest, ListChunksRequest, - SetEnabledRequest, TriggerDigestRequest, + set_enabled_rpc, GetChunkRequest, IngestRequest, ListChunksRequest, SetEnabledRequest, }; use openhuman_core::openhuman::memory_tree::tree::set_summary_embedding; use openhuman_core::openhuman::memory_tree::tree::store as tree_store; @@ -435,137 +433,6 @@ async fn llm_extractor_recovers_spans_topics_strict_filters_and_retry_paths() { assert!(empty_after_bad_json.entities.is_empty()); } -#[tokio::test] -async fn topic_retrieval_merges_topic_root_leaf_and_summary_hits_with_rerank_edges() { - let tmp = TempDir::new().expect("tempdir"); - let cfg = test_config(&tmp); - let entity_id = "topic:phoenix"; - let summary = seed_topic_summary( - &cfg, - entity_id, - "summary:round18-root", - 0.30, - 1_700_000_000_000, - ); - set_summary_embedding(&cfg, &summary.id, &one_hot(0)).expect("set summary embedding"); - - let newer_chunk = sample_chunk( - &cfg, - "slack:#round18", - 1, - "Phoenix rollout update from Alice.", - 1_800_000_000_000, - ); - set_chunk_embedding(&cfg, &newer_chunk.id, &one_hot(1)).expect("set chunk embedding"); - let older_chunk = sample_chunk( - &cfg, - "slack:#round18", - 2, - "Older Phoenix note from Bob.", - 1_500_000_000_000, - ); - - let newer_entity = CanonicalEntity { - canonical_id: entity_id.into(), - kind: EntityKind::Topic, - surface: "Phoenix".into(), - span_start: 0, - span_end: 7, - score: 0.95, - }; - let older_entity = CanonicalEntity { - score: 0.90, - ..newer_entity.clone() - }; - index_entity( - &cfg, - &newer_entity, - &newer_chunk.id, - "leaf", - newer_chunk.metadata.timestamp.timestamp_millis(), - None, - ) - .expect("index newer leaf"); - index_entity( - &cfg, - &older_entity, - &older_chunk.id, - "leaf", - older_chunk.metadata.timestamp.timestamp_millis(), - Some("missing-tree"), - ) - .expect("index older leaf"); - index_entity( - &cfg, - &CanonicalEntity { - score: 0.80, - ..newer_entity.clone() - }, - &summary.id, - "summary", - summary.time_range_end.timestamp_millis(), - Some(&summary.tree_id), - ) - .expect("index summary duplicate"); - index_entity( - &cfg, - &newer_entity, - "missing-leaf-row", - "leaf", - Utc::now().timestamp_millis(), - None, - ) - .expect("index stale row"); - index_entity( - &cfg, - &newer_entity, - "missing-summary-row", - "summary", - Utc::now().timestamp_millis(), - Some(&summary.tree_id), - ) - .expect("index stale summary row"); - - let raw_hits = lookup_entity(&cfg, entity_id, Some(10)).expect("lookup entity"); - assert!(raw_hits.iter().any(|hit| hit.node_id == "missing-leaf-row")); - - let by_score = query_topic(&cfg, entity_id, None, None, 10) - .await - .expect("query topic"); - assert_eq!(by_score.hits.len(), 3); - assert_eq!(by_score.total, 3); - assert_eq!(by_score.hits[0].node_id, newer_chunk.id); - assert!(by_score.hits.iter().any(|hit| hit.node_id == summary.id)); - assert!(by_score - .hits - .iter() - .any(|hit| hit.node_id == older_chunk.id && hit.tree_scope == "slack:#round18")); - - let truncated = query_topic(&cfg, entity_id, None, None, 2) - .await - .expect("query limit"); - assert_eq!(truncated.hits.len(), 2); - assert!(truncated.truncated); - - let windowed = query_topic(&cfg, entity_id, Some(1), None, 10) - .await - .expect("query with narrow window"); - assert!(windowed.hits.is_empty()); - - let semantic = query_topic(&cfg, entity_id, None, Some("prefer embedded rows"), 10) - .await - .expect("semantic query"); - assert_eq!(semantic.hits.len(), 3); - assert!( - semantic - .hits - .iter() - .position(|hit| hit.node_id == older_chunk.id) - .unwrap() - > 0 - ); -} - #[tokio::test] async fn memory_tree_rpc_status_set_enabled_backfill_and_ingest_errors() { let tmp = TempDir::new().expect("tempdir"); @@ -653,36 +520,6 @@ async fn memory_tree_rpc_status_set_enabled_backfill_and_ingest_errors() { assert!(changed.changed); assert_eq!(changed.mode, "auto"); - let digest = trigger_digest_rpc( - &cfg, - TriggerDigestRequest { - date_iso: Some("2026-05-28".into()), - }, - ) - .await - .expect("trigger digest") - .value; - assert_eq!(digest.date_iso, "2026-05-28"); - let duplicate = trigger_digest_rpc( - &cfg, - TriggerDigestRequest { - date_iso: Some("2026-05-28".into()), - }, - ) - .await - .expect("trigger duplicate") - .value; - assert!(!duplicate.enqueued); - let invalid = trigger_digest_rpc( - &cfg, - TriggerDigestRequest { - date_iso: Some("05/28/2026".into()), - }, - ) - .await - .unwrap_err(); - assert!(invalid.contains("invalid date_iso")); - jobs::enqueue( &cfg, &NewJob::reembed_backfill(&ReembedBackfillPayload { diff --git a/tests/worker_c_modules_e2e.rs b/tests/worker_c_modules_e2e.rs index 99dbed70f..3e6d33d55 100644 --- a/tests/worker_c_modules_e2e.rs +++ b/tests/worker_c_modules_e2e.rs @@ -932,10 +932,7 @@ async fn memory_memory_tree_and_sources_controller_surfaces_are_reachable() { "openhuman.memory_tree_reset_tree", "openhuman.memory_tree_wipe_all", "openhuman.memory_tree_set_enabled", - "openhuman.memory_tree_trigger_digest", "openhuman.memory_tree_query_source", - "openhuman.memory_tree_query_global", - "openhuman.memory_tree_query_topic", "openhuman.memory_tree_search_entities", "openhuman.memory_tree_drill_down", "openhuman.memory_tree_fetch_leaves",