admin/openhuman
1
0
Fork 0
mirror of https://github.com/tinyhumansai/openhuman.git synced 2026-07-27 21:08:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(memory-queue): classify host-FS I/O errors to stop the tree_jobs Sentry flood (#4284) (#4287)

Browse Source
This commit is contained in:
oxoxDev
2026-06-30 19:53:36 +05:30
committed by GitHub
parent 928ce4d5c6
commit d7bee77e33
18 changed files with 434 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/src/lib/i18n/ar.ts
+2
View File
@@ -5903,6 +5903,8 @@ const messages: TranslationMap = {
'لا يتوفر مزوّد تلخيص لميزة إنشاء أشجار التلخيص. فعّل الذكاء الاصطناعي المحلي (Ollama)، أو فعّل تلخيص السحابة في الإعدادات → الذكاء الاصطناعي → الذاكرة.',
'memory.health.remediation.empty_input_refused':
'تم تخطي عنصر ذاكرة لأن نصه كان فارغًا. لا حاجة لأي إجراء — تستمر العناصر الجديدة في التضمين بشكل طبيعي.',
'memory.health.remediation.storage_unavailable':
'تعذّر على OpenHuman الكتابة إلى تخزين الذاكرة — يبدو أن القرص أو بطاقة SD معطوبة أو ممتلئة أو للقراءة فقط. تحقّق من محرك الأقراص وحرّر بعض المساحة؛ ستُستأنف معالجة الذاكرة تلقائيًا بمجرد أن يصبح التخزين قابلاً للكتابة مرة أخرى.',
'memory.health.remediation.transient':
'حدث خطأ مؤقت أدى إلى مقاطعة معالجة الذاكرة. ستتم إعادة المحاولة تلقائيًا.',
'memory.health.remediation.unknown':
app/src/lib/i18n/bn.ts
+2
View File
@@ -6026,6 +6026,8 @@ const messages: TranslationMap = {
'সারাংশ ট্রি তৈরির জন্য কোনও সারাংশ প্রদানকারী উপলব্ধ নেই। স্থানীয় AI (Ollama) সক্ষম করুন, অথবা সেটিংস → AI → মেমরিতে ক্লাউড সারাংশ সক্ষম করুন।',
'memory.health.remediation.empty_input_refused':
'একটি মেমরি আইটেম এড়িয়ে যাওয়া হয়েছে কারণ এর পাঠ্য খালি ছিল। কোনো পদক্ষেপের প্রয়োজন নেই — নতুন আইটেমগুলি স্বাভাবিকভাবে এমবেড করতে থাকে।',
'memory.health.remediation.storage_unavailable':
'OpenHuman তার মেমরি স্টোরেজে লিখতে পারছে না — ডিস্ক বা SD কার্ডটি ত্রুটিপূর্ণ, পূর্ণ বা শুধু-পঠনযোগ্য বলে মনে হচ্ছে। ড্রাইভটি পরীক্ষা করুন এবং কিছু জায়গা খালি করুন; স্টোরেজ আবার লেখার উপযোগী হলে মেমরি প্রক্রিয়াকরণ স্বয়ংক্রিয়ভাবে পুনরায় শুরু হবে।',
'memory.health.remediation.transient':
'একটি অস্থায়ী ত্রুটি মেমরি প্রক্রিয়াকরণে বাধা দিয়েছে। স্বয়ংক্রিয়ভাবে পুনরায় চেষ্টা করা হবে।',
'memory.health.remediation.unknown':
app/src/lib/i18n/de.ts
+2
View File
@@ -6180,6 +6180,8 @@ const messages: TranslationMap = {
'Für „Zusammenfassungsbäume erstellen” ist kein Zusammenfassungsanbieter verfügbar. Aktiviere die lokale KI (Ollama) oder aktiviere die Cloud-Zusammenfassung unter Einstellungen → KI → Speicher.',
'memory.health.remediation.empty_input_refused':
'Ein Speicherelement wurde übersprungen, weil sein Text leer war. Keine Aktion erforderlich — neue Einträge werden weiterhin normal eingebettet.',
'memory.health.remediation.storage_unavailable':
'OpenHuman kann nicht in seinen Speicher schreiben — die Festplatte oder SD-Karte scheint defekt, voll oder schreibgeschützt zu sein. Überprüfe das Laufwerk und gib Speicherplatz frei; die Speicherverarbeitung wird automatisch fortgesetzt, sobald der Speicher wieder beschreibbar ist.',
'memory.health.remediation.transient':
'Ein vorübergehender Fehler hat die Speicherverarbeitung unterbrochen. Es wird automatisch erneut versucht.',
'memory.health.remediation.unknown':
app/src/lib/i18n/en.ts
+2
View File
@@ -1041,6 +1041,8 @@ const en: TranslationMap = {
'No summarization provider is available for Build Summary Trees. Enable local AI (Ollama), or enable cloud summarization in Settings → AI → Memory.',
'memory.health.remediation.empty_input_refused':
'A memory item was skipped because its text was empty. No action needed — newer items continue to embed normally.',
'memory.health.remediation.storage_unavailable':
"OpenHuman can't write to its memory storage — the disk or SD card appears to be failing, full, or read-only. Check the drive and free up space; memory processing resumes automatically once storage is writable again.",
'memory.health.remediation.transient':
'A temporary error interrupted memory processing. It will retry automatically.',
'memory.health.remediation.unknown':
app/src/lib/i18n/es.ts
+2
View File
@@ -6142,6 +6142,8 @@ const messages: TranslationMap = {
'No hay ningún proveedor de resúmenes disponible para Crear árboles de resumen. Activa la IA local (Ollama) o activa el resumen en la nube en Configuración → IA → Memoria.',
'memory.health.remediation.empty_input_refused':
'Se omitió un elemento de memoria porque su texto estaba vacío. No se requiere ninguna acción — los elementos nuevos siguen incrustándose con normalidad.',
'memory.health.remediation.storage_unavailable':
'OpenHuman no puede escribir en su almacenamiento de memoria: el disco o la tarjeta SD parece estar fallando, lleno o de solo lectura. Revisa la unidad y libera espacio; el procesamiento de la memoria se reanudará automáticamente cuando el almacenamiento vuelva a permitir escritura.',
'memory.health.remediation.transient':
'Un error temporal interrumpió el procesamiento de la memoria. Se reintentará automáticamente.',
'memory.health.remediation.unknown':
app/src/lib/i18n/fr.ts
+2
View File
@@ -6163,6 +6163,8 @@ const messages: TranslationMap = {
"Aucun fournisseur de résumé n'est disponible pour Créer des arbres de résumé. Activez l'IA locale (Ollama) ou activez la synthèse cloud dans Paramètres → IA → Mémoire.",
'memory.health.remediation.empty_input_refused':
"Un élément de mémoire a été ignoré car son texte était vide. Aucune action requise — les nouveaux éléments continuent de s'intégrer normalement.",
'memory.health.remediation.storage_unavailable':
"OpenHuman ne peut pas écrire dans son stockage de mémoire — le disque ou la carte SD semble défaillant, plein ou en lecture seule. Vérifiez le lecteur et libérez de l'espace ; le traitement de la mémoire reprendra automatiquement une fois le stockage de nouveau accessible en écriture.",
'memory.health.remediation.transient':
'Une erreur temporaire a interrompu le traitement de la mémoire. Une nouvelle tentative aura lieu automatiquement.',
'memory.health.remediation.unknown':
app/src/lib/i18n/hi.ts
+2
View File
@@ -6029,6 +6029,8 @@ const messages: TranslationMap = {
'सारांश ट्री बनाएँ के लिए कोई सारांश प्रदाता उपलब्ध नहीं है। स्थानीय AI (Ollama) सक्षम करें, या सेटिंग्स → AI → मेमोरी में क्लाउड सारांश सक्षम करें।',
'memory.health.remediation.empty_input_refused':
'एक मेमोरी आइटम छोड़ दिया गया क्योंकि उसका टेक्स्ट खाली था। कोई कार्रवाई आवश्यक नहीं — नए आइटम सामान्य रूप से एम्बेड होते रहेंगे।',
'memory.health.remediation.storage_unavailable':
'OpenHuman अपने मेमोरी स्टोरेज में नहीं लिख पा रहा है — डिस्क या SD कार्ड खराब, भरा हुआ या केवल-पढ़ने योग्य लगता है। ड्राइव जाँचें और कुछ जगह खाली करें; स्टोरेज के फिर से लिखने योग्य होते ही मेमोरी प्रोसेसिंग अपने आप फिर शुरू हो जाएगी।',
'memory.health.remediation.transient':
'एक अस्थायी त्रुटि ने मेमोरी प्रोसेसिंग को बाधित किया। स्वचालित रूप से पुनः प्रयास किया जाएगा।',
'memory.health.remediation.unknown':
app/src/lib/i18n/id.ts
+2
View File
@@ -6045,6 +6045,8 @@ const messages: TranslationMap = {
'Tidak ada penyedia ringkasan yang tersedia untuk Buat Pohon Ringkasan. Aktifkan AI lokal (Ollama), atau aktifkan ringkasan cloud di Pengaturan → AI → Memori.',
'memory.health.remediation.empty_input_refused':
'Item memori dilewati karena teksnya kosong. Tidak diperlukan tindakan — item baru tetap disematkan seperti biasa.',
'memory.health.remediation.storage_unavailable':
'OpenHuman tidak dapat menulis ke penyimpanan memorinya — disk atau kartu SD tampaknya rusak, penuh, atau hanya-baca. Periksa drive dan kosongkan ruang; pemrosesan memori akan dilanjutkan secara otomatis setelah penyimpanan dapat ditulisi kembali.',
'memory.health.remediation.transient':
'Kesalahan sementara mengganggu pemrosesan memori. Akan dicoba lagi secara otomatis.',
'memory.health.remediation.unknown':
app/src/lib/i18n/it.ts
+2
View File
@@ -6129,6 +6129,8 @@ const messages: TranslationMap = {
"Nessun provider di riepilogo è disponibile per Crea alberi di riepilogo. Abilita l'IA locale (Ollama) o abilita il riepilogo cloud in Impostazioni → IA → Memoria.",
'memory.health.remediation.empty_input_refused':
'Un elemento di memoria è stato saltato perché il suo testo era vuoto. Nessuna azione necessaria — i nuovi elementi continuano a essere incorporati normalmente.',
'memory.health.remediation.storage_unavailable':
"OpenHuman non riesce a scrivere nella sua memoria — il disco o la scheda SD sembra guasto, pieno o di sola lettura. Controlla l'unità e libera spazio; l'elaborazione della memoria riprenderà automaticamente quando lo spazio di archiviazione tornerà scrivibile.",
'memory.health.remediation.transient':
"Un errore temporaneo ha interrotto l'elaborazione della memoria. Verrà riprovato automaticamente.",
'memory.health.remediation.unknown':
app/src/lib/i18n/ko.ts
+2
View File
@@ -5966,6 +5966,8 @@ const messages: TranslationMap = {
'요약 트리 만들기에 사용할 수 있는 요약 제공자가 없습니다. 로컬 AI(Ollama)를 활성화하거나, 설정 → AI → 메모리에서 클라우드 요약을 활성화하세요.',
'memory.health.remediation.empty_input_refused':
'텍스트가 비어 있어 메모리 항목이 건너뛰어졌습니다. 조치가 필요하지 않습니다 — 새 항목은 정상적으로 임베딩됩니다.',
'memory.health.remediation.storage_unavailable':
'OpenHuman이 메모리 저장소에 쓸 수 없습니다 — 디스크 또는 SD 카드가 손상되었거나 가득 찼거나 읽기 전용인 것 같습니다. 드라이브를 확인하고 공간을 확보하세요. 저장소에 다시 쓸 수 있게 되면 메모리 처리가 자동으로 재개됩니다.',
'memory.health.remediation.transient':
'일시적인 오류로 메모리 처리가 중단되었습니다. 자동으로 다시 시도됩니다.',
'memory.health.remediation.unknown':
app/src/lib/i18n/pl.ts
+2
View File
@@ -6113,6 +6113,8 @@ const messages: TranslationMap = {
'Brak dostępnego dostawcy podsumowań dla funkcji Twórz drzewa podsumowań. Włącz lokalną AI (Ollama) lub włącz podsumowywanie w chmurze w Ustawienia → AI → Pamięć.',
'memory.health.remediation.empty_input_refused':
'Pominięto element pamięci, ponieważ jego tekst był pusty. Żadne działanie nie jest wymagane — nowe elementy są nadal osadzane normalnie.',
'memory.health.remediation.storage_unavailable':
'OpenHuman nie może zapisywać w pamięci — dysk lub karta SD wydaje się uszkodzona, pełna lub tylko do odczytu. Sprawdź dysk i zwolnij miejsce; przetwarzanie pamięci wznowi się automatycznie, gdy zapis znów będzie możliwy.',
'memory.health.remediation.transient':
'Tymczasowy błąd przerwał przetwarzanie pamięci. Ponowna próba nastąpi automatycznie.',
'memory.health.remediation.unknown':
app/src/lib/i18n/pt.ts
+2
View File
@@ -6124,6 +6124,8 @@ const messages: TranslationMap = {
'Nenhum provedor de resumo está disponível para Criar árvores de resumo. Ative a IA local (Ollama) ou ative o resumo na nuvem em Configurações → IA → Memória.',
'memory.health.remediation.empty_input_refused':
'Um item de memória foi ignorado porque o texto estava vazio. Nenhuma ação necessária — itens novos continuam a ser incorporados normalmente.',
'memory.health.remediation.storage_unavailable':
'O OpenHuman não consegue gravar no seu armazenamento de memória — o disco ou cartão SD parece estar com falha, cheio ou somente leitura. Verifique a unidade e libere espaço; o processamento da memória será retomado automaticamente assim que o armazenamento voltar a permitir gravação.',
'memory.health.remediation.transient':
'Um erro temporário interrompeu o processamento da memória. Será repetido automaticamente.',
'memory.health.remediation.unknown':
app/src/lib/i18n/ru.ts
+2
View File
@@ -6083,6 +6083,8 @@ const messages: TranslationMap = {
'Нет доступного поставщика суммаризации для «Построить деревья сводок». Включите локальный ИИ (Ollama) или включите облачную суммаризацию в разделе Настройки → ИИ → Память.',
'memory.health.remediation.empty_input_refused':
'Элемент памяти пропущен, так как его текст был пуст. Действия не требуются — новые элементы продолжают встраиваться как обычно.',
'memory.health.remediation.storage_unavailable':
'OpenHuman не может записывать в хранилище памяти — диск или SD-карта, похоже, неисправны, заполнены или доступны только для чтения. Проверьте накопитель и освободите место; обработка памяти возобновится автоматически, как только запись снова станет возможной.',
'memory.health.remediation.transient':
'Временная ошибка прервала обработку памяти. Повтор произойдёт автоматически.',
'memory.health.remediation.unknown':
app/src/lib/i18n/zh-CN.ts
+2
View File
@@ -5720,6 +5720,8 @@ const messages: TranslationMap = {
'没有可用于构建摘要树的摘要提供方。请启用本地 AI（Ollama），或在设置 → AI → 记忆中启用云端摘要。',
'memory.health.remediation.empty_input_refused':
'由于文本为空，一项记忆已被跳过。无需操作 — 新条目继续正常嵌入。',
'memory.health.remediation.storage_unavailable':
'OpenHuman 无法写入其记忆存储 — 磁盘或 SD 卡似乎已损坏、已满或为只读。请检查驱动器并释放空间；存储恢复可写后，记忆处理将自动继续。',
'memory.health.remediation.transient': '临时错误中断了记忆处理。将自动重试。',
'memory.health.remediation.unknown': '记忆处理遇到问题。请在设置 → AI 中检查配置。',
// Chat — agent-generated artifacts (#2779)
src/openhuman/memory_queue/worker.rs
+178 -3
View File
@@ -23,7 +23,9 @@ use crate::openhuman::memory_queue::store::{
recover_stale_locks, release_running_locks, DEFAULT_LOCK_DURATION_MS,
};
use crate::openhuman::memory_queue::types::{Job, JobKind, JobOutcome};
use crate::openhuman::memory_tree::health::PipelineFailure;
use crate::openhuman::memory_tree::health::{
clear_storage_degraded, mark_storage_degraded, FailureCode, PipelineFailure,
};
/// Number of concurrent job-worker tasks. Each worker claims one job
/// at a time via `claim_next` (atomic UPDATE under SQLite WAL with
@@ -55,6 +57,15 @@ static STARTED: std::sync::Once = std::sync::Once::new();
/// (Sentry TAURI-RUST-E93: 1,633 events in ~17 min from one host).
static CORRUPT_REPORTED: AtomicBool = AtomicBool::new(false);
/// Process-wide latch so a persistent host-filesystem failure (EIO/ENOSPC/
/// EROFS on the memory_tree dir/DB path) is reported to Sentry **once**, not on
/// every poll from every worker. Set on the first host-I/O failure; cleared on
/// the next successful claim (storage recovered) so a genuinely-new, later
/// failure can still page once. Without this, 4 workers re-polling a dead disk
/// flood the dashboard (Sentry CORE-RUST-19J: ~10k events in ~50 min from one
/// Raspberry Pi with a failing SD card).
static STORAGE_IO_REPORTED: AtomicBool = AtomicBool::new(false);
/// Notify any idle workers so they re-poll immediately instead of waiting
/// out [`POLL_INTERVAL`]. Cheap no-op before [`start`] has run.
pub fn wake_workers() {
@@ -116,8 +127,24 @@ pub fn start(config: Config) {
tokio::spawn(async move {
loop {
match run_once(&cfg).await {
Ok(true) => continue,
Ok(false) => {
Ok(processed) => {
// A successful claim proves the memory_tree DB
// opened, so the host filesystem is healthy again.
// Clear any prior host-I/O degradation so the status
// banner self-heals and a genuinely-new later failure
// can page once more. Guarded on the latch swap so the
// clear + info log fire only on the recovery edge, not
// every poll.
if STORAGE_IO_REPORTED.swap(false, Ordering::Relaxed) {
clear_storage_degraded();
log::info!(
"[memory::jobs] worker {idx} storage recovered; \
cleared host-I/O degraded flag"
);
}
if processed {
continue;
}
tokio::select! {
_ = notify.notified() => {}
_ = tokio::time::sleep(POLL_INTERVAL) => {}
@@ -184,6 +211,42 @@ pub fn start(config: Config) {
// rebuild succeeds.
recover_corrupt_db_once(idx, &err, &cfg);
tokio::time::sleep(Duration::from_secs(300)).await;
} else if is_host_io_error(&err) {
// Persistent host-filesystem failure (EIO 5 /
// ENOSPC 28 / EROFS 30) creating or opening the
// memory_tree dir/DB — e.g. a failing or
// disconnected SD card, or a volume the kernel
// remounted read-only. Like SQLITE_FULL/CORRUPT
// this is a persistent host condition only the
// user can clear (reseat/replace/free storage),
// so re-polling every second and paging Sentry on
// each failure floods the dashboard (CORE-RUST-19J:
// ~10k events in ~50 min from one Raspberry Pi).
//
// Resolution, not just suppression: mark the
// memory_tree degraded with `StorageUnavailable`
// so the status panel shows the user an actionable
// "check your disk" banner — they own the only
// lever. Report ONCE (process-wide latch) for dev
// telemetry, then back off 300s and stay silent.
// Jobs stay `ready` and resume when storage
// returns; the degraded flag + latch clear on the
// next successful claim.
mark_storage_degraded(FailureCode::StorageUnavailable);
if !STORAGE_IO_REPORTED.swap(true, Ordering::Relaxed) {
crate::core::observability::report_error(
&err,
"memory",
"tree_jobs_worker_host_io",
&[("worker_idx", &idx.to_string())],
);
}
log::warn!(
"[memory::jobs] worker {idx} hit host filesystem I/O error \
(EIO/ENOSPC/EROFS — failing or read-only storage), \
backing off 300s: {err:#}"
);
tokio::time::sleep(Duration::from_secs(300)).await;
} else {
crate::core::observability::report_error(
&err,
@@ -466,6 +529,40 @@ fn is_sqlite_corrupt(err: &anyhow::Error) -> bool {
msg.contains("database disk image is malformed") || msg.contains("file is not a database")
}
/// Classify whether an error is a **persistent host-filesystem failure** —
/// `std::fs::create_dir_all` / file open returning an OS-level I/O error on the
/// memory_tree path. Matches the three persistent, user-only-fixable POSIX
/// codes:
/// - **EIO `5`** — the block device can't service I/O (failing/disconnected SD
/// card or USB drive). This is the CORE-RUST-19J signal:
/// `"Failed to create memory_tree dir: …: Input/output error (os error 5)"`.
/// - **ENOSPC `28`** — no space left at the *filesystem* layer (distinct from
/// `SQLITE_FULL`, which is the SQLite-write code handled separately).
/// - **EROFS `30`** — read-only filesystem; Linux remounts a failing SD card
/// read-only, so this is the common next stage of the same Pi failure.
///
/// Unlike the SQLite busy/transient family, these are persistent host
/// conditions the app has no lever to fix: re-polling every second and paging
/// Sentry on each failure turns one dead disk into a flood. The worker backs
/// off long, surfaces a `StorageUnavailable` degradation to the user, and stays
/// silent after a single report.
///
/// Matching on `raw_os_error()` is platform-stable. The text fallback covers
/// the case where the `io::Error` was flattened to a plain `anyhow!` string
/// across `.context()` layers (anyhow renders `"… (os error N)"`); it anchors
/// on the unambiguous os-error number, not the loose phrase, so a network
/// "input/output" string can't false-positive. A non-OS error has
/// `raw_os_error() == None` and matches neither path.
fn is_host_io_error(err: &anyhow::Error) -> bool {
if let Some(io_err) = err.downcast_ref::<std::io::Error>() {
if matches!(io_err.raw_os_error(), Some(5) | Some(28) | Some(30)) {
return true;
}
}
let msg = format!("{err:#}").to_ascii_lowercase();
msg.contains("(os error 5)") || msg.contains("(os error 28)") || msg.contains("(os error 30)")
}
/// Handle a confirmed `SQLITE_CORRUPT` failure from the worker loop: report it
/// to Sentry **once** (process-wide [`CORRUPT_REPORTED`] latch, not per-poll
/// across the workers) and drive the quarantine+rebuild recovery in
@@ -865,6 +962,84 @@ mod tests {
)));
}
// ── is_host_io_error tests (CORE-RUST-19J) ───────────────────────────────
/// EIO (`os error 5`) is the CORE-RUST-19J signal: `create_dir_all` on a
/// failing/disconnected SD card. Must classify so the worker surfaces a
/// `StorageUnavailable` degradation + backs off long instead of paging
/// Sentry every second.
#[test]
fn is_host_io_error_matches_eio() {
let err = anyhow::Error::from(std::io::Error::from_raw_os_error(5));
assert!(is_host_io_error(&err));
}
/// ENOSPC (28, filesystem-level out-of-space on `create_dir`) and EROFS (30,
/// kernel-remounted-read-only — the common next stage of a dying SD card)
/// are the same persistent, user-only-fixable host condition.
#[test]
fn is_host_io_error_matches_enospc_and_erofs() {
for code in [28, 30] {
let err = anyhow::Error::from(std::io::Error::from_raw_os_error(code));
assert!(
is_host_io_error(&err),
"os error {code} must classify as host I/O"
);
}
}
/// The production shape: the `io::Error` bubbles out of `open_and_init`
/// wrapped in `.with_context("Failed to create memory_tree dir: …")` then
/// the `with_connection` layer. The downcast must still find it through the
/// anyhow context chain (regression guard: don't rely on the top-level type).
#[test]
fn is_host_io_error_matches_through_context_layers() {
let wrapped = anyhow::Error::from(std::io::Error::from_raw_os_error(5))
.context("Failed to create memory_tree dir: /home/x/.openhuman-workspace/workspace/memory_tree")
.context("with_connection closure failed");
assert!(is_host_io_error(&wrapped));
}
/// Text fallback: when no `io::Error` is available to downcast (flattened to
/// a plain `anyhow!` string), the exact flattened CORE-RUST-19J message is
/// still classified via the os-error-number anchor.
#[test]
fn is_host_io_error_text_fallback() {
let err = anyhow::anyhow!(
"Failed to create memory_tree dir: /home/x/.openhuman-workspace/workspace/memory_tree: \
Input/output error (os error 5)"
);
assert!(is_host_io_error(&err));
}
/// Permission-denied (13), not-found (2), a SQLite disk-full failure (its
/// own arm), and unrelated errors must NOT be swallowed as host I/O — those
/// are real bugs / handled elsewhere and must keep reporting.
#[test]
fn is_host_io_error_does_not_match_other_errors() {
// EACCES — a genuine permission bug, not failing hardware.
assert!(!is_host_io_error(&anyhow::Error::from(
std::io::Error::from_raw_os_error(13)
)));
// ENOENT.
assert!(!is_host_io_error(&anyhow::Error::from(
std::io::Error::from_raw_os_error(2)
)));
// SQLITE_FULL stays in is_sqlite_disk_full's arm, not here.
let disk_full = rusqlite::Error::SqliteFailure(
rusqlite::ffi::Error {
code: rusqlite::ErrorCode::DiskFull,
extended_code: 13,
},
Some("database or disk is full".into()),
);
assert!(!is_host_io_error(&anyhow::Error::from(disk_full)));
// Unrelated.
assert!(!is_host_io_error(&anyhow::anyhow!(
"upstream returned 500: internal server error"
)));
}
/// The worker's corruption arm must quarantine a malformed image and rebuild
/// an empty, queryable schema so the queue resumes — exercising the
/// report-once + recover path the live loop runs.
src/openhuman/memory_tree/health/doctor.rs
+55
View File
@@ -109,6 +109,31 @@ pub fn run_doctor(config: &Config) -> DoctorReport {
let mut stages = Vec::new();
// 0. Storage health — the foundational layer. If the host filesystem can't
// service the memory_tree path (EIO/ENOSPC/EROFS on dir-create / DB
// open, flagged by the queue worker's host-I/O arm), nothing downstream
// can run. Pushed FIRST so it becomes `first_blocking_cause` and the
// status panel leads the user to the disk fix instead of a misleading
// "configure embeddings" message. Only the user owns this lever.
if degraded.storage {
let cause = degraded
.cause
.clone()
.filter(|c| c.code == FailureCode::StorageUnavailable)
.unwrap_or_else(|| PipelineFailure::new(FailureCode::StorageUnavailable));
stages.push(StageHealth::bad(
"storage",
cause,
"memory storage path is unavailable — the host filesystem returned a \
persistent I/O error (failing/read-only disk or SD card)",
));
} else {
stages.push(StageHealth::ok(
"storage",
"memory storage path is writable",
));
}
// 1. Routing/config sanity — is *any* embeddings provider configured?
// (`build_write_embedder` skips embedding when none is, so this is the
// most common "empty wiki" root cause.)
@@ -399,6 +424,36 @@ mod tests {
);
}
/// A host-FS storage failure must surface as the doctor's
/// `first_blocking_cause` (stage 0), outranking everything else — even a
/// fully-misconfigured embeddings setup — so the user is told to fix their
/// disk, not their provider config.
#[test]
fn storage_failure_is_first_blocking_cause() {
let _g = super::super::test_guard();
let (_tmp, mut cfg) = test_config();
// Deliberately also break embeddings so we prove storage wins.
cfg.embeddings_provider = None;
cfg.local_ai.runtime_enabled = false;
super::super::mark_storage_degraded(FailureCode::StorageUnavailable);
let report = run_doctor(&cfg);
assert!(!report.healthy);
let cause = report.first_blocking_cause.expect("should have a cause");
assert_eq!(
cause.code,
FailureCode::StorageUnavailable,
"storage must outrank the embeddings misconfig"
);
let storage = report
.stages
.iter()
.find(|s| s.stage == "storage")
.expect("storage stage present");
assert!(!storage.ok);
assert!(report.degraded.storage);
}
#[test]
fn report_serde_roundtrips() {
let _g = super::super::test_guard();
src/openhuman/memory_tree/health/mod.rs
+109 -6
View File
@@ -81,6 +81,15 @@ pub enum FailureCode {
/// `OpenHumanCloudEmbedding::embed` starts with
/// `"<name> embed: refusing empty/whitespace input ..."`.
EmptyInputRefused,
/// The host filesystem cannot service the memory_tree path — `create_dir`
/// / DB open returned a persistent OS-level I/O error (EIO `5`, ENOSPC
/// `28`, EROFS `30`), e.g. a failing/disconnected SD card or a volume the
/// kernel remounted read-only. Unrecoverable from inside the app: only the
/// user can reseat/replace/free the storage. Distinct from the embeddings
/// provider faults above and from the SQLite-level `SQLITE_FULL` /
/// `SQLITE_CORRUPT` handled in the queue worker — this is the
/// directory/DB-init layer below them.
StorageUnavailable,
/// Catch-all transient failure (network 5xx, timeout, truncated JSON).
Transient,
}
@@ -98,6 +107,7 @@ impl FailureCode {
Self::ExtractionTimeout => "extraction_timeout",
Self::SummarizerUnavailable => "summarizer_unavailable",
Self::EmptyInputRefused => "empty_input_refused",
Self::StorageUnavailable => "storage_unavailable",
Self::Transient => "transient",
}
}
@@ -113,6 +123,7 @@ impl FailureCode {
"extraction_timeout" => Self::ExtractionTimeout,
"summarizer_unavailable" => Self::SummarizerUnavailable,
"empty_input_refused" => Self::EmptyInputRefused,
"storage_unavailable" => Self::StorageUnavailable,
"transient" => Self::Transient,
_ => return None,
})
@@ -139,6 +150,7 @@ impl FailureCode {
Self::ExtractionTimeout => "memory.health.remediation.extraction_timeout",
Self::SummarizerUnavailable => "memory.health.remediation.summarizer_unavailable",
Self::EmptyInputRefused => "memory.health.remediation.empty_input_refused",
Self::StorageUnavailable => "memory.health.remediation.storage_unavailable",
Self::Transient => "memory.health.remediation.transient",
}
}
@@ -333,6 +345,13 @@ pub struct DegradedState {
/// True when extraction yielded empty across the board so the wiki has
/// no entity/topic structure.
pub structure: bool,
/// True when the memory_tree's own storage path is unusable — the host
/// filesystem returned a persistent I/O error on dir-create / DB open
/// (EIO/ENOSPC/EROFS). This is the most severe degradation: the pipeline
/// can't even open its DB, so nothing else runs. `#[serde(default)]` keeps
/// the wire format backward-compatible (older clients omit it → `false`).
#[serde(default)]
pub storage: bool,
/// The cause of the most significant degradation, when known.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cause: Option<PipelineFailure>,
@@ -341,7 +360,7 @@ pub struct DegradedState {
impl DegradedState {
/// True when any degradation is present.
pub fn is_degraded(&self) -> bool {
self.semantic_recall || self.structure
self.semantic_recall || self.structure || self.storage
}
}
@@ -360,12 +379,17 @@ use std::sync::atomic::{AtomicBool, AtomicU8, Ordering};
static SEMANTIC_RECALL_DEGRADED: AtomicBool = AtomicBool::new(false);
static STRUCTURE_DEGRADED: AtomicBool = AtomicBool::new(false);
/// The host filesystem can't service the memory_tree path (EIO/ENOSPC/EROFS).
/// Set by the queue worker's host-I/O arm; cleared on the next successful
/// claim (storage recovered). Most severe — outranks recall/structure.
static STORAGE_DEGRADED: AtomicBool = AtomicBool::new(false);
/// Per-flag degradation cause as a `FailureCode` discriminant (0 = none).
/// Tracked separately per flag so clearing one degradation can't leave the
/// other reporting a stale cause (e.g. mark recall, mark structure, clear
/// structure → recall must still report its OWN cause, not structure's).
static SEMANTIC_RECALL_CAUSE: AtomicU8 = AtomicU8::new(0);
static STRUCTURE_CAUSE: AtomicU8 = AtomicU8::new(0);
static STORAGE_CAUSE: AtomicU8 = AtomicU8::new(0);
fn code_to_u8(code: FailureCode) -> u8 {
match code {
@@ -379,6 +403,7 @@ fn code_to_u8(code: FailureCode) -> u8 {
FailureCode::SummarizerUnavailable => 8,
FailureCode::Transient => 9,
FailureCode::EmptyInputRefused => 10,
FailureCode::StorageUnavailable => 11,
}
}
@@ -394,6 +419,7 @@ fn u8_to_code(v: u8) -> Option<FailureCode> {
8 => FailureCode::SummarizerUnavailable,
9 => FailureCode::Transient,
10 => FailureCode::EmptyInputRefused,
11 => FailureCode::StorageUnavailable,
_ => return None,
})
}
@@ -428,6 +454,24 @@ pub fn clear_structure_degraded() {
STRUCTURE_CAUSE.store(0, Ordering::Relaxed);
}
/// Record that the memory_tree storage path is unusable — the host filesystem
/// returned a persistent I/O error (EIO/ENOSPC/EROFS) on dir-create / DB open.
/// `cause` is typically [`FailureCode::StorageUnavailable`]. Set by the queue
/// worker's host-I/O arm so the status surface tells the user to check their
/// disk; idempotent / cheap.
pub fn mark_storage_degraded(cause: FailureCode) {
STORAGE_DEGRADED.store(true, Ordering::Relaxed);
STORAGE_CAUSE.store(code_to_u8(cause), Ordering::Relaxed);
}
/// Clear the storage degraded flag — call when a claim succeeds (the DB opened,
/// so the host filesystem recovered), so the surface self-heals. Clears only
/// this flag's cause.
pub fn clear_storage_degraded() {
STORAGE_DEGRADED.store(false, Ordering::Relaxed);
STORAGE_CAUSE.store(0, Ordering::Relaxed);
}
/// Test-only serialization + reset for the process-global degraded flags.
///
/// The flags are a single process-wide signal, so tests across *different*
@@ -444,8 +488,10 @@ pub fn test_guard() -> std::sync::MutexGuard<'static, ()> {
.unwrap_or_else(|p| p.into_inner());
SEMANTIC_RECALL_DEGRADED.store(false, Ordering::Relaxed);
STRUCTURE_DEGRADED.store(false, Ordering::Relaxed);
STORAGE_DEGRADED.store(false, Ordering::Relaxed);
SEMANTIC_RECALL_CAUSE.store(0, Ordering::Relaxed);
STRUCTURE_CAUSE.store(0, Ordering::Relaxed);
STORAGE_CAUSE.store(0, Ordering::Relaxed);
g
}
@@ -455,11 +501,16 @@ pub fn test_guard() -> std::sync::MutexGuard<'static, ()> {
pub fn current_degraded_state() -> DegradedState {
let semantic_recall = SEMANTIC_RECALL_DEGRADED.load(Ordering::Relaxed);
let structure = STRUCTURE_DEGRADED.load(Ordering::Relaxed);
let storage = STORAGE_DEGRADED.load(Ordering::Relaxed);
// Each flag carries its own cause; pick the most actionable one to surface.
// Structure degradation (extraction failing → empty wiki) is reported first
// because it's the more severe "built but useless" symptom; otherwise the
// recall cause. Either way the cause reflects a CURRENTLY-active flag.
let cause = if structure {
// Storage degradation is reported first — the host FS can't open the DB, so
// it's the foundational failure beneath both recall and structure (no point
// telling the user "configure embeddings" when the disk is dying). Then
// structure (extraction failing → empty wiki), then recall. Either way the
// cause reflects a CURRENTLY-active flag.
let cause = if storage {
u8_to_code(STORAGE_CAUSE.load(Ordering::Relaxed)).map(PipelineFailure::new)
} else if structure {
u8_to_code(STRUCTURE_CAUSE.load(Ordering::Relaxed)).map(PipelineFailure::new)
} else if semantic_recall {
u8_to_code(SEMANTIC_RECALL_CAUSE.load(Ordering::Relaxed)).map(PipelineFailure::new)
@@ -469,6 +520,7 @@ pub fn current_degraded_state() -> DegradedState {
DegradedState {
semantic_recall,
structure,
storage,
cause,
}
}
@@ -477,7 +529,7 @@ pub fn current_degraded_state() -> DegradedState {
mod tests {
use super::*;
const ALL_CODES: [FailureCode; 10] = [
const ALL_CODES: [FailureCode; 11] = [
FailureCode::BudgetExhausted,
FailureCode::AuthMissing,
FailureCode::AuthInvalid,
@@ -487,6 +539,7 @@ mod tests {
FailureCode::ExtractionTimeout,
FailureCode::SummarizerUnavailable,
FailureCode::EmptyInputRefused,
FailureCode::StorageUnavailable,
FailureCode::Transient,
];
@@ -751,4 +804,54 @@ mod tests {
assert!(!s.is_degraded());
assert!(s.cause.is_none());
}
/// `StorageUnavailable` is the foundational host-FS failure: unrecoverable,
/// with its own remediation key.
#[test]
fn storage_unavailable_is_unrecoverable_with_key() {
let f = PipelineFailure::new(FailureCode::StorageUnavailable);
assert_eq!(f.class, FailureClass::Unrecoverable);
assert!(f.is_unrecoverable());
assert_eq!(
f.remediation_key,
"memory.health.remediation.storage_unavailable"
);
// discriminant round-trips through the per-flag u8 mapping.
assert_eq!(
u8_to_code(code_to_u8(FailureCode::StorageUnavailable)),
Some(FailureCode::StorageUnavailable)
);
}
/// Storage degradation outranks both structure and recall in
/// `current_degraded_state` — the host can't open the DB, so the disk fix
/// is the one actionable thing to surface. Clearing storage falls back to
/// the next-most-severe active cause (structure), each keeping its own.
#[test]
fn storage_degradation_outranks_structure_and_recall() {
let _g = test_guard(); // resets all flags + causes
mark_semantic_recall_degraded(FailureCode::EmbeddingsUnconfigured);
mark_structure_degraded(FailureCode::ExtractionTimeout);
mark_storage_degraded(FailureCode::StorageUnavailable);
// All three active → storage wins.
let s = current_degraded_state();
assert!(s.storage && s.structure && s.semantic_recall);
assert!(s.is_degraded());
assert_eq!(
s.cause.as_ref().map(|c| c.code),
Some(FailureCode::StorageUnavailable)
);
// Clear storage → structure becomes the surfaced cause (its OWN, not
// storage's stale one).
clear_storage_degraded();
let s = current_degraded_state();
assert!(!s.storage && s.structure);
assert_eq!(
s.cause.as_ref().map(|c| c.code),
Some(FailureCode::ExtractionTimeout)
);
}
}
src/openhuman/memory_tree/tree/rpc.rs
+64
View File
@@ -661,6 +661,19 @@ fn derive_pipeline_status(
Some(format!("scheduler gate mode = {}", mode.as_str())),
);
}
// Host storage is unusable (EIO/ENOSPC/EROFS on the memory_tree path). This
// is a foundational, unrecoverable error — the DB can't even open, so it
// outranks the per-content recall/structure degradation below AND fires
// regardless of `total_chunks` (on a dead disk we may not be able to count
// chunks at all). Only the user can fix it (reseat/replace/free storage);
// the actionable remediation text rides the `StorageUnavailable`
// remediation key surfaced by the doctor's `first_blocking_cause`.
if degraded.storage {
return (
"error".to_string(),
Some("memory storage unavailable — check your disk / SD card".to_string()),
);
}
// #3365: split the failed bucket by class. Only an UNRECOVERABLE failure
// (budget / auth / dim-mismatch) is a hard `error` the user must act on —
// it stays parked and can't self-heal. Transient failures are auto-requeued
@@ -1093,13 +1106,21 @@ mod tests {
let recall_degraded = DegradedState {
semantic_recall: true,
structure: false,
storage: false,
cause: Some(PipelineFailure::new(FailureCode::EmbeddingsUnconfigured)),
};
let structure_degraded = DegradedState {
semantic_recall: false,
structure: true,
storage: false,
cause: Some(PipelineFailure::new(FailureCode::ExtractionTimeout)),
};
let storage_degraded = DegradedState {
semantic_recall: false,
structure: false,
storage: true,
cause: Some(PipelineFailure::new(FailureCode::StorageUnavailable)),
};
// Args: (is_paused, mode, is_syncing, failed, failed_unrecoverable,
// total_chunks, &degraded).
@@ -1117,6 +1138,49 @@ mod tests {
assert_eq!(s, "paused");
assert!(reason.unwrap().contains("off"));
// paused still beats a storage failure (user explicitly stood the
// worker down; the flag won't be freshly set anyway).
let (s, _) = derive_pipeline_status(
true,
SchedulerGateMode::Off,
false,
0,
0,
0,
&storage_degraded,
);
assert_eq!(s, "paused", "paused beats storage");
// storage failure → error, and it fires even with ZERO chunks (unlike
// recall/structure degradation, which is content-relative) — a dead
// disk is broken regardless of how much content exists.
let (s, reason) = derive_pipeline_status(
false,
SchedulerGateMode::Auto,
false,
0,
0,
0, // no chunks — must still surface
&storage_degraded,
);
assert_eq!(
s, "error",
"storage failure is a hard error at any chunk count"
);
assert!(reason.unwrap().contains("storage"));
// storage outranks transient-failed degradation too.
let (s, _) = derive_pipeline_status(
false,
SchedulerGateMode::Auto,
true,
3,
0,
100,
&storage_degraded,
);
assert_eq!(s, "error", "storage beats transient-degraded");
// error beats degraded / syncing / running / idle — but ONLY for
// unrecoverable failures (#3365).
let (s, reason) = derive_pipeline_status(