admin/openhuman
1
0
Fork 0
mirror of https://github.com/tinyhumansai/openhuman.git synced 2026-07-30 23:14:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(local_ai): default to cloud fallback on <8GB RAM devices (#589)

Browse Source 
* Enhance draft message handling in streaming edits

- Introduced a new `draft_sent` field in the `StreamingState` struct to track when a draft message has been posted, decoupling the existence of a draft from the ability to edit it.
- Updated the `flush_streaming_edit` function to set `draft_sent` upon posting a draft, ensuring that duplicate messages are not sent if the backend fails to return an ID.
- Modified the `finalize_channel_reply` function to prevent sending a fresh message if a draft was already posted without an ID, improving user experience by avoiding duplicate bubbles.

These changes enhance the robustness of message handling during streaming edits, ensuring a smoother interaction for users.

* feat(onboarding): enhance LocalAIStep for low-RAM device handling

- Added logic to determine if the device is below the RAM threshold, defaulting to a cloud AI fallback if necessary.
- Introduced a new UI for low-RAM devices, informing users about the cloud mode and providing options to continue with cloud AI or force-enable local AI.
- Updated the `ensureRecommendedLocalAiPresetIfNeeded` function to return a `recommend_disabled` flag based on device RAM.
- Enhanced tests to cover new cloud fallback UI and local AI consent handling.

These changes improve the onboarding experience by providing clearer options based on device capabilities.

* style: apply prettier formatting to LocalAIStep

* feat(local_ai): unlock model selection + add Disabled/cloud fallback option

- Remove MVP ceiling that clamped selection to the 2-4 GB tier, in bootstrap
  and in the apply_preset RPC — users can now pick any tier.
- Add special "disabled" tier string to apply_preset that toggles
  config.local_ai.enabled = false so the app uses the cloud summarizer.
- Presets RPC now returns local_ai_enabled so the UI can render the
  currently-active state correctly.
- Rewrite DeviceCapabilitySection: tiers are now clickable buttons that
  call apply_preset; adds a "Disabled — Cloud fallback" card at the top
  marked "Recommended" on low-RAM devices and "Active" when enabled=false.
- Remove the MVP message copy from the model tier panel.
- Remove unread-count badge from the bottom tab bar.

* refactor(onboarding): streamline LocalAIStep and update local AI preset handling

- Removed the `ensureRecommendedLocalAiPresetIfNeeded` function call from `LocalAIStep`, replacing it with `openhumanLocalAiPresets` to directly fetch preset information.
- Updated the logic in `ensureRecommendedLocalAiPresetIfNeeded` to ensure the recommended tier is applied correctly based on user consent, improving clarity in the local AI setup process.
- Enhanced tests to mock the new `openhumanLocalAiPresets` function, ensuring coverage for low-RAM device scenarios and local AI consent handling.
- Updated documentation in the `schemas.rs` file to reflect changes in the data structure returned by the presets RPC.

These changes improve the onboarding experience by providing a more direct and efficient method for managing local AI presets based on device capabilities.

* test(LocalAIStep): improve mock implementation for local AI presets

- Refactored the mock for `openhumanLocalAiPresets` to enhance readability and maintainability.
- Ensured the mock returns consistent device information and preset details for testing scenarios.
- This change supports better test coverage and clarity in the LocalAIStep component's behavior during onboarding.

* fix(local_ai): preserve hadSelectedTier semantics in bootstrap return

hadSelectedTier / selectedTier represent the incoming state ("was a tier
already selected before this call"), not the post-apply state. Setting
both to the just-applied tier broke the existing localAiBootstrap
contract and the unit test that encodes it.

The Rust-side persistence fix is independent: removing the
recommend_disabled short-circuit ensures openhumanLocalAiApplyPreset
actually writes the selected tier to disk on the opt-in path, so
config_with_recommended_tier_if_unselected() honors the user's choice.
This commit is contained in:
Steven Enamakel
2026-04-16 10:20:28 -07:00
committed by GitHub
parent 923c920ad5
commit 7db71408bd
9 changed files with 463 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/openhuman/local_ai/presets.rs
+44 -22
View File
@@ -33,6 +33,24 @@ pub enum ModelTier {
/// selection is re-enabled post-MVP.
pub const MVP_MAX_TIER: ModelTier = ModelTier::Ram2To4Gb;
/// Minimum host RAM (in whole GB) below which the **default** is to skip
/// local inference and use the cloud summarizer instead. The user can still
/// override this and opt into local AI via settings.
pub const MIN_RAM_GB_FOR_LOCAL_AI: u64 = 8;
/// Returns `true` when the device has enough RAM that local AI should be
/// enabled by default. Below the floor we recommend cloud fallback instead.
pub fn device_supports_local_ai(device: &DeviceProfile) -> bool {
device.total_ram_gb() >= MIN_RAM_GB_FOR_LOCAL_AI
}
/// Returns `true` when the device is below the RAM floor and local AI should
/// default to disabled (cloud fallback). This is a **recommendation**, not a
/// hard gate — the user can still opt in.
pub fn should_default_to_cloud_fallback(device: &DeviceProfile) -> bool {
!device_supports_local_ai(device)
}
impl ModelTier {
pub fn as_str(&self) -> &'static str {
match self {
@@ -178,12 +196,9 @@ pub fn preset_for_tier(tier: ModelTier) -> Option<ModelPreset> {
}
/// Recommend a tier based on device capabilities.
///
/// The recommendation is capped at [`MVP_MAX_TIER`] so that auto-provisioning
/// never selects a model above the MVP ceiling, regardless of available RAM.
pub fn recommend_tier(device: &DeviceProfile) -> ModelTier {
let ram_gb = device.total_ram_gb();
let uncapped = if ram_gb >= 16 {
let tier = if ram_gb >= 16 {
ModelTier::Ram16PlusGb
} else if ram_gb >= 8 {
ModelTier::Ram8To16Gb
@@ -194,17 +209,6 @@ pub fn recommend_tier(device: &DeviceProfile) -> ModelTier {
} else {
ModelTier::Ram1Gb
};
let tier = if uncapped.is_mvp_allowed() {
uncapped
} else {
tracing::debug!(
ram_gb,
?uncapped,
capped_to = ?MVP_MAX_TIER,
"[local_ai] capping recommended tier to MVP ceiling"
);
MVP_MAX_TIER
};
tracing::debug!(ram_gb, ?tier, "[local_ai] recommended model tier");
tier
}
@@ -318,13 +322,12 @@ mod tests {
}
#[test]
fn recommend_tier_by_ram_capped_at_mvp_max() {
// All devices are capped at the single MVP tier (Ram2To4Gb).
assert_eq!(recommend_tier(&test_device(1)), MVP_MAX_TIER);
assert_eq!(recommend_tier(&test_device(3)), MVP_MAX_TIER);
assert_eq!(recommend_tier(&test_device(4)), MVP_MAX_TIER);
assert_eq!(recommend_tier(&test_device(8)), MVP_MAX_TIER);
assert_eq!(recommend_tier(&test_device(32)), MVP_MAX_TIER);
fn recommend_tier_scales_with_ram() {
assert_eq!(recommend_tier(&test_device(1)), ModelTier::Ram1Gb);
assert_eq!(recommend_tier(&test_device(3)), ModelTier::Ram2To4Gb);
assert_eq!(recommend_tier(&test_device(4)), ModelTier::Ram4To8Gb);
assert_eq!(recommend_tier(&test_device(8)), ModelTier::Ram8To16Gb);
assert_eq!(recommend_tier(&test_device(32)), ModelTier::Ram16PlusGb);
}
#[test]
@@ -381,6 +384,25 @@ mod tests {
assert_eq!(vision_mode_for_config(&config), VisionMode::Bundled);
}
#[test]
fn device_supports_local_ai_honors_min_ram_floor() {
assert!(!device_supports_local_ai(&test_device(1)));
assert!(!device_supports_local_ai(&test_device(4)));
assert!(!device_supports_local_ai(&test_device(7)));
assert!(device_supports_local_ai(&test_device(8)));
assert!(device_supports_local_ai(&test_device(16)));
assert!(device_supports_local_ai(&test_device(64)));
}
#[test]
fn should_default_to_cloud_fallback_below_floor() {
assert!(should_default_to_cloud_fallback(&test_device(1)));
assert!(should_default_to_cloud_fallback(&test_device(4)));
assert!(should_default_to_cloud_fallback(&test_device(7)));
assert!(!should_default_to_cloud_fallback(&test_device(8)));
assert!(!should_default_to_cloud_fallback(&test_device(16)));
}
#[test]
fn built_in_vision_modes_match_expectations() {
let mut config = LocalAiConfig::default();
src/openhuman/local_ai/schemas.rs
+30 -14
View File
@@ -437,7 +437,11 @@ pub fn schemas(function: &str) -> ControllerSchema {
inputs: vec![],
outputs: vec![json_output(
"presets",
"Presets, recommended tier, current tier.",
"Object containing: presets (array of ModelPreset), recommended_tier (string), \
current_tier (string), selected_tier (string | null), device (DeviceProfile), \
recommend_disabled (boolean — true when the device is below the RAM floor and \
cloud fallback is the recommended default), local_ai_enabled (boolean — mirrors \
config.local_ai.enabled so the UI can render the active state when disabled).",
)],
},
"local_ai_apply_preset" => ControllerSchema {
@@ -759,12 +763,16 @@ fn handle_local_ai_presets(_params: Map<String, Value>) -> ControllerFuture {
preset_count = presets.len(),
"[local_ai] presets: returning"
);
let recommend_disabled =
crate::openhuman::local_ai::presets::should_default_to_cloud_fallback(&device);
let value = serde_json::json!({
"presets": presets,
"recommended_tier": recommended,
"current_tier": current,
"selected_tier": selected_tier,
"device": device,
"recommend_disabled": recommend_disabled,
"local_ai_enabled": config.local_ai.enabled,
});
Ok(value)
})
@@ -776,10 +784,26 @@ fn handle_local_ai_apply_preset(params: Map<String, Value>) -> ControllerFuture
let tier_str = p.tier.trim().to_ascii_lowercase();
tracing::debug!(tier = %tier_str, "[local_ai] apply_preset: parsing tier");
// Special "disabled" tier: turn local_ai off and route AI to cloud.
if tier_str == "disabled" {
let mut config = config_rpc::load_config_with_timeout().await?;
config.local_ai.enabled = false;
config.local_ai.selected_tier = Some("disabled".to_string());
config
.save()
.await
.map_err(|e| format!("save config: {e}"))?;
tracing::debug!("[local_ai] apply_preset: local_ai disabled (cloud fallback)");
return Ok(serde_json::json!({
"applied_tier": "disabled",
"local_ai_enabled": false,
}));
}
let tier = crate::openhuman::local_ai::presets::ModelTier::from_str_opt(&tier_str)
.ok_or_else(|| {
format!(
"invalid tier '{}': expected one of ram_1gb, ram_2_4gb, ram_4_8gb, ram_8_16gb, ram_16_plus_gb",
"invalid tier '{}': expected one of disabled, ram_1gb, ram_2_4gb, ram_4_8gb, ram_8_16gb, ram_16_plus_gb",
tier_str
)
})?;
@@ -788,19 +812,10 @@ fn handle_local_ai_apply_preset(params: Map<String, Value>) -> ControllerFuture
return Err("cannot apply 'custom' tier; set model IDs directly".to_string());
}
if !tier.is_mvp_allowed() {
tracing::debug!(
tier = %tier_str,
"[local_ai] apply_preset: rejected — tier exceeds MVP ceiling"
);
return Err(format!(
"model tier '{}' is not available in MVP; only tiers up to {} are allowed",
tier_str,
crate::openhuman::local_ai::presets::MVP_MAX_TIER.as_str()
));
}
let mut config = config_rpc::load_config_with_timeout().await?;
// Re-enable local AI in case it was previously disabled via the
// "disabled" tier, so the user can switch back to local inference.
config.local_ai.enabled = true;
crate::openhuman::local_ai::presets::apply_preset_to_config(&mut config.local_ai, tier);
config
.save()
@@ -815,6 +830,7 @@ fn handle_local_ai_apply_preset(params: Map<String, Value>) -> ControllerFuture
"embedding_model_id": config.local_ai.embedding_model_id,
"quantization": config.local_ai.quantization,
"vision_mode": crate::openhuman::local_ai::presets::vision_mode_for_config(&config.local_ai),
"local_ai_enabled": true,
}))
})
}
src/openhuman/local_ai/service/bootstrap.rs
+88 -48
View File
@@ -274,28 +274,31 @@ fn config_with_recommended_tier_if_unselected(config: &Config, device: &DevicePr
let current_tier =
crate::openhuman::local_ai::presets::current_tier_from_config(&config.local_ai);
// If a tier is already selected, check whether it exceeds the MVP ceiling.
// If so, clamp it down to the recommended (MVP-capped) tier.
// When the user has *not* explicitly picked a tier and the device is below
// the RAM floor, default to disabled (cloud fallback). The user can still
// opt in via Settings. This check runs first so even a "detected" tier from
// matching defaults gets overridden on low-RAM hosts.
if selected_tier.is_none()
&& crate::openhuman::local_ai::presets::should_default_to_cloud_fallback(device)
{
tracing::debug!(
total_ram_gb = device.total_ram_gb(),
min_required_gb = crate::openhuman::local_ai::presets::MIN_RAM_GB_FOR_LOCAL_AI,
"[local_ai] bootstrap: device below RAM floor, defaulting to disabled (cloud fallback)"
);
let mut effective_config = config.clone();
effective_config.local_ai.enabled = false;
return effective_config;
}
// If the user has already picked a tier (explicit or matching defaults),
// honor it. No MVP ceiling — users can pick any tier.
if selected_tier.is_some()
|| !matches!(
current_tier,
crate::openhuman::local_ai::presets::ModelTier::Custom
)
{
if !current_tier.is_mvp_allowed() {
let recommended = crate::openhuman::local_ai::presets::recommend_tier(device);
tracing::warn!(
?current_tier,
?recommended,
"[local_ai] bootstrap: selected tier exceeds MVP ceiling, clamping to recommended"
);
let mut effective_config = config.clone();
crate::openhuman::local_ai::presets::apply_preset_to_config(
&mut effective_config.local_ai,
recommended,
);
return effective_config;
}
return config.clone();
}
@@ -360,51 +363,88 @@ mod tests {
assert!(!service.should_run_memory_autosummary(&config));
}
#[test]
fn bootstrap_clamps_non_mvp_tier_when_selection_missing() {
let config = Config::default();
// Default config matches Ram8To16Gb which is not MVP-allowed,
// so bootstrap clamps it to the MVP-capped recommended tier.
let device = DeviceProfile {
total_ram_bytes: 4 * 1024 * 1024 * 1024,
fn test_device(ram_gb: u64) -> DeviceProfile {
DeviceProfile {
total_ram_bytes: ram_gb * 1024 * 1024 * 1024,
cpu_count: 4,
cpu_brand: String::new(),
os_name: String::new(),
os_version: String::new(),
has_gpu: false,
gpu_description: None,
};
let effective = config_with_recommended_tier_if_unselected(&config, &device);
assert_eq!(
effective.local_ai.selected_tier.as_deref(),
Some("ram_2_4gb")
);
assert_eq!(effective.local_ai.chat_model_id, "gemma3:1b-it-qat");
}
}
#[test]
fn bootstrap_clamps_non_mvp_selected_tier() {
fn bootstrap_defaults_to_disabled_on_low_ram_device() {
let config = Config::default();
let device = test_device(4);
let effective = config_with_recommended_tier_if_unselected(&config, &device);
assert!(
!effective.local_ai.enabled,
"local_ai.enabled must default to false on <8 GB device"
);
}
#[test]
fn bootstrap_keeps_enabled_on_sufficient_ram_device() {
let config = Config::default();
let device = test_device(16);
let effective = config_with_recommended_tier_if_unselected(&config, &device);
assert!(
effective.local_ai.enabled,
"local_ai.enabled must stay true on >= 8 GB device"
);
}
#[test]
fn bootstrap_respects_explicit_tier_on_low_ram_device() {
let mut config = Config::default();
config.local_ai.selected_tier = Some("ram_2_4gb".to_string());
crate::openhuman::local_ai::presets::apply_preset_to_config(
&mut config.local_ai,
crate::openhuman::local_ai::presets::ModelTier::Ram2To4Gb,
);
let device = test_device(4);
let effective = config_with_recommended_tier_if_unselected(&config, &device);
// User explicitly chose a tier, so local_ai stays enabled.
assert!(
effective.local_ai.enabled,
"explicit tier selection must be honored even on low-RAM device"
);
}
#[test]
fn bootstrap_keeps_default_tier_when_selection_missing_and_ram_sufficient() {
// Default config already matches a preset (Ram8To16Gb) and device has
// enough RAM, so bootstrap returns it untouched (no auto-change).
let config = Config::default();
let device = test_device(8);
let effective = config_with_recommended_tier_if_unselected(&config, &device);
assert!(effective.local_ai.enabled);
assert_eq!(
effective.local_ai.chat_model_id,
config.local_ai.chat_model_id
);
}
#[test]
fn bootstrap_honors_explicit_non_mvp_tier_selection() {
let mut config = Config::default();
config.local_ai.selected_tier = Some("high".to_string());
let device = DeviceProfile {
total_ram_bytes: 4 * 1024 * 1024 * 1024,
cpu_count: 4,
cpu_brand: String::new(),
os_name: String::new(),
os_version: String::new(),
has_gpu: false,
gpu_description: None,
};
let device = test_device(8);
let effective = config_with_recommended_tier_if_unselected(&config, &device);
// "high" maps to Ram16PlusGb which is not MVP-allowed, so it gets clamped.
assert_eq!(
effective.local_ai.selected_tier.as_deref(),
Some("ram_2_4gb")
);
assert_eq!(effective.local_ai.chat_model_id, "gemma3:1b-it-qat");
// User explicitly picked a high-tier preset — honor it, no MVP clamp.
assert_eq!(effective.local_ai.selected_tier.as_deref(), Some("high"));
}
}