diff --git a/docs/TEST-COVERAGE-MATRIX.md b/docs/TEST-COVERAGE-MATRIX.md index e244e6fcd..1595f504c 100644 --- a/docs/TEST-COVERAGE-MATRIX.md +++ b/docs/TEST-COVERAGE-MATRIX.md @@ -190,6 +190,7 @@ Canonical mapping of every product feature to its test source(s). Drives gap-fil | 4.3.2 | Permission-Based Execution | RU+WD | `src/openhuman/tools/`, `skill-execution-flow.spec.ts` | ✅ | | | 4.3.3 | Tool Failure Handling | WD | `skill-execution-flow.spec.ts` | ✅ | | | 4.3.4 | Subagent Mascot Visualization | VU | `app/src/features/human/SubMascotLayer.test.tsx`, `app/src/features/human/HumanPage.test.tsx` | ✅ | Renders spawned/completed/failed subagent timeline rows as colored companion mascots with activity bubbles | +| 4.3.5 | Image Tool Contracts | RU | `src/openhuman/image/` | ✅ | High-level `image_generation` / `view_image` schema, gating, serialization, prompt guidance, and contract e2e coverage for #2984 | --- diff --git a/gitbooks/SUMMARY.md b/gitbooks/SUMMARY.md index 59d7558ee..dd84739ad 100644 --- a/gitbooks/SUMMARY.md +++ b/gitbooks/SUMMARY.md @@ -21,6 +21,7 @@ * [Local AI (optional)](features/model-routing/local-ai.md) * [Available Tools](features/native-tools/README.md) * [Web Search](features/native-tools/web-search.md) + * [Image Tools](features/native-tools/image-tools.md) * [Web Scraper](features/native-tools/web-scraper.md) * [Coder](features/native-tools/coder.md) * [Browser & Computer Control](features/native-tools/browser-and-computer.md) diff --git a/gitbooks/features/native-tools/image-tools.md b/gitbooks/features/native-tools/image-tools.md new file mode 100644 index 000000000..fafc4bd3a --- /dev/null +++ b/gitbooks/features/native-tools/image-tools.md @@ -0,0 +1,81 @@ +# Image Tools + +OpenHuman's image contract gives agents a stable way to reason about +image generation and local image inspection without tying the prompt surface to a +single provider runtime. + +## Scope + +The contract lives in the top-level `src/openhuman/image/` module and +currently covers two model-facing tools: + +| Tool | Purpose | Permission | Output | +| --- | --- | --- | --- | +| `image_generation` | Generate or edit raster images from a prompt. | Write | Local generated-media artifact paths. | +| `view_image` | Load a local image file into model-visible image context. | Read-only | Image content visible to the model. | + +This layer is intentionally high level. Existing lower-level tools still own +their concrete behavior: + +- `image_info` reads local image metadata and optional base64 text. +- Browser screenshot tooling captures pages and writes image files. +- Agent multimodal preparation normalizes `[IMAGE:...]` markers for providers + that accept image data. + +The image layer defines names, schemas, gating, and prompt rules so +agents can make consistent decisions as runtimes add direct support. + +## `image_generation` + +`image_generation` is a hosted provider capability. The Rust core should not +pretend to be an image renderer when no provider supports it. When enabled, the +runtime should: + +1. Validate any `input_image_path` through the same local-file policy used for + image viewing. +2. Send the prompt and optional edit image to the hosted image provider. +3. Persist returned bytes under a session-scoped generated-media root, or under + an approved caller-provided `output_path`. +4. Return saved artifact paths so the final assistant answer can reference them. + +The schema includes `prompt`, optional `output_path`, optional `size`, optional +`input_image_path`, and `output_format` (`png`, `webp`, `jpeg`). + +## `view_image` + +`view_image` loads pixels from a local file into model-visible context. Use it +when text metadata is insufficient: screenshots, UI review, OCR, diagrams, +charts, visual diffs, and generated-image inspection. + +The runtime must keep local file boundaries explicit: + +- Allow paths in the approved workspace. +- Allow paths created during the current session. +- Allow paths explicitly referenced by the user or trusted tool output. +- Deny paths outside policy, and do not silently attach unrelated local images. + +The schema includes `path` and optional `detail` (`auto`, `high`, `original`). +Use `original` only when full-resolution inspection is necessary. + +## Prompt Guidance + +Prompt rendering should include image guidance only when at least one +media tool is enabled. The guidance should tell agents: + +- Use `view_image` when pixels are needed, not for ordinary file metadata. +- Use `image_generation` for requested raster image creation or edits. +- Provide an output path when the destination matters. +- Mention generated artifact paths in final answers. +- Respect local image boundaries before attaching a file to model context. + +## Tests + +The module has focused Rust tests for: + +- JSON schema shape for `image_generation`. +- JSON schema shape for `view_image`. +- Independent gating of generation vs local viewing. +- End-to-end contract rendering from config to specs and prompt guidance. + +Future runtime PRs should add provider-specific execution tests next to the +runtime adapter, not in the hosted contract module. diff --git a/src/openhuman/image/README.md b/src/openhuman/image/README.md new file mode 100644 index 000000000..cbbc17034 --- /dev/null +++ b/src/openhuman/image/README.md @@ -0,0 +1,43 @@ +# image + +High-level contracts for image-capable agent runtimes. + +This module does not execute image generation or pixel inspection directly. It +defines the stable model-facing contracts that provider/runtime adapters can +expose when image capabilities are available. + +## Responsibilities + +- Define the `image_generation` contract for hosted raster image creation and + edits. +- Define the `view_image` contract for loading local image files into + model-visible image context. +- Gate contract exposure by runtime support and local policy: + - generated image writes + - local image reads +- Render concise prompt guidance for agents when image tools are available. +- Keep schema/serialization contracts covered by focused Rust tests. + +## Module Shape + +| File | Role | +| --- | --- | +| `mod.rs` | Export-only module entrypoint. | +| `types.rs` | Shared descriptors, permission/config types, and gating helpers. | +| `image_generation.rs` | `image_generation` schema and output-format contract. | +| `image_view.rs` | `view_image` schema and detail-level contract. | +| `prompt.rs` | Agent prompt guidance for enabled image tools. | +| `tests.rs` | Contract-level e2e tests across config, schemas, and prompt output. | + +## Notes + +Existing lower-level image helpers remain separate: + +- `image_info` reads metadata/base64 text. +- Browser screenshot tooling captures page images. +- Multimodal `[IMAGE:...]` preparation normalizes image references for + image-capable providers. + +Future execution adapters should depend on these contracts and live next to the +runtime/provider implementation that actually performs the hosted call or model +attachment. diff --git a/src/openhuman/image/image_generation.rs b/src/openhuman/image/image_generation.rs new file mode 100644 index 000000000..6e6476b98 --- /dev/null +++ b/src/openhuman/image/image_generation.rs @@ -0,0 +1,132 @@ +//! Contract for the hosted `image_generation` tool. +//! +//! The tool is a hosted/provider capability, not a local Rust image renderer. +//! Runtimes that support it should persist generated bytes under a +//! session-scoped generated-media root, then return stable file paths in tool +//! output so final answers can reference concrete artifacts. + +use serde::{Deserialize, Serialize}; +use serde_json::json; + +use super::{ImagePermission, ImageToolSpec}; + +/// Stable model-facing tool name. +pub const IMAGE_GENERATION_TOOL_NAME: &str = "image_generation"; + +/// Generated image file format requested from the hosted provider. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ImageGenerationOutputFormat { + Png, + Webp, + Jpeg, +} + +impl ImageGenerationOutputFormat { + pub fn as_str(self) -> &'static str { + match self { + Self::Png => "png", + Self::Webp => "webp", + Self::Jpeg => "jpeg", + } + } +} + +/// Build the hosted `image_generation` model-facing contract. +pub fn image_generation_spec(output_format: ImageGenerationOutputFormat) -> ImageToolSpec { + ImageToolSpec { + name: IMAGE_GENERATION_TOOL_NAME.to_string(), + description: "Generate or edit raster images from a prompt and store each result as a local artifact path for the user.".to_string(), + parameters: json!({ + "type": "object", + "properties": { + "prompt": { + "type": "string", + "description": "Detailed visual prompt or edit instruction for the hosted image model." + }, + "output_path": { + "type": "string", + "description": "Optional workspace-relative or approved absolute output path. When omitted, the runtime chooses a generated-media path." + }, + "size": { + "type": "string", + "description": "Optional output size such as 1024x1024, 1536x1024, or provider default." + }, + "input_image_path": { + "type": "string", + "description": "Optional local image path to edit. The runtime must validate local-file access before attaching it." + }, + "output_format": { + "type": "string", + "enum": ["png", "webp", "jpeg"], + "default": output_format.as_str(), + "description": "Requested persisted file format." + } + }, + "required": ["prompt"] + }), + permission: ImagePermission::Write, + model_visible_image_output: false, + writes_files: true, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn image_generation_schema_declares_output_path_and_format() { + let spec = image_generation_spec(ImageGenerationOutputFormat::Webp); + + assert_eq!(spec.name, "image_generation"); + assert!(spec.description.contains("Generate or edit raster images")); + assert_eq!(spec.permission, ImagePermission::Write); + assert!(!spec.model_visible_image_output); + assert!(spec.writes_files); + assert_eq!(spec.parameters["required"], serde_json::json!(["prompt"])); + assert!(spec.parameters["properties"].get("output_path").is_some()); + assert_eq!( + spec.parameters["properties"]["output_format"]["default"], + "webp" + ); + } + + #[test] + fn output_format_serializes_as_snake_case() { + assert_eq!(ImageGenerationOutputFormat::Png.as_str(), "png"); + assert_eq!(ImageGenerationOutputFormat::Webp.as_str(), "webp"); + assert_eq!(ImageGenerationOutputFormat::Jpeg.as_str(), "jpeg"); + + assert_eq!( + serde_json::to_value(ImageGenerationOutputFormat::Webp).unwrap(), + serde_json::json!("webp") + ); + assert_eq!( + serde_json::from_value::(serde_json::json!("jpeg")) + .unwrap(), + ImageGenerationOutputFormat::Jpeg + ); + } + + #[test] + fn image_generation_schema_keeps_edit_and_size_inputs_optional() { + let spec = image_generation_spec(ImageGenerationOutputFormat::Png); + let properties = spec.parameters["properties"].as_object().unwrap(); + + assert_eq!( + properties.keys().cloned().collect::>(), + vec![ + "input_image_path".to_string(), + "output_format".to_string(), + "output_path".to_string(), + "prompt".to_string(), + "size".to_string() + ] + ); + assert_eq!( + properties["output_format"]["enum"], + serde_json::json!(["png", "webp", "jpeg"]) + ); + } +} diff --git a/src/openhuman/image/image_view.rs b/src/openhuman/image/image_view.rs new file mode 100644 index 000000000..85a5e899d --- /dev/null +++ b/src/openhuman/image/image_view.rs @@ -0,0 +1,104 @@ +//! Contract for the hosted `view_image` tool. +//! +//! `view_image` bridges local image files into model-visible image content. It +//! is distinct from `image_info`: metadata extraction can stay textual, while +//! `view_image` asks the runtime to load pixels into the conversation context. + +use serde::{Deserialize, Serialize}; +use serde_json::json; + +use super::{ImagePermission, ImageToolSpec}; + +/// Stable model-facing tool name. +pub const IMAGE_VIEW_TOOL_NAME: &str = "view_image"; + +/// Requested image-detail level for model-visible inspection. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ImageDetail { + Auto, + High, + Original, +} + +impl ImageDetail { + pub fn as_str(self) -> &'static str { + match self { + Self::Auto => "auto", + Self::High => "high", + Self::Original => "original", + } + } +} + +/// Build the hosted `view_image` model-facing contract. +pub fn image_view_spec() -> ImageToolSpec { + ImageToolSpec { + name: IMAGE_VIEW_TOOL_NAME.to_string(), + description: "Load a local image file into model-visible image context for inspection, OCR, UI review, or visual reasoning.".to_string(), + parameters: json!({ + "type": "object", + "properties": { + "path": { + "type": "string", + "description": "Local image path, absolute or relative to the approved workspace." + }, + "detail": { + "type": "string", + "enum": ["auto", "high", "original"], + "default": ImageDetail::Auto.as_str(), + "description": "Inspection detail. Use original only when full resolution is necessary." + } + }, + "required": ["path"] + }), + permission: ImagePermission::ReadOnly, + model_visible_image_output: true, + writes_files: false, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn view_image_schema_requires_path_and_marks_model_visible_output() { + let spec = image_view_spec(); + + assert_eq!(spec.name, "view_image"); + assert!(spec.description.contains("model-visible image context")); + assert_eq!(spec.permission, ImagePermission::ReadOnly); + assert!(spec.model_visible_image_output); + assert!(!spec.writes_files); + assert_eq!(spec.parameters["required"], serde_json::json!(["path"])); + assert_eq!(spec.parameters["properties"]["detail"]["default"], "auto"); + } + + #[test] + fn detail_names_match_prompt_contract() { + assert_eq!(ImageDetail::Auto.as_str(), "auto"); + assert_eq!(ImageDetail::High.as_str(), "high"); + assert_eq!(ImageDetail::Original.as_str(), "original"); + + assert_eq!( + serde_json::to_value(ImageDetail::Original).unwrap(), + serde_json::json!("original") + ); + assert_eq!( + serde_json::from_value::(serde_json::json!("high")).unwrap(), + ImageDetail::High + ); + } + + #[test] + fn view_image_schema_lists_supported_detail_levels() { + let spec = image_view_spec(); + + assert_eq!( + spec.parameters["properties"]["detail"]["enum"], + serde_json::json!(["auto", "high", "original"]) + ); + assert_eq!(spec.parameters["properties"]["path"]["type"], "string"); + } +} diff --git a/src/openhuman/image/mod.rs b/src/openhuman/image/mod.rs new file mode 100644 index 000000000..51ad4a672 --- /dev/null +++ b/src/openhuman/image/mod.rs @@ -0,0 +1,37 @@ +//! Image-tool contracts for model-facing agents. +//! +//! This module is intentionally a high-level contract layer. OpenHuman already +//! has lower-level image helpers (`image_info`, browser screenshots, and +//! multimodal `[IMAGE:...]` normalization). The image layer defines the +//! stable tool names, schema, gating, and prompt guidance that agents should see +//! when a runtime can provide Codex-like media tools. +//! +//! The two first-class contracts are: +//! +//! - [`image_generation`] — create or edit raster images and return stored file +//! references. +//! - [`image_view`] — attach a local image file as model-visible image content +//! so the agent can inspect it. +//! +//! Keeping this contract separate from execution lets provider runtimes adopt +//! the surface incrementally without duplicating business logic in the tools +//! registry. + +pub mod image_generation; +pub mod image_view; +pub mod prompt; +pub mod types; + +pub use image_generation::{ + image_generation_spec, ImageGenerationOutputFormat, IMAGE_GENERATION_TOOL_NAME, +}; +pub use image_view::{image_view_spec, ImageDetail, IMAGE_VIEW_TOOL_NAME}; +pub use prompt::{render_image_prompt_guidance, ImagePromptOptions}; +pub use types::{ + image_specs, is_image_tool_gated, ImagePermission, ImageToolConfig, ImageToolSpec, + IMAGE_TOOL_NAMES, +}; + +#[cfg(test)] +#[path = "tests.rs"] +mod tests; diff --git a/src/openhuman/image/prompt.rs b/src/openhuman/image/prompt.rs new file mode 100644 index 000000000..fad137c2c --- /dev/null +++ b/src/openhuman/image/prompt.rs @@ -0,0 +1,160 @@ +//! Prompt guidance for image tools. + +use super::{ImageToolConfig, IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME}; + +/// Rendering options for image prompt guidance. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ImagePromptOptions { + /// Include final-answer artifact-reference guidance for generated images. + pub include_final_answer_rules: bool, + /// Include local-file privacy boundaries for viewed images. + pub include_local_file_boundaries: bool, +} + +impl Default for ImagePromptOptions { + fn default() -> Self { + Self { + include_final_answer_rules: true, + include_local_file_boundaries: true, + } + } +} + +/// Render concise model guidance for enabled image tools. +pub fn render_image_prompt_guidance( + config: &ImageToolConfig, + options: &ImagePromptOptions, +) -> String { + let image_generation_available = + config.image_generation_enabled && config.generated_image_writes_allowed; + let image_view_available = config.image_view_enabled && config.local_image_reads_allowed; + + if !image_generation_available && !image_view_available { + return String::new(); + } + + let mut out = String::from("## Image Tools\n\n"); + + if image_view_available { + out.push_str(&format!( + "- Use `{IMAGE_VIEW_TOOL_NAME}` when image pixels are needed for UI review, OCR, chart inspection, visual comparison, or understanding a local screenshot.\n" + )); + if options.include_local_file_boundaries { + out.push_str( + "- Only view local images that are in the approved workspace, were created during this session, or were explicitly referenced by the user or trusted tool output.\n", + ); + } + } + + if image_generation_available { + out.push_str(&format!( + "- Use `{IMAGE_GENERATION_TOOL_NAME}` for requested raster image creation or image edits; provide a specific prompt and an output path when the destination matters.\n" + )); + if options.include_final_answer_rules { + out.push_str( + "- After generating images, mention the saved artifact path in the final answer so the user can find it.\n", + ); + } + } + + out.trim_end().to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::openhuman::image::ImageGenerationOutputFormat; + + #[test] + fn prompt_guidance_is_empty_when_no_image_tools_are_enabled() { + let rendered = render_image_prompt_guidance( + &ImageToolConfig::default(), + &ImagePromptOptions::default(), + ); + + assert!(rendered.is_empty()); + } + + #[test] + fn prompt_guidance_renders_image_generation_and_view_rules() { + let config = ImageToolConfig { + image_generation_enabled: true, + image_view_enabled: true, + image_generation_output_format: ImageGenerationOutputFormat::Png, + local_image_reads_allowed: true, + generated_image_writes_allowed: true, + }; + + let rendered = render_image_prompt_guidance(&config, &ImagePromptOptions::default()); + + assert!(rendered.contains("## Image Tools")); + assert!(rendered.contains("`view_image`")); + assert!(rendered.contains("`image_generation`")); + assert!(rendered.contains("saved artifact path")); + } + + #[test] + fn prompt_guidance_can_omit_optional_rule_text() { + let config = ImageToolConfig { + image_generation_enabled: true, + image_view_enabled: true, + image_generation_output_format: ImageGenerationOutputFormat::Png, + local_image_reads_allowed: true, + generated_image_writes_allowed: true, + }; + let options = ImagePromptOptions { + include_final_answer_rules: false, + include_local_file_boundaries: false, + }; + + let rendered = render_image_prompt_guidance(&config, &options); + + assert!(rendered.contains("`view_image`")); + assert!(rendered.contains("`image_generation`")); + assert!(!rendered.contains("approved workspace")); + assert!(!rendered.contains("saved artifact path")); + } + + #[test] + fn prompt_guidance_respects_policy_gates() { + let config = ImageToolConfig { + image_generation_enabled: true, + image_view_enabled: true, + image_generation_output_format: ImageGenerationOutputFormat::Png, + local_image_reads_allowed: false, + generated_image_writes_allowed: false, + }; + + let rendered = render_image_prompt_guidance(&config, &ImagePromptOptions::default()); + + assert!(rendered.is_empty()); + } + + #[test] + fn prompt_guidance_renders_single_available_tool() { + let generation_only = ImageToolConfig { + image_generation_enabled: true, + image_view_enabled: false, + image_generation_output_format: ImageGenerationOutputFormat::Png, + local_image_reads_allowed: true, + generated_image_writes_allowed: true, + }; + let view_only = ImageToolConfig { + image_generation_enabled: false, + image_view_enabled: true, + image_generation_output_format: ImageGenerationOutputFormat::Png, + local_image_reads_allowed: true, + generated_image_writes_allowed: true, + }; + + let generation_rendered = + render_image_prompt_guidance(&generation_only, &ImagePromptOptions::default()); + let view_rendered = + render_image_prompt_guidance(&view_only, &ImagePromptOptions::default()); + + assert!(generation_rendered.contains("`image_generation`")); + assert!(!generation_rendered.contains("`view_image`")); + assert!(view_rendered.contains("`view_image`")); + assert!(!view_rendered.contains("`image_generation`")); + } +} diff --git a/src/openhuman/image/tests.rs b/src/openhuman/image/tests.rs new file mode 100644 index 000000000..f80dcc2ef --- /dev/null +++ b/src/openhuman/image/tests.rs @@ -0,0 +1,120 @@ +use super::*; + +#[test] +fn image_specs_gate_each_tool_independently() { + let config = ImageToolConfig { + image_generation_enabled: true, + image_view_enabled: true, + local_image_reads_allowed: false, + generated_image_writes_allowed: true, + ..ImageToolConfig::default() + }; + + let specs = image_specs(&config); + + assert_eq!(specs.len(), 1); + assert_eq!(specs[0].name, IMAGE_GENERATION_TOOL_NAME); + assert!(!is_image_tool_gated(IMAGE_GENERATION_TOOL_NAME, &config)); + assert!(is_image_tool_gated(IMAGE_VIEW_TOOL_NAME, &config)); +} + +#[test] +fn image_specs_hide_generation_when_writes_are_blocked() { + let config = ImageToolConfig { + image_generation_enabled: true, + image_view_enabled: true, + local_image_reads_allowed: true, + generated_image_writes_allowed: false, + ..ImageToolConfig::default() + }; + + let specs = image_specs(&config); + + assert_eq!(specs.len(), 1); + assert_eq!(specs[0].name, IMAGE_VIEW_TOOL_NAME); + assert!(is_image_tool_gated(IMAGE_GENERATION_TOOL_NAME, &config)); + assert!(!is_image_tool_gated(IMAGE_VIEW_TOOL_NAME, &config)); +} + +#[test] +fn image_specs_are_empty_when_runtime_support_is_disabled() { + let config = ImageToolConfig { + local_image_reads_allowed: true, + generated_image_writes_allowed: true, + ..ImageToolConfig::default() + }; + + assert!(image_specs(&config).is_empty()); + assert!(is_image_tool_gated("unknown_tool", &config)); +} + +#[test] +fn image_e2e_contract_renders_specs_and_prompt_guidance() { + let config = ImageToolConfig { + image_generation_enabled: true, + image_view_enabled: true, + image_generation_output_format: ImageGenerationOutputFormat::Jpeg, + local_image_reads_allowed: true, + generated_image_writes_allowed: true, + }; + + let specs = image_specs(&config); + let names = specs + .iter() + .map(|spec| spec.name.as_str()) + .collect::>(); + let prompt = render_image_prompt_guidance(&config, &ImagePromptOptions::default()); + + assert_eq!( + names, + vec![IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME] + ); + assert_eq!( + specs[0].parameters["properties"]["output_format"]["default"], + "jpeg" + ); + assert_eq!( + specs[1].parameters["properties"]["detail"]["default"], + "auto" + ); + assert!(prompt.contains("## Image Tools")); + assert!(prompt.contains(IMAGE_GENERATION_TOOL_NAME)); + assert!(prompt.contains(IMAGE_VIEW_TOOL_NAME)); +} + +#[test] +fn known_image_tool_names_stay_stable() { + assert_eq!( + IMAGE_TOOL_NAMES, + [IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME] + ); +} + +#[test] +fn image_spec_serializes_for_schema_catalogs() { + let spec = image_generation_spec(ImageGenerationOutputFormat::Png); + let encoded = serde_json::to_value(&spec).unwrap(); + + assert_eq!(encoded["name"], IMAGE_GENERATION_TOOL_NAME); + assert_eq!(encoded["permission"], "write"); + assert_eq!(encoded["writes_files"], true); + assert_eq!(encoded["model_visible_image_output"], false); + + let decoded: ImageToolSpec = serde_json::from_value(encoded).unwrap(); + assert_eq!(decoded, spec); +} + +#[test] +fn image_config_default_is_closed_by_capability() { + let config = ImageToolConfig::default(); + + assert!(!config.image_generation_enabled); + assert!(!config.image_view_enabled); + assert_eq!( + config.image_generation_output_format, + ImageGenerationOutputFormat::Png + ); + assert!(config.local_image_reads_allowed); + assert!(config.generated_image_writes_allowed); + assert!(image_specs(&config).is_empty()); +} diff --git a/src/openhuman/image/types.rs b/src/openhuman/image/types.rs new file mode 100644 index 000000000..77e647ea9 --- /dev/null +++ b/src/openhuman/image/types.rs @@ -0,0 +1,87 @@ +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use super::{ + image_generation_spec, image_view_spec, ImageGenerationOutputFormat, + IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME, +}; + +/// Model-facing image tool names used for filtering and policy decisions. +pub const IMAGE_TOOL_NAMES: [&str; 2] = [IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME]; + +/// A provider/runtime independent image tool descriptor. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ImageToolSpec { + /// Stable model-facing tool name. + pub name: String, + /// Concise tool description injected into prompt/tool catalogues. + pub description: String, + /// JSON Schema object for tool arguments. + pub parameters: Value, + /// Execution permission required by OpenHuman policy gates. + pub permission: ImagePermission, + /// Whether the tool payload is expected to become model-visible image + /// content rather than plain text. + pub model_visible_image_output: bool, + /// Whether execution writes files into an output directory. + pub writes_files: bool, +} + +/// Permission class for image tools. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ImagePermission { + /// Metadata or read-only local inspection. + ReadOnly, + /// Creates or edits generated media files. + Write, +} + +/// Session/runtime switches that decide which image tools are exposed. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ImageToolConfig { + /// Runtime supports hosted image generation. + pub image_generation_enabled: bool, + /// Runtime supports local image attachment/viewing. + pub image_view_enabled: bool, + /// Desired output format for generated images. + pub image_generation_output_format: ImageGenerationOutputFormat, + /// Whether the current filesystem policy allows workspace image reads. + pub local_image_reads_allowed: bool, + /// Whether generated files may be written under the configured output root. + pub generated_image_writes_allowed: bool, +} + +impl Default for ImageToolConfig { + fn default() -> Self { + Self { + image_generation_enabled: false, + image_view_enabled: false, + image_generation_output_format: ImageGenerationOutputFormat::Png, + local_image_reads_allowed: true, + generated_image_writes_allowed: true, + } + } +} + +/// Build the image specs visible to an agent for this runtime. +pub fn image_specs(config: &ImageToolConfig) -> Vec { + let mut specs = Vec::new(); + + if config.image_generation_enabled && config.generated_image_writes_allowed { + specs.push(image_generation_spec(config.image_generation_output_format)); + } + + if config.image_view_enabled && config.local_image_reads_allowed { + specs.push(image_view_spec()); + } + + specs +} + +/// Return true when an image tool should be hidden from a session. +pub fn is_image_tool_gated(tool_name: &str, config: &ImageToolConfig) -> bool { + !image_specs(config) + .iter() + .any(|spec| spec.name == tool_name) +} diff --git a/src/openhuman/mod.rs b/src/openhuman/mod.rs index ebde9b8e6..1721c034c 100644 --- a/src/openhuman/mod.rs +++ b/src/openhuman/mod.rs @@ -47,6 +47,7 @@ pub mod encryption; pub mod health; pub mod heartbeat; pub mod http_host; +pub mod image; pub mod inference; pub mod integrations; pub mod javascript;