mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
feat(image): add image tool contracts (#2997)
This commit is contained in:
Steven Enamakel
@@ -190,6 +190,7 @@ Canonical mapping of every product feature to its test source(s). Drives gap-fil
|
||||
| 4.3.2 | Permission-Based Execution | RU+WD | `src/openhuman/tools/`, `skill-execution-flow.spec.ts` | ✅ | |
|
||||
| 4.3.3 | Tool Failure Handling | WD | `skill-execution-flow.spec.ts` | ✅ | |
|
||||
| 4.3.4 | Subagent Mascot Visualization | VU | `app/src/features/human/SubMascotLayer.test.tsx`, `app/src/features/human/HumanPage.test.tsx` | ✅ | Renders spawned/completed/failed subagent timeline rows as colored companion mascots with activity bubbles |
|
||||
| 4.3.5 | Image Tool Contracts | RU | `src/openhuman/image/` | ✅ | High-level `image_generation` / `view_image` schema, gating, serialization, prompt guidance, and contract e2e coverage for #2984 |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
* [Local AI (optional)](features/model-routing/local-ai.md)
|
||||
* [Available Tools](features/native-tools/README.md)
|
||||
* [Web Search](features/native-tools/web-search.md)
|
||||
* [Image Tools](features/native-tools/image-tools.md)
|
||||
* [Web Scraper](features/native-tools/web-scraper.md)
|
||||
* [Coder](features/native-tools/coder.md)
|
||||
* [Browser & Computer Control](features/native-tools/browser-and-computer.md)
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
# Image Tools
|
||||
|
||||
OpenHuman's image contract gives agents a stable way to reason about
|
||||
image generation and local image inspection without tying the prompt surface to a
|
||||
single provider runtime.
|
||||
|
||||
## Scope
|
||||
|
||||
The contract lives in the top-level `src/openhuman/image/` module and
|
||||
currently covers two model-facing tools:
|
||||
|
||||
| Tool | Purpose | Permission | Output |
|
||||
| --- | --- | --- | --- |
|
||||
| `image_generation` | Generate or edit raster images from a prompt. | Write | Local generated-media artifact paths. |
|
||||
| `view_image` | Load a local image file into model-visible image context. | Read-only | Image content visible to the model. |
|
||||
|
||||
This layer is intentionally high level. Existing lower-level tools still own
|
||||
their concrete behavior:
|
||||
|
||||
- `image_info` reads local image metadata and optional base64 text.
|
||||
- Browser screenshot tooling captures pages and writes image files.
|
||||
- Agent multimodal preparation normalizes `[IMAGE:...]` markers for providers
|
||||
that accept image data.
|
||||
|
||||
The image layer defines names, schemas, gating, and prompt rules so
|
||||
agents can make consistent decisions as runtimes add direct support.
|
||||
|
||||
## `image_generation`
|
||||
|
||||
`image_generation` is a hosted provider capability. The Rust core should not
|
||||
pretend to be an image renderer when no provider supports it. When enabled, the
|
||||
runtime should:
|
||||
|
||||
1. Validate any `input_image_path` through the same local-file policy used for
|
||||
image viewing.
|
||||
2. Send the prompt and optional edit image to the hosted image provider.
|
||||
3. Persist returned bytes under a session-scoped generated-media root, or under
|
||||
an approved caller-provided `output_path`.
|
||||
4. Return saved artifact paths so the final assistant answer can reference them.
|
||||
|
||||
The schema includes `prompt`, optional `output_path`, optional `size`, optional
|
||||
`input_image_path`, and `output_format` (`png`, `webp`, `jpeg`).
|
||||
|
||||
## `view_image`
|
||||
|
||||
`view_image` loads pixels from a local file into model-visible context. Use it
|
||||
when text metadata is insufficient: screenshots, UI review, OCR, diagrams,
|
||||
charts, visual diffs, and generated-image inspection.
|
||||
|
||||
The runtime must keep local file boundaries explicit:
|
||||
|
||||
- Allow paths in the approved workspace.
|
||||
- Allow paths created during the current session.
|
||||
- Allow paths explicitly referenced by the user or trusted tool output.
|
||||
- Deny paths outside policy, and do not silently attach unrelated local images.
|
||||
|
||||
The schema includes `path` and optional `detail` (`auto`, `high`, `original`).
|
||||
Use `original` only when full-resolution inspection is necessary.
|
||||
|
||||
## Prompt Guidance
|
||||
|
||||
Prompt rendering should include image guidance only when at least one
|
||||
media tool is enabled. The guidance should tell agents:
|
||||
|
||||
- Use `view_image` when pixels are needed, not for ordinary file metadata.
|
||||
- Use `image_generation` for requested raster image creation or edits.
|
||||
- Provide an output path when the destination matters.
|
||||
- Mention generated artifact paths in final answers.
|
||||
- Respect local image boundaries before attaching a file to model context.
|
||||
|
||||
## Tests
|
||||
|
||||
The module has focused Rust tests for:
|
||||
|
||||
- JSON schema shape for `image_generation`.
|
||||
- JSON schema shape for `view_image`.
|
||||
- Independent gating of generation vs local viewing.
|
||||
- End-to-end contract rendering from config to specs and prompt guidance.
|
||||
|
||||
Future runtime PRs should add provider-specific execution tests next to the
|
||||
runtime adapter, not in the hosted contract module.
|
||||
@@ -0,0 +1,43 @@
|
||||
# image
|
||||
|
||||
High-level contracts for image-capable agent runtimes.
|
||||
|
||||
This module does not execute image generation or pixel inspection directly. It
|
||||
defines the stable model-facing contracts that provider/runtime adapters can
|
||||
expose when image capabilities are available.
|
||||
|
||||
## Responsibilities
|
||||
|
||||
- Define the `image_generation` contract for hosted raster image creation and
|
||||
edits.
|
||||
- Define the `view_image` contract for loading local image files into
|
||||
model-visible image context.
|
||||
- Gate contract exposure by runtime support and local policy:
|
||||
- generated image writes
|
||||
- local image reads
|
||||
- Render concise prompt guidance for agents when image tools are available.
|
||||
- Keep schema/serialization contracts covered by focused Rust tests.
|
||||
|
||||
## Module Shape
|
||||
|
||||
| File | Role |
|
||||
| --- | --- |
|
||||
| `mod.rs` | Export-only module entrypoint. |
|
||||
| `types.rs` | Shared descriptors, permission/config types, and gating helpers. |
|
||||
| `image_generation.rs` | `image_generation` schema and output-format contract. |
|
||||
| `image_view.rs` | `view_image` schema and detail-level contract. |
|
||||
| `prompt.rs` | Agent prompt guidance for enabled image tools. |
|
||||
| `tests.rs` | Contract-level e2e tests across config, schemas, and prompt output. |
|
||||
|
||||
## Notes
|
||||
|
||||
Existing lower-level image helpers remain separate:
|
||||
|
||||
- `image_info` reads metadata/base64 text.
|
||||
- Browser screenshot tooling captures page images.
|
||||
- Multimodal `[IMAGE:...]` preparation normalizes image references for
|
||||
image-capable providers.
|
||||
|
||||
Future execution adapters should depend on these contracts and live next to the
|
||||
runtime/provider implementation that actually performs the hosted call or model
|
||||
attachment.
|
||||
@@ -0,0 +1,132 @@
|
||||
//! Contract for the hosted `image_generation` tool.
|
||||
//!
|
||||
//! The tool is a hosted/provider capability, not a local Rust image renderer.
|
||||
//! Runtimes that support it should persist generated bytes under a
|
||||
//! session-scoped generated-media root, then return stable file paths in tool
|
||||
//! output so final answers can reference concrete artifacts.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::json;
|
||||
|
||||
use super::{ImagePermission, ImageToolSpec};
|
||||
|
||||
/// Stable model-facing tool name.
|
||||
pub const IMAGE_GENERATION_TOOL_NAME: &str = "image_generation";
|
||||
|
||||
/// Generated image file format requested from the hosted provider.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ImageGenerationOutputFormat {
|
||||
Png,
|
||||
Webp,
|
||||
Jpeg,
|
||||
}
|
||||
|
||||
impl ImageGenerationOutputFormat {
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Png => "png",
|
||||
Self::Webp => "webp",
|
||||
Self::Jpeg => "jpeg",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the hosted `image_generation` model-facing contract.
|
||||
pub fn image_generation_spec(output_format: ImageGenerationOutputFormat) -> ImageToolSpec {
|
||||
ImageToolSpec {
|
||||
name: IMAGE_GENERATION_TOOL_NAME.to_string(),
|
||||
description: "Generate or edit raster images from a prompt and store each result as a local artifact path for the user.".to_string(),
|
||||
parameters: json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"prompt": {
|
||||
"type": "string",
|
||||
"description": "Detailed visual prompt or edit instruction for the hosted image model."
|
||||
},
|
||||
"output_path": {
|
||||
"type": "string",
|
||||
"description": "Optional workspace-relative or approved absolute output path. When omitted, the runtime chooses a generated-media path."
|
||||
},
|
||||
"size": {
|
||||
"type": "string",
|
||||
"description": "Optional output size such as 1024x1024, 1536x1024, or provider default."
|
||||
},
|
||||
"input_image_path": {
|
||||
"type": "string",
|
||||
"description": "Optional local image path to edit. The runtime must validate local-file access before attaching it."
|
||||
},
|
||||
"output_format": {
|
||||
"type": "string",
|
||||
"enum": ["png", "webp", "jpeg"],
|
||||
"default": output_format.as_str(),
|
||||
"description": "Requested persisted file format."
|
||||
}
|
||||
},
|
||||
"required": ["prompt"]
|
||||
}),
|
||||
permission: ImagePermission::Write,
|
||||
model_visible_image_output: false,
|
||||
writes_files: true,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn image_generation_schema_declares_output_path_and_format() {
|
||||
let spec = image_generation_spec(ImageGenerationOutputFormat::Webp);
|
||||
|
||||
assert_eq!(spec.name, "image_generation");
|
||||
assert!(spec.description.contains("Generate or edit raster images"));
|
||||
assert_eq!(spec.permission, ImagePermission::Write);
|
||||
assert!(!spec.model_visible_image_output);
|
||||
assert!(spec.writes_files);
|
||||
assert_eq!(spec.parameters["required"], serde_json::json!(["prompt"]));
|
||||
assert!(spec.parameters["properties"].get("output_path").is_some());
|
||||
assert_eq!(
|
||||
spec.parameters["properties"]["output_format"]["default"],
|
||||
"webp"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn output_format_serializes_as_snake_case() {
|
||||
assert_eq!(ImageGenerationOutputFormat::Png.as_str(), "png");
|
||||
assert_eq!(ImageGenerationOutputFormat::Webp.as_str(), "webp");
|
||||
assert_eq!(ImageGenerationOutputFormat::Jpeg.as_str(), "jpeg");
|
||||
|
||||
assert_eq!(
|
||||
serde_json::to_value(ImageGenerationOutputFormat::Webp).unwrap(),
|
||||
serde_json::json!("webp")
|
||||
);
|
||||
assert_eq!(
|
||||
serde_json::from_value::<ImageGenerationOutputFormat>(serde_json::json!("jpeg"))
|
||||
.unwrap(),
|
||||
ImageGenerationOutputFormat::Jpeg
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_generation_schema_keeps_edit_and_size_inputs_optional() {
|
||||
let spec = image_generation_spec(ImageGenerationOutputFormat::Png);
|
||||
let properties = spec.parameters["properties"].as_object().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
properties.keys().cloned().collect::<Vec<_>>(),
|
||||
vec![
|
||||
"input_image_path".to_string(),
|
||||
"output_format".to_string(),
|
||||
"output_path".to_string(),
|
||||
"prompt".to_string(),
|
||||
"size".to_string()
|
||||
]
|
||||
);
|
||||
assert_eq!(
|
||||
properties["output_format"]["enum"],
|
||||
serde_json::json!(["png", "webp", "jpeg"])
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
//! Contract for the hosted `view_image` tool.
|
||||
//!
|
||||
//! `view_image` bridges local image files into model-visible image content. It
|
||||
//! is distinct from `image_info`: metadata extraction can stay textual, while
|
||||
//! `view_image` asks the runtime to load pixels into the conversation context.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::json;
|
||||
|
||||
use super::{ImagePermission, ImageToolSpec};
|
||||
|
||||
/// Stable model-facing tool name.
|
||||
pub const IMAGE_VIEW_TOOL_NAME: &str = "view_image";
|
||||
|
||||
/// Requested image-detail level for model-visible inspection.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ImageDetail {
|
||||
Auto,
|
||||
High,
|
||||
Original,
|
||||
}
|
||||
|
||||
impl ImageDetail {
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Auto => "auto",
|
||||
Self::High => "high",
|
||||
Self::Original => "original",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the hosted `view_image` model-facing contract.
|
||||
pub fn image_view_spec() -> ImageToolSpec {
|
||||
ImageToolSpec {
|
||||
name: IMAGE_VIEW_TOOL_NAME.to_string(),
|
||||
description: "Load a local image file into model-visible image context for inspection, OCR, UI review, or visual reasoning.".to_string(),
|
||||
parameters: json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": "Local image path, absolute or relative to the approved workspace."
|
||||
},
|
||||
"detail": {
|
||||
"type": "string",
|
||||
"enum": ["auto", "high", "original"],
|
||||
"default": ImageDetail::Auto.as_str(),
|
||||
"description": "Inspection detail. Use original only when full resolution is necessary."
|
||||
}
|
||||
},
|
||||
"required": ["path"]
|
||||
}),
|
||||
permission: ImagePermission::ReadOnly,
|
||||
model_visible_image_output: true,
|
||||
writes_files: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn view_image_schema_requires_path_and_marks_model_visible_output() {
|
||||
let spec = image_view_spec();
|
||||
|
||||
assert_eq!(spec.name, "view_image");
|
||||
assert!(spec.description.contains("model-visible image context"));
|
||||
assert_eq!(spec.permission, ImagePermission::ReadOnly);
|
||||
assert!(spec.model_visible_image_output);
|
||||
assert!(!spec.writes_files);
|
||||
assert_eq!(spec.parameters["required"], serde_json::json!(["path"]));
|
||||
assert_eq!(spec.parameters["properties"]["detail"]["default"], "auto");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn detail_names_match_prompt_contract() {
|
||||
assert_eq!(ImageDetail::Auto.as_str(), "auto");
|
||||
assert_eq!(ImageDetail::High.as_str(), "high");
|
||||
assert_eq!(ImageDetail::Original.as_str(), "original");
|
||||
|
||||
assert_eq!(
|
||||
serde_json::to_value(ImageDetail::Original).unwrap(),
|
||||
serde_json::json!("original")
|
||||
);
|
||||
assert_eq!(
|
||||
serde_json::from_value::<ImageDetail>(serde_json::json!("high")).unwrap(),
|
||||
ImageDetail::High
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn view_image_schema_lists_supported_detail_levels() {
|
||||
let spec = image_view_spec();
|
||||
|
||||
assert_eq!(
|
||||
spec.parameters["properties"]["detail"]["enum"],
|
||||
serde_json::json!(["auto", "high", "original"])
|
||||
);
|
||||
assert_eq!(spec.parameters["properties"]["path"]["type"], "string");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
//! Image-tool contracts for model-facing agents.
|
||||
//!
|
||||
//! This module is intentionally a high-level contract layer. OpenHuman already
|
||||
//! has lower-level image helpers (`image_info`, browser screenshots, and
|
||||
//! multimodal `[IMAGE:...]` normalization). The image layer defines the
|
||||
//! stable tool names, schema, gating, and prompt guidance that agents should see
|
||||
//! when a runtime can provide Codex-like media tools.
|
||||
//!
|
||||
//! The two first-class contracts are:
|
||||
//!
|
||||
//! - [`image_generation`] — create or edit raster images and return stored file
|
||||
//! references.
|
||||
//! - [`image_view`] — attach a local image file as model-visible image content
|
||||
//! so the agent can inspect it.
|
||||
//!
|
||||
//! Keeping this contract separate from execution lets provider runtimes adopt
|
||||
//! the surface incrementally without duplicating business logic in the tools
|
||||
//! registry.
|
||||
|
||||
pub mod image_generation;
|
||||
pub mod image_view;
|
||||
pub mod prompt;
|
||||
pub mod types;
|
||||
|
||||
pub use image_generation::{
|
||||
image_generation_spec, ImageGenerationOutputFormat, IMAGE_GENERATION_TOOL_NAME,
|
||||
};
|
||||
pub use image_view::{image_view_spec, ImageDetail, IMAGE_VIEW_TOOL_NAME};
|
||||
pub use prompt::{render_image_prompt_guidance, ImagePromptOptions};
|
||||
pub use types::{
|
||||
image_specs, is_image_tool_gated, ImagePermission, ImageToolConfig, ImageToolSpec,
|
||||
IMAGE_TOOL_NAMES,
|
||||
};
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "tests.rs"]
|
||||
mod tests;
|
||||
@@ -0,0 +1,160 @@
|
||||
//! Prompt guidance for image tools.
|
||||
|
||||
use super::{ImageToolConfig, IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME};
|
||||
|
||||
/// Rendering options for image prompt guidance.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ImagePromptOptions {
|
||||
/// Include final-answer artifact-reference guidance for generated images.
|
||||
pub include_final_answer_rules: bool,
|
||||
/// Include local-file privacy boundaries for viewed images.
|
||||
pub include_local_file_boundaries: bool,
|
||||
}
|
||||
|
||||
impl Default for ImagePromptOptions {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
include_final_answer_rules: true,
|
||||
include_local_file_boundaries: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Render concise model guidance for enabled image tools.
|
||||
pub fn render_image_prompt_guidance(
|
||||
config: &ImageToolConfig,
|
||||
options: &ImagePromptOptions,
|
||||
) -> String {
|
||||
let image_generation_available =
|
||||
config.image_generation_enabled && config.generated_image_writes_allowed;
|
||||
let image_view_available = config.image_view_enabled && config.local_image_reads_allowed;
|
||||
|
||||
if !image_generation_available && !image_view_available {
|
||||
return String::new();
|
||||
}
|
||||
|
||||
let mut out = String::from("## Image Tools\n\n");
|
||||
|
||||
if image_view_available {
|
||||
out.push_str(&format!(
|
||||
"- Use `{IMAGE_VIEW_TOOL_NAME}` when image pixels are needed for UI review, OCR, chart inspection, visual comparison, or understanding a local screenshot.\n"
|
||||
));
|
||||
if options.include_local_file_boundaries {
|
||||
out.push_str(
|
||||
"- Only view local images that are in the approved workspace, were created during this session, or were explicitly referenced by the user or trusted tool output.\n",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if image_generation_available {
|
||||
out.push_str(&format!(
|
||||
"- Use `{IMAGE_GENERATION_TOOL_NAME}` for requested raster image creation or image edits; provide a specific prompt and an output path when the destination matters.\n"
|
||||
));
|
||||
if options.include_final_answer_rules {
|
||||
out.push_str(
|
||||
"- After generating images, mention the saved artifact path in the final answer so the user can find it.\n",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
out.trim_end().to_string()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::openhuman::image::ImageGenerationOutputFormat;
|
||||
|
||||
#[test]
|
||||
fn prompt_guidance_is_empty_when_no_image_tools_are_enabled() {
|
||||
let rendered = render_image_prompt_guidance(
|
||||
&ImageToolConfig::default(),
|
||||
&ImagePromptOptions::default(),
|
||||
);
|
||||
|
||||
assert!(rendered.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prompt_guidance_renders_image_generation_and_view_rules() {
|
||||
let config = ImageToolConfig {
|
||||
image_generation_enabled: true,
|
||||
image_view_enabled: true,
|
||||
image_generation_output_format: ImageGenerationOutputFormat::Png,
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: true,
|
||||
};
|
||||
|
||||
let rendered = render_image_prompt_guidance(&config, &ImagePromptOptions::default());
|
||||
|
||||
assert!(rendered.contains("## Image Tools"));
|
||||
assert!(rendered.contains("`view_image`"));
|
||||
assert!(rendered.contains("`image_generation`"));
|
||||
assert!(rendered.contains("saved artifact path"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prompt_guidance_can_omit_optional_rule_text() {
|
||||
let config = ImageToolConfig {
|
||||
image_generation_enabled: true,
|
||||
image_view_enabled: true,
|
||||
image_generation_output_format: ImageGenerationOutputFormat::Png,
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: true,
|
||||
};
|
||||
let options = ImagePromptOptions {
|
||||
include_final_answer_rules: false,
|
||||
include_local_file_boundaries: false,
|
||||
};
|
||||
|
||||
let rendered = render_image_prompt_guidance(&config, &options);
|
||||
|
||||
assert!(rendered.contains("`view_image`"));
|
||||
assert!(rendered.contains("`image_generation`"));
|
||||
assert!(!rendered.contains("approved workspace"));
|
||||
assert!(!rendered.contains("saved artifact path"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prompt_guidance_respects_policy_gates() {
|
||||
let config = ImageToolConfig {
|
||||
image_generation_enabled: true,
|
||||
image_view_enabled: true,
|
||||
image_generation_output_format: ImageGenerationOutputFormat::Png,
|
||||
local_image_reads_allowed: false,
|
||||
generated_image_writes_allowed: false,
|
||||
};
|
||||
|
||||
let rendered = render_image_prompt_guidance(&config, &ImagePromptOptions::default());
|
||||
|
||||
assert!(rendered.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prompt_guidance_renders_single_available_tool() {
|
||||
let generation_only = ImageToolConfig {
|
||||
image_generation_enabled: true,
|
||||
image_view_enabled: false,
|
||||
image_generation_output_format: ImageGenerationOutputFormat::Png,
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: true,
|
||||
};
|
||||
let view_only = ImageToolConfig {
|
||||
image_generation_enabled: false,
|
||||
image_view_enabled: true,
|
||||
image_generation_output_format: ImageGenerationOutputFormat::Png,
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: true,
|
||||
};
|
||||
|
||||
let generation_rendered =
|
||||
render_image_prompt_guidance(&generation_only, &ImagePromptOptions::default());
|
||||
let view_rendered =
|
||||
render_image_prompt_guidance(&view_only, &ImagePromptOptions::default());
|
||||
|
||||
assert!(generation_rendered.contains("`image_generation`"));
|
||||
assert!(!generation_rendered.contains("`view_image`"));
|
||||
assert!(view_rendered.contains("`view_image`"));
|
||||
assert!(!view_rendered.contains("`image_generation`"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn image_specs_gate_each_tool_independently() {
|
||||
let config = ImageToolConfig {
|
||||
image_generation_enabled: true,
|
||||
image_view_enabled: true,
|
||||
local_image_reads_allowed: false,
|
||||
generated_image_writes_allowed: true,
|
||||
..ImageToolConfig::default()
|
||||
};
|
||||
|
||||
let specs = image_specs(&config);
|
||||
|
||||
assert_eq!(specs.len(), 1);
|
||||
assert_eq!(specs[0].name, IMAGE_GENERATION_TOOL_NAME);
|
||||
assert!(!is_image_tool_gated(IMAGE_GENERATION_TOOL_NAME, &config));
|
||||
assert!(is_image_tool_gated(IMAGE_VIEW_TOOL_NAME, &config));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_specs_hide_generation_when_writes_are_blocked() {
|
||||
let config = ImageToolConfig {
|
||||
image_generation_enabled: true,
|
||||
image_view_enabled: true,
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: false,
|
||||
..ImageToolConfig::default()
|
||||
};
|
||||
|
||||
let specs = image_specs(&config);
|
||||
|
||||
assert_eq!(specs.len(), 1);
|
||||
assert_eq!(specs[0].name, IMAGE_VIEW_TOOL_NAME);
|
||||
assert!(is_image_tool_gated(IMAGE_GENERATION_TOOL_NAME, &config));
|
||||
assert!(!is_image_tool_gated(IMAGE_VIEW_TOOL_NAME, &config));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_specs_are_empty_when_runtime_support_is_disabled() {
|
||||
let config = ImageToolConfig {
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: true,
|
||||
..ImageToolConfig::default()
|
||||
};
|
||||
|
||||
assert!(image_specs(&config).is_empty());
|
||||
assert!(is_image_tool_gated("unknown_tool", &config));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_e2e_contract_renders_specs_and_prompt_guidance() {
|
||||
let config = ImageToolConfig {
|
||||
image_generation_enabled: true,
|
||||
image_view_enabled: true,
|
||||
image_generation_output_format: ImageGenerationOutputFormat::Jpeg,
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: true,
|
||||
};
|
||||
|
||||
let specs = image_specs(&config);
|
||||
let names = specs
|
||||
.iter()
|
||||
.map(|spec| spec.name.as_str())
|
||||
.collect::<Vec<_>>();
|
||||
let prompt = render_image_prompt_guidance(&config, &ImagePromptOptions::default());
|
||||
|
||||
assert_eq!(
|
||||
names,
|
||||
vec![IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME]
|
||||
);
|
||||
assert_eq!(
|
||||
specs[0].parameters["properties"]["output_format"]["default"],
|
||||
"jpeg"
|
||||
);
|
||||
assert_eq!(
|
||||
specs[1].parameters["properties"]["detail"]["default"],
|
||||
"auto"
|
||||
);
|
||||
assert!(prompt.contains("## Image Tools"));
|
||||
assert!(prompt.contains(IMAGE_GENERATION_TOOL_NAME));
|
||||
assert!(prompt.contains(IMAGE_VIEW_TOOL_NAME));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn known_image_tool_names_stay_stable() {
|
||||
assert_eq!(
|
||||
IMAGE_TOOL_NAMES,
|
||||
[IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_spec_serializes_for_schema_catalogs() {
|
||||
let spec = image_generation_spec(ImageGenerationOutputFormat::Png);
|
||||
let encoded = serde_json::to_value(&spec).unwrap();
|
||||
|
||||
assert_eq!(encoded["name"], IMAGE_GENERATION_TOOL_NAME);
|
||||
assert_eq!(encoded["permission"], "write");
|
||||
assert_eq!(encoded["writes_files"], true);
|
||||
assert_eq!(encoded["model_visible_image_output"], false);
|
||||
|
||||
let decoded: ImageToolSpec = serde_json::from_value(encoded).unwrap();
|
||||
assert_eq!(decoded, spec);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn image_config_default_is_closed_by_capability() {
|
||||
let config = ImageToolConfig::default();
|
||||
|
||||
assert!(!config.image_generation_enabled);
|
||||
assert!(!config.image_view_enabled);
|
||||
assert_eq!(
|
||||
config.image_generation_output_format,
|
||||
ImageGenerationOutputFormat::Png
|
||||
);
|
||||
assert!(config.local_image_reads_allowed);
|
||||
assert!(config.generated_image_writes_allowed);
|
||||
assert!(image_specs(&config).is_empty());
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
|
||||
use super::{
|
||||
image_generation_spec, image_view_spec, ImageGenerationOutputFormat,
|
||||
IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME,
|
||||
};
|
||||
|
||||
/// Model-facing image tool names used for filtering and policy decisions.
|
||||
pub const IMAGE_TOOL_NAMES: [&str; 2] = [IMAGE_GENERATION_TOOL_NAME, IMAGE_VIEW_TOOL_NAME];
|
||||
|
||||
/// A provider/runtime independent image tool descriptor.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct ImageToolSpec {
|
||||
/// Stable model-facing tool name.
|
||||
pub name: String,
|
||||
/// Concise tool description injected into prompt/tool catalogues.
|
||||
pub description: String,
|
||||
/// JSON Schema object for tool arguments.
|
||||
pub parameters: Value,
|
||||
/// Execution permission required by OpenHuman policy gates.
|
||||
pub permission: ImagePermission,
|
||||
/// Whether the tool payload is expected to become model-visible image
|
||||
/// content rather than plain text.
|
||||
pub model_visible_image_output: bool,
|
||||
/// Whether execution writes files into an output directory.
|
||||
pub writes_files: bool,
|
||||
}
|
||||
|
||||
/// Permission class for image tools.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum ImagePermission {
|
||||
/// Metadata or read-only local inspection.
|
||||
ReadOnly,
|
||||
/// Creates or edits generated media files.
|
||||
Write,
|
||||
}
|
||||
|
||||
/// Session/runtime switches that decide which image tools are exposed.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ImageToolConfig {
|
||||
/// Runtime supports hosted image generation.
|
||||
pub image_generation_enabled: bool,
|
||||
/// Runtime supports local image attachment/viewing.
|
||||
pub image_view_enabled: bool,
|
||||
/// Desired output format for generated images.
|
||||
pub image_generation_output_format: ImageGenerationOutputFormat,
|
||||
/// Whether the current filesystem policy allows workspace image reads.
|
||||
pub local_image_reads_allowed: bool,
|
||||
/// Whether generated files may be written under the configured output root.
|
||||
pub generated_image_writes_allowed: bool,
|
||||
}
|
||||
|
||||
impl Default for ImageToolConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
image_generation_enabled: false,
|
||||
image_view_enabled: false,
|
||||
image_generation_output_format: ImageGenerationOutputFormat::Png,
|
||||
local_image_reads_allowed: true,
|
||||
generated_image_writes_allowed: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the image specs visible to an agent for this runtime.
|
||||
pub fn image_specs(config: &ImageToolConfig) -> Vec<ImageToolSpec> {
|
||||
let mut specs = Vec::new();
|
||||
|
||||
if config.image_generation_enabled && config.generated_image_writes_allowed {
|
||||
specs.push(image_generation_spec(config.image_generation_output_format));
|
||||
}
|
||||
|
||||
if config.image_view_enabled && config.local_image_reads_allowed {
|
||||
specs.push(image_view_spec());
|
||||
}
|
||||
|
||||
specs
|
||||
}
|
||||
|
||||
/// Return true when an image tool should be hidden from a session.
|
||||
pub fn is_image_tool_gated(tool_name: &str, config: &ImageToolConfig) -> bool {
|
||||
!image_specs(config)
|
||||
.iter()
|
||||
.any(|spec| spec.name == tool_name)
|
||||
}
|
||||
@@ -47,6 +47,7 @@ pub mod encryption;
|
||||
pub mod health;
|
||||
pub mod heartbeat;
|
||||
pub mod http_host;
|
||||
pub mod image;
|
||||
pub mod inference;
|
||||
pub mod integrations;
|
||||
pub mod javascript;
|
||||
|
||||
Reference in New Issue
Block a user