mirror of
https://github.com/tinyhumansai/openhuman.git
synced 2026-07-27 21:08:00 +00:00
* feat(debug): add skills debug script and E2E tests - Introduced a new script `debug-skill.sh` for running end-to-end tests on skills, allowing users to easily test specific skills with customizable parameters. - Added comprehensive integration tests in `skills_debug_e2e.rs` to validate the full lifecycle of skills, including discovery, starting, tool listing, and execution. - Enhanced logging and error handling in the tests to improve observability and debugging capabilities. These additions facilitate better testing and debugging of skills, improving the overall development workflow. * feat(tests): add end-to-end tests for Skills RPC over HTTP JSON-RPC - Introduced a new test file `skills_rpc_e2e.rs` to validate the full stack of skill operations via HTTP JSON-RPC. - Implemented comprehensive tests covering skill discovery, starting, tool listing, and execution, ensuring robust functionality. - Enhanced logging for better observability during test execution, facilitating easier debugging and validation of skill interactions. These tests improve the reliability and maintainability of the skills framework by ensuring all critical operations are thoroughly validated. * refactor(tests): update RPC method names in end-to-end tests for skills - Changed RPC method names in `skills_rpc_e2e.rs` to use the new `openhuman` prefix, reflecting the updated API structure. - Updated corresponding test assertions to ensure consistency with the new method names. - Enhanced logging messages to align with the new method naming conventions, improving clarity during test execution. These changes ensure that the end-to-end tests accurately reflect the current API and improve maintainability. * feat(debug): add live debugging script and corresponding tests for Notion skill - Introduced `debug-notion-live.sh` script to facilitate debugging of the Notion skill with a live backend, including health checks and OAuth proxy testing. - Added `skills_notion_live.rs` test file to validate the Notion skill's functionality using real data and backend interactions. - Enhanced logging and error handling in both the script and tests to improve observability and debugging capabilities. These additions streamline the debugging process and ensure the Notion skill operates correctly with live data. * feat(env): enhance environment configuration for debugging scripts - Updated `.env.example` to include a new `JWT_TOKEN` variable for session management in debugging scripts. - Modified `debug-notion-live.sh` and `debug-skill.sh` scripts to load environment variables from `.env`, improving flexibility and usability. - Enhanced error handling in the scripts to ensure required variables are set, providing clearer feedback during execution. These changes streamline the debugging process for skills by ensuring necessary configurations are easily managed and accessible. * feat(tests): add disconnect flow test for skills - Introduced a new end-to-end test `skill_disconnect_flow` to validate the disconnect process for skills, mirroring the expected frontend behavior. - The test covers the stopping of a skill, handling OAuth credentials, and verifying cleanup after a disconnect. - Enhanced logging throughout the test to improve observability and debugging capabilities. These additions ensure that the disconnect flow is properly validated, improving the reliability of skill interactions. * fix(skills): revoke OAuth credentials on skill disconnect disconnectSkill() was only stopping the skill and resetting setup_complete, leaving oauth_credential.json on disk. On restart the stale credential would be restored, causing confusing auth state. Now sends oauth/revoked RPC before stopping so the event loop deletes the credential file and clears memory. Also adds revokeOAuth() and disableSkill() to the skills RPC API layer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: apply cargo fmt to skill debug tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(tests): improve skills directory discovery and error handling - Renamed `find_skills_dir` to `try_find_skills_dir`, returning an `Option<PathBuf>` to handle cases where the skills directory is not found. - Introduced a macro `require_skills_dir!` to simplify the usage of skills directory discovery in tests, providing clearer error messages when the directory is unavailable. - Updated multiple test functions to utilize the new macro, enhancing readability and maintainability of the test code. These changes improve the robustness of the skills directory discovery process and streamline the test setup. * refactor(tests): enhance skills directory discovery with improved error handling - Renamed `find_skills_dir` to `try_find_skills_dir`, returning an `Option<PathBuf>` to better handle cases where the skills directory is not found. - Introduced a new macro `require_skills_dir!` to streamline the usage of skills directory discovery in tests, providing clearer error messages when the directory is unavailable. - Updated test functions to utilize the new macro, improving code readability and maintainability. These changes enhance the robustness of the skills directory discovery process and simplify test setup. * fix(tests): skip skill tests gracefully when skills dir unavailable Tests that require the openhuman-skills repo now return early with a SKIPPED message instead of panicking when the directory is not found. Fixes CI failures where the skills repo is not checked out. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(skills): harden disconnect flow, test assertions, and secret redaction - disconnectSkill: read stored credentialId from snapshot and pass it to oauth/revoked for correct memory bucket cleanup; add host-side fallback to delete oauth_credential.json when the runtime is already stopped. - revokeOAuth: make integrationId required (no more "default" fabrication); add removePersistedOAuthCredential helper for host-side cleanup. - skills_debug_e2e: hard-assert oauth_credential.json is deleted after oauth/revoked instead of soft logging. - skills_notion_live: gate behind RUN_LIVE_NOTION=1; require all env vars (BACKEND_URL, JWT_TOKEN, CREDENTIAL_ID, SKILLS_DATA_DIR); redact JWT and credential file contents from logs. - skills_rpc_e2e: check_result renamed to assert_rpc_ok and now panics on JSON-RPC errors so protocol regressions fail fast. - debug-notion-live.sh: capture cargo exit code separately from grep/head to avoid spurious failures under set -euo pipefail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: apply cargo fmt to skills_notion_live.rs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
320 lines
11 KiB
Rust
320 lines
11 KiB
Rust
//! Notion skill live debug test — uses real data directory and backend.
|
|
//!
|
|
//! This test uses:
|
|
//! - The real ~/.openhuman/skills_data/notion/ directory (with oauth_credential.json)
|
|
//! - BACKEND_URL env var for the OAuth proxy
|
|
//! - JWT_TOKEN env var for authentication
|
|
//!
|
|
//! Run:
|
|
//! BACKEND_URL=https://staging-api.alphahuman.xyz \
|
|
//! JWT_TOKEN=<jwt> \
|
|
//! cargo test --test skills_notion_live -- --nocapture
|
|
|
|
use std::path::PathBuf;
|
|
use std::sync::Arc;
|
|
use std::time::Duration;
|
|
|
|
use serde_json::{json, Value};
|
|
|
|
use openhuman_core::openhuman::skills::qjs_engine::{set_global_engine, RuntimeEngine};
|
|
|
|
fn try_find_skills_dir() -> Option<PathBuf> {
|
|
if let Ok(dir) = std::env::var("SKILL_DEBUG_DIR") {
|
|
let p = PathBuf::from(&dir);
|
|
return if p.exists() { Some(p) } else { None };
|
|
}
|
|
let cwd = std::env::current_dir().expect("cwd");
|
|
for candidate in &[
|
|
"../openhuman-skills/skills",
|
|
"openhuman-skills/skills",
|
|
"../alphahuman/skills/skills",
|
|
] {
|
|
let p = cwd.join(candidate);
|
|
if p.exists() {
|
|
return Some(p.canonicalize().unwrap());
|
|
}
|
|
}
|
|
if let Some(parent) = cwd.parent() {
|
|
for entry in std::fs::read_dir(parent).into_iter().flatten().flatten() {
|
|
let c = entry.path().join("skills/skills");
|
|
if c.join("notion/manifest.json").exists() {
|
|
return Some(c.canonicalize().unwrap());
|
|
}
|
|
}
|
|
}
|
|
None
|
|
}
|
|
|
|
macro_rules! require_skills_dir {
|
|
() => {
|
|
match try_find_skills_dir() {
|
|
Some(dir) => dir,
|
|
None => {
|
|
eprintln!("SKIPPED: no skills directory available");
|
|
return;
|
|
}
|
|
}
|
|
};
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn notion_live_with_real_data() {
|
|
// Opt-in: only runs when RUN_LIVE_NOTION=1 is set explicitly.
|
|
if std::env::var("RUN_LIVE_NOTION").unwrap_or_default() != "1" {
|
|
eprintln!("SKIPPED: set RUN_LIVE_NOTION=1 to run this live integration test");
|
|
return;
|
|
}
|
|
|
|
let _ = env_logger::builder()
|
|
.filter_level(log::LevelFilter::Info)
|
|
.is_test(true)
|
|
.try_init();
|
|
|
|
let backend_url =
|
|
std::env::var("BACKEND_URL").expect("BACKEND_URL must be set for live Notion test");
|
|
let jwt_token = std::env::var("JWT_TOKEN").expect("JWT_TOKEN must be set for live Notion test");
|
|
let credential_id =
|
|
std::env::var("CREDENTIAL_ID").expect("CREDENTIAL_ID must be set for live Notion test");
|
|
let skills_dir = require_skills_dir!();
|
|
|
|
let real_data_dir = PathBuf::from(
|
|
std::env::var("SKILLS_DATA_DIR").expect("SKILLS_DATA_DIR must be set for live Notion test"),
|
|
);
|
|
|
|
let sep = "=".repeat(60);
|
|
eprintln!("\n{sep}");
|
|
eprintln!(" Notion Live Debug (real data dir)");
|
|
eprintln!("{sep}");
|
|
eprintln!(" Backend: {backend_url}");
|
|
eprintln!(" JWT: <redacted, {} bytes>", jwt_token.len());
|
|
eprintln!(" Credential ID: {credential_id}");
|
|
eprintln!(" Skills dir: {}", skills_dir.display());
|
|
eprintln!(" Data dir: {}", real_data_dir.display());
|
|
|
|
// Check oauth_credential.json exists (don't log contents — may contain secrets)
|
|
let cred_path = real_data_dir.join("notion/oauth_credential.json");
|
|
if cred_path.exists() {
|
|
let size = std::fs::metadata(&cred_path).map(|m| m.len()).unwrap_or(0);
|
|
eprintln!(" OAuth cred: present ({size} bytes)");
|
|
} else {
|
|
eprintln!(" OAuth cred: NOT FOUND at {}", cred_path.display());
|
|
eprintln!(" (Skill will start without OAuth — tools that need API access will fail)");
|
|
}
|
|
|
|
// ── Step 1: Raw backend check ──
|
|
eprintln!("\n--- Step 1: Backend Health ---");
|
|
let client = reqwest::Client::builder()
|
|
.timeout(Duration::from_secs(10))
|
|
.build()
|
|
.unwrap();
|
|
|
|
let health = client
|
|
.get(format!("{backend_url}/settings"))
|
|
.header("Authorization", format!("Bearer {jwt_token}"))
|
|
.send()
|
|
.await;
|
|
|
|
match health {
|
|
Ok(resp) => {
|
|
let status = resp.status();
|
|
let body = resp.text().await.unwrap_or_default();
|
|
eprintln!(" GET /settings → HTTP {status}");
|
|
if status.is_success() {
|
|
eprintln!(" ✓ Backend reachable, JWT valid");
|
|
eprintln!(" Body: {}...", &body[..body.len().min(200)]);
|
|
} else if status.as_u16() == 502 {
|
|
eprintln!(" ✗ Backend DOWN (502 Bad Gateway)");
|
|
eprintln!(" The staging server is unreachable. OAuth proxy will fail.");
|
|
eprintln!(" Continuing to test skill lifecycle anyway...");
|
|
} else {
|
|
eprintln!(" ⚠ HTTP {status}: {body}");
|
|
}
|
|
}
|
|
Err(e) => {
|
|
eprintln!(" ✗ Connection failed: {e}");
|
|
}
|
|
}
|
|
|
|
// ── Step 2: Raw proxy check ──
|
|
eprintln!("\n--- Step 2: OAuth Proxy Check ---");
|
|
let proxy_url = format!("{backend_url}/proxy/by-id/{credential_id}/v1/users?page_size=1");
|
|
eprintln!(" GET {proxy_url}");
|
|
|
|
let proxy = client
|
|
.get(&proxy_url)
|
|
.header("Authorization", format!("Bearer {jwt_token}"))
|
|
.header("Content-Type", "application/json")
|
|
.send()
|
|
.await;
|
|
|
|
match proxy {
|
|
Ok(resp) => {
|
|
let status = resp.status();
|
|
let body = resp.text().await.unwrap_or_default();
|
|
eprintln!(" HTTP {status}");
|
|
if status.is_success() {
|
|
eprintln!(" ✓ Notion API accessible via proxy");
|
|
eprintln!(" Body: {}...", &body[..body.len().min(300)]);
|
|
} else {
|
|
eprintln!(
|
|
" ✗ Proxy returned {status}: {}...",
|
|
&body[..body.len().min(200)]
|
|
);
|
|
}
|
|
}
|
|
Err(e) => {
|
|
eprintln!(" ✗ Proxy request failed: {e}");
|
|
}
|
|
}
|
|
|
|
// ── Step 3: Start skill with real data dir ──
|
|
eprintln!("\n--- Step 3: Start Notion Skill (real data dir) ---");
|
|
let engine = RuntimeEngine::new(real_data_dir.clone()).expect("engine");
|
|
let engine = Arc::new(engine);
|
|
engine.set_skills_source_dir(skills_dir.clone());
|
|
set_global_engine(engine.clone());
|
|
|
|
let start = engine.start_skill("notion").await;
|
|
match &start {
|
|
Ok(snap) => {
|
|
eprintln!(" ✓ Started — status: {:?}", snap.status);
|
|
eprintln!(" Tools: {}", snap.tools.len());
|
|
eprintln!(" Published state:");
|
|
for (k, v) in &snap.state {
|
|
if k.contains("status")
|
|
|| k.contains("error")
|
|
|| k.contains("auth")
|
|
|| k == "workspaceName"
|
|
|| k == "is_initialized"
|
|
{
|
|
eprintln!(" {k} = {v}");
|
|
}
|
|
}
|
|
}
|
|
Err(e) => {
|
|
eprintln!(" ✗ Start failed: {e}");
|
|
panic!("Skill start failed");
|
|
}
|
|
}
|
|
|
|
// ── Step 4: sync-status tool ──
|
|
eprintln!("\n--- Step 4: sync-status tool ---");
|
|
let sync_status = tokio::time::timeout(
|
|
Duration::from_secs(15),
|
|
engine.call_tool("notion", "sync-status", json!({})),
|
|
)
|
|
.await;
|
|
|
|
match sync_status {
|
|
Ok(Ok(result)) => {
|
|
for content in &result.content {
|
|
match content {
|
|
openhuman_core::openhuman::skills::types::ToolContent::Text { text } => {
|
|
// Parse and pretty-print
|
|
if let Ok(v) = serde_json::from_str::<Value>(text) {
|
|
eprintln!(" ✓ sync-status:");
|
|
eprintln!(
|
|
" connected: {}",
|
|
v.get("connected").unwrap_or(&json!(null))
|
|
);
|
|
eprintln!(
|
|
" workspace: {}",
|
|
v.get("workspace_name").unwrap_or(&json!(null))
|
|
);
|
|
eprintln!(
|
|
" last_sync: {}",
|
|
v.get("last_sync_time").unwrap_or(&json!(null))
|
|
);
|
|
eprintln!(
|
|
" last_sync_error: {}",
|
|
v.get("last_sync_error").unwrap_or(&json!(null))
|
|
);
|
|
if let Some(totals) = v.get("totals") {
|
|
eprintln!(" totals: {totals}");
|
|
}
|
|
} else {
|
|
eprintln!(" ✓ Raw: {text}");
|
|
}
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
}
|
|
Ok(Err(e)) => eprintln!(" ✗ Error: {e}"),
|
|
Err(_) => eprintln!(" ✗ TIMED OUT"),
|
|
}
|
|
|
|
// ── Step 5: search tool (needs OAuth) ──
|
|
eprintln!("\n--- Step 5: search tool (needs OAuth proxy) ---");
|
|
let search = tokio::time::timeout(
|
|
Duration::from_secs(30),
|
|
engine.call_tool("notion", "search", json!({"query": "test", "page_size": 3})),
|
|
)
|
|
.await;
|
|
|
|
match search {
|
|
Ok(Ok(result)) => {
|
|
eprintln!(" is_error: {}", result.is_error);
|
|
for content in &result.content {
|
|
match content {
|
|
openhuman_core::openhuman::skills::types::ToolContent::Text { text } => {
|
|
eprintln!(" Result: {}...", &text[..text.len().min(500)]);
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
}
|
|
Ok(Err(e)) => eprintln!(" ✗ Error: {e}"),
|
|
Err(_) => eprintln!(" ✗ TIMED OUT"),
|
|
}
|
|
|
|
// ── Step 6: list-all-pages tool (needs OAuth) ──
|
|
eprintln!("\n--- Step 6: list-all-pages (needs OAuth proxy) ---");
|
|
let pages = tokio::time::timeout(
|
|
Duration::from_secs(30),
|
|
engine.call_tool("notion", "list-all-pages", json!({"page_size": 3})),
|
|
)
|
|
.await;
|
|
|
|
match pages {
|
|
Ok(Ok(result)) => {
|
|
eprintln!(" is_error: {}", result.is_error);
|
|
for content in &result.content {
|
|
match content {
|
|
openhuman_core::openhuman::skills::types::ToolContent::Text { text } => {
|
|
eprintln!(" Result: {}...", &text[..text.len().min(500)]);
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
}
|
|
Ok(Err(e)) => eprintln!(" ✗ Error: {e}"),
|
|
Err(_) => eprintln!(" ✗ TIMED OUT"),
|
|
}
|
|
|
|
// ── Step 7: Final state ──
|
|
eprintln!("\n--- Step 7: Final Skill State ---");
|
|
if let Some(snap) = engine.get_skill_state("notion") {
|
|
eprintln!(" Status: {:?}", snap.status);
|
|
for (k, v) in &snap.state {
|
|
if k.contains("status")
|
|
|| k.contains("error")
|
|
|| k.contains("auth")
|
|
|| k == "workspaceName"
|
|
{
|
|
eprintln!(" {k} = {v}");
|
|
}
|
|
}
|
|
}
|
|
|
|
// ── Cleanup ──
|
|
eprintln!("\n--- Stop ---");
|
|
let _ = engine.stop_skill("notion").await;
|
|
eprintln!(" Done");
|
|
|
|
let sep = "=".repeat(60);
|
|
eprintln!("\n{sep}");
|
|
eprintln!(" COMPLETE");
|
|
eprintln!("{sep}\n");
|
|
}
|