Commit Graph
153 Commits
Author SHA1 Message Date
43b3a59033 feat(evals): TauBench V2 native integration + GAIA eval configs (#162)
* feat(evals): add TauBench V2 native integration + GAIA eval configs

Integrate TauBench V2 (τ²-bench) multi-turn customer service benchmark
natively through OpenJarvis's inference engine. The agent's LLM calls go
through OpenJarvis while tau2-bench handles the orchestration, user
simulation, domain tools, database, and evaluation.

## TauBench Integration

Architecture: JarvisHalfDuplexAgent bridges OpenJarvis's engine into
tau2's Orchestrator as a drop-in agent replacement. This enables
testing how well OpenJarvis's Intelligence + Engine handles multi-turn
customer service tasks with real tool calling and database mutations.

Key features:
- Native OpenJarvis engine for agent LLM calls
- tau2's UserSimulator for realistic customer interactions
- Domain tools (airline, retail, telecom) with mock databases
- Full evaluation: DB state checks, action matching, NL assertions
- Test-split filtering for leaderboard-comparable results
- Pass^k multi-trial support (default 3 trials per task)
- Qwen thinking-mode disabled for clean tool call parsing
- Gemini thought_signature handling for multi-turn conversations

Files:
- datasets/taubench.py: Dataset provider with test-split filtering
- execution/taubench_env.py: JarvisHalfDuplexAgent + simulation runner
- scorers/taubench.py: Scorer reading tau2 evaluation rewards
- CLI registration and KNOWN_BENCHMARKS update

## Results (test split, pass^3, 60 tasks)

| Model              | TauBench | Leaderboard |
|--------------------|----------|-------------|
| Claude Opus 4.6    | 86.67%   | 84.8%       |
| Nemotron-3-Super   | 86.67%   | —           |
| Qwen3.5-397B       | 81.67%   | 95.6%       |
| GPT-5.4            | 81.67%   | 91.5%       |
| Qwen3.5-122B       | 80.00%   | 93.6%       |
| Qwen3.5-35B        | 77.27%   | 89.2%       |
| Gemini 3.1 Pro     | 58.33%   | ~87%        |

## GAIA Eval Configs

Added configs for GPT-5.4, Gemini 3.1 Pro, Nemotron, Qwen 122B,
Qwen 35B, and existing GAIA rerun configs for multiple models.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: lint errors in taubench integration

Remove unused imports and sort import blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: E501 line too long in slack_connector.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove unrelated GAIA configs from PR

Keep only TauBench configs that were created and tested in this PR.
GAIA configs are pre-existing or belong in a separate PR.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 14:59:20 -07:00
Jon Saad-FalconandGitHub 09d1cc3681 Merge pull request #152 from open-jarvis/feat/security-hardening
feat: security hardening — layered boundary enforcement
2026-03-28 22:15:22 -07:00
Jon Saad-FalconandClaude Opus 4.6 0cbd0e1dac fix: update existing tests for new secure defaults
test_config.py: SecurityConfig.mode default changed from "warn" to "redact"
test_config_phase3.py: ServerConfig.host default changed from "0.0.0.0" to "127.0.0.1"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 22:02:02 -07:00
Jon Saad-FalconandClaude Opus 4.6 f122f5a699 fix: tests work without Rust extension and server extras
- BoundaryGuard degrades gracefully when Rust scanners unavailable
- BoundaryGuard tests use lightweight mock scanners (no Rust needed)
- Server-dependent tests use pytest.importorskip for starlette/fastapi
- Fix import ordering in test files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 21:46:53 -07:00
Jon Saad-FalconandClaude Opus 4.6 57ecd5c77f feat: security profiles — personal, shared, server presets with user overrides
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 21:30:27 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 6aca5eb559 feat: add Content-Security-Policy header to API responses
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:27:38 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 60317ffcba feat: get_tool_credential — scoped credential access without env pollution
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:26:42 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 687ad30f9e feat: SanitizingFormatter — auto-redact credentials in all log output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:24:57 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 9909dde780 feat: secure_mkdir and secure_create helpers for restrictive file permissions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 20:57:25 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 472938cc2b fix: fail-closed webhook validation — reject when SDK missing or secret unconfigured
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 20:56:24 -07:00
Jon Saad-FalconandClaude Opus 4.6 d2dbd4c2a2 feat: wire BoundaryGuard into ToolExecutor for external tool scanning
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 20:28:33 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 677ece6fb3 feat: tag engines with is_cloud and tools with is_local for boundary scanning
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 20:26:26 -07:00
Jon Saad-FalconandClaude Opus 4.6 7c1f983891 feat: BoundaryGuard module — scan/redact/block at device exit points
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 20:24:07 -07:00
Jon Saad-FalconandClaude Opus 4.6 63a32d20d6 feat: secure server defaults — loopback binding, redact mode, rate limiting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 20:00:49 -07:00
krypticmouseandClaude Opus 4.6 76aab9c269 fix: guard fastapi import in test_sendblue_webhook.py with importorskip
CI installs `--extra dev` but not `--extra server`, so fastapi is
unavailable. All other server tests use pytest.importorskip("fastapi")
to skip gracefully — this file was missing the guard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 23:49:12 +00:00
krypticmouseandClaude Opus 4.6 caf0e59f16 fix: remove unused imports flagged by ruff across 5 files
Pre-existing lint errors that caused CI failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 23:39:35 +00:00
Jon Saad-FalconandClaude Opus 4.6 1252531551 test: add 26 tests for SendBlue channel and webhook
tests/channels/test_sendblue.py (18 tests):
- Init from params and env vars, no-credentials error state
- Connect/disconnect lifecycle
- Send: success, from_number inclusion, API error, network error,
  no-credentials, event emission
- Webhook handler: incoming triggers handlers, outbound ignored,
  empty content ignored, event emission, handler crash safety
- Properties: from_number, list_channels

tests/server/test_sendblue_webhook.py (8 tests):
- Webhook: incoming 200, outbound ignored, empty ignored, missing
  from_number ignored, secret validation (reject + accept), no bridge
- Health: ready=true when wired, ready=false when not

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 14:42:43 -07:00
Jon Saad-FalconandClaude Opus 4.6 1add224599 test: add 59 tests for system prompt, Slack messaging, and connector health
- test_deep_research_prompt.py (8 tests): date injection, time, day of week,
  dynamic generation, response types, tool descriptions, /no_think, Jarvis name
- test_slack_messaging.py (7 tests): connect with/without tokens, error state,
  disconnect, handler registration, send without token
- test_connector_health.py (44 tests): all 11 connectors instantiate, have
  required methods, report not-connected without creds, have metadata.
  Plus KnowledgeStore data presence checks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 14:35:58 -07:00
Jon Saad-FalconandClaude Opus 4.6 62b98d903b feat: add proper Google OAuth flow with localhost callback server
Add exchange_google_token() and run_oauth_flow() to oauth.py for the
full authorization code exchange. Update gdrive, gcalendar, and
gcontacts connectors to trigger the browser-based OAuth flow when a
client_id:client_secret pair is provided, prefer access_token over raw
token, and require an actual access_token for is_connected(). auth_url()
now returns the Cloud Console credentials page when no client_id is
stored.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 16:45:03 -07:00
Jon Saad-FalconandClaude Opus 4.6 258feff64e fix: skip torch-dependent tests when torch not installed in CI
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 14:01:30 -07:00
Jon Saad-FalconandClaude Opus 4.6 db64abec23 fix: add schedule_value to template, skip server tests when fastapi missing
- personal_deep_research.toml: add schedule_value="" for template test compat
- test_deep_research_tools_wiring: skip when fastapi not installed in CI

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 13:42:43 -07:00
Jon Saad-FalconandClaude Opus 4.6 869d9256fd fix: resolve all lint errors after merge with main
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 13:33:42 -07:00
Jon Saad-FalconandClaude Opus 4.6 bce8a2fe12 merge: resolve conflicts with main — keep both DeepResearch tools + MCP functions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 13:09:12 -07:00
Jon Saad-FalconandGitHub b55b93a9c5 feat: simplify agent wizard, wire tools, add smart defaults (#149)
feat: simplify agent wizard, wire tools, add smart defaults
2026-03-27 12:54:30 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 37ad345b2d feat: add iMessage AppleScript daemon for iPhone-to-agent messaging
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 12:53:59 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 f75bd469cc feat: wire ChannelBridge to route messages to DeepResearchAgent
Add deep_research_agent parameter to ChannelBridge.__init__ and update
_handle_chat to try DeepResearchAgent first, falling back to system.ask()
when no research agent is configured.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 12:53:25 -07:00
Jon Saad-FalconandClaude Opus 4.6 17de84b9e6 feat: merge channel gateway from PR #78 — ChannelBridge, webhooks, sessions, auth
Cherry-picks the mobile channel gateway infrastructure:
- ChannelBridge orchestrator for multi-channel routing
- Webhook endpoints for Twilio SMS, BlueBubbles (iMessage), WhatsApp
- SessionStore for per-sender conversation tracking
- API key authentication middleware
- TwilioSMSChannel adapter
- CLI commands: jarvis auth, jarvis tunnel

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 12:49:48 -07:00
Jon Saad-Falcon 5a37e9c774 fix: remove unused TestClient import in recommended model tests 2026-03-27 12:48:31 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 add0abde41 feat: wire DeepResearch tools in managed agent streaming endpoint
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 12:40:16 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 e4466146ce feat: add /v1/recommended-model endpoint
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 12:27:24 -07:00
Jon Saad-FalconandClaude Opus 4.6 7366681f62 feat: wire tools from agent config to executor via ToolRegistry
Resolve tool names from config["tools"] into actual tool instances
using ToolRegistry, inject runtime deps, and pass them to the agent
constructor instead of hardcoded tools=[].

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 11:32:57 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 ccea853e5e feat: expand system_prompt_template with instruction on agent creation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 11:28:18 -07:00
b39dbedcc4 feat: fix external MCP server integration and add streaming tool-call support
Rebased and cleaned-up version of PR #113 by @mricharz, resolved against
current main (including Codex engine, Gemini thought_signature, and
agent manager fixes merged since the original PR).

MCP Transport & Client:
- StreamableHTTPTransport with session tracking, SSE parsing, timeouts
- MCPClient.initialize() sends proper MCP handshake (protocolVersion,
  capabilities, clientInfo) + notifications/initialized
- Fix StdioTransport constructor: command=[command] + args
- MCPRequest.to_dict() with notification support (id=None)

External MCP Discovery:
- _discover_external_mcp supports both url (HTTP) and command (stdio)
- Per-server include_tools / exclude_tools filtering
- MCP clients persisted on JarvisSystem for runtime lifetime

Streaming Tool-Call Support (stream_full):
- StreamChunk dataclass in _stubs.py (content, tool_calls, finish_reason, usage)
- Default stream_full() on InferenceEngine ABC wraps stream() for backward compat
- _OpenAICompatibleEngine.stream_full() with SSE parsing
- CloudEngine: _stream_full_openai (OpenAI/OpenRouter/MiniMax/Codex routing)
               _stream_full_anthropic (event-based → OpenAI delta format)
- InstrumentedEngine, MultiEngine: stream_full delegation
- GuardrailsEngine: stream_full with post-hoc security scanning (FIXED:
  original PR bypassed output scanning — now accumulates and scans like stream())

Other improvements:
- _prepare_anthropic_messages() extracted to eliminate duplication
- Default tool_choice=auto when tools are provided (OpenAI compat engines)
- MCP tool injection into managed agent streaming path
- Documentation: docs/user-guide/mcp-external-servers.md

Tests: ~59 new tests across 8 test files, all passing.

Closes PR #113

Co-Authored-By: mricharz <mricharz@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 18:09:54 +00:00
krypticmouseandClaude Opus 4.6 63ae942116 feat: add Codex cloud engine for ChatGPT Plus/Pro subscribers
Adds `codex/` prefixed model support using the OpenAI Responses API —
the same protocol used by zeroclaw and other Codex-compatible tools.

Live-tested against gpt-5-mini-2025-08-07 with:
- Generate (non-streaming): confirmed working
- System prompt → instructions mapping: confirmed working
- SSE streaming: confirmed working (9 chunks)
- End-to-end via `jarvis ask`: confirmed working

Implementation:
- Default endpoint: api.openai.com/v1/responses (standard API key)
- Override via OPENAI_CODEX_BASE_URL for ChatGPT OAuth tokens
  (e.g. chatgpt.com/backend-api/codex)
- Auth via OPENAI_CODEX_API_KEY env var
- Responses API format: input array, instructions field, output_text extraction
- Handles reasoning+message output blocks correctly
- SSE streaming parses response.output_text.delta events

Models: codex/gpt-4o, codex/gpt-4o-mini, codex/o3-mini,
        codex/gpt-5-mini, codex/gpt-5-mini-2025-08-07

Usage:
  export OPENAI_CODEX_API_KEY="your-api-key-or-oauth-token"
  jarvis ask "Hello" --model codex/gpt-5-mini-2025-08-07

Closes #134

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 01:54:36 +00:00
Jon Saad-FalconandClaude Sonnet 4.6 a1626960c1 feat: rewrite DeepResearch system prompt, wire SQL + scan + think tools, increase max_turns to 8
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 18:32:25 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 02862614a4 feat: add scan_chunks tool for LM-powered semantic grep
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 18:30:23 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 5ea5df93d4 feat: add knowledge_sql tool for read-only SQL aggregation queries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 18:29:22 -07:00
4d05475ec4 feat: enhance security scan with DNS check, JSON output, Rich UI, and API endpoint
Incorporates the useful new features from PR #135 (by @gridworks) on top
of the existing PrivacyScanner implementation:

- Add DNS configuration check (macOS, via scutil --dns)
- Add --json flag to `jarvis scan` for machine-readable output
- Add --no-scan flag to `jarvis init` to skip the post-init audit
- Expand remote-access process list (ngrok, tailscaled, cloudflared, ZeroTier)
- Upgrade `jarvis scan` output from plain text to Rich table
- Add GET /v1/security/scan API endpoint
- Add tests for all new features (30 tests, all passing)

Closes #133

Co-Authored-By: gridworks <5502067+gridworks@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 01:00:38 +00:00
Jon Saad-FalconandGitHub 7f336d7679 Merge pull request #130 from jbergant/fix/qwen35-model-catalog-mlx-129
fix: correct Qwen3.5 model sizes and MLX repos in catalog
2026-03-26 17:50:44 -07:00
Jon Saad-FalconandClaude Opus 4.6 e788d1a416 feat: extend deep-research-setup with token source detection and interactive connect
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 17:16:53 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 0f03059eec feat: add Outlook connector as IMAP subclass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 17:16:46 -07:00
Jon Saad-FalconandClaude Opus 4.6 326e6f9779 fix: expose state_db param in ingest_sources, isolate test from global state
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 16:32:43 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 bebb604a23 feat: add jarvis deep-research-setup CLI command
Auto-detects local sources (Apple Notes, iMessage, Obsidian), ingests
them into ~/.openjarvis/knowledge.db via IngestionPipeline + SyncEngine,
and launches an interactive Deep Research chat session with Qwen3.5 via
Ollama.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 16:27:26 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 744da567db fix: Apple Notes protobuf extraction handles HTML too, fix ZTITLE1 column
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 16:15:58 -07:00
krypticmouseandClaude Sonnet 4.6 bfa5132d1d feat: add SyncScheduler for periodic incremental sync
Implements SyncScheduler that runs a daemon background thread to call
SyncEngine.sync() on all registered connectors at a configurable interval.
Provides run_once() as a synchronous helper for testing.  Disconnected
connectors are skipped each cycle; errors are logged without stopping the
loop.  4 tests covering run_once, disconnected skip, start/stop, and
per-connector chunk counts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 21:29:50 +00:00
krypticmouseandClaude Sonnet 4.6 526cd4d3fb feat: add WhatsApp chat export connector
Implements WhatsAppConnector that reads exported .txt chat files from a
directory, parses the standard WhatsApp export line format with regex,
and yields one Document per chat file with participants, timestamps, and
since filtering.  Registers under "whatsapp" in ConnectorRegistry with
MCP tools for whatsapp_search_messages and whatsapp_get_chat.  10 tests
covering parsing, multi-file, since filtering, is_connected, and registry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 21:29:44 +00:00
krypticmouseandClaude Sonnet 4.6 b776c40507 feat: add Dropbox connector with file listing and download
Implements DropboxConnector using the Dropbox API v2 (list_folder +
download endpoints). Module-level API functions (_dropbox_api_list_folder,
_dropbox_api_download) are kept mockable. Downloads text-extractable
file types (.txt, .md, .csv, .py, etc.) and stores binary/unknown files
as metadata-only Documents.
Includes 6 tests covering not_connected, auth_url, sync (mocked),
disconnect, mcp_tools (dropbox_search_files / dropbox_get_file /
dropbox_list_recent), and registry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 21:23:45 +00:00
krypticmouseandClaude Sonnet 4.6 b4a1e824d8 feat: add Apple Notes connector with gzipped HTML extraction
Reads from the macOS NoteStore.sqlite database using read-only SQLite.
Decompresses ZDATA blobs (gzip + HTML) to yield plain-text Documents.
Includes 9 tests covering sync, decompression, doc_type, disconnect,
mcp_tools (notes_search / notes_get_note), and registry.
Updates connectors/__init__.py auto-import block.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 21:23:38 +00:00
krypticmouseandClaude Sonnet 4.6 dc4c773316 feat: wire Gmail since param to after: search operator
Remove the ARG002 noqa suppression and translate the `since` datetime
parameter into a Gmail `after:<epoch>` query string passed to
`_gmail_api_list_messages`, so incremental syncs only fetch messages
newer than the requested cutoff.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 21:18:52 +00:00
krypticmouseandClaude Sonnet 4.6 ee1805ceaf feat: wire attachment text extraction into IngestionPipeline
Add optional `attachment_store` parameter to IngestionPipeline. When
provided, attachments are stored as blobs in AttachmentStore and their
text (plain/markdown/csv via decode, PDF via pdfplumber) is chunked and
indexed in the KnowledgeStore with attachment provenance metadata.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-26 21:18:47 +00:00