Commit Graph
449 Commits
Author SHA1 Message Date
dc2a2edbcd fix: data source connect flow UX, obsidian/gcalendar sync bugs, agent timeout
- Frontend: add progress stages (Connecting → Authenticating → Connected →
  Syncing) with spinner and progress bar to the data source connect flow.
  Previously sources would silently stay "Not connected" after setup.
  Show error message on failure instead of swallowing exceptions.

- Obsidian connector: use timezone-aware datetime (tz=timezone.utc) in
  fromtimestamp() to fix "can't compare offset-naive and offset-aware
  datetimes" crash during incremental sync.

- Google Calendar connector: catch HTTPStatusError when listing events
  for individual calendars (e.g. US Holidays returning 404) so one
  inaccessible calendar doesn't crash the entire sync.

- Agent SSE timeout: increase progress queue timeout from 120s to 600s
  so complex multi-hop deep research queries aren't killed mid-execution
  on slower local models.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 21:24:36 -07:00
c4f85d435f fix(evals): add tool_choice=auto + fix traces thread safety (#163)
Two fixes for eval accuracy and stability:

1. TauBench agent: add tool_choice="auto" to match tau2's native
   LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
   below leaderboard because the models don't receive explicit
   tool-calling guidance.

2. SystemBuilder: apply self._traces flag to config.traces.enabled.
   Previously builder.traces(False) was a no-op — traces stayed
   enabled, creating SQLite connections in the main thread that
   crashed when accessed from ThreadPoolExecutor worker threads
   in GAIA evals ("SQLite objects created in a thread can only be
   used in that same thread").

Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 21:24:36 -07:00
450397a2ad feat(evals): TauBench V2 native integration + GAIA eval configs (#162)
* feat(evals): add TauBench V2 native integration + GAIA eval configs

Integrate TauBench V2 (τ²-bench) multi-turn customer service benchmark
natively through OpenJarvis's inference engine. The agent's LLM calls go
through OpenJarvis while tau2-bench handles the orchestration, user
simulation, domain tools, database, and evaluation.

## TauBench Integration

Architecture: JarvisHalfDuplexAgent bridges OpenJarvis's engine into
tau2's Orchestrator as a drop-in agent replacement. This enables
testing how well OpenJarvis's Intelligence + Engine handles multi-turn
customer service tasks with real tool calling and database mutations.

Key features:
- Native OpenJarvis engine for agent LLM calls
- tau2's UserSimulator for realistic customer interactions
- Domain tools (airline, retail, telecom) with mock databases
- Full evaluation: DB state checks, action matching, NL assertions
- Test-split filtering for leaderboard-comparable results
- Pass^k multi-trial support (default 3 trials per task)
- Qwen thinking-mode disabled for clean tool call parsing
- Gemini thought_signature handling for multi-turn conversations

Files:
- datasets/taubench.py: Dataset provider with test-split filtering
- execution/taubench_env.py: JarvisHalfDuplexAgent + simulation runner
- scorers/taubench.py: Scorer reading tau2 evaluation rewards
- CLI registration and KNOWN_BENCHMARKS update

## Results (test split, pass^3, 60 tasks)

| Model              | TauBench | Leaderboard |
|--------------------|----------|-------------|
| Claude Opus 4.6    | 86.67%   | 84.8%       |
| Nemotron-3-Super   | 86.67%   | —           |
| Qwen3.5-397B       | 81.67%   | 95.6%       |
| GPT-5.4            | 81.67%   | 91.5%       |
| Qwen3.5-122B       | 80.00%   | 93.6%       |
| Qwen3.5-35B        | 77.27%   | 89.2%       |
| Gemini 3.1 Pro     | 58.33%   | ~87%        |

## GAIA Eval Configs

Added configs for GPT-5.4, Gemini 3.1 Pro, Nemotron, Qwen 122B,
Qwen 35B, and existing GAIA rerun configs for multiple models.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: lint errors in taubench integration

Remove unused imports and sort import blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: E501 line too long in slack_connector.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: remove unrelated GAIA configs from PR

Keep only TauBench configs that were created and tested in this PR.
GAIA configs are pre-existing or belong in a separate PR.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 21:24:36 -07:00
mrTSB 59dcbb9cc4 make cloud models on desktop app work 2026-04-01 21:22:39 -07:00
mrTSB f8c37fe85e make cloud models on desktop app work 2026-04-01 21:22:25 -07:00
Jon Saad-FalconandClaude Opus 4.6 90ef8338d6 fix: use Z suffix for Granola API created_after date format
Granola API rejects +00:00 timezone offset and bare ISO timestamps.
Use strftime with explicit Z suffix which the API accepts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 14:35:17 -07:00
Jon Saad-FalconandClaude Opus 4.6 91205fda2b fix: remove duplicate Gmail connector, fix sync UI state tracking
- Remove gmail (REST API) from registered connectors, keep only gmail_imap
- Track background sync state at router level so status endpoint reports
  "syncing" while background thread is alive, "error" with message on failure
- Fixes sync UI bouncing back to "Sync Now" immediately after clicking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-29 14:27:31 -07:00
Jon Saad-FalconandClaude Opus 4.6 a57813fcfa feat: overhaul install + setup UX for data sources, messaging, and desktop app
- Skip SetupWizard on launch, go straight to Chat page
- Add Data Sources page with sidebar nav (separate tabs for data sources + messaging channels)
- Add "Connect your data" banner + quick-action buttons on Chat empty state
- Add hint on deep research agent pointing to Data Sources + Messaging tabs
- Consistent naming: "Data Sources" and "Messaging Channels" everywhere
- Fix Apple Notes / iMessage setup (remove broken system prefs link)
- Fix Slack data source: auto-join public channels, rate limit retry, is_member filter
- Add channels:join scope to all Slack manifests + docs
- Fix Gmail: use gmail_imap connector (IMAP + app password), increase limit to 5000
- Fix sync endpoint: run in background thread, return immediately
- Show sync progress with progress bar, error messages, Sync Now / Re-sync / Retry buttons
- Add triggerSync() API + SyncStatusDisplay component
- Rewrite all connector setup instructions with precise click-by-click steps
- Notion: share all pages at once via top-level page sharing
- Obsidian: show how to find vault path via Obsidian UI or Finder
- SendBlue: add link to API Credentials page, add ngrok webhook step
- Slack messaging: add Copy button for JSON manifest
- Add OpenJarvis Slack icon asset
- Shorten deep research template description
- Fix desktop app port (8222 → 8000 to match server default)
- Disable auto-updater for local dev builds
- Register Slack, Outlook, GCalendar connectors in __init__.py
- Auto-create default agent when setting up messaging channels

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 22:44:59 -07:00
Jon Saad-FalconandGitHub 09d1cc3681 Merge pull request #152 from open-jarvis/feat/security-hardening
feat: security hardening — layered boundary enforcement
2026-03-28 22:15:22 -07:00
Jon Saad-FalconandGitHub b06e1ebfe9 Merge pull request #151 from open-jarvis/fix/frontend-node-engine-requirement
fix: add Node >= 20 engine requirement to frontend package.json
2026-03-28 22:15:08 -07:00
Jon Saad-FalconandClaude Opus 4.6 096b718417 chore: remove spec and plan docs from PR
These are internal planning artifacts, not part of the deliverable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 22:08:07 -07:00
Jon Saad-FalconandClaude Opus 4.6 0cbd0e1dac fix: update existing tests for new secure defaults
test_config.py: SecurityConfig.mode default changed from "warn" to "redact"
test_config_phase3.py: ServerConfig.host default changed from "0.0.0.0" to "127.0.0.1"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 22:02:02 -07:00
Jon Saad-FalconandClaude Opus 4.6 f122f5a699 fix: tests work without Rust extension and server extras
- BoundaryGuard degrades gracefully when Rust scanners unavailable
- BoundaryGuard tests use lightweight mock scanners (no Rust needed)
- Server-dependent tests use pytest.importorskip for starlette/fastapi
- Fix import ordering in test files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 21:46:53 -07:00
Jon Saad-FalconandClaude Opus 4.6 8e75fcb232 feat: non-loopback auth enforcement and CORS origin passthrough
Add check_bind_safety() to auth_middleware — refuses to bind non-loopback
without API key. Add cors_origins parameter to create_app() so CORS uses
config values instead of hardcoded wildcard "*".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 21:46:45 -07:00
Jon Saad-FalconandClaude Opus 4.6 52f6aaf5bb fix: lint fixes for security hardening — line length and import order
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 21:32:20 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 0193e07d83 feat: jarvis doctor checks for security profile configuration
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:31:09 -07:00
Jon Saad-FalconandClaude Opus 4.6 57ecd5c77f feat: security profiles — personal, shared, server presets with user overrides
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 21:30:27 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 6aca5eb559 feat: add Content-Security-Policy header to API responses
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:27:38 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 79a46df747 feat: startup credential audit log and CORS wildcard warning
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:27:30 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 60317ffcba feat: get_tool_credential — scoped credential access without env pollution
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:26:42 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 687ad30f9e feat: SanitizingFormatter — auto-redact credentials in all log output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:24:57 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 85fbd2bd8c fix: enforce 0o600/0o700 permissions on all database and data files
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 21:20:01 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 9909dde780 feat: secure_mkdir and secure_create helpers for restrictive file permissions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 20:57:25 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 472938cc2b fix: fail-closed webhook validation — reject when SDK missing or secret unconfigured
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 20:56:24 -07:00
Jon Saad-FalconandClaude Opus 4.6 d2dbd4c2a2 feat: wire BoundaryGuard into ToolExecutor for external tool scanning
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 20:28:33 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 677ece6fb3 feat: tag engines with is_cloud and tools with is_local for boundary scanning
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-28 20:26:26 -07:00
Jon Saad-FalconandClaude Opus 4.6 7c1f983891 feat: BoundaryGuard module — scan/redact/block at device exit points
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 20:24:07 -07:00
Jon Saad-FalconandClaude Opus 4.6 63a32d20d6 feat: secure server defaults — loopback binding, redact mode, rate limiting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 20:00:49 -07:00
Jon Saad-FalconandClaude Opus 4.6 44c52edfcd docs: security hardening implementation plan — 15 tasks with TDD
Detailed plan covering network exposure, boundary guard, webhook
validation, file permissions, log sanitization, credential scoping,
CORS hardening, and security profiles. Each task has failing tests
first, then implementation, then verification.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 19:26:59 -07:00
ANarayanandClaude Opus 4.6 bbcda62ed4 fix: add Node >= 20 engine requirement to frontend package.json
Tailwind CSS v4.2 (@tailwindcss/oxide) requires Node >= 20 for native
bindings. Without this, `npm install` succeeds but `vite` fails at
runtime with a missing native binding error.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-28 18:51:37 -07:00
Jon Saad-FalconandClaude Opus 4.6 e842976d96 docs: security hardening design spec — layered boundary enforcement
Approved design for comprehensive security hardening covering network
exposure defaults, boundary guard scanning, webhook fail-closed validation,
file permissions, credential handling, log sanitization, and security profiles.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 18:29:51 -07:00
Jon Saad-Falcon 9ab2d37c82 fix: shorten Personal Deep Research template description to match others 2026-03-28 17:41:05 -07:00
Jon Saad-FalconandClaude Opus 4.6 0e9d7005cb docs: update roadmap + create cross-platform testing plan
Roadmap: updated messaging status (iMessage/SMS + Slack working,
WhatsApp Baileys blocked, Meta Cloud API planned).

Testing plan: 12 query types across 3 platforms (Interact, Slack,
iMessage/SMS) with execution phases and success criteria.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 17:34:30 -07:00
Jon Saad-Falcon 4ab6139d52 Merge branch 'main' of https://github.com/open-jarvis/OpenJarvis 2026-03-28 17:22:36 -07:00
Jon Saad-FalconandClaude Opus 4.6 290746b3e0 docs: restructure channels guide — Messaging Channels + Data Connectors
Added SendBlue (iMessage/SMS) and Slack messaging setup with detailed
step-by-step instructions, App Manifest JSON, webhook registration,
and troubleshooting tables. Reorganized doc into two sections:
Messaging Channels (talk to agent) and Data Connectors (search data).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 17:21:43 -07:00
Jon Saad-Falcon fa87bbb2d6 fix: remove whatsapp bridge node_modules from git, add to gitignore 2026-03-28 17:18:13 -07:00
Jon Saad-FalconandClaude Opus 4.6 bc31cce732 feat: simplify Messaging tab to iMessage/SMS (SendBlue) + Slack only
Removed WhatsApp (Baileys blocked by WhatsApp servers) and Twilio SMS
(SendBlue handles SMS as fallback automatically). Updated Slack setup
instructions with the App Manifest JSON for one-step configuration.

Three channels: iMessage + SMS (SendBlue), Slack (Socket Mode)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 17:17:51 -07:00
Jon Saad-FalconandGitHub 7a03bf5794 Merge pull request #150 from open-jarvis/fix/windows-git-detection
fix: Windows git detection in desktop app
2026-03-28 16:56:33 -07:00
krypticmouseandClaude Opus 4.6 76aab9c269 fix: guard fastapi import in test_sendblue_webhook.py with importorskip
CI installs `--extra dev` but not `--extra server`, so fastapi is
unavailable. All other server tests use pytest.importorskip("fastapi")
to skip gracefully — this file was missing the guard.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 23:49:12 +00:00
krypticmouseandClaude Opus 4.6 eeb66a1606 fix: remaining unused import lint errors in slack_daemon and agent_manager_routes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 23:39:51 +00:00
krypticmouseandClaude Opus 4.6 caf0e59f16 fix: remove unused imports flagged by ruff across 5 files
Pre-existing lint errors that caused CI failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 23:39:35 +00:00
krypticmouseandClaude Opus 4.6 e3039c0d0a fix: Windows git detection in desktop app resolve_bin()
The Tauri desktop app's resolve_bin("git") function checked incorrect
paths for Git on Windows (e.g. {ProgramFiles}\git\git.exe) and had no
PATH fallback, causing "git not found" even when Git is installed.

Fixes:
- Add correct Git for Windows paths: {ProgramFiles}\Git\cmd\git.exe,
  {ProgramFiles(x86)}\Git\cmd\git.exe, {LOCALAPPDATA}\Programs\Git\cmd\
- Add Scoop package manager path: {home}\scoop\shims\git.exe
- Add PATH fallback using `where.exe` on Windows and `which` on Unix,
  so any binary on PATH is found even if not at a hardcoded location

Closes #128

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 23:39:35 +00:00
Jon Saad-FalconandClaude Opus 4.6 eea5f65ea0 feat: wire Twilio SMS handler with agent + ack + response sending
The Twilio webhook now:
1. Sends "Message received! Working on it now..." via Twilio API
2. Routes to DeepResearchAgent (via bridge or direct fallback)
3. Sends the research response back as SMS
4. Handles missing bridge gracefully
5. Formats response for SMS (strips markdown, 1500 char limit)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 16:27:32 -07:00
Jon Saad-FalconandClaude Opus 4.6 6233ae1aa1 feat: Slack daemon as subprocess for reliable Socket Mode
Replaces the flaky inline bolt_app creation with a proper subprocess
daemon (slack_daemon.py). The daemon persists independently of the
HTTP request lifecycle, solving the Socket Mode connection dropping.

Features: thread replies, progress updates, message queue, Slack
formatting, PID file for lifecycle management.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 15:57:49 -07:00
Jon Saad-FalconandClaude Opus 4.6 9a6b1c1623 feat: reply in thread for Slack messages to reduce DM clutter
All bot replies (acknowledgment, progress updates, final response)
now use thread_ts to reply in the thread of the user's message.
Keeps the DM conversation clean — one thread per question.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 15:12:36 -07:00
Jon Saad-FalconandClaude Opus 4.6 590e8b90a3 feat: convert markdown to Slack formatting instead of stripping it
Converts: ## Header → *Header* (bold), **bold** → *bold*,
*italic* → _italic_, ~~strike~~ → ~strike~, [text](url) → <url|text>.
Removes LaTeX and HTML. System prompt updated to use simple markdown
since it gets auto-converted per platform.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 15:06:57 -07:00
Jon Saad-FalconandClaude Opus 4.6 d3c7c59c8d fix: improve SendBlue UX — queue tracking, periodic reminders, SMS formatting
1. Ack message: "Message received! Working on it now..."
2. Queue awareness: if another message arrives while processing,
   "Message received! Message X in queue, will respond ASAP"
3. Periodic reminders every 60s: "Still working! Will reply ASAP"
4. Response formatting: _format_for_sms() strips markdown headers,
   bold/italic, code blocks, LaTeX, image syntax, collapses blanks
5. System prompt: instructs agent to use plain text, short paragraphs,
   simple dashes — no markdown/LaTeX that breaks in iMessage/SMS

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 15:06:01 -07:00
Jon Saad-FalconandClaude Opus 4.6 2ef6615184 feat: improve Slack messaging UX — progress updates, queue, clean formatting
1. Initial: "Message received! Working on it now..."
2. Every 60s while processing: "Still working! Will reply ASAP"
3. New message while busy: "Message received! Message X in queue of Y"
4. Responses stripped of markdown (headers, bold, code blocks) for
   clean Slack display
5. System prompt updated to prefer plain text over complex markdown

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 14:59:03 -07:00
Jon Saad-FalconandClaude Opus 4.6 1252531551 test: add 26 tests for SendBlue channel and webhook
tests/channels/test_sendblue.py (18 tests):
- Init from params and env vars, no-credentials error state
- Connect/disconnect lifecycle
- Send: success, from_number inclusion, API error, network error,
  no-credentials, event emission
- Webhook handler: incoming triggers handlers, outbound ignored,
  empty content ignored, event emission, handler crash safety
- Properties: from_number, list_channels

tests/server/test_sendblue_webhook.py (8 tests):
- Webhook: incoming 200, outbound ignored, empty ignored, missing
  from_number ignored, secret validation (reject + accept), no bridge
- Health: ready=true when wired, ready=false when not

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 14:42:43 -07:00
Jon Saad-FalconandClaude Opus 4.6 1add224599 test: add 59 tests for system prompt, Slack messaging, and connector health
- test_deep_research_prompt.py (8 tests): date injection, time, day of week,
  dynamic generation, response types, tool descriptions, /no_think, Jarvis name
- test_slack_messaging.py (7 tests): connect with/without tokens, error state,
  disconnect, handler registration, send without token
- test_connector_health.py (44 tests): all 11 connectors instantiate, have
  required methods, report not-connected without creds, have metadata.
  Plus KnowledgeStore data presence checks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 14:35:58 -07:00