CI's lint job ran ruff check but never ruff format --check, letting format drift land silently (79 files had drifted from the pinned ruff 0.15.1). Add the ruff format --check step to ci.yml, reformat the 79 drifted files with the pinned ruff (mechanical only — verified AST-identical to before across all files, no logic changes), and add a Makefile whose test target mirrors the actual CI lane.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#388. Parse Google Calendar all-day events from start.date instead of stamping them with the current time; treat generic next/upcoming calendar-event queries as gcalendar timeline requests that return nearest-future events first (UTC-normalized, instant-aware comparison that handles tz offsets and all-day events); and update the research planner guidance to route such queries with sources=[gcalendar] + a today-onward time_range. Real in-memory KnowledgeStore integration tests cover ordering, tz normalization, all-day inclusion, and source narrowing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pasting a Google Client ID / Secret never completed OAuth: Drive (and its
Google siblings) accepted the credentials, showed no error, opened no browser,
and never appeared in Data Sources. Root cause is three coupled defects, all
reproduced at the unit level against main with a FastAPI TestClient (no Google
creds, network-free):
(A/B) POST /connect routed a `client_id:client_secret` pair into the
connector's handle_callback, which spawned a daemon thread that popped a
browser and ran its own localhost:8789 callback server. That thread fails
silently in the bundled desktop context (`except Exception: pass`), so the
connector never gained an access_token; /connect returned status "pending"
and the UI's 20x2s poll timed out with no error.
Fix: in POST /connect, an OAuth `client_id:client_secret` pair now persists
the client credentials to every Google credential file and returns an
`oauth_required` directive pointing at the in-process server flow, instead of
the silent background thread. The Google connectors' handle_callback no longer
spawns the browser thread for the pair case — it only persists the creds; the
server's /oauth/start -> /oauth/callback owns the consent round-trip.
(C) The would-be-correct server flow was itself broken: under
`from __future__ import annotations` plus a `Request` import local to the
router factory, FastAPI could not resolve the stringized `request: Request`
annotation. /oauth/start returned HTTP 422 (request mis-bound as a query
param) and /oauth/callback injected None -> AttributeError on
`request.base_url`. Fix: import `Request` at module scope and make the
callback's `request` a required injected dependency.
A malformed/blank client pair now raises HTTP 400 with the provider setup URL
instead of a perpetual silent "pending" (REVIEW.md silent-failure discipline).
Frontend: DataSourcesPage now opens the server OAuth window when /connect
returns `oauth_required`, then polls until connected; connect errors surface the
backend detail; the Drive setup steps document the "Web application" OAuth
client + server-callback redirect URI the in-process flow requires.
Tests (run on the main venv, hermetic — no ~/.openjarvis pollution):
- test_oauth_flow.py: the three handle_callback tests now assert NO browser is
opened and only client creds are persisted (was: assert background flow ran).
- test_connectors_router_oauth.py (new): reproduces + fixes all three defects via
TestClient with mocked token exchange; parametrized over gdrive/gcalendar/
gcontacts/gmail/google_tasks to prove the shared OAuth path is fixed for every
sibling and that a single consent writes the access_token to all six Google
credential files and flips is_connected() to True.
Full tests/connectors suite: 355 passed.
Relationship to PR #510: #510 rewrites all of these files (account-scoped
retrieval) but still carries all three defects. This fix is intentionally scoped
to the OAuth path and does not modify oauth.py, to minimize collision. A
maintainer can either merge this and rebase #510 on top, or port these changes
into #510. See PR body for details.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: close KnowledgeStore connections in connectors_router endpoints
KnowledgeStore opens a persistent SQLite connection in __init__ but was
never closed at the three instantiation sites in connectors_router.py,
leaking one file descriptor (plus its WAL handle) per call. The worst
offender is _connector_summary(), which is invoked per-connector on
every GET /connectors poll from the frontend — after ~100 polls the
backend hits the default macOS per-process FD limit (256) and asyncio's
accept() starts failing with OSError: [Errno 24] Too many open files.
Add __enter__/__exit__ to KnowledgeStore (close() already existed) and
wrap all three instantiation sites in `with` blocks so the connection
is released deterministically when the scope exits.
- connectors/store.py: add context manager protocol
- server/connectors_router.py: use `with KnowledgeStore() as store:`
in _connector_summary, _ingest, and _run_sync
* test: cover KnowledgeStore context manager + connectors_router close leak
- test_store.py: add two tests for the new __enter__/__exit__ protocol
verifying the connection closes on normal exit and on exceptions.
- test_connectors_router.py: add a regression test that monkeypatches
KnowledgeStore.close to count invocations and asserts every store
opened by GET /v1/connectors is paired with a close.
Also fix a pre-existing bug in the test_connectors_router fixture: the
router is created with prefix="/v1/connectors" internally (line 92 of
connectors_router.py), and the fixture was wrapping it again with
prefix="/v1", producing "/v1/v1/connectors". All 6 existing router
tests were failing with 404 before this fix. 5 of them now pass; the
remaining failure (test_trigger_sync) is an unrelated KeyError on a
missing response field and is out of scope for this PR.
* test: drop connectors_router regression test — skipped in CI anyway
The regression test added in the previous commit relies on FastAPI
being importable, but CI's dev-extra install does not include fastapi
(it lives in the `server` optional group). All tests in
test_connectors_router.py silently skip on CI with "fastapi not
installed", so the regression test would never actually execute there.
Revert test_connectors_router.py to the upstream main version. The
fixture bug I fixed (double prefix) and the connection-leak regression
test both deserve a separate PR scoped to test infrastructure — that
PR should either add fastapi to dev deps, split server tests into
their own CI job, or both.
The KnowledgeStore context manager tests in test_store.py are kept
because they have no fastapi dependency and will run in CI.
* feat: add Apple Contacts connector (macOS AddressBook)
Reads directly from ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb
in read-only mode. Extracts names, phone numbers, emails, postal addresses, URLs,
social profiles, and notes for each contact. Requires Full Disk Access like
iMessage and Apple Notes connectors.
- New connector: src/openjarvis/connectors/apple_contacts.py
- Registered in connectors/__init__.py
- Added frontend catalog entry (PIM category) and icon mapping
- MCP tools: contacts_search, contacts_get_contact
* test: add comprehensive tests for Apple Contacts connector
15 tests covering: is_connected (exists/missing), sync yields all real
contacts (skips system rows), extracts all field types (phone, email,
address, URL, social, notes), cleans Apple label markup, org-only
contacts, minimal contacts, since filter, sync_status tracking,
structured metadata, disconnect, mcp_tools, registry, empty DB,
missing DB. Uses fake SQLite database — no real Contacts DB needed.
* fix: scan iCloud/Exchange source databases for all contacts
The main AddressBook database only contains locally-created contacts.
Synced contacts (iCloud, Exchange, etc.) live under
Sources/<UUID>/AddressBook-v22.abcddb. Now scans all source databases
and deduplicates by ZUNIQUEID across sources.
Adds tests for multi-source scanning and cross-source deduplication.
- test_deep_research_prompt.py (8 tests): date injection, time, day of week,
dynamic generation, response types, tool descriptions, /no_think, Jarvis name
- test_slack_messaging.py (7 tests): connect with/without tokens, error state,
disconnect, handler registration, send without token
- test_connector_health.py (44 tests): all 11 connectors instantiate, have
required methods, report not-connected without creds, have metadata.
Plus KnowledgeStore data presence checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add exchange_google_token() and run_oauth_flow() to oauth.py for the
full authorization code exchange. Update gdrive, gcalendar, and
gcontacts connectors to trigger the browser-based OAuth flow when a
client_id:client_secret pair is provided, prefer access_token over raw
token, and require an actual access_token for is_connected(). auth_url()
now returns the Cloud Console credentials page when no client_id is
stored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements SyncScheduler that runs a daemon background thread to call
SyncEngine.sync() on all registered connectors at a configurable interval.
Provides run_once() as a synchronous helper for testing. Disconnected
connectors are skipped each cycle; errors are logged without stopping the
loop. 4 tests covering run_once, disconnected skip, start/stop, and
per-connector chunk counts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements WhatsAppConnector that reads exported .txt chat files from a
directory, parses the standard WhatsApp export line format with regex,
and yields one Document per chat file with participants, timestamps, and
since filtering. Registers under "whatsapp" in ConnectorRegistry with
MCP tools for whatsapp_search_messages and whatsapp_get_chat. 10 tests
covering parsing, multi-file, since filtering, is_connected, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove the ARG002 noqa suppression and translate the `since` datetime
parameter into a Gmail `after:<epoch>` query string passed to
`_gmail_api_list_messages`, so incremental syncs only fetch messages
newer than the requested cutoff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add optional `attachment_store` parameter to IngestionPipeline. When
provided, attachments are stored as blobs in AttachmentStore and their
text (plain/markdown/csv via decode, PDF via pdfplumber) is chunked and
indexed in the KnowledgeStore with attachment provenance metadata.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add disk-persistent storage for ColBERT token-level embeddings so the
reranker can reuse pre-computed embeddings across queries instead of
re-encoding every document on every search.
- EmbeddingStore: individual .pt files per chunk with SQLite index for
O(1) lookup, graceful degradation when torch is not installed
- ColBERTReranker: checks EmbeddingStore for cached embeddings before
falling back to docFromText(), stores newly computed embeddings
- KnowledgeStore: ensures chunk_id is always present in retrieval
metadata (both at store time and as a backfill in retrieve)
- 18 tests covering round-trip persistence, deletion, torch-absent
degradation, and reranker cache-hit/miss behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements AttachmentStore that writes blobs to {base_dir}/{sha[:2]}/{sha}
with a SQLite metadata index tracking filename, MIME type, size, and the
accumulated list of source_doc_ids for each content-identical blob.
Includes 7 unit tests covering SHA-256 return, file path layout,
idempotent dedup, multi-source tracking, metadata retrieval, content
round-trip, and missing-blob None return.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parse the last_sync checkpoint timestamp into a datetime and pass it as
the `since` argument to connector.sync(), enabling subsequent syncs to
fetch only new items rather than a full re-fetch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs found via live API testing:
- List notes API returns "notes" key, not "data" (was returning 0 results)
- Transcript speaker field is a dict {"source": "microphone"}, not a string
Verified: 112 real meeting notes synced, 2,226 chunks indexed and searchable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Slack connector now skips bot_id messages and non-content subtypes
(message_changed, message_deleted, bot_message, channel_join, channel_leave).
Found by comparing against hermes-agent reference implementation.
Live smoke test indexes 4,467 chunks from real docs/ markdown files and
verifies the full pipeline: Obsidian → SyncEngine → KnowledgeStore → search.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Break long lines in test_notion.py, test_store.py, and test_sync_engine.py
to comply with the 88-character E501 limit enforced by ruff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements IMessageConnector (auth_type='local') that opens ~/Library/Messages/chat.db
read-only, converts Apple nanosecond timestamps to UTC datetimes, and yields one
Document per message with handle-based author resolution and chat display names.
Includes 8 tests using a temporary SQLite DB and 2 MCP tools (search + get conversation).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GContactsConnector using the People API v1 to paginate
contacts (names, emails, phones, orgs) and yield them as Documents
with doc_type="contact". Includes 6 unit tests and auto-registration
in the connectors __init__.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GCalendarConnector (registered as "gcalendar") following the
gmail.py pattern: module-level API functions for calendarList and events.list,
_format_event helper for human-readable content, paginated sync across all
calendars, and three MCP tools (get_events_today, search_events, next_meeting).
Includes 6 tests covering not_connected, auth_url scope, sync document fields,
disconnect, mcp_tools, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GDriveConnector using the Drive API v3 with OAuth, paginated
file listing, Google Workspace export (Docs→text/plain, Sheets→text/csv,
Slides→text/plain), and 3 MCP tools. Adds auto-registration in __init__.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements SlackConnector that syncs channel message history via the Slack
Web API, with OAuth credential storage, user-map resolution, and 3 MCP tools.
Includes 7 tests covering connectivity, auth, sync document correctness,
disconnect, MCP tools, and registry registration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GranolaConnector that syncs meeting notes from the Granola
public API, combining AI-generated summaries with full speaker transcripts
into searchable Documents. Includes cursor-based pagination, created_after
filtering, 2 MCP tools, and 8 tests covering all connector behaviours.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements NotionConnector that syncs pages via the Notion REST API,
renders block content (paragraph, headings, lists, code, quote, divider,
to_do) to markdown, and registers with ConnectorRegistry. Includes 8
tests covering auth, sync, rendering, disconnect, MCP tools, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire up ObsidianConnector and GmailConnector via auto-import in the
connectors __init__.py so they register on package import. Add a full
end-to-end integration test covering the Obsidian vault → SyncEngine →
KnowledgeStore → knowledge_search pipeline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GmailConnector registered under 'gmail' in ConnectorRegistry,
a shared oauth.py helper (build_google_auth_url, load/save/delete_tokens)
reusable by Drive/Calendar/Contacts, and 7 fully mocked pytest tests
covering auth state, sync document extraction, disconnect, mcp_tools,
and registry lookup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements ObsidianConnector (filesystem auth, no OAuth) that walks a vault
directory for .md/.markdown/.txt files, parses YAML frontmatter via a
dependency-free parser, skips hidden dirs and binary files, and yields
Document objects with doc_type="note" and obsidian:// deep-link URLs.
Exposes an obsidian_search_notes MCP ToolSpec. 9 tests cover connection
state, vault traversal, hidden-dir/binary filtering, frontmatter extraction,
and registry wiring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduces SyncEngine that wraps IngestionPipeline with a SQLite state
database (sync_state.db) for checkpoint/resume: cursors and item counts
are persisted after every 100-document batch and on completion/error.
Adds 4 tests covering single-connector ingestion, checkpoint accuracy,
unsynced-connector None return, and multi-connector source filtering.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements IngestionPipeline that deduplicates Documents by doc_id (both
in-memory and by loading existing doc_ids from the store on init), chunks
content via SemanticChunker, and persists all chunks with full provenance
metadata to KnowledgeStore. 8 tests cover single-doc ingestion, dedup across
calls and batches, persistence across pipeline instances, long-doc multi-chunk
splitting, atomic event chunking, multi-source filtering, and return-value accuracy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements SemanticChunker that splits text based on doc_type (event/contact
as single chunks, email on reply boundaries, message on double-newlines,
document/note on ## headings → paragraphs → sentences). ChunkResult carries
sequential 0-based indexes and inherits parent metadata; section headings are
added as chunk metadata. 16 tests covering all splitting strategies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements KnowledgeStore (extends MemoryBackend) for Deep Research with a
rich SQLite/FTS5 schema supporting source, doc_type, author, participants,
timestamp, thread_id, url, and chunk_index columns; BM25 ranking via FTS5
with porter unicode61 tokenizer; filtered retrieval by source, doc_type,
author, since, and until; MEMORY_STORE/MEMORY_RETRIEVE event emission; WAL
journal mode; and 15 isolated tests using tmp_path fixtures.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
