* fix: close KnowledgeStore connections in connectors_router endpoints
KnowledgeStore opens a persistent SQLite connection in __init__ but was
never closed at the three instantiation sites in connectors_router.py,
leaking one file descriptor (plus its WAL handle) per call. The worst
offender is _connector_summary(), which is invoked per-connector on
every GET /connectors poll from the frontend — after ~100 polls the
backend hits the default macOS per-process FD limit (256) and asyncio's
accept() starts failing with OSError: [Errno 24] Too many open files.
Add __enter__/__exit__ to KnowledgeStore (close() already existed) and
wrap all three instantiation sites in `with` blocks so the connection
is released deterministically when the scope exits.
- connectors/store.py: add context manager protocol
- server/connectors_router.py: use `with KnowledgeStore() as store:`
in _connector_summary, _ingest, and _run_sync
* test: cover KnowledgeStore context manager + connectors_router close leak
- test_store.py: add two tests for the new __enter__/__exit__ protocol
verifying the connection closes on normal exit and on exceptions.
- test_connectors_router.py: add a regression test that monkeypatches
KnowledgeStore.close to count invocations and asserts every store
opened by GET /v1/connectors is paired with a close.
Also fix a pre-existing bug in the test_connectors_router fixture: the
router is created with prefix="/v1/connectors" internally (line 92 of
connectors_router.py), and the fixture was wrapping it again with
prefix="/v1", producing "/v1/v1/connectors". All 6 existing router
tests were failing with 404 before this fix. 5 of them now pass; the
remaining failure (test_trigger_sync) is an unrelated KeyError on a
missing response field and is out of scope for this PR.
* test: drop connectors_router regression test — skipped in CI anyway
The regression test added in the previous commit relies on FastAPI
being importable, but CI's dev-extra install does not include fastapi
(it lives in the `server` optional group). All tests in
test_connectors_router.py silently skip on CI with "fastapi not
installed", so the regression test would never actually execute there.
Revert test_connectors_router.py to the upstream main version. The
fixture bug I fixed (double prefix) and the connection-leak regression
test both deserve a separate PR scoped to test infrastructure — that
PR should either add fastapi to dev deps, split server tests into
their own CI job, or both.
The KnowledgeStore context manager tests in test_store.py are kept
because they have no fastapi dependency and will run in CI.
* feat: add Apple Contacts connector (macOS AddressBook)
Reads directly from ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb
in read-only mode. Extracts names, phone numbers, emails, postal addresses, URLs,
social profiles, and notes for each contact. Requires Full Disk Access like
iMessage and Apple Notes connectors.
- New connector: src/openjarvis/connectors/apple_contacts.py
- Registered in connectors/__init__.py
- Added frontend catalog entry (PIM category) and icon mapping
- MCP tools: contacts_search, contacts_get_contact
* test: add comprehensive tests for Apple Contacts connector
15 tests covering: is_connected (exists/missing), sync yields all real
contacts (skips system rows), extracts all field types (phone, email,
address, URL, social, notes), cleans Apple label markup, org-only
contacts, minimal contacts, since filter, sync_status tracking,
structured metadata, disconnect, mcp_tools, registry, empty DB,
missing DB. Uses fake SQLite database — no real Contacts DB needed.
* fix: scan iCloud/Exchange source databases for all contacts
The main AddressBook database only contains locally-created contacts.
Synced contacts (iCloud, Exchange, etc.) live under
Sources/<UUID>/AddressBook-v22.abcddb. Now scans all source databases
and deduplicates by ZUNIQUEID across sources.
Adds tests for multi-source scanning and cross-source deduplication.
- test_deep_research_prompt.py (8 tests): date injection, time, day of week,
dynamic generation, response types, tool descriptions, /no_think, Jarvis name
- test_slack_messaging.py (7 tests): connect with/without tokens, error state,
disconnect, handler registration, send without token
- test_connector_health.py (44 tests): all 11 connectors instantiate, have
required methods, report not-connected without creds, have metadata.
Plus KnowledgeStore data presence checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add exchange_google_token() and run_oauth_flow() to oauth.py for the
full authorization code exchange. Update gdrive, gcalendar, and
gcontacts connectors to trigger the browser-based OAuth flow when a
client_id:client_secret pair is provided, prefer access_token over raw
token, and require an actual access_token for is_connected(). auth_url()
now returns the Cloud Console credentials page when no client_id is
stored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements SyncScheduler that runs a daemon background thread to call
SyncEngine.sync() on all registered connectors at a configurable interval.
Provides run_once() as a synchronous helper for testing. Disconnected
connectors are skipped each cycle; errors are logged without stopping the
loop. 4 tests covering run_once, disconnected skip, start/stop, and
per-connector chunk counts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements WhatsAppConnector that reads exported .txt chat files from a
directory, parses the standard WhatsApp export line format with regex,
and yields one Document per chat file with participants, timestamps, and
since filtering. Registers under "whatsapp" in ConnectorRegistry with
MCP tools for whatsapp_search_messages and whatsapp_get_chat. 10 tests
covering parsing, multi-file, since filtering, is_connected, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove the ARG002 noqa suppression and translate the `since` datetime
parameter into a Gmail `after:<epoch>` query string passed to
`_gmail_api_list_messages`, so incremental syncs only fetch messages
newer than the requested cutoff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add optional `attachment_store` parameter to IngestionPipeline. When
provided, attachments are stored as blobs in AttachmentStore and their
text (plain/markdown/csv via decode, PDF via pdfplumber) is chunked and
indexed in the KnowledgeStore with attachment provenance metadata.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add disk-persistent storage for ColBERT token-level embeddings so the
reranker can reuse pre-computed embeddings across queries instead of
re-encoding every document on every search.
- EmbeddingStore: individual .pt files per chunk with SQLite index for
O(1) lookup, graceful degradation when torch is not installed
- ColBERTReranker: checks EmbeddingStore for cached embeddings before
falling back to docFromText(), stores newly computed embeddings
- KnowledgeStore: ensures chunk_id is always present in retrieval
metadata (both at store time and as a backfill in retrieve)
- 18 tests covering round-trip persistence, deletion, torch-absent
degradation, and reranker cache-hit/miss behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements AttachmentStore that writes blobs to {base_dir}/{sha[:2]}/{sha}
with a SQLite metadata index tracking filename, MIME type, size, and the
accumulated list of source_doc_ids for each content-identical blob.
Includes 7 unit tests covering SHA-256 return, file path layout,
idempotent dedup, multi-source tracking, metadata retrieval, content
round-trip, and missing-blob None return.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parse the last_sync checkpoint timestamp into a datetime and pass it as
the `since` argument to connector.sync(), enabling subsequent syncs to
fetch only new items rather than a full re-fetch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs found via live API testing:
- List notes API returns "notes" key, not "data" (was returning 0 results)
- Transcript speaker field is a dict {"source": "microphone"}, not a string
Verified: 112 real meeting notes synced, 2,226 chunks indexed and searchable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Slack connector now skips bot_id messages and non-content subtypes
(message_changed, message_deleted, bot_message, channel_join, channel_leave).
Found by comparing against hermes-agent reference implementation.
Live smoke test indexes 4,467 chunks from real docs/ markdown files and
verifies the full pipeline: Obsidian → SyncEngine → KnowledgeStore → search.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Break long lines in test_notion.py, test_store.py, and test_sync_engine.py
to comply with the 88-character E501 limit enforced by ruff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements IMessageConnector (auth_type='local') that opens ~/Library/Messages/chat.db
read-only, converts Apple nanosecond timestamps to UTC datetimes, and yields one
Document per message with handle-based author resolution and chat display names.
Includes 8 tests using a temporary SQLite DB and 2 MCP tools (search + get conversation).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GContactsConnector using the People API v1 to paginate
contacts (names, emails, phones, orgs) and yield them as Documents
with doc_type="contact". Includes 6 unit tests and auto-registration
in the connectors __init__.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GCalendarConnector (registered as "gcalendar") following the
gmail.py pattern: module-level API functions for calendarList and events.list,
_format_event helper for human-readable content, paginated sync across all
calendars, and three MCP tools (get_events_today, search_events, next_meeting).
Includes 6 tests covering not_connected, auth_url scope, sync document fields,
disconnect, mcp_tools, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GDriveConnector using the Drive API v3 with OAuth, paginated
file listing, Google Workspace export (Docs→text/plain, Sheets→text/csv,
Slides→text/plain), and 3 MCP tools. Adds auto-registration in __init__.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements SlackConnector that syncs channel message history via the Slack
Web API, with OAuth credential storage, user-map resolution, and 3 MCP tools.
Includes 7 tests covering connectivity, auth, sync document correctness,
disconnect, MCP tools, and registry registration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GranolaConnector that syncs meeting notes from the Granola
public API, combining AI-generated summaries with full speaker transcripts
into searchable Documents. Includes cursor-based pagination, created_after
filtering, 2 MCP tools, and 8 tests covering all connector behaviours.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements NotionConnector that syncs pages via the Notion REST API,
renders block content (paragraph, headings, lists, code, quote, divider,
to_do) to markdown, and registers with ConnectorRegistry. Includes 8
tests covering auth, sync, rendering, disconnect, MCP tools, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire up ObsidianConnector and GmailConnector via auto-import in the
connectors __init__.py so they register on package import. Add a full
end-to-end integration test covering the Obsidian vault → SyncEngine →
KnowledgeStore → knowledge_search pipeline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GmailConnector registered under 'gmail' in ConnectorRegistry,
a shared oauth.py helper (build_google_auth_url, load/save/delete_tokens)
reusable by Drive/Calendar/Contacts, and 7 fully mocked pytest tests
covering auth state, sync document extraction, disconnect, mcp_tools,
and registry lookup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements ObsidianConnector (filesystem auth, no OAuth) that walks a vault
directory for .md/.markdown/.txt files, parses YAML frontmatter via a
dependency-free parser, skips hidden dirs and binary files, and yields
Document objects with doc_type="note" and obsidian:// deep-link URLs.
Exposes an obsidian_search_notes MCP ToolSpec. 9 tests cover connection
state, vault traversal, hidden-dir/binary filtering, frontmatter extraction,
and registry wiring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduces SyncEngine that wraps IngestionPipeline with a SQLite state
database (sync_state.db) for checkpoint/resume: cursors and item counts
are persisted after every 100-document batch and on completion/error.
Adds 4 tests covering single-connector ingestion, checkpoint accuracy,
unsynced-connector None return, and multi-connector source filtering.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements IngestionPipeline that deduplicates Documents by doc_id (both
in-memory and by loading existing doc_ids from the store on init), chunks
content via SemanticChunker, and persists all chunks with full provenance
metadata to KnowledgeStore. 8 tests cover single-doc ingestion, dedup across
calls and batches, persistence across pipeline instances, long-doc multi-chunk
splitting, atomic event chunking, multi-source filtering, and return-value accuracy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements SemanticChunker that splits text based on doc_type (event/contact
as single chunks, email on reply boundaries, message on double-newlines,
document/note on ## headings → paragraphs → sentences). ChunkResult carries
sequential 0-based indexes and inherits parent metadata; section headings are
added as chunk metadata. 16 tests covering all splitting strategies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements KnowledgeStore (extends MemoryBackend) for Deep Research with a
rich SQLite/FTS5 schema supporting source, doc_type, author, participants,
timestamp, thread_id, url, and chunk_index columns; BM25 ranking via FTS5
with porter unicode61 tokenizer; filtered retrieval by source, doc_type,
author, since, and until; MEMORY_STORE/MEMORY_RETRIEVE event emission; WAL
journal mode; and 15 isolated tests using tmp_path fixtures.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
