Add disk-persistent storage for ColBERT token-level embeddings so the
reranker can reuse pre-computed embeddings across queries instead of
re-encoding every document on every search.
- EmbeddingStore: individual .pt files per chunk with SQLite index for
O(1) lookup, graceful degradation when torch is not installed
- ColBERTReranker: checks EmbeddingStore for cached embeddings before
falling back to docFromText(), stores newly computed embeddings
- KnowledgeStore: ensures chunk_id is always present in retrieval
metadata (both at store time and as a backfill in retrieve)
- 18 tests covering round-trip persistence, deletion, torch-absent
degradation, and reranker cache-hit/miss behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new POST /{connector_id}/sync endpoint alongside the existing GET
sync-status endpoint. The new endpoint validates the connector is registered
and connected, then runs SyncEngine.sync() and returns chunks_indexed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements AttachmentStore that writes blobs to {base_dir}/{sha[:2]}/{sha}
with a SQLite metadata index tracking filename, MIME type, size, and the
accumulated list of source_doc_ids for each content-identical blob.
Includes 7 unit tests covering SHA-256 return, file path layout,
idempotent dedup, multi-source tracking, metadata retrieval, content
round-trip, and missing-blob None return.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parse the last_sync checkpoint timestamp into a datetime and pass it as
the `since` argument to connector.sync(), enabling subsequent syncs to
fetch only new items rather than a full re-fetch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3-task plan: wire since param for incremental sync, content-addressed
AttachmentStore with SHA-256 dedup, POST /sync trigger endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SetupWizard orchestrates the pick→connect→ingest→ready state machine with a
step indicator; SetupScreen now transitions to the wizard after all boot checks
pass (phase === 'ready') instead of immediately calling onReady().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SourcePicker renders a grouped card grid for selecting data sources; SourceConnectFlow
provides per-source auth panels (OAuth, filesystem, local); IngestDashboard polls sync
status every 2s with progress bars; ReadyScreen shows a celebration screen with
context-aware starter queries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements create_connectors_router() with GET /connectors, GET
/connectors/{id}, POST /connectors/{id}/connect, POST
/connectors/{id}/disconnect, and GET /connectors/{id}/sync endpoints.
Includes 6 passing tests covering list, detail, 404, connect, disconnect,
and sync status.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Exposes /v1/connectors/* REST API for desktop wizard and CLI:
list, detail, connect, disconnect, sync status.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds tests/agents/test_channel_agent_integration.py covering the full
stack from KnowledgeStore + IngestionPipeline through TwoStageRetriever,
KnowledgeSearchTool, and DeepResearchAgent into ChannelAgent/FakeChannel:
- test_quick_query_inline_response: asserts inline reply with meeting info
and no openjarvis:// escalation link
- test_deep_query_escalation_link: asserts openjarvis:// link is present
when engine issues a tool call and returns a long report
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds classify_query() heuristic (quick vs deep) and ChannelAgent that
bridges BaseChannel messages to any agent, delivering inline replies for
quick/short responses and preview+openjarvis:// escalation links for deep
or long responses. 18 tests cover classifier cases and agent behaviour
including non-blocking handler and error recovery.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3-task plan: query classifier, ChannelAgent with thread pool and
escalation links, integration test with real KnowledgeStore.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use Checkpoint instead of Searcher (doesn't need a pre-built index)
- Set gpus=0 in ColBERTConfig for CPU fallback
- Use cosine_similarity MaxSim scoring via queryFromText/docFromText
- Verified: reranking changes result order on 2,082 real Granola chunks
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests the full path from multi-source Document ingestion through
IngestionPipeline -> KnowledgeStore -> TwoStageRetriever ->
KnowledgeSearchTool -> DeepResearchAgent to a cited report, and
verifies cross-platform retrieval returns results from >= 2 sources.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Multi-hop research agent that uses native function calling (OpenAI
tool_calls format) to search personal data across sources via
KnowledgeSearchTool, cross-references results, and produces narrative
answers with inline source citations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add optional retriever parameter to KnowledgeSearchTool so it can
delegate to TwoStageRetriever (BM25 + reranking) when supplied, falling
back to direct KnowledgeStore retrieval otherwise.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Simpler alternative to OAuth-based Gmail connector. Uses Python's
built-in imaplib + email modules (no dependencies). Just needs an
email address and Google app password.
Live tested: 50 emails synced, 138 chunks indexed, search working.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two bugs found via live API testing:
- List notes API returns "notes" key, not "data" (was returning 0 results)
- Transcript speaker field is a dict {"source": "microphone"}, not a string
Verified: 112 real meeting notes synced, 2,226 chunks indexed and searchable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Slack connector now skips bot_id messages and non-content subtypes
(message_changed, message_deleted, bot_message, channel_join, channel_leave).
Found by comparing against hermes-agent reference implementation.
Live smoke test indexes 4,467 chunks from real docs/ markdown files and
verifies the full pipeline: Obsidian → SyncEngine → KnowledgeStore → search.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Break long lines in test_notion.py, test_store.py, and test_sync_engine.py
to comply with the 88-character E501 limit enforced by ruff.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements IMessageConnector (auth_type='local') that opens ~/Library/Messages/chat.db
read-only, converts Apple nanosecond timestamps to UTC datetimes, and yields one
Document per message with handle-based author resolution and chat display names.
Includes 8 tests using a temporary SQLite DB and 2 MCP tools (search + get conversation).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GContactsConnector using the People API v1 to paginate
contacts (names, emails, phones, orgs) and yield them as Documents
with doc_type="contact". Includes 6 unit tests and auto-registration
in the connectors __init__.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GCalendarConnector (registered as "gcalendar") following the
gmail.py pattern: module-level API functions for calendarList and events.list,
_format_event helper for human-readable content, paginated sync across all
calendars, and three MCP tools (get_events_today, search_events, next_meeting).
Includes 6 tests covering not_connected, auth_url scope, sync document fields,
disconnect, mcp_tools, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GDriveConnector using the Drive API v3 with OAuth, paginated
file listing, Google Workspace export (Docs→text/plain, Sheets→text/csv,
Slides→text/plain), and 3 MCP tools. Adds auto-registration in __init__.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements SlackConnector that syncs channel message history via the Slack
Web API, with OAuth credential storage, user-map resolution, and 3 MCP tools.
Includes 7 tests covering connectivity, auth, sync document correctness,
disconnect, MCP tools, and registry registration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GranolaConnector that syncs meeting notes from the Granola
public API, combining AI-generated summaries with full speaker transcripts
into searchable Documents. Includes cursor-based pagination, created_after
filtering, 2 MCP tools, and 8 tests covering all connector behaviours.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements NotionConnector that syncs pages via the Notion REST API,
renders block content (paragraph, headings, lists, code, quote, divider,
to_do) to markdown, and registers with ConnectorRegistry. Includes 8
tests covering auth, sync, rendering, disconnect, MCP tools, and registry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire up ObsidianConnector and GmailConnector via auto-import in the
connectors __init__.py so they register on package import. Add a full
end-to-end integration test covering the Obsidian vault → SyncEngine →
KnowledgeStore → knowledge_search pipeline.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds connect_cmd.py with --list, --disconnect, --sync, and positional source
routing (filesystem vs OAuth), registers it in the CLI, and covers all
behaviours with 5 passing unit tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements KnowledgeSearchTool registered under "knowledge_search" that wraps
KnowledgeStore with optional filters (source, doc_type, author, since, until,
top_k) and formats results with source attribution for agent consumption.
Includes 8 unit tests covering basic search, filter-by-source, filter-by-author,
no-results, empty-query, no-store, spec parameters, and registry checks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements GmailConnector registered under 'gmail' in ConnectorRegistry,
a shared oauth.py helper (build_google_auth_url, load/save/delete_tokens)
reusable by Drive/Calendar/Contacts, and 7 fully mocked pytest tests
covering auth state, sync document extraction, disconnect, mcp_tools,
and registry lookup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements ObsidianConnector (filesystem auth, no OAuth) that walks a vault
directory for .md/.markdown/.txt files, parses YAML frontmatter via a
dependency-free parser, skips hidden dirs and binary files, and yields
Document objects with doc_type="note" and obsidian:// deep-link URLs.
Exposes an obsidian_search_notes MCP ToolSpec. 9 tests cover connection
state, vault traversal, hidden-dir/binary filtering, frontmatter extraction,
and registry wiring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduces SyncEngine that wraps IngestionPipeline with a SQLite state
database (sync_state.db) for checkpoint/resume: cursors and item counts
are persisted after every 100-document batch and on completion/error.
Adds 4 tests covering single-connector ingestion, checkpoint accuracy,
unsynced-connector None return, and multi-connector source filtering.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements IngestionPipeline that deduplicates Documents by doc_id (both
in-memory and by loading existing doc_ids from the store on init), chunks
content via SemanticChunker, and persists all chunks with full provenance
metadata to KnowledgeStore. 8 tests cover single-doc ingestion, dedup across
calls and batches, persistence across pipeline instances, long-doc multi-chunk
splitting, atomic event chunking, multi-source filtering, and return-value accuracy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements SemanticChunker that splits text based on doc_type (event/contact
as single chunks, email on reply boundaries, message on double-newlines,
document/note on ## headings → paragraphs → sentences). ChunkResult carries
sequential 0-based indexes and inherits parent metadata; section headings are
added as chunk metadata. 16 tests covering all splitting strategies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements KnowledgeStore (extends MemoryBackend) for Deep Research with a
rich SQLite/FTS5 schema supporting source, doc_type, author, participants,
timestamp, thread_id, url, and chunk_index columns; BM25 ranking via FTS5
with porter unicode61 tokenizer; filtered retrieval by source, doc_type,
author, since, and until; MEMORY_STORE/MEMORY_RETRIEVE event emission; WAL
journal mode; and 15 isolated tests using tmp_path fixtures.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Document that pygemma v0.1.3 completion() does not accept
temperature/max_tokens (params accepted for ABC compliance)
- Add model-mismatch warning to stream() for consistency with generate()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move mid-file imports to top of test file to resolve E402 violations and apply ruff formatting to both gemma_cpp.py and test_gemma_cpp.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
