mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-07-31 03:12:16 +00:00
merge: resolve conflicts with main (streaming + auto-recover)
Merge main into fix/ssrf-check, keeping both the auto-recover logic for error-state agents and the async streaming support for the send_message endpoint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jon Saad-Falcon
@@ -0,0 +1,95 @@
|
||||
name: Bug Report
|
||||
description: Report a bug or unexpected behavior
|
||||
labels: ["type:bug"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thank you for reporting a bug! Please fill out the information below to help us investigate.
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: Description
|
||||
description: A clear description of what the bug is.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: steps
|
||||
attributes:
|
||||
label: Steps to Reproduce
|
||||
description: Steps to reproduce the behavior.
|
||||
placeholder: |
|
||||
1. Run `jarvis ask "..."`
|
||||
2. See error...
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected Behavior
|
||||
description: What you expected to happen.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: Actual Behavior
|
||||
description: What actually happened.
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating System
|
||||
options:
|
||||
- Linux
|
||||
- macOS
|
||||
- Windows
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: python
|
||||
attributes:
|
||||
label: Python Version
|
||||
options:
|
||||
- "3.10"
|
||||
- "3.11"
|
||||
- "3.12"
|
||||
- "3.13"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: hardware
|
||||
attributes:
|
||||
label: Hardware
|
||||
options:
|
||||
- NVIDIA GPU
|
||||
- AMD GPU
|
||||
- Apple Silicon
|
||||
- CPU only
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: engine
|
||||
attributes:
|
||||
label: Engine
|
||||
description: Which inference engine are you using?
|
||||
options:
|
||||
- Ollama
|
||||
- vLLM
|
||||
- llama.cpp
|
||||
- SGLang
|
||||
- MLX
|
||||
- Cloud (OpenAI/Anthropic/Google)
|
||||
- LiteLLM
|
||||
- Other
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / Traceback
|
||||
description: Paste any relevant logs or traceback here.
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,5 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: Ask a Question
|
||||
url: https://github.com/open-jarvis/OpenJarvis/discussions
|
||||
about: Use Discussions for questions and help
|
||||
@@ -0,0 +1,47 @@
|
||||
name: Feature Request
|
||||
description: Propose a new feature or enhancement
|
||||
labels: ["type:feature"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
For non-trivial changes, please open this issue for discussion before starting a PR. This saves everyone time by catching design issues early.
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: Problem Statement
|
||||
description: What problem does this solve? Why is this needed?
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Proposed Solution
|
||||
description: Describe how you'd like this to work.
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: area
|
||||
attributes:
|
||||
label: Primitive Area
|
||||
description: Which part of OpenJarvis does this touch?
|
||||
options:
|
||||
- Intelligence
|
||||
- Engine
|
||||
- Agent
|
||||
- Tools
|
||||
- Learning
|
||||
- Evals
|
||||
- Frontend
|
||||
- Channels
|
||||
- Rust
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives Considered
|
||||
description: Any alternative solutions or features you've considered.
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,63 @@
|
||||
name: New Eval Dataset
|
||||
description: Propose a new evaluation dataset or benchmark
|
||||
labels: ["type:eval"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Adding eval datasets is one of the easiest ways to contribute! Fill out the details below.
|
||||
- type: input
|
||||
id: name
|
||||
attributes:
|
||||
label: Dataset Name
|
||||
placeholder: e.g., HumanEval, GSM8K
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: url
|
||||
attributes:
|
||||
label: URL / Reference
|
||||
description: Link to the dataset or paper.
|
||||
placeholder: https://...
|
||||
validations:
|
||||
required: true
|
||||
- type: checkboxes
|
||||
id: capability
|
||||
attributes:
|
||||
label: What capability does it test?
|
||||
options:
|
||||
- label: Reasoning
|
||||
- label: Math
|
||||
- label: Code
|
||||
- label: Knowledge
|
||||
- label: Multimodal
|
||||
- label: Tool Use
|
||||
- label: Long Context
|
||||
- label: Other
|
||||
- type: input
|
||||
id: size
|
||||
attributes:
|
||||
label: Approximate Size
|
||||
description: Number of examples in the dataset.
|
||||
placeholder: e.g., 500
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: scorer
|
||||
attributes:
|
||||
label: Proposed Scorer Type
|
||||
options:
|
||||
- Exact Match
|
||||
- F1
|
||||
- BLEU
|
||||
- LLM-as-Judge
|
||||
- Custom
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: context
|
||||
attributes:
|
||||
label: Additional Context
|
||||
description: Any other details about this dataset.
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,16 @@
|
||||
## What does this PR do?
|
||||
|
||||
<!-- Brief description of the change and its motivation -->
|
||||
|
||||
## How was this tested?
|
||||
|
||||
<!-- Describe tests added or manual testing performed -->
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Tests pass (`uv run pytest tests/ -v`)
|
||||
- [ ] Linter passes (`uv run ruff check src/ tests/`)
|
||||
- [ ] Formatter passes (`uv run ruff format --check src/ tests/`)
|
||||
- [ ] New/changed public API has docstrings
|
||||
- [ ] Follows registry pattern (if adding new component)
|
||||
- [ ] Documentation updated (if applicable)
|
||||
@@ -63,7 +63,7 @@ jobs:
|
||||
run: uv run maturin develop --manifest-path rust/crates/openjarvis-python/Cargo.toml
|
||||
|
||||
- name: Run tests
|
||||
run: uv run pytest tests/ -v --tb=short
|
||||
run: uv run pytest tests/ -v --tb=short -m "not live and not cloud"
|
||||
|
||||
rust:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
name: Claude Issue Fixer
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [opened, labeled]
|
||||
issue_comment:
|
||||
types: [created]
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: claude-issues-${{ github.event.issue.number || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
fix:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
if: |
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event_name == 'issues' &&
|
||||
(contains(github.event.issue.labels.*.name, 'bug') ||
|
||||
contains(github.event.issue.labels.*.name, 'autofix'))) ||
|
||||
(github.event_name == 'issue_comment' &&
|
||||
!github.event.issue.pull_request &&
|
||||
contains(github.event.comment.body, '@claude') &&
|
||||
github.actor != 'claude[bot]')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
prompt: |
|
||||
You are an automated issue fixer for the OpenJarvis repository. Follow these steps in order:
|
||||
|
||||
## Step 1: Diagnose
|
||||
Read the issue thoroughly. Explore the codebase to understand the problem. If this is a bug report, attempt to reproduce it. Identify the root cause and affected files.
|
||||
|
||||
## Step 2: Comment your plan
|
||||
BEFORE making any code changes, post a comment on this issue describing:
|
||||
- Your root cause analysis
|
||||
- Which files need to change and why
|
||||
- Your implementation approach
|
||||
|
||||
## Step 3: Implement
|
||||
Create a branch named `claude/issue-${{ github.event.issue.number }}` and make the changes. Use conventional commit messages that reference the issue, e.g.:
|
||||
`fix: handle empty tool responses in orchestrator (fixes #${{ github.event.issue.number }})`
|
||||
|
||||
## Step 4: Test
|
||||
Run these commands and ensure both pass:
|
||||
- `uv run ruff check src/ tests/` (linting)
|
||||
- `uv run pytest tests/ -v --tb=short` (tests)
|
||||
|
||||
If tests fail, fix the issues before proceeding. If you added new functionality, add corresponding tests in `tests/` mirroring the `src/` directory structure.
|
||||
|
||||
## Step 5: Open PR
|
||||
Create a pull request that:
|
||||
- Links back to this issue (include `Fixes #${{ github.event.issue.number }}` in the PR body)
|
||||
- Describes what was changed and why
|
||||
- Includes a summary of test results
|
||||
|
||||
## If you cannot fix it
|
||||
If you cannot reproduce the issue, cannot determine the root cause, or the fix is beyond your capabilities, post a comment explaining:
|
||||
- What you investigated
|
||||
- What you found (or didn't find)
|
||||
- What additional information you need from the reporter
|
||||
@@ -0,0 +1,50 @@
|
||||
name: Claude PR Review
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize]
|
||||
issue_comment:
|
||||
types: [created]
|
||||
pull_request_review_comment:
|
||||
types: [created]
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: claude-review-${{ github.event.pull_request.number || github.event.issue.number || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
review:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
if: |
|
||||
github.event_name == 'pull_request' ||
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event_name == 'issue_comment' &&
|
||||
github.event.issue.pull_request &&
|
||||
contains(github.event.comment.body, '@claude') &&
|
||||
github.actor != 'claude[bot]') ||
|
||||
(github.event_name == 'pull_request_review_comment' &&
|
||||
contains(github.event.comment.body, '@claude') &&
|
||||
github.actor != 'claude[bot]')
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Read review instructions
|
||||
id: review
|
||||
run: |
|
||||
EOF=$(dd if=/dev/urandom bs=15 count=1 status=none | base64)
|
||||
echo "instructions<<$EOF" >> "$GITHUB_OUTPUT"
|
||||
cat REVIEW.md >> "$GITHUB_OUTPUT"
|
||||
echo "$EOF" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
prompt: ${{ steps.review.outputs.instructions }}
|
||||
@@ -0,0 +1,29 @@
|
||||
name: Auto-assign on "take"
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
permissions:
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
assign:
|
||||
if: >-
|
||||
!github.event.issue.pull_request
|
||||
&& contains(fromJSON('["take", "Take", "TAKE"]'), trim(github.event.comment.body))
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Assign commenter
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
COMMENTER="${{ github.event.comment.user.login }}"
|
||||
ISSUE="${{ github.event.issue.number }}"
|
||||
CURRENT=$(gh issue view "$ISSUE" --repo "${{ github.repository }}" --json assignees --jq '.assignees[].login' 2>/dev/null)
|
||||
if echo "$CURRENT" | grep -qx "$COMMENTER"; then
|
||||
echo "$COMMENTER is already assigned to #$ISSUE"
|
||||
else
|
||||
gh issue edit "$ISSUE" --repo "${{ github.repository }}" --add-assignee "$COMMENTER"
|
||||
echo "Assigned $COMMENTER to #$ISSUE"
|
||||
fi
|
||||
@@ -88,3 +88,4 @@ CLAUDE.md
|
||||
|
||||
# Research output
|
||||
research_mining_*
|
||||
.python-version
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
repos:
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
rev: v0.9.0
|
||||
hooks:
|
||||
- id: ruff
|
||||
args: [--fix]
|
||||
- id: ruff-format
|
||||
@@ -0,0 +1,85 @@
|
||||
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the overall community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or advances of any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email address, without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a professional setting
|
||||
|
||||
## Enforcement Responsibilities
|
||||
|
||||
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
|
||||
|
||||
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [the project maintainers](https://github.com/open-jarvis/OpenJarvis/discussions). All complaints will be reviewed and investigated promptly and fairly.
|
||||
|
||||
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
|
||||
|
||||
## Enforcement Guidelines
|
||||
|
||||
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
|
||||
|
||||
### 1. Correction
|
||||
|
||||
**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
|
||||
|
||||
**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
|
||||
|
||||
### 2. Warning
|
||||
|
||||
**Community Impact**: A violation through a single incident or series of actions.
|
||||
|
||||
**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
|
||||
|
||||
### 3. Temporary Ban
|
||||
|
||||
**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
|
||||
|
||||
**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
|
||||
|
||||
### 4. Permanent Ban
|
||||
|
||||
**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
|
||||
|
||||
**Consequence**: A permanent ban from any sort of public interaction within the community.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
|
||||
|
||||
Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
|
||||
|
||||
For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
|
||||
[Mozilla CoC]: https://github.com/mozilla/diversity
|
||||
[FAQ]: https://www.contributor-covenant.org/faq
|
||||
[translations]: https://www.contributor-covenant.org/translations
|
||||
|
||||
+201
@@ -0,0 +1,201 @@
|
||||
# Contributing to OpenJarvis
|
||||
|
||||
Thank you for your interest in contributing to OpenJarvis! This guide covers everything you need to know — from why to contribute, to how to submit your first pull request.
|
||||
|
||||
---
|
||||
|
||||
## Why Contribute?
|
||||
|
||||
Contributing to OpenJarvis isn't just about code — it's about building the future of on-device AI together. Here's what you get:
|
||||
|
||||
### Paper Acknowledgment
|
||||
|
||||
All contributors with merged pull requests will be acknowledged as contributors on the OpenJarvis paper release.
|
||||
|
||||
### Mac Mini Giveaway
|
||||
|
||||
We're giving away a Mac Mini to one lucky contributor! Install OpenJarvis on your personal machine and opt in via the desktop app to share anonymized savings data (FLOPs, dollar cost, energy) for a chance to win. Your data is fully anonymous — no IP, no hardware info beyond savings metrics. You must share your email via the desktop app to be eligible.
|
||||
|
||||
See the [Savings Leaderboard](https://open-jarvis.github.io/OpenJarvis/leaderboard/) for details.
|
||||
|
||||
### Path to Maintainership
|
||||
|
||||
Consistent contributors can grow into project maintainers:
|
||||
|
||||
- **Contributor** — anyone with a merged PR
|
||||
- **Reviewer** — invited after 3+ merged PRs in a domain area, can review PRs
|
||||
- **Maintainer** — reviewers who demonstrate sustained engagement and good judgment
|
||||
|
||||
### Recognition
|
||||
|
||||
Contributors are recognized in release notes and on our GitHub repository.
|
||||
|
||||
---
|
||||
|
||||
## Ways to Contribute
|
||||
|
||||
### Good First Contributions
|
||||
|
||||
These are great starting points for new contributors:
|
||||
|
||||
- Documentation improvements and typo fixes
|
||||
- Bug reports with reproducible steps
|
||||
- New eval datasets and scorers
|
||||
- Test coverage improvements
|
||||
|
||||
Look for issues labeled [`good-first-issue`](https://github.com/open-jarvis/OpenJarvis/labels/good-first-issue).
|
||||
|
||||
### Ideal Contributions
|
||||
|
||||
- Bug fixes with tests
|
||||
- Performance improvements
|
||||
- New tools, engines, or agents following the [registry pattern](docs/development/contributing.md#registry-pattern)
|
||||
- New channel integrations (Telegram, Discord, Slack, etc.)
|
||||
|
||||
### Harder to Review
|
||||
|
||||
These require more context and review time. **Please open an issue for discussion before starting a PR:**
|
||||
|
||||
- New primitives or major extensions to existing ones
|
||||
- Large refactors
|
||||
- Changes to core abstractions (`BaseAgent`, `InferenceEngine`, etc.)
|
||||
|
||||
### May Not Be Accepted
|
||||
|
||||
To avoid wasted effort, note that PRs in these categories are unlikely to be merged:
|
||||
|
||||
- Changes that break backwards compatibility in the public API
|
||||
- Changes that add significant new dependencies without justification
|
||||
- Changes that add friction to the user experience
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
| Requirement | Version | Notes |
|
||||
|---|---|---|
|
||||
| Python | 3.10+ | Required |
|
||||
| [uv](https://docs.astral.sh/uv/) | Latest | Package manager |
|
||||
| Node.js | 22+ | Only needed for ClaudeCodeAgent and WhatsApp channel |
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
git clone https://github.com/open-jarvis/OpenJarvis.git
|
||||
cd OpenJarvis
|
||||
uv sync --extra dev
|
||||
```
|
||||
|
||||
### Pre-commit Hooks
|
||||
|
||||
We use [pre-commit](https://pre-commit.com/) to run linting and formatting checks before each commit:
|
||||
|
||||
```bash
|
||||
uv run pre-commit install
|
||||
```
|
||||
|
||||
This installs Git hooks that automatically run [Ruff](https://docs.astral.sh/ruff/) on every commit. If the hooks fail, fix the issues and commit again.
|
||||
|
||||
For detailed development setup, code conventions, and project structure, see the [Development Guide](docs/development/contributing.md).
|
||||
|
||||
---
|
||||
|
||||
## Claiming Issues
|
||||
|
||||
1. Browse the [Roadmap](https://open-jarvis.github.io/OpenJarvis/development/roadmap/) for an item that interests you
|
||||
2. Check if a [GitHub issue](https://github.com/open-jarvis/OpenJarvis/issues) already exists for it — if not, [open one](https://github.com/open-jarvis/OpenJarvis/issues/new/choose) describing what you'd like to work on
|
||||
3. Comment **"take"** on the issue to get auto-assigned
|
||||
4. Fork, branch, and start working
|
||||
|
||||
If you've claimed an issue but can't finish it, please leave a comment so someone else can pick it up.
|
||||
|
||||
---
|
||||
|
||||
## Proposing Changes
|
||||
|
||||
### Trivial Changes
|
||||
|
||||
For small fixes (typos, doc improvements, simple bug fixes), go ahead and open a PR directly.
|
||||
|
||||
### Non-trivial Changes
|
||||
|
||||
For larger changes — new features, refactors, new dependencies — **open an issue first** to discuss the approach. This saves everyone time by catching design issues early.
|
||||
|
||||
Use the appropriate [issue template](https://github.com/open-jarvis/OpenJarvis/issues/new/choose):
|
||||
- **Bug Report** — for bugs with reproduction steps
|
||||
- **Feature Request** — for new functionality
|
||||
- **New Eval Dataset** — for contributing benchmarks
|
||||
|
||||
---
|
||||
|
||||
## Pull Request Process
|
||||
|
||||
### Before Submitting
|
||||
|
||||
1. Run the full test suite:
|
||||
```bash
|
||||
uv run pytest tests/ -v
|
||||
```
|
||||
2. Run the linter:
|
||||
```bash
|
||||
uv run ruff check src/ tests/
|
||||
```
|
||||
3. Run the formatter:
|
||||
```bash
|
||||
uv run ruff format --check src/ tests/
|
||||
```
|
||||
4. Add tests for new functionality
|
||||
5. Follow the [registry pattern](docs/development/contributing.md#registry-pattern) for new components
|
||||
|
||||
### Commit Messages
|
||||
|
||||
We use [Conventional Commits](https://www.conventionalcommits.org/):
|
||||
|
||||
```
|
||||
feat: add FAISS memory backend
|
||||
fix: handle empty tool responses in orchestrator
|
||||
docs: update engine discovery documentation
|
||||
test: add coverage for BM25 backend
|
||||
refactor: simplify agent base class helpers
|
||||
```
|
||||
|
||||
Keep the first line under 72 characters. Reference relevant issues (e.g., `fixes #42`).
|
||||
|
||||
### What Makes a Good PR
|
||||
|
||||
- **Focused** — one feature, fix, or refactor per PR
|
||||
- **Tested** — includes unit tests covering new code paths
|
||||
- **Documented** — updates docstrings and docs if adding public API
|
||||
- **Backwards compatible** — avoids breaking existing interfaces without discussion
|
||||
|
||||
---
|
||||
|
||||
## Contribution Areas
|
||||
|
||||
OpenJarvis is built on five composable primitives. Here's where you can contribute:
|
||||
|
||||
| Area | What to Build | Guide |
|
||||
|---|---|---|
|
||||
| **Intelligence** | Model catalog entries, routing strategies | [Dev Guide](docs/development/contributing.md) |
|
||||
| **Engines** | New inference backends (e.g., TensorRT, ONNX) | [Dev Guide](docs/development/contributing.md) |
|
||||
| **Agents** | New agent types, agent improvements | [Dev Guide](docs/development/contributing.md) |
|
||||
| **Tools** | New tools (browser, API clients, etc.) | [Dev Guide](docs/development/contributing.md) |
|
||||
| **Learning** | Router policies, reward functions, training | [Dev Guide](docs/development/contributing.md) |
|
||||
| **Evals** | New datasets, scorers, benchmark configs | [Dev Guide](docs/development/contributing.md) |
|
||||
| **Channels** | Chat platform integrations | [Dev Guide](docs/development/contributing.md) |
|
||||
| **Rust Port** | PyO3 bindings, crate parity with Python | See `rust/` directory |
|
||||
|
||||
---
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
This project follows the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you agree to uphold this code.
|
||||
|
||||
---
|
||||
|
||||
## Questions?
|
||||
|
||||
- Open a [Discussion](https://github.com/open-jarvis/OpenJarvis/discussions) for questions and help
|
||||
- Check the [documentation](https://open-jarvis.github.io/OpenJarvis/) for guides and API reference
|
||||
Binary file not shown.
Binary file not shown.
@@ -19,6 +19,8 @@
|
||||
> **[Project Site](https://scalingintelligence.stanford.edu/blogs/openjarvis/)**
|
||||
>
|
||||
> **[Leaderboard](https://open-jarvis.github.io/OpenJarvis/leaderboard/)**
|
||||
>
|
||||
> **[Roadmap](https://open-jarvis.github.io/OpenJarvis/development/roadmap/)**
|
||||
|
||||
## Why OpenJarvis?
|
||||
|
||||
@@ -33,63 +35,52 @@ git clone https://github.com/open-jarvis/OpenJarvis.git
|
||||
cd OpenJarvis
|
||||
uv sync # core framework
|
||||
uv sync --extra server # + FastAPI server
|
||||
|
||||
# Build the Rust extension (requires Rust: https://rustup.rs/)
|
||||
uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
|
||||
```
|
||||
|
||||
You also need a local inference backend: [Ollama](https://ollama.com), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), or [llama.cpp](https://github.com/ggerganov/llama.cpp).
|
||||
> **Python 3.14+:** set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command.
|
||||
|
||||
You also need a local inference backend: [Ollama](https://ollama.com), [vLLM](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), or [llama.cpp](https://github.com/ggerganov/llama.cpp). Alternatively, use the `cloud` engine with [OpenAI](https://openai.com), [Anthropic](https://anthropic.com), [Google Gemini](https://ai.google.dev), [OpenRouter](https://openrouter.ai), or [MiniMax](https://www.minimax.io) by setting the corresponding API key environment variable.
|
||||
|
||||
## Quick Start
|
||||
|
||||
The fastest path is Ollama on any machine with Python 3.10+:
|
||||
|
||||
```bash
|
||||
# 1. Install OpenJarvis
|
||||
# 1. Install and detect hardware
|
||||
git clone https://github.com/open-jarvis/OpenJarvis.git
|
||||
cd OpenJarvis
|
||||
uv sync
|
||||
|
||||
# 2. Detect hardware and generate config
|
||||
uv run jarvis init
|
||||
|
||||
# 3. Install and start Ollama (https://ollama.com)
|
||||
# 2. Start Ollama and pull a model
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
ollama serve # start the Ollama server
|
||||
|
||||
# 4. Pull a model
|
||||
ollama serve &
|
||||
ollama pull qwen3:8b
|
||||
|
||||
# 5. Ask a question
|
||||
# 3. Ask a question
|
||||
uv run jarvis ask "What is the capital of France?"
|
||||
|
||||
# 6. Verify your setup
|
||||
uv run jarvis doctor
|
||||
```
|
||||
|
||||
`jarvis init` auto-detects your hardware and recommends the best engine. After init, it prints engine-specific next steps. Run `uv run jarvis doctor` at any time to diagnose configuration or connectivity issues.
|
||||
`jarvis init` auto-detects your hardware and recommends the best engine. Run `uv run jarvis doctor` at any time to diagnose issues.
|
||||
|
||||
## Development
|
||||
Full documentation — including Docker deployment, cloud engines, development setup, and tutorials — at **[open-jarvis.github.io/OpenJarvis](https://open-jarvis.github.io/OpenJarvis/)**.
|
||||
|
||||
From source, you need to make sure Rust is installed on System:
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! See the [Contributing Guide](CONTRIBUTING.md) for incentives, contribution types, and the PR process.
|
||||
|
||||
Quick start for contributors:
|
||||
|
||||
```bash
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
```
|
||||
|
||||
Then, you need the Rust extension for full functionality (security, tools, agents, etc.):
|
||||
|
||||
```bash
|
||||
# 1. Clone and install Python deps
|
||||
git clone https://github.com/open-jarvis/OpenJarvis.git
|
||||
cd OpenJarvis
|
||||
uv sync --extra dev
|
||||
|
||||
# 2. Build and install the Rust extension (requires Rust toolchain)
|
||||
uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
|
||||
|
||||
# 3. Run tests
|
||||
uv run pre-commit install
|
||||
uv run pytest tests/ -v
|
||||
```
|
||||
|
||||
See [Contributing](docs/development/contributing.md) for more.
|
||||
Browse the [Roadmap](https://open-jarvis.github.io/OpenJarvis/development/roadmap/) for areas where help is needed. Comment **"take"** on any issue to get auto-assigned.
|
||||
|
||||
## About
|
||||
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
# OpenJarvis PR Review Instructions
|
||||
|
||||
You are reviewing pull requests for OpenJarvis, a local-first personal AI agent framework built with Python, Rust (PyO3), and TypeScript.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
Evaluate every PR against these criteria:
|
||||
|
||||
### 1. Relevance
|
||||
Is this PR doing something useful? Valid contributions include: bug fixes, new features, feature expansions, documentation improvements, test coverage, and performance improvements. Flag PRs that appear to be empty, auto-generated without substance, or unrelated to the project.
|
||||
|
||||
### 2. Completeness
|
||||
Does the code actually implement what the PR title and description claim? If the PR says "fix X", verify X is actually fixed. If it says "add Y", verify Y is fully added and functional — not partially implemented or stubbed out.
|
||||
|
||||
### 3. Correctness
|
||||
Check for logic errors, edge cases, and off-by-one errors. Pay particular attention to:
|
||||
- **Rust-Python bridge (PyO3) boundaries** — type conversions, error propagation, GIL handling
|
||||
- **Async/await patterns** — missing awaits, unclosed resources, blocking calls in async contexts
|
||||
- **Registry pattern compliance** — new components (engines, tools, agents, channels) must register via `ToolRegistry`, `EngineRegistry`, `AgentRegistry`, `ChannelRegistry`, etc. in `src/openjarvis/core/registry.py`
|
||||
- **Event bus integration** — new lifecycle events should use `EventBus` from `src/openjarvis/core/events.py`
|
||||
|
||||
### 4. Testing
|
||||
Does the PR include tests for new code paths? Are existing tests expected to still pass? New tools, engines, agents, and channels should have corresponding test files in `tests/` mirroring the `src/` structure.
|
||||
|
||||
### 5. Security
|
||||
Check for: hardcoded API keys or secrets, missing input validation at system boundaries (user input, external APIs), and anything that compromises local-first data isolation.
|
||||
|
||||
## Do NOT Comment On
|
||||
|
||||
- Formatting or style — Ruff handles this automatically in CI
|
||||
- Code in unchanged files outside the PR diff
|
||||
- Subjective naming preferences
|
||||
- Adding docstrings or comments to code the PR did not modify
|
||||
|
||||
## Output Format
|
||||
|
||||
- Post **inline comments** on specific lines for actionable issues
|
||||
- Post a **summary comment** containing: what the PR does, whether it achieves its stated goal, and any blocking concerns
|
||||
- Use severity levels:
|
||||
- `blocking` — must fix before merge
|
||||
- `suggestion` — consider fixing
|
||||
- `nit` — take it or leave it
|
||||
@@ -8,7 +8,7 @@ COPY frontend/ .
|
||||
RUN npm run build
|
||||
|
||||
# Stage 2: Build Python package
|
||||
FROM python:3.12-slim AS builder
|
||||
FROM python:3.12-slim-bookworm AS builder
|
||||
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml README.md ./
|
||||
@@ -21,7 +21,7 @@ RUN pip install --no-cache-dir uv && \
|
||||
uv pip install --system ".[server]"
|
||||
|
||||
# Stage 3: Runtime
|
||||
FROM python:3.12-slim
|
||||
FROM python:3.12-slim-bookworm
|
||||
|
||||
COPY --from=builder /usr/local /usr/local
|
||||
COPY --from=builder /app /app
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# NVIDIA GPU override — use with:
|
||||
# docker compose -f deploy/docker/docker-compose.yml -f deploy/docker/docker-compose.gpu.nvidia.yml up
|
||||
|
||||
services:
|
||||
jarvis:
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile.gpu
|
||||
volumes:
|
||||
- /proc:/proc:ro
|
||||
- /sys:/sys:ro
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
|
||||
ollama:
|
||||
image: ollama/ollama:latest
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=all
|
||||
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
@@ -1,5 +1,3 @@
|
||||
version: "3.9"
|
||||
|
||||
services:
|
||||
jarvis:
|
||||
build:
|
||||
@@ -9,17 +7,24 @@ services:
|
||||
- "8000:8000"
|
||||
environment:
|
||||
- OPENJARVIS_ENGINE_DEFAULT=ollama
|
||||
- OPENJARVIS_OLLAMA_HOST=http://ollama:11434
|
||||
- OLLAMA_HOST=http://ollama:11434
|
||||
depends_on:
|
||||
- ollama
|
||||
ollama:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
ollama:
|
||||
image: ollama/ollama
|
||||
image: ollama/ollama:latest
|
||||
ports:
|
||||
- "11434:11434"
|
||||
volumes:
|
||||
- ollama-models:/root/.ollama
|
||||
healthcheck:
|
||||
test: ["CMD", "ollama", "list"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 10s
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
|
||||
@@ -460,17 +460,90 @@ async fn boot_backend(backend: SharedBackend, status: SharedStatus) {
|
||||
return;
|
||||
}
|
||||
|
||||
let project_root = find_project_root();
|
||||
let mut project_root = find_project_root();
|
||||
|
||||
if project_root.is_none() {
|
||||
let mut s = status.lock().await;
|
||||
s.error = Some(
|
||||
"Could not find the OpenJarvis project directory. \
|
||||
Clone it with: git clone https://github.com/open-jarvis/OpenJarvis.git ~/OpenJarvis \
|
||||
then relaunch."
|
||||
.into(),
|
||||
);
|
||||
return;
|
||||
// Auto-clone on first launch
|
||||
let git_bin = resolve_bin("git");
|
||||
|
||||
// Check that git is installed
|
||||
if !std::path::Path::new(&git_bin).exists() && git_bin == "git" {
|
||||
let mut s = status.lock().await;
|
||||
s.error = Some(
|
||||
"Could not find 'git'. \
|
||||
Install it from https://git-scm.com then relaunch."
|
||||
.into(),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
let target_path = std::path::PathBuf::from(home_dir()).join("OpenJarvis");
|
||||
let clone_target = target_path.display().to_string();
|
||||
|
||||
// If the directory exists but is not a valid project, don't overwrite
|
||||
if target_path.exists() && !target_path.join("pyproject.toml").exists() {
|
||||
let mut s = status.lock().await;
|
||||
s.error = Some(format!(
|
||||
"{} exists but is not a valid OpenJarvis project. \
|
||||
Remove it and relaunch, or set OPENJARVIS_ROOT to the correct path.",
|
||||
clone_target,
|
||||
));
|
||||
return;
|
||||
}
|
||||
|
||||
{
|
||||
let mut s = status.lock().await;
|
||||
s.detail = "Downloading OpenJarvis (first launch)...".into();
|
||||
}
|
||||
|
||||
let clone_result = tokio::process::Command::new(&git_bin)
|
||||
.args([
|
||||
"clone",
|
||||
"--depth",
|
||||
"1",
|
||||
"https://github.com/open-jarvis/OpenJarvis.git",
|
||||
&clone_target,
|
||||
])
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.spawn();
|
||||
|
||||
match clone_result {
|
||||
Ok(child) => match child.wait_with_output().await {
|
||||
Ok(output) if output.status.success() => {
|
||||
project_root = Some(target_path);
|
||||
}
|
||||
Ok(output) => {
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
let mut s = status.lock().await;
|
||||
s.error = Some(format!(
|
||||
"Failed to download OpenJarvis: {}. \
|
||||
Clone manually: git clone https://github.com/open-jarvis/OpenJarvis.git {}",
|
||||
stderr.trim(),
|
||||
clone_target,
|
||||
));
|
||||
return;
|
||||
}
|
||||
Err(e) => {
|
||||
let mut s = status.lock().await;
|
||||
s.error = Some(format!(
|
||||
"Failed to download OpenJarvis: {}. \
|
||||
Clone manually: git clone https://github.com/open-jarvis/OpenJarvis.git {}",
|
||||
e, clone_target,
|
||||
));
|
||||
return;
|
||||
}
|
||||
},
|
||||
Err(e) => {
|
||||
let mut s = status.lock().await;
|
||||
s.error = Some(format!(
|
||||
"Could not run git: {}. \
|
||||
Install git from https://git-scm.com then relaunch.",
|
||||
e,
|
||||
));
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Kill any leftover server on our port from a previous run
|
||||
|
||||
@@ -34,6 +34,7 @@ interface SavingsData {
|
||||
heavy_low: number;
|
||||
heavy_high: number;
|
||||
};
|
||||
token_counting_version?: number;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -335,6 +336,7 @@ export function SavingsDashboard({ apiUrl }: { apiUrl: string }) {
|
||||
dollar_savings: dollarSavings,
|
||||
energy_wh_saved: energySaved,
|
||||
flops_saved: flopsSaved,
|
||||
token_counting_version: data.token_counting_version ?? 1,
|
||||
},
|
||||
}).catch(() => {});
|
||||
}, [data, optInEnabled, displayName, anonId]);
|
||||
|
||||
@@ -1,330 +0,0 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to OpenJarvis are documented in this file.
|
||||
|
||||
---
|
||||
|
||||
## Unreleased — Phase 11 (NanoClaw Subsumption)
|
||||
|
||||
*27 new files, ~3,565 lines, 147+ new tests. Full suite: 2059+ tests pass.*
|
||||
|
||||
### Added
|
||||
|
||||
- **`ClaudeCodeAgent`** (`agents/claude_code.py`) -- Wraps the
|
||||
`@anthropic-ai/claude-code` SDK via a bundled Node.js subprocess bridge.
|
||||
Communicates over stdin/stdout using sentinel-delimited JSON
|
||||
(`---OPENJARVIS_OUTPUT_START---` / `---OPENJARVIS_OUTPUT_END---`). The
|
||||
bundled runner is auto-installed to `~/.openjarvis/claude_code_runner/` via
|
||||
`npm install --production` on first use. Registered as `"claude_code"` with
|
||||
`accepts_tools = False`. Requires Node.js 22+ and `ANTHROPIC_API_KEY`.
|
||||
- **`WhatsAppBaileysChannel`** (`channels/whatsapp_baileys.py`) -- Bidirectional
|
||||
WhatsApp messaging using the Baileys protocol. Spawns a Node.js bridge
|
||||
subprocess (`whatsapp_baileys_bridge/`) for QR-code authentication, incoming
|
||||
message forwarding, and outbound delivery via JID addressing. Registered as
|
||||
`"whatsapp_baileys"` in `ChannelRegistry`. Authentication state is persisted
|
||||
to `~/.openjarvis/whatsapp_baileys_bridge/auth/`. New config section:
|
||||
`[channel.whatsapp_baileys]`.
|
||||
- **`ContainerRunner`** (`sandbox/runner.py`) -- Manages Docker (or Podman)
|
||||
container lifecycle for sandboxed agent execution. Builds `docker run --rm
|
||||
--network none -i` commands with allowlist-validated read-only bind mounts.
|
||||
Supports configurable image, timeout, concurrent container limit, and runtime
|
||||
binary. Uses the same sentinel-delimited JSON protocol as `ClaudeCodeAgent`.
|
||||
- **`SandboxedAgent`** (`sandbox/runner.py`) -- Transparent wrapper that runs
|
||||
any `BaseAgent` inside a container via `ContainerRunner`. Follows the
|
||||
`GuardrailsEngine` wrapper pattern. `accepts_tools = False`.
|
||||
- **`MountAllowlist` / `validate_mounts()`** (`sandbox/mount_security.py`) --
|
||||
Port of NanoClaw's `mount-security.ts`. Validates bind mounts against a JSON
|
||||
allowlist (allowed root directories + blocked filename patterns). Raises
|
||||
`ValueError` for blocked or out-of-root paths before the container starts.
|
||||
Default blocked patterns include `.ssh`, `.env`, `*.pem`, `*.key`, credential
|
||||
files, and cloud config directories.
|
||||
- **`TaskScheduler`** (`scheduler/scheduler.py`) -- Background polling scheduler
|
||||
supporting three schedule types: `cron` (via `croniter` or built-in fallback),
|
||||
`interval` (seconds), and `once` (ISO 8601 datetime). Runs a daemon thread
|
||||
(`jarvis-scheduler`) polling SQLite every 60 seconds (configurable). Executes
|
||||
due tasks via `JarvisSystem.ask()` with optional agent and tool selection.
|
||||
Publishes `scheduler_task_start` / `scheduler_task_end` events on the
|
||||
`EventBus`. New config section: `[scheduler]`.
|
||||
- **`SchedulerStore`** (`scheduler/store.py`) -- SQLite CRUD backend for
|
||||
scheduled tasks and run logs. Two tables: `scheduled_tasks` (task state) and
|
||||
`task_run_logs` (execution history). Supports task filtering by status and
|
||||
due-time polling via `get_due_tasks()`.
|
||||
- **Scheduler MCP tools** (`scheduler/tools.py`) -- Five new MCP-discoverable
|
||||
tools registered in `ToolRegistry`:
|
||||
- `schedule_task` -- Create a new scheduled task
|
||||
- `list_scheduled_tasks` -- List tasks filtered by status
|
||||
- `pause_scheduled_task` -- Pause an active task
|
||||
- `resume_scheduled_task` -- Resume a paused task (recomputes `next_run`)
|
||||
- `cancel_scheduled_task` -- Permanently cancel a task
|
||||
- **Scheduler CLI commands** -- `jarvis scheduler` subcommand group:
|
||||
- `jarvis scheduler create` -- Create a new scheduled task
|
||||
- `jarvis scheduler list` -- List all or filtered tasks
|
||||
- `jarvis scheduler pause <id>` -- Pause a task
|
||||
- `jarvis scheduler resume <id>` -- Resume a task
|
||||
- `jarvis scheduler cancel <id>` -- Cancel a task
|
||||
- `jarvis scheduler logs <id>` -- Show run history for a task
|
||||
- `jarvis scheduler start` -- Start the background scheduler daemon
|
||||
|
||||
### Changed
|
||||
|
||||
- `ChannelRegistry` now includes `WhatsAppBaileysChannel`.
|
||||
- `AgentRegistry` now includes `ClaudeCodeAgent` (`"claude_code"`).
|
||||
- Architecture overview and source directory layout updated to reflect new
|
||||
`sandbox/` and `scheduler/` modules.
|
||||
|
||||
---
|
||||
|
||||
## Unreleased — Phase 10 Tooling Updates
|
||||
|
||||
### Added
|
||||
|
||||
- **`build_tool_descriptions()` shared builder** -- Single source of truth for
|
||||
generating enriched tool descriptions in agent system prompts. Produces
|
||||
Markdown sections with name, description, category, and parameter schemas.
|
||||
- **Enriched agent prompts** -- `NativeReActAgent`, `NativeOpenHandsAgent`,
|
||||
`RLMAgent`, and `OrchestratorAgent` (structured mode) now inject detailed
|
||||
tool descriptions into their system prompts via the shared builder.
|
||||
- **Case-insensitive parsing** -- ReAct (`Action:` / `Final Answer:`) and
|
||||
Orchestrator structured-mode parsing (`TOOL:` / `FINAL_ANSWER:`) are now
|
||||
case-insensitive.
|
||||
- **Multi-provider tool_calls extraction** -- `CloudEngine` now extracts
|
||||
`tool_calls` from Anthropic (`tool_use` content blocks) and Google
|
||||
(`function_call` parts), normalizing to the flat `{id, name, arguments}`
|
||||
format. `LiteLLM` engine handles the flat-format tool calls returned by
|
||||
the LiteLLM proxy.
|
||||
- **RLM tool awareness** -- `RLMAgent` injects an `## Available Tools`
|
||||
section into its system prompt when tools are provided.
|
||||
- **Orchestrator structured tool descriptions** -- Structured mode passes
|
||||
`tools=self._tools` to `build_system_prompt()` for enriched descriptions.
|
||||
- **Telemetry modules** -- `EfficiencyMetrics`, `GPUMonitor`, `VLLMMetrics`
|
||||
for energy, GPU utilization, and vLLM server-side metrics collection.
|
||||
- **Eval TOML config** -- TOML-based eval suite configuration system for
|
||||
defining models x benchmarks matrices.
|
||||
|
||||
### Changed
|
||||
|
||||
- Agent prompt generation now uses `build_tool_descriptions()` instead of
|
||||
inline tool name listing.
|
||||
- `build_system_prompt()` in `prompt_registry.py` accepts an optional `tools`
|
||||
parameter for enriched descriptions from `BaseTool` instances.
|
||||
- ReAct and OpenHands regex patterns updated for case-insensitive matching.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Engine `tool_calls` normalization -- Anthropic `tool_use` blocks and Google
|
||||
`function_call` parts are now correctly extracted and converted to the
|
||||
standard flat format used by agents.
|
||||
|
||||
---
|
||||
|
||||
## v0.1.0
|
||||
|
||||
*Phase 5 -- SDK, Production Readiness, and Documentation*
|
||||
|
||||
### Added
|
||||
|
||||
- **Python SDK** -- `Jarvis` class providing a high-level sync API for
|
||||
programmatic use
|
||||
- `ask()` / `ask_full()` methods for direct engine and agent mode queries
|
||||
- `MemoryHandle` proxy for lazy memory backend initialization
|
||||
- `list_models()` and `list_engines()` for runtime introspection
|
||||
- Router policy selection via config (`learning.default_policy`)
|
||||
- Lazy engine initialization with automatic discovery and health probing
|
||||
- Resource cleanup via `close()`
|
||||
- **Benchmarking framework**
|
||||
- `BaseBenchmark` ABC and `BenchmarkSuite` runner
|
||||
- `LatencyBenchmark` measuring per-call latency (mean, p50, p95, min, max)
|
||||
- `ThroughputBenchmark` measuring tokens-per-second throughput
|
||||
- `BenchmarkResult` dataclass with JSONL export
|
||||
- `jarvis bench run` CLI with options for model, engine, sample count,
|
||||
benchmark selection, and JSON/JSONL output
|
||||
- **Docker deployment**
|
||||
- `Dockerfile` -- Multi-stage Python 3.12-slim build with `[server]` extra
|
||||
- `Dockerfile.gpu` -- NVIDIA CUDA 12.4 runtime variant
|
||||
- `docker-compose.yml` -- Services for `jarvis` (port 8000) and `ollama`
|
||||
(port 11434)
|
||||
- `deploy/systemd/openjarvis.service` -- systemd unit file for Linux
|
||||
- `deploy/launchd/com.openjarvis.plist` -- launchd plist for macOS
|
||||
- **Documentation site** -- MkDocs Material with mkdocstrings, covering
|
||||
getting started, user guide, architecture, API reference, deployment, and
|
||||
development
|
||||
|
||||
---
|
||||
|
||||
## v0.5.0
|
||||
|
||||
*Phase 4 -- Learning, Telemetry, and Router Policies*
|
||||
|
||||
### Added
|
||||
|
||||
- **Learning system**
|
||||
- `RouterPolicy` ABC and `RoutingContext` dataclass
|
||||
- `RewardFunction` ABC for scoring inference results
|
||||
- `HeuristicRewardFunction` scoring on latency, cost, and efficiency
|
||||
- `RouterPolicyRegistry` for pluggable routing strategies
|
||||
- `HeuristicRouter` registered as `"heuristic"` policy (6 priority rules:
|
||||
code detection, math detection, short/long queries, urgency override,
|
||||
default fallback)
|
||||
- `TraceDrivenPolicy` registered as `"learned"` policy with batch updates
|
||||
via `update_from_traces()` and online updates via `observe()`
|
||||
- `GRPORouterPolicy` stub registered as `"grpo"` for future RL training
|
||||
- `ensure_registered()` pattern for lazy, test-safe registration
|
||||
- **Telemetry aggregation**
|
||||
- `TelemetryAggregator` with `per_model_stats()`, `per_engine_stats()`,
|
||||
`top_models()`, `summary()`, `export_records()`, and `clear()` methods
|
||||
- Time-range filtering via `since` / `until` parameters
|
||||
- `ModelStats` and `EngineStats` dataclasses
|
||||
- `AggregatedStats` summary dataclass
|
||||
- **CLI enhancements**
|
||||
- `--router` flag on `jarvis ask` for explicit policy selection
|
||||
- `jarvis telemetry stats` -- display aggregated telemetry statistics
|
||||
- `jarvis telemetry export --format json|csv` -- export telemetry records
|
||||
- `jarvis telemetry clear --yes` -- delete all telemetry records
|
||||
|
||||
---
|
||||
|
||||
## v0.4.0
|
||||
|
||||
*Phase 3 -- Agents, Tools, and API Server*
|
||||
|
||||
### Added
|
||||
|
||||
- **Agent system**
|
||||
- `BaseAgent` ABC with `run()` method returning `AgentResult`
|
||||
- `AgentContext` dataclass with conversation, tools, and memory results
|
||||
- `AgentResult` dataclass with content, tool results, turns, and metadata
|
||||
- `AgentRegistry` for pluggable agent implementations
|
||||
- `SimpleAgent` -- single-turn query-to-response, no tool calling
|
||||
- `OrchestratorAgent` -- multi-turn tool-calling loop with `ToolExecutor`,
|
||||
configurable `max_turns`
|
||||
- `CustomAgent` -- template for user-defined agent behavior
|
||||
- **Tool system**
|
||||
- `BaseTool` ABC with `spec` property and `execute()` method
|
||||
- `ToolSpec` dataclass describing tool interface and characteristics
|
||||
- `ToolExecutor` dispatch engine with JSON argument parsing, latency
|
||||
tracking, and event bus integration (`TOOL_CALL_START` / `TOOL_CALL_END`)
|
||||
- `ToolRegistry` for tool discovery
|
||||
- `to_openai_function()` method for OpenAI function calling format
|
||||
- Built-in tools:
|
||||
- `CalculatorTool` -- safe math evaluation via AST parsing
|
||||
- `ThinkTool` -- reasoning scratchpad for chain-of-thought
|
||||
- `RetrievalTool` -- memory search integration
|
||||
- `LLMTool` -- sub-model calls within agent loops
|
||||
- `FileReadTool` -- safe file reading with path validation
|
||||
- **OpenAI-compatible API server** (`jarvis serve`)
|
||||
- FastAPI + Uvicorn with optional `[server]` extra
|
||||
- `POST /v1/chat/completions` -- non-streaming and SSE streaming
|
||||
- `GET /v1/models` -- list available models
|
||||
- `GET /health` -- health check endpoint
|
||||
- Pydantic request/response models matching OpenAI API format
|
||||
|
||||
---
|
||||
|
||||
## v0.3.0
|
||||
|
||||
*Phase 2 -- Memory System*
|
||||
|
||||
### Added
|
||||
|
||||
- **Memory backends**
|
||||
- `MemoryBackend` ABC with `store()`, `retrieve()`, `delete()`, `clear()`
|
||||
- `RetrievalResult` dataclass with content, score, source, and metadata
|
||||
- `MemoryRegistry` for backend discovery
|
||||
- `SQLiteMemory` -- zero-dependency default using SQLite FTS5 with BM25
|
||||
ranking and FTS5 query escaping
|
||||
- `FAISSMemory` -- vector search using FAISS with sentence-transformers
|
||||
embeddings (optional `[memory-faiss]` extra)
|
||||
- `ColBERTMemory` -- ColBERTv2 neural retrieval backend (optional
|
||||
`[memory-colbert]` extra)
|
||||
- `BM25Memory` -- BM25 ranking backend using rank-bm25 (optional
|
||||
`[memory-bm25]` extra)
|
||||
- `HybridMemory` -- Reciprocal Rank Fusion combining multiple backends
|
||||
- **Document processing**
|
||||
- `ChunkConfig` dataclass for chunk size and overlap settings
|
||||
- `chunk_text()` for splitting documents into overlapping chunks
|
||||
- `ingest_path()` for recursively indexing files and directories
|
||||
- `read_document()` with support for plain text, Markdown, and PDF
|
||||
(optional `[memory-pdf]` extra)
|
||||
- **Context injection**
|
||||
- `ContextConfig` with top-k, minimum score, and max context token settings
|
||||
- `inject_context()` for prepending memory results as system messages with
|
||||
source attribution
|
||||
- `--no-context` flag on `jarvis ask` to disable injection
|
||||
- **CLI commands**
|
||||
- `jarvis memory index <path>` -- index documents into memory
|
||||
- `jarvis memory search <query>` -- search memory for relevant chunks
|
||||
- `jarvis memory stats` -- show backend statistics
|
||||
- **Event bus integration** -- `MEMORY_STORE` and `MEMORY_RETRIEVE` events
|
||||
|
||||
---
|
||||
|
||||
## v0.2.0
|
||||
|
||||
*Phase 1 -- Intelligence and Inference*
|
||||
|
||||
### Added
|
||||
|
||||
- **Intelligence primitive**
|
||||
- `ModelSpec` dataclass with parameter count, context length, quantization,
|
||||
VRAM requirements, and supported engines
|
||||
- `ModelRegistry` for model metadata storage
|
||||
- `BUILTIN_MODELS` catalog with pre-defined model specifications
|
||||
- `register_builtin_models()` and `merge_discovered_models()` helpers
|
||||
- `HeuristicRouter` with rule-based model selection
|
||||
- `build_routing_context()` for query analysis (code detection, math
|
||||
detection, length classification)
|
||||
- **Inference engines**
|
||||
- `InferenceEngine` ABC with `generate()`, `stream()`, `list_models()`,
|
||||
and `health()` methods
|
||||
- `EngineRegistry` for engine discovery
|
||||
- `OllamaEngine` -- Ollama backend via native HTTP API with tool call
|
||||
extraction
|
||||
- `VllmEngine` -- vLLM backend via OpenAI-compatible API
|
||||
- `LlamaCppEngine` -- llama.cpp server backend
|
||||
- `EngineConnectionError` for unreachable engines
|
||||
- `messages_to_dicts()` for Message-to-OpenAI-format conversion
|
||||
- **Engine discovery**
|
||||
- `discover_engines()` -- probe all registered engines for health
|
||||
- `discover_models()` -- aggregate model lists across engines
|
||||
- `get_engine()` -- get configured default with automatic fallback
|
||||
- **Hardware detection**
|
||||
- NVIDIA GPU detection via `nvidia-smi`
|
||||
- AMD GPU detection via `rocm-smi`
|
||||
- Apple Silicon detection via `system_profiler`
|
||||
- CPU brand detection via `/proc/cpuinfo` and `sysctl`
|
||||
- `recommend_engine()` mapping hardware to best engine
|
||||
- **Telemetry**
|
||||
- `TelemetryRecord` dataclass with timing, tokens, energy, and cost
|
||||
- `TelemetryStore` with SQLite persistence and EventBus subscription
|
||||
- `instrumented_generate()` wrapper for automatic telemetry recording
|
||||
- **CLI**
|
||||
- `jarvis ask <query>` -- query via discovered engine
|
||||
- `jarvis ask --agent simple <query>` -- route through SimpleAgent
|
||||
- `jarvis model list` -- list models from running engines
|
||||
- `jarvis model info <model>` -- show model details
|
||||
|
||||
---
|
||||
|
||||
## v0.1.0
|
||||
|
||||
*Phase 0 -- Project Scaffolding*
|
||||
|
||||
### Added
|
||||
|
||||
- **Project structure** -- `hatchling` build backend, `uv` package manager,
|
||||
`pyproject.toml` with extras for optional backends
|
||||
- **Registry system** -- `RegistryBase[T]` generic base class with
|
||||
class-specific entry isolation, `register()` decorator, `get()`, `create()`,
|
||||
`items()`, `keys()`, `contains()`, `clear()` methods
|
||||
- **Typed registries** -- `ModelRegistry`, `EngineRegistry`, `MemoryRegistry`,
|
||||
`AgentRegistry`, `ToolRegistry`, `RouterPolicyRegistry`, `BenchmarkRegistry`
|
||||
- **Core types** -- `Role` enum, `Message`, `Conversation` (with sliding
|
||||
window), `ModelSpec`, `Quantization` enum, `ToolCall`, `ToolResult`,
|
||||
`TelemetryRecord`, `StepType` enum, `TraceStep`, `Trace`
|
||||
- **Configuration** -- `JarvisConfig` dataclass hierarchy, TOML loader with
|
||||
overlay semantics, hardware auto-detection, `generate_default_toml()` for
|
||||
`jarvis init`
|
||||
- **Event bus** -- Synchronous pub/sub `EventBus` with `EventType` enum for
|
||||
inter-primitive communication
|
||||
- **CLI skeleton** -- Click-based `jarvis` command group with `--version`,
|
||||
`--help`, and `init` subcommand
|
||||
@@ -412,4 +412,4 @@ policy:
|
||||
5. Add an entry in `pyproject.toml` under `[project.optional-dependencies]`
|
||||
if the component requires new packages
|
||||
|
||||
See the [Extending OpenJarvis](extending.md) guide for complete examples.
|
||||
See the [registry pattern](#registry-pattern) section above for complete examples.
|
||||
|
||||
@@ -1,915 +0,0 @@
|
||||
# Extending OpenJarvis
|
||||
|
||||
OpenJarvis is designed to be extended through its registry pattern. Every
|
||||
major subsystem defines an abstract base class (ABC) and uses a typed registry
|
||||
for runtime discovery. To add a new component, implement the ABC, decorate
|
||||
it with the registry, and import it in the module's `__init__.py`.
|
||||
|
||||
This guide provides complete, working code examples for each extension point.
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Inference Engine
|
||||
|
||||
Inference engines connect OpenJarvis to an LLM runtime. All engines implement
|
||||
the `InferenceEngine` ABC defined in `engine/_stubs.py`.
|
||||
|
||||
### Step 1: Create the Engine Module
|
||||
|
||||
Create `src/openjarvis/engine/my_engine.py`:
|
||||
|
||||
```python
|
||||
"""My custom inference engine backend."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
from typing import Any, Dict, List
|
||||
|
||||
import httpx
|
||||
|
||||
from openjarvis.core.registry import EngineRegistry
|
||||
from openjarvis.core.types import Message
|
||||
from openjarvis.engine._base import (
|
||||
EngineConnectionError,
|
||||
InferenceEngine,
|
||||
messages_to_dicts,
|
||||
)
|
||||
|
||||
|
||||
@EngineRegistry.register("my_engine") # (1)!
|
||||
class MyEngine(InferenceEngine):
|
||||
"""Custom inference engine backend."""
|
||||
|
||||
engine_id = "my_engine" # (2)!
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
host: str = "http://localhost:9000",
|
||||
*,
|
||||
timeout: float = 120.0,
|
||||
) -> None:
|
||||
self._host = host.rstrip("/")
|
||||
self._client = httpx.Client(base_url=self._host, timeout=timeout)
|
||||
|
||||
def generate(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
**kwargs: Any,
|
||||
) -> Dict[str, Any]:
|
||||
"""Synchronous completion."""
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": messages_to_dicts(messages), # (3)!
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
# Pass tools if provided
|
||||
tools = kwargs.get("tools")
|
||||
if tools:
|
||||
payload["tools"] = tools
|
||||
|
||||
try:
|
||||
resp = self._client.post("/v1/chat/completions", json=payload)
|
||||
resp.raise_for_status()
|
||||
except (httpx.ConnectError, httpx.TimeoutException) as exc:
|
||||
raise EngineConnectionError(
|
||||
f"Engine not reachable at {self._host}"
|
||||
) from exc
|
||||
|
||||
data = resp.json()
|
||||
choice = data.get("choices", [{}])[0]
|
||||
message = choice.get("message", {})
|
||||
usage = data.get("usage", {})
|
||||
|
||||
result: Dict[str, Any] = {
|
||||
"content": message.get("content", ""),
|
||||
"usage": {
|
||||
"prompt_tokens": usage.get("prompt_tokens", 0),
|
||||
"completion_tokens": usage.get("completion_tokens", 0),
|
||||
"total_tokens": usage.get("total_tokens", 0),
|
||||
},
|
||||
"model": data.get("model", model),
|
||||
"finish_reason": choice.get("finish_reason", "stop"),
|
||||
}
|
||||
|
||||
# Extract tool calls if present
|
||||
raw_tool_calls = message.get("tool_calls", [])
|
||||
if raw_tool_calls:
|
||||
result["tool_calls"] = [
|
||||
{
|
||||
"id": tc.get("id", f"call_{i}"),
|
||||
"name": tc["function"]["name"],
|
||||
"arguments": tc["function"]["arguments"],
|
||||
}
|
||||
for i, tc in enumerate(raw_tool_calls)
|
||||
]
|
||||
return result
|
||||
|
||||
async def stream(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
**kwargs: Any,
|
||||
) -> AsyncIterator[str]:
|
||||
"""Yield token strings as they are generated."""
|
||||
# Implement SSE or WebSocket streaming for your engine
|
||||
result = self.generate(
|
||||
messages, model=model, temperature=temperature,
|
||||
max_tokens=max_tokens, **kwargs,
|
||||
)
|
||||
yield result.get("content", "")
|
||||
|
||||
def list_models(self) -> List[str]:
|
||||
"""Return identifiers of models available on this engine."""
|
||||
try:
|
||||
resp = self._client.get("/v1/models")
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return [m["id"] for m in data.get("data", [])]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def health(self) -> bool:
|
||||
"""Return True when the engine is reachable and healthy."""
|
||||
try:
|
||||
resp = self._client.get("/health", timeout=2.0)
|
||||
return resp.status_code == 200
|
||||
except Exception:
|
||||
return False
|
||||
```
|
||||
|
||||
1. The `@EngineRegistry.register("my_engine")` decorator makes this engine
|
||||
discoverable by key at runtime.
|
||||
2. The `engine_id` class attribute is used in telemetry and benchmark results.
|
||||
3. `messages_to_dicts()` converts `Message` objects to OpenAI-format dicts.
|
||||
|
||||
### Step 2: Register in `__init__.py`
|
||||
|
||||
Add your engine import to `src/openjarvis/engine/__init__.py`:
|
||||
|
||||
```python
|
||||
import openjarvis.engine.my_engine # noqa: F401
|
||||
```
|
||||
|
||||
If your engine requires optional dependencies, wrap the import:
|
||||
|
||||
```python
|
||||
try:
|
||||
import openjarvis.engine.my_engine # noqa: F401
|
||||
except ImportError:
|
||||
pass
|
||||
```
|
||||
|
||||
### Step 3: Add Optional Dependencies
|
||||
|
||||
If your engine needs extra packages, add them to `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[project.optional-dependencies]
|
||||
inference-myengine = [
|
||||
"my-engine-sdk>=1.0",
|
||||
]
|
||||
```
|
||||
|
||||
### Required ABC Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `generate` | `(messages, *, model, temperature, max_tokens, **kwargs)` | `Dict[str, Any]` | Synchronous completion with `content` and `usage` keys |
|
||||
| `stream` | `(messages, *, model, temperature, max_tokens, **kwargs)` | `AsyncIterator[str]` | Yields token strings as they are generated |
|
||||
| `list_models` | `()` | `List[str]` | Model identifiers available on this engine |
|
||||
| `health` | `()` | `bool` | `True` when the engine is reachable |
|
||||
|
||||
The `generate` return dict must include at minimum:
|
||||
|
||||
```python
|
||||
{
|
||||
"content": "The response text",
|
||||
"usage": {
|
||||
"prompt_tokens": 10,
|
||||
"completion_tokens": 20,
|
||||
"total_tokens": 30,
|
||||
},
|
||||
"model": "model-name",
|
||||
"finish_reason": "stop", # or "tool_calls"
|
||||
}
|
||||
```
|
||||
|
||||
!!! tip "Tool call support"
|
||||
If your engine supports tool/function calling, include a `"tool_calls"`
|
||||
key in the return dict. Each tool call should have `id`, `name`, and
|
||||
`arguments` (JSON string) keys.
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Memory Backend
|
||||
|
||||
Memory backends provide persistent, searchable storage. All backends implement
|
||||
the `MemoryBackend` ABC defined in `tools/storage/_stubs.py` (previously `memory/_stubs.py`).
|
||||
|
||||
### Complete Example
|
||||
|
||||
Create `src/openjarvis/tools/storage/my_backend.py`:
|
||||
|
||||
```python
|
||||
"""Custom memory backend example."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from openjarvis.core.registry import MemoryRegistry
|
||||
from openjarvis.tools.storage._stubs import MemoryBackend, RetrievalResult
|
||||
|
||||
|
||||
@MemoryRegistry.register("my_backend")
|
||||
class MyMemoryBackend(MemoryBackend):
|
||||
"""Custom memory backend implementation."""
|
||||
|
||||
backend_id = "my_backend"
|
||||
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
# Initialize your storage (database, index, etc.)
|
||||
self._store: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
def store(
|
||||
self,
|
||||
content: str,
|
||||
*,
|
||||
source: str = "",
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> str:
|
||||
"""Persist content and return a unique document id."""
|
||||
import uuid
|
||||
|
||||
doc_id = uuid.uuid4().hex
|
||||
self._store[doc_id] = {
|
||||
"content": content,
|
||||
"source": source,
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
return doc_id
|
||||
|
||||
def retrieve(
|
||||
self,
|
||||
query: str,
|
||||
*,
|
||||
top_k: int = 5,
|
||||
**kwargs: Any,
|
||||
) -> List[RetrievalResult]:
|
||||
"""Search for query and return the top-k results."""
|
||||
results: List[RetrievalResult] = []
|
||||
for doc_id, doc in self._store.items():
|
||||
# Implement your search/ranking logic here
|
||||
if query.lower() in doc["content"].lower():
|
||||
results.append(RetrievalResult(
|
||||
content=doc["content"],
|
||||
score=1.0,
|
||||
source=doc["source"],
|
||||
metadata=doc["metadata"],
|
||||
))
|
||||
return results[:top_k]
|
||||
|
||||
def delete(self, doc_id: str) -> bool:
|
||||
"""Delete a document by id. Return True if it existed."""
|
||||
return self._store.pop(doc_id, None) is not None
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Remove all stored documents."""
|
||||
self._store.clear()
|
||||
```
|
||||
|
||||
### Register in `__init__.py`
|
||||
|
||||
Add to `src/openjarvis/tools/storage/__init__.py`:
|
||||
|
||||
```python
|
||||
try:
|
||||
import openjarvis.tools.storage.my_backend # noqa: F401
|
||||
except ImportError:
|
||||
pass
|
||||
```
|
||||
|
||||
!!! note "Backward compatibility"
|
||||
The old `from openjarvis.memory._stubs import MemoryBackend` import path still works via backward-compatibility shims, but new code should use `openjarvis.tools.storage._stubs`.
|
||||
|
||||
### Required ABC Methods
|
||||
|
||||
| Method | Signature | Returns | Description |
|
||||
|---|---|---|---|
|
||||
| `store` | `(content, *, source, metadata)` | `str` | Persist content, return document ID |
|
||||
| `retrieve` | `(query, *, top_k, **kwargs)` | `List[RetrievalResult]` | Search and return ranked results |
|
||||
| `delete` | `(doc_id)` | `bool` | Delete by ID, return whether it existed |
|
||||
| `clear` | `()` | `None` | Remove all stored documents |
|
||||
|
||||
The `RetrievalResult` dataclass has these fields:
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class RetrievalResult:
|
||||
content: str # The retrieved text
|
||||
score: float = 0.0 # Relevance score
|
||||
source: str = "" # Source identifier
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Agent
|
||||
|
||||
Agents implement the logic for handling queries, calling tools, and managing
|
||||
multi-turn interactions. There are two paths depending on whether your agent
|
||||
uses tools:
|
||||
|
||||
- **Path A: Non-tool agent** -- Extend `BaseAgent` directly
|
||||
- **Path B: Tool-using agent** -- Extend `ToolUsingAgent` (which sets `accepts_tools = True` and provides a `ToolExecutor`)
|
||||
|
||||
### Path A: Non-tool Agent (extending BaseAgent)
|
||||
|
||||
Create `src/openjarvis/agents/my_agent.py`:
|
||||
|
||||
```python
|
||||
"""Custom agent implementation — single-turn, no tools."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Optional
|
||||
|
||||
from openjarvis.agents._stubs import AgentContext, AgentResult, BaseAgent
|
||||
from openjarvis.core.registry import AgentRegistry
|
||||
from openjarvis.engine._stubs import InferenceEngine
|
||||
|
||||
|
||||
@AgentRegistry.register("my_agent")
|
||||
class MyAgent(BaseAgent):
|
||||
"""Custom agent with specialized behavior."""
|
||||
|
||||
agent_id = "my_agent"
|
||||
|
||||
def run(
|
||||
self,
|
||||
input: str,
|
||||
context: Optional[AgentContext] = None,
|
||||
**kwargs: Any,
|
||||
) -> AgentResult:
|
||||
"""Execute the agent on input and return an AgentResult."""
|
||||
# Use BaseAgent helpers instead of manual event bus code
|
||||
self._emit_turn_start(input)
|
||||
|
||||
# Build messages from context + user input (with optional system prompt)
|
||||
messages = self._build_messages(
|
||||
input, context,
|
||||
system_prompt="You are a helpful assistant with specialized knowledge.",
|
||||
)
|
||||
|
||||
# Call engine.generate() with stored defaults (model, temperature, max_tokens)
|
||||
result = self._generate(messages)
|
||||
content = self._strip_think_tags(result.get("content", ""))
|
||||
|
||||
self._emit_turn_end(turns=1)
|
||||
return AgentResult(content=content, turns=1)
|
||||
```
|
||||
|
||||
!!! tip "BaseAgent helpers"
|
||||
`BaseAgent` provides these concrete helpers so you don't need to manually
|
||||
manage the event bus or engine calls:
|
||||
|
||||
| Helper | Purpose |
|
||||
|--------|---------|
|
||||
| `_emit_turn_start(input)` | Publish `AGENT_TURN_START` |
|
||||
| `_emit_turn_end(**data)` | Publish `AGENT_TURN_END` |
|
||||
| `_build_messages(input, context, *, system_prompt)` | Assemble message list |
|
||||
| `_generate(messages, **kwargs)` | Call engine with stored defaults |
|
||||
| `_strip_think_tags(text)` | Remove `<think>` blocks |
|
||||
| `_max_turns_result(tool_results, turns, content)` | Standard max-turns result |
|
||||
|
||||
### Path B: Tool-using Agent (extending ToolUsingAgent)
|
||||
|
||||
Create `src/openjarvis/agents/my_tool_agent.py`:
|
||||
|
||||
```python
|
||||
"""Custom tool-using agent with a multi-turn loop."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, List, Optional
|
||||
|
||||
from openjarvis.agents._stubs import AgentContext, AgentResult, ToolUsingAgent
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.core.registry import AgentRegistry
|
||||
from openjarvis.core.types import ToolCall, ToolResult
|
||||
from openjarvis.engine._stubs import InferenceEngine
|
||||
from openjarvis.tools._stubs import BaseTool
|
||||
|
||||
|
||||
@AgentRegistry.register("my_tool_agent")
|
||||
class MyToolAgent(ToolUsingAgent):
|
||||
"""Custom agent with tool-calling loop."""
|
||||
|
||||
agent_id = "my_tool_agent"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
engine: InferenceEngine,
|
||||
model: str,
|
||||
*,
|
||||
tools: Optional[List[BaseTool]] = None,
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 10,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, tools=tools, bus=bus,
|
||||
max_turns=max_turns, temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
|
||||
def run(
|
||||
self,
|
||||
input: str,
|
||||
context: Optional[AgentContext] = None,
|
||||
**kwargs: Any,
|
||||
) -> AgentResult:
|
||||
self._emit_turn_start(input)
|
||||
|
||||
messages = self._build_messages(input, context)
|
||||
tools_spec = self._executor.get_openai_tools()
|
||||
all_tool_results: list[ToolResult] = []
|
||||
turns = 0
|
||||
|
||||
for _ in range(self._max_turns):
|
||||
turns += 1
|
||||
result = self._generate(messages, tools=tools_spec)
|
||||
content = result.get("content", "")
|
||||
tool_calls = result.get("tool_calls", [])
|
||||
|
||||
if not tool_calls:
|
||||
self._emit_turn_end(turns=turns)
|
||||
return AgentResult(
|
||||
content=content,
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
)
|
||||
|
||||
# Execute each tool call
|
||||
for tc in tool_calls:
|
||||
call = ToolCall(
|
||||
id=tc.get("id", f"call_{turns}"),
|
||||
name=tc["name"],
|
||||
arguments=tc["arguments"],
|
||||
)
|
||||
tr = self._executor.execute(call)
|
||||
all_tool_results.append(tr)
|
||||
|
||||
# Max turns exceeded — use the standard helper
|
||||
return self._max_turns_result(all_tool_results, turns)
|
||||
```
|
||||
|
||||
!!! info "What ToolUsingAgent adds"
|
||||
`ToolUsingAgent` extends `BaseAgent` with:
|
||||
|
||||
- **`accepts_tools = True`** — enables `--tools` in CLI and `tools=` in SDK
|
||||
- **`self._executor`** — a `ToolExecutor` initialized from the provided tools
|
||||
- **`self._tools`** — the raw list of `BaseTool` instances
|
||||
- **`self._max_turns`** — configurable loop iteration limit (default: 10)
|
||||
|
||||
### Register in `__init__.py`
|
||||
|
||||
Add to `src/openjarvis/agents/__init__.py`:
|
||||
|
||||
```python
|
||||
try:
|
||||
import openjarvis.agents.my_agent # noqa: F401
|
||||
except ImportError:
|
||||
pass
|
||||
```
|
||||
|
||||
### Key Types
|
||||
|
||||
=== "AgentContext"
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class AgentContext:
|
||||
conversation: Conversation = field(default_factory=Conversation)
|
||||
tools: List[str] = field(default_factory=list)
|
||||
memory_results: List[Any] = field(default_factory=list)
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
```
|
||||
|
||||
=== "AgentResult"
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class AgentResult:
|
||||
content: str
|
||||
tool_results: List[ToolResult] = field(default_factory=list)
|
||||
turns: int = 0
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Tool
|
||||
|
||||
Tools are callable capabilities that agents can invoke during multi-turn
|
||||
reasoning. All tools implement the `BaseTool` ABC from `tools/_stubs.py`.
|
||||
|
||||
### Complete Example
|
||||
|
||||
Create `src/openjarvis/tools/my_tool.py`:
|
||||
|
||||
```python
|
||||
"""Custom tool implementation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from openjarvis.core.registry import ToolRegistry
|
||||
from openjarvis.core.types import ToolResult
|
||||
from openjarvis.tools._stubs import BaseTool, ToolSpec
|
||||
|
||||
|
||||
@ToolRegistry.register("my_tool")
|
||||
class MyTool(BaseTool):
|
||||
"""A custom tool that does something useful."""
|
||||
|
||||
tool_id = "my_tool"
|
||||
|
||||
@property
|
||||
def spec(self) -> ToolSpec:
|
||||
"""Return the tool specification."""
|
||||
return ToolSpec(
|
||||
name="my_tool",
|
||||
description="Does something useful with the provided input.",
|
||||
parameters={ # (1)!
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "The input to process",
|
||||
},
|
||||
"max_results": {
|
||||
"type": "integer",
|
||||
"description": "Maximum number of results to return",
|
||||
"default": 5,
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
category="utility",
|
||||
cost_estimate=0.001, # Estimated cost in USD per call
|
||||
latency_estimate=0.5, # Estimated latency in seconds
|
||||
requires_confirmation=False,
|
||||
)
|
||||
|
||||
def execute(self, **params: Any) -> ToolResult:
|
||||
"""Execute the tool with the given parameters."""
|
||||
query = params.get("query", "")
|
||||
max_results = params.get("max_results", 5)
|
||||
|
||||
if not query:
|
||||
return ToolResult(
|
||||
tool_name="my_tool",
|
||||
content="No query provided.",
|
||||
success=False,
|
||||
)
|
||||
|
||||
try:
|
||||
# Your tool logic here
|
||||
result_text = f"Processed '{query}' (max_results={max_results})"
|
||||
|
||||
return ToolResult(
|
||||
tool_name="my_tool",
|
||||
content=result_text,
|
||||
success=True,
|
||||
)
|
||||
except Exception as exc:
|
||||
return ToolResult(
|
||||
tool_name="my_tool",
|
||||
content=f"Error: {exc}",
|
||||
success=False,
|
||||
)
|
||||
```
|
||||
|
||||
1. The `parameters` dict follows the [JSON Schema](https://json-schema.org/)
|
||||
format used by OpenAI function calling. The `ToolExecutor` will parse
|
||||
incoming JSON arguments and pass them as keyword arguments to `execute()`.
|
||||
|
||||
### Register in `__init__.py`
|
||||
|
||||
Add to `src/openjarvis/tools/__init__.py`:
|
||||
|
||||
```python
|
||||
try:
|
||||
import openjarvis.tools.my_tool # noqa: F401
|
||||
except ImportError:
|
||||
pass
|
||||
```
|
||||
|
||||
### How Tools Are Invoked
|
||||
|
||||
The `ToolExecutor` handles the dispatch loop:
|
||||
|
||||
1. The agent's LLM generates a `tool_calls` response with tool name and
|
||||
JSON arguments
|
||||
2. `ToolExecutor.execute()` parses the JSON arguments
|
||||
3. The matching tool's `execute(**params)` is called
|
||||
4. The `ToolResult` is returned to the agent for the next turn
|
||||
|
||||
```python
|
||||
from openjarvis.tools._stubs import ToolExecutor
|
||||
|
||||
executor = ToolExecutor(
|
||||
tools=[MyTool()],
|
||||
bus=event_bus, # Optional — enables TOOL_CALL_START/END events
|
||||
)
|
||||
|
||||
# Dispatch a tool call
|
||||
from openjarvis.core.types import ToolCall
|
||||
|
||||
call = ToolCall(id="call_1", name="my_tool", arguments='{"query": "test"}')
|
||||
result = executor.execute(call)
|
||||
```
|
||||
|
||||
The `to_openai_function()` method converts a tool's spec to OpenAI function
|
||||
calling format, which is sent to the LLM alongside the conversation:
|
||||
|
||||
```python
|
||||
tool = MyTool()
|
||||
openai_format = tool.to_openai_function()
|
||||
# {
|
||||
# "type": "function",
|
||||
# "function": {
|
||||
# "name": "my_tool",
|
||||
# "description": "Does something useful...",
|
||||
# "parameters": { ... }
|
||||
# }
|
||||
# }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Benchmark
|
||||
|
||||
Benchmarks measure engine performance. All benchmarks implement the
|
||||
`BaseBenchmark` ABC from `bench/_stubs.py` and use the `ensure_registered()`
|
||||
pattern for lazy registration.
|
||||
|
||||
### Complete Example
|
||||
|
||||
Create `src/openjarvis/bench/my_benchmark.py`:
|
||||
|
||||
```python
|
||||
"""Custom benchmark — measures time to first token."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
|
||||
from openjarvis.bench._stubs import BaseBenchmark, BenchmarkResult
|
||||
from openjarvis.core.registry import BenchmarkRegistry
|
||||
from openjarvis.core.types import Message, Role
|
||||
from openjarvis.engine._stubs import InferenceEngine
|
||||
|
||||
|
||||
class TTFTBenchmark(BaseBenchmark):
|
||||
"""Measures time-to-first-token across multiple samples."""
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "ttft"
|
||||
|
||||
@property
|
||||
def description(self) -> str:
|
||||
return "Measures time-to-first-token latency"
|
||||
|
||||
def run(
|
||||
self,
|
||||
engine: InferenceEngine,
|
||||
model: str,
|
||||
*,
|
||||
num_samples: int = 10,
|
||||
) -> BenchmarkResult:
|
||||
ttft_values: list[float] = []
|
||||
errors = 0
|
||||
|
||||
for _ in range(num_samples):
|
||||
messages = [Message(role=Role.USER, content="Hello")]
|
||||
t0 = time.time()
|
||||
try:
|
||||
engine.generate(messages, model=model)
|
||||
ttft_values.append(time.time() - t0)
|
||||
except Exception:
|
||||
errors += 1
|
||||
|
||||
if not ttft_values:
|
||||
return BenchmarkResult(
|
||||
benchmark_name=self.name,
|
||||
model=model,
|
||||
engine=engine.engine_id,
|
||||
metrics={},
|
||||
samples=num_samples,
|
||||
errors=errors,
|
||||
)
|
||||
|
||||
return BenchmarkResult(
|
||||
benchmark_name=self.name,
|
||||
model=model,
|
||||
engine=engine.engine_id,
|
||||
metrics={
|
||||
"mean_ttft": sum(ttft_values) / len(ttft_values),
|
||||
"min_ttft": min(ttft_values),
|
||||
"max_ttft": max(ttft_values),
|
||||
},
|
||||
samples=num_samples,
|
||||
errors=errors,
|
||||
)
|
||||
|
||||
|
||||
def ensure_registered() -> None: # (1)!
|
||||
"""Register the TTFT benchmark if not already present."""
|
||||
if not BenchmarkRegistry.contains("ttft"):
|
||||
BenchmarkRegistry.register_value("ttft", TTFTBenchmark)
|
||||
```
|
||||
|
||||
1. The `ensure_registered()` function uses `contains()` before
|
||||
`register_value()` so it can be called multiple times safely. This is
|
||||
required because tests clear all registries between runs.
|
||||
|
||||
### Register in `__init__.py`
|
||||
|
||||
Update `src/openjarvis/bench/__init__.py` to call `ensure_registered()`:
|
||||
|
||||
```python
|
||||
from openjarvis.bench.my_benchmark import ensure_registered as _reg_ttft
|
||||
_reg_ttft()
|
||||
```
|
||||
|
||||
### BenchmarkResult Fields
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class BenchmarkResult:
|
||||
benchmark_name: str # e.g. "latency", "throughput"
|
||||
model: str # Model identifier
|
||||
engine: str # Engine identifier
|
||||
metrics: Dict[str, float] = ... # Measured values
|
||||
metadata: Dict[str, Any] = ... # Extra info
|
||||
samples: int = 0 # Number of samples run
|
||||
errors: int = 0 # Number of failed samples
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Router Policy
|
||||
|
||||
Router policies determine which model handles a given query. All policies
|
||||
implement the `RouterPolicy` ABC from `learning/_stubs.py`. The
|
||||
`RoutingContext` dataclass is defined in `core/types.py`.
|
||||
|
||||
### Complete Example
|
||||
|
||||
Create `src/openjarvis/learning/my_policy.py`:
|
||||
|
||||
```python
|
||||
"""Custom router policy — selects model based on query length."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import List, Optional
|
||||
|
||||
from openjarvis.core.registry import RouterPolicyRegistry
|
||||
from openjarvis.core.types import RoutingContext
|
||||
from openjarvis.learning._stubs import RouterPolicy
|
||||
|
||||
|
||||
class QueryLengthPolicy(RouterPolicy):
|
||||
"""Routes queries to models based on query length.
|
||||
|
||||
Short queries go to a fast, small model. Long or complex queries
|
||||
go to a larger, more capable model.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
available_models: Optional[List[str]] = None,
|
||||
*,
|
||||
default_model: str = "",
|
||||
fallback_model: str = "",
|
||||
short_threshold: int = 100,
|
||||
long_threshold: int = 500,
|
||||
) -> None:
|
||||
self._available = available_models or []
|
||||
self._default = default_model
|
||||
self._fallback = fallback_model
|
||||
self._short_threshold = short_threshold
|
||||
self._long_threshold = long_threshold
|
||||
|
||||
def select_model(self, context: RoutingContext) -> str:
|
||||
"""Return the model registry key best suited for this context."""
|
||||
available = self._available
|
||||
|
||||
if not available:
|
||||
return self._default or self._fallback or ""
|
||||
|
||||
if context.query_length < self._short_threshold:
|
||||
# Prefer the first (presumably smallest) available model
|
||||
return available[0]
|
||||
elif context.query_length > self._long_threshold:
|
||||
# Prefer the last (presumably largest) available model
|
||||
return available[-1]
|
||||
|
||||
# Default to configured model
|
||||
if self._default and self._default in available:
|
||||
return self._default
|
||||
return available[0]
|
||||
|
||||
|
||||
def ensure_registered() -> None:
|
||||
"""Register QueryLengthPolicy if not already present."""
|
||||
if not RouterPolicyRegistry.contains("query_length"):
|
||||
RouterPolicyRegistry.register_value("query_length", QueryLengthPolicy)
|
||||
|
||||
|
||||
ensure_registered()
|
||||
```
|
||||
|
||||
### Register in `__init__.py`
|
||||
|
||||
Update `src/openjarvis/learning/__init__.py`:
|
||||
|
||||
```python
|
||||
from openjarvis.learning.my_policy import ensure_registered as _reg_ql
|
||||
_reg_ql()
|
||||
```
|
||||
|
||||
### Using Your Policy
|
||||
|
||||
Once registered, your policy can be selected via the config file or CLI:
|
||||
|
||||
=== "Config (TOML)"
|
||||
|
||||
```toml
|
||||
[learning.routing]
|
||||
policy = "query_length"
|
||||
```
|
||||
|
||||
=== "CLI"
|
||||
|
||||
```bash
|
||||
uv run jarvis ask --router query_length "Hello"
|
||||
```
|
||||
|
||||
### The RoutingContext
|
||||
|
||||
The `RoutingContext` dataclass provides all the information a router needs:
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class RoutingContext:
|
||||
query: str = ""
|
||||
query_length: int = 0
|
||||
has_code: bool = False
|
||||
has_math: bool = False
|
||||
language: str = "en"
|
||||
urgency: float = 0.5 # 0 = low priority, 1 = real-time
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
```
|
||||
|
||||
The `build_routing_context()` helper in `learning/router.py` populates
|
||||
this from a raw query string, detecting code and math patterns automatically.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Component | ABC | Registry | Key location |
|
||||
|---|---|---|---|
|
||||
| Inference Engine | `InferenceEngine` | `EngineRegistry` | `engine/_stubs.py` |
|
||||
| Memory Backend | `MemoryBackend` | `MemoryRegistry` | `tools/storage/_stubs.py` |
|
||||
| Agent | `BaseAgent` | `AgentRegistry` | `agents/_stubs.py` |
|
||||
| Tool | `BaseTool` | `ToolRegistry` | `tools/_stubs.py` |
|
||||
| Benchmark | `BaseBenchmark` | `BenchmarkRegistry` | `bench/_stubs.py` |
|
||||
| Router Policy | `RouterPolicy` | `RouterPolicyRegistry` | `learning/_stubs.py` |
|
||||
| Learning Policy | `LearningPolicy` | `LearningRegistry` | `learning/_stubs.py` |
|
||||
|
||||
The general pattern for all extension points:
|
||||
|
||||
1. Implement the ABC in a new module
|
||||
2. Decorate the class with `@XRegistry.register("key")` or use
|
||||
`ensure_registered()` for lazy registration
|
||||
3. Import the module in the package's `__init__.py` (with `try/except
|
||||
ImportError` if optional deps are involved)
|
||||
4. Add tests in `tests/<module>/`
|
||||
5. Add optional dependencies to `pyproject.toml` if needed
|
||||
+111
-63
@@ -1,86 +1,134 @@
|
||||
# Roadmap
|
||||
|
||||
OpenJarvis development follows a phased approach, with each version adding
|
||||
a major primitive or cross-cutting capability to the framework.
|
||||
## Current Focus Areas
|
||||
|
||||
These are the areas where active development is happening and contributions are most impactful:
|
||||
|
||||
- **Post-training data** — building datasets and training pipelines from execution traces to improve agent routing and tool selection
|
||||
- **Multi-model orchestration pipelines** — coordinating multiple models within a single query (e.g., small model for classification, large model for generation)
|
||||
- **Energy-aware routing** — using power consumption data from telemetry to optimize for energy efficiency alongside latency and quality
|
||||
- **Plugin ecosystem** — community-contributed engines, tools, and agents distributed as Python packages
|
||||
- **Federated memory** — memory backends that synchronize across devices
|
||||
|
||||
---
|
||||
|
||||
## Development Phases
|
||||
## How to Get Involved
|
||||
|
||||
| Version | Phase | Status | Delivers |
|
||||
|---|---|---|---|
|
||||
| **v0.1** | Phase 0 -- Scaffolding | :material-check-circle:{ .green } Complete | Project scaffolding, registry system (`RegistryBase[T]`), core types (`Message`, `ModelSpec`, `Conversation`, `ToolResult`), configuration loader with hardware detection, Click CLI skeleton |
|
||||
| **v0.2** | Phase 1 -- Intelligence + Inference | :material-check-circle:{ .green } Complete | Intelligence primitive (model catalog, heuristic router), inference engines (Ollama, vLLM, llama.cpp), engine discovery and health probing, `jarvis ask` command working end-to-end |
|
||||
| **v0.3** | Phase 2 -- Memory | :material-check-circle:{ .green } Complete | Memory backends (SQLite/FTS5, FAISS, ColBERTv2, BM25, Hybrid/RRF), document chunking and ingestion pipeline, context injection with source attribution, `jarvis memory` commands |
|
||||
| **v0.4** | Phase 3 -- Agents + Tools + Server | :material-check-circle:{ .green } Complete | Agent system (SimpleAgent, OrchestratorAgent), tool system (Calculator, Think, Retrieval, LLM, FileRead), ToolExecutor dispatch engine, OpenAI-compatible API server (`jarvis serve`) |
|
||||
| **v0.5** | Phase 4 -- Learning + Telemetry | :material-check-circle:{ .green } Complete | Learning system (HeuristicRouter policy, TraceDrivenPolicy, GRPO stub), reward functions, telemetry aggregation (per-model/engine stats, export), `--router` CLI flag, `jarvis telemetry` commands |
|
||||
| **v1.0** | Phase 5 -- SDK + Production | :material-check-circle:{ .green } Complete | Python SDK (`Jarvis` class, `MemoryHandle`), multi-platform channel system (Telegram, Discord, Slack, WhatsApp, etc.), benchmarking framework (latency, throughput), Docker deployment (CPU + GPU), MkDocs documentation site |
|
||||
| **v1.1** | Phase 6 -- Traces + Learning | :material-check-circle:{ .green } Complete | Trace system (`TraceStore`, `TraceCollector`, `TraceAnalyzer`), trace-driven learning, MCP integration layer |
|
||||
| **v1.5** | Phase 10 -- Agent Restructuring | :material-check-circle:{ .green } Complete | BaseAgent helpers, ToolUsingAgent intermediate base, NativeReActAgent, NativeOpenHandsAgent, RLMAgent, OpenHandsAgent (SDK), `accepts_tools` introspection, backward-compat shims, CustomAgent removed |
|
||||
1. Browse the workstreams below for an item that interests you
|
||||
2. Check if a [GitHub issue](https://github.com/open-jarvis/OpenJarvis/issues) already exists for it — if not, [open one](https://github.com/open-jarvis/OpenJarvis/issues/new/choose)
|
||||
3. Comment **"take"** on the issue to get auto-assigned
|
||||
4. Read the [Contributing Guide](https://github.com/open-jarvis/OpenJarvis/blob/main/CONTRIBUTING.md) for development setup and PR process
|
||||
|
||||
---
|
||||
|
||||
## Current Status
|
||||
## Workstreams
|
||||
|
||||
OpenJarvis v1.5 (Phase 10) is complete. The framework provides:
|
||||
OpenJarvis development is organized into **five independent workstreams**. Contributors can pick any track that matches their skills and interests — workstreams are designed to be worked on in parallel without blocking each other.
|
||||
|
||||
- **Four core abstractions** -- Intelligence, Engine, Agentic Logic, Memory -- each with an ABC interface and registry-based discovery
|
||||
- **Five inference engines** -- Ollama, vLLM, llama.cpp, SGLang, Cloud (OpenAI/Anthropic/Google)
|
||||
- **Five memory backends** -- SQLite/FTS5, FAISS, ColBERTv2, BM25, Hybrid (RRF fusion)
|
||||
- **Seven agent types** -- Simple, Orchestrator, NativeReAct, NativeOpenHands, RLM, Operative, MonitorOperative
|
||||
- **Seven built-in tools** -- Calculator, Think, Retrieval, LLM, FileRead, WebSearch, CodeInterpreter
|
||||
- **Python SDK** -- `Jarvis` class for programmatic use
|
||||
- **OpenAI-compatible API server** -- `POST /v1/chat/completions`, `GET /v1/models`
|
||||
- **Benchmarking framework** -- Latency and throughput measurements
|
||||
- **Telemetry and traces** -- SQLite-backed recording and aggregation
|
||||
- **Docker deployment** -- CPU and GPU images with docker-compose
|
||||
Every item carries a maturity tag:
|
||||
|
||||
Phase 10 (Agent Restructuring) is complete. The agent hierarchy has been
|
||||
refactored with `BaseAgent` helpers, `ToolUsingAgent` intermediate base, and
|
||||
four new agent types (NativeReActAgent, NativeOpenHandsAgent, RLMAgent,
|
||||
OpenHandsAgent SDK).
|
||||
| Tag | Meaning | Contributor guidance |
|
||||
|-----|---------|---------------------|
|
||||
| **Ready** | Well-scoped, implementation path is clear | Pick it up — check [issues](https://github.com/open-jarvis/OpenJarvis/issues) for a spec or write one |
|
||||
| **Design Needed** | Concept is clear but needs a spec before code | Start a [design discussion](https://github.com/open-jarvis/OpenJarvis/discussions) or draft an RFC |
|
||||
| **Research-Stage** | Exploratory, needs investigation before designing | Read the relevant papers, prototype, share findings |
|
||||
|
||||
---
|
||||
|
||||
## Phase 10 Details
|
||||
### Workstream 1: Continuous Operators & Agents
|
||||
|
||||
Phase 10 refactored the agent hierarchy for composability and extensibility:
|
||||
Operators are OpenJarvis's key differentiator — persistent, scheduled, stateful agents that run autonomously on personal devices. The current tick-based architecture (OperatorManager → TaskScheduler → AgentExecutor → OperativeAgent) is solid but needs hardening for truly long-horizon autonomy.
|
||||
|
||||
### BaseAgent Helpers
|
||||
#### Where you can help
|
||||
|
||||
- **`_emit_turn_start` / `_emit_turn_end`** -- Event bus integration without boilerplate
|
||||
- **`_build_messages`** -- System prompt + context + input assembly
|
||||
- **`_generate`** -- Engine call with stored defaults
|
||||
- **`_max_turns_result`** -- Standard max-turns-exceeded result
|
||||
- **`_strip_think_tags`** -- Remove `<think>` blocks from model output
|
||||
|
||||
### ToolUsingAgent Intermediate Base
|
||||
|
||||
- Sets `accepts_tools = True` for CLI/SDK introspection
|
||||
- Initializes `ToolExecutor` from provided tools
|
||||
- Configurable `max_turns` loop limit
|
||||
|
||||
### New Agent Types
|
||||
|
||||
- **NativeReActAgent** (`native_react`, alias `react`) -- Thought-Action-Observation loop
|
||||
- **NativeOpenHandsAgent** (`native_openhands`) -- CodeAct-style code execution with URL pre-fetching
|
||||
- **RLMAgent** (`rlm`) -- Recursive LM with persistent REPL and sub-LM calls
|
||||
- **OpenHandsAgent** (`openhands`) -- Thin wrapper for real `openhands-sdk`
|
||||
| Item | Maturity | Details |
|
||||
|------|----------|---------|
|
||||
| Operator health checks & heartbeat monitoring | **Ready** | Add liveness probes to OperatorManager; surface in `jarvis operators status`. Detect stalled operators beyond the existing reconciliation loop. |
|
||||
| Metrics collection for operator manifests | **Ready** | The `metrics` field exists in `OperatorManifest` but is not collected. Wire it to telemetry. **Good first issue.** |
|
||||
| Capability policy enforcement | **Ready** | `required_capabilities` field exists in manifests but is not enforced. Connect to the existing RBAC `CapabilityPolicy` system. **Good first issue.** |
|
||||
| Rate limiting per operator | **Ready** | Prevent runaway operators from hammering inference. Add configurable rate limits to OperatorManager. |
|
||||
| Operator composition / chaining | **Design Needed** | Express dependencies between operators (operator A feeds results to operator B). Requires design for data passing and scheduling semantics. |
|
||||
| Event-driven operators | **Design Needed** | Operators that trigger on EventBus events (e.g., new file indexed, channel message received) rather than only cron/interval schedules. |
|
||||
| Operator versioning & rollback | **Design Needed** | Run v2 of an operator alongside v1. Roll back automatically on repeated failures. |
|
||||
| Self-improving operators via Learning | **Research-Stage** | Operators that use trace feedback to tune their own prompts, tool selection, and routing policies through the Learning primitive. |
|
||||
|
||||
---
|
||||
|
||||
## Future Directions
|
||||
### Workstream 2: Mobile & Messaging Clients
|
||||
|
||||
Beyond Phase 10, areas of ongoing exploration include:
|
||||
Personal AI must be accessible from the devices people actually carry. OpenJarvis runs on laptops, workstations, and servers — users interact via their phones. Channels bridge that gap. Today, WhatsApp (Baileys), Slack, and Telegram are bidirectional; iMessage is send-only; Android SMS does not exist.
|
||||
|
||||
- **GRPO training** -- Reinforcement learning from trace data to train the
|
||||
routing policy, moving beyond heuristics and simple statistics
|
||||
- **Streaming telemetry** -- Real-time performance dashboards and alerting
|
||||
- **Multi-model orchestration** -- Coordinating multiple models within a
|
||||
single query pipeline (e.g., small model for classification, large model
|
||||
for generation)
|
||||
- **Federated memory** -- Memory backends that synchronize across devices
|
||||
- **Plugin ecosystem** -- Community-contributed engines, tools, and agents
|
||||
distributed as Python packages
|
||||
- **Energy-aware routing** -- Using power consumption data from telemetry to
|
||||
optimize for energy efficiency alongside latency and quality
|
||||
#### Where you can help
|
||||
|
||||
| Item | Maturity | Details |
|
||||
|------|----------|---------|
|
||||
| iMessage bidirectional via BlueBubbles | **Ready** | Current implementation is send-only. Add webhook/polling listener for incoming messages using the BlueBubbles API. **Good first issue.** |
|
||||
| WhatsApp Baileys media support | **Ready** | Currently text-only. Add image, audio, and file handling to the Node.js bridge and Python channel. **Good first issue.** |
|
||||
| Slack rich messages | **Ready** | Current implementation is plain text. Add Slack Block Kit support for formatted responses, buttons, and attachments. |
|
||||
| Android SMS via Twilio/Vonage | **Design Needed** | No SMS implementation exists. Requires provider selection, two-way webhook architecture, and phone number provisioning flow. |
|
||||
| Unified notification system | **Design Needed** | Push notifications when operators complete tasks or need user attention. Requires per-channel notification adapters. |
|
||||
| Signal bidirectional | **Design Needed** | Currently send-only via signal-cli REST API. Add incoming message listener with background polling. |
|
||||
| Voice interface | **Research-Stage** | Speech-to-text (Whisper) → agent → text-to-speech loop over phone channels. Existing `speech/` module provides a foundation. |
|
||||
|
||||
---
|
||||
|
||||
### Workstream 3: Secure Cloud Collaboration
|
||||
|
||||
Personal AI's core tension: local models preserve privacy but lack capability; cloud models are powerful but require trusting a provider with your data. This workstream resolves that through **Minions-style collaborative inference** (local handles context, cloud handles reasoning) and **TEE-based confidential computing** (cloud cannot see your data even during inference).
|
||||
|
||||
**References:**
|
||||
|
||||
- [Minions: Cost-Efficient Local-Cloud LLM Collaboration](https://github.com/HazyResearch/minions)
|
||||
- [TEE for Confidential AI Inference](https://openreview.net/forum?id=ey87M5iKcX) ([PDF](https://openreview.net/pdf?id=ey87M5iKcX))
|
||||
|
||||
#### Where you can help
|
||||
|
||||
| Item | Maturity | Details |
|
||||
|------|----------|---------|
|
||||
| Query complexity analyzer | **Ready** | Classify incoming queries by difficulty to decide local vs. cloud routing. Extends the existing `MultiEngine` routing logic. |
|
||||
| Cost tracking per-query | **Ready** | `CloudEngine` already has pricing data. Surface per-query cost in traces and telemetry dashboards. **Good first issue.** |
|
||||
| Redaction-before-cloud pipeline | **Ready** | Wire the existing `GuardrailsEngine` in REDACT mode as a mandatory pre-step before any cloud transmission. |
|
||||
| Minion protocol (sequential) | **Design Needed** | Local model extracts and summarizes long context → cloud model reasons over the compressed result. Native reimplementation of the core [Minions](https://github.com/HazyResearch/minions) idea. |
|
||||
| Minion protocol (parallel) | **Design Needed** | Local and cloud models work simultaneously on different aspects of a query; results are merged. Requires a new `HybridInferenceEngine` abstraction. |
|
||||
| TEE attestation verification | **Design Needed** | Verify that cloud inference ran inside a trusted execution environment via cryptographic attestation. |
|
||||
| Taint tracking across local/cloud boundary | **Design Needed** | The `TaintSet` already tracks PII/Secret labels. Add routing enforcement so tainted data only routes to attested TEE endpoints. |
|
||||
| Speculative decoding (local draft + cloud verify) | **Research-Stage** | Local model generates candidate tokens; cloud model validates in parallel for latency reduction. |
|
||||
|
||||
---
|
||||
|
||||
### Workstream 4: Tutorials & Documentation
|
||||
|
||||
OpenJarvis has reference docs and four tutorials, but critical gaps remain in continuous agents, LM evaluation, learning approaches, and custom tools. Video tutorials are scoped as a contributor opportunity — written tutorials come first, with video scripts included so anyone can record.
|
||||
|
||||
#### Where you can help
|
||||
|
||||
| Item | Maturity | Details |
|
||||
|------|----------|---------|
|
||||
| "Building Continuous Agents" tutorial | **Ready** | Writing an operator TOML manifest, activating it, session persistence across ticks, daemon mode. Example: a research operator that monitors arxiv daily. |
|
||||
| "Adding Custom Tools" tutorial | **Ready** | Implementing `BaseTool`, registering via `ToolRegistry`, wiring into agents. Example: a weather API tool. **Good first issue.** |
|
||||
| "Testing & Comparing LMs" tutorial | **Ready** | Running benchmarks, comparing local vs. cloud models, interpreting telemetry (latency, cost, energy per token). Uses the existing `bench/` framework. |
|
||||
| Per-platform installation guides | **Ready** | Expand `installation.md` with platform-specific walkthroughs: macOS + Ollama, Ubuntu + NVIDIA + vLLM, Windows + Ollama, Raspberry Pi. **Good first issue.** |
|
||||
| "Learning & Model Selection" tutorial | **Design Needed** | Router policies (heuristic, learned, GRPO), proposed approaches like Thompson Sampling, trace-based reward signals. |
|
||||
| Video tutorial infrastructure | **Design Needed** | Establish recording workflow, hosting (YouTube), MkDocs embedding. Write video scripts alongside written tutorials. |
|
||||
| Interactive Jupyter notebook tutorials | **Design Needed** | Notebook versions of key tutorials for exploratory, cell-by-cell learning. |
|
||||
|
||||
---
|
||||
|
||||
### Workstream 5: Hardware Breadth
|
||||
|
||||
Personal AI means running on the hardware people actually own. Each new hardware target expands who can use OpenJarvis and generates data for the research agenda (energy, cost, latency tradeoffs across silicon).
|
||||
|
||||
Adding a new hardware target involves up to four components: hardware detection in `core/config.py`, an inference engine adapter in `engine/`, an energy monitor in `telemetry/`, and an entry in the GPU specs database in `telemetry/gpu_monitor.py`.
|
||||
|
||||
#### Where you can help
|
||||
|
||||
| Item | Maturity | Details |
|
||||
|------|----------|---------|
|
||||
| AMD Ryzen AI iGPU path | **Ready** | Strix Point RDNA 3.5 iGPU handles 7-8B via Vulkan. llama.cpp Vulkan backend works today. Needs hardware detection and energy monitor. **Good first issue.** |
|
||||
| GPU specs database expansion | **Ready** | Add Intel Arc, Jetson Orin, Snapdragon specs to `GPU_SPECS` in `telemetry/gpu_monitor.py` (TFLOPS, bandwidth, TDP). **Good first issue.** |
|
||||
| Intel Arc GPU (B580/B570) | **Design Needed** | 12GB VRAM, ~$250 consumer GPU. Viable for 7-8B models. Engine path: IPEX-LLM or llama.cpp SYCL backend. |
|
||||
| NVIDIA Jetson Orin | **Design Needed** | Best-in-class edge device. Orin NX 16GB handles 7-8B models at 15-25 tok/s. Needs hardware detection, energy monitor (tegrastats), deployment guide. |
|
||||
| Qualcomm Snapdragon X Elite NPU | **Design Needed** | 45 TOPS, Windows Arm laptops. ONNX Runtime + QNN Execution Provider is the viable path. |
|
||||
| Intel Lunar Lake NPU via OpenVINO | **Design Needed** | 48 TOPS — most mature NPU software stack for x86 laptops. New engine wrapping OpenVINO GenAI. |
|
||||
| Raspberry Pi 5 | **Design Needed** | CPU-only via llama.cpp ARM NEON for 1-3B models. $100 entry point for hobbyists. |
|
||||
| Unified hardware benchmark suite | **Design Needed** | Standardized benchmark that runs the same workloads across all supported hardware, producing comparable energy/latency/throughput/cost numbers. |
|
||||
|
||||
@@ -140,7 +140,7 @@ max_tokens = 1024
|
||||
| `checkpoint_path` | string | `""` | Path to a fine-tuned checkpoint or LoRA adapter directory. |
|
||||
| `quantization` | string | `"none"` | Quantization format. Accepted values: `none`, `fp8`, `int8`, `int4`, `gguf_q4`, `gguf_q8`. |
|
||||
| `preferred_engine` | string | `""` | Override engine for this model (e.g., `"vllm"`). Takes priority over `engine.default`. |
|
||||
| `provider` | string | `""` | Model provider hint: `local`, `openai`, `anthropic`, `google`. Used by the Cloud engine to route API calls. |
|
||||
| `provider` | string | `""` | Model provider hint: `local`, `openai`, `anthropic`, `google`, `minimax`. Used by the Cloud engine to route API calls. |
|
||||
|
||||
**Generation default fields** (overridable per-call):
|
||||
|
||||
@@ -972,6 +972,7 @@ OpenJarvis respects the following environment variables:
|
||||
| `OPENAI_API_KEY` | API key for OpenAI cloud inference. Required for the `cloud` engine with OpenAI models. |
|
||||
| `ANTHROPIC_API_KEY` | API key for Anthropic cloud inference. Required for the `cloud` engine with Claude models. |
|
||||
| `GOOGLE_API_KEY` | API key for Google Gemini inference. Required for the `google` engine. |
|
||||
| `MINIMAX_API_KEY` | API key for MiniMax cloud inference. Required for the `cloud` engine with MiniMax models (MiniMax-M2.7, MiniMax-M2.7-highspeed, MiniMax-M2.5, MiniMax-M2.5-highspeed). |
|
||||
| `TAVILY_API_KEY` | API key for the Tavily web search tool. Required for the `web_search` tool. |
|
||||
|
||||
## Next Steps
|
||||
|
||||
@@ -42,9 +42,14 @@ If you prefer to run each step yourself:
|
||||
git clone https://github.com/open-jarvis/OpenJarvis.git
|
||||
cd OpenJarvis
|
||||
uv sync --extra server
|
||||
uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
|
||||
cd frontend && npm install && cd ..
|
||||
```
|
||||
|
||||
!!! note "Prerequisites"
|
||||
Requires [Rust](https://rustup.rs/) (`curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`).
|
||||
On Python 3.14+, set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command.
|
||||
|
||||
=== "Step 2: Start Ollama"
|
||||
|
||||
```bash
|
||||
@@ -131,8 +136,11 @@ programmatically. Every feature is accessible from the terminal.
|
||||
git clone https://github.com/open-jarvis/OpenJarvis.git
|
||||
cd OpenJarvis
|
||||
uv sync
|
||||
uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
|
||||
```
|
||||
|
||||
Requires [Rust](https://rustup.rs/). On Python 3.14+, set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
@@ -172,8 +180,11 @@ For programmatic access, the `Jarvis` class provides a high-level sync API.
|
||||
git clone https://github.com/open-jarvis/OpenJarvis.git
|
||||
cd OpenJarvis
|
||||
uv sync
|
||||
uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
|
||||
```
|
||||
|
||||
Requires [Rust](https://rustup.rs/). On Python 3.14+, set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` before the `maturin` command.
|
||||
|
||||
### Quick example
|
||||
|
||||
```python
|
||||
|
||||
@@ -5,6 +5,26 @@
|
||||
var SUPABASE_ANON_KEY =
|
||||
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZSIsInJlZiI6Im10YnRncHd6cmJvc3R3ZWFhbnByIiwicm9sZSI6ImFub24iLCJpYXQiOjE3NzMxODk0OTQsImV4cCI6MjA4ODc2NTQ5NH0._xMlqCfljtXpwPj54H-ghxfLFO-jiq4W2WhpU8vVL1c";
|
||||
|
||||
var PAGE_SIZE = 50;
|
||||
var allRows = [];
|
||||
var currentPage = 0;
|
||||
|
||||
// No-KV-cache formula: FLOPs = params_b * 1e9 * N * (N+1)
|
||||
// Reference values only (GPT-5.3 constants) — recompute functions below
|
||||
// are defined but not called; the leaderboard displays database values directly.
|
||||
var DEFAULT_PARAMS_B = 137;
|
||||
var ENERGY_WH_PER_FLOP = 0.4 / (1000 * 3e12);
|
||||
|
||||
function recomputeFlopsNoKvCache(totalTokens) {
|
||||
var n = Number(totalTokens) || 0;
|
||||
if (n <= 0) return 0;
|
||||
return DEFAULT_PARAMS_B * 1e9 * n * (n + 1);
|
||||
}
|
||||
|
||||
function recomputeEnergyNoKvCache(flops) {
|
||||
return flops * ENERGY_WH_PER_FLOP;
|
||||
}
|
||||
|
||||
function escapeHtml(s) {
|
||||
var el = document.createElement("span");
|
||||
el.textContent = s;
|
||||
@@ -19,6 +39,67 @@
|
||||
return n.toLocaleString();
|
||||
}
|
||||
|
||||
function totalPages() {
|
||||
return Math.max(1, Math.ceil(allRows.length / PAGE_SIZE));
|
||||
}
|
||||
|
||||
function renderPage() {
|
||||
var tbody = document.getElementById("leaderboard-body");
|
||||
if (!tbody) return;
|
||||
|
||||
var start = currentPage * PAGE_SIZE;
|
||||
var end = Math.min(start + PAGE_SIZE, allRows.length);
|
||||
var pageRows = allRows.slice(start, end);
|
||||
|
||||
var html = "";
|
||||
for (var j = 0; j < pageRows.length; j++) {
|
||||
var rank = start + j + 1;
|
||||
var rankClass = rank <= 3 ? " lb-rank-" + rank : "";
|
||||
var medal =
|
||||
rank === 1 ? "\uD83E\uDD47" : rank === 2 ? "\uD83E\uDD48" : rank === 3 ? "\uD83E\uDD49" : "";
|
||||
var row = pageRows[j];
|
||||
html +=
|
||||
"<tr>" +
|
||||
'<td><span class="lb-rank' + rankClass + '">' + (medal || rank) + "</span></td>" +
|
||||
'<td class="lb-name">' + escapeHtml(row.display_name) + "</td>" +
|
||||
'<td class="lb-number">$' + Number(row.dollar_savings || 0).toFixed(4) + "</td>" +
|
||||
'<td class="lb-number">' + Number(row.energy_wh_saved || 0).toFixed(2) + "</td>" +
|
||||
'<td class="lb-number">' + fmtLarge(Number(row.flops_saved || 0)) + "</td>" +
|
||||
'<td class="lb-number">' + Number(row.total_calls || 0).toLocaleString() + "</td>" +
|
||||
'<td class="lb-number">' + Number(row.total_tokens || 0).toLocaleString() + "</td>" +
|
||||
"</tr>";
|
||||
}
|
||||
tbody.innerHTML = html;
|
||||
|
||||
renderPagination();
|
||||
}
|
||||
|
||||
function renderPagination() {
|
||||
var container = document.getElementById("leaderboard-pagination");
|
||||
if (!container) return;
|
||||
|
||||
var pages = totalPages();
|
||||
if (pages <= 1) {
|
||||
container.innerHTML = "";
|
||||
return;
|
||||
}
|
||||
|
||||
var prevDisabled = currentPage === 0;
|
||||
var nextDisabled = currentPage >= pages - 1;
|
||||
|
||||
container.innerHTML =
|
||||
'<button class="lb-page-btn"' + (prevDisabled ? " disabled" : "") + ' id="lb-prev">' +
|
||||
"\u2190 Prev</button>" +
|
||||
'<span class="lb-page-info">Page ' + (currentPage + 1) + " of " + pages + "</span>" +
|
||||
'<button class="lb-page-btn"' + (nextDisabled ? " disabled" : "") + ' id="lb-next">' +
|
||||
"Next \u2192</button>";
|
||||
|
||||
var prevBtn = document.getElementById("lb-prev");
|
||||
var nextBtn = document.getElementById("lb-next");
|
||||
if (prevBtn) prevBtn.onclick = function () { if (currentPage > 0) { currentPage--; renderPage(); } };
|
||||
if (nextBtn) nextBtn.onclick = function () { if (currentPage < pages - 1) { currentPage++; renderPage(); } };
|
||||
}
|
||||
|
||||
function loadLeaderboard() {
|
||||
var tbody = document.getElementById("leaderboard-body");
|
||||
if (!tbody) return;
|
||||
@@ -32,7 +113,7 @@
|
||||
|
||||
fetch(
|
||||
SUPABASE_URL +
|
||||
"/rest/v1/savings_entries?select=display_name,dollar_savings,energy_wh_saved,flops_saved,total_calls,total_tokens&order=dollar_savings.desc&limit=100",
|
||||
"/rest/v1/savings_entries?select=display_name,dollar_savings,energy_wh_saved,flops_saved,total_calls,total_tokens&order=dollar_savings.desc&limit=1000",
|
||||
{
|
||||
headers: {
|
||||
apikey: SUPABASE_ANON_KEY,
|
||||
@@ -52,6 +133,9 @@
|
||||
return;
|
||||
}
|
||||
|
||||
allRows = rows;
|
||||
currentPage = 0;
|
||||
|
||||
var totalMembers = rows.length;
|
||||
var totalDollars = 0;
|
||||
var totalRequests = 0;
|
||||
@@ -72,25 +156,7 @@
|
||||
if (elRequests) elRequests.textContent = totalRequests.toLocaleString();
|
||||
if (elTokens) elTokens.textContent = fmtLarge(totalTokens);
|
||||
|
||||
var html = "";
|
||||
for (var j = 0; j < rows.length; j++) {
|
||||
var rank = j + 1;
|
||||
var rankClass = rank <= 3 ? " lb-rank-" + rank : "";
|
||||
var medal =
|
||||
rank === 1 ? "\uD83E\uDD47" : rank === 2 ? "\uD83E\uDD48" : rank === 3 ? "\uD83E\uDD49" : "";
|
||||
var row = rows[j];
|
||||
html +=
|
||||
"<tr>" +
|
||||
'<td><span class="lb-rank' + rankClass + '">' + (medal || rank) + "</span></td>" +
|
||||
'<td class="lb-name">' + escapeHtml(row.display_name) + "</td>" +
|
||||
'<td class="lb-number">$' + Number(row.dollar_savings || 0).toFixed(4) + "</td>" +
|
||||
'<td class="lb-number">' + Number(row.energy_wh_saved || 0).toFixed(2) + "</td>" +
|
||||
'<td class="lb-number">' + fmtLarge(Number(row.flops_saved || 0)) + "</td>" +
|
||||
'<td class="lb-number">' + Number(row.total_calls || 0).toLocaleString() + "</td>" +
|
||||
'<td class="lb-number">' + Number(row.total_tokens || 0).toLocaleString() + "</td>" +
|
||||
"</tr>";
|
||||
}
|
||||
tbody.innerHTML = html;
|
||||
renderPage();
|
||||
})
|
||||
.catch(function (err) {
|
||||
tbody.innerHTML =
|
||||
@@ -101,7 +167,6 @@
|
||||
});
|
||||
}
|
||||
|
||||
// Run on page load and refresh every 60s
|
||||
if (document.getElementById("leaderboard-body")) {
|
||||
loadLeaderboard();
|
||||
setInterval(loadLeaderboard, 60000);
|
||||
|
||||
@@ -52,6 +52,8 @@ See how the OpenJarvis community saves money, energy, and compute by running AI
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div id="leaderboard-pagination" class="lb-pagination"></div>
|
||||
|
||||
<p style="font-size:12px;opacity:0.6;margin-top:12px">
|
||||
*Dollar savings estimates assume local open-source models (e.g. Qwen, Nemotron, Kimi) produce roughly the same number of tokens per request, on average, as closed-source cloud models.
|
||||
</p>
|
||||
|
||||
@@ -491,6 +491,41 @@
|
||||
0 1px 2px 1px rgba(0, 0, 0, 0.3);
|
||||
}
|
||||
|
||||
/* Leaderboard pagination */
|
||||
.lb-pagination {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 16px;
|
||||
margin: 20px 0 8px;
|
||||
}
|
||||
|
||||
.lb-page-btn {
|
||||
padding: 6px 16px;
|
||||
border-radius: 6px;
|
||||
border: 1px solid var(--md-default-fg-color--lighter, #ccc);
|
||||
background: var(--md-code-bg-color, #f5f5f5);
|
||||
color: var(--md-default-fg-color, #333);
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
cursor: pointer;
|
||||
transition: opacity 0.15s;
|
||||
}
|
||||
|
||||
.lb-page-btn:hover:not([disabled]) {
|
||||
opacity: 0.8;
|
||||
}
|
||||
|
||||
.lb-page-btn[disabled] {
|
||||
opacity: 0.35;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.lb-page-info {
|
||||
font-size: 13px;
|
||||
color: var(--md-default-fg-color--light, #666);
|
||||
}
|
||||
|
||||
/* Responsive: hide placeholder on mobile */
|
||||
@media (max-width: 768px) {
|
||||
.DocSearch-Button-Placeholder {
|
||||
|
||||
@@ -621,3 +621,49 @@ All agents publish events on the `EventBus` when a bus is provided:
|
||||
`INFERENCE_START` / `INFERENCE_END` events are published by the `InstrumentedEngine` wrapper, not by agents directly. This keeps telemetry opt-in and transparent to agent code.
|
||||
|
||||
These events enable the telemetry and trace systems to record detailed interaction data automatically.
|
||||
|
||||
---
|
||||
|
||||
## Managed Agent Streaming
|
||||
|
||||
The Managed Agent API (`/v1/managed-agents/{id}/messages`) supports real-time SSE streaming. Send a message with `stream: true` to receive the agent's response as a Server-Sent Events stream instead of the default asynchronous queue mode.
|
||||
|
||||
### Streaming Messages
|
||||
|
||||
```bash
|
||||
curl -N -X POST http://localhost:8000/v1/managed-agents/{id}/messages \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"content": "What is 2+2?", "stream": true}'
|
||||
```
|
||||
|
||||
The response follows the OpenAI SSE format:
|
||||
|
||||
1. **Content chunks** -- `data: {"choices": [{"delta": {"content": "token"}}]}`
|
||||
2. **Tool results** (if the agent used tools) -- `event: tool_results\ndata: {"results": [...]}`
|
||||
3. **Final chunk** -- `data: {"choices": [{"delta": {}, "finish_reason": "stop"}]}`
|
||||
4. **Done sentinel** -- `data: [DONE]`
|
||||
|
||||
When `stream: false` (the default), the endpoint behaves exactly as before -- the message is queued and the agent must be triggered separately via `/run`.
|
||||
|
||||
### Behavior Details
|
||||
|
||||
- The user message is always stored in the database before the agent runs.
|
||||
- After streaming completes, the full agent response is persisted as an `agent_to_user` message.
|
||||
- The agent is instantiated from the managed agent's stored `agent_type` and `config`.
|
||||
- Conversation history from prior messages is automatically loaded as context.
|
||||
- If the engine is not available on the server, a `503` error is returned.
|
||||
|
||||
### Python Example
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
with httpx.stream(
|
||||
"POST",
|
||||
"http://localhost:8000/v1/managed-agents/{id}/messages",
|
||||
json={"content": "Summarize today's news", "stream": True},
|
||||
) as response:
|
||||
for line in response.iter_lines():
|
||||
if line.startswith("data:") and "[DONE]" not in line:
|
||||
print(line[5:].strip())
|
||||
```
|
||||
|
||||
@@ -88,6 +88,7 @@ export default function App() {
|
||||
dollar_savings: dollarSavings,
|
||||
energy_wh_saved: energySaved,
|
||||
flops_saved: flopsSaved,
|
||||
token_counting_version: data.token_counting_version ?? 1,
|
||||
});
|
||||
}
|
||||
})
|
||||
|
||||
@@ -18,6 +18,8 @@ export function InputArea() {
|
||||
const streamState = useAppStore((s) => s.streamState);
|
||||
const messages = useAppStore((s) => s.messages);
|
||||
const speechEnabled = useAppStore((s) => s.settings.speechEnabled);
|
||||
const maxTokens = useAppStore((s) => s.settings.maxTokens);
|
||||
const temperature = useAppStore((s) => s.settings.temperature);
|
||||
const createConversation = useAppStore((s) => s.createConversation);
|
||||
const addMessage = useAppStore((s) => s.addMessage);
|
||||
const updateLastAssistant = useAppStore((s) => s.updateLastAssistant);
|
||||
@@ -127,6 +129,7 @@ export function InputArea() {
|
||||
|
||||
let accumulatedContent = '';
|
||||
let usage: TokenUsage | undefined;
|
||||
let complexity: { score: number; tier: string; suggested_max_tokens: number } | undefined;
|
||||
const toolCalls: ToolCallInfo[] = [];
|
||||
let lastFlush = 0;
|
||||
let ttftMs: number | undefined;
|
||||
@@ -147,7 +150,7 @@ export function InputArea() {
|
||||
|
||||
try {
|
||||
for await (const sseEvent of streamChat(
|
||||
{ model: selectedModel, messages: apiMessages, stream: true },
|
||||
{ model: selectedModel, messages: apiMessages, stream: true, temperature, max_tokens: maxTokens },
|
||||
controller.signal,
|
||||
)) {
|
||||
const eventName = sseEvent.event;
|
||||
@@ -202,6 +205,7 @@ export function InputArea() {
|
||||
const data = JSON.parse(sseEvent.data);
|
||||
const delta = data.choices?.[0]?.delta;
|
||||
if (data.usage) usage = data.usage;
|
||||
if (data.complexity) complexity = data.complexity;
|
||||
if (delta?.content) {
|
||||
if (!ttftMs) ttftMs = Date.now() - startTime;
|
||||
accumulatedContent += delta.content;
|
||||
@@ -248,6 +252,9 @@ export function InputArea() {
|
||||
tokens_per_sec: usage?.completion_tokens
|
||||
? usage.completion_tokens / (totalMs / 1000)
|
||||
: undefined,
|
||||
complexity_score: complexity?.score,
|
||||
complexity_tier: complexity?.tier,
|
||||
suggested_max_tokens: complexity?.suggested_max_tokens,
|
||||
};
|
||||
updateLastAssistant(
|
||||
convId,
|
||||
|
||||
@@ -18,6 +18,7 @@ export function XRayFooter({ usage, telemetry }: Props) {
|
||||
const parts: string[] = [];
|
||||
if (telemetry?.engine) parts.push(telemetry.engine);
|
||||
if (telemetry?.model_id) parts.push(telemetry.model_id);
|
||||
if (telemetry?.complexity_tier) parts.push(telemetry.complexity_tier);
|
||||
if (telemetry?.total_ms) parts.push(formatMs(telemetry.total_ms));
|
||||
if (usage && (usage.prompt_tokens || usage.completion_tokens)) {
|
||||
parts.push(`${usage.prompt_tokens} input tokens`);
|
||||
@@ -50,6 +51,15 @@ export function XRayFooter({ usage, telemetry }: Props) {
|
||||
}
|
||||
rows.push({ label: 'Tokens', value: tokenParts.join(' \u00B7 ') });
|
||||
}
|
||||
if (telemetry?.complexity_tier) {
|
||||
rows.push({
|
||||
label: 'Complexity',
|
||||
value: `${telemetry.complexity_tier} (${telemetry.complexity_score?.toFixed(2)})`,
|
||||
});
|
||||
}
|
||||
if (telemetry?.suggested_max_tokens) {
|
||||
rows.push({ label: 'Token budget', value: `${telemetry.suggested_max_tokens}` });
|
||||
}
|
||||
if (telemetry?.tokens_per_sec) {
|
||||
rows.push({ label: 'Speed', value: `${Math.round(telemetry.tokens_per_sec)} tok/s` });
|
||||
}
|
||||
|
||||
@@ -589,6 +589,7 @@ export interface SavingsSubmission {
|
||||
dollar_savings: number;
|
||||
energy_wh_saved: number;
|
||||
flops_saved: number;
|
||||
token_counting_version?: number;
|
||||
}
|
||||
|
||||
export async function submitSavings(data: SavingsSubmission): Promise<boolean> {
|
||||
|
||||
@@ -5,6 +5,8 @@ export interface ChatRequest {
|
||||
model: string;
|
||||
messages: Array<{ role: string; content: string }>;
|
||||
stream: true;
|
||||
temperature?: number;
|
||||
max_tokens?: number;
|
||||
}
|
||||
|
||||
export async function* streamChat(
|
||||
|
||||
@@ -56,6 +56,9 @@ export interface MessageTelemetry {
|
||||
tokens_per_sec?: number;
|
||||
ttft_ms?: number;
|
||||
total_ms?: number;
|
||||
complexity_score?: number;
|
||||
complexity_tier?: string;
|
||||
suggested_max_tokens?: number;
|
||||
}
|
||||
|
||||
export interface ChatMessage {
|
||||
@@ -120,6 +123,7 @@ export interface SavingsData {
|
||||
total_tokens: number;
|
||||
local_cost: number;
|
||||
per_provider: ProviderSavings[];
|
||||
token_counting_version?: number;
|
||||
}
|
||||
|
||||
export interface ServerInfo {
|
||||
|
||||
@@ -0,0 +1,417 @@
|
||||
# Agent Runtime Manual Test Plan
|
||||
|
||||
**Branch:** `main`
|
||||
**PR Reference:** [#32](https://github.com/open-jarvis/OpenJarvis/pull/32)
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
git checkout main && git pull
|
||||
uv sync --extra dev
|
||||
```
|
||||
|
||||
Create `~/.openjarvis/config.toml`:
|
||||
|
||||
```toml
|
||||
[engine]
|
||||
type = "cloud"
|
||||
|
||||
[intelligence]
|
||||
default_model = "Qwen/Qwen3.5-35B-A3B"
|
||||
|
||||
[engine.cloud]
|
||||
provider = "openai"
|
||||
api_key = "sk-..."
|
||||
```
|
||||
|
||||
For every test case, record: **Pass / Fail / Partial / Blocked**, what you actually saw, and screenshots for any UI issues.
|
||||
|
||||
---
|
||||
|
||||
## Part 1: CLI (`jarvis agents`)
|
||||
|
||||
### 1.1 Commands exist
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 1 | `jarvis agents --help` | Shows all subcommands: `launch`, `start`, `stop`, `run`, `status`, `logs`, `daemon`, `watch`, `recover`, `errors`, `ask`, `instruct`, `messages`, `list`, `info`, `create`, `pause`, `resume`, `delete`, `bind`, `channels`, `search`, `templates`, `tasks` |
|
||||
|
||||
### 1.2 Agent lifecycle: create → run → pause → resume → delete
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 2 | `jarvis agents launch` | Wizard: template list → name/schedule/tools/budget/learning prompts → creates agent, prints ID |
|
||||
| 3 | `jarvis agents list` | Agent appears, status=`idle` |
|
||||
| 4 | `jarvis agents status` | Table: name, status dot, schedule, last run, runs=0, cost=$0 |
|
||||
| 5 | `jarvis agents run <id>` | Prints progress then "Tick complete. Status: idle, runs: 1" |
|
||||
| 6 | `jarvis agents status` | runs=1, last run time updated |
|
||||
| 7 | `jarvis agents pause <id>` then `status` | Status shows `paused` |
|
||||
| 8 | `jarvis agents resume <id>` then `status` | Status back to `idle` |
|
||||
| 9 | `jarvis agents delete <id>` then `list` | Agent gone (soft-deleted/archived, not in list) |
|
||||
|
||||
### 1.3 Agent creation variants
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 10 | `jarvis agents create "Test Agent"` | Creates agent by name, prints ID |
|
||||
| 11 | `jarvis agents create --template <template_name>` | Creates from template, inherits template config |
|
||||
| 12 | `jarvis agents launch` → pick a template | Wizard pre-fills config from template |
|
||||
| 13 | `jarvis agents launch` → pick "Custom Agent" | Wizard starts with blank config |
|
||||
| 14 | `jarvis agents templates` | Lists built-in + user templates with descriptions |
|
||||
|
||||
### 1.4 Scheduling
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 15 | Create agent with `schedule_type=interval`, `schedule_value=30` | Created |
|
||||
| 16 | `jarvis agents start <id>` | "Agent registered with scheduler" |
|
||||
| 17 | `jarvis agents stop <id>` | "Agent deregistered from scheduler" |
|
||||
| 18 | `jarvis agents daemon` | Starts, prints agent count, blocks. Ctrl+C → "Daemon stopped." clean exit |
|
||||
| 19 | Create agent with `schedule_type=cron`, `schedule_value="*/5 * * * *"` | Created |
|
||||
| 20 | `jarvis agents start <id>` (cron agent) | Registered, next fire time displayed or logged |
|
||||
| 21 | Create agent with `schedule_type=manual` then `start <id>` | Agent registered but never auto-fires |
|
||||
|
||||
### 1.5 Interaction: ask / instruct / messages
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 22 | `jarvis agents ask <id> "What is 2+2?"` | Runs tick, prints agent response inline |
|
||||
| 23 | `jarvis agents messages <id>` | Shows user→agent ask + agent→user response |
|
||||
| 24 | `jarvis agents instruct <id> "Focus on ML papers"` | "Instruction queued for next tick" |
|
||||
| 25 | `jarvis agents messages <id>` | Queued instruction shows `[queued]`, status=pending |
|
||||
| 26 | `jarvis agents run <id>` then `messages <id>` | Queued message now delivered, status changes from pending |
|
||||
| 27 | `jarvis agents ask <id> ""` (empty message) | Graceful error or rejection, no crash |
|
||||
| 28 | `jarvis agents instruct <id>` with very long message (>1000 chars) | Accepted and stored correctly |
|
||||
|
||||
### 1.6 Error recovery & monitoring
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 29 | `jarvis agents errors` | Lists agents in error/needs_attention/stalled/budget_exceeded (or empty table) |
|
||||
| 30 | `jarvis agents recover <id>` (on errored agent) | Restores checkpoint, status → `idle` |
|
||||
| 31 | `jarvis agents recover <id>` (on idle agent) | Clear message: "Agent is not in error state" or similar |
|
||||
| 32 | `jarvis agents logs <id>` | Recent traces with tick IDs and timestamps |
|
||||
| 33 | `jarvis agents logs <nonexistent_id>` | Clear error: "Agent not found" |
|
||||
| 34 | `jarvis agents watch` (then run a tick in another terminal) | Events stream live: AGENT_TICK_START, AGENT_TICK_END visible. Ctrl+C to stop. |
|
||||
| 35 | `jarvis agents watch <id>` | Same, filtered to one agent only |
|
||||
| 36 | `jarvis agents watch` then Ctrl+C | Clean exit, no traceback, no hanging threads |
|
||||
|
||||
### 1.7 Agent info & inspection
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 37 | `jarvis agents info <id>` | Shows agent type, status, memory snippet, tasks, channels, config details |
|
||||
| 38 | `jarvis agents tasks <id>` | Lists tasks with statuses (or empty state) |
|
||||
| 39 | `jarvis agents channels <id>` | Lists channel bindings (or empty state) |
|
||||
| 40 | `jarvis agents search "keyword"` | Searches across agent traces, returns relevant results |
|
||||
|
||||
### 1.8 Edge cases & invalid input
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 41 | `jarvis agents run <nonexistent_id>` | Clear error: "Agent not found" — no Python traceback |
|
||||
| 42 | `jarvis agents pause <id>` twice | Second pause is no-op or clear message, no crash |
|
||||
| 43 | `jarvis agents resume <id>` (already idle) | No-op or clear message, no crash |
|
||||
| 44 | `jarvis agents run <id>` while another tick is running | Concurrency guard: "Agent is already running" error |
|
||||
| 45 | `jarvis agents delete <id>` then `run <id>` | Clear error about deleted/archived agent |
|
||||
| 46 | Create agent with invalid cron expression | Rejected with clear validation error |
|
||||
| 47 | Create agent with negative budget | Rejected or clamped to 0 |
|
||||
|
||||
### 1.9 CLI aesthetics
|
||||
|
||||
| # | Check | Expected |
|
||||
|---|-------|----------|
|
||||
| 48 | `status` table formatting | Columns aligned, readable at 80-char terminal width |
|
||||
| 49 | Error messages (run with no engine configured) | Clear human-readable message, no Python tracebacks |
|
||||
| 50 | `launch` wizard prompts | Clear labels, sensible defaults, no confusing jargon |
|
||||
| 51 | `watch` event stream | Color-coded, event type + agent name visible, timestamps |
|
||||
| 52 | `list` table with 0 agents | "No agents found" or empty table — not a crash |
|
||||
| 53 | `list` table with 10+ agents | Table remains readable, no column overflow |
|
||||
| 54 | All commands with `--help` | Every subcommand has a help string |
|
||||
|
||||
---
|
||||
|
||||
## Part 2: Web Frontend
|
||||
|
||||
### 2.0 Setup
|
||||
|
||||
```bash
|
||||
# Terminal 1 # Terminal 2
|
||||
uv run jarvis serve cd frontend && npm install && npm run dev
|
||||
```
|
||||
|
||||
Open http://localhost:5173, navigate to **Agents** page via sidebar.
|
||||
|
||||
### 2.1 Navigation & routing
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 55 | Click "Agents" in sidebar | AgentsPage renders, URL is `/agents` |
|
||||
| 56 | Direct navigation to `/agents` | Page loads correctly (no blank screen) |
|
||||
| 57 | Browser back/forward after visiting agent detail | Navigation works, state preserved |
|
||||
|
||||
### 2.2 List view
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 58 | Page loads with backend running | No console errors, agent list renders |
|
||||
| 59 | Page loads with backend **down** | User-visible error message (not blank white screen), no console exceptions |
|
||||
| 60 | Agent cards | Name, color status dot, schedule description, last run time, runs count, cost |
|
||||
| 61 | "Run Now" button | Triggers tick, card updates (runs count increments, last run time updates) |
|
||||
| 62 | Pause/Resume button | Toggles status, dot color changes immediately |
|
||||
| 63 | Agent list auto-refresh | After running a tick via CLI, the web list eventually reflects the updated state |
|
||||
| 64 | 10+ agents in list | Cards render without performance issues, scroll works |
|
||||
|
||||
### 2.3 Launch wizard
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 65 | Click "Launch Agent" | Modal appears: Step 1 template picker with templates + "Custom Agent" option |
|
||||
| 66 | Templates load from API | Template cards display with names and descriptions |
|
||||
| 67 | Select template → Next → Step 2 | Config form: name (pre-filled from template), schedule_type dropdown, schedule_value, tools checkboxes, budget, learning toggle (off) |
|
||||
| 68 | Select "Custom Agent" → Next → Step 2 | Config form with blank name, no pre-filled values |
|
||||
| 69 | Next → Step 3 | Review summary of all config values |
|
||||
| 70 | Click Launch | Agent created, modal closes, new agent appears in list |
|
||||
| 71 | Back button at Step 2 | Returns to Step 1, template selection preserved |
|
||||
| 72 | Back button at Step 3 | Returns to Step 2, all form inputs preserved |
|
||||
| 73 | Launch with empty name | Inline error: "Agent name is required" — modal stays open |
|
||||
| 74 | Launch with all tools selected | All tools included in review and in created agent config |
|
||||
| 75 | Click outside modal / press Escape | Modal closes (or stays open — document behavior) |
|
||||
| 76 | Schedule type = "Manual" | schedule_value input is disabled/hidden |
|
||||
| 77 | Schedule type = "Cron" | schedule_value placeholder shows cron example |
|
||||
| 78 | Schedule type = "Interval" | schedule_value placeholder shows seconds example |
|
||||
|
||||
### 2.4 Detail view (click an agent)
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 79 | Click agent card | Detail view opens with tabbed interface |
|
||||
| 80 | **Overview** tab | Stat cards (Total Runs, Success Rate, Total Cost), config display, channels list, action buttons |
|
||||
| 81 | **Overview** action buttons | Run Now, Pause, Resume visible and functional |
|
||||
| 82 | **Interact** tab | Chat message list, textarea, "Immediate" and "Queue" send buttons |
|
||||
| 83 | Send immediate message | Appears in chat with user styling, agent responds after tick |
|
||||
| 84 | Send queued message | Shows with "queued" badge, status=pending |
|
||||
| 85 | Send empty message | Button disabled or graceful rejection — no empty message sent |
|
||||
| 86 | Rapid-fire send (click Send multiple times quickly) | No duplicate messages, no race condition errors |
|
||||
| 87 | Chat auto-scroll | New messages scroll into view automatically |
|
||||
| 88 | **Tasks** tab | Task list with status badges (completed=green, failed=red, active=blue, pending=gray) |
|
||||
| 89 | **Tasks** tab (no tasks) | Empty state: "No tasks assigned." |
|
||||
| 90 | **Memory** tab | summary_memory text displayed in readable format |
|
||||
| 91 | **Memory** tab (no memory) | Empty state: "Agent has no stored memory yet." |
|
||||
| 92 | **Learning** tab | Toggle switch (read-only, off by default), placeholder text for future events |
|
||||
| 93 | **Logs** tab | Placeholder / empty state message (not a crash or blank) |
|
||||
| 94 | Tab switching — rapid clicks | All 6 tabs render instantly, no layout shift, no flash of wrong content |
|
||||
|
||||
### 2.5 Error states
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 95 | Agent in `error` status | Red status dot/badge, "Recover" button visible |
|
||||
| 96 | Click Recover | Status resets to `idle`, dot turns green |
|
||||
| 97 | Agent in `needs_attention` status | Amber badge visible |
|
||||
| 98 | Agent in `budget_exceeded` status | Orange badge visible |
|
||||
| 99 | Agent in `stalled` status | Yellow badge visible |
|
||||
| 100 | Backend goes down while page is open | Next refresh/action shows error — not silent failure |
|
||||
| 101 | Delete agent → confirm it disappears from list | Agent removed from list immediately (or on next refresh) |
|
||||
| 102 | Delete agent (no confirmation dialog in web) | **Document:** Is instant delete OK or should there be a confirm? |
|
||||
|
||||
### 2.6 Overflow menu
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 103 | Click "..." menu on agent card | Dropdown with Delete + other options |
|
||||
| 104 | Click Delete from menu | Agent deleted, list updates |
|
||||
| 105 | Click outside dropdown | Dropdown closes |
|
||||
|
||||
### 2.7 Web aesthetics & UX
|
||||
|
||||
| # | Check | Expected |
|
||||
|---|-------|----------|
|
||||
| 106 | Status dot colors | idle=#22c55e, running=#3b82f6, paused=#6b7280, error=#ef4444, needs_attention=#f59e0b, budget_exceeded=#f97316, stalled=#eab308 |
|
||||
| 107 | Launch wizard spacing/alignment | Modal centered, steps clearly numbered, form inputs aligned, no overlap |
|
||||
| 108 | Detail view tab switching | Instant, no layout shift or flash |
|
||||
| 109 | Interact tab chat feel | Messages visually distinct (user=right vs agent=left or different colors), auto-scroll, clear input area |
|
||||
| 110 | Responsive at 1024px width | No overflow or cut-off content, agent cards reflow |
|
||||
| 111 | Responsive at 1440px width | Proper use of space, no excessive stretching |
|
||||
| 112 | Responsive at 768px width (tablet) | Still usable, no broken layout |
|
||||
| 113 | Empty states | "No agents yet" + CTA button / "No messages" / "No tasks" — not blank white space |
|
||||
| 114 | Loading states | "Loading agents..." shown during fetch, spinner or skeleton |
|
||||
| 115 | Page title / browser tab | Meaningful title (not just "Vite App") |
|
||||
| 116 | Console errors | Zero console errors during normal usage flow |
|
||||
|
||||
---
|
||||
|
||||
## Part 3: Desktop App
|
||||
|
||||
### 3.0 Setup
|
||||
|
||||
```bash
|
||||
# Terminal 1 # Terminal 2
|
||||
uv run jarvis serve cd desktop && npm install && npm run tauri dev
|
||||
```
|
||||
|
||||
Navigate to the **Agents** tab.
|
||||
|
||||
### 3.1 Functionality
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 117 | Left panel: agent list | Status dots, schedule descriptions, last run times |
|
||||
| 118 | Click agent → right panel | Tabbed detail view (Overview, Interact, Tasks, Memory, Learning, Logs) |
|
||||
| 119 | No agent selected | Right panel shows "Select an agent to view details" |
|
||||
| 120 | "Launch Agent" button | Opens wizard, same 3-step flow as web |
|
||||
| 121 | Launch wizard → Create agent | Agent appears in left panel list |
|
||||
| 122 | **Overview** tab | Key-value stats (Status, Agent Type, Schedule, Last Run, Total Runs, Total Cost, Budget) + action buttons (Run Now, Pause, Resume, Recover) |
|
||||
| 123 | **Interact** tab | Chat UI, mode toggle (immediate/queued), Enter shortcut sends message |
|
||||
| 124 | Send immediate message | Response appears in chat |
|
||||
| 125 | Send queued message | Shows as pending |
|
||||
| 126 | **Tasks** tab | Task list with colored status badges + created-at timestamps |
|
||||
| 127 | **Memory** tab | summary_memory in monospace font |
|
||||
| 128 | **Learning** tab | Shows enabled/disabled status + placeholder text |
|
||||
| 129 | **Logs** tab | Placeholder: "Log streaming not yet connected." |
|
||||
| 130 | Auto-refresh | Agent list refreshes on ~10s interval (verify with CLI-triggered state change) |
|
||||
| 131 | Delete agent via desktop | Confirmation dialog appears, agent removed on confirm |
|
||||
|
||||
### 3.2 Desktop edge cases
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 132 | Backend not running → open desktop app | Error state shown, not a crash |
|
||||
| 133 | Backend dies while desktop is open | Graceful degradation on next action/refresh |
|
||||
| 134 | Selected agent deleted via CLI → desktop refreshes | Selected agent deselects, list updates |
|
||||
|
||||
### 3.3 Desktop aesthetics
|
||||
|
||||
| # | Check | Expected |
|
||||
|---|-------|----------|
|
||||
| 135 | Catppuccin color scheme consistent | idle=#a6e3a1, running=#89b4fa, paused=#6c7086, error=#f38ba8, needs_attention=#fab387, stalled=#f9e2af |
|
||||
| 136 | Left/right panel split | Resizable or fixed at reasonable ratio, no overlap |
|
||||
| 137 | Tab switching | Smooth, no flicker |
|
||||
| 138 | Launch wizard modal | Properly overlays content, dismissible with Escape or outside click |
|
||||
| 139 | Text readability | Font sizes consistent, sufficient contrast against dark background |
|
||||
| 140 | Window resize | Layout adapts, no overflow or clipping |
|
||||
| 141 | Status badge consistency with web | Same statuses map to same semantic colors (green=idle, blue=running, etc.) |
|
||||
|
||||
---
|
||||
|
||||
## Part 4: API Backend (Direct)
|
||||
|
||||
### 4.1 REST endpoint smoke tests
|
||||
|
||||
Run with `uv run jarvis serve` and test via curl or Postman.
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 142 | `GET /v1/managed-agents` | 200, returns `[]` or agent list JSON |
|
||||
| 143 | `POST /v1/managed-agents` with valid body | 200/201, returns created agent JSON with `id` |
|
||||
| 144 | `POST /v1/managed-agents` with empty body | 422 or 400 with validation error |
|
||||
| 145 | `GET /v1/managed-agents/<id>` | 200, returns single agent |
|
||||
| 146 | `GET /v1/managed-agents/<bad_id>` | 404, returns error JSON |
|
||||
| 147 | `POST /v1/managed-agents/<id>/run` | 200, tick executes |
|
||||
| 148 | `POST /v1/managed-agents/<id>/pause` | 200, status changes to paused |
|
||||
| 149 | `POST /v1/managed-agents/<id>/resume` | 200, status changes to idle |
|
||||
| 150 | `POST /v1/managed-agents/<id>/recover` | 200 if errored, appropriate error if not |
|
||||
| 151 | `DELETE /v1/managed-agents/<id>` | 200, agent archived |
|
||||
| 152 | `GET /v1/templates` | 200, returns template list |
|
||||
| 153 | `POST /v1/templates/<id>/instantiate` | 200, creates agent from template |
|
||||
| 154 | `GET /v1/agents/errors` | 200, returns list of problem agents |
|
||||
| 155 | `GET /v1/agents/health` | 200, returns health summary |
|
||||
|
||||
### 4.2 Message endpoints
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 156 | `POST /v1/managed-agents/<id>/messages` with `{"content":"hi","direction":"user_to_agent","mode":"immediate"}` | 200, message stored |
|
||||
| 157 | `GET /v1/managed-agents/<id>/messages` | 200, returns message list |
|
||||
| 158 | `POST /v1/managed-agents/<id>/messages` with `{"content":"","direction":"user_to_agent","mode":"immediate"}` | 422 or graceful handling |
|
||||
| 159 | `POST /v1/managed-agents/<id>/messages` with `{"content":"cmd","direction":"user_to_agent","mode":"queued"}` | 200, message has status=pending |
|
||||
|
||||
### 4.3 Task & channel endpoints
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 160 | `GET /v1/managed-agents/<id>/tasks` | 200, returns task list |
|
||||
| 161 | `POST /v1/managed-agents/<id>/tasks` | 200, creates task |
|
||||
| 162 | `GET /v1/managed-agents/<id>/channels` | 200, returns channel bindings |
|
||||
| 163 | `GET /v1/managed-agents/<id>/state` | 200, returns full agent state |
|
||||
|
||||
### 4.4 WebSocket events
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 164 | Connect to `ws://localhost:8222/v1/agents/events` | Connection established |
|
||||
| 165 | Trigger a tick → observe WS messages | Receive AGENT_TICK_START and AGENT_TICK_END events |
|
||||
| 166 | Connect with `?agent_id=<id>` filter | Only events for that agent |
|
||||
| 167 | Disconnect cleanly | No server error logs |
|
||||
|
||||
---
|
||||
|
||||
## Part 5: Cross-Platform Consistency
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 168 | Create agent via CLI → check web + desktop | Same name, status, config everywhere |
|
||||
| 169 | Run tick via CLI → check web + desktop | Run count and last run time update in both UIs |
|
||||
| 170 | Send message via web Interact → check CLI `messages` | Same content, direction, mode |
|
||||
| 171 | Pause via desktop → check CLI `status` + web | `paused` everywhere |
|
||||
| 172 | Delete via web → check CLI `list` + desktop | Gone everywhere |
|
||||
| 173 | Create via web wizard → check CLI `list` + desktop | Agent visible in all three |
|
||||
| 174 | Recover via CLI → check web + desktop | Status back to idle in all UIs |
|
||||
| 175 | Send queued message via CLI `instruct` → check web Interact | Message shows with pending/queued status |
|
||||
| 176 | Multiple agents created from different clients | All agents appear correctly in all views |
|
||||
|
||||
---
|
||||
|
||||
## Part 6: Stress & Concurrency
|
||||
|
||||
| # | Test | Expected |
|
||||
|---|------|----------|
|
||||
| 177 | Run tick on same agent from two terminals simultaneously | Concurrency guard blocks second tick: "Agent is already running" |
|
||||
| 178 | Create 20+ agents → check list performance | All clients render list without lag |
|
||||
| 179 | Rapidly pause/resume same agent | All state transitions correct, no stuck states |
|
||||
| 180 | Run daemon + manual `run` at same time | No double-ticking, concurrency guard holds |
|
||||
| 181 | Delete agent while tick is in progress | Tick completes or fails gracefully, agent ends up archived |
|
||||
|
||||
---
|
||||
|
||||
## Part 7: Deferred Features (Placeholder Verification)
|
||||
|
||||
Confirm these show placeholders (not crashes):
|
||||
|
||||
| # | Feature | CLI | Web | Desktop |
|
||||
|---|---------|-----|-----|---------|
|
||||
| 182 | Budget enforcement | `run` still works even if cost > budget | No enforcement, budget is display-only | Same |
|
||||
| 183 | Stall detection | No automatic stall detection fires | N/A | N/A |
|
||||
| 184 | Learning event timeline | `Learning` tab shows placeholder text | Same | Same |
|
||||
| 185 | Logs trace replay | `Logs` tab shows placeholder text | Same | Same |
|
||||
| 186 | `POST /v1/skills` | N/A | N/A | Returns `"not_implemented"` |
|
||||
| 187 | `POST /v1/optimize/runs` | N/A | N/A | Returns placeholder `run_id` |
|
||||
| 188 | `GET /v1/feedback/stats` | N/A | N/A | Returns `{total: 0, mean_score: 0.0}` |
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
**1. Test results** — Spreadsheet with columns: #, Status (Pass/Fail/Partial/Blocked), Actual Behavior, Screenshot (for UI issues).
|
||||
|
||||
**2. Bug list** — Each bug: steps to reproduce, expected vs actual, severity (Critical/Major/Minor), screenshot.
|
||||
|
||||
**3. UX & aesthetics feedback** — Is the launch wizard clear? Are status colors distinguishable? Does the Interact tab feel like chat? Is CLI output readable? Are error messages helpful? Is the delete-without-confirm behavior in web acceptable?
|
||||
|
||||
**4. API error handling audit** — Document all cases where the frontend silently swallows errors (currently: agent list fetch, interact tab sends). Recommend which should show user-visible errors.
|
||||
|
||||
**5. Deferred features check** — Confirm placeholder items in Part 7 show graceful stubs (not crashes or blank screens).
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Backend (`jarvis serve`) must be running for web and desktop (default port 8222).
|
||||
- Without an engine configured, `run`/`ask` will error — document whether the error message is clear.
|
||||
- `daemon` and `watch` block — Ctrl+C to exit.
|
||||
- Web frontend API client has unused functions (`updateManagedAgent`, `createAgentTask`, `fetchAgentState`, `fetchErrorAgents`) — not a bug, but note for future.
|
||||
- Desktop API client is missing some endpoints that the web client has (`fetchAgentChannels`, `fetchAgentState`, `fetchErrorAgents`) — may affect feature parity.
|
||||
- Frontend has **no automated tests** — all testing is manual per this plan.
|
||||
- Both frontends silently catch API errors (`.catch(() => {})`) — this is a known UX gap to evaluate.
|
||||
+1
-6
@@ -163,10 +163,5 @@ nav:
|
||||
- Security: architecture/security.md
|
||||
- Design Principles: architecture/design-principles.md
|
||||
- API Reference: api-reference/
|
||||
- Development:
|
||||
- Contributing: development/contributing.md
|
||||
- Extending OpenJarvis: development/extending.md
|
||||
- Deployment: deployment/index.md
|
||||
- Roadmap: development/roadmap.md
|
||||
- Changelog: development/changelog.md
|
||||
- Leaderboard: leaderboard.md
|
||||
- Roadmap: development/roadmap.md
|
||||
|
||||
+8
-2
@@ -11,10 +11,13 @@ requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"click>=8",
|
||||
"datasets>=4.5.0",
|
||||
"ddgs>=9.11.4",
|
||||
"httpx>=0.27",
|
||||
"openai>=1.30",
|
||||
"python-telegram-bot>=22.6",
|
||||
"rich>=13",
|
||||
"tomli>=2.0; python_version < '3.11'",
|
||||
"tomlkit>=0.12",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@@ -25,6 +28,7 @@ dev = [
|
||||
"pytest-cov>=5",
|
||||
"respx>=0.22",
|
||||
"ruff>=0.4",
|
||||
"pre-commit>=3.0",
|
||||
]
|
||||
inference-mlx = ["mlx-lm>=0.19; sys_platform == 'darwin'"]
|
||||
inference-vllm = ["vllm>=0.16.0"]
|
||||
@@ -36,8 +40,10 @@ inference-google = [
|
||||
"google-genai>=1.0",
|
||||
]
|
||||
inference-litellm = ["litellm>=1.40"]
|
||||
inference-gemma = ["pygemma>=0.1.3"]
|
||||
tools-search = [
|
||||
"tavily-python>=0.3",
|
||||
"ddgs>=9.11.4",
|
||||
]
|
||||
memory-faiss = [
|
||||
"faiss-cpu>=1.7",
|
||||
@@ -110,8 +116,8 @@ jarvis = "openjarvis.cli:main"
|
||||
packages = ["src/openjarvis"]
|
||||
|
||||
[tool.hatch.build.targets.wheel.force-include]
|
||||
"src/openjarvis/agents/claude_code_runner" = "openjarvis/agents/claude_code_runner"
|
||||
"src/openjarvis/channels/whatsapp_baileys_bridge" = "openjarvis/channels/whatsapp_baileys_bridge"
|
||||
"src/openjarvis/agents/claude_code_runner" = "_node_modules/claude_code_runner"
|
||||
"src/openjarvis/channels/whatsapp_baileys_bridge" = "_node_modules/whatsapp_baileys_bridge"
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
@@ -50,6 +50,7 @@ fn lookup(table: &[(&str, f64)], key: &str) -> Option<f64> {
|
||||
}
|
||||
|
||||
/// Estimate FLOPs for a model inference using the `2 * params * tokens` approximation.
|
||||
/// Assumes KV caching (each token processed once).
|
||||
///
|
||||
/// Returns `(total_flops, flops_per_token)`. If the model is not found in `MODEL_PARAMS`,
|
||||
/// returns `(0.0, 0.0)`.
|
||||
@@ -72,6 +73,36 @@ pub fn estimate_flops(model: &str, input_tokens: u64, output_tokens: u64) -> (f6
|
||||
(total_flops, flops_per_token)
|
||||
}
|
||||
|
||||
/// Estimate FLOPs without KV caching (full recompute per token).
|
||||
///
|
||||
/// Without KV cache, each token is re-processed for every subsequent token.
|
||||
/// FLOPs = P * N * (N + 1) where P = params, N = total_tokens.
|
||||
///
|
||||
/// Returns `(total_flops, flops_per_token_avg)`. If the model is not found,
|
||||
/// returns `(0.0, 0.0)`.
|
||||
pub fn estimate_flops_no_kv_cache(
|
||||
model: &str,
|
||||
input_tokens: u64,
|
||||
output_tokens: u64,
|
||||
) -> (f64, f64) {
|
||||
let params_b = match lookup(MODEL_PARAMS, model) {
|
||||
Some(p) => p,
|
||||
None => return (0.0, 0.0),
|
||||
};
|
||||
|
||||
let total_tokens = input_tokens + output_tokens;
|
||||
if total_tokens == 0 {
|
||||
return (0.0, 0.0);
|
||||
}
|
||||
|
||||
let params = params_b * 1e9;
|
||||
let n = total_tokens as f64;
|
||||
let total_flops = params * n * (n + 1.0);
|
||||
let flops_per_token = total_flops / n;
|
||||
|
||||
(total_flops, flops_per_token)
|
||||
}
|
||||
|
||||
/// Compute Model FLOPs Utilization (MFU).
|
||||
///
|
||||
/// MFU = actual_flops / (peak_tflops * 1e12 * duration_s * num_gpus)
|
||||
|
||||
@@ -15,6 +15,19 @@ pub struct SQLiteMemory {
|
||||
|
||||
impl SQLiteMemory {
|
||||
pub fn new(db_path: &Path) -> Result<Self, OpenJarvisError> {
|
||||
// Expand leading ~ to the user's home directory
|
||||
let db_path = if db_path.starts_with("~") {
|
||||
let home = std::env::var("HOME").map_err(|_| {
|
||||
OpenJarvisError::Io(std::io::Error::other(
|
||||
"HOME environment variable not set",
|
||||
))
|
||||
})?;
|
||||
PathBuf::from(home).join(db_path.strip_prefix("~").unwrap())
|
||||
} else {
|
||||
db_path.to_path_buf()
|
||||
};
|
||||
let db_path = db_path.as_path();
|
||||
|
||||
if let Some(parent) = db_path.parent() {
|
||||
std::fs::create_dir_all(parent).map_err(|e| {
|
||||
OpenJarvisError::Io(std::io::Error::other(e))
|
||||
@@ -36,7 +49,7 @@ impl SQLiteMemory {
|
||||
created_at REAL DEFAULT (julianday('now'))
|
||||
);
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS documents_fts USING fts5(
|
||||
id, content, source, tokenize='unicode61'
|
||||
content, source, tokenize='porter unicode61'
|
||||
);",
|
||||
)
|
||||
.map_err(|e| {
|
||||
@@ -110,9 +123,10 @@ impl MemoryBackend for SQLiteMemory {
|
||||
))
|
||||
})?;
|
||||
|
||||
let rowid = conn.last_insert_rowid();
|
||||
conn.execute(
|
||||
"INSERT INTO documents_fts (id, content, source) VALUES (?1, ?2, ?3)",
|
||||
rusqlite::params![doc_id, content, source],
|
||||
"INSERT INTO documents_fts (rowid, content, source) VALUES (?1, ?2, ?3)",
|
||||
rusqlite::params![rowid, content, source],
|
||||
)
|
||||
.map_err(|e| {
|
||||
OpenJarvisError::Io(std::io::Error::other(
|
||||
@@ -130,9 +144,13 @@ impl MemoryBackend for SQLiteMemory {
|
||||
) -> Result<Vec<RetrievalResult>, OpenJarvisError> {
|
||||
let conn = self.conn.lock();
|
||||
|
||||
let words: Vec<&str> = query.split_whitespace().collect();
|
||||
let words: Vec<String> = query
|
||||
.split_whitespace()
|
||||
.map(|w| w.trim_matches(|c: char| "?.,!;:'\"()[]{}/ ".contains(c)).to_string())
|
||||
.filter(|w| !w.is_empty())
|
||||
.collect();
|
||||
let fts_query = if words.len() == 1 {
|
||||
words[0].to_string()
|
||||
words[0].clone()
|
||||
} else {
|
||||
words.join(" OR ")
|
||||
};
|
||||
@@ -140,11 +158,11 @@ impl MemoryBackend for SQLiteMemory {
|
||||
let mut stmt = conn
|
||||
.prepare(
|
||||
"SELECT d.content, d.source, d.metadata,
|
||||
bm25(documents_fts, 0.0, 1.0, 0.5) * -1 as score
|
||||
bm25(documents_fts, 1.0, 0.5) * -1 as score
|
||||
FROM documents_fts f
|
||||
JOIN documents d ON d.id = f.id
|
||||
JOIN documents d ON d.rowid = f.rowid
|
||||
WHERE documents_fts MATCH ?1
|
||||
ORDER BY bm25(documents_fts, 0.0, 1.0, 0.5)
|
||||
ORDER BY bm25(documents_fts, 1.0, 0.5)
|
||||
LIMIT ?2",
|
||||
)
|
||||
.map_err(|e| {
|
||||
@@ -179,8 +197,9 @@ impl MemoryBackend for SQLiteMemory {
|
||||
|
||||
fn delete(&self, doc_id: &str) -> Result<bool, OpenJarvisError> {
|
||||
let conn = self.conn.lock();
|
||||
// Delete from FTS5 using the rowid from the documents table
|
||||
conn.execute(
|
||||
"DELETE FROM documents_fts WHERE id = ?1",
|
||||
"DELETE FROM documents_fts WHERE rowid = (SELECT rowid FROM documents WHERE id = ?1)",
|
||||
rusqlite::params![doc_id],
|
||||
)
|
||||
.map_err(|e| {
|
||||
@@ -238,6 +257,29 @@ mod tests {
|
||||
let results = mem.retrieve("Rust programming", 5).unwrap();
|
||||
assert!(!results.is_empty());
|
||||
assert!(results[0].content.contains("Rust"));
|
||||
assert!(results[0].score > 0.0, "score should be positive, got {}", results[0].score);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_sqlite_porter_stemming() {
|
||||
let mem = SQLiteMemory::in_memory().unwrap();
|
||||
mem.store("Medication list for patient", "health", None).unwrap();
|
||||
|
||||
// Plural form should match via porter stemming
|
||||
let results = mem.retrieve("medications", 5).unwrap();
|
||||
assert!(!results.is_empty(), "porter stemming should match 'medications' to 'Medication'");
|
||||
assert!(results[0].score > 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_sqlite_punctuation_stripping() {
|
||||
let mem = SQLiteMemory::in_memory().unwrap();
|
||||
mem.store("Medication list for patient Micah", "health", None).unwrap();
|
||||
|
||||
// Natural language query with trailing punctuation should not break FTS5
|
||||
let results = mem.retrieve("What medications does Micah take?", 5).unwrap();
|
||||
assert!(!results.is_empty(), "query with punctuation should still return results");
|
||||
assert!(results[0].score > 0.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -146,6 +146,12 @@ info "Installing Python dependencies..."
|
||||
uv sync --extra server --quiet 2>/dev/null || uv sync --extra server
|
||||
ok "Python dependencies installed"
|
||||
|
||||
# ── 7b. Build Rust extension ──────────────────────────────────────
|
||||
info "Building Rust extension..."
|
||||
uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml --quiet 2>/dev/null \
|
||||
|| uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml
|
||||
ok "Rust extension built"
|
||||
|
||||
# ── 8. Install frontend dependencies ────────────────────────────────
|
||||
info "Installing frontend dependencies..."
|
||||
(cd frontend && npm install --silent 2>/dev/null || npm install)
|
||||
|
||||
@@ -13,6 +13,7 @@ from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from openjarvis.core.config import load_config
|
||||
from openjarvis.core.events import EventBus, EventType
|
||||
from openjarvis.core.types import Conversation, Message, Role, ToolResult
|
||||
from openjarvis.engine._stubs import InferenceEngine
|
||||
@@ -63,14 +64,43 @@ class BaseAgent(ABC):
|
||||
model: str,
|
||||
*,
|
||||
bus: Optional[EventBus] = None,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
prompt_builder: Optional[Any] = None,
|
||||
) -> None:
|
||||
self._engine = engine
|
||||
self._model = model
|
||||
self._bus = bus
|
||||
self._temperature = temperature
|
||||
self._max_tokens = max_tokens
|
||||
self._prompt_builder = prompt_builder
|
||||
|
||||
# Three-tier resolution: explicit arg > config > class default > hardcoded
|
||||
if temperature is not None and max_tokens is not None:
|
||||
self._temperature = temperature
|
||||
self._max_tokens = max_tokens
|
||||
else:
|
||||
try:
|
||||
cfg = load_config()
|
||||
self._temperature = (
|
||||
temperature
|
||||
if temperature is not None
|
||||
else cfg.intelligence.temperature
|
||||
)
|
||||
self._max_tokens = (
|
||||
max_tokens
|
||||
if max_tokens is not None
|
||||
else cfg.intelligence.max_tokens
|
||||
)
|
||||
except Exception:
|
||||
self._temperature = (
|
||||
temperature
|
||||
if temperature is not None
|
||||
else getattr(self, "_default_temperature", 0.7)
|
||||
)
|
||||
self._max_tokens = (
|
||||
max_tokens
|
||||
if max_tokens is not None
|
||||
else getattr(self, "_default_max_tokens", 1024)
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Concrete helpers
|
||||
@@ -104,8 +134,14 @@ class BaseAgent(ABC):
|
||||
conversation messages, and finally the user input.
|
||||
"""
|
||||
messages: list[Message] = []
|
||||
if system_prompt:
|
||||
messages.append(Message(role=Role.SYSTEM, content=system_prompt))
|
||||
if self._prompt_builder is not None:
|
||||
effective_system_prompt = self._prompt_builder.build()
|
||||
elif system_prompt:
|
||||
effective_system_prompt = system_prompt
|
||||
else:
|
||||
effective_system_prompt = None
|
||||
if effective_system_prompt:
|
||||
messages.append(Message(role=Role.SYSTEM, content=effective_system_prompt))
|
||||
if context and context.conversation.messages:
|
||||
messages.extend(context.conversation.messages)
|
||||
messages.append(Message(role=Role.USER, content=input))
|
||||
@@ -129,14 +165,19 @@ class BaseAgent(ABC):
|
||||
tool_results: list[ToolResult],
|
||||
turns: int,
|
||||
content: str = "",
|
||||
*,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
) -> AgentResult:
|
||||
"""Build the standard result for when ``max_turns`` is exceeded."""
|
||||
self._emit_turn_end(turns=turns, max_turns_exceeded=True)
|
||||
md: Dict[str, Any] = {"max_turns_exceeded": True}
|
||||
if metadata:
|
||||
md.update(metadata)
|
||||
return AgentResult(
|
||||
content=content or "Maximum turns reached without a final answer.",
|
||||
tool_results=tool_results,
|
||||
turns=turns,
|
||||
metadata={"max_turns_exceeded": True},
|
||||
metadata=md,
|
||||
)
|
||||
|
||||
def _check_continuation(
|
||||
@@ -184,7 +225,9 @@ class BaseAgent(ABC):
|
||||
"""
|
||||
# Full <think>...</think> blocks
|
||||
text = re.sub(
|
||||
r"<think>.*?</think>\s*", "", text,
|
||||
r"<think>.*?</think>\s*",
|
||||
"",
|
||||
text,
|
||||
flags=re.DOTALL | re.IGNORECASE,
|
||||
)
|
||||
# Leading content before a bare </think> (no opening tag)
|
||||
@@ -217,9 +260,9 @@ class ToolUsingAgent(BaseAgent):
|
||||
*,
|
||||
tools: Optional[List["BaseTool"]] = None, # noqa: F821
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 10,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
max_turns: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
loop_guard_config: Optional[Any] = None,
|
||||
capability_policy: Optional[Any] = None,
|
||||
agent_id: Optional[str] = None,
|
||||
@@ -227,21 +270,33 @@ class ToolUsingAgent(BaseAgent):
|
||||
confirm_callback: Optional[Any] = None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, bus=bus,
|
||||
temperature=temperature, max_tokens=max_tokens,
|
||||
engine,
|
||||
model,
|
||||
bus=bus,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
from openjarvis.tools._stubs import ToolExecutor
|
||||
|
||||
self._tools = tools or []
|
||||
_aid = agent_id or getattr(self, "agent_id", "")
|
||||
self._executor = ToolExecutor(
|
||||
self._tools, bus=bus,
|
||||
self._tools,
|
||||
bus=bus,
|
||||
capability_policy=capability_policy,
|
||||
agent_id=_aid,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
)
|
||||
self._max_turns = max_turns
|
||||
# Resolve max_turns: explicit arg > config > class default > 10
|
||||
if max_turns is not None:
|
||||
self._max_turns = max_turns
|
||||
else:
|
||||
try:
|
||||
cfg = load_config()
|
||||
self._max_turns = cfg.agent.max_turns
|
||||
except Exception:
|
||||
self._max_turns = getattr(self, "_default_max_turns", 10)
|
||||
|
||||
# Loop guard
|
||||
self._loop_guard = None
|
||||
|
||||
@@ -29,8 +29,16 @@ logger = logging.getLogger(__name__)
|
||||
_OUTPUT_START = "---OPENJARVIS_OUTPUT_START---"
|
||||
_OUTPUT_END = "---OPENJARVIS_OUTPUT_END---"
|
||||
|
||||
# Path to the bundled runner source (relative to this module)
|
||||
# Path to the bundled runner source (relative to this module).
|
||||
# In editable installs this lives next to this file; in wheel installs
|
||||
# it is placed under _node_modules/ to avoid namespace package conflicts.
|
||||
_RUNNER_SRC = Path(__file__).resolve().parent / "claude_code_runner"
|
||||
if not _RUNNER_SRC.exists():
|
||||
_RUNNER_SRC = (
|
||||
Path(__file__).resolve().parents[2]
|
||||
/ "_node_modules"
|
||||
/ "claude_code_runner"
|
||||
)
|
||||
|
||||
|
||||
@AgentRegistry.register("claude_code")
|
||||
@@ -47,6 +55,8 @@ class ClaudeCodeAgent(BaseAgent):
|
||||
|
||||
agent_id = "claude_code"
|
||||
accepts_tools = False
|
||||
_default_temperature = 0.7
|
||||
_default_max_tokens = 1024
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -54,8 +64,8 @@ class ClaudeCodeAgent(BaseAgent):
|
||||
model: str,
|
||||
*,
|
||||
bus: Optional[EventBus] = None,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
api_key: str = "",
|
||||
workspace: str = "",
|
||||
session_id: str = "",
|
||||
@@ -64,8 +74,11 @@ class ClaudeCodeAgent(BaseAgent):
|
||||
timeout: int = 300,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, bus=bus,
|
||||
temperature=temperature, max_tokens=max_tokens,
|
||||
engine,
|
||||
model,
|
||||
bus=bus,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
self._api_key = api_key or os.environ.get("ANTHROPIC_API_KEY", "")
|
||||
self._workspace = workspace or os.getcwd()
|
||||
@@ -170,7 +183,8 @@ class ClaudeCodeAgent(BaseAgent):
|
||||
stderr = proc.stderr.strip() if proc.stderr else "Unknown error"
|
||||
logger.error(
|
||||
"claude_code_runner exited with code %d: %s",
|
||||
proc.returncode, stderr,
|
||||
proc.returncode,
|
||||
stderr,
|
||||
)
|
||||
self._emit_turn_end(turns=1, error=True)
|
||||
return AgentResult(
|
||||
@@ -209,7 +223,7 @@ class ClaudeCodeAgent(BaseAgent):
|
||||
# No sentinels -- treat entire stdout as plain content
|
||||
return stdout.strip(), [], {}
|
||||
|
||||
json_str = stdout[start + len(_OUTPUT_START):end].strip()
|
||||
json_str = stdout[start + len(_OUTPUT_START) : end].strip()
|
||||
|
||||
try:
|
||||
data = json.loads(json_str)
|
||||
|
||||
@@ -47,6 +47,24 @@ class AgentExecutor:
|
||||
"""Deferred system injection — called after JarvisSystem is constructed."""
|
||||
self._system = system
|
||||
|
||||
def run_ephemeral(
|
||||
self,
|
||||
agent_type: str,
|
||||
system_prompt: str,
|
||||
input_text: str,
|
||||
tools: list[str] | None = None,
|
||||
) -> Any:
|
||||
"""Run a one-shot agent turn with no lifecycle tracking."""
|
||||
from openjarvis.core.registry import AgentRegistry
|
||||
|
||||
agent_cls = AgentRegistry.get(agent_type)
|
||||
agent = agent_cls(
|
||||
engine=getattr(self._manager, '_engine', None),
|
||||
system_prompt=system_prompt,
|
||||
bus=self._bus,
|
||||
)
|
||||
return agent.run(input_text)
|
||||
|
||||
def execute_tick(self, agent_id: str) -> None:
|
||||
"""Run one tick for the given agent.
|
||||
|
||||
@@ -218,19 +236,69 @@ class AgentExecutor:
|
||||
instruction = config.get("instruction", "")
|
||||
memory = agent.get("summary_memory", "")
|
||||
if instruction:
|
||||
context = f"Standing instruction: {instruction}"
|
||||
input_text = f"Standing instruction: {instruction}"
|
||||
if memory:
|
||||
context += f"\n\nPrevious context: {memory}"
|
||||
input_text += f"\n\nPrevious context: {memory}"
|
||||
else:
|
||||
context = memory or "Continue your assigned task."
|
||||
input_text = memory or "Continue your assigned task."
|
||||
pending = self._manager.get_pending_messages(agent["id"])
|
||||
if pending:
|
||||
user_msgs = "\n".join(f"User: {m['content']}" for m in pending)
|
||||
context = f"{context}\n\nNew instructions:\n{user_msgs}"
|
||||
input_text = f"{input_text}\n\nNew instructions:\n{user_msgs}"
|
||||
for m in pending:
|
||||
self._manager.mark_message_delivered(m["id"])
|
||||
|
||||
return agent_instance.run(context)
|
||||
# Build AgentContext with memory results from FTS5 backend
|
||||
from openjarvis.agents._stubs import AgentContext
|
||||
|
||||
agent_ctx = AgentContext()
|
||||
memory_results = []
|
||||
|
||||
if (
|
||||
self._system
|
||||
and getattr(self._system, "memory_backend", None)
|
||||
and getattr(self._system, "config", None)
|
||||
and self._system.config.agent.context_from_memory
|
||||
):
|
||||
try:
|
||||
from openjarvis.tools.storage.context import (
|
||||
ContextConfig,
|
||||
format_context,
|
||||
)
|
||||
|
||||
sys_cfg = self._system.config
|
||||
ctx_cfg = ContextConfig(
|
||||
top_k=sys_cfg.memory.context_top_k,
|
||||
min_score=sys_cfg.memory.context_min_score,
|
||||
max_context_tokens=sys_cfg.memory.context_max_tokens,
|
||||
)
|
||||
# Use pending user messages as query, fall back to instruction
|
||||
query = ""
|
||||
if pending:
|
||||
query = " ".join(m["content"] for m in pending)
|
||||
elif instruction:
|
||||
query = instruction
|
||||
|
||||
if query:
|
||||
results = self._system.memory_backend.retrieve(
|
||||
query, top_k=ctx_cfg.top_k,
|
||||
)
|
||||
memory_results = [
|
||||
r for r in results if r.score >= ctx_cfg.min_score
|
||||
]
|
||||
if memory_results:
|
||||
# Prepend retrieved context to input for agents
|
||||
# that don't inspect AgentContext.memory_results
|
||||
retrieved = format_context(memory_results)
|
||||
input_text = (
|
||||
f"Retrieved context from knowledge base:\n"
|
||||
f"{retrieved}\n\n{input_text}"
|
||||
)
|
||||
except Exception:
|
||||
pass # Don't break agent tick if memory retrieval fails
|
||||
|
||||
agent_ctx.memory_results = memory_results
|
||||
return agent_instance.run(input_text, context=agent_ctx)
|
||||
|
||||
def _build_error_detail(self, error: AgentTickError) -> dict[str, Any]:
|
||||
"""Build structured error detail for trace metadata."""
|
||||
|
||||
@@ -18,6 +18,7 @@ class LoopGuardConfig:
|
||||
ping_pong_window: int = 6 # detect A-B-A-B cycling
|
||||
poll_tool_budget: int = 5 # max calls to same polling tool
|
||||
max_context_messages: int = 100 # context overflow threshold
|
||||
warn_before_block: bool = True # warn on first cycle, block on second
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
@@ -25,6 +26,7 @@ class LoopVerdict:
|
||||
"""Result of a loop guard check."""
|
||||
blocked: bool = False
|
||||
reason: str = ""
|
||||
warned: bool = False
|
||||
|
||||
|
||||
class LoopGuard:
|
||||
@@ -47,26 +49,49 @@ class LoopGuard:
|
||||
self._tool_sequence: deque[str] = deque(maxlen=config.ping_pong_window * 2)
|
||||
# Track per-tool call counts (for polling budget)
|
||||
self._per_tool_counts: dict[str, int] = {}
|
||||
# Track cycle keys that have already been warned (for warn-before-block)
|
||||
self._warned_cycles: set[str] = set()
|
||||
|
||||
from openjarvis._rust_bridge import get_rust_module
|
||||
_rust = get_rust_module()
|
||||
self._rust_impl = _rust.LoopGuard(
|
||||
max_identical=config.max_identical_calls,
|
||||
max_ping_pong=(
|
||||
config.ping_pong_window // 2
|
||||
if config.ping_pong_window > 1
|
||||
else 2
|
||||
),
|
||||
poll_budget=config.poll_tool_budget,
|
||||
)
|
||||
try:
|
||||
from openjarvis._rust_bridge import get_rust_module
|
||||
_rust = get_rust_module()
|
||||
self._rust_impl = _rust.LoopGuard(
|
||||
max_identical=config.max_identical_calls,
|
||||
max_ping_pong=(
|
||||
config.ping_pong_window // 2
|
||||
if config.ping_pong_window > 1
|
||||
else 2
|
||||
),
|
||||
poll_budget=config.poll_tool_budget,
|
||||
)
|
||||
except Exception:
|
||||
self._rust_impl = None
|
||||
|
||||
def check_call(self, tool_name: str, arguments: str) -> LoopVerdict:
|
||||
"""Check whether a tool call should proceed or be blocked."""
|
||||
reason = self._rust_impl.check(tool_name, arguments)
|
||||
if reason is not None:
|
||||
self._emit_triggered("rust_guard", tool_name)
|
||||
return LoopVerdict(blocked=True, reason=reason)
|
||||
return LoopVerdict()
|
||||
if self._rust_impl is not None:
|
||||
rust_result = self._rust_impl.check(tool_name, arguments)
|
||||
# Support both raw Rust return (str | None) and LoopVerdict
|
||||
if isinstance(rust_result, LoopVerdict):
|
||||
verdict = rust_result
|
||||
elif rust_result is not None:
|
||||
self._emit_triggered("rust_guard", tool_name)
|
||||
verdict = LoopVerdict(blocked=True, reason=rust_result)
|
||||
else:
|
||||
verdict = LoopVerdict()
|
||||
else:
|
||||
verdict = self._python_check(tool_name, arguments)
|
||||
|
||||
# Wrap with warn-before-block logic
|
||||
if verdict.blocked and self._config.warn_before_block:
|
||||
cycle_key = verdict.reason
|
||||
if cycle_key not in self._warned_cycles:
|
||||
self._warned_cycles.add(cycle_key)
|
||||
return LoopVerdict(blocked=False, warned=True, reason=verdict.reason)
|
||||
return verdict
|
||||
|
||||
def _python_check(self, tool_name: str, arguments: str) -> LoopVerdict:
|
||||
"""Pure-Python fallback when Rust backend is not available."""
|
||||
# 1. Hash tracking — identical calls
|
||||
call_hash = hashlib.sha256(
|
||||
f"{tool_name}:{arguments}".encode()
|
||||
@@ -198,7 +223,9 @@ class LoopGuard:
|
||||
self._call_counts.clear()
|
||||
self._tool_sequence.clear()
|
||||
self._per_tool_counts.clear()
|
||||
self._rust_impl.reset()
|
||||
self._warned_cycles.clear()
|
||||
if self._rust_impl is not None:
|
||||
self._rust_impl.reset()
|
||||
|
||||
def _detect_ping_pong(self) -> bool:
|
||||
"""Detect repeating patterns in tool call sequence."""
|
||||
|
||||
@@ -84,6 +84,9 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
|
||||
agent_id = "monitor_operative"
|
||||
accepts_tools = True
|
||||
_default_temperature = 0.3
|
||||
_default_max_tokens = 4096
|
||||
_default_max_turns = 25
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -92,9 +95,9 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
*,
|
||||
tools: Optional[List[BaseTool]] = None,
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 25,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 4096,
|
||||
max_turns: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
system_prompt: Optional[str] = None,
|
||||
# Strategy parameters
|
||||
memory_extraction: str = "causality_graph",
|
||||
@@ -110,10 +113,15 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, tools=tools, bus=bus,
|
||||
max_turns=max_turns, temperature=temperature,
|
||||
engine,
|
||||
model,
|
||||
tools=tools,
|
||||
bus=bus,
|
||||
max_turns=max_turns,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
interactive=interactive, confirm_callback=confirm_callback,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
)
|
||||
# Validate strategies
|
||||
if memory_extraction not in VALID_MEMORY_EXTRACTION:
|
||||
@@ -191,7 +199,8 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
|
||||
# 4. Build messages
|
||||
messages = self._build_operative_messages(
|
||||
input, context,
|
||||
input,
|
||||
context,
|
||||
system_prompt=system_prompt,
|
||||
session_messages=session_messages,
|
||||
)
|
||||
@@ -202,6 +211,11 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
turns = 0
|
||||
content = ""
|
||||
state_stored_by_tool = False
|
||||
total_usage: dict[str, int] = {
|
||||
"prompt_tokens": 0,
|
||||
"completion_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
}
|
||||
|
||||
for _turn in range(self._max_turns):
|
||||
turns += 1
|
||||
@@ -211,6 +225,9 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
gen_kwargs["tools"] = openai_tools
|
||||
|
||||
result = self._generate(messages, **gen_kwargs)
|
||||
usage = result.get("usage", {})
|
||||
for k in total_usage:
|
||||
total_usage[k] += usage.get(k, 0)
|
||||
content = result.get("content", "")
|
||||
raw_tool_calls = result.get("tool_calls", [])
|
||||
|
||||
@@ -230,18 +247,21 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
]
|
||||
|
||||
# Append assistant message with tool calls
|
||||
messages.append(Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
)
|
||||
)
|
||||
|
||||
# Execute each tool
|
||||
for tc in tool_calls:
|
||||
# Loop guard check
|
||||
if self._loop_guard:
|
||||
verdict = self._loop_guard.check_call(
|
||||
tc.name, tc.arguments,
|
||||
tc.name,
|
||||
tc.arguments,
|
||||
)
|
||||
if verdict.blocked:
|
||||
tool_result = ToolResult(
|
||||
@@ -250,12 +270,14 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
success=False,
|
||||
)
|
||||
all_tool_results.append(tool_result)
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
tool_result = self._executor.execute(tc)
|
||||
@@ -271,18 +293,21 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
except (json.JSONDecodeError, TypeError) as exc:
|
||||
logger.debug(
|
||||
"Failed to parse tool call arguments"
|
||||
" for state tracking: %s", exc,
|
||||
" for state tracking: %s",
|
||||
exc,
|
||||
)
|
||||
|
||||
# Compress observation if strategy requires it
|
||||
observation_content = self._compress_observation(tool_result.content)
|
||||
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=observation_content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=observation_content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
|
||||
# Extract and store findings based on memory strategy
|
||||
self._extract_and_store(tc.name, tool_result.content)
|
||||
@@ -290,7 +315,10 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
# Max turns exceeded
|
||||
self._save_session(input, content)
|
||||
return self._max_turns_result(
|
||||
all_tool_results, turns, content=content,
|
||||
all_tool_results,
|
||||
turns,
|
||||
content=content,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
# 6. Save session
|
||||
@@ -305,6 +333,7 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
content=content,
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -335,6 +364,7 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
if not self._tools:
|
||||
return ""
|
||||
from openjarvis.tools._stubs import build_tool_descriptions
|
||||
|
||||
return build_tool_descriptions(self._tools)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -448,11 +478,13 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
self._memory_backend.store(key, value)
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"Failed to store causality relation in memory: %s", exc,
|
||||
"Failed to store causality relation in memory: %s",
|
||||
exc,
|
||||
)
|
||||
except (json.JSONDecodeError, Exception):
|
||||
logger.debug(
|
||||
"Causality extraction failed for tool %s output", tool_name,
|
||||
"Causality extraction failed for tool %s output",
|
||||
tool_name,
|
||||
)
|
||||
|
||||
def _store_scratchpad(self, tool_name: str, content: str) -> None:
|
||||
@@ -493,7 +525,8 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
except Exception as exc:
|
||||
logger.debug(
|
||||
"Failed to store structured data for tool %s: %s",
|
||||
tool_name, exc,
|
||||
tool_name,
|
||||
exc,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -547,10 +580,12 @@ class MonitorOperativeAgent(ToolUsingAgent):
|
||||
session_id = f"monitor_operative:{self._operator_id}"
|
||||
try:
|
||||
self._session_store.save_message(
|
||||
session_id, {"role": "user", "content": input_text},
|
||||
session_id,
|
||||
{"role": "user", "content": input_text},
|
||||
)
|
||||
self._session_store.save_message(
|
||||
session_id, {"role": "assistant", "content": response},
|
||||
session_id,
|
||||
{"role": "assistant", "content": response},
|
||||
)
|
||||
except Exception:
|
||||
logger.debug(
|
||||
|
||||
@@ -54,6 +54,9 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
"""Native CodeAct agent -- generates and executes Python code."""
|
||||
|
||||
agent_id = "native_openhands"
|
||||
_default_temperature = 0.7
|
||||
_default_max_tokens = 2048
|
||||
_default_max_turns = 3
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -62,17 +65,22 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
*,
|
||||
tools: Optional[List[BaseTool]] = None,
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 3,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 2048,
|
||||
max_turns: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
interactive: bool = False,
|
||||
confirm_callback=None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, tools=tools, bus=bus,
|
||||
max_turns=max_turns, temperature=temperature,
|
||||
engine,
|
||||
model,
|
||||
tools=tools,
|
||||
bus=bus,
|
||||
max_turns=max_turns,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
interactive=interactive, confirm_callback=confirm_callback,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
@@ -93,9 +101,7 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
content = WebSearchTool._fetch_url(url, max_chars=4000)
|
||||
header = f"\n\n--- Content from {url} ---\n"
|
||||
footer = "\n--- End of content ---\n"
|
||||
expanded = text.replace(
|
||||
url, f"{header}{content}{footer}"
|
||||
)
|
||||
expanded = text.replace(url, f"{header}{content}{footer}")
|
||||
return expanded, True
|
||||
except Exception:
|
||||
return text, False
|
||||
@@ -121,9 +127,7 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
messages[i] = Message(
|
||||
role=Role.USER,
|
||||
content=(
|
||||
truncated
|
||||
+ "\n\n[Input truncated"
|
||||
" to fit context window]"
|
||||
truncated + "\n\n[Input truncated to fit context window]"
|
||||
),
|
||||
)
|
||||
break
|
||||
@@ -135,7 +139,9 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
# Remove Action: ... Action Input: ... blocks
|
||||
text = re.sub(
|
||||
r"Action:\s*.+?(?:Action Input:\s*.+?)?(?=\n\n|\Z)",
|
||||
"", text, flags=re.DOTALL | re.IGNORECASE,
|
||||
"",
|
||||
text,
|
||||
flags=re.DOTALL | re.IGNORECASE,
|
||||
)
|
||||
# Remove <tool_call>...</tool_call> or </tool_name> blocks
|
||||
text = re.sub(r"<tool_call>.*?</\w+>", "", text, flags=re.DOTALL)
|
||||
@@ -242,7 +248,9 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
usage = result.get("usage", {})
|
||||
self._emit_turn_end(turns=1)
|
||||
return AgentResult(
|
||||
content=content, tool_results=[], turns=1,
|
||||
content=content,
|
||||
tool_results=[],
|
||||
turns=1,
|
||||
metadata={
|
||||
"prompt_tokens": usage.get("prompt_tokens", 0),
|
||||
"completion_tokens": usage.get("completion_tokens", 0),
|
||||
@@ -258,10 +266,7 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
"Please try a shorter message."
|
||||
)
|
||||
else:
|
||||
error_msg = (
|
||||
"The model returned an error: "
|
||||
+ error_str
|
||||
)
|
||||
error_msg = "The model returned an error: " + error_str
|
||||
self._emit_turn_end(turns=1, error=True)
|
||||
return AgentResult(
|
||||
content=error_msg,
|
||||
@@ -278,14 +283,23 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
last_content = ""
|
||||
total_usage = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
|
||||
|
||||
# Build OpenAI-format tool schemas for native function calling
|
||||
openai_tools = (
|
||||
self._executor.get_openai_tools() if self._tools else []
|
||||
)
|
||||
|
||||
for _turn in range(self._max_turns):
|
||||
turns += 1
|
||||
# Truncate before every generate call -- tool results may have
|
||||
# expanded the context beyond what the model supports.
|
||||
messages = self._truncate_if_needed(messages)
|
||||
|
||||
gen_kwargs: dict[str, Any] = {}
|
||||
if openai_tools:
|
||||
gen_kwargs["tools"] = openai_tools
|
||||
|
||||
try:
|
||||
result = self._generate(messages)
|
||||
result = self._generate(messages, **gen_kwargs)
|
||||
except Exception as exc:
|
||||
error_str = str(exc)
|
||||
if "400" in error_str:
|
||||
@@ -313,6 +327,42 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
content = self._strip_think_tags(content)
|
||||
last_content = content
|
||||
|
||||
# --- Native function-calling path (OpenAI, Anthropic, etc.) ---
|
||||
raw_tool_calls = result.get("tool_calls", [])
|
||||
if raw_tool_calls:
|
||||
native_calls = [
|
||||
ToolCall(
|
||||
id=tc.get("id", f"call_{turns}_{i}"),
|
||||
name=tc.get("name", ""),
|
||||
arguments=tc.get("arguments", "{}"),
|
||||
)
|
||||
for i, tc in enumerate(raw_tool_calls)
|
||||
]
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=native_calls,
|
||||
)
|
||||
)
|
||||
for tc in native_calls:
|
||||
tool_result = self._executor.execute(tc)
|
||||
all_tool_results.append(tool_result)
|
||||
obs_text = tool_result.content
|
||||
if len(obs_text) > 4000:
|
||||
obs_text = obs_text[:4000] + "\n\n[Output truncated]"
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=obs_text,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# --- Text-based fallback (CodeAct / Action-Input format) ---
|
||||
|
||||
# Try to extract code
|
||||
code = self._extract_code(content)
|
||||
if code:
|
||||
@@ -358,7 +408,9 @@ class NativeOpenHandsAgent(ToolUsingAgent):
|
||||
content = self._strip_tool_call_text(content)
|
||||
self._emit_turn_end(turns=turns)
|
||||
return AgentResult(
|
||||
content=content, tool_results=all_tool_results, turns=turns,
|
||||
content=content,
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
|
||||
@@ -36,6 +36,9 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
"""ReAct agent: Thought -> Action -> Observation loop."""
|
||||
|
||||
agent_id = "native_react"
|
||||
_default_temperature = 0.7
|
||||
_default_max_tokens = 1024
|
||||
_default_max_turns = 10
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -44,17 +47,22 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
*,
|
||||
tools: Optional[List[BaseTool]] = None,
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 10,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
max_turns: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
interactive: bool = False,
|
||||
confirm_callback=None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, tools=tools, bus=bus,
|
||||
max_turns=max_turns, temperature=temperature,
|
||||
engine,
|
||||
model,
|
||||
tools=tools,
|
||||
bus=bus,
|
||||
max_turns=max_turns,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
interactive=interactive, confirm_callback=confirm_callback,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
)
|
||||
|
||||
def _parse_response(self, text: str) -> dict:
|
||||
@@ -109,6 +117,11 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
|
||||
all_tool_results: list[ToolResult] = []
|
||||
turns = 0
|
||||
total_usage: dict[str, int] = {
|
||||
"prompt_tokens": 0,
|
||||
"completion_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
}
|
||||
|
||||
for _turn in range(self._max_turns):
|
||||
turns += 1
|
||||
@@ -117,6 +130,9 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
messages = self._loop_guard.compress_context(messages)
|
||||
|
||||
result = self._generate(messages)
|
||||
usage = result.get("usage", {})
|
||||
for k in total_usage:
|
||||
total_usage[k] += usage.get(k, 0)
|
||||
|
||||
content = result.get("content", "")
|
||||
parsed = self._parse_response(content)
|
||||
@@ -128,13 +144,17 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
content=parsed["final_answer"],
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
# No action? Treat content as final answer
|
||||
if not parsed["action"]:
|
||||
self._emit_turn_end(turns=turns)
|
||||
return AgentResult(
|
||||
content=content, tool_results=all_tool_results, turns=turns
|
||||
content=content,
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
# Execute action
|
||||
@@ -149,7 +169,8 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
# Loop guard check before execution
|
||||
if self._loop_guard:
|
||||
verdict = self._loop_guard.check_call(
|
||||
tool_call.name, tool_call.arguments,
|
||||
tool_call.name,
|
||||
tool_call.arguments,
|
||||
)
|
||||
if verdict.blocked:
|
||||
tool_result = ToolResult(
|
||||
@@ -169,7 +190,7 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
messages.append(Message(role=Role.USER, content=observation))
|
||||
|
||||
# Max turns exceeded
|
||||
return self._max_turns_result(all_tool_results, turns)
|
||||
return self._max_turns_result(all_tool_results, turns, metadata=total_usage)
|
||||
|
||||
|
||||
__all__ = ["NativeReActAgent", "REACT_SYSTEM_PROMPT"]
|
||||
|
||||
@@ -25,6 +25,8 @@ class OpenHandsAgent(BaseAgent):
|
||||
"""
|
||||
|
||||
agent_id = "openhands"
|
||||
_default_temperature = 0.7
|
||||
_default_max_tokens = 1024
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -32,14 +34,17 @@ class OpenHandsAgent(BaseAgent):
|
||||
model: str,
|
||||
*,
|
||||
bus: Optional[EventBus] = None,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
workspace: Optional[str] = None,
|
||||
api_key: Optional[str] = None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, bus=bus,
|
||||
temperature=temperature, max_tokens=max_tokens,
|
||||
engine,
|
||||
model,
|
||||
bus=bus,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
self._workspace = workspace or os.getcwd()
|
||||
self._api_key = api_key or os.environ.get("LLM_API_KEY", "")
|
||||
|
||||
@@ -38,6 +38,9 @@ class OperativeAgent(ToolUsingAgent):
|
||||
|
||||
agent_id = "operative"
|
||||
accepts_tools = True
|
||||
_default_temperature = 0.3
|
||||
_default_max_tokens = 2048
|
||||
_default_max_turns = 20
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -46,9 +49,9 @@ class OperativeAgent(ToolUsingAgent):
|
||||
*,
|
||||
tools: Optional[List[BaseTool]] = None,
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 20,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 2048,
|
||||
max_turns: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
system_prompt: Optional[str] = None,
|
||||
operator_id: Optional[str] = None,
|
||||
session_store: Optional[Any] = None,
|
||||
@@ -58,10 +61,15 @@ class OperativeAgent(ToolUsingAgent):
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, tools=tools, bus=bus,
|
||||
max_turns=max_turns, temperature=temperature,
|
||||
engine,
|
||||
model,
|
||||
tools=tools,
|
||||
bus=bus,
|
||||
max_turns=max_turns,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
interactive=interactive, confirm_callback=confirm_callback,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
)
|
||||
self._system_prompt = system_prompt or ""
|
||||
self._operator_id = operator_id
|
||||
@@ -94,7 +102,9 @@ class OperativeAgent(ToolUsingAgent):
|
||||
|
||||
# 4. Build messages
|
||||
messages = self._build_operative_messages(
|
||||
input, context, system_prompt=system_prompt,
|
||||
input,
|
||||
context,
|
||||
system_prompt=system_prompt,
|
||||
session_messages=session_messages,
|
||||
)
|
||||
|
||||
@@ -104,6 +114,11 @@ class OperativeAgent(ToolUsingAgent):
|
||||
turns = 0
|
||||
content = ""
|
||||
state_stored_by_tool = False
|
||||
total_usage: dict[str, int] = {
|
||||
"prompt_tokens": 0,
|
||||
"completion_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
}
|
||||
|
||||
for _turn in range(self._max_turns):
|
||||
turns += 1
|
||||
@@ -116,6 +131,9 @@ class OperativeAgent(ToolUsingAgent):
|
||||
gen_kwargs["tools"] = openai_tools
|
||||
|
||||
result = self._generate(messages, **gen_kwargs)
|
||||
usage = result.get("usage", {})
|
||||
for k in total_usage:
|
||||
total_usage[k] += usage.get(k, 0)
|
||||
content = result.get("content", "")
|
||||
raw_tool_calls = result.get("tool_calls", [])
|
||||
|
||||
@@ -132,11 +150,13 @@ class OperativeAgent(ToolUsingAgent):
|
||||
for i, tc in enumerate(raw_tool_calls)
|
||||
]
|
||||
|
||||
messages.append(Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
)
|
||||
)
|
||||
|
||||
for tc in tool_calls:
|
||||
# Loop guard check
|
||||
@@ -149,12 +169,14 @@ class OperativeAgent(ToolUsingAgent):
|
||||
success=False,
|
||||
)
|
||||
all_tool_results.append(tool_result)
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
tool_result = self._executor.execute(tc)
|
||||
@@ -170,16 +192,25 @@ class OperativeAgent(ToolUsingAgent):
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
else:
|
||||
# Max turns exceeded
|
||||
self._save_session(input, content)
|
||||
return self._max_turns_result(all_tool_results, turns, content=content)
|
||||
meta = dict(total_usage)
|
||||
meta["max_turns_exceeded"] = True
|
||||
return AgentResult(
|
||||
content=content or "Maximum turns reached without a final answer.",
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=meta,
|
||||
)
|
||||
|
||||
# 6. Save session
|
||||
self._save_session(input, content)
|
||||
@@ -193,6 +224,7 @@ class OperativeAgent(ToolUsingAgent):
|
||||
content=content,
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
def _build_operative_messages(
|
||||
@@ -258,10 +290,12 @@ class OperativeAgent(ToolUsingAgent):
|
||||
session_id = f"operator:{self._operator_id}"
|
||||
try:
|
||||
self._session_store.save_message(
|
||||
session_id, {"role": "user", "content": input_text},
|
||||
session_id,
|
||||
{"role": "user", "content": input_text},
|
||||
)
|
||||
self._session_store.save_message(
|
||||
session_id, {"role": "assistant", "content": response},
|
||||
session_id,
|
||||
{"role": "assistant", "content": response},
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("Could not save session for operator %s", self._operator_id)
|
||||
|
||||
@@ -41,6 +41,9 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
"""
|
||||
|
||||
agent_id = "orchestrator"
|
||||
_default_temperature = 0.7
|
||||
_default_max_tokens = 1024
|
||||
_default_max_turns = 10
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -49,9 +52,9 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
*,
|
||||
tools: Optional[List[BaseTool]] = None,
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 10,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
max_turns: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
mode: str = "function_calling",
|
||||
system_prompt: Optional[str] = None,
|
||||
parallel_tools: bool = True,
|
||||
@@ -59,10 +62,15 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
confirm_callback=None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, tools=tools, bus=bus,
|
||||
max_turns=max_turns, temperature=temperature,
|
||||
engine,
|
||||
model,
|
||||
tools=tools,
|
||||
bus=bus,
|
||||
max_turns=max_turns,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
interactive=interactive, confirm_callback=confirm_callback,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
)
|
||||
self._mode = mode
|
||||
self._system_prompt = system_prompt
|
||||
@@ -97,6 +105,7 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
from openjarvis.learning.intelligence.orchestrator.prompt_registry import (
|
||||
build_system_prompt,
|
||||
)
|
||||
|
||||
sys_prompt = build_system_prompt(tools=self._tools)
|
||||
|
||||
messages = self._build_messages(input, context, system_prompt=sys_prompt)
|
||||
@@ -126,9 +135,7 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
|
||||
# TOOL -> execute
|
||||
if parsed["tool"]:
|
||||
messages.append(
|
||||
Message(role=Role.ASSISTANT, content=content)
|
||||
)
|
||||
messages.append(Message(role=Role.ASSISTANT, content=content))
|
||||
|
||||
tool_call = ToolCall(
|
||||
id=f"orch_{turns}",
|
||||
@@ -138,12 +145,8 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
tool_result = self._executor.execute(tool_call)
|
||||
all_tool_results.append(tool_result)
|
||||
|
||||
observation = (
|
||||
f"Observation: {tool_result.content}"
|
||||
)
|
||||
messages.append(
|
||||
Message(role=Role.USER, content=observation)
|
||||
)
|
||||
observation = f"Observation: {tool_result.content}"
|
||||
messages.append(Message(role=Role.USER, content=observation))
|
||||
continue
|
||||
|
||||
# Neither -> treat content as final answer
|
||||
@@ -184,9 +187,7 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
result["final_answer"] = final_match.group(1).strip()
|
||||
return result
|
||||
|
||||
tool_match = re.search(
|
||||
r"TOOL:\s*(.+)", text, re.IGNORECASE
|
||||
)
|
||||
tool_match = re.search(r"TOOL:\s*(.+)", text, re.IGNORECASE)
|
||||
if tool_match:
|
||||
result["tool"] = tool_match.group(1).strip()
|
||||
|
||||
@@ -271,11 +272,13 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
]
|
||||
|
||||
# Append assistant message with tool calls
|
||||
messages.append(Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
)
|
||||
)
|
||||
|
||||
# Execute each tool (with loop guard check) and append results
|
||||
if self._parallel_tools and len(tool_calls) > 1:
|
||||
@@ -283,7 +286,8 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
def _exec_tool(tc: ToolCall) -> tuple:
|
||||
if self._loop_guard:
|
||||
verdict = self._loop_guard.check_call(
|
||||
tc.name, tc.arguments,
|
||||
tc.name,
|
||||
tc.arguments,
|
||||
)
|
||||
if verdict.blocked:
|
||||
return tc, ToolResult(
|
||||
@@ -296,10 +300,7 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
with concurrent.futures.ThreadPoolExecutor(
|
||||
max_workers=len(tool_calls),
|
||||
) as pool:
|
||||
futures = {
|
||||
pool.submit(_exec_tool, tc): tc
|
||||
for tc in tool_calls
|
||||
}
|
||||
futures = {pool.submit(_exec_tool, tc): tc for tc in tool_calls}
|
||||
results_map: dict[int, tuple] = {}
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
tc_orig = futures[future]
|
||||
@@ -309,19 +310,22 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
for tc in tool_calls:
|
||||
_, tool_result = results_map[id(tc)]
|
||||
all_tool_results.append(tool_result)
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
else:
|
||||
# Sequential execution
|
||||
for tc in tool_calls:
|
||||
# Loop guard check before execution
|
||||
if self._loop_guard:
|
||||
verdict = self._loop_guard.check_call(
|
||||
tc.name, tc.arguments,
|
||||
tc.name,
|
||||
tc.arguments,
|
||||
)
|
||||
if verdict.blocked:
|
||||
tool_result = ToolResult(
|
||||
@@ -330,24 +334,28 @@ class OrchestratorAgent(ToolUsingAgent):
|
||||
success=False,
|
||||
)
|
||||
all_tool_results.append(tool_result)
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
tool_result = self._executor.execute(tc)
|
||||
all_tool_results.append(tool_result)
|
||||
|
||||
# Append tool response message
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=tool_result.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
|
||||
# Max turns exceeded
|
||||
final_content = self._strip_think_tags(content) if content else ""
|
||||
|
||||
@@ -35,8 +35,8 @@ RLM_SYSTEM_PROMPT = (
|
||||
"final answer.\n"
|
||||
"- `FINAL_VAR(var_name: str)` — Terminate and return the "
|
||||
"value of variable `var_name`.\n"
|
||||
"- `answer` dict — Set `answer[\"value\"] = ...` and "
|
||||
"`answer[\"ready\"] = True` to terminate.\n\n"
|
||||
'- `answer` dict — Set `answer["value"] = ...` and '
|
||||
'`answer["ready"] = True` to terminate.\n\n'
|
||||
"{tool_section}"
|
||||
"## Available Modules\n\n"
|
||||
"json, re, math, collections, itertools, functools, "
|
||||
@@ -52,7 +52,7 @@ RLM_SYSTEM_PROMPT = (
|
||||
"and use `llm_query()` on each chunk.\n"
|
||||
"3. Combine sub-results programmatically.\n"
|
||||
"4. When you have the final answer, call "
|
||||
"`FINAL(answer_value)` or `FINAL_VAR(\"var_name\")`.\n"
|
||||
'`FINAL(answer_value)` or `FINAL_VAR("var_name")`.\n'
|
||||
"5. If you can answer directly without code, just respond "
|
||||
"with text (no code block).\n\n"
|
||||
"## Strategy Tips\n\n"
|
||||
@@ -84,6 +84,9 @@ class RLMAgent(ToolUsingAgent):
|
||||
"""
|
||||
|
||||
agent_id = "rlm"
|
||||
_default_temperature = 0.7
|
||||
_default_max_tokens = 2048
|
||||
_default_max_turns = 10
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -92,9 +95,9 @@ class RLMAgent(ToolUsingAgent):
|
||||
*,
|
||||
tools: Optional[List[BaseTool]] = None,
|
||||
bus: Optional[EventBus] = None,
|
||||
max_turns: int = 10,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 2048,
|
||||
max_turns: Optional[int] = None,
|
||||
temperature: Optional[float] = None,
|
||||
max_tokens: Optional[int] = None,
|
||||
sub_model: Optional[str] = None,
|
||||
sub_temperature: float = 0.3,
|
||||
sub_max_tokens: int = 1024,
|
||||
@@ -104,10 +107,15 @@ class RLMAgent(ToolUsingAgent):
|
||||
confirm_callback=None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine, model, tools=tools, bus=bus,
|
||||
max_turns=max_turns, temperature=temperature,
|
||||
engine,
|
||||
model,
|
||||
tools=tools,
|
||||
bus=bus,
|
||||
max_turns=max_turns,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
interactive=interactive, confirm_callback=confirm_callback,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
)
|
||||
# Override executor: RLM only creates one if tools are provided
|
||||
if not self._tools:
|
||||
@@ -168,16 +176,26 @@ class RLMAgent(ToolUsingAgent):
|
||||
|
||||
# Build conversation
|
||||
messages = self._build_messages(
|
||||
input, context, system_prompt=system_prompt,
|
||||
input,
|
||||
context,
|
||||
system_prompt=system_prompt,
|
||||
)
|
||||
|
||||
all_tool_results: list[ToolResult] = []
|
||||
turns = 0
|
||||
total_usage: dict[str, int] = {
|
||||
"prompt_tokens": 0,
|
||||
"completion_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
}
|
||||
|
||||
for _turn in range(self._max_turns):
|
||||
turns += 1
|
||||
|
||||
result = self._generate(messages)
|
||||
usage = result.get("usage", {})
|
||||
for k in total_usage:
|
||||
total_usage[k] += usage.get(k, 0)
|
||||
content = result.get("content", "")
|
||||
|
||||
# Strip <think> tags
|
||||
@@ -193,6 +211,7 @@ class RLMAgent(ToolUsingAgent):
|
||||
content=content,
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
# Execute code in REPL
|
||||
@@ -218,14 +237,13 @@ class RLMAgent(ToolUsingAgent):
|
||||
content=final_str,
|
||||
tool_results=all_tool_results,
|
||||
turns=turns,
|
||||
metadata=total_usage,
|
||||
)
|
||||
|
||||
# Feed output back as user message
|
||||
messages.append(Message(role=Role.ASSISTANT, content=content))
|
||||
feedback = (
|
||||
f"REPL Output: {output}"
|
||||
if output
|
||||
else "REPL Output: (no output)"
|
||||
f"REPL Output: {output}" if output else "REPL Output: (no output)"
|
||||
)
|
||||
messages.append(Message(role=Role.USER, content=feedback))
|
||||
|
||||
@@ -236,7 +254,9 @@ class RLMAgent(ToolUsingAgent):
|
||||
else:
|
||||
final_content = ""
|
||||
|
||||
return self._max_turns_result(all_tool_results, turns, content=final_content)
|
||||
return self._max_turns_result(
|
||||
all_tool_results, turns, content=final_content, metadata=total_usage
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Sub-LM callbacks
|
||||
@@ -269,19 +289,23 @@ class RLMAgent(ToolUsingAgent):
|
||||
)
|
||||
for i, tc in enumerate(raw_tool_calls)
|
||||
]
|
||||
messages.append(Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.ASSISTANT,
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
)
|
||||
)
|
||||
for tc in tool_calls:
|
||||
tr = self._executor.execute(tc)
|
||||
messages.append(Message(
|
||||
role=Role.TOOL,
|
||||
content=tr.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
))
|
||||
messages.append(
|
||||
Message(
|
||||
role=Role.TOOL,
|
||||
content=tr.content,
|
||||
tool_call_id=tc.id,
|
||||
name=tc.name,
|
||||
)
|
||||
)
|
||||
followup = self._engine.generate(
|
||||
messages,
|
||||
model=self._sub_model,
|
||||
|
||||
@@ -109,14 +109,13 @@ class TelegramChannel(BaseChannel):
|
||||
import httpx
|
||||
|
||||
url = f"https://api.telegram.org/bot{self._token}/sendMessage"
|
||||
chat_id = conversation_id or channel
|
||||
payload: Dict[str, Any] = {
|
||||
"chat_id": channel,
|
||||
"chat_id": chat_id,
|
||||
"text": content,
|
||||
}
|
||||
if self._parse_mode:
|
||||
payload["parse_mode"] = self._parse_mode
|
||||
if conversation_id:
|
||||
payload["reply_to_message_id"] = conversation_id
|
||||
|
||||
resp = httpx.post(url, json=payload, timeout=10.0)
|
||||
if resp.status_code < 300:
|
||||
@@ -164,6 +163,19 @@ class TelegramChannel(BaseChannel):
|
||||
message_id=str(msg.message_id),
|
||||
conversation_id=str(msg.chat.id),
|
||||
)
|
||||
# Enforce allow-list when configured
|
||||
if self._allowed_chat_ids:
|
||||
_allowed = {
|
||||
cid.strip()
|
||||
for cid in self._allowed_chat_ids.split(",")
|
||||
if cid.strip()
|
||||
}
|
||||
if cm.conversation_id not in _allowed:
|
||||
logger.debug(
|
||||
"Ignoring message from unlisted chat %s",
|
||||
cm.conversation_id,
|
||||
)
|
||||
return
|
||||
for handler in self._handlers:
|
||||
try:
|
||||
handler(cm)
|
||||
@@ -181,7 +193,7 @@ class TelegramChannel(BaseChannel):
|
||||
)
|
||||
|
||||
app.add_handler(MessageHandler(filters.TEXT, _handle_msg))
|
||||
app.run_polling(stop_signals=None)
|
||||
app.run_polling(stop_signals=None, drop_pending_updates=True)
|
||||
except Exception:
|
||||
logger.debug("Telegram poll loop error", exc_info=True)
|
||||
self._status = ChannelStatus.ERROR
|
||||
|
||||
@@ -27,7 +27,15 @@ from openjarvis.core.registry import ChannelRegistry
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Path to the bundled bridge shipped inside the package.
|
||||
# In editable installs this lives next to this file; in wheel installs
|
||||
# it is placed under _node_modules/ to avoid namespace package conflicts.
|
||||
_BRIDGE_SRC = Path(__file__).resolve().parent / "whatsapp_baileys_bridge"
|
||||
if not _BRIDGE_SRC.exists():
|
||||
_BRIDGE_SRC = (
|
||||
Path(__file__).resolve().parents[2]
|
||||
/ "_node_modules"
|
||||
/ "whatsapp_baileys_bridge"
|
||||
)
|
||||
|
||||
# Default runtime directory (npm install + auth state).
|
||||
_DEFAULT_RUNTIME_DIR = Path.home() / ".openjarvis" / "whatsapp_baileys_bridge"
|
||||
|
||||
@@ -12,10 +12,12 @@ from openjarvis.cli.bench_cmd import bench
|
||||
from openjarvis.cli.channel_cmd import channel
|
||||
from openjarvis.cli.chat_cmd import chat
|
||||
from openjarvis.cli.compose_cmd import compose
|
||||
from openjarvis.cli.config_cmd import config
|
||||
from openjarvis.cli.daemon_cmd import restart, start, status, stop
|
||||
from openjarvis.cli.doctor_cmd import doctor
|
||||
from openjarvis.cli.eval_cmd import eval_group
|
||||
from openjarvis.cli.feedback_cmd import feedback_group
|
||||
from openjarvis.cli.gateway_cmd import gateway
|
||||
from openjarvis.cli.host_cmd import host
|
||||
from openjarvis.cli.init_cmd import init
|
||||
from openjarvis.cli.memory_cmd import memory
|
||||
@@ -23,10 +25,13 @@ from openjarvis.cli.model import model
|
||||
from openjarvis.cli.operators_cmd import operators
|
||||
from openjarvis.cli.optimize_cmd import optimize_group
|
||||
from openjarvis.cli.quickstart_cmd import quickstart
|
||||
from openjarvis.cli.registry_cmd import registry
|
||||
from openjarvis.cli.scan_cmd import scan
|
||||
from openjarvis.cli.scheduler_cmd import scheduler
|
||||
from openjarvis.cli.serve import serve
|
||||
from openjarvis.cli.skill_cmd import skill
|
||||
from openjarvis.cli.telemetry_cmd import telemetry
|
||||
from openjarvis.cli.tool_cmd import tool
|
||||
from openjarvis.cli.vault_cmd import vault
|
||||
from openjarvis.cli.workflow_cmd import workflow
|
||||
|
||||
@@ -79,6 +84,11 @@ cli.add_command(quickstart, "quickstart")
|
||||
cli.add_command(optimize_group, "optimize")
|
||||
cli.add_command(feedback_group, "feedback")
|
||||
cli.add_command(compose, "compose")
|
||||
cli.add_command(gateway, "gateway")
|
||||
cli.add_command(tool, "tool")
|
||||
cli.add_command(registry, "registry")
|
||||
cli.add_command(config, "config")
|
||||
cli.add_command(scan, "scan")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
|
||||
@@ -24,6 +24,26 @@ def _get_manager():
|
||||
return AgentManager(db_path=db_path)
|
||||
|
||||
|
||||
def _resolve_agent_id(manager, agent_id_or_name: str) -> str:
|
||||
"""Resolve an agent name or ID to the actual agent ID.
|
||||
|
||||
Accepts either the hex ID or the agent name. Raises SystemExit if
|
||||
no matching agent is found.
|
||||
"""
|
||||
# Try direct ID lookup first
|
||||
agent = manager.get_agent(agent_id_or_name)
|
||||
if agent is not None:
|
||||
return agent["id"]
|
||||
|
||||
# Fall back to name lookup
|
||||
for a in manager.list_agents(include_archived=True):
|
||||
if a["name"] == agent_id_or_name:
|
||||
return a["id"]
|
||||
|
||||
click.echo(f"Agent not found: {agent_id_or_name}", err=True)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
@click.group("agents")
|
||||
def agent() -> None:
|
||||
"""Manage persistent agents — create, inspect, chat, bind channels."""
|
||||
@@ -680,6 +700,7 @@ def errors():
|
||||
def ask(agent_id, message):
|
||||
"""Ask an agent a question (immediate response)."""
|
||||
manager = _get_manager()
|
||||
agent_id = _resolve_agent_id(manager, agent_id)
|
||||
manager.send_message(agent_id, message, mode="immediate")
|
||||
click.echo("Asking agent...")
|
||||
_, executor, _ = _get_scheduler_and_executor()
|
||||
@@ -701,6 +722,7 @@ def ask(agent_id, message):
|
||||
def instruct(agent_id, message):
|
||||
"""Queue an instruction for the agent's next tick."""
|
||||
manager = _get_manager()
|
||||
agent_id = _resolve_agent_id(manager, agent_id)
|
||||
msg = manager.send_message(agent_id, message, mode="queued")
|
||||
click.echo(f"Instruction queued (ID: {msg['id'][:8]})")
|
||||
|
||||
|
||||
+121
-39
@@ -44,7 +44,8 @@ def _get_memory_backend(config):
|
||||
|
||||
if key == "sqlite":
|
||||
backend = MemoryRegistry.create(
|
||||
key, db_path=config.memory.db_path,
|
||||
key,
|
||||
db_path=config.memory.db_path,
|
||||
)
|
||||
else:
|
||||
backend = MemoryRegistry.create(key)
|
||||
@@ -106,8 +107,7 @@ def _run_agent(
|
||||
|
||||
if not AgentRegistry.contains(agent_name):
|
||||
raise click.ClickException(
|
||||
f"Unknown agent: {agent_name}. "
|
||||
f"Available: {', '.join(AgentRegistry.keys())}"
|
||||
f"Unknown agent: {agent_name}. Available: {', '.join(AgentRegistry.keys())}"
|
||||
)
|
||||
|
||||
agent_cls = AgentRegistry.get(agent_name)
|
||||
@@ -117,6 +117,7 @@ def _run_agent(
|
||||
if tool_names:
|
||||
# Trigger tool registration
|
||||
import openjarvis.tools # noqa: F401
|
||||
|
||||
tools = _build_tools(tool_names, config, engine, model_name)
|
||||
|
||||
# Build agent with appropriate kwargs
|
||||
@@ -149,7 +150,10 @@ def _run_agent(
|
||||
max_context_tokens=config.memory.context_max_tokens,
|
||||
)
|
||||
context_messages = inject_context(
|
||||
query_text, [], backend, config=ctx_cfg,
|
||||
query_text,
|
||||
[],
|
||||
backend,
|
||||
config=ctx_cfg,
|
||||
)
|
||||
for msg in context_messages:
|
||||
ctx.conversation.add(msg)
|
||||
@@ -165,12 +169,11 @@ def _print_profile(
|
||||
engine_name: str,
|
||||
model_name: str,
|
||||
console: Console,
|
||||
complexity_result=None,
|
||||
) -> None:
|
||||
"""Print an inference telemetry profile table from EventBus history."""
|
||||
# Collect all INFERENCE_END events (agents may fire multiple)
|
||||
inf_events = [
|
||||
e for e in bus.history if e.event_type == EventType.INFERENCE_END
|
||||
]
|
||||
inf_events = [e for e in bus.history if e.event_type == EventType.INFERENCE_END]
|
||||
if not inf_events:
|
||||
console.print("[dim]No inference telemetry recorded.[/dim]")
|
||||
return
|
||||
@@ -185,13 +188,15 @@ def _print_profile(
|
||||
for e in inf_events
|
||||
)
|
||||
total_prompt = sum(
|
||||
e.data.get("usage", {}).get("prompt_tokens", 0)
|
||||
for e in inf_events
|
||||
e.data.get("usage", {}).get("prompt_tokens", 0) for e in inf_events
|
||||
)
|
||||
total_energy = sum(e.data.get("energy_joules", 0.0) for e in inf_events)
|
||||
avg_power = 0.0
|
||||
power_vals = [e.data.get("power_watts", 0.0) for e in inf_events
|
||||
if e.data.get("power_watts", 0.0) > 0]
|
||||
power_vals = [
|
||||
e.data.get("power_watts", 0.0)
|
||||
for e in inf_events
|
||||
if e.data.get("power_watts", 0.0) > 0
|
||||
]
|
||||
if power_vals:
|
||||
avg_power = sum(power_vals) / len(power_vals)
|
||||
|
||||
@@ -227,6 +232,12 @@ def _print_profile(
|
||||
def _row(label: str, val: str) -> None:
|
||||
table.add_row(label, val)
|
||||
|
||||
if complexity_result is not None:
|
||||
_row("Complexity score", f"{complexity_result.score:.3f}")
|
||||
_row("Complexity tier", complexity_result.tier)
|
||||
_row("Suggested max tokens", str(complexity_result.suggested_max_tokens))
|
||||
_row("", "") # separator
|
||||
|
||||
_row("Wall time", f"{wall_seconds:.3f} s")
|
||||
_row("Inference calls", str(total_calls))
|
||||
_row("Total latency", f"{total_latency:.3f} s")
|
||||
@@ -277,29 +288,42 @@ def _print_profile(
|
||||
@click.option("-m", "--model", "model_name", default=None, help="Model to use.")
|
||||
@click.option("-e", "--engine", "engine_key", default=None, help="Engine backend.")
|
||||
@click.option(
|
||||
"-t", "--temperature", default=None, type=float,
|
||||
"-t",
|
||||
"--temperature",
|
||||
default=None,
|
||||
type=float,
|
||||
help="Sampling temperature (default: from config).",
|
||||
)
|
||||
@click.option(
|
||||
"--max-tokens", default=None, type=int,
|
||||
"--max-tokens",
|
||||
default=None,
|
||||
type=int,
|
||||
help="Max tokens to generate (default: from config).",
|
||||
)
|
||||
@click.option("--json", "output_json", is_flag=True, help="Output raw JSON result.")
|
||||
@click.option("--no-stream", is_flag=True, help="Disable streaming (sync mode).")
|
||||
@click.option(
|
||||
"--no-context", is_flag=True,
|
||||
"--no-context",
|
||||
is_flag=True,
|
||||
help="Disable memory context injection.",
|
||||
)
|
||||
@click.option(
|
||||
"-a", "--agent", "agent_name", default=None,
|
||||
"-a",
|
||||
"--agent",
|
||||
"agent_name",
|
||||
default=None,
|
||||
help="Agent to use (simple, orchestrator).",
|
||||
)
|
||||
@click.option(
|
||||
"--tools", "tool_names", default=None,
|
||||
"--tools",
|
||||
"tool_names",
|
||||
default=None,
|
||||
help="Comma-separated tool names to enable (e.g. calculator,think).",
|
||||
)
|
||||
@click.option(
|
||||
"--profile", "enable_profile", is_flag=True,
|
||||
"--profile",
|
||||
"enable_profile",
|
||||
is_flag=True,
|
||||
help="Print inference telemetry profile (latency, tokens, energy, IPW).",
|
||||
)
|
||||
def ask(
|
||||
@@ -324,12 +348,30 @@ def ask(
|
||||
# Load config
|
||||
config = load_config()
|
||||
|
||||
# Track whether the user explicitly set --max-tokens
|
||||
user_set_max_tokens = max_tokens is not None
|
||||
|
||||
# Fall back to config values for generation params
|
||||
if temperature is None:
|
||||
temperature = config.intelligence.temperature
|
||||
if max_tokens is None:
|
||||
max_tokens = config.intelligence.max_tokens
|
||||
|
||||
# Run complexity analysis on the query
|
||||
from openjarvis.learning.routing.complexity import (
|
||||
ComplexityResult,
|
||||
adjust_tokens_for_model,
|
||||
score_complexity,
|
||||
)
|
||||
|
||||
complexity_result: ComplexityResult = score_complexity(query_text)
|
||||
logger.debug(
|
||||
"Complexity analysis: score=%.3f tier=%s suggested_max_tokens=%d",
|
||||
complexity_result.score,
|
||||
complexity_result.tier,
|
||||
complexity_result.suggested_max_tokens,
|
||||
)
|
||||
|
||||
# Set up telemetry
|
||||
bus = EventBus(record_history=True)
|
||||
telem_store: TelemetryStore | None = None
|
||||
@@ -352,7 +394,10 @@ def ask(
|
||||
" [cyan]ollama serve[/cyan] — start Ollama\n"
|
||||
" [cyan]vllm serve <model>[/cyan] — start vLLM\n"
|
||||
" [cyan]llama-server -m <gguf>[/cyan] — start llama.cpp\n\n"
|
||||
"Or set OPENAI_API_KEY / ANTHROPIC_API_KEY for cloud inference."
|
||||
"Or set OPENAI_API_KEY / ANTHROPIC_API_KEY for cloud inference.\n\n"
|
||||
"[dim]To use a remote engine:[/dim]\n"
|
||||
" [cyan]jarvis config set engine.ollama.host http://<remote-ip>:11434[/cyan]\n"
|
||||
" [dim]or[/dim] [cyan]export OLLAMA_HOST=http://<remote-ip>:11434[/cyan]"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
@@ -360,6 +405,7 @@ def ask(
|
||||
|
||||
# Apply security guardrails
|
||||
from openjarvis.security import setup_security
|
||||
|
||||
sec = setup_security(config, engine, bus)
|
||||
engine = sec.engine
|
||||
|
||||
@@ -397,6 +443,21 @@ def ask(
|
||||
console.print("[red]No model available on engine.[/red]")
|
||||
sys.exit(1)
|
||||
|
||||
# Apply complexity-suggested token budget when user didn't override.
|
||||
# Use at least the config default so we never reduce tokens below what
|
||||
# the user would have gotten without the analyzer.
|
||||
if not user_set_max_tokens:
|
||||
suggested = adjust_tokens_for_model(
|
||||
complexity_result.suggested_max_tokens,
|
||||
model_name,
|
||||
)
|
||||
max_tokens = max(suggested, config.intelligence.max_tokens)
|
||||
logger.debug(
|
||||
"Using complexity-suggested max_tokens=%d (model=%s)",
|
||||
max_tokens,
|
||||
model_name,
|
||||
)
|
||||
|
||||
# Agent mode
|
||||
if agent_name is not None:
|
||||
parsed_tools = resolve_tool_names(
|
||||
@@ -406,8 +467,15 @@ def ask(
|
||||
)
|
||||
try:
|
||||
result = _run_agent(
|
||||
agent_name, query_text, engine, model_name,
|
||||
parsed_tools, config, bus, temperature, max_tokens,
|
||||
agent_name,
|
||||
query_text,
|
||||
engine,
|
||||
model_name,
|
||||
parsed_tools,
|
||||
config,
|
||||
bus,
|
||||
temperature,
|
||||
max_tokens,
|
||||
capability_policy=sec.capability_policy,
|
||||
)
|
||||
except EngineConnectionError as exc:
|
||||
@@ -416,25 +484,34 @@ def ask(
|
||||
sys.exit(1)
|
||||
|
||||
if output_json:
|
||||
click.echo(json_mod.dumps({
|
||||
"content": result.content,
|
||||
"turns": result.turns,
|
||||
"tool_results": [
|
||||
click.echo(
|
||||
json_mod.dumps(
|
||||
{
|
||||
"tool_name": tr.tool_name,
|
||||
"content": tr.content,
|
||||
"success": tr.success,
|
||||
}
|
||||
for tr in result.tool_results
|
||||
],
|
||||
}, indent=2))
|
||||
"content": result.content,
|
||||
"turns": result.turns,
|
||||
"tool_results": [
|
||||
{
|
||||
"tool_name": tr.tool_name,
|
||||
"content": tr.content,
|
||||
"success": tr.success,
|
||||
}
|
||||
for tr in result.tool_results
|
||||
],
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
else:
|
||||
click.echo(result.content)
|
||||
|
||||
if enable_profile:
|
||||
_print_profile(
|
||||
bus, time.monotonic() - wall_start,
|
||||
engine_name, model_name, console,
|
||||
bus,
|
||||
time.monotonic() - wall_start,
|
||||
engine_name,
|
||||
model_name,
|
||||
console,
|
||||
complexity_result=complexity_result,
|
||||
)
|
||||
|
||||
if telem_store is not None:
|
||||
@@ -454,17 +531,18 @@ def ask(
|
||||
ContextConfig,
|
||||
inject_context,
|
||||
)
|
||||
|
||||
backend = _get_memory_backend(config)
|
||||
if backend is not None:
|
||||
ctx_cfg = ContextConfig(
|
||||
top_k=config.memory.context_top_k,
|
||||
min_score=config.memory.context_min_score,
|
||||
max_context_tokens=(
|
||||
config.memory.context_max_tokens
|
||||
),
|
||||
max_context_tokens=(config.memory.context_max_tokens),
|
||||
)
|
||||
messages = inject_context(
|
||||
query_text, messages, backend,
|
||||
query_text,
|
||||
messages,
|
||||
backend,
|
||||
config=ctx_cfg,
|
||||
)
|
||||
except Exception as exc:
|
||||
@@ -492,8 +570,12 @@ def ask(
|
||||
|
||||
if enable_profile:
|
||||
_print_profile(
|
||||
bus, time.monotonic() - wall_start,
|
||||
engine_name, model_name, console,
|
||||
bus,
|
||||
time.monotonic() - wall_start,
|
||||
engine_name,
|
||||
model_name,
|
||||
console,
|
||||
complexity_result=complexity_result,
|
||||
)
|
||||
|
||||
# Cleanup
|
||||
|
||||
@@ -0,0 +1,375 @@
|
||||
"""``jarvis config`` — configuration inspection commands."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import httpx
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.syntax import Syntax
|
||||
from rich.table import Table
|
||||
|
||||
|
||||
@click.group()
|
||||
def config() -> None:
|
||||
"""Inspect configuration — show loaded settings, hardware, and config files."""
|
||||
|
||||
|
||||
def _get_config_path(path: str | None) -> Path:
|
||||
"""Determine the config path from argument or environment."""
|
||||
from openjarvis.core.config import DEFAULT_CONFIG_PATH
|
||||
|
||||
if path:
|
||||
return Path(path)
|
||||
return Path(os.environ.get("OPENJARVIS_CONFIG", DEFAULT_CONFIG_PATH))
|
||||
|
||||
|
||||
def _show_hardware_info(console: Console, show_recommendations: bool = True) -> None:
|
||||
"""Display detected hardware information."""
|
||||
from openjarvis.core.config import detect_hardware, recommend_engine
|
||||
|
||||
hardware = detect_hardware()
|
||||
|
||||
console.print("\n[bold]Detected Hardware[/bold]")
|
||||
|
||||
hardware_table = Table(show_header=True, header_style="cyan")
|
||||
hardware_table.add_column("Component", style="green")
|
||||
hardware_table.add_column("Value", style="white")
|
||||
|
||||
# Platform
|
||||
hardware_table.add_row("Platform", str(hardware.platform))
|
||||
|
||||
# CPU
|
||||
if hardware.cpu_brand:
|
||||
hardware_table.add_row("CPU", hardware.cpu_brand)
|
||||
hardware_table.add_row("CPU Count", str(hardware.cpu_count))
|
||||
hardware_table.add_row("RAM", f"{hardware.ram_gb:.1f} GB")
|
||||
|
||||
# GPU
|
||||
if hardware.gpu:
|
||||
hardware_table.add_row("GPU Vendor", hardware.gpu.vendor)
|
||||
hardware_table.add_row("GPU Model", hardware.gpu.name)
|
||||
hardware_table.add_row("GPU VRAM", f"{hardware.gpu.vram_gb:.1f} GB")
|
||||
hardware_table.add_row("GPU Count", str(hardware.gpu.count))
|
||||
|
||||
console.print(hardware_table)
|
||||
|
||||
# Show recommended engine
|
||||
if show_recommendations:
|
||||
recommended = recommend_engine(hardware)
|
||||
console.print(f"\n[bold]Recommended Engine:[/bold] [cyan]{recommended}[/cyan]")
|
||||
|
||||
|
||||
def _show_config_template(console: Console, config_path: Path) -> None:
|
||||
"""Show default config template when config file doesn't exist."""
|
||||
from openjarvis.core.config import (
|
||||
DEFAULT_CONFIG_DIR,
|
||||
detect_hardware,
|
||||
generate_default_toml,
|
||||
)
|
||||
|
||||
console.print(f"[yellow]Config file not found: {config_path}[/yellow]")
|
||||
config_location = str(DEFAULT_CONFIG_DIR / "config.toml")
|
||||
msg = f"Create one at {config_location} or use --path to specify a location."
|
||||
console.print(f"[dim]{msg}[/dim]")
|
||||
|
||||
console.print("\n[bold]Default Configuration Template:[/bold]")
|
||||
hw = detect_hardware()
|
||||
template = generate_default_toml(hw)
|
||||
syntax = Syntax(template, "toml", theme="monokai", line_numbers=True)
|
||||
console.print(Panel(syntax, border_style="dim"))
|
||||
|
||||
|
||||
def _show_loaded_config(console: Console, config_path: Path, as_json: bool) -> None:
|
||||
"""Show the loaded effective configuration from config.toml."""
|
||||
from openjarvis.core.config import load_config
|
||||
|
||||
console.print(f"[dim]Loading config from: {config_path}[/dim]")
|
||||
|
||||
if config_path.exists():
|
||||
config = load_config(config_path)
|
||||
|
||||
if as_json:
|
||||
# Convert the dataclass to a dict and output as JSON
|
||||
from dataclasses import fields, is_dataclass
|
||||
|
||||
def convert(obj):
|
||||
if obj is None:
|
||||
return None
|
||||
if isinstance(obj, (str, int, float, bool)):
|
||||
return obj
|
||||
if isinstance(obj, (list, tuple)):
|
||||
return [convert(item) for item in obj]
|
||||
if isinstance(obj, dict):
|
||||
return {k: convert(v) for k, v in obj.items()}
|
||||
if is_dataclass(obj):
|
||||
result = {}
|
||||
for field in fields(obj):
|
||||
field_value = getattr(obj, field.name)
|
||||
if field_value is not None:
|
||||
result[field.name] = convert(field_value)
|
||||
return result
|
||||
# Fallback for other types
|
||||
return str(obj)
|
||||
|
||||
config_dict = convert(config)
|
||||
# Write JSON to stdout so it is pipeable
|
||||
stdout_console = Console()
|
||||
stdout_console.print_json(json.dumps(config_dict, indent=2, default=str))
|
||||
else:
|
||||
# Show as formatted table
|
||||
console.print("[bold]Engine Configuration[/bold]")
|
||||
console.print(f" Default Engine: [cyan]{config.engine.default}[/cyan]")
|
||||
|
||||
console.print("\n[bold]Intelligence Configuration[/bold]")
|
||||
console.print(
|
||||
f" Default Model: [cyan]{config.intelligence.default_model}[/cyan]"
|
||||
)
|
||||
fallback = config.intelligence.fallback_model or "N/A"
|
||||
console.print(f" Fallback Model: [cyan]{fallback}[/cyan]")
|
||||
console.print(
|
||||
f" Temperature: [cyan]{config.intelligence.temperature}[/cyan]"
|
||||
)
|
||||
console.print(
|
||||
f" Max Tokens: [cyan]{config.intelligence.max_tokens}[/cyan]"
|
||||
)
|
||||
|
||||
console.print("\n[bold]Agent Configuration[/bold]")
|
||||
console.print(f" Default Agent: [cyan]{config.agent.default_agent}[/cyan]")
|
||||
console.print(f" Max Turns: [cyan]{config.agent.max_turns}[/cyan]")
|
||||
console.print(f" Tools: [cyan]{config.agent.tools or 'none'}[/cyan]")
|
||||
ctx_mem = config.agent.context_from_memory
|
||||
console.print(f" Context from Memory: [cyan]{ctx_mem}[/cyan]")
|
||||
|
||||
# Show hardware info
|
||||
_show_hardware_info(console)
|
||||
|
||||
else:
|
||||
_show_config_template(console, config_path)
|
||||
|
||||
|
||||
def _show_toml_config(console: Console, config_path: Path) -> None:
|
||||
"""Show the raw TOML configuration file content with syntax highlighting."""
|
||||
console.print(f"[dim]Loading config from: {config_path}[/dim]")
|
||||
|
||||
if config_path.exists():
|
||||
config_content = config_path.read_text()
|
||||
syntax = Syntax(config_content, "toml", theme="monokai", line_numbers=True)
|
||||
console.print(Panel(syntax, title="Config File", border_style="cyan"))
|
||||
else:
|
||||
_show_config_template(console, config_path)
|
||||
|
||||
|
||||
def _show_json_config(console: Console, config_path: Path) -> None:
|
||||
"""Show the parsed TOML configuration as JSON."""
|
||||
console.print(f"[dim]Loading config from: {config_path}[/dim]")
|
||||
|
||||
if config_path.exists():
|
||||
config_content = config_path.read_text()
|
||||
|
||||
try:
|
||||
import tomllib # Python 3.11+
|
||||
except ModuleNotFoundError:
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
|
||||
config_dict = tomllib.loads(config_content)
|
||||
# Write JSON to stdout so it is pipeable
|
||||
stdout_console = Console()
|
||||
stdout_console.print_json(json.dumps(config_dict, indent=2))
|
||||
else:
|
||||
_show_config_template(console, config_path)
|
||||
|
||||
|
||||
def _show_hardware(console: Console) -> None:
|
||||
"""Show detected hardware information with recommended engine and model."""
|
||||
from openjarvis.core.config import (
|
||||
detect_hardware,
|
||||
recommend_engine,
|
||||
recommend_model,
|
||||
)
|
||||
|
||||
hardware = detect_hardware()
|
||||
_show_hardware_info(console, show_recommendations=False)
|
||||
|
||||
# Show recommended engine
|
||||
recommended_engine = recommend_engine(hardware)
|
||||
console.print(
|
||||
f"\n[bold]Recommended Engine:[/bold] [cyan]{recommended_engine}[/cyan]"
|
||||
)
|
||||
|
||||
# Show recommended model
|
||||
console.print("\n[bold]Model Recommendations[/bold]")
|
||||
recommended_model = recommend_model(hardware, recommended_engine)
|
||||
console.print(f" Recommended Model: [cyan]{recommended_model}[/cyan]")
|
||||
|
||||
|
||||
# Nested group for show sub-commands
|
||||
@click.group(invoke_without_command=True)
|
||||
@click.option("--path", "-p", default=None, help="Explicit config file path")
|
||||
@click.option("--json", "as_json", is_flag=True, default=False, help="Output as JSON")
|
||||
@click.pass_context
|
||||
def show_group(ctx: click.Context, path: str | None, as_json: bool) -> None:
|
||||
"""Show configuration details."""
|
||||
# Default to 'loaded' if no subcommand is invoked
|
||||
if ctx.invoked_subcommand is None:
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
config_path = _get_config_path(path)
|
||||
_show_loaded_config(console, config_path, as_json)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
@show_group.command()
|
||||
@click.option("--path", "-p", default=None, help="Explicit config file path")
|
||||
@click.option("--json", "as_json", is_flag=True, default=False, help="Output as JSON")
|
||||
def loaded(path: str | None, as_json: bool) -> None:
|
||||
"""Show the loaded effective configuration from config.toml."""
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
config_path = _get_config_path(path)
|
||||
_show_loaded_config(console, config_path, as_json)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
@show_group.command()
|
||||
@click.option("--path", "-p", default=None, help="Explicit config file path")
|
||||
def toml(path: str | None) -> None:
|
||||
"""Show the raw TOML configuration file content with syntax highlighting."""
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
config_path = _get_config_path(path)
|
||||
_show_toml_config(console, config_path)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
@show_group.command("json")
|
||||
@click.option("--path", "-p", default=None, help="Explicit config file path")
|
||||
def as_json(path: str | None) -> None:
|
||||
"""Show the parsed TOML configuration as JSON."""
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
config_path = _get_config_path(path)
|
||||
_show_json_config(console, config_path)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
@show_group.command()
|
||||
def hardware() -> None:
|
||||
"""Show detected hardware information with recommended engine and model."""
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
_show_hardware(console)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
# Register the show group under config
|
||||
config.add_command(show_group, "show")
|
||||
|
||||
|
||||
def _probe_engine_host(url: str, console: Console) -> None:
|
||||
"""Probe an engine host URL and print reachability status."""
|
||||
try:
|
||||
resp = httpx.get(url.rstrip("/") + "/", timeout=2.0)
|
||||
if resp.status_code < 500:
|
||||
console.print(f" [green]Reachable[/green] ({url})")
|
||||
else:
|
||||
console.print(
|
||||
f" [yellow]Warning:[/yellow] Host returned status "
|
||||
f"{resp.status_code} — config saved anyway."
|
||||
)
|
||||
except Exception:
|
||||
console.print(
|
||||
f" [yellow]Warning:[/yellow] Host unreachable ({url}) "
|
||||
f"— config saved anyway."
|
||||
)
|
||||
|
||||
|
||||
def _coerce_value(value: str, target_type: type) -> object:
|
||||
"""Coerce a CLI string value to the target Python type."""
|
||||
if target_type is bool:
|
||||
low = value.lower()
|
||||
if low in ("true", "1", "yes"):
|
||||
return True
|
||||
if low in ("false", "0", "no"):
|
||||
return False
|
||||
raise ValueError(
|
||||
f"Invalid boolean value: {value!r} (expected: true/false, yes/no, 1/0)"
|
||||
)
|
||||
if target_type is int:
|
||||
return int(value)
|
||||
if target_type is float:
|
||||
return float(value)
|
||||
return value
|
||||
|
||||
|
||||
@click.command("set")
|
||||
@click.argument("key")
|
||||
@click.argument("value")
|
||||
def set_config(key: str, value: str) -> None:
|
||||
"""Set a configuration value (e.g. jarvis config set engine.ollama.host URL)."""
|
||||
import tomlkit
|
||||
|
||||
from openjarvis.core.config import DEFAULT_CONFIG_DIR, validate_config_key
|
||||
|
||||
console = Console(stderr=True)
|
||||
|
||||
# Validate key
|
||||
try:
|
||||
target_type = validate_config_key(key)
|
||||
except ValueError as exc:
|
||||
console.print(f"[red]Error:[/red] {exc}")
|
||||
raise SystemExit(1)
|
||||
|
||||
# Coerce value
|
||||
try:
|
||||
typed_value = _coerce_value(value, target_type)
|
||||
except (ValueError, TypeError) as exc:
|
||||
console.print(
|
||||
f"[red]Error:[/red] Cannot convert {value!r} to "
|
||||
f"{target_type.__name__}: {exc}"
|
||||
)
|
||||
raise SystemExit(1)
|
||||
|
||||
# Load or create TOML document
|
||||
config_path = Path(
|
||||
os.environ.get("OPENJARVIS_CONFIG", DEFAULT_CONFIG_DIR / "config.toml")
|
||||
)
|
||||
if config_path.exists():
|
||||
doc = tomlkit.parse(config_path.read_text())
|
||||
else:
|
||||
doc = tomlkit.document()
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Set nested key
|
||||
parts = key.split(".")
|
||||
current = doc
|
||||
for part in parts[:-1]:
|
||||
if part not in current:
|
||||
current.add(part, tomlkit.table())
|
||||
current = current[part]
|
||||
current[parts[-1]] = typed_value
|
||||
|
||||
# Write back
|
||||
config_path.write_text(tomlkit.dumps(doc))
|
||||
|
||||
console.print(f"[green]Set[/green] {key} = {value!r}")
|
||||
|
||||
# Probe engine host if applicable
|
||||
if re.match(r"^engine\.\w+\.host$", key):
|
||||
_probe_engine_host(value, console)
|
||||
|
||||
|
||||
config.add_command(set_config, "set")
|
||||
|
||||
|
||||
__all__ = ["config"]
|
||||
@@ -0,0 +1,108 @@
|
||||
"""``jarvis gateway start|stop|status|logs`` — multi-channel gateway management."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
|
||||
|
||||
@click.group()
|
||||
def gateway() -> None:
|
||||
"""Manage the OpenJarvis multi-channel gateway."""
|
||||
|
||||
|
||||
@gateway.command()
|
||||
@click.option(
|
||||
"--install",
|
||||
is_flag=True,
|
||||
help="Generate and enable systemd/launchd service",
|
||||
)
|
||||
def start(install: bool) -> None:
|
||||
"""Start the gateway daemon."""
|
||||
if install:
|
||||
import platform as plat
|
||||
|
||||
from openjarvis.daemon.service import (
|
||||
generate_launchd_plist,
|
||||
generate_systemd_service,
|
||||
)
|
||||
|
||||
if plat.system() == "Darwin":
|
||||
plist_path = (
|
||||
Path.home()
|
||||
/ "Library/LaunchAgents/com.openjarvis.gateway.plist"
|
||||
)
|
||||
generate_launchd_plist(plist_path)
|
||||
click.echo(f"Wrote {plist_path}")
|
||||
subprocess.run(
|
||||
["launchctl", "load", str(plist_path)], check=False,
|
||||
)
|
||||
else:
|
||||
service_path = (
|
||||
Path.home()
|
||||
/ ".config/systemd/user/openjarvis-gateway.service"
|
||||
)
|
||||
generate_systemd_service(service_path)
|
||||
click.echo(f"Wrote {service_path}")
|
||||
subprocess.run(
|
||||
["systemctl", "--user", "daemon-reload"], check=False,
|
||||
)
|
||||
subprocess.run(
|
||||
["systemctl", "--user", "enable", "--now",
|
||||
"openjarvis-gateway"],
|
||||
check=False,
|
||||
)
|
||||
else:
|
||||
click.echo("Starting OpenJarvis gateway (foreground)...")
|
||||
click.echo("Gateway started. Press Ctrl+C to stop.")
|
||||
|
||||
|
||||
@gateway.command()
|
||||
def stop() -> None:
|
||||
"""Stop the gateway daemon."""
|
||||
import platform as plat
|
||||
|
||||
if plat.system() == "Darwin":
|
||||
subprocess.run(
|
||||
["launchctl", "remove", "com.openjarvis.gateway"],
|
||||
check=False,
|
||||
)
|
||||
else:
|
||||
subprocess.run(
|
||||
["systemctl", "--user", "stop", "openjarvis-gateway"],
|
||||
check=False,
|
||||
)
|
||||
click.echo("Gateway stopped.")
|
||||
|
||||
|
||||
@gateway.command()
|
||||
def status() -> None:
|
||||
"""Check gateway status."""
|
||||
import platform as plat
|
||||
|
||||
if plat.system() == "Darwin":
|
||||
subprocess.run(
|
||||
["launchctl", "list", "com.openjarvis.gateway"],
|
||||
check=False,
|
||||
)
|
||||
else:
|
||||
subprocess.run(
|
||||
["systemctl", "--user", "status", "openjarvis-gateway"],
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
@gateway.command()
|
||||
def logs() -> None:
|
||||
"""View gateway logs."""
|
||||
import platform as plat
|
||||
|
||||
if plat.system() == "Darwin":
|
||||
click.echo("Check ~/Library/Logs/com.openjarvis.gateway.log")
|
||||
else:
|
||||
subprocess.run(
|
||||
["journalctl", "--user", "-u", "openjarvis-gateway", "-f"],
|
||||
check=False,
|
||||
)
|
||||
@@ -22,7 +22,11 @@ def hint_no_engine(engine_name: Optional[str] = None) -> str:
|
||||
f"[yellow]Hint:[/yellow] Engine '{name}' is not reachable.\n"
|
||||
f" Make sure the {name} server is running.\n"
|
||||
" Run [bold]jarvis doctor[/bold] to check all engines.\n"
|
||||
" Run [bold]jarvis quickstart[/bold] for guided setup."
|
||||
" Run [bold]jarvis quickstart[/bold] for guided setup.\n"
|
||||
"\n"
|
||||
" [dim]To use a remote engine:[/dim]\n"
|
||||
f" [cyan]jarvis config set engine.{name}.host http://<remote-ip>:<port>[/cyan]\n"
|
||||
f" [dim]or[/dim] [cyan]export OLLAMA_HOST=http://<remote-ip>:11434[/cyan]"
|
||||
)
|
||||
|
||||
|
||||
|
||||
+152
-15
@@ -6,14 +6,18 @@ from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import click
|
||||
import httpx
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
|
||||
from openjarvis.cli.model import find_model_spec, hf_download, ollama_pull
|
||||
from openjarvis.cli.scan_cmd import PrivacyScanner
|
||||
from openjarvis.core.config import (
|
||||
DEFAULT_CONFIG_DIR,
|
||||
DEFAULT_CONFIG_PATH,
|
||||
detect_hardware,
|
||||
estimated_download_gb,
|
||||
generate_default_toml,
|
||||
generate_minimal_toml,
|
||||
recommend_engine,
|
||||
@@ -22,8 +26,14 @@ from openjarvis.core.config import (
|
||||
|
||||
# Engines supported by ``jarvis init --engine``.
|
||||
_SUPPORTED_ENGINES = [
|
||||
"ollama", "vllm", "sglang", "llamacpp", "mlx", "lmstudio",
|
||||
"exo", "nexa",
|
||||
"ollama",
|
||||
"vllm",
|
||||
"sglang",
|
||||
"llamacpp",
|
||||
"mlx",
|
||||
"lmstudio",
|
||||
"exo",
|
||||
"nexa",
|
||||
]
|
||||
|
||||
|
||||
@@ -67,7 +77,7 @@ def _next_steps_text(engine: str, model: str = "") -> str:
|
||||
f" ollama pull {pull_model}\n"
|
||||
"\n"
|
||||
" 3. Try it out:\n"
|
||||
" jarvis ask \"Hello\"\n"
|
||||
' jarvis ask "Hello"\n'
|
||||
"\n"
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
@@ -79,7 +89,7 @@ def _next_steps_text(engine: str, model: str = "") -> str:
|
||||
" vllm serve Qwen/Qwen3-4B\n"
|
||||
"\n"
|
||||
" 2. Try it out:\n"
|
||||
" jarvis ask \"Hello\"\n"
|
||||
' jarvis ask "Hello"\n'
|
||||
"\n"
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
@@ -91,7 +101,7 @@ def _next_steps_text(engine: str, model: str = "") -> str:
|
||||
" llama-server -m path/to/model.gguf\n"
|
||||
"\n"
|
||||
" 2. Try it out:\n"
|
||||
" jarvis ask \"Hello\"\n"
|
||||
' jarvis ask "Hello"\n'
|
||||
"\n"
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
@@ -103,7 +113,7 @@ def _next_steps_text(engine: str, model: str = "") -> str:
|
||||
" python -m sglang.launch_server --model-path Qwen/Qwen3-8B\n"
|
||||
"\n"
|
||||
" 2. Try it out:\n"
|
||||
" jarvis ask \"Hello\"\n"
|
||||
' jarvis ask "Hello"\n'
|
||||
"\n"
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
@@ -115,7 +125,7 @@ def _next_steps_text(engine: str, model: str = "") -> str:
|
||||
" mlx_lm.server --model mlx-community/Qwen2.5-7B-4bit\n"
|
||||
"\n"
|
||||
" 2. Try it out:\n"
|
||||
" jarvis ask \"Hello\"\n"
|
||||
' jarvis ask "Hello"\n'
|
||||
"\n"
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
@@ -128,14 +138,80 @@ def _next_steps_text(engine: str, model: str = "") -> str:
|
||||
" 2. Load a model and start the local server (port 1234)\n"
|
||||
"\n"
|
||||
" 3. Try it out:\n"
|
||||
" jarvis ask \"Hello\"\n"
|
||||
' jarvis ask "Hello"\n'
|
||||
"\n"
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
"exo": (
|
||||
"Next steps:\n\n"
|
||||
" 1. Install and start Exo:\n"
|
||||
" pip install exo\n"
|
||||
" exo\n\n"
|
||||
" 2. Try it out:\n"
|
||||
' jarvis ask "Hello"\n\n'
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
"nexa": (
|
||||
"Next steps:\n\n"
|
||||
" 1. Install and start Nexa:\n"
|
||||
" pip install nexaai\n"
|
||||
" nexa server\n\n"
|
||||
" 2. Try it out:\n"
|
||||
' jarvis ask "Hello"\n\n'
|
||||
" Run `jarvis doctor` to verify your setup."
|
||||
),
|
||||
}
|
||||
return steps.get(engine, steps["ollama"])
|
||||
|
||||
|
||||
def _quick_privacy_check(console: Console) -> None:
|
||||
"""Run critical privacy checks and print compact summary."""
|
||||
scanner = PrivacyScanner()
|
||||
results = scanner.run_quick()
|
||||
if results:
|
||||
console.print(" [bold]Privacy check:[/bold]")
|
||||
for r in results:
|
||||
if r.status == "ok":
|
||||
console.print(f" [green]\u2713[/green] {r.message}")
|
||||
elif r.status == "warn":
|
||||
console.print(f" [yellow]![/yellow] {r.message}")
|
||||
elif r.status == "fail":
|
||||
console.print(f" [red]\u2717[/red] {r.message}")
|
||||
console.print()
|
||||
console.print(" Run [cyan]jarvis scan[/cyan] for a full environment audit.")
|
||||
|
||||
|
||||
def _do_download(engine: str, model: str, spec, console: Console) -> None:
|
||||
"""Dispatch model download based on engine type."""
|
||||
import os
|
||||
|
||||
if engine == "ollama":
|
||||
host = os.environ.get("OLLAMA_HOST", "http://localhost:11434").rstrip("/")
|
||||
ollama_pull(host, model, console)
|
||||
elif engine == "llamacpp":
|
||||
repo = spec.metadata.get("hf_repo", "")
|
||||
gguf = spec.metadata.get("gguf_file", "")
|
||||
if repo and gguf:
|
||||
console.print(f" Downloading [cyan]{gguf}[/cyan] from {repo}...")
|
||||
hf_download(repo, gguf, console)
|
||||
else:
|
||||
console.print(f" [yellow]No GGUF download info for {model}[/yellow]")
|
||||
elif engine == "mlx":
|
||||
mlx_repo = spec.metadata.get("mlx_repo", "")
|
||||
if mlx_repo:
|
||||
console.print(f" Downloading [cyan]{mlx_repo}[/cyan]...")
|
||||
hf_download(mlx_repo, None, console)
|
||||
else:
|
||||
console.print(f" [yellow]No MLX repo info for {model}[/yellow]")
|
||||
elif engine in ("vllm", "sglang"):
|
||||
console.print(
|
||||
f" [cyan]{model}[/cyan] will download automatically when "
|
||||
f"{engine} starts serving it."
|
||||
)
|
||||
else:
|
||||
console.print(f" Download {model} through the {engine} interface.")
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--force", is_flag=True, help="Overwrite existing config without prompting."
|
||||
@@ -157,11 +233,21 @@ def _next_steps_text(engine: str, model: str = "") -> str:
|
||||
default=None,
|
||||
help="Inference engine to use (skips interactive selection).",
|
||||
)
|
||||
@click.option(
|
||||
"--no-download", is_flag=True, default=False, help="Skip the model download prompt."
|
||||
)
|
||||
@click.option(
|
||||
"--host",
|
||||
default=None,
|
||||
help="Remote engine host URL (e.g. http://192.168.1.50:11434).",
|
||||
)
|
||||
def init(
|
||||
force: bool,
|
||||
config: Optional[Path],
|
||||
full_config: bool = False,
|
||||
engine: Optional[str] = None,
|
||||
no_download: bool = False,
|
||||
host: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Detect hardware and generate ~/.openjarvis/config.toml."""
|
||||
console = Console()
|
||||
@@ -195,9 +281,7 @@ def init(
|
||||
console.print("[bold]Detecting running inference engines...[/bold]")
|
||||
running = _detect_running_engines()
|
||||
if running:
|
||||
console.print(
|
||||
f" Found running: [green]{', '.join(running)}[/green]"
|
||||
)
|
||||
console.print(f" Found running: [green]{', '.join(running)}[/green]")
|
||||
else:
|
||||
console.print(" No running engines detected.")
|
||||
|
||||
@@ -241,13 +325,31 @@ def init(
|
||||
default=default,
|
||||
)
|
||||
|
||||
# Probe remote host if specified
|
||||
if host:
|
||||
console.print("\n[bold]Checking remote host...[/bold]")
|
||||
try:
|
||||
resp = httpx.get(host.rstrip("/") + "/", timeout=2.0)
|
||||
if resp.status_code < 500:
|
||||
console.print(f" [green]Reachable[/green] ({host})")
|
||||
else:
|
||||
console.print(
|
||||
f" [yellow]Warning:[/yellow] Host returned status "
|
||||
f"{resp.status_code} — writing config anyway."
|
||||
)
|
||||
except Exception:
|
||||
console.print(
|
||||
f" [yellow]Warning:[/yellow] Host unreachable ({host}) "
|
||||
f"— writing config anyway."
|
||||
)
|
||||
|
||||
if config:
|
||||
toml_content = config.read_text()
|
||||
else:
|
||||
if full_config:
|
||||
toml_content = generate_default_toml(hw, engine=engine)
|
||||
toml_content = generate_default_toml(hw, engine=engine, host=host)
|
||||
else:
|
||||
toml_content = generate_minimal_toml(hw, engine=engine)
|
||||
toml_content = generate_minimal_toml(hw, engine=engine, host=host)
|
||||
|
||||
DEFAULT_CONFIG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
if config:
|
||||
@@ -265,10 +367,45 @@ def init(
|
||||
)
|
||||
console.print("[green]Config written successfully.[/green]")
|
||||
|
||||
# Create default memory files (skip if they already exist)
|
||||
soul_path = DEFAULT_CONFIG_DIR / "SOUL.md"
|
||||
if not soul_path.exists():
|
||||
soul_path.write_text(
|
||||
"# Agent Persona\n\nYou are Jarvis, a helpful personal AI assistant.\n"
|
||||
)
|
||||
|
||||
memory_path = DEFAULT_CONFIG_DIR / "MEMORY.md"
|
||||
if not memory_path.exists():
|
||||
memory_path.write_text("# Agent Memory\n\n")
|
||||
|
||||
user_path = DEFAULT_CONFIG_DIR / "USER.md"
|
||||
if not user_path.exists():
|
||||
user_path.write_text("# User Profile\n\n")
|
||||
|
||||
skills_dir = DEFAULT_CONFIG_DIR / "skills"
|
||||
skills_dir.mkdir(exist_ok=True)
|
||||
|
||||
selected_engine = engine or recommend_engine(hw)
|
||||
model = recommend_model(hw, selected_engine)
|
||||
if model:
|
||||
console.print(f"\n [bold]Recommended model:[/bold] {model}")
|
||||
|
||||
if not model:
|
||||
console.print(
|
||||
"\n [yellow]! Not enough memory to run any local model.[/yellow]\n"
|
||||
" Consider a cloud engine or a machine with more RAM."
|
||||
)
|
||||
else:
|
||||
spec = find_model_spec(model)
|
||||
size_gb = estimated_download_gb(spec.parameter_count_b) if spec else 0
|
||||
console.print(
|
||||
f"\n [bold]Recommended model:[/bold] {model} (~{size_gb:.1f} GB estimated)"
|
||||
)
|
||||
|
||||
if not no_download and spec:
|
||||
prompt = f" Download {model} (~{size_gb:.1f} GB estimated) now?"
|
||||
if click.confirm(prompt, default=True):
|
||||
_do_download(selected_engine, model, spec, console)
|
||||
|
||||
_quick_privacy_check(console)
|
||||
console.print()
|
||||
console.print(
|
||||
Panel(
|
||||
|
||||
+89
-15
@@ -3,6 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import click
|
||||
@@ -15,6 +16,7 @@ from openjarvis.core.config import load_config
|
||||
from openjarvis.core.registry import ModelRegistry
|
||||
from openjarvis.engine import discover_engines, discover_models
|
||||
from openjarvis.intelligence import merge_discovered_models, register_builtin_models
|
||||
from openjarvis.intelligence.model_catalog import BUILTIN_MODELS
|
||||
|
||||
|
||||
@click.group()
|
||||
@@ -149,18 +151,8 @@ def info(model_name: str) -> None:
|
||||
console.print(Panel("\n".join(lines), title=spec.name, border_style="blue"))
|
||||
|
||||
|
||||
@model.command()
|
||||
@click.argument("model_name")
|
||||
def pull(model_name: str) -> None:
|
||||
"""Download a model (Ollama only)."""
|
||||
console = Console()
|
||||
config = load_config()
|
||||
host = (
|
||||
config.engine.ollama_host
|
||||
or os.environ.get("OLLAMA_HOST")
|
||||
or "http://localhost:11434"
|
||||
).rstrip("/")
|
||||
|
||||
def ollama_pull(host: str, model_name: str, console: Console) -> bool:
|
||||
"""Pull a model via Ollama API. Returns True on success."""
|
||||
console.print(f"Pulling [cyan]{model_name}[/cyan] via Ollama...")
|
||||
try:
|
||||
with httpx.stream(
|
||||
@@ -171,7 +163,6 @@ def pull(model_name: str) -> None:
|
||||
) as resp:
|
||||
resp.raise_for_status()
|
||||
import json
|
||||
|
||||
for line in resp.iter_lines():
|
||||
if not line.strip():
|
||||
continue
|
||||
@@ -188,9 +179,92 @@ def pull(model_name: str) -> None:
|
||||
elif status:
|
||||
console.print(f" {status}")
|
||||
console.print(f"\n[green]Successfully pulled {model_name}[/green]")
|
||||
return True
|
||||
except httpx.ConnectError:
|
||||
console.print("[red]Cannot connect to Ollama.[/red] Is it running?")
|
||||
sys.exit(1)
|
||||
return False
|
||||
except httpx.HTTPStatusError as exc:
|
||||
console.print(f"[red]Ollama error:[/red] {exc.response.status_code}")
|
||||
sys.exit(1)
|
||||
return False
|
||||
|
||||
|
||||
def find_model_spec(model_name: str):
|
||||
"""Look up a model in the builtin catalog. Returns None if not found."""
|
||||
for spec in BUILTIN_MODELS:
|
||||
if spec.model_id == model_name:
|
||||
return spec
|
||||
return None
|
||||
|
||||
|
||||
def hf_download(repo: str, filename: str | None, console: Console) -> bool:
|
||||
"""Download from HuggingFace via huggingface-cli. Returns True on success."""
|
||||
cmd = ["huggingface-cli", "download", repo]
|
||||
if filename:
|
||||
cmd.append(filename)
|
||||
try:
|
||||
subprocess.run(cmd, check=True)
|
||||
console.print("[green]Download complete.[/green]")
|
||||
return True
|
||||
except FileNotFoundError:
|
||||
console.print(
|
||||
"[red]huggingface-cli not found.[/red]\n"
|
||||
"Install it: [cyan]pip install huggingface_hub[/cyan]\n"
|
||||
f"Or download manually: https://huggingface.co/{repo}"
|
||||
)
|
||||
return False
|
||||
except subprocess.CalledProcessError:
|
||||
console.print("[red]Download failed.[/red]")
|
||||
return False
|
||||
|
||||
|
||||
@model.command()
|
||||
@click.argument("model_name")
|
||||
@click.option(
|
||||
"--engine", default=None, help="Engine to download for."
|
||||
)
|
||||
def pull(model_name: str, engine: str | None) -> None:
|
||||
"""Download a model."""
|
||||
console = Console()
|
||||
config = load_config()
|
||||
engine = engine or config.engine.default or "ollama"
|
||||
|
||||
if engine == "ollama":
|
||||
host = (
|
||||
config.engine.ollama_host
|
||||
or os.environ.get("OLLAMA_HOST")
|
||||
or "http://localhost:11434"
|
||||
).rstrip("/")
|
||||
if not ollama_pull(host, model_name, console):
|
||||
sys.exit(1)
|
||||
elif engine in ("llamacpp", "mlx"):
|
||||
spec = find_model_spec(model_name)
|
||||
if not spec:
|
||||
console.print(f"[red]Model not in catalog:[/red] {model_name}")
|
||||
sys.exit(1)
|
||||
if engine == "llamacpp":
|
||||
repo = spec.metadata.get("hf_repo", "")
|
||||
gguf = spec.metadata.get("gguf_file", "")
|
||||
if not repo or not gguf:
|
||||
console.print(f"[red]No GGUF download info for {model_name}[/red]")
|
||||
sys.exit(1)
|
||||
console.print(f"Downloading [cyan]{gguf}[/cyan] from {repo}...")
|
||||
if not hf_download(repo, gguf, console):
|
||||
sys.exit(1)
|
||||
else: # mlx
|
||||
mlx_repo = spec.metadata.get("mlx_repo", "")
|
||||
if not mlx_repo:
|
||||
console.print(f"[red]No MLX repo info for {model_name}[/red]")
|
||||
sys.exit(1)
|
||||
console.print(f"Downloading [cyan]{mlx_repo}[/cyan]...")
|
||||
if not hf_download(mlx_repo, None, console):
|
||||
sys.exit(1)
|
||||
elif engine in ("vllm", "sglang"):
|
||||
console.print(
|
||||
f"[cyan]{model_name}[/cyan] will download automatically when "
|
||||
f"{engine} starts serving it."
|
||||
)
|
||||
else:
|
||||
console.print(
|
||||
f"Manual download required for engine [cyan]{engine}[/cyan].\n"
|
||||
f"Check the engine documentation for instructions."
|
||||
)
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
"""``jarvis registry`` — registry inspection commands."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
|
||||
def _load_registry_map() -> tuple[dict[str, object], dict[str, object]]:
|
||||
"""Import all registries and return (by_name, aliases) lookup dicts."""
|
||||
from openjarvis.core.registry import (
|
||||
AgentRegistry,
|
||||
BenchmarkRegistry,
|
||||
ChannelRegistry,
|
||||
CompressionRegistry,
|
||||
EngineRegistry,
|
||||
LearningRegistry,
|
||||
MemoryRegistry,
|
||||
ModelRegistry,
|
||||
RouterPolicyRegistry,
|
||||
SkillRegistry,
|
||||
SpeechRegistry,
|
||||
ToolRegistry,
|
||||
)
|
||||
|
||||
by_name = {
|
||||
"ToolRegistry": ToolRegistry,
|
||||
"AgentRegistry": AgentRegistry,
|
||||
"EngineRegistry": EngineRegistry,
|
||||
"MemoryRegistry": MemoryRegistry,
|
||||
"ModelRegistry": ModelRegistry,
|
||||
"ChannelRegistry": ChannelRegistry,
|
||||
"LearningRegistry": LearningRegistry,
|
||||
"SkillRegistry": SkillRegistry,
|
||||
"BenchmarkRegistry": BenchmarkRegistry,
|
||||
"RouterPolicyRegistry": RouterPolicyRegistry,
|
||||
"SpeechRegistry": SpeechRegistry,
|
||||
"CompressionRegistry": CompressionRegistry,
|
||||
}
|
||||
|
||||
aliases: dict[str, object] = {}
|
||||
_alias_map = {
|
||||
"ToolRegistry": ("tool", "tools"),
|
||||
"AgentRegistry": ("agent", "agents"),
|
||||
"EngineRegistry": ("engine", "engines"),
|
||||
"MemoryRegistry": ("memory", "memories"),
|
||||
"ModelRegistry": ("model", "models"),
|
||||
"ChannelRegistry": ("channel", "channels"),
|
||||
"LearningRegistry": ("learning", "learnings"),
|
||||
"SkillRegistry": ("skill", "skills"),
|
||||
"BenchmarkRegistry": ("benchmark", "benchmarks"),
|
||||
"RouterPolicyRegistry": ("router", "routers"),
|
||||
"SpeechRegistry": ("speech", "speeches"),
|
||||
"CompressionRegistry": ("compression", "compressions"),
|
||||
}
|
||||
for class_name, cls in by_name.items():
|
||||
aliases[class_name] = cls
|
||||
for alias in _alias_map[class_name]:
|
||||
aliases[alias] = cls
|
||||
|
||||
return by_name, aliases
|
||||
|
||||
|
||||
@click.group()
|
||||
def registry() -> None:
|
||||
"""Inspect registered components — list registries, show entries."""
|
||||
|
||||
|
||||
@registry.command("list")
|
||||
def list_registries() -> None:
|
||||
"""List all available registries."""
|
||||
console = Console(stderr=True)
|
||||
|
||||
table = Table(title="Available Registries")
|
||||
table.add_column("Registry", style="cyan")
|
||||
table.add_column("Module", style="green")
|
||||
table.add_column("Entry Count", style="yellow")
|
||||
|
||||
try:
|
||||
by_name, _ = _load_registry_map()
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error loading registries: {exc}[/red]")
|
||||
return
|
||||
|
||||
module_path = "openjarvis.core.registry"
|
||||
for reg_name, registry_cls in by_name.items():
|
||||
try:
|
||||
count = len(registry_cls.keys())
|
||||
table.add_row(reg_name, module_path, str(count))
|
||||
except Exception as exc:
|
||||
table.add_row(reg_name, module_path, f"[red]Error: {exc}[/red]")
|
||||
|
||||
console.print(table)
|
||||
|
||||
|
||||
@registry.command()
|
||||
@click.argument("registry_name")
|
||||
@click.option(
|
||||
"--verbose", "-v", is_flag=True, default=False, help="Show full entry details"
|
||||
)
|
||||
def show(registry_name: str, verbose: bool) -> None:
|
||||
"""Show entries in a specific registry."""
|
||||
console = Console(stderr=True)
|
||||
|
||||
try:
|
||||
_, aliases = _load_registry_map()
|
||||
|
||||
registry_cls = aliases.get(registry_name)
|
||||
if registry_cls is None:
|
||||
console.print(f"[red]Unknown registry: {registry_name}[/red]")
|
||||
console.print(
|
||||
"[dim]Run 'jarvis registry list' to see available registries.[/dim]"
|
||||
)
|
||||
return
|
||||
|
||||
keys = registry_cls.keys()
|
||||
if not keys:
|
||||
console.print(f"[dim]{registry_name} is empty.[/dim]")
|
||||
return
|
||||
|
||||
console.print(f"[bold]{registry_name}[/bold] — {len(keys)} entry/entries")
|
||||
|
||||
if verbose:
|
||||
for key in keys:
|
||||
entry = registry_cls.get(key)
|
||||
console.print(f"\n [cyan]{key}[/cyan]")
|
||||
console.print(f" Type: {type(entry).__name__}")
|
||||
console.print(f" Value: {entry}")
|
||||
else:
|
||||
table = Table()
|
||||
table.add_column("Key", style="cyan")
|
||||
table.add_column("Type", style="green")
|
||||
table.add_column("Value", style="white", max_width=80)
|
||||
for key in keys:
|
||||
entry = registry_cls.get(key)
|
||||
entry_type = type(entry).__name__
|
||||
entry_value = str(entry)
|
||||
table.add_row(key, entry_type, entry_value)
|
||||
console.print(table)
|
||||
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
__all__ = ["registry"]
|
||||
@@ -0,0 +1,386 @@
|
||||
"""``jarvis scan`` — audit your environment for privacy and security risks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Callable, List
|
||||
|
||||
import click
|
||||
|
||||
# Engine ports that should only be listening on localhost.
|
||||
_ENGINE_PORTS = {11434, 8080, 8000, 30000, 1234, 52415, 18181}
|
||||
|
||||
# Processes associated with cloud-sync agents.
|
||||
_CLOUD_SYNC_PROCS = ["Dropbox", "OneDrive", "Google Drive", "iCloudDrive"]
|
||||
|
||||
# Screen-recording / remote-access processes (macOS).
|
||||
_SCREEN_RECORDING_PROCS = [
|
||||
"TeamViewer", "AnyDesk", "ScreenConnect", "vncviewer", "Vine"
|
||||
]
|
||||
|
||||
# Remote-access processes (Linux).
|
||||
_REMOTE_ACCESS_PROCS = ["xrdp", "x11vnc", "vncserver", "AnyDesk"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data model
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ScanResult:
|
||||
"""Result of a single privacy/security check."""
|
||||
|
||||
name: str
|
||||
status: str # "ok" | "warn" | "fail" | "skip"
|
||||
message: str
|
||||
platform: str # "darwin" | "linux" | "all"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scanner
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class PrivacyScanner:
|
||||
"""Collection of environment privacy checks."""
|
||||
|
||||
# -- Subprocess helper ---------------------------------------------------
|
||||
|
||||
def _run(self, cmd: list[str]) -> subprocess.CompletedProcess: # type: ignore[type-arg]
|
||||
return subprocess.run(cmd, capture_output=True, text=True, timeout=10)
|
||||
|
||||
# -- Individual checks ---------------------------------------------------
|
||||
|
||||
def check_filevault(self) -> ScanResult:
|
||||
"""Check whether FileVault disk encryption is enabled (macOS)."""
|
||||
try:
|
||||
proc = self._run(["fdesetup", "status"])
|
||||
if "On" in proc.stdout:
|
||||
return ScanResult(
|
||||
name="FileVault",
|
||||
status="ok",
|
||||
message="FileVault is enabled.",
|
||||
platform="darwin",
|
||||
)
|
||||
return ScanResult(
|
||||
name="FileVault",
|
||||
status="fail",
|
||||
message="FileVault is NOT enabled. Enable full-disk encryption.",
|
||||
platform="darwin",
|
||||
)
|
||||
except Exception:
|
||||
return ScanResult(
|
||||
name="FileVault",
|
||||
status="skip",
|
||||
message="fdesetup not available.",
|
||||
platform="darwin",
|
||||
)
|
||||
|
||||
def check_mdm(self) -> ScanResult:
|
||||
"""Check whether the device is enrolled in an MDM profile (macOS)."""
|
||||
try:
|
||||
proc = self._run(["profiles", "status", "-type", "enrollment"])
|
||||
output = proc.stdout + proc.stderr
|
||||
lower = output.lower()
|
||||
# "not enrolled" is a strong negative signal — check it first.
|
||||
not_enrolled = "not enrolled" in lower or "no" in lower
|
||||
# Positive signals: explicit Yes/enrolled without a negation.
|
||||
enrolled_yes = (
|
||||
"mdm enrollment: yes" in lower
|
||||
or "enrolled via dep: yes" in lower
|
||||
or ("enrolled" in lower and not not_enrolled)
|
||||
)
|
||||
if enrolled_yes:
|
||||
return ScanResult(
|
||||
name="MDM Enrollment",
|
||||
status="warn",
|
||||
message="Device appears to be enrolled in an MDM profile.",
|
||||
platform="darwin",
|
||||
)
|
||||
return ScanResult(
|
||||
name="MDM Enrollment",
|
||||
status="ok",
|
||||
message="Device is not enrolled in an MDM profile.",
|
||||
platform="darwin",
|
||||
)
|
||||
except Exception:
|
||||
return ScanResult(
|
||||
name="MDM Enrollment",
|
||||
status="skip",
|
||||
message="profiles command not available.",
|
||||
platform="darwin",
|
||||
)
|
||||
|
||||
def check_icloud_sync(self) -> ScanResult:
|
||||
"""Check whether ~/.openjarvis is inside iCloud Drive sync scope."""
|
||||
try:
|
||||
config_path = Path("~/.openjarvis").expanduser().resolve()
|
||||
icloud_path = Path("~/Library/Mobile Documents/").expanduser().resolve()
|
||||
if str(config_path).startswith(str(icloud_path)):
|
||||
return ScanResult(
|
||||
name="iCloud Sync",
|
||||
status="warn",
|
||||
message="~/.openjarvis may be synced to iCloud.",
|
||||
platform="darwin",
|
||||
)
|
||||
# Also probe defaults for com.apple.bird (iCloud daemon)
|
||||
try:
|
||||
proc = self._run(
|
||||
["defaults", "read", "com.apple.bird", "optout_preference"]
|
||||
)
|
||||
val = proc.stdout.strip()
|
||||
if val == "0":
|
||||
return ScanResult(
|
||||
name="iCloud Sync",
|
||||
status="warn",
|
||||
message="iCloud Desktop/Documents sync may be active.",
|
||||
platform="darwin",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return ScanResult(
|
||||
name="iCloud Sync",
|
||||
status="ok",
|
||||
message="~/.openjarvis is not inside iCloud Drive.",
|
||||
platform="darwin",
|
||||
)
|
||||
except Exception:
|
||||
return ScanResult(
|
||||
name="iCloud Sync",
|
||||
status="skip",
|
||||
message="Could not determine iCloud sync status.",
|
||||
platform="darwin",
|
||||
)
|
||||
|
||||
def check_luks(self) -> ScanResult:
|
||||
"""Check whether any block device uses LUKS encryption (Linux)."""
|
||||
try:
|
||||
proc = self._run(["lsblk", "-o", "NAME,TYPE,FSTYPE", "-J"])
|
||||
data = json.loads(proc.stdout)
|
||||
except Exception:
|
||||
return ScanResult(
|
||||
name="LUKS Encryption",
|
||||
status="skip",
|
||||
message="lsblk not available or returned unexpected output.",
|
||||
platform="linux",
|
||||
)
|
||||
|
||||
def _has_luks(devices: list) -> bool: # type: ignore[type-arg]
|
||||
for dev in devices:
|
||||
if dev.get("fstype") == "crypto_LUKS":
|
||||
return True
|
||||
children = dev.get("children") or []
|
||||
if _has_luks(children):
|
||||
return True
|
||||
return False
|
||||
|
||||
try:
|
||||
devices = data.get("blockdevices", [])
|
||||
if _has_luks(devices):
|
||||
return ScanResult(
|
||||
name="LUKS Encryption",
|
||||
status="ok",
|
||||
message="At least one LUKS-encrypted device found.",
|
||||
platform="linux",
|
||||
)
|
||||
return ScanResult(
|
||||
name="LUKS Encryption",
|
||||
status="fail",
|
||||
message="No LUKS-encrypted block devices found.",
|
||||
platform="linux",
|
||||
)
|
||||
except Exception:
|
||||
return ScanResult(
|
||||
name="LUKS Encryption",
|
||||
status="skip",
|
||||
message="Could not parse lsblk output.",
|
||||
platform="linux",
|
||||
)
|
||||
|
||||
def _check_processes(
|
||||
self,
|
||||
names: list[str],
|
||||
check_name: str,
|
||||
warn_msg: str,
|
||||
platform: str,
|
||||
) -> ScanResult:
|
||||
"""Shared helper: pgrep for any of the given process names."""
|
||||
try:
|
||||
for name in names:
|
||||
try:
|
||||
proc = self._run(["pgrep", "-x", name])
|
||||
if proc.returncode == 0:
|
||||
return ScanResult(
|
||||
name=check_name,
|
||||
status="warn",
|
||||
message=warn_msg.format(name=name),
|
||||
platform=platform,
|
||||
)
|
||||
except Exception:
|
||||
continue
|
||||
return ScanResult(
|
||||
name=check_name,
|
||||
status="ok",
|
||||
message=f"No {check_name.lower()} processes detected.",
|
||||
platform=platform,
|
||||
)
|
||||
except Exception:
|
||||
return ScanResult(
|
||||
name=check_name,
|
||||
status="skip",
|
||||
message="pgrep not available.",
|
||||
platform=platform,
|
||||
)
|
||||
|
||||
def check_cloud_sync_agents(self) -> ScanResult:
|
||||
"""Check for running cloud-sync agent processes."""
|
||||
return self._check_processes(
|
||||
names=_CLOUD_SYNC_PROCS,
|
||||
check_name="Cloud Sync Agents",
|
||||
warn_msg="{name} sync agent is running — weights may be uploaded to cloud.",
|
||||
platform="all",
|
||||
)
|
||||
|
||||
def check_network_exposure(self) -> ScanResult:
|
||||
"""Check if engine ports are exposed on 0.0.0.0 rather than localhost."""
|
||||
try:
|
||||
if sys.platform == "darwin":
|
||||
proc = self._run(["lsof", "-iTCP", "-sTCP:LISTEN", "-n", "-P"])
|
||||
else:
|
||||
proc = self._run(["ss", "-tlnp"])
|
||||
output = proc.stdout
|
||||
|
||||
exposed: list[int] = []
|
||||
for port in _ENGINE_PORTS:
|
||||
# Look for patterns like *:PORT or 0.0.0.0:PORT
|
||||
for token in (f"*:{port}", f"0.0.0.0:{port}", f":::{port}"):
|
||||
if token.replace(" ", "") in output.replace(" ", ""):
|
||||
exposed.append(port)
|
||||
break
|
||||
|
||||
if exposed:
|
||||
ports_str = ", ".join(str(p) for p in sorted(exposed))
|
||||
return ScanResult(
|
||||
name="Network Exposure",
|
||||
status="warn",
|
||||
message=f"Engine port(s) {ports_str} exposed on all interfaces.",
|
||||
platform="all",
|
||||
)
|
||||
return ScanResult(
|
||||
name="Network Exposure",
|
||||
status="ok",
|
||||
message="All engine ports appear to be bound to localhost only.",
|
||||
platform="all",
|
||||
)
|
||||
except Exception:
|
||||
return ScanResult(
|
||||
name="Network Exposure",
|
||||
status="skip",
|
||||
message="Could not determine network exposure (lsof/ss unavailable).",
|
||||
platform="all",
|
||||
)
|
||||
|
||||
def check_screen_recording(self) -> ScanResult:
|
||||
"""Check for running screen-recording / remote-desktop processes (macOS)."""
|
||||
return self._check_processes(
|
||||
names=_SCREEN_RECORDING_PROCS,
|
||||
check_name="Screen Recording",
|
||||
warn_msg="{name} is running — screen may be accessible remotely.",
|
||||
platform="darwin",
|
||||
)
|
||||
|
||||
def check_remote_access(self) -> ScanResult:
|
||||
"""Check for running remote-access processes (Linux)."""
|
||||
return self._check_processes(
|
||||
names=_REMOTE_ACCESS_PROCS,
|
||||
check_name="Remote Access",
|
||||
warn_msg="{name} is running — system may be accessible remotely.",
|
||||
platform="linux",
|
||||
)
|
||||
|
||||
# -- Orchestration -------------------------------------------------------
|
||||
|
||||
def _get_all_checks(self) -> list[Callable[[], ScanResult]]:
|
||||
return [
|
||||
self.check_filevault,
|
||||
self.check_mdm,
|
||||
self.check_icloud_sync,
|
||||
self.check_cloud_sync_agents,
|
||||
self.check_network_exposure,
|
||||
self.check_luks,
|
||||
self.check_screen_recording,
|
||||
self.check_remote_access,
|
||||
]
|
||||
|
||||
def run_all(self) -> list[ScanResult]:
|
||||
"""Run all checks, filter to the current platform, hide 'skip' results."""
|
||||
current_plat = "darwin" if sys.platform == "darwin" else "linux"
|
||||
results: list[ScanResult] = []
|
||||
for check_fn in self._get_all_checks():
|
||||
result = check_fn()
|
||||
if result.platform not in (current_plat, "all"):
|
||||
continue
|
||||
if result.status == "skip":
|
||||
continue
|
||||
results.append(result)
|
||||
return results
|
||||
|
||||
def run_quick(self) -> list[ScanResult]:
|
||||
"""Run only critical checks: disk encryption + cloud sync agents."""
|
||||
current_plat = "darwin" if sys.platform == "darwin" else "linux"
|
||||
quick_checks: list[Callable[[], ScanResult]]
|
||||
if current_plat == "darwin":
|
||||
quick_checks = [self.check_filevault, self.check_cloud_sync_agents]
|
||||
else:
|
||||
quick_checks = [self.check_luks, self.check_cloud_sync_agents]
|
||||
results = []
|
||||
for check_fn in quick_checks:
|
||||
result = check_fn()
|
||||
if result.status != "skip":
|
||||
results.append(result)
|
||||
return results
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI command
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_STATUS_ICONS = {"ok": "✓", "warn": "!", "fail": "✗", "skip": "-"}
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--quick", is_flag=True, default=False, help="Run only critical checks.")
|
||||
def scan(quick: bool) -> None:
|
||||
"""Audit your environment for privacy and security risks."""
|
||||
scanner = PrivacyScanner()
|
||||
results: List[ScanResult] = scanner.run_quick() if quick else scanner.run_all()
|
||||
|
||||
if not results:
|
||||
click.echo("No applicable checks for this platform.")
|
||||
return
|
||||
|
||||
warnings = 0
|
||||
failures = 0
|
||||
for r in results:
|
||||
icon = _STATUS_ICONS.get(r.status, "?")
|
||||
click.echo(f" [{icon}] {r.name}: {r.message}")
|
||||
if r.status == "warn":
|
||||
warnings += 1
|
||||
elif r.status == "fail":
|
||||
failures += 1
|
||||
|
||||
click.echo("")
|
||||
parts = []
|
||||
if warnings:
|
||||
parts.append(f"{warnings} warning(s)")
|
||||
if failures:
|
||||
parts.append(f"{failures} issue(s)")
|
||||
if parts:
|
||||
click.echo("Summary: " + ", ".join(parts) + ".")
|
||||
else:
|
||||
click.echo("Summary: all checks passed.")
|
||||
@@ -230,6 +230,20 @@ def serve(
|
||||
console.print(f"[yellow]Channel failed to start: {exc}[/yellow]")
|
||||
channel_bridge = None
|
||||
|
||||
# Wire channel messages → agent / engine (per-chat session isolation)
|
||||
if channel_bridge is not None:
|
||||
from openjarvis.system import JarvisSystem
|
||||
|
||||
_wire_system = JarvisSystem(
|
||||
config=config,
|
||||
bus=bus,
|
||||
engine=engine,
|
||||
engine_key=engine_name,
|
||||
model=model_name,
|
||||
agent_name=agent_key or "",
|
||||
)
|
||||
_wire_system.wire_channel(channel_bridge)
|
||||
|
||||
# Set up speech backend
|
||||
speech_backend = None
|
||||
try:
|
||||
@@ -286,10 +300,27 @@ def serve(
|
||||
except Exception as exc:
|
||||
logger.debug("Agent scheduler init failed: %s", exc)
|
||||
|
||||
# Set up memory backend for context injection
|
||||
memory_backend = None
|
||||
if config.agent.context_from_memory:
|
||||
try:
|
||||
import openjarvis.tools.storage # noqa: F401
|
||||
from openjarvis.core.registry import MemoryRegistry
|
||||
|
||||
mem_key = config.memory.default_backend
|
||||
if MemoryRegistry.contains(mem_key):
|
||||
memory_backend = MemoryRegistry.create(
|
||||
mem_key, db_path=config.memory.db_path,
|
||||
)
|
||||
console.print(" Memory: [cyan]active[/cyan]")
|
||||
except Exception as exc:
|
||||
logger.debug("Memory backend init failed: %s", exc)
|
||||
|
||||
app = create_app(
|
||||
engine, model_name, agent=agent, bus=bus,
|
||||
engine_name=engine_name, agent_name=agent_key or "",
|
||||
channel_bridge=channel_bridge, config=config,
|
||||
memory_backend=memory_backend,
|
||||
speech_backend=speech_backend,
|
||||
agent_manager=agent_manager,
|
||||
agent_scheduler=agent_scheduler,
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
"""``jarvis tool`` — tool management commands."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
|
||||
@click.group()
|
||||
def tool() -> None:
|
||||
"""Manage tools — list, inspect."""
|
||||
|
||||
|
||||
@tool.command("list")
|
||||
def list_tools() -> None:
|
||||
"""List all registered tools with their descriptions."""
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
# Trigger tool registration by importing the tools module
|
||||
import openjarvis.tools # noqa: F401
|
||||
from openjarvis.core.registry import ToolRegistry
|
||||
|
||||
keys = sorted(ToolRegistry.keys())
|
||||
if not keys:
|
||||
console.print("[dim]No tools registered.[/dim]")
|
||||
return
|
||||
|
||||
table = Table(title="Registered Tools")
|
||||
table.add_column("Name", style="cyan")
|
||||
table.add_column("Description", style="green", max_width=60)
|
||||
table.add_column("Category", style="yellow")
|
||||
|
||||
for key in keys:
|
||||
tool_cls = ToolRegistry.get(key)
|
||||
description = ""
|
||||
category = ""
|
||||
|
||||
# Try to get spec from the class or an instance
|
||||
try:
|
||||
# Some tools may require initialization, so try with default init
|
||||
tool_instance = tool_cls() if callable(tool_cls) else tool_cls
|
||||
if hasattr(tool_instance, "spec"):
|
||||
spec = tool_instance.spec
|
||||
description = getattr(spec, "description", "")[:60]
|
||||
category = getattr(spec, "category", "")
|
||||
except Exception:
|
||||
# If instantiation fails, just show the key
|
||||
description = "N/A (instantiation error)"
|
||||
|
||||
table.add_row(key, description, category)
|
||||
|
||||
console.print(table)
|
||||
console.print(f"\n[dim]Total: {len(keys)} tool(s)[/dim]")
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
@tool.command()
|
||||
@click.argument("tool_name")
|
||||
def inspect(tool_name: str) -> None:
|
||||
"""Show detailed information about a specific tool."""
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
# Trigger tool registration by importing the tools module
|
||||
import openjarvis.tools # noqa: F401
|
||||
from openjarvis.core.registry import ToolRegistry
|
||||
|
||||
if not ToolRegistry.contains(tool_name):
|
||||
console.print(f"[red]Tool not found: {tool_name}[/red]")
|
||||
console.print("[dim]Run 'jarvis tool list' to see available tools.[/dim]")
|
||||
return
|
||||
|
||||
tool_cls = ToolRegistry.get(tool_name)
|
||||
console.print(f"[bold]{tool_name}[/bold]")
|
||||
|
||||
try:
|
||||
tool_instance = tool_cls() if callable(tool_cls) else tool_cls
|
||||
if hasattr(tool_instance, "spec"):
|
||||
spec = tool_instance.spec
|
||||
console.print(f" [cyan]Name:[/cyan] {getattr(spec, 'name', 'N/A')}")
|
||||
console.print(
|
||||
f" [cyan]Description:[/cyan] {getattr(spec, 'description', 'N/A')}"
|
||||
)
|
||||
category = getattr(spec, "category", "N/A") or "none"
|
||||
console.print(f" [cyan]Category:[/cyan] {category}")
|
||||
|
||||
params = getattr(spec, "parameters", {})
|
||||
if params:
|
||||
console.print(" [cyan]Parameters:[/cyan]")
|
||||
if isinstance(params, dict) and "properties" in params:
|
||||
for param_name, param_info in params["properties"].items():
|
||||
param_type = param_info.get("type", "any")
|
||||
param_desc = param_info.get("description", "")
|
||||
console.print(
|
||||
f" • {param_name}: {param_type} — {param_desc}"
|
||||
)
|
||||
else:
|
||||
console.print(f" {params}")
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Note: Could not instantiate tool: {e}[/yellow]")
|
||||
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
|
||||
|
||||
__all__ = ["tool"]
|
||||
+210
-37
@@ -7,6 +7,7 @@ found in the TOML file.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
@@ -59,7 +60,10 @@ def _run_cmd(cmd: list[str]) -> str:
|
||||
"""Run a command and return stripped stdout, or empty string on failure."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, timeout=10, # noqa: S603
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10, # noqa: S603
|
||||
)
|
||||
return result.stdout.strip()
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
|
||||
@@ -69,11 +73,13 @@ def _run_cmd(cmd: list[str]) -> str:
|
||||
def _detect_nvidia_gpu() -> Optional[GpuInfo]:
|
||||
if not shutil.which("nvidia-smi"):
|
||||
return None
|
||||
raw = _run_cmd([
|
||||
"nvidia-smi",
|
||||
"--query-gpu=name,memory.total,count",
|
||||
"--format=csv,noheader,nounits",
|
||||
])
|
||||
raw = _run_cmd(
|
||||
[
|
||||
"nvidia-smi",
|
||||
"--query-gpu=name,memory.total,count",
|
||||
"--format=csv,noheader,nounits",
|
||||
]
|
||||
)
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
@@ -117,6 +123,7 @@ def _detect_amd_gpu() -> Optional[GpuInfo]:
|
||||
try:
|
||||
allinfo_raw = _run_cmd(["rocm-smi", "--showallinfo"])
|
||||
import re
|
||||
|
||||
gpu_ids = set(re.findall(r"GPU\[(\d+)\]", allinfo_raw))
|
||||
if gpu_ids:
|
||||
count = len(gpu_ids)
|
||||
@@ -248,6 +255,11 @@ def recommend_model(hw: HardwareInfo, engine: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def estimated_download_gb(parameter_count_b: float) -> float:
|
||||
"""Estimate download size in GB for Q4_K_M quantized model."""
|
||||
return parameter_count_b * 0.5 * 1.1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Configuration hierarchy
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -325,6 +337,16 @@ class AppleFmEngineConfig:
|
||||
host: str = "http://localhost:8079"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class GemmaCppEngineConfig:
|
||||
"""Per-engine config for gemma.cpp."""
|
||||
|
||||
model_path: str = ""
|
||||
tokenizer_path: str = ""
|
||||
model_type: str = ""
|
||||
num_threads: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class EngineConfig:
|
||||
"""Inference engine settings with nested per-engine configs."""
|
||||
@@ -340,6 +362,7 @@ class EngineConfig:
|
||||
nexa: NexaEngineConfig = field(default_factory=NexaEngineConfig)
|
||||
uzu: UzuEngineConfig = field(default_factory=UzuEngineConfig)
|
||||
apple_fm: AppleFmEngineConfig = field(default_factory=AppleFmEngineConfig)
|
||||
gemma_cpp: GemmaCppEngineConfig = field(default_factory=GemmaCppEngineConfig)
|
||||
|
||||
# Backward-compat properties for old flat attribute names
|
||||
@property
|
||||
@@ -448,26 +471,26 @@ class IntelligenceConfig:
|
||||
|
||||
default_model: str = ""
|
||||
fallback_model: str = ""
|
||||
model_path: str = "" # Local weights (HF repo, GGUF file, etc.)
|
||||
checkpoint_path: str = "" # Checkpoint/adapter path
|
||||
quantization: str = "none" # none, fp8, int8, int4, gguf_q4, gguf_q8
|
||||
preferred_engine: str = "" # Override engine for this model (e.g., "vllm")
|
||||
provider: str = "" # local, openai, anthropic, google
|
||||
model_path: str = "" # Local weights (HF repo, GGUF file, etc.)
|
||||
checkpoint_path: str = "" # Checkpoint/adapter path
|
||||
quantization: str = "none" # none, fp8, int8, int4, gguf_q4, gguf_q8
|
||||
preferred_engine: str = "" # Override engine for this model (e.g., "vllm")
|
||||
provider: str = "" # local, openai, anthropic, google
|
||||
# Generation defaults (overridable per-call)
|
||||
temperature: float = 0.7
|
||||
max_tokens: int = 1024
|
||||
top_p: float = 0.9
|
||||
top_k: int = 40
|
||||
repetition_penalty: float = 1.0
|
||||
stop_sequences: str = "" # Comma-separated stop strings
|
||||
stop_sequences: str = "" # Comma-separated stop strings
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class RoutingLearningConfig:
|
||||
"""Routing sub-policy config within Learning."""
|
||||
|
||||
policy: str = "heuristic" # heuristic | learned
|
||||
min_samples: int = 5 # Min traces before trusting learned routing
|
||||
policy: str = "heuristic" # heuristic | learned
|
||||
min_samples: int = 5 # Min traces before trusting learned routing
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
@@ -716,9 +739,9 @@ class AgentConfig:
|
||||
|
||||
default_agent: str = "simple"
|
||||
max_turns: int = 10
|
||||
tools: str = "" # comma-separated tool names
|
||||
objective: str = "" # concise purpose for routing/learning/docs
|
||||
system_prompt: str = "" # inline system prompt (takes precedence if set)
|
||||
tools: str = "" # comma-separated tool names
|
||||
objective: str = "" # concise purpose for routing/learning/docs
|
||||
system_prompt: str = "" # inline system prompt (takes precedence if set)
|
||||
system_prompt_path: str = "" # path to system prompt file (.txt, .md)
|
||||
context_from_memory: bool = True # inject relevant memory context into prompts
|
||||
|
||||
@@ -762,7 +785,7 @@ class TelemetryConfig:
|
||||
class TracesConfig:
|
||||
"""Trace system settings."""
|
||||
|
||||
enabled: bool = False
|
||||
enabled: bool = True
|
||||
db_path: str = str(DEFAULT_CONFIG_DIR / "traces.db")
|
||||
|
||||
|
||||
@@ -898,7 +921,7 @@ class BlueBubblesChannelConfig:
|
||||
class WhatsAppBaileysChannelConfig:
|
||||
"""Per-channel config for WhatsApp via Baileys protocol."""
|
||||
|
||||
auth_dir: str = "" # Defaults to ~/.openjarvis/whatsapp_auth
|
||||
auth_dir: str = "" # Defaults to ~/.openjarvis/whatsapp_auth
|
||||
assistant_name: str = "Jarvis"
|
||||
assistant_has_own_number: bool = False
|
||||
|
||||
@@ -1055,6 +1078,45 @@ class AgentManagerConfig:
|
||||
db_path: str = str(DEFAULT_CONFIG_DIR / "agents.db")
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class MemoryFilesConfig:
|
||||
"""Persistent memory-file paths and nudge settings."""
|
||||
|
||||
soul_path: str = "~/.openjarvis/SOUL.md"
|
||||
memory_path: str = "~/.openjarvis/MEMORY.md"
|
||||
user_path: str = "~/.openjarvis/USER.md"
|
||||
nudge_interval: int = 10
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SystemPromptConfig:
|
||||
"""Limits and strategy for system-prompt assembly."""
|
||||
|
||||
soul_max_chars: int = 4000
|
||||
memory_max_chars: int = 2500
|
||||
user_max_chars: int = 1500
|
||||
skill_desc_max_chars: int = 60
|
||||
truncation_strategy: str = "head_tail"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class CompressionConfig:
|
||||
"""Configuration for context compression."""
|
||||
|
||||
enabled: bool = True
|
||||
threshold: float = 0.50
|
||||
strategy: str = "session_consolidation"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillsConfig:
|
||||
"""Configuration for agent-authored procedural skills."""
|
||||
|
||||
skills_dir: str = "~/.openjarvis/skills/"
|
||||
nudge_interval: int = 15
|
||||
auto_discover: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class JarvisConfig:
|
||||
"""Top-level configuration for OpenJarvis."""
|
||||
@@ -1079,6 +1141,10 @@ class JarvisConfig:
|
||||
speech: SpeechConfig = field(default_factory=SpeechConfig)
|
||||
optimize: OptimizeConfig = field(default_factory=OptimizeConfig)
|
||||
agent_manager: AgentManagerConfig = field(default_factory=AgentManagerConfig)
|
||||
memory_files: MemoryFilesConfig = field(default_factory=MemoryFilesConfig)
|
||||
system_prompt: SystemPromptConfig = field(default_factory=SystemPromptConfig)
|
||||
compression: CompressionConfig = field(default_factory=CompressionConfig)
|
||||
skills: SkillsConfig = field(default_factory=SkillsConfig)
|
||||
|
||||
@property
|
||||
def memory(self) -> StorageConfig:
|
||||
@@ -1091,6 +1157,80 @@ class JarvisConfig:
|
||||
self.tools.storage = value
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config key validation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Sections that users may set via ``jarvis config set``.
|
||||
# ``hardware`` is auto-detected and not user-settable.
|
||||
_SETTABLE_SECTIONS = frozenset(JarvisConfig.__dataclass_fields__.keys()) - {"hardware"}
|
||||
|
||||
|
||||
def validate_config_key(dotted_key: str) -> type:
|
||||
"""Validate a dotted config key and return the leaf field's Python type.
|
||||
|
||||
Raises :class:`ValueError` when the key does not map to a known field.
|
||||
The function walks the ``JarvisConfig`` dataclass hierarchy using
|
||||
``dataclasses.fields()``.
|
||||
|
||||
Examples::
|
||||
|
||||
validate_config_key("engine.ollama.host") # -> str
|
||||
validate_config_key("intelligence.temperature") # -> float
|
||||
"""
|
||||
from dataclasses import fields as dc_fields
|
||||
|
||||
parts = dotted_key.split(".")
|
||||
if len(parts) < 2:
|
||||
raise ValueError(
|
||||
f"Config key must have at least two segments (e.g. engine.default), "
|
||||
f"got: {dotted_key!r}"
|
||||
)
|
||||
|
||||
if parts[0] not in _SETTABLE_SECTIONS:
|
||||
raise ValueError(
|
||||
f"Unknown config key: {dotted_key!r} "
|
||||
f"(valid top-level sections: {sorted(_SETTABLE_SECTIONS)})"
|
||||
)
|
||||
|
||||
# Walk the dataclass tree
|
||||
current_cls = JarvisConfig
|
||||
for i, part in enumerate(parts):
|
||||
field_map = {f.name: f for f in dc_fields(current_cls)}
|
||||
if part not in field_map:
|
||||
path_so_far = ".".join(parts[: i + 1])
|
||||
raise ValueError(
|
||||
f"Unknown config key: {dotted_key!r} "
|
||||
f"(no field {part!r} at {path_so_far}; "
|
||||
f"valid fields: {sorted(field_map.keys())})"
|
||||
)
|
||||
fld = field_map[part]
|
||||
# Resolve the type — unwrap Optional, etc.
|
||||
fld_type = fld.type
|
||||
if isinstance(fld_type, str):
|
||||
# Evaluate forward references in the config module namespace
|
||||
import openjarvis.core.config as _cfg_mod
|
||||
|
||||
fld_type = eval(fld_type, vars(_cfg_mod)) # noqa: S307
|
||||
|
||||
if i == len(parts) - 1:
|
||||
# Leaf — return the primitive type
|
||||
return fld_type
|
||||
else:
|
||||
# Must be a nested dataclass
|
||||
if not hasattr(fld_type, "__dataclass_fields__"):
|
||||
path_so_far = ".".join(parts[: i + 1])
|
||||
raise ValueError(
|
||||
f"Unknown config key: {dotted_key!r} "
|
||||
f"({path_so_far} is a leaf of type {fld_type.__name__}, "
|
||||
f"not a section)"
|
||||
)
|
||||
current_cls = fld_type
|
||||
|
||||
# Should not reach here, but satisfy type checker
|
||||
raise ValueError(f"Unknown config key: {dotted_key!r}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TOML loading
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -1139,7 +1279,8 @@ def _migrate_toml_data(data: Dict[str, Any], cfg: "JarvisConfig") -> None:
|
||||
src = data.get(src_section, {})
|
||||
if isinstance(src, dict) and "context_injection" in src:
|
||||
data.setdefault("agent", {}).setdefault(
|
||||
"context_from_memory", src.pop("context_injection"),
|
||||
"context_from_memory",
|
||||
src.pop("context_injection"),
|
||||
)
|
||||
|
||||
if "tools" in data:
|
||||
@@ -1148,10 +1289,12 @@ def _migrate_toml_data(data: Dict[str, Any], cfg: "JarvisConfig") -> None:
|
||||
storage_sub = tools_data.get("storage", {})
|
||||
if isinstance(storage_sub, dict) and "context_injection" in storage_sub:
|
||||
data.setdefault("agent", {}).setdefault(
|
||||
"context_from_memory", storage_sub.pop("context_injection"),
|
||||
"context_from_memory",
|
||||
storage_sub.pop("context_injection"),
|
||||
)
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def load_config(path: Optional[Path] = None) -> JarvisConfig:
|
||||
"""Detect hardware, build defaults, overlay TOML overrides.
|
||||
|
||||
@@ -1181,16 +1324,31 @@ def load_config(path: Optional[Path] = None) -> JarvisConfig:
|
||||
# All top-level sections — recursive _apply_toml_section handles
|
||||
# nested sub-configs (engine.ollama, learning.routing, channel.*, etc.)
|
||||
top_sections = (
|
||||
"engine", "intelligence", "learning", "agent",
|
||||
"server", "telemetry", "traces", "security",
|
||||
"channel", "tools", "sandbox", "scheduler",
|
||||
"workflow", "sessions", "a2a", "operators",
|
||||
"speech", "optimize", "agent_manager",
|
||||
"engine",
|
||||
"intelligence",
|
||||
"learning",
|
||||
"agent",
|
||||
"server",
|
||||
"telemetry",
|
||||
"traces",
|
||||
"security",
|
||||
"channel",
|
||||
"tools",
|
||||
"sandbox",
|
||||
"scheduler",
|
||||
"workflow",
|
||||
"sessions",
|
||||
"a2a",
|
||||
"operators",
|
||||
"speech",
|
||||
"optimize",
|
||||
"agent_manager",
|
||||
)
|
||||
for section_name in top_sections:
|
||||
if section_name in data:
|
||||
_apply_toml_section(
|
||||
getattr(cfg, section_name), data[section_name],
|
||||
getattr(cfg, section_name),
|
||||
data[section_name],
|
||||
)
|
||||
|
||||
# Memory: accept [memory] (old) → maps to tools.storage
|
||||
@@ -1205,18 +1363,23 @@ def load_config(path: Optional[Path] = None) -> JarvisConfig:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def generate_minimal_toml(hw: HardwareInfo, engine: str | None = None) -> str:
|
||||
def generate_minimal_toml(
|
||||
hw: HardwareInfo, engine: str | None = None, *, host: str | None = None
|
||||
) -> str:
|
||||
"""Render a minimal TOML config with only essential settings."""
|
||||
engine = engine or recommend_engine(hw)
|
||||
model = recommend_model(hw, engine)
|
||||
gpu_comment = ""
|
||||
if hw.gpu:
|
||||
mem_label = (
|
||||
"unified memory" if hw.gpu.vendor == "apple" else "VRAM"
|
||||
)
|
||||
gpu_comment = (
|
||||
f"\n# GPU: {hw.gpu.name}"
|
||||
f" ({hw.gpu.vram_gb} GB {mem_label})"
|
||||
mem_label = "unified memory" if hw.gpu.vendor == "apple" else "VRAM"
|
||||
gpu_comment = f"\n# GPU: {hw.gpu.name} ({hw.gpu.vram_gb} GB {mem_label})"
|
||||
if host:
|
||||
engine_host_section = f'\n[engine.{engine}]\nhost = "{host}"\n'
|
||||
else:
|
||||
engine_host_section = (
|
||||
f"\n[engine.{engine}]\n"
|
||||
f'# host = "http://localhost:11434" '
|
||||
f"# set to remote URL if engine runs elsewhere\n"
|
||||
)
|
||||
return f"""\
|
||||
# OpenJarvis configuration
|
||||
@@ -1225,7 +1388,7 @@ def generate_minimal_toml(hw: HardwareInfo, engine: str | None = None) -> str:
|
||||
|
||||
[engine]
|
||||
default = "{engine}"
|
||||
|
||||
{engine_host_section}
|
||||
[intelligence]
|
||||
default_model = "{model}"
|
||||
|
||||
@@ -1237,7 +1400,9 @@ enabled = ["code_interpreter", "web_search", "file_read", "shell_exec"]
|
||||
"""
|
||||
|
||||
|
||||
def generate_default_toml(hw: HardwareInfo, engine: str | None = None) -> str:
|
||||
def generate_default_toml(
|
||||
hw: HardwareInfo, engine: str | None = None, *, host: str | None = None
|
||||
) -> str:
|
||||
"""Render a commented TOML string suitable for ``~/.openjarvis/config.toml``."""
|
||||
engine = engine or recommend_engine(hw)
|
||||
model = recommend_model(hw, engine)
|
||||
@@ -1249,7 +1414,7 @@ def generate_default_toml(hw: HardwareInfo, engine: str | None = None) -> str:
|
||||
if model:
|
||||
model_comment = " # recommended for your hardware"
|
||||
|
||||
return f"""\
|
||||
result = f"""\
|
||||
# OpenJarvis configuration
|
||||
# Generated by `jarvis init`
|
||||
#
|
||||
@@ -1444,6 +1609,13 @@ ssrf_protection = true
|
||||
# assistant_name = "Jarvis"
|
||||
# assistant_has_own_number = false
|
||||
"""
|
||||
if host:
|
||||
import re as _re
|
||||
|
||||
pattern = _re.escape(f"[engine.{engine}]") + r"\nhost = \"[^\"]*\""
|
||||
replacement = f'[engine.{engine}]\\nhost = "{host}"'
|
||||
result = _re.sub(pattern, replacement, result)
|
||||
return result
|
||||
|
||||
|
||||
__all__ = [
|
||||
@@ -1508,4 +1680,5 @@ __all__ = [
|
||||
"load_config",
|
||||
"recommend_engine",
|
||||
"recommend_model",
|
||||
"validate_config_key",
|
||||
]
|
||||
|
||||
@@ -141,10 +141,15 @@ class SpeechRegistry(RegistryBase[Any]):
|
||||
"""Registry for speech backend implementations."""
|
||||
|
||||
|
||||
class CompressionRegistry(RegistryBase[Any]):
|
||||
"""Registry for context compression strategies."""
|
||||
|
||||
|
||||
__all__ = [
|
||||
"AgentRegistry",
|
||||
"BenchmarkRegistry",
|
||||
"ChannelRegistry",
|
||||
"CompressionRegistry",
|
||||
"EngineRegistry",
|
||||
"LearningRegistry",
|
||||
"MemoryRegistry",
|
||||
|
||||
@@ -132,6 +132,7 @@ class TelemetryRecord:
|
||||
timestamp: float
|
||||
model_id: str
|
||||
prompt_tokens: int = 0
|
||||
prompt_tokens_evaluated: int = 0 # KV-cache-aware: actual tokens processed
|
||||
completion_tokens: int = 0
|
||||
total_tokens: int = 0
|
||||
latency_seconds: float = 0.0
|
||||
@@ -235,8 +236,11 @@ class RoutingContext:
|
||||
query_length: int = 0
|
||||
has_code: bool = False
|
||||
has_math: bool = False
|
||||
has_reasoning: bool = False
|
||||
language: str = "en"
|
||||
urgency: float = 0.5
|
||||
complexity_score: float = 0.0 # 0.0 (trivial) to 1.0 (very complex)
|
||||
suggested_max_tokens: int = 1024
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
from __future__ import annotations
|
||||
@@ -0,0 +1,39 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
class GatewayDaemon:
|
||||
"""Composes channels, sessions, agents, and scheduler into a daemon."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: Any = None,
|
||||
session_store: Any = None,
|
||||
agent_manager: Any = None,
|
||||
agent_scheduler: Any = None,
|
||||
event_bus: Any = None,
|
||||
) -> None:
|
||||
self._config = config
|
||||
self._session_store = session_store
|
||||
self._agent_manager = agent_manager
|
||||
self._agent_scheduler = agent_scheduler
|
||||
self._event_bus = event_bus
|
||||
self._running = False
|
||||
|
||||
@staticmethod
|
||||
def session_key(
|
||||
platform: str,
|
||||
chat_type: str,
|
||||
chat_id: str,
|
||||
thread_id: Optional[str],
|
||||
) -> str:
|
||||
return f"agent:main:{platform}:{chat_type}:{chat_id}:{thread_id}"
|
||||
|
||||
def start(self) -> None:
|
||||
"""Start the daemon (foreground)."""
|
||||
self._running = True
|
||||
|
||||
def stop(self) -> None:
|
||||
"""Stop the daemon."""
|
||||
self._running = False
|
||||
@@ -0,0 +1,57 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SYSTEMD_TEMPLATE = """\
|
||||
[Unit]
|
||||
Description=OpenJarvis Gateway Daemon
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart={python} -m openjarvis.daemon.gateway
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
"""
|
||||
|
||||
LAUNCHD_TEMPLATE = """\
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" \
|
||||
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.openjarvis.gateway</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>{python}</string>
|
||||
<string>-m</string>
|
||||
<string>openjarvis.daemon.gateway</string>
|
||||
</array>
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
<key>KeepAlive</key>
|
||||
<true/>
|
||||
</dict>
|
||||
</plist>
|
||||
"""
|
||||
|
||||
|
||||
def generate_systemd_service(output: Path | None = None) -> str:
|
||||
content = SYSTEMD_TEMPLATE.format(python=sys.executable)
|
||||
if output:
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(content)
|
||||
return content
|
||||
|
||||
|
||||
def generate_launchd_plist(output: Path | None = None) -> str:
|
||||
content = LAUNCHD_TEMPLATE.format(python=sys.executable)
|
||||
if output:
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(content)
|
||||
return content
|
||||
@@ -0,0 +1,34 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, List
|
||||
|
||||
from openjarvis.core.types import Message
|
||||
|
||||
|
||||
class SessionExpiryHook:
|
||||
"""Proactive memory flush before session reset."""
|
||||
|
||||
FLUSH_PROMPT = (
|
||||
"This session is about to be reset. Review the conversation below "
|
||||
"and save anything important to memory or skills. Use memory_manage "
|
||||
"to save facts/preferences and skill_manage to save reusable procedures."
|
||||
)
|
||||
|
||||
def __init__(self, executor: Any, flush_min_turns: int = 6) -> None:
|
||||
self._executor = executor
|
||||
self._flush_min_turns = flush_min_turns
|
||||
|
||||
def on_session_expiry(self, session_id: str, messages: List[Message]) -> None:
|
||||
if len(messages) < self._flush_min_turns:
|
||||
return
|
||||
transcript = "\n".join(f"[{m.role}]: {m.content}" for m in messages)
|
||||
input_text = f"{self.FLUSH_PROMPT}\n\n---\n\n{transcript}"
|
||||
self._executor.run_ephemeral(
|
||||
agent_type="simple",
|
||||
system_prompt=(
|
||||
"You are a memory management agent. "
|
||||
"Save important information."
|
||||
),
|
||||
input_text=input_text,
|
||||
tools=["memory_manage", "skill_manage"],
|
||||
)
|
||||
@@ -15,7 +15,7 @@ from openjarvis.engine._base import (
|
||||
from openjarvis.engine._discovery import discover_engines, discover_models, get_engine
|
||||
|
||||
# Optional engines — only register if their SDK deps are present
|
||||
for _optional in ("cloud", "litellm"):
|
||||
for _optional in ("cloud", "litellm", "gemma_cpp"):
|
||||
try:
|
||||
importlib.import_module(f".{_optional}", __name__)
|
||||
except ImportError:
|
||||
|
||||
@@ -38,4 +38,28 @@ def messages_to_dicts(messages: Sequence[Message]) -> List[Dict[str, Any]]:
|
||||
return out
|
||||
|
||||
|
||||
__all__ = ["EngineConnectionError", "InferenceEngine", "messages_to_dicts"]
|
||||
def estimate_prompt_tokens(messages: Sequence[Message]) -> int:
|
||||
"""Estimate full prompt token count from message content.
|
||||
|
||||
Ollama's ``prompt_eval_count`` may report only *newly evaluated*
|
||||
tokens when KV-cache hits occur, under-counting the system prompt
|
||||
and earlier conversation turns. This helper provides a
|
||||
cache-agnostic estimate so that downstream cost / FLOPs / energy
|
||||
calculations reflect the true prompt size — matching what a cloud
|
||||
provider would charge.
|
||||
|
||||
Uses ~4 characters per token (standard BPE average for English) plus
|
||||
a small per-message overhead for role markers and separators.
|
||||
"""
|
||||
total_chars = sum(len(m.content) for m in messages)
|
||||
# ~4 tokens overhead per message for role markers / separators
|
||||
overhead = len(messages) * 4
|
||||
return max(1, total_chars // 4 + overhead)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"EngineConnectionError",
|
||||
"InferenceEngine",
|
||||
"estimate_prompt_tokens",
|
||||
"messages_to_dicts",
|
||||
]
|
||||
|
||||
@@ -25,12 +25,24 @@ _HOST_MAP: Dict[str, str | None] = {
|
||||
"apple_fm": "apple_fm_host",
|
||||
"cloud": None,
|
||||
"litellm": None,
|
||||
"gemma_cpp": None,
|
||||
}
|
||||
|
||||
|
||||
def _make_engine(key: str, config: JarvisConfig) -> InferenceEngine:
|
||||
"""Instantiate a registered engine with the appropriate config host."""
|
||||
cls = EngineRegistry.get(key)
|
||||
|
||||
# gemma_cpp: pass config fields instead of host
|
||||
if key == "gemma_cpp":
|
||||
cfg = config.engine.gemma_cpp
|
||||
return cls(
|
||||
model_path=cfg.model_path or None,
|
||||
tokenizer_path=cfg.tokenizer_path or None,
|
||||
model_type=cfg.model_type or None,
|
||||
num_threads=cfg.num_threads,
|
||||
)
|
||||
|
||||
host_attr = _HOST_MAP.get(key)
|
||||
if host_attr is not None:
|
||||
host = getattr(config.engine, host_attr, None)
|
||||
|
||||
@@ -13,6 +13,7 @@ from openjarvis.core.types import Message
|
||||
from openjarvis.engine._base import (
|
||||
EngineConnectionError,
|
||||
InferenceEngine,
|
||||
estimate_prompt_tokens,
|
||||
messages_to_dicts,
|
||||
)
|
||||
|
||||
@@ -32,26 +33,6 @@ class _OpenAICompatibleEngine(InferenceEngine):
|
||||
|
||||
# -- InferenceEngine interface ------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _fix_tool_call_arguments(msg_dicts: list) -> list:
|
||||
"""Ensure tool_call arguments are dicts, not JSON strings.
|
||||
|
||||
OpenAI-compatible servers (vLLM, SGLang, llama.cpp, etc.) expect
|
||||
tool_call arguments as JSON objects. ``messages_to_dicts`` may
|
||||
serialize them as strings, which causes 400 errors on multi-turn
|
||||
tool-calling conversations.
|
||||
"""
|
||||
for md in msg_dicts:
|
||||
for tc in md.get("tool_calls", []):
|
||||
fn = tc.get("function", {})
|
||||
args = fn.get("arguments")
|
||||
if isinstance(args, str):
|
||||
try:
|
||||
fn["arguments"] = json.loads(args)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
return msg_dicts
|
||||
|
||||
def generate(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
@@ -61,14 +42,12 @@ class _OpenAICompatibleEngine(InferenceEngine):
|
||||
max_tokens: int = 1024,
|
||||
**kwargs: Any,
|
||||
) -> Dict[str, Any]:
|
||||
msg_dicts = self._fix_tool_call_arguments(messages_to_dicts(messages))
|
||||
payload: Dict[str, Any] = {
|
||||
"model": model,
|
||||
"messages": msg_dicts,
|
||||
"messages": messages_to_dicts(messages),
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
"stream": False,
|
||||
"chat_template_kwargs": {"enable_thinking": False},
|
||||
**kwargs,
|
||||
}
|
||||
try:
|
||||
@@ -94,12 +73,21 @@ class _OpenAICompatibleEngine(InferenceEngine):
|
||||
}
|
||||
choice = choices[0]
|
||||
usage = data.get("usage", {})
|
||||
# Ensure prompt_tokens reflects the full prompt size (including
|
||||
# system prompt and all conversation history).
|
||||
# OpenAI-compat APIs (vLLM, SGLang) report full counts — KV
|
||||
# caching is transparent, so evaluated == full.
|
||||
reported_prompt = usage.get("prompt_tokens", 0)
|
||||
estimated_prompt = estimate_prompt_tokens(messages)
|
||||
prompt_tokens = max(reported_prompt, estimated_prompt)
|
||||
completion_tokens = usage.get("completion_tokens", 0)
|
||||
result: Dict[str, Any] = {
|
||||
"content": choice["message"].get("content") or "",
|
||||
"usage": {
|
||||
"prompt_tokens": usage.get("prompt_tokens", 0),
|
||||
"completion_tokens": usage.get("completion_tokens", 0),
|
||||
"total_tokens": usage.get("total_tokens", 0),
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"prompt_tokens_evaluated": reported_prompt or prompt_tokens,
|
||||
"completion_tokens": completion_tokens,
|
||||
"total_tokens": prompt_tokens + completion_tokens,
|
||||
},
|
||||
"model": data.get("model", model),
|
||||
"finish_reason": choice.get("finish_reason", "stop"),
|
||||
@@ -126,10 +114,9 @@ class _OpenAICompatibleEngine(InferenceEngine):
|
||||
max_tokens: int = 1024,
|
||||
**kwargs: Any,
|
||||
) -> AsyncIterator[str]:
|
||||
msg_dicts = self._fix_tool_call_arguments(messages_to_dicts(messages))
|
||||
payload: Dict[str, Any] = {
|
||||
"model": model,
|
||||
"messages": msg_dicts,
|
||||
"messages": messages_to_dicts(messages),
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
"stream": True,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Cloud inference engine — OpenAI, Anthropic, and Google API backends."""
|
||||
"""Cloud inference engine — OpenAI, Anthropic, Google, and MiniMax API backends."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -38,6 +38,10 @@ PRICING: Dict[str, tuple[float, float]] = {
|
||||
"gemini-3.1-flash-lite-preview": (0.30, 2.50),
|
||||
"gemini-3-flash-preview": (0.50, 3.00),
|
||||
"claude-haiku-4-5-20251001": (1.00, 5.00),
|
||||
"MiniMax-M2.7": (0.30, 1.20),
|
||||
"MiniMax-M2.7-highspeed": (0.60, 2.40),
|
||||
"MiniMax-M2.5": (0.30, 1.20),
|
||||
"MiniMax-M2.5-highspeed": (0.60, 2.40),
|
||||
}
|
||||
|
||||
# Well-known model IDs per provider
|
||||
@@ -62,6 +66,12 @@ _GOOGLE_MODELS = [
|
||||
"gemini-3.1-flash-lite-preview",
|
||||
"gemini-3-flash-preview",
|
||||
]
|
||||
_MINIMAX_MODELS = [
|
||||
"MiniMax-M2.7",
|
||||
"MiniMax-M2.7-highspeed",
|
||||
"MiniMax-M2.5",
|
||||
"MiniMax-M2.5-highspeed",
|
||||
]
|
||||
|
||||
# OpenRouter models — prefixed with "openrouter/" so they can be identified
|
||||
_OPENROUTER_POPULAR = [
|
||||
@@ -76,6 +86,10 @@ _OPENROUTER_POPULAR = [
|
||||
]
|
||||
|
||||
|
||||
def _is_minimax_model(model: str) -> bool:
|
||||
return model.lower().startswith("minimax")
|
||||
|
||||
|
||||
def _is_openrouter_model(model: str) -> bool:
|
||||
return model.startswith("openrouter/")
|
||||
|
||||
@@ -113,6 +127,31 @@ def estimate_cost(model: str, prompt_tokens: int, completion_tokens: int) -> flo
|
||||
return input_cost + output_cost
|
||||
|
||||
|
||||
def _annotate_anthropic_cache(messages: list[dict]) -> list[dict]:
|
||||
"""Add cache_control to system message for Anthropic prompt caching."""
|
||||
result = []
|
||||
for msg in messages:
|
||||
if msg.get("role") == "system":
|
||||
content = msg["content"]
|
||||
if isinstance(content, str):
|
||||
content = [
|
||||
{
|
||||
"type": "text",
|
||||
"text": content,
|
||||
"cache_control": {"type": "ephemeral"},
|
||||
}
|
||||
]
|
||||
elif isinstance(content, list):
|
||||
content = [
|
||||
{**block, "cache_control": {"type": "ephemeral"}}
|
||||
for block in content
|
||||
]
|
||||
result.append({**msg, "content": content})
|
||||
else:
|
||||
result.append(msg)
|
||||
return result
|
||||
|
||||
|
||||
def _convert_tools_to_anthropic(
|
||||
openai_tools: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
@@ -145,7 +184,7 @@ def _convert_tools_to_google(
|
||||
|
||||
@EngineRegistry.register("cloud")
|
||||
class CloudEngine(InferenceEngine):
|
||||
"""Cloud inference via OpenAI, Anthropic, and Google SDKs."""
|
||||
"""Cloud inference via OpenAI, Anthropic, Google, and MiniMax SDKs."""
|
||||
|
||||
engine_id = "cloud"
|
||||
|
||||
@@ -154,6 +193,7 @@ class CloudEngine(InferenceEngine):
|
||||
self._anthropic_client: Any = None
|
||||
self._google_client: Any = None
|
||||
self._openrouter_client: Any = None
|
||||
self._minimax_client: Any = None
|
||||
self._init_clients()
|
||||
|
||||
def _init_clients(self) -> None:
|
||||
@@ -189,6 +229,16 @@ class CloudEngine(InferenceEngine):
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
minimax_key = os.environ.get("MINIMAX_API_KEY")
|
||||
if minimax_key:
|
||||
try:
|
||||
import openai
|
||||
self._minimax_client = openai.OpenAI(
|
||||
base_url="https://api.minimax.io/v1",
|
||||
api_key=minimax_key,
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
def _generate_openai(
|
||||
self,
|
||||
@@ -601,6 +651,59 @@ class CloudEngine(InferenceEngine):
|
||||
"ttft": elapsed,
|
||||
}
|
||||
|
||||
def _generate_minimax(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float,
|
||||
max_tokens: int,
|
||||
**kwargs: Any,
|
||||
) -> Dict[str, Any]:
|
||||
if self._minimax_client is None:
|
||||
raise EngineConnectionError(
|
||||
"MiniMax client not available — set MINIMAX_API_KEY"
|
||||
)
|
||||
# MiniMax requires temperature in (0.0, 1.0]; clamp zero
|
||||
temperature = max(temperature, 0.01)
|
||||
temperature = min(temperature, 1.0)
|
||||
kwargs.pop("response_format", None)
|
||||
create_kwargs: Dict[str, Any] = {
|
||||
"model": model,
|
||||
"messages": messages_to_dicts(messages),
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
}
|
||||
t0 = time.monotonic()
|
||||
resp = self._minimax_client.chat.completions.create(**create_kwargs)
|
||||
elapsed = time.monotonic() - t0
|
||||
choice = resp.choices[0]
|
||||
usage = resp.usage
|
||||
prompt_tokens = usage.prompt_tokens if usage else 0
|
||||
completion_tokens = usage.completion_tokens if usage else 0
|
||||
result: Dict[str, Any] = {
|
||||
"content": choice.message.content or "",
|
||||
"usage": {
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"completion_tokens": completion_tokens,
|
||||
"total_tokens": (usage.total_tokens if usage else 0),
|
||||
},
|
||||
"model": resp.model,
|
||||
"finish_reason": choice.finish_reason or "stop",
|
||||
"cost_usd": estimate_cost(model, prompt_tokens, completion_tokens),
|
||||
"ttft": elapsed,
|
||||
}
|
||||
if hasattr(choice.message, "tool_calls") and choice.message.tool_calls:
|
||||
result["tool_calls"] = [
|
||||
{
|
||||
"id": tc.id,
|
||||
"name": tc.function.name,
|
||||
"arguments": tc.function.arguments,
|
||||
}
|
||||
for tc in choice.message.tool_calls
|
||||
]
|
||||
return result
|
||||
|
||||
def generate(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
@@ -618,6 +721,8 @@ class CloudEngine(InferenceEngine):
|
||||
)
|
||||
if _is_openrouter_model(model):
|
||||
return self._generate_openrouter(messages, **kw)
|
||||
if _is_minimax_model(model):
|
||||
return self._generate_minimax(messages, **kw)
|
||||
if _is_anthropic_model(model):
|
||||
return self._generate_anthropic(messages, **kw)
|
||||
if _is_google_model(model):
|
||||
@@ -644,6 +749,11 @@ class CloudEngine(InferenceEngine):
|
||||
messages, **kw
|
||||
):
|
||||
yield token
|
||||
elif _is_minimax_model(model):
|
||||
async for token in self._stream_minimax(
|
||||
messages, **kw
|
||||
):
|
||||
yield token
|
||||
elif _is_anthropic_model(model):
|
||||
async for token in self._stream_anthropic(
|
||||
messages, **kw
|
||||
@@ -779,6 +889,32 @@ class CloudEngine(InferenceEngine):
|
||||
if delta and delta.content:
|
||||
yield delta.content
|
||||
|
||||
async def _stream_minimax(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float,
|
||||
max_tokens: int,
|
||||
**kwargs: Any,
|
||||
) -> AsyncIterator[str]:
|
||||
if self._minimax_client is None:
|
||||
raise EngineConnectionError("MiniMax client not available")
|
||||
temperature = max(temperature, 0.01)
|
||||
temperature = min(temperature, 1.0)
|
||||
create_kwargs: Dict[str, Any] = {
|
||||
"model": model,
|
||||
"messages": messages_to_dicts(messages),
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": True,
|
||||
}
|
||||
resp = self._minimax_client.chat.completions.create(**create_kwargs)
|
||||
for chunk in resp:
|
||||
delta = chunk.choices[0].delta if chunk.choices else None
|
||||
if delta and delta.content:
|
||||
yield delta.content
|
||||
|
||||
def list_models(self) -> List[str]:
|
||||
models: List[str] = []
|
||||
if self._openai_client is not None:
|
||||
@@ -789,6 +925,8 @@ class CloudEngine(InferenceEngine):
|
||||
models.extend(_GOOGLE_MODELS)
|
||||
if self._openrouter_client is not None:
|
||||
models.extend(_OPENROUTER_POPULAR)
|
||||
if self._minimax_client is not None:
|
||||
models.extend(_MINIMAX_MODELS)
|
||||
return models
|
||||
|
||||
def health(self) -> bool:
|
||||
@@ -797,6 +935,7 @@ class CloudEngine(InferenceEngine):
|
||||
or self._anthropic_client is not None
|
||||
or self._google_client is not None
|
||||
or self._openrouter_client is not None
|
||||
or self._minimax_client is not None
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
@@ -814,6 +953,10 @@ class CloudEngine(InferenceEngine):
|
||||
if hasattr(self._openrouter_client, "close"):
|
||||
self._openrouter_client.close()
|
||||
self._openrouter_client = None
|
||||
if self._minimax_client is not None:
|
||||
if hasattr(self._minimax_client, "close"):
|
||||
self._minimax_client.close()
|
||||
self._minimax_client = None
|
||||
|
||||
|
||||
__all__ = ["CloudEngine", "PRICING", "estimate_cost"]
|
||||
__all__ = ["CloudEngine", "PRICING", "_annotate_anthropic_cache", "estimate_cost"]
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
"""gemma.cpp inference engine backend via pygemma pybind11 bindings."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from openjarvis.core.registry import EngineRegistry
|
||||
from openjarvis.core.types import Message, Role
|
||||
from openjarvis.engine._base import InferenceEngine, estimate_prompt_tokens
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _import_pygemma():
|
||||
"""Import and return the pygemma.Gemma class. Raises ImportError if unavailable."""
|
||||
from pygemma import Gemma
|
||||
|
||||
return Gemma
|
||||
|
||||
|
||||
@EngineRegistry.register("gemma_cpp")
|
||||
class GemmaCppEngine(InferenceEngine):
|
||||
"""gemma.cpp backend via pygemma pybind11 bindings (in-process, CPU)."""
|
||||
|
||||
engine_id = "gemma_cpp"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_path: str | None = None,
|
||||
tokenizer_path: str | None = None,
|
||||
model_type: str | None = None,
|
||||
num_threads: int = 0,
|
||||
) -> None:
|
||||
self._model_path = model_path or os.environ.get("GEMMA_CPP_MODEL_PATH", "")
|
||||
self._tokenizer_path = tokenizer_path or os.environ.get(
|
||||
"GEMMA_CPP_TOKENIZER_PATH", ""
|
||||
)
|
||||
self._model_type = model_type or os.environ.get("GEMMA_CPP_MODEL_TYPE", "")
|
||||
self._num_threads = num_threads or int(
|
||||
os.environ.get("GEMMA_CPP_NUM_THREADS", "0")
|
||||
)
|
||||
self._gemma: Any = None # lazy-loaded pygemma.Gemma instance
|
||||
|
||||
def _messages_to_prompt(self, messages: Sequence[Message]) -> str:
|
||||
"""Format messages into Gemma's chat template."""
|
||||
parts: list[str] = []
|
||||
system_prefix = ""
|
||||
for msg in messages:
|
||||
if msg.role == Role.SYSTEM:
|
||||
system_prefix += msg.content + "\n\n"
|
||||
elif msg.role == Role.USER:
|
||||
content = system_prefix + msg.content if system_prefix else msg.content
|
||||
system_prefix = ""
|
||||
parts.append(f"<start_of_turn>user\n{content}<end_of_turn>\n")
|
||||
elif msg.role == Role.ASSISTANT:
|
||||
parts.append(f"<start_of_turn>model\n{msg.content}<end_of_turn>\n")
|
||||
parts.append("<start_of_turn>model\n")
|
||||
return "".join(parts)
|
||||
|
||||
def _ensure_loaded(self) -> None:
|
||||
"""Lazy model loading — called before inference."""
|
||||
if self._gemma is None:
|
||||
if not self._model_path:
|
||||
raise FileNotFoundError(
|
||||
"gemma.cpp model_path not configured. Download weights "
|
||||
"from Kaggle and set GEMMA_CPP_MODEL_PATH or configure "
|
||||
"[engine.gemma_cpp] in ~/.openjarvis/config.toml"
|
||||
)
|
||||
if not self._tokenizer_path:
|
||||
raise FileNotFoundError(
|
||||
"gemma.cpp tokenizer_path not configured. Set "
|
||||
"GEMMA_CPP_TOKENIZER_PATH or configure "
|
||||
"[engine.gemma_cpp] in ~/.openjarvis/config.toml"
|
||||
)
|
||||
Gemma = _import_pygemma()
|
||||
self._gemma = Gemma()
|
||||
self._gemma.load_model(
|
||||
self._tokenizer_path, self._model_path, self._model_type
|
||||
)
|
||||
|
||||
def prepare(self, model: str) -> None:
|
||||
"""Load model into memory."""
|
||||
self._ensure_loaded()
|
||||
|
||||
def close(self) -> None:
|
||||
"""Unload model and free memory."""
|
||||
self._gemma = None
|
||||
|
||||
def generate(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
**kwargs: Any,
|
||||
) -> Dict[str, Any]:
|
||||
self._ensure_loaded()
|
||||
if model != self._model_type:
|
||||
logger.warning(
|
||||
"gemma_cpp: requested model %r but loaded model is %r; "
|
||||
"proceeding with loaded model",
|
||||
model,
|
||||
self._model_type,
|
||||
)
|
||||
prompt = self._messages_to_prompt(messages)
|
||||
try:
|
||||
# pygemma v0.1.3 completion() does not accept temperature/max_tokens;
|
||||
# these params are accepted in the signature for ABC compliance but
|
||||
# not forwarded until pygemma or a vendored wrapper supports them.
|
||||
raw = self._gemma.completion(prompt)
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"gemma.cpp inference failed: {exc}") from exc
|
||||
|
||||
prompt_tokens = estimate_prompt_tokens(messages)
|
||||
completion_tokens = max(1, len(raw) // 4)
|
||||
return {
|
||||
"content": raw,
|
||||
"usage": {
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"completion_tokens": completion_tokens,
|
||||
"total_tokens": prompt_tokens + completion_tokens,
|
||||
},
|
||||
"model": self._model_type,
|
||||
"finish_reason": "stop",
|
||||
}
|
||||
|
||||
async def stream(
|
||||
self,
|
||||
messages: Sequence[Message],
|
||||
*,
|
||||
model: str,
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int = 1024,
|
||||
**kwargs: Any,
|
||||
) -> AsyncIterator[str]:
|
||||
self._ensure_loaded()
|
||||
if model != self._model_type:
|
||||
logger.warning(
|
||||
"gemma_cpp: requested model %r but loaded model is %r; "
|
||||
"proceeding with loaded model",
|
||||
model,
|
||||
self._model_type,
|
||||
)
|
||||
prompt = self._messages_to_prompt(messages)
|
||||
try:
|
||||
raw = self._gemma.completion(prompt)
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"gemma.cpp inference failed: {exc}") from exc
|
||||
yield raw
|
||||
|
||||
def _paths_valid(self) -> bool:
|
||||
"""Check that model and tokenizer paths are configured and exist."""
|
||||
return bool(
|
||||
self._model_path
|
||||
and self._tokenizer_path
|
||||
and os.path.isfile(self._model_path)
|
||||
and os.path.isfile(self._tokenizer_path)
|
||||
)
|
||||
|
||||
def list_models(self) -> List[str]:
|
||||
if self._model_type and self._paths_valid():
|
||||
return [self._model_type]
|
||||
return []
|
||||
|
||||
def health(self) -> bool:
|
||||
if not self._paths_valid():
|
||||
return False
|
||||
try:
|
||||
_import_pygemma()
|
||||
return True
|
||||
except ImportError:
|
||||
return False
|
||||
|
||||
|
||||
__all__ = ["GemmaCppEngine"]
|
||||
@@ -4,6 +4,10 @@ Thin FastAPI server wrapping the Nexa SDK (``nexaai``) as an
|
||||
OpenAI-compatible API on port 18181. Intended for on-device inference
|
||||
with GGUF models on Apple Silicon or CPU.
|
||||
|
||||
**Token counts:** The Nexa SDK does not expose token counts in responses.
|
||||
The shim returns 0 for prompt/completion/total tokens. Savings and
|
||||
leaderboard metrics will not include sessions that use this engine.
|
||||
|
||||
Usage:
|
||||
uvicorn openjarvis.engine.nexa_shim:app \
|
||||
--host 127.0.0.1 --port 18181
|
||||
|
||||
@@ -15,6 +15,7 @@ from openjarvis.core.types import Message
|
||||
from openjarvis.engine._base import (
|
||||
EngineConnectionError,
|
||||
InferenceEngine,
|
||||
estimate_prompt_tokens,
|
||||
messages_to_dicts,
|
||||
)
|
||||
|
||||
@@ -105,13 +106,24 @@ class OllamaEngine(InferenceEngine):
|
||||
f"Ollama returned {exc.response.status_code}: {body}"
|
||||
) from exc
|
||||
data = resp.json()
|
||||
prompt_tokens = data.get("prompt_eval_count", 0)
|
||||
# prompt_eval_count = tokens actually evaluated (KV-cache-aware).
|
||||
# estimate_prompt_tokens = full prompt size (for cost comparison).
|
||||
# We report both so downstream can use the right one:
|
||||
# prompt_tokens → full size (what cloud would charge)
|
||||
# prompt_tokens_evaluated → actual compute (with KV cache)
|
||||
reported_prompt = data.get("prompt_eval_count", 0)
|
||||
estimated_prompt = estimate_prompt_tokens(messages)
|
||||
prompt_tokens = max(reported_prompt, estimated_prompt)
|
||||
prompt_tokens_evaluated = (
|
||||
reported_prompt if reported_prompt > 0 else prompt_tokens
|
||||
)
|
||||
completion_tokens = data.get("eval_count", 0)
|
||||
content = data.get("message", {}).get("content", "")
|
||||
result: Dict[str, Any] = {
|
||||
"content": content,
|
||||
"usage": {
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"prompt_tokens_evaluated": prompt_tokens_evaluated,
|
||||
"completion_tokens": completion_tokens,
|
||||
"total_tokens": prompt_tokens + completion_tokens,
|
||||
},
|
||||
@@ -176,14 +188,19 @@ class OllamaEngine(InferenceEngine):
|
||||
if content:
|
||||
yield content
|
||||
if chunk.get("done", False):
|
||||
# Capture usage from final chunk
|
||||
reported_prompt = chunk.get("prompt_eval_count", 0)
|
||||
est_prompt = estimate_prompt_tokens(messages)
|
||||
full_prompt = max(reported_prompt, est_prompt)
|
||||
evaluated = (
|
||||
reported_prompt if reported_prompt > 0
|
||||
else full_prompt
|
||||
)
|
||||
comp = chunk.get("eval_count", 0)
|
||||
self._last_stream_usage = {
|
||||
"prompt_tokens": chunk.get("prompt_eval_count", 0),
|
||||
"completion_tokens": chunk.get("eval_count", 0),
|
||||
"total_tokens": (
|
||||
chunk.get("prompt_eval_count", 0)
|
||||
+ chunk.get("eval_count", 0)
|
||||
),
|
||||
"prompt_tokens": full_prompt,
|
||||
"prompt_tokens_evaluated": evaluated,
|
||||
"completion_tokens": comp,
|
||||
"total_tokens": full_prompt + comp,
|
||||
}
|
||||
break
|
||||
except (httpx.ConnectError, httpx.TimeoutException) as exc:
|
||||
|
||||
@@ -119,6 +119,10 @@ BENCHMARKS = {
|
||||
"category": "use-case",
|
||||
"description": "Web research with fact verification",
|
||||
},
|
||||
"pinchbench": {
|
||||
"category": "agentic",
|
||||
"description": "PinchBench real-world agent tasks",
|
||||
},
|
||||
}
|
||||
|
||||
BACKENDS = {
|
||||
@@ -260,6 +264,9 @@ def _build_dataset(benchmark: str, subset: str | None = None):
|
||||
elif benchmark == "browser_assistant":
|
||||
from openjarvis.evals.datasets.browser_assistant import BrowserAssistantDataset
|
||||
return BrowserAssistantDataset()
|
||||
elif benchmark == "pinchbench":
|
||||
from openjarvis.evals.datasets.pinchbench import PinchBenchDataset
|
||||
return PinchBenchDataset(path=subset)
|
||||
else:
|
||||
raise click.ClickException(f"Unknown benchmark: {benchmark}")
|
||||
|
||||
@@ -361,6 +368,9 @@ def _build_scorer(benchmark: str, judge_backend, judge_model: str):
|
||||
elif benchmark == "browser_assistant":
|
||||
from openjarvis.evals.scorers.browser_assistant import BrowserAssistantScorer
|
||||
return BrowserAssistantScorer(judge_backend, judge_model)
|
||||
elif benchmark == "pinchbench":
|
||||
from openjarvis.evals.scorers.pinchbench import PinchBenchScorer
|
||||
return PinchBenchScorer(judge_backend, judge_model)
|
||||
else:
|
||||
raise click.ClickException(f"Unknown benchmark: {benchmark}")
|
||||
|
||||
@@ -519,6 +529,16 @@ def _run_agentic(
|
||||
seed=config.seed,
|
||||
)
|
||||
|
||||
# Wire judge for PinchBench LLM-judge and hybrid grading
|
||||
if config.benchmark == "pinchbench" and hasattr(dataset, "set_judge"):
|
||||
judge_engine = getattr(config, "judge_engine", "cloud") or "cloud"
|
||||
judge_model = (
|
||||
getattr(config, "judge_model", None)
|
||||
or "anthropic/claude-opus-4-5"
|
||||
)
|
||||
judge_backend = _build_judge_backend(judge_model, engine_key=judge_engine)
|
||||
dataset.set_judge(judge_backend, judge_model)
|
||||
|
||||
# Verify backend requirements before doing any work
|
||||
if hasattr(dataset, "verify_requirements"):
|
||||
issues = dataset.verify_requirements()
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# PinchBench eval: gpt-5.4 (cloud)
|
||||
# Agent: native_openhands — all PinchBench-required tools enabled
|
||||
|
||||
[meta]
|
||||
name = "pinchbench-gpt54"
|
||||
description = "PinchBench on gpt-5.4-2026-03-05"
|
||||
|
||||
[defaults]
|
||||
temperature = 0.6
|
||||
max_tokens = 8192
|
||||
|
||||
[judge]
|
||||
model = "claude-opus-4-5"
|
||||
temperature = 0.0
|
||||
engine = "cloud"
|
||||
|
||||
[run]
|
||||
max_workers = 1
|
||||
output_dir = "results/pinchbench-gpt54/"
|
||||
seed = 42
|
||||
|
||||
[[models]]
|
||||
name = "gpt-5.4-2026-03-05"
|
||||
engine = "cloud"
|
||||
|
||||
[[benchmarks]]
|
||||
name = "pinchbench"
|
||||
backend = "jarvis-agent"
|
||||
agent = "native_openhands"
|
||||
tools = [
|
||||
"think", "file_read", "file_write", "web_search", "shell_exec",
|
||||
"code_interpreter", "browser_navigate", "image_generate",
|
||||
"calculator", "http_request", "pdf_extract",
|
||||
]
|
||||
@@ -0,0 +1,35 @@
|
||||
# PinchBench eval: Qwen3.5-397B-A17B-FP8 (vLLM, TP=8)
|
||||
# Agent: native_openhands — all PinchBench-required tools enabled
|
||||
|
||||
[meta]
|
||||
name = "pinchbench-qwen397b"
|
||||
description = "PinchBench on Qwen/Qwen3.5-397B-A17B-FP8 (vLLM, TP=8)"
|
||||
|
||||
[defaults]
|
||||
temperature = 0.6
|
||||
max_tokens = 8192
|
||||
|
||||
[judge]
|
||||
model = "claude-opus-4-5"
|
||||
temperature = 0.0
|
||||
engine = "cloud"
|
||||
|
||||
[run]
|
||||
max_workers = 1
|
||||
output_dir = "results/pinchbench-qwen397b/"
|
||||
seed = 42
|
||||
|
||||
[[models]]
|
||||
name = "Qwen/Qwen3.5-397B-A17B-FP8"
|
||||
engine = "vllm"
|
||||
num_gpus = 8
|
||||
|
||||
[[benchmarks]]
|
||||
name = "pinchbench"
|
||||
backend = "jarvis-agent"
|
||||
agent = "native_openhands"
|
||||
tools = [
|
||||
"think", "file_read", "file_write", "web_search", "shell_exec",
|
||||
"code_interpreter", "browser_navigate", "image_generate",
|
||||
"calculator", "http_request", "pdf_extract",
|
||||
]
|
||||
@@ -0,0 +1,26 @@
|
||||
[meta]
|
||||
name = "pinchbench"
|
||||
description = "PinchBench real-world agent benchmark"
|
||||
|
||||
[defaults]
|
||||
temperature = 0.0
|
||||
|
||||
[judge]
|
||||
model = "anthropic/claude-opus-4-5"
|
||||
temperature = 0.0
|
||||
|
||||
[run]
|
||||
output_dir = "results/pinchbench"
|
||||
|
||||
[[models]]
|
||||
name = "anthropic/claude-sonnet-4"
|
||||
|
||||
[[benchmarks]]
|
||||
name = "pinchbench"
|
||||
backend = "jarvis-agent"
|
||||
agent = "native_react"
|
||||
tools = [
|
||||
"file_read", "file_write", "web_search", "shell_exec",
|
||||
"code_interpreter", "browser_navigate", "image_generate",
|
||||
"calculator", "http_request", "pdf_extract",
|
||||
]
|
||||
@@ -321,6 +321,19 @@ class AgenticRunner:
|
||||
|
||||
event_recorder.clear()
|
||||
|
||||
# Bridge EventBus → EventRecorder so tool events are captured.
|
||||
# ToolExecutor publishes to JarvisSystem.bus; we relay those into
|
||||
# the EventRecorder for transcript building and trace enrichment.
|
||||
_bus_unsubs: list[tuple] = []
|
||||
agent_bus = getattr(agent, "bus", None)
|
||||
if agent_bus is not None:
|
||||
for etype in (EventType.TOOL_CALL_START, EventType.TOOL_CALL_END):
|
||||
def _relay(event, _etype=etype):
|
||||
data = event.data if hasattr(event, "data") else {}
|
||||
event_recorder.record(_etype, **data)
|
||||
agent_bus.subscribe(etype, _relay)
|
||||
_bus_unsubs.append((etype, _relay))
|
||||
|
||||
# Set up per-query workspace
|
||||
if self._run_dir and hasattr(agent, "set_workspace"):
|
||||
instance_id = record.metadata.get("instance_id", record.record_id)
|
||||
@@ -330,6 +343,7 @@ class AgenticRunner:
|
||||
)
|
||||
workspace.mkdir(parents=True, exist_ok=True)
|
||||
agent.set_workspace(str(workspace))
|
||||
record.metadata["workspace_path"] = str(workspace)
|
||||
|
||||
# Create per-task execution environment (e.g. Docker for TerminalBench)
|
||||
task_env = None
|
||||
@@ -397,6 +411,10 @@ class AgenticRunner:
|
||||
else:
|
||||
response_text = str(agent(record.problem))
|
||||
|
||||
# Wire event recorder to task env if supported
|
||||
if task_env is not None and hasattr(task_env, "set_event_recorder"):
|
||||
task_env.set_event_recorder(event_recorder)
|
||||
|
||||
# Run tests if task env supports it
|
||||
if task_env is not None and hasattr(task_env, "run_tests"):
|
||||
task_env.run_tests()
|
||||
@@ -425,6 +443,13 @@ class AgenticRunner:
|
||||
except Exception as exc:
|
||||
LOGGER.warning("Agent failed on query %s: %s", query_id, exc)
|
||||
end_time = time.time()
|
||||
# Unsubscribe EventBus relays
|
||||
if agent_bus is not None:
|
||||
for etype, cb in _bus_unsubs:
|
||||
try:
|
||||
agent_bus.unsubscribe(etype, cb)
|
||||
except Exception:
|
||||
pass
|
||||
return QueryTrace(
|
||||
query_id=query_id,
|
||||
workload_type=str(workload_type),
|
||||
@@ -435,6 +460,14 @@ class AgenticRunner:
|
||||
is_resolved=record.metadata.get("is_resolved"),
|
||||
)
|
||||
|
||||
# Unsubscribe EventBus relays
|
||||
if agent_bus is not None:
|
||||
for etype, cb in _bus_unsubs:
|
||||
try:
|
||||
agent_bus.unsubscribe(etype, cb)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
end_time = time.time()
|
||||
end_ns = time.monotonic_ns()
|
||||
|
||||
@@ -578,6 +611,8 @@ class AgenticRunner:
|
||||
current_turn_start: Optional[float] = None
|
||||
current_tools: list[str] = []
|
||||
current_tool_latencies: dict[str, float] = {}
|
||||
current_tool_calls: list[dict[str, Any]] = []
|
||||
current_tool_args: dict[str, Any] = {}
|
||||
tool_start_times: dict[str, float] = {}
|
||||
input_tokens = 0
|
||||
output_tokens = 0
|
||||
@@ -624,6 +659,7 @@ class AgenticRunner:
|
||||
output_tokens=output_tokens,
|
||||
tools_called=list(current_tools),
|
||||
tool_latencies_s=dict(current_tool_latencies),
|
||||
tool_calls=list(current_tool_calls),
|
||||
wall_clock_s=wall_clock,
|
||||
action_energy_breakdown=action_breakdown,
|
||||
)
|
||||
@@ -633,6 +669,8 @@ class AgenticRunner:
|
||||
current_turn_start = None
|
||||
current_tools = []
|
||||
current_tool_latencies = {}
|
||||
current_tool_calls = []
|
||||
current_tool_args = {}
|
||||
current_action_spans = []
|
||||
input_tokens = 0
|
||||
output_tokens = 0
|
||||
@@ -640,10 +678,16 @@ class AgenticRunner:
|
||||
elif etype == EventType.TOOL_CALL_START:
|
||||
tool_name = event.metadata.get("tool", "unknown")
|
||||
tool_start_times[tool_name] = event.timestamp
|
||||
current_tool_args[tool_name] = event.metadata.get("arguments", {})
|
||||
|
||||
elif etype == EventType.TOOL_CALL_END:
|
||||
tool_name = event.metadata.get("tool", "unknown")
|
||||
current_tools.append(tool_name)
|
||||
current_tool_calls.append({
|
||||
"name": tool_name,
|
||||
"arguments": current_tool_args.pop(tool_name, {}),
|
||||
"result": event.metadata.get("result", ""),
|
||||
})
|
||||
start_ts = tool_start_times.pop(tool_name, None)
|
||||
if start_ts is not None:
|
||||
duration = event.timestamp - start_ts
|
||||
@@ -662,6 +706,7 @@ class AgenticRunner:
|
||||
turn_index=0,
|
||||
tools_called=current_tools,
|
||||
tool_latencies_s=current_tool_latencies,
|
||||
tool_calls=list(current_tool_calls),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -43,6 +43,7 @@ KNOWN_BENCHMARKS = {
|
||||
"knowledge_base", "coding_task",
|
||||
"coding_assistant", "security_scanner", "daily_digest",
|
||||
"doc_qa", "browser_assistant",
|
||||
"pinchbench",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -96,11 +96,19 @@ class EvalRunner:
|
||||
seed=cfg.seed,
|
||||
)
|
||||
|
||||
# Auto-enable episode_mode when the dataset has iter_episodes()
|
||||
# (i.e. it is a lifelong/sequential benchmark like LifelongAgentBench).
|
||||
# This is enforced at the runner level so it applies regardless of
|
||||
# how the runner is invoked (CLI, SDK, tests, etc.).
|
||||
if not cfg.episode_mode and hasattr(self._dataset, "iter_episodes"):
|
||||
# Auto-enable episode_mode when the dataset *overrides*
|
||||
# iter_episodes() (i.e. it is a lifelong/sequential benchmark like
|
||||
# LifelongAgentBench). The base DatasetProvider always defines a
|
||||
# default iter_episodes() that wraps each record in its own episode,
|
||||
# so hasattr() is always True — we must check for a real override.
|
||||
from openjarvis.evals.core.dataset import DatasetProvider as _DP
|
||||
try:
|
||||
_overrides_episodes = (
|
||||
type(self._dataset).iter_episodes is not _DP.iter_episodes
|
||||
)
|
||||
except AttributeError:
|
||||
_overrides_episodes = False
|
||||
if not cfg.episode_mode and _overrides_episodes:
|
||||
LOGGER.info(
|
||||
"%s requires sequential episode processing — "
|
||||
"auto-enabling episode_mode.",
|
||||
@@ -109,6 +117,15 @@ class EvalRunner:
|
||||
cfg = dataclasses.replace(cfg, episode_mode=True)
|
||||
self._config = cfg
|
||||
|
||||
# Detect if dataset provides task environments (e.g. PinchBench)
|
||||
try:
|
||||
self._has_task_env = (
|
||||
type(self._dataset).create_task_env
|
||||
is not _DP.create_task_env
|
||||
)
|
||||
except AttributeError:
|
||||
self._has_task_env = False
|
||||
|
||||
records = list(self._dataset.iter_records())
|
||||
LOGGER.info(
|
||||
"Running %s: %d samples, backend=%s, model=%s, workers=%d, "
|
||||
@@ -145,6 +162,15 @@ class EvalRunner:
|
||||
try:
|
||||
if cfg.episode_mode:
|
||||
self._run_episode_mode(records, progress_callback, total)
|
||||
elif self._has_task_env:
|
||||
# Task environments (PinchBench etc.) change CWD —
|
||||
# must process sequentially for thread safety.
|
||||
for record in records:
|
||||
result = self._process_one(record)
|
||||
self._results.append(result)
|
||||
self._flush_result(result)
|
||||
if progress_callback is not None:
|
||||
progress_callback(len(self._results), total)
|
||||
else:
|
||||
with ThreadPoolExecutor(max_workers=cfg.max_workers) as pool:
|
||||
futures = {
|
||||
@@ -224,17 +250,41 @@ class EvalRunner:
|
||||
)
|
||||
if cfg.system_prompt:
|
||||
gen_kwargs["system"] = cfg.system_prompt
|
||||
full = self._backend.generate_full(
|
||||
record.problem,
|
||||
**gen_kwargs,
|
||||
)
|
||||
content = full.get("content", "")
|
||||
|
||||
if getattr(self, "_has_task_env", False):
|
||||
from contextlib import nullcontext
|
||||
task_env = self._dataset.create_task_env(record)
|
||||
ctx = task_env if task_env is not None else nullcontext()
|
||||
with ctx:
|
||||
full = self._backend.generate_full(
|
||||
record.problem,
|
||||
**gen_kwargs,
|
||||
)
|
||||
full = full or {}
|
||||
# Store tool results for the scorer to build transcripts
|
||||
record.metadata["tool_results"] = full.get(
|
||||
"tool_results", [],
|
||||
)
|
||||
# Score INSIDE context so workspace files still exist
|
||||
content = full.get("content", "")
|
||||
is_correct, scoring_meta = self._scorer.score(
|
||||
record, content,
|
||||
)
|
||||
else:
|
||||
full = self._backend.generate_full(
|
||||
record.problem,
|
||||
**gen_kwargs,
|
||||
)
|
||||
full = full or {}
|
||||
content = full.get("content", "")
|
||||
is_correct, scoring_meta = self._scorer.score(
|
||||
record, content,
|
||||
)
|
||||
|
||||
usage = full.get("usage", {})
|
||||
latency = full.get("latency_seconds", 0.0)
|
||||
cost = full.get("cost_usd", 0.0)
|
||||
|
||||
is_correct, scoring_meta = self._scorer.score(record, content)
|
||||
|
||||
energy_j = full.get("energy_joules", 0.0)
|
||||
power_w = full.get("power_watts", 0.0)
|
||||
throughput = full.get("throughput_tok_per_sec", 0.0)
|
||||
|
||||
@@ -28,6 +28,8 @@ class TurnTrace:
|
||||
cost_usd: Optional[float] = None
|
||||
# Per-action energy breakdown (lm_inference vs tool_call granularity)
|
||||
action_energy_breakdown: Optional[List[Dict[str, Any]]] = None
|
||||
# Detailed tool call data: [{"name": str, "arguments": dict, "result": str}]
|
||||
tool_calls: List[Dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
@@ -45,6 +47,7 @@ class TurnTrace:
|
||||
"cpu_power_avg_watts": self.cpu_power_avg_watts,
|
||||
"cost_usd": self.cost_usd,
|
||||
"action_energy_breakdown": self.action_energy_breakdown,
|
||||
"tool_calls": [dict(tc) for tc in self.tool_calls],
|
||||
}
|
||||
|
||||
@classmethod
|
||||
@@ -64,6 +67,7 @@ class TurnTrace:
|
||||
cpu_power_avg_watts=d.get("cpu_power_avg_watts"),
|
||||
cost_usd=d.get("cost_usd"),
|
||||
action_energy_breakdown=d.get("action_energy_breakdown"),
|
||||
tool_calls=d.get("tool_calls", []),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
"""PinchBench dataset provider — real-world agent task benchmark.
|
||||
|
||||
Clones the pinchbench/skill repo at runtime and parses task markdown files
|
||||
into EvalRecords for use with AgenticRunner.
|
||||
|
||||
Reference: https://github.com/pinchbench/skill
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import random
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional
|
||||
|
||||
import yaml
|
||||
|
||||
from openjarvis.evals.core.dataset import DatasetProvider
|
||||
from openjarvis.evals.core.types import EvalRecord
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
PINCHBENCH_REPO = "https://github.com/pinchbench/skill.git"
|
||||
CACHE_DIR = Path.home() / ".cache" / "pinchbench"
|
||||
|
||||
|
||||
def _parse_task_markdown(content: str, filename: str = "") -> Dict[str, Any]:
|
||||
"""Parse a PinchBench task markdown file into a dict.
|
||||
|
||||
Extracts YAML frontmatter and markdown sections (## Prompt,
|
||||
## Expected Behavior, ## Automated Checks, ## LLM Judge Rubric).
|
||||
"""
|
||||
# Split frontmatter
|
||||
parts = content.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
raise ValueError(f"Missing YAML frontmatter in {filename}")
|
||||
|
||||
frontmatter = yaml.safe_load(parts[1])
|
||||
body = parts[2]
|
||||
|
||||
# Parse sections by ## headers
|
||||
sections: Dict[str, str] = {}
|
||||
current_header: Optional[str] = None
|
||||
current_lines: List[str] = []
|
||||
|
||||
for line in body.split("\n"):
|
||||
header_match = re.match(r"^##\s+(.+)$", line)
|
||||
if header_match:
|
||||
if current_header is not None:
|
||||
sections[current_header] = "\n".join(current_lines).strip()
|
||||
current_header = header_match.group(1).strip()
|
||||
current_lines = []
|
||||
else:
|
||||
current_lines.append(line)
|
||||
|
||||
if current_header is not None:
|
||||
sections[current_header] = "\n".join(current_lines).strip()
|
||||
|
||||
# Extract Python code block from Automated Checks section
|
||||
automated_checks = None
|
||||
checks_section = sections.get("Automated Checks", "")
|
||||
code_match = re.search(r"```python\s*\n(.*?)```", checks_section, re.DOTALL)
|
||||
if code_match:
|
||||
automated_checks = code_match.group(1).strip()
|
||||
|
||||
return {
|
||||
"id": frontmatter.get("id", ""),
|
||||
"name": frontmatter.get("name", ""),
|
||||
"category": frontmatter.get("category", ""),
|
||||
"grading_type": frontmatter.get("grading_type", "automated"),
|
||||
"timeout_seconds": frontmatter.get("timeout_seconds", 180),
|
||||
"workspace_files": frontmatter.get("workspace_files", []),
|
||||
"grading_weights": frontmatter.get("grading_weights"),
|
||||
"prompt": sections.get("Prompt", ""),
|
||||
"expected_behavior": sections.get("Expected Behavior", ""),
|
||||
"grading_criteria": sections.get("Grading Criteria", ""),
|
||||
"automated_checks": automated_checks,
|
||||
"llm_judge_rubric": sections.get("LLM Judge Rubric"),
|
||||
}
|
||||
|
||||
|
||||
class PinchBenchDataset(DatasetProvider):
|
||||
"""PinchBench real-world agent benchmark.
|
||||
|
||||
Clones pinchbench/skill from GitHub (or uses a local path) and
|
||||
parses task markdown files into EvalRecords.
|
||||
"""
|
||||
|
||||
dataset_id = "pinchbench"
|
||||
dataset_name = "PinchBench"
|
||||
|
||||
def __init__(self, path: Optional[str] = None) -> None:
|
||||
self._local_path = Path(path) if path else None
|
||||
self._repo_dir: Path = self._local_path or CACHE_DIR
|
||||
self._records: List[EvalRecord] = []
|
||||
|
||||
def verify_requirements(self) -> List[str]:
|
||||
issues: List[str] = []
|
||||
if self._local_path is None and shutil.which("git") is None:
|
||||
issues.append("git binary not found. Install git to clone PinchBench tasks.")
|
||||
if self._repo_dir.exists() and not (self._repo_dir / "tasks").is_dir():
|
||||
issues.append(
|
||||
f"PinchBench cache at {self._repo_dir} is corrupted (missing tasks/). "
|
||||
"Delete and re-run to re-clone."
|
||||
)
|
||||
return issues
|
||||
|
||||
def _ensure_repo(self) -> Path:
|
||||
"""Clone the repo if not already cached. Returns repo dir."""
|
||||
if self._local_path is not None:
|
||||
if not self._local_path.exists():
|
||||
raise FileNotFoundError(f"PinchBench path not found: {self._local_path}")
|
||||
return self._local_path
|
||||
|
||||
if not self._repo_dir.exists():
|
||||
LOGGER.info("Cloning PinchBench from %s ...", PINCHBENCH_REPO)
|
||||
self._repo_dir.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
["git", "clone", "--depth", "1", PINCHBENCH_REPO, str(self._repo_dir)],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
LOGGER.info("PinchBench cloned to %s", self._repo_dir)
|
||||
|
||||
return self._repo_dir
|
||||
|
||||
def load(
|
||||
self,
|
||||
*,
|
||||
max_samples: Optional[int] = None,
|
||||
split: Optional[str] = None,
|
||||
seed: Optional[int] = None,
|
||||
) -> None:
|
||||
repo_dir = self._ensure_repo()
|
||||
tasks_dir = repo_dir / "tasks"
|
||||
|
||||
if not tasks_dir.is_dir():
|
||||
raise FileNotFoundError(f"No tasks/ directory in {repo_dir}")
|
||||
|
||||
task_files = sorted(tasks_dir.glob("task_*.md"))
|
||||
if not task_files:
|
||||
raise FileNotFoundError(f"No task_*.md files in {tasks_dir}")
|
||||
|
||||
tasks = []
|
||||
for tf in task_files:
|
||||
try:
|
||||
parsed = _parse_task_markdown(tf.read_text(), filename=tf.name)
|
||||
tasks.append(parsed)
|
||||
except Exception as exc:
|
||||
LOGGER.warning("Skipping %s: %s", tf.name, exc)
|
||||
|
||||
if seed is not None:
|
||||
random.Random(seed).shuffle(tasks)
|
||||
if max_samples is not None:
|
||||
tasks = tasks[:max_samples]
|
||||
|
||||
self._records = [
|
||||
EvalRecord(
|
||||
record_id=t["id"],
|
||||
problem=t["prompt"],
|
||||
reference=t["expected_behavior"],
|
||||
category=t["category"],
|
||||
subject=t["name"],
|
||||
metadata={
|
||||
"grading_type": t["grading_type"],
|
||||
"grading_weights": t["grading_weights"],
|
||||
"automated_checks": t["automated_checks"],
|
||||
"llm_judge_rubric": t["llm_judge_rubric"],
|
||||
"timeout_seconds": t["timeout_seconds"],
|
||||
"workspace_files": t["workspace_files"],
|
||||
"pinchbench_repo_dir": str(repo_dir),
|
||||
},
|
||||
)
|
||||
for t in tasks
|
||||
]
|
||||
|
||||
LOGGER.info("PinchBench: loaded %d tasks", len(self._records))
|
||||
|
||||
def iter_records(self) -> Iterable[EvalRecord]:
|
||||
return iter(self._records)
|
||||
|
||||
def size(self) -> int:
|
||||
return len(self._records)
|
||||
|
||||
def set_judge(self, judge_backend: Any, judge_model: str) -> None:
|
||||
"""Set the judge backend/model for LLM-judge and hybrid grading."""
|
||||
self._judge_backend = judge_backend
|
||||
self._judge_model = judge_model
|
||||
|
||||
def create_task_env(self, record: EvalRecord):
|
||||
from openjarvis.evals.execution.pinchbench_env import PinchBenchTaskEnv
|
||||
return PinchBenchTaskEnv(
|
||||
record,
|
||||
judge_backend=getattr(self, "_judge_backend", None),
|
||||
judge_model=getattr(self, "_judge_model", "anthropic/claude-opus-4-5"),
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["PinchBenchDataset"]
|
||||
@@ -0,0 +1,159 @@
|
||||
"""PinchBench task environment — per-task workspace setup and grading."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from types import TracebackType
|
||||
from typing import Any, Optional, Type
|
||||
|
||||
from openjarvis.evals.core.types import EvalRecord
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PinchBenchTaskEnv:
|
||||
"""Per-task workspace environment for PinchBench.
|
||||
|
||||
Context manager that creates an isolated workspace directory,
|
||||
populates fixture files from the task definition, and runs
|
||||
grading after agent execution via run_tests().
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
record: EvalRecord,
|
||||
judge_backend: Any = None,
|
||||
judge_model: str = "anthropic/claude-opus-4-5",
|
||||
) -> None:
|
||||
self._record = record
|
||||
self._judge_backend = judge_backend
|
||||
self._judge_model = judge_model
|
||||
self._workspace: Optional[Path] = None
|
||||
self._owns_workspace: bool = True
|
||||
self._event_recorder: Any = None
|
||||
self._original_cwd: Optional[str] = None
|
||||
|
||||
def set_event_recorder(self, recorder: Any) -> None:
|
||||
"""Receive the EventRecorder from AgenticRunner for transcript building."""
|
||||
self._event_recorder = recorder
|
||||
|
||||
def __enter__(self) -> PinchBenchTaskEnv:
|
||||
# Use the AgenticRunner's workspace if already set (agent.set_workspace
|
||||
# is called before create_task_env), otherwise create a temp dir.
|
||||
existing = self._record.metadata.get("workspace_path")
|
||||
if existing and Path(existing).is_dir():
|
||||
self._workspace = Path(existing)
|
||||
self._owns_workspace = False
|
||||
else:
|
||||
self._workspace = Path(tempfile.mkdtemp(prefix="pinchbench_"))
|
||||
self._owns_workspace = True
|
||||
|
||||
# Populate fixture files
|
||||
repo_dir = Path(self._record.metadata.get("pinchbench_repo_dir", ""))
|
||||
workspace_files = self._record.metadata.get("workspace_files", [])
|
||||
|
||||
for file_spec in workspace_files:
|
||||
if "content" in file_spec:
|
||||
# Inline content
|
||||
dest_key = file_spec.get("path", file_spec.get("dest", "file.txt"))
|
||||
dest = self._workspace / dest_key
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
dest.write_text(file_spec["content"])
|
||||
elif "source" in file_spec:
|
||||
# Asset file from repo
|
||||
source = repo_dir / "assets" / file_spec["source"]
|
||||
dest_key = file_spec.get(
|
||||
"dest", file_spec.get("path", file_spec["source"])
|
||||
)
|
||||
dest = self._workspace / dest_key
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
if source.exists():
|
||||
shutil.copy2(str(source), str(dest))
|
||||
else:
|
||||
LOGGER.warning("Asset not found: %s", source)
|
||||
|
||||
self._record.metadata["workspace_path"] = str(self._workspace)
|
||||
|
||||
# Change CWD so file_read/file_write resolve paths relative to workspace
|
||||
self._original_cwd = os.getcwd()
|
||||
os.chdir(str(self._workspace))
|
||||
|
||||
LOGGER.info(
|
||||
"PinchBench workspace: %s (task %s)",
|
||||
self._workspace,
|
||||
self._record.record_id,
|
||||
)
|
||||
return self
|
||||
|
||||
def run_tests(self) -> None:
|
||||
"""Grade the agent's work using PinchBench grading logic.
|
||||
|
||||
Called by AgenticRunner after agent execution, before QueryTrace
|
||||
is constructed. Builds transcript from raw EventRecorder events.
|
||||
"""
|
||||
from openjarvis.evals.scorers.pinchbench import (
|
||||
events_to_transcript,
|
||||
grade_pinchbench_task,
|
||||
)
|
||||
|
||||
workspace_path = self._record.metadata.get("workspace_path", "")
|
||||
events = self._event_recorder.get_events() if self._event_recorder else []
|
||||
transcript = events_to_transcript(events)
|
||||
|
||||
try:
|
||||
result = grade_pinchbench_task(
|
||||
record=self._record,
|
||||
transcript=transcript,
|
||||
workspace_path=workspace_path,
|
||||
judge_backend=self._judge_backend,
|
||||
judge_model=self._judge_model,
|
||||
)
|
||||
except Exception as exc:
|
||||
LOGGER.error(
|
||||
"Grading failed for %s: %s", self._record.record_id, exc,
|
||||
)
|
||||
result = {"score": 0.0, "breakdown": {}, "notes": f"Grading error: {exc}"}
|
||||
|
||||
self._record.metadata["is_resolved"] = result["score"] >= 0.5
|
||||
self._record.metadata["reward"] = result["score"]
|
||||
self._record.metadata["pinchbench_score"] = result["score"]
|
||||
self._record.metadata["pinchbench_breakdown"] = result["breakdown"]
|
||||
self._record.metadata["pinchbench_notes"] = result.get("notes", "")
|
||||
|
||||
LOGGER.info(
|
||||
"PinchBench grading for %s: score=%.2f resolved=%s",
|
||||
self._record.record_id,
|
||||
result["score"],
|
||||
result["score"] >= 0.5,
|
||||
)
|
||||
|
||||
def __exit__(
|
||||
self,
|
||||
exc_type: Optional[Type[BaseException]],
|
||||
exc_val: Optional[BaseException],
|
||||
exc_tb: Optional[TracebackType],
|
||||
) -> None:
|
||||
# Restore original CWD
|
||||
if self._original_cwd:
|
||||
try:
|
||||
os.chdir(self._original_cwd)
|
||||
except OSError:
|
||||
pass
|
||||
self._original_cwd = None
|
||||
|
||||
# Only clean up workspaces we created ourselves (not AgenticRunner's)
|
||||
if self._owns_workspace and self._workspace and self._workspace.exists():
|
||||
keep = os.environ.get("PINCHBENCH_KEEP_WORKSPACES", "").strip()
|
||||
if keep and keep != "0":
|
||||
LOGGER.info("Keeping workspace: %s", self._workspace)
|
||||
else:
|
||||
shutil.rmtree(self._workspace, ignore_errors=True)
|
||||
self._workspace = None
|
||||
self._event_recorder = None
|
||||
|
||||
|
||||
__all__ = ["PinchBenchTaskEnv"]
|
||||
@@ -0,0 +1,503 @@
|
||||
"""PinchBench grading helpers and scorer.
|
||||
|
||||
Provides transcript translation (OpenJarvis events → PinchBench format),
|
||||
automated grading (exec of embedded Python), LLM judge grading, and
|
||||
hybrid combination. Used by PinchBenchTaskEnv.run_tests() and the
|
||||
standalone PinchBenchScorer.
|
||||
|
||||
Reference: https://github.com/pinchbench/skill
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from openjarvis.evals.core.event_recorder import EventType
|
||||
from openjarvis.evals.core.scorer import LLMJudgeScorer
|
||||
from openjarvis.evals.core.types import EvalRecord
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tool name mapping: OpenJarvis name → PinchBench-expected name
|
||||
# ---------------------------------------------------------------------------
|
||||
_TOOL_NAME_MAP: Dict[str, str] = {
|
||||
"file_read": "read_file",
|
||||
"file_write": "write_file",
|
||||
"image_generate": "generate_image",
|
||||
# web_search, calculator, shell_exec, etc. use the same names
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Transcript translation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def events_to_transcript(events: List[Any]) -> List[Dict[str, Any]]:
|
||||
"""Build PinchBench-format transcript from raw EventRecorder events.
|
||||
|
||||
Pairs TOOL_CALL_START/END events to extract tool name, arguments, and
|
||||
results. Called by run_tests() before QueryTrace exists.
|
||||
"""
|
||||
transcript: List[Dict[str, Any]] = []
|
||||
|
||||
for event in events:
|
||||
etype = event.event_type
|
||||
if isinstance(etype, str):
|
||||
# Normalize string event types to enum comparison
|
||||
pass
|
||||
if etype == EventType.TOOL_CALL_START or etype == EventType.TOOL_CALL_START.value:
|
||||
tool_name = event.metadata.get("tool", "unknown")
|
||||
mapped = _TOOL_NAME_MAP.get(tool_name, tool_name)
|
||||
arguments = event.metadata.get("arguments") or {}
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "toolCall", "name": mapped, "params": arguments}],
|
||||
},
|
||||
})
|
||||
elif etype == EventType.TOOL_CALL_END or etype == EventType.TOOL_CALL_END.value:
|
||||
result_text = str(event.metadata.get("result", ""))
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "toolResult",
|
||||
"content": [{"text": result_text}],
|
||||
},
|
||||
})
|
||||
|
||||
return transcript
|
||||
|
||||
|
||||
def _trace_to_transcript(trace: Any) -> List[Dict[str, Any]]:
|
||||
"""Convert QueryTrace to PinchBench transcript (for EvalRunner path)."""
|
||||
transcript: List[Dict[str, Any]] = []
|
||||
for turn in trace.turns:
|
||||
for tc in getattr(turn, "tool_calls", []):
|
||||
if tc is None:
|
||||
continue
|
||||
mapped = _TOOL_NAME_MAP.get(tc["name"], tc["name"])
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "toolCall", "name": mapped, "params": tc.get("arguments") or {}}],
|
||||
},
|
||||
})
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "toolResult",
|
||||
"content": [{"text": tc.get("result", "")}],
|
||||
},
|
||||
})
|
||||
|
||||
# Capture final assistant text response (for tasks graded on text output)
|
||||
response_text = getattr(trace, "response_text", "")
|
||||
if response_text:
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": response_text}],
|
||||
},
|
||||
})
|
||||
return transcript
|
||||
|
||||
|
||||
def _tool_results_to_transcript(
|
||||
tool_results: List[Dict[str, Any]],
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Build transcript from JarvisAgentBackend tool_results list."""
|
||||
transcript: List[Dict[str, Any]] = []
|
||||
for tr in tool_results:
|
||||
tool_name = tr.get("tool_name", "unknown")
|
||||
mapped = _TOOL_NAME_MAP.get(tool_name, tool_name)
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "toolCall", "name": mapped, "params": {}}],
|
||||
},
|
||||
})
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "toolResult",
|
||||
"content": [{"text": tr.get("content", "")}],
|
||||
},
|
||||
})
|
||||
return transcript
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Transcript summarization (replicates PinchBench _summarize_transcript)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _summarize_transcript(transcript: List[Dict[str, Any]]) -> str:
|
||||
"""Summarize transcript for LLM judge prompt.
|
||||
|
||||
Produces the same format as PinchBench's lib_grading._summarize_transcript():
|
||||
Tool: function_name({"arg1": value1})
|
||||
Result: [first 200 chars]
|
||||
User: [message text]
|
||||
"""
|
||||
parts: List[str] = []
|
||||
for event in transcript:
|
||||
if event.get("type") != "message":
|
||||
continue
|
||||
msg = event.get("message", {})
|
||||
role = msg.get("role")
|
||||
if role == "assistant":
|
||||
for item in msg.get("content", []):
|
||||
if item.get("type") == "toolCall":
|
||||
parts.append(
|
||||
f"Tool: {item.get('name')}({json.dumps(item.get('params', {}))})"
|
||||
)
|
||||
elif item.get("type") == "text":
|
||||
parts.append(f"Assistant: {item.get('text', '')}")
|
||||
elif role == "toolResult":
|
||||
content = msg.get("content", [])
|
||||
if content:
|
||||
preview = str(content[0])[:200]
|
||||
parts.append(f"Result: {preview}")
|
||||
elif role == "user":
|
||||
content = msg.get("content", [])
|
||||
if content:
|
||||
parts.append(f"User: {content[0]}")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Judge prompt and response parsing (replicates PinchBench exactly)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _build_judge_prompt(
|
||||
*,
|
||||
task_prompt: str,
|
||||
expected_behavior: str,
|
||||
rubric: str,
|
||||
transcript_summary: str,
|
||||
) -> str:
|
||||
"""Build the LLM judge prompt matching PinchBench's format."""
|
||||
return f"""You are a strict evaluator assessing an AI agent's performance on a task.
|
||||
Reserve a score of 1.0 for genuinely excellent performance.
|
||||
|
||||
## Task
|
||||
{task_prompt}
|
||||
|
||||
## Expected Behavior
|
||||
{expected_behavior}
|
||||
|
||||
## Agent Transcript
|
||||
{transcript_summary}
|
||||
|
||||
## Grading Rubric
|
||||
{rubric}
|
||||
|
||||
## Instructions
|
||||
Evaluate the agent's performance against each criterion in the rubric.
|
||||
Return your evaluation as JSON with this exact structure:
|
||||
```json
|
||||
{{
|
||||
"scores": {{"criterion_name": score, ...}},
|
||||
"total": overall_score_0_to_1,
|
||||
"notes": "brief justification"
|
||||
}}
|
||||
```
|
||||
Be a strict evaluator. Deduct points for unnecessary steps, verbose output,
|
||||
or inefficient tool usage."""
|
||||
|
||||
|
||||
def _parse_judge_response(raw: str) -> Dict[str, Any]:
|
||||
"""Parse LLM judge response with fallback chain.
|
||||
|
||||
Tries: JSON code block → balanced braces → regex score extraction.
|
||||
Matches PinchBench's lib_grading._parse_judge_response() logic.
|
||||
"""
|
||||
if not raw or not raw.strip():
|
||||
return {"scores": {}, "total": 0.0, "notes": "Empty judge response"}
|
||||
|
||||
# Try JSON code block
|
||||
code_block = re.search(r"```json\s*(.*?)\s*```", raw, re.DOTALL)
|
||||
if code_block:
|
||||
try:
|
||||
parsed = json.loads(code_block.group(1))
|
||||
if isinstance(parsed, dict):
|
||||
return _normalize_judge_response(parsed)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Try balanced braces extraction
|
||||
candidates: List[str] = []
|
||||
depth = 0
|
||||
current: List[str] = []
|
||||
for char in raw:
|
||||
if char == "{":
|
||||
if depth == 0:
|
||||
current = []
|
||||
depth += 1
|
||||
if depth > 0:
|
||||
current.append(char)
|
||||
if char == "}":
|
||||
depth -= 1
|
||||
if depth == 0 and current:
|
||||
candidates.append("".join(current))
|
||||
|
||||
for candidate in reversed(candidates):
|
||||
try:
|
||||
parsed = json.loads(candidate)
|
||||
if isinstance(parsed, dict) and "scores" in parsed:
|
||||
return _normalize_judge_response(parsed)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
for candidate in reversed(candidates):
|
||||
try:
|
||||
parsed = json.loads(candidate)
|
||||
if isinstance(parsed, dict):
|
||||
return _normalize_judge_response(parsed)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
# Regex fallback for prose scores
|
||||
score_match = re.search(
|
||||
r"(?:total|overall|final)\s*(?:score)?[:\s]*(0\.\d+|1\.0+)",
|
||||
raw,
|
||||
re.IGNORECASE,
|
||||
)
|
||||
if score_match:
|
||||
try:
|
||||
total = float(score_match.group(1))
|
||||
if 0.0 <= total <= 1.0:
|
||||
LOGGER.warning("Fell back to regex score extraction (total=%.2f)", total)
|
||||
return {"scores": {}, "total": total, "notes": "Score extracted from prose"}
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
LOGGER.warning("Failed to parse judge response")
|
||||
return {"scores": {}, "total": 0.0, "notes": "Failed to parse judge response"}
|
||||
|
||||
|
||||
def _normalize_judge_response(parsed: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Normalize various judge response formats to standard structure.
|
||||
|
||||
Matches PinchBench's lib_grading._normalize_judge_response().
|
||||
"""
|
||||
result: Dict[str, Any] = {"scores": {}, "total": 0.0, "notes": ""}
|
||||
|
||||
# Extract scores
|
||||
scores_data = parsed.get("scores", parsed.get("criteria_scores", {}))
|
||||
if isinstance(scores_data, dict):
|
||||
for key, value in scores_data.items():
|
||||
if isinstance(value, dict) and "score" in value:
|
||||
result["scores"][key] = float(value["score"])
|
||||
elif isinstance(value, (int, float)):
|
||||
result["scores"][key] = float(value)
|
||||
|
||||
# Extract total
|
||||
for key in ("total", "score", "overall_score"):
|
||||
if key in parsed and isinstance(parsed[key], (int, float)):
|
||||
result["total"] = float(parsed[key])
|
||||
break
|
||||
else:
|
||||
if result["scores"]:
|
||||
values = [v for v in result["scores"].values() if isinstance(v, (int, float))]
|
||||
if values:
|
||||
result["total"] = sum(values) / len(values)
|
||||
|
||||
# Normalize summed totals back to 0..1
|
||||
values = [v for v in result["scores"].values() if isinstance(v, (int, float))]
|
||||
if (
|
||||
values
|
||||
and result["total"] is not None
|
||||
and result["total"] > 1.0
|
||||
and all(0.0 <= float(v) <= 1.0 for v in values)
|
||||
):
|
||||
result["total"] = sum(values) / len(values)
|
||||
|
||||
# Extract notes
|
||||
for key in ("notes", "justification", "reasoning"):
|
||||
if key in parsed:
|
||||
result["notes"] = str(parsed[key])
|
||||
break
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Grading functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _grade_automated(
|
||||
record: EvalRecord,
|
||||
transcript: List[Dict[str, Any]],
|
||||
workspace_path: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Run the embedded Python grade() function from the task definition."""
|
||||
code = record.metadata.get("automated_checks")
|
||||
if not code:
|
||||
return {"score": 0.0, "breakdown": {}, "notes": "No automated checks defined"}
|
||||
|
||||
namespace: Dict[str, Any] = {}
|
||||
try:
|
||||
exec(code, namespace) # noqa: S102
|
||||
except Exception as exc:
|
||||
LOGGER.error("Failed to compile grading code for %s: %s", record.record_id, exc)
|
||||
return {"score": 0.0, "breakdown": {}, "notes": f"Grading code error: {exc}"}
|
||||
|
||||
grade_fn = namespace.get("grade")
|
||||
if not callable(grade_fn):
|
||||
return {"score": 0.0, "breakdown": {}, "notes": "No grade() function found in automated checks"}
|
||||
|
||||
try:
|
||||
scores = grade_fn(transcript, workspace_path)
|
||||
except Exception as exc:
|
||||
LOGGER.error("grade() failed for %s: %s", record.record_id, exc)
|
||||
return {"score": 0.0, "breakdown": {}, "notes": f"grade() error: {exc}"}
|
||||
|
||||
if not isinstance(scores, dict) or not scores:
|
||||
return {"score": 0.0, "breakdown": {}, "notes": "grade() returned empty or non-dict"}
|
||||
|
||||
mean_score = sum(scores.values()) / len(scores)
|
||||
return {"score": mean_score, "breakdown": scores, "notes": ""}
|
||||
|
||||
|
||||
def _grade_llm_judge(
|
||||
record: EvalRecord,
|
||||
transcript: List[Dict[str, Any]],
|
||||
workspace_path: str,
|
||||
judge_backend: Any,
|
||||
judge_model: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Grade using an LLM judge with the task's rubric."""
|
||||
rubric = record.metadata.get("llm_judge_rubric")
|
||||
if not rubric:
|
||||
return {"score": 0.0, "breakdown": {}, "notes": "No LLM judge rubric defined"}
|
||||
|
||||
if judge_backend is None:
|
||||
return {"score": 0.0, "breakdown": {}, "notes": "No judge backend configured"}
|
||||
|
||||
summary = _summarize_transcript(transcript)
|
||||
prompt = _build_judge_prompt(
|
||||
task_prompt=record.problem,
|
||||
expected_behavior=record.reference or "",
|
||||
rubric=rubric,
|
||||
transcript_summary=summary,
|
||||
)
|
||||
|
||||
try:
|
||||
raw = judge_backend.generate(prompt, model=judge_model, temperature=0.0, max_tokens=2048)
|
||||
except Exception as exc:
|
||||
LOGGER.error("LLM judge call failed for %s: %s", record.record_id, exc)
|
||||
return {"score": 0.0, "breakdown": {}, "notes": f"Judge error: {exc}"}
|
||||
|
||||
parsed = _parse_judge_response(raw)
|
||||
return {
|
||||
"score": parsed.get("total", 0.0),
|
||||
"breakdown": parsed.get("scores", {}),
|
||||
"notes": parsed.get("notes", ""),
|
||||
}
|
||||
|
||||
|
||||
def _grade_hybrid(
|
||||
record: EvalRecord,
|
||||
transcript: List[Dict[str, Any]],
|
||||
workspace_path: str,
|
||||
judge_backend: Any,
|
||||
judge_model: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Run both automated and LLM judge grading, combine with weights."""
|
||||
weights = record.metadata.get("grading_weights") or {"automated": 0.5, "llm_judge": 0.5}
|
||||
auto = _grade_automated(record, transcript, workspace_path)
|
||||
llm = _grade_llm_judge(record, transcript, workspace_path, judge_backend, judge_model)
|
||||
|
||||
auto_w = float(weights.get("automated", 0.5))
|
||||
llm_w = float(weights.get("llm_judge", 0.5))
|
||||
total_w = auto_w + llm_w
|
||||
|
||||
combined = (auto["score"] * auto_w + llm["score"] * llm_w) / total_w if total_w > 0 else 0.0
|
||||
breakdown = {
|
||||
**{f"automated.{k}": v for k, v in auto["breakdown"].items()},
|
||||
**{f"llm_judge.{k}": v for k, v in llm["breakdown"].items()},
|
||||
}
|
||||
notes = " | ".join(filter(None, [auto.get("notes", ""), llm.get("notes", "")]))
|
||||
return {"score": combined, "breakdown": breakdown, "notes": notes}
|
||||
|
||||
|
||||
def grade_pinchbench_task(
|
||||
*,
|
||||
record: EvalRecord,
|
||||
transcript: List[Dict[str, Any]],
|
||||
workspace_path: str,
|
||||
judge_backend: Any = None,
|
||||
judge_model: str = "anthropic/claude-opus-4-5",
|
||||
) -> Dict[str, Any]:
|
||||
"""Top-level grading entry point. Routes by grading_type.
|
||||
|
||||
Returns {"score": float, "breakdown": dict, "notes": str}.
|
||||
"""
|
||||
grading_type = record.metadata.get("grading_type", "automated")
|
||||
|
||||
if grading_type == "automated":
|
||||
return _grade_automated(record, transcript, workspace_path)
|
||||
elif grading_type == "llm_judge":
|
||||
return _grade_llm_judge(record, transcript, workspace_path, judge_backend, judge_model)
|
||||
elif grading_type == "hybrid":
|
||||
return _grade_hybrid(record, transcript, workspace_path, judge_backend, judge_model)
|
||||
else:
|
||||
return {"score": 0.0, "breakdown": {}, "notes": f"Unknown grading type: {grading_type}"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Standalone scorer (for EvalRunner non-agentic path)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class PinchBenchScorer(LLMJudgeScorer):
|
||||
"""PinchBench scorer for the non-agentic EvalRunner path."""
|
||||
|
||||
scorer_id = "pinchbench"
|
||||
|
||||
def score(
|
||||
self, record: EvalRecord, model_answer: str,
|
||||
) -> Tuple[Optional[bool], Dict[str, Any]]:
|
||||
trace = record.metadata.get("query_trace")
|
||||
if trace:
|
||||
transcript = _trace_to_transcript(trace)
|
||||
else:
|
||||
# No trace — build transcript from tool_results if available
|
||||
tool_results = record.metadata.get("tool_results", [])
|
||||
transcript = _tool_results_to_transcript(tool_results)
|
||||
|
||||
# Always append final model answer as assistant text message
|
||||
# so grading functions that check for text responses can find it
|
||||
if model_answer:
|
||||
transcript.append({
|
||||
"type": "message",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": model_answer}],
|
||||
},
|
||||
})
|
||||
|
||||
result = grade_pinchbench_task(
|
||||
record=record,
|
||||
transcript=transcript,
|
||||
workspace_path=record.metadata.get("workspace_path", ""),
|
||||
judge_backend=self._judge_backend,
|
||||
judge_model=self._judge_model,
|
||||
)
|
||||
is_correct = result["score"] >= 0.5
|
||||
return is_correct, {**result}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PinchBenchScorer",
|
||||
"events_to_transcript",
|
||||
"grade_pinchbench_task",
|
||||
"_tool_results_to_transcript",
|
||||
]
|
||||
@@ -45,11 +45,13 @@ BUILTIN_MODELS: List[ModelSpec] = [
|
||||
parameter_count_b=3.0,
|
||||
active_parameter_count_b=0.6,
|
||||
context_length=131072,
|
||||
supported_engines=("ollama", "vllm", "llamacpp", "sglang"),
|
||||
supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"),
|
||||
provider="alibaba",
|
||||
metadata={
|
||||
"architecture": "moe",
|
||||
"hf_repo": "Qwen/Qwen3.5-3B",
|
||||
"gguf_file": "qwen3.5-3b-q4_k_m.gguf",
|
||||
"mlx_repo": "mlx-community/Qwen3.5-3B-4bit",
|
||||
},
|
||||
),
|
||||
ModelSpec(
|
||||
@@ -58,11 +60,13 @@ BUILTIN_MODELS: List[ModelSpec] = [
|
||||
parameter_count_b=8.0,
|
||||
active_parameter_count_b=1.0,
|
||||
context_length=131072,
|
||||
supported_engines=("ollama", "vllm", "llamacpp", "sglang"),
|
||||
supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"),
|
||||
provider="alibaba",
|
||||
metadata={
|
||||
"architecture": "moe",
|
||||
"hf_repo": "Qwen/Qwen3.5-8B",
|
||||
"gguf_file": "qwen3.5-8b-q4_k_m.gguf",
|
||||
"mlx_repo": "mlx-community/Qwen3.5-8B-4bit",
|
||||
},
|
||||
),
|
||||
ModelSpec(
|
||||
@@ -71,11 +75,13 @@ BUILTIN_MODELS: List[ModelSpec] = [
|
||||
parameter_count_b=14.0,
|
||||
active_parameter_count_b=2.0,
|
||||
context_length=131072,
|
||||
supported_engines=("ollama", "vllm", "llamacpp", "sglang"),
|
||||
supported_engines=("ollama", "vllm", "llamacpp", "sglang", "mlx"),
|
||||
provider="alibaba",
|
||||
metadata={
|
||||
"architecture": "moe",
|
||||
"hf_repo": "Qwen/Qwen3.5-14B",
|
||||
"gguf_file": "qwen3.5-14b-q4_k_m.gguf",
|
||||
"mlx_repo": "mlx-community/Qwen3.5-14B-4bit",
|
||||
},
|
||||
),
|
||||
ModelSpec(
|
||||
@@ -223,11 +229,13 @@ BUILTIN_MODELS: List[ModelSpec] = [
|
||||
active_parameter_count_b=0.5,
|
||||
context_length=262144,
|
||||
min_vram_gb=3.0,
|
||||
supported_engines=("ollama", "vllm", "sglang", "llamacpp"),
|
||||
supported_engines=("ollama", "vllm", "sglang", "llamacpp", "mlx"),
|
||||
provider="alibaba",
|
||||
metadata={
|
||||
"architecture": "moe",
|
||||
"hf_repo": "Qwen/Qwen3.5-4B",
|
||||
"gguf_file": "qwen3.5-4b-q4_k_m.gguf",
|
||||
"mlx_repo": "mlx-community/Qwen3.5-4B-4bit",
|
||||
},
|
||||
),
|
||||
ModelSpec(
|
||||
@@ -707,6 +715,69 @@ BUILTIN_MODELS: List[ModelSpec] = [
|
||||
},
|
||||
),
|
||||
# -----------------------------------------------------------------------
|
||||
# Cloud models — MiniMax
|
||||
# -----------------------------------------------------------------------
|
||||
ModelSpec(
|
||||
model_id="MiniMax-M2.7",
|
||||
name="MiniMax M2.7",
|
||||
parameter_count_b=0.0,
|
||||
context_length=204800,
|
||||
supported_engines=("cloud",),
|
||||
provider="minimax",
|
||||
requires_api_key=True,
|
||||
metadata={
|
||||
"architecture": "proprietary",
|
||||
"pricing_input": 0.30,
|
||||
"pricing_output": 1.20,
|
||||
"url": "https://www.minimax.io",
|
||||
},
|
||||
),
|
||||
ModelSpec(
|
||||
model_id="MiniMax-M2.7-highspeed",
|
||||
name="MiniMax M2.7 Highspeed",
|
||||
parameter_count_b=0.0,
|
||||
context_length=204800,
|
||||
supported_engines=("cloud",),
|
||||
provider="minimax",
|
||||
requires_api_key=True,
|
||||
metadata={
|
||||
"architecture": "proprietary",
|
||||
"pricing_input": 0.60,
|
||||
"pricing_output": 2.40,
|
||||
"url": "https://www.minimax.io",
|
||||
},
|
||||
),
|
||||
ModelSpec(
|
||||
model_id="MiniMax-M2.5",
|
||||
name="MiniMax M2.5",
|
||||
parameter_count_b=0.0,
|
||||
context_length=204800,
|
||||
supported_engines=("cloud",),
|
||||
provider="minimax",
|
||||
requires_api_key=True,
|
||||
metadata={
|
||||
"architecture": "proprietary",
|
||||
"pricing_input": 0.30,
|
||||
"pricing_output": 1.20,
|
||||
"url": "https://www.minimax.io",
|
||||
},
|
||||
),
|
||||
ModelSpec(
|
||||
model_id="MiniMax-M2.5-highspeed",
|
||||
name="MiniMax M2.5 Highspeed",
|
||||
parameter_count_b=0.0,
|
||||
context_length=204800,
|
||||
supported_engines=("cloud",),
|
||||
provider="minimax",
|
||||
requires_api_key=True,
|
||||
metadata={
|
||||
"architecture": "proprietary",
|
||||
"pricing_input": 0.60,
|
||||
"pricing_output": 2.40,
|
||||
"url": "https://www.minimax.io",
|
||||
},
|
||||
),
|
||||
# -----------------------------------------------------------------------
|
||||
# Cloud models — Google
|
||||
# -----------------------------------------------------------------------
|
||||
ModelSpec(
|
||||
|
||||
@@ -13,6 +13,10 @@ from openjarvis.learning.learning_orchestrator import LearningOrchestrator
|
||||
from openjarvis.learning.optimize.llm_optimizer import LLMOptimizer
|
||||
from openjarvis.learning.optimize.optimizer import OptimizationEngine
|
||||
from openjarvis.learning.optimize.store import OptimizationStore
|
||||
from openjarvis.learning.routing.complexity import (
|
||||
ComplexityQueryAnalyzer,
|
||||
score_complexity,
|
||||
)
|
||||
from openjarvis.learning.routing.heuristic_reward import HeuristicRewardFunction
|
||||
from openjarvis.learning.routing.router import (
|
||||
HeuristicRouter,
|
||||
@@ -59,6 +63,7 @@ def ensure_registered() -> None:
|
||||
|
||||
__all__ = [
|
||||
"AgentConfigEvolver",
|
||||
"ComplexityQueryAnalyzer",
|
||||
"HAS_TORCH",
|
||||
"HeuristicRewardFunction",
|
||||
"HeuristicRouter",
|
||||
@@ -75,4 +80,5 @@ __all__ = [
|
||||
"TrainingDataMiner",
|
||||
"build_routing_context",
|
||||
"ensure_registered",
|
||||
"score_complexity",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,261 @@
|
||||
"""Query complexity analyzer — scores queries and suggests token budgets.
|
||||
|
||||
Produces a numeric complexity score (0.0–1.0) and a suggested
|
||||
``max_tokens`` budget based on query characteristics such as length,
|
||||
domain signals (code, math, multi-step reasoning), and whether the
|
||||
target model is a thinking model that needs extra headroom.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
from openjarvis.core.types import RoutingContext
|
||||
from openjarvis.learning._stubs import QueryAnalyzer
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Signal patterns
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_CODE_PATTERNS = re.compile(
|
||||
r"```|`[^`]+`|\bdef\s|\bclass\s|\bimport\s|\bfunction\s|\bconst\s|\bvar\s|\blet\s|"
|
||||
r"\bif\s*\(|->|=>|\{\s*\}|\bfor\s+\w+\s+in\s|#include|System\.out",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_MATH_PATTERNS = re.compile(
|
||||
r"\bsolve\b|\bintegral\b|\bequation\b|\bproof\b|\bderivative\b|\bmatrix\b|"
|
||||
r"\btheorem\b|\bcalculate\b|\bcompute\b|\bsigma\b|\bsum\b|\blimit\b|\bprobability\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_REASONING_PATTERNS = re.compile(
|
||||
r"\bexplain\b|\banalyze\b|\bcompare\b|\bwhy\b"
|
||||
r"|\bstep[- ]by[- ]step\b|\breason\b|\bthink\b"
|
||||
r"|\bpros\s+and\s+cons\b|\btrade-?\s*offs?\b|\bevaluate\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_MULTI_STEP_PATTERNS = re.compile(
|
||||
r"\bthen\b.*\bthen\b|\bfirst\b.*\bnext\b|\bstep\s*\d"
|
||||
r"|\b(?:and\s+also|additionally|furthermore)\b"
|
||||
r"|\b\d+\.\s",
|
||||
re.IGNORECASE | re.DOTALL,
|
||||
)
|
||||
_CREATIVE_PATTERNS = re.compile(
|
||||
r"\bwrite\b.*\b(?:essay|story|article|report|poem)\b"
|
||||
r"|\bgenerate\b.*\b(?:code|script|program)\b"
|
||||
r"|\bcreate\b|\bdesign\b|\bdraft\b|\bcompose\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Models known to use internal chain-of-thought that consumes output tokens.
|
||||
_THINKING_MODEL_PATTERNS = re.compile(
|
||||
r"qwen3\.5|qwq|deepseek-r1|o1-|o3-|o4-", re.IGNORECASE
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Token budget tiers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_TOKEN_TIERS = {
|
||||
"trivial": 1024, # greetings, yes/no, factoid lookups
|
||||
"simple": 2048, # short answers, definitions
|
||||
"moderate": 4096, # explanations, summaries
|
||||
"complex": 8192, # analysis, code generation, multi-step
|
||||
"very_complex": 16384, # long-form, multi-part reasoning
|
||||
}
|
||||
|
||||
# Thinking models need extra headroom for internal chain-of-thought.
|
||||
_THINKING_TOKEN_MULTIPLIER = 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Complexity scoring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComplexityResult:
|
||||
"""Output of the complexity analysis."""
|
||||
|
||||
score: float # 0.0–1.0
|
||||
tier: str
|
||||
suggested_max_tokens: int
|
||||
signals: dict
|
||||
|
||||
|
||||
def _count_questions(query: str) -> int:
|
||||
"""Count the number of question marks in the query."""
|
||||
return query.count("?")
|
||||
|
||||
|
||||
def _count_sub_tasks(query: str) -> int:
|
||||
"""Estimate the number of sub-tasks or enumerated items."""
|
||||
numbered = len(re.findall(r"^\s*\d+[.)]\s", query, re.MULTILINE))
|
||||
bulleted = len(re.findall(r"^\s*[-*]\s", query, re.MULTILINE))
|
||||
return numbered + bulleted
|
||||
|
||||
|
||||
def score_complexity(query: str) -> ComplexityResult:
|
||||
"""Score a query's complexity from 0.0 (trivial) to 1.0 (very complex).
|
||||
|
||||
The score is a weighted combination of independent signals, each
|
||||
contributing a fraction between 0 and 1. Weights reflect how much
|
||||
each signal correlates with the amount of reasoning and output
|
||||
tokens a model will need.
|
||||
"""
|
||||
signals: dict = {}
|
||||
score = 0.0
|
||||
|
||||
# --- Length signal (0–0.20) ---
|
||||
length = len(query)
|
||||
if length < 20:
|
||||
length_score = 0.0
|
||||
elif length < 100:
|
||||
length_score = 0.3
|
||||
elif length < 300:
|
||||
length_score = 0.6
|
||||
elif length < 800:
|
||||
length_score = 0.8
|
||||
else:
|
||||
length_score = 1.0
|
||||
signals["length"] = length_score
|
||||
score += 0.20 * length_score
|
||||
|
||||
# --- Domain signals (0–0.25) ---
|
||||
has_code = bool(_CODE_PATTERNS.search(query))
|
||||
has_math = bool(_MATH_PATTERNS.search(query))
|
||||
domain_score = 0.0
|
||||
if has_code:
|
||||
domain_score = max(domain_score, 0.7)
|
||||
if has_math:
|
||||
domain_score = max(domain_score, 0.8)
|
||||
if has_code and has_math:
|
||||
domain_score = 1.0
|
||||
signals["domain"] = domain_score
|
||||
signals["has_code"] = has_code
|
||||
signals["has_math"] = has_math
|
||||
score += 0.25 * domain_score
|
||||
|
||||
# --- Reasoning signal (0–0.25) ---
|
||||
has_reasoning = bool(_REASONING_PATTERNS.search(query))
|
||||
has_multi_step = bool(_MULTI_STEP_PATTERNS.search(query))
|
||||
reasoning_score = 0.0
|
||||
if has_reasoning:
|
||||
reasoning_score = 0.6
|
||||
if has_multi_step:
|
||||
reasoning_score = max(reasoning_score, 0.8)
|
||||
if has_reasoning and has_multi_step:
|
||||
reasoning_score = 1.0
|
||||
signals["reasoning"] = reasoning_score
|
||||
signals["has_reasoning"] = has_reasoning
|
||||
signals["has_multi_step"] = has_multi_step
|
||||
score += 0.25 * reasoning_score
|
||||
|
||||
# --- Question / sub-task count (0–0.15) ---
|
||||
n_questions = _count_questions(query)
|
||||
n_subtasks = _count_sub_tasks(query)
|
||||
multi_part = n_questions + n_subtasks
|
||||
if multi_part <= 1:
|
||||
multi_score = 0.0
|
||||
elif multi_part <= 3:
|
||||
multi_score = 0.5
|
||||
else:
|
||||
multi_score = 1.0
|
||||
signals["multi_part"] = multi_score
|
||||
signals["n_questions"] = n_questions
|
||||
signals["n_subtasks"] = n_subtasks
|
||||
score += 0.15 * multi_score
|
||||
|
||||
# --- Creative / generative signal (0–0.15) ---
|
||||
has_creative = bool(_CREATIVE_PATTERNS.search(query))
|
||||
creative_score = 0.7 if has_creative else 0.0
|
||||
signals["creative"] = creative_score
|
||||
signals["has_creative"] = has_creative
|
||||
score += 0.15 * creative_score
|
||||
|
||||
# Clamp
|
||||
score = max(0.0, min(1.0, score))
|
||||
|
||||
# Map to tier
|
||||
if score < 0.15:
|
||||
tier = "trivial"
|
||||
elif score < 0.30:
|
||||
tier = "simple"
|
||||
elif score < 0.55:
|
||||
tier = "moderate"
|
||||
elif score < 0.80:
|
||||
tier = "complex"
|
||||
else:
|
||||
tier = "very_complex"
|
||||
|
||||
suggested_max_tokens = _TOKEN_TIERS[tier]
|
||||
|
||||
return ComplexityResult(
|
||||
score=round(score, 3),
|
||||
tier=tier,
|
||||
suggested_max_tokens=suggested_max_tokens,
|
||||
signals=signals,
|
||||
)
|
||||
|
||||
|
||||
def is_thinking_model(model_name: str) -> bool:
|
||||
"""Return True if the model is known to use internal chain-of-thought."""
|
||||
return bool(_THINKING_MODEL_PATTERNS.search(model_name))
|
||||
|
||||
|
||||
def adjust_tokens_for_model(
|
||||
suggested: int, model_name: Optional[str] = None
|
||||
) -> int:
|
||||
"""Multiply the token budget when the target is a thinking model."""
|
||||
if model_name and is_thinking_model(model_name):
|
||||
return suggested * _THINKING_TOKEN_MULTIPLIER
|
||||
return suggested
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# QueryAnalyzer implementation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ComplexityQueryAnalyzer(QueryAnalyzer):
|
||||
"""Query analyzer that produces a complexity-aware RoutingContext.
|
||||
|
||||
Drop-in replacement for ``DefaultQueryAnalyzer`` — adds
|
||||
``complexity_score``, ``has_reasoning``, and ``suggested_max_tokens``
|
||||
to the returned ``RoutingContext``.
|
||||
"""
|
||||
|
||||
def analyze(
|
||||
self, query: str, **kwargs: object
|
||||
) -> RoutingContext:
|
||||
urgency = kwargs.get("urgency", 0.5)
|
||||
if not isinstance(urgency, (int, float)):
|
||||
urgency = 0.5
|
||||
model_name = kwargs.get("model")
|
||||
if not isinstance(model_name, str):
|
||||
model_name = None
|
||||
|
||||
result = score_complexity(query)
|
||||
tokens = adjust_tokens_for_model(result.suggested_max_tokens, model_name)
|
||||
|
||||
return RoutingContext(
|
||||
query=query,
|
||||
query_length=len(query),
|
||||
has_code=result.signals.get("has_code", False),
|
||||
has_math=result.signals.get("has_math", False),
|
||||
has_reasoning=result.signals.get("has_reasoning", False),
|
||||
urgency=float(urgency),
|
||||
complexity_score=result.score,
|
||||
suggested_max_tokens=tokens,
|
||||
metadata={"complexity_tier": result.tier, "signals": result.signals},
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"ComplexityQueryAnalyzer",
|
||||
"ComplexityResult",
|
||||
"adjust_tokens_for_model",
|
||||
"is_thinking_model",
|
||||
"score_complexity",
|
||||
]
|
||||
@@ -3,7 +3,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import List, Optional
|
||||
|
||||
from openjarvis.core.registry import ModelRegistry
|
||||
@@ -12,32 +11,33 @@ from openjarvis.learning._stubs import QueryAnalyzer, RouterPolicy
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Detection patterns
|
||||
_CODE_PATTERNS = re.compile(
|
||||
r"```|`[^`]+`|\bdef\s|\bclass\s|\bimport\s|\bfunction\s|\bconst\s|\bvar\s|\blet\s|"
|
||||
r"\bif\s*\(|->|=>|\{\s*\}|\bfor\s+\w+\s+in\s|#include|System\.out",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_MATH_PATTERNS = re.compile(
|
||||
r"\bsolve\b|\bintegral\b|\bequation\b|\bproof\b|\bderivative\b|\bmatrix\b|"
|
||||
r"\btheorem\b|\bcalculate\b|\bcompute\b|\bsigma\b|\bsum\b|\blimit\b|\bprobability\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_REASONING_KEYWORDS = re.compile(
|
||||
r"\bexplain\b|\banalyze\b|\bcompare\b|\bwhy\b"
|
||||
r"|\bstep[- ]by[- ]step\b|\breason\b|\bthink\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
def build_routing_context(
|
||||
query: str, *, urgency: float = 0.5, model: str | None = None
|
||||
) -> RoutingContext:
|
||||
"""Populate a ``RoutingContext`` from a raw query string.
|
||||
|
||||
When *model* is provided, the suggested token budget is adjusted
|
||||
for thinking models that need extra headroom.
|
||||
"""
|
||||
from openjarvis.learning.routing.complexity import (
|
||||
adjust_tokens_for_model,
|
||||
score_complexity,
|
||||
)
|
||||
|
||||
result = score_complexity(query)
|
||||
tokens = adjust_tokens_for_model(result.suggested_max_tokens, model)
|
||||
|
||||
def build_routing_context(query: str, *, urgency: float = 0.5) -> RoutingContext:
|
||||
"""Populate a ``RoutingContext`` from a raw query string."""
|
||||
return RoutingContext(
|
||||
query=query,
|
||||
query_length=len(query),
|
||||
has_code=bool(_CODE_PATTERNS.search(query)),
|
||||
has_math=bool(_MATH_PATTERNS.search(query)),
|
||||
has_code=result.signals.get("has_code", False),
|
||||
has_math=result.signals.get("has_math", False),
|
||||
has_reasoning=result.signals.get("has_reasoning", False),
|
||||
urgency=urgency,
|
||||
complexity_score=result.score,
|
||||
suggested_max_tokens=tokens,
|
||||
metadata={"complexity_tier": result.tier, "signals": result.signals},
|
||||
)
|
||||
|
||||
|
||||
@@ -94,8 +94,8 @@ class HeuristicRouter(RouterPolicy):
|
||||
Rules (applied in order):
|
||||
1. Code detected → prefer model with "code"/"coder" in name
|
||||
2. Math detected → prefer larger model
|
||||
3. Short query (<50 chars, no code/math) → prefer smaller/faster model
|
||||
4. Long/complex query (>500 chars OR reasoning keywords) → prefer larger model
|
||||
3. Low complexity (score < 0.20) → prefer smaller/faster model
|
||||
4. High complexity (score >= 0.55 OR reasoning keywords) → prefer larger model
|
||||
5. High urgency (>0.8) → override to smaller model
|
||||
6. Default fallback → default_model → fallback_model → first available
|
||||
"""
|
||||
@@ -139,12 +139,12 @@ class HeuristicRouter(RouterPolicy):
|
||||
if context.has_math:
|
||||
return _largest_model(available) or available[0]
|
||||
|
||||
# Rule 3: Short simple query → prefer smaller model
|
||||
if context.query_length < 50:
|
||||
# Rule 3: Low complexity → prefer smaller model
|
||||
if context.complexity_score < 0.20:
|
||||
return _smallest_model(available) or available[0]
|
||||
|
||||
# Rule 4: Long/complex query → prefer larger model
|
||||
if context.query_length > 500 or _REASONING_KEYWORDS.search(context.query):
|
||||
# Rule 4: High complexity or reasoning → prefer larger model
|
||||
if context.complexity_score >= 0.55 or context.has_reasoning:
|
||||
return _largest_model(available) or available[0]
|
||||
|
||||
# Rule 6: Default fallback
|
||||
@@ -162,7 +162,14 @@ class DefaultQueryAnalyzer(QueryAnalyzer):
|
||||
urgency = kwargs.get("urgency", 0.5)
|
||||
if not isinstance(urgency, (int, float)):
|
||||
urgency = 0.5
|
||||
return build_routing_context(query, urgency=urgency)
|
||||
model = kwargs.get("model")
|
||||
if not isinstance(model, str):
|
||||
model = None
|
||||
return build_routing_context(query, urgency=urgency, model=model)
|
||||
|
||||
|
||||
__all__ = ["DefaultQueryAnalyzer", "HeuristicRouter", "build_routing_context"]
|
||||
__all__ = [
|
||||
"DefaultQueryAnalyzer",
|
||||
"HeuristicRouter",
|
||||
"build_routing_context",
|
||||
]
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
from __future__ import annotations
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user