Metadata-Version: 2.4
Name: okareo-mcp
Version: 0.0.31
Summary: Okareo MCP server for AI model evaluation
Author-email: Okareo <support@okareo.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/okareo-ai/okareo-mcp-beta
Project-URL: Documentation, https://docs.okareo.com/docs/mcp/introduction
Project-URL: Repository, https://github.com/okareo-ai/okareo-mcp-beta
Project-URL: Issues, https://github.com/okareo-ai/okareo-mcp-beta/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
Requires-Dist: mcp[cli]
Requires-Dist: okareo>=0.0.128
Requires-Dist: httpx
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"

# Okareo MCP Server

The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.

For detailed documentation, see the [Okareo MCP docs](https://docs.okareo.com/docs/mcp/introduction).

## Prerequisites

- An Okareo API key from [app.okareo.com](https://app.okareo.com)
- A copilot that supports MCP servers (Claude Code, Cursor, or VS Code)
- Python 3.10–3.12 (for local install modes)

---

### Step 1: Set Your API Key

```bash
export OKAREO_API_KEY="your-api-key"
```

Add this to your `~/.zshrc` or `~/.bash_profile` for persistence.

### Step 2: Configure Your Copilot

#### Claude Code

Add to `.mcp.json`:

```json
{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"],
      "env": {
        "OKAREO_API_KEY": "${OKAREO_API_KEY}"
      }
    }
  }
}
```

No pre-install needed — `uvx` handles it automatically.

#### Cursor

Add to `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"]
    }
  }
}
```

Cursor inherits `OKAREO_API_KEY` from your shell environment.

#### Alternative: pip install

If you don't have `uv` installed:

```bash
pip install okareo-mcp
```

Then use `"command": "okareo-mcp"` instead of `"command": "uvx"` with `"args": ["okareo-mcp"]`.

---

## Configuration Reference

| Variable | Default | Description |
|----------|---------|-------------|
| `OKAREO_API_KEY` | *(required)* | Your Okareo API key |
| `OKAREO_BASE_URL` | `https://api.okareo.com` | Override for on-prem Okareo backend |
| `TRANSPORT` | `stdio` | Transport: `stdio` (local) or `sse` (Docker) |
| `PORT` | `8000` | Port for SSE transport |

---

## Available Tools

### Scenarios

| Tool | Description |
|------|-------------|
| `save_scenario` | Save a named scenario from rows of input/result data (idempotent) |
| `list_scenarios` | List all scenarios in the project with names, IDs, and row counts |
| `get_scenario` | Retrieve a scenario's metadata and all data rows by name or ID |
| `create_scenario_version` | Create a new version of an existing scenario with updated data |
| `preview_delete_scenario` | Preview what will be deleted before removing a scenario |
| `delete_scenario` | Permanently delete a scenario and all related test data |

### Generation Models

| Tool | Description |
|------|-------------|
| `list_available_llms` | Browse available LLMs from the Okareo registry |
| `register_generation_model` | Register a generation model for testing by selecting an LLM from the registry |
| `list_generation_models` | List all registered generation models in the project |
| `get_generation_model` | Read detailed information about a registered generation model |
| `update_generation_model` | Change the LLM a registered generation model points to |
| `delete_generation_model` | Remove a registered generation model and all its related test data |

### Tests & Evaluations

| Tool | Description |
|------|-------------|
| `list_checks` | List available quality checks (built-in and custom) for evaluating model outputs |
| `run_test` | Run a quality test that evaluates a model against a scenario using specified checks |
| `list_test_runs` | List past test runs with optional filters (model, scenario, simulation-only) |
| `get_test_run_results` | Load detailed per-row results of a test run or simulation by ID or name |

### Simulations (Multi-Turn)

| Tool | Description |
|------|-------------|
| `create_or_update_target` | Create or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio) |
| `get_target` | Retrieve a Target's configuration by name (all types) |
| `list_targets` | List all simulation targets (voice and custom_endpoint) in the project |
| `delete_target` | Remove a simulation target and all its related test data |
| `create_or_update_driver` | Define a simulated user persona that will interact with your target |
| `get_driver` | Retrieve a Driver's full configuration including the persona prompt |
| `list_drivers` | List all Driver personas in the project |
| `run_simulation` | Run a multi-turn conversation evaluation (or rerun a previous one with overrides) |
| `list_simulations` | List past simulation runs with optional filters (target, scenario, limit) |

### Documentation & Templates

| Tool | Description |
|------|-------------|
| `get_docs` | Query the Okareo documentation system for conceptual or user-legible explanations |
| `get_templates` | Retrieve prompt templates for common Okareo patterns (works offline) |

---

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| `okareo-mcp: command not found` | Not installed or not in PATH | Run `pip install -e .` (dev) or use `uvx okareo-mcp` (user) |
| Server exits with API key error | `OKAREO_API_KEY` not set | Export it: `export OKAREO_API_KEY="..."` |
| `pip install` fails on Python 3.13+ | Okareo SDK requires Python <3.13 | Use Python 3.10–3.12 |
| Copilot can't connect (Docker) | Wrong URL | Ensure URL ends with `/sse` and port matches |
| Cursor doesn't pick up API key | Cursor launched from Dock, not terminal | Launch Cursor from terminal: `cursor .` |
