Metadata-Version: 2.4
Name: orchestra-mcp
Version: 0.1.3
Summary: MCP server testing and orchestration tool
Project-URL: Homepage, https://ahaan.world/orchestra
Project-URL: Documentation, https://ahaan.world/orchestra/docs
Project-URL: Changelog, https://ahaan.world/orchestra/docs/changelog
Author: Ahaan Chaudhuri
License: Proprietary
License-File: LICENSE
Keywords: automation,mcp,model-context-protocol,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: certifi>=2024.0.0
Requires-Dist: jsonpath-ng>=1.6.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.9.0
Provides-Extra: chuk
Requires-Dist: chuk-mcp-client-oauth; extra == 'chuk'
Provides-Extra: dev
Requires-Dist: chuk-mcp-client-oauth; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: truststore>=0.9; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: server
Requires-Dist: fastapi>=0.135.3; extra == 'server'
Requires-Dist: truststore>=0.9; extra == 'server'
Requires-Dist: uvicorn>=0.44.0; extra == 'server'
Description-Content-Type: text/markdown

# 🎵 Orchestra

**Automated testing for MCP servers — declarative, fast, CI/CD-ready.**

[![License: Proprietary](https://img.shields.io/badge/License-Proprietary-red.svg)](LICENSE)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/)

The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is becoming the standard way to connect AI models to external tools and data. Orchestra is a CLI tool that lets you write declarative YAML test suites for any MCP server — whether you're building one or integrating one into your stack.

**Write tests once. Run them anywhere. Catch regressions before they reach production.**

---

## Why Orchestra?

MCP servers are proliferating fast, but tooling for testing them is still early. Orchestra fills that gap:

- **No code required** — define tests in readable YAML
- **Works with any MCP server** — STDIO (local subprocess), HTTP (Streamable HTTP), and SSE transports
- **Assertions that understand MCP** — JSONPath queries + first-class `isError` detection
- **CI/CD native** — exit codes, quiet mode, JSON reports

---

## Install

Orchestra is closed-source. The source is proprietary (see [LICENSE](LICENSE)),
but the CLI is distributed as a wheel on PyPI so it can be installed in CI
without the desktop app:

```bash
uv tool install orchestra-mcp     # or: pipx install orchestra-mcp
orchestra --help
```

Installing it as a *tool* rather than into the project environment keeps
Orchestra's dependencies from colliding with the MCP server under test.

The desktop app bundles the same CLI; **Get Started → Install the orchestra
command-line tool** links it onto your PATH.

For development in this repo:

```bash
uv sync --extra server --extra dev
uv run orchestra --help
```

---

## Quickstart

**New to Orchestra? Use the interactive builder:**

```bash
orchestra new schemas/my_test.yaml
```

The wizard guides you through transport, auth, and your first test step. Time to first passing test: ~3 minutes.

**Or write YAML directly:**

```yaml
# schemas/memory_test.yaml
version: 1
name: "Memory Server Test"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]

steps:
  - id: create_entity
    type: tool_call
    tool: "create_entities"
    input:
      entities:
        - name: "TestUser"
          entityType: "person"
          observations: ["Loves testing"]
    save: "$"
    delay_ms: 1000

  - id: verify_created
    type: assert
    from: "create_entity"
    check:
      op: "jsonpath_exists"
      path: "$.content[0].text"
```

```bash
orchestra run schemas/memory_test.yaml
```

```
============================================================
 Running: Memory Server Test
 Server: stdio | Steps: 2
============================================================

📡 Connecting to MCP server...
 ✅ Connected to memory-server v0.6.3

▶ Step: create_entity (tool_call)
  ✅ Success

▶ Step: verify_created (assert)
  Check: jsonpath_exists at $.content[0].text
  ✅ Passed

═══════════════════════════════════════════════════════════
 Status: ✅ PASSED  |  Duration: 1247ms  |  2/2 steps
═══════════════════════════════════════════════════════════
📁 Report saved: reports/abc123.json
```

---

## Features

| Feature | Description |
|---|---|
| 🎯 **Interactive builder** | `orchestra new` walks through setup — no YAML knowledge needed |
| 🔍 **Server discovery** | `orchestra inspect` reveals all tools and their schemas |
| 🌐 **Multi-transport** | STDIO, Streamable HTTP, SSE |
| 🔒 **Auth support** | Bearer, API Key, Basic, **OAuth** (browser login + saved session) |
| ✅ **Rich assertions** | JSONPath queries, `is_error` / `no_error`, length checks |
| ⚡ **Rate limit handling** | Per-step `delay_ms` for API rate limits |
| 📊 **JSON reports** | Structured reports with run IDs, timestamps, step results |
| 🚀 **CI/CD ready** | Exit codes for pass/fail, `--quiet` mode for automation |
| 🔐 **Secrets-safe** | `{{env.VAR}}` interpolation — no hardcoded credentials |

---

## CLI Reference

```bash
orchestra new [output_file]          # Interactive collection builder
orchestra inspect <server.yaml>      # Discover tools and schemas
orchestra run <collection.yaml>      # Execute a test collection
orchestra validate <collection.yaml> # Validate schema without running
orchestra info                       # Version and environment info
orchestra auth login <mcp-url>       # OAuth sign-in (saves tokens for run/inspect)
orchestra auth logout <url|profile>  # Remove stored OAuth session
```

### Run options

```bash
orchestra run schemas/my_test.yaml --show-responses   # Print full JSON responses
orchestra run schemas/my_test.yaml --quiet            # Errors only
orchestra run schemas/my_test.yaml --output json      # Machine-readable output
orchestra run schemas/my_test.yaml --no-report        # Skip saving report file
```

---

## Transports

**Local (STDIO)** — runs the server as a subprocess:
```yaml
server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]
  env:
    API_KEY: "{{env.MY_API_KEY}}"
```

**Remote (HTTP)** — Streamable HTTP:
```yaml
server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"
```

**SSE:**
```yaml
server:
  transport: "sse"
  url: "http://localhost:3001"
```

---

## Authentication

**OAuth (MCP HTTP servers with authorization metadata)** — run once, then use `type: oauth` in YAML:

```bash
orchestra auth login https://mcp.example.com
# Optional: --profile myapp   # if set, add oauth_profile: "myapp" in YAML
# Optional: --scopes "scope1 scope2"
```

Tokens are stored under `~/.config/orchestra/oauth_sessions.json` (POSIX mode `600`). This file-backed store is used by default for both CLI and the desktop GUI so OAuth works consistently across processes. Set `ORCHESTRA_OAUTH_PROVIDER=chuk` only if you prefer chuk's platform credential store instead (requires the optional extra: `pip install "orchestra-mcp[chuk]"`). If **dynamic client registration** fails, set `ORCHESTRA_OAUTH_CLIENT_ID` (and `ORCHESTRA_OAUTH_CLIENT_SECRET` if required).

```yaml
server:
  transport: "http"
  url: "https://mcp.example.com"
  auth:
    type: "oauth"
    # oauth_profile: "myapp"  # only if you used --profile on login
```

```yaml
# Bearer token
auth:
  type: "bearer"
  token: "{{env.API_TOKEN}}"

# API key
auth:
  type: "api_key"
  key: "{{env.API_KEY}}"

# Basic auth
auth:
  type: "basic"
  username: "{{env.USERNAME}}"
  password: "{{env.PASSWORD}}"
```

---

## Assertions

| Operator | Description |
|---|---|
| `jsonpath_exists` | Path exists in response |
| `jsonpath_eq` | Value equals expected |
| `jsonpath_contains` | String or array contains value |
| `jsonpath_len_eq` | Array length equals N |
| `jsonpath_len_gte` | Array length ≥ N |
| `jsonpath_len_lte` | Array length ≤ N |
| `is_error` | MCP response has `isError: true` |
| `no_error` | MCP response has no error |

```yaml
- id: check_result
  type: assert
  from: "my_step"
  check:
    op: "jsonpath_contains"
    path: "$.content[0].text"
    value: "expected string"
```

**MCP error detection** — Orchestra distinguishes between JSON-RPC transport errors and tool-level errors (`isError: true` in the response body), so you can assert on both:

```yaml
- id: expect_failure
  type: assert
  from: "bad_call"
  check:
    op: "is_error"   # Passes if the tool itself returned an error
```

---

## Reports

Every run saves a structured JSON report:

```json
{
  "run_id": "abc123-def456",
  "collection_name": "My Test",
  "status": "passed",
  "duration_ms": 1234,
  "server": { "name": "memory-server", "version": "0.6.3" },
  "steps": [
    {
      "id": "create_entity",
      "type": "tool_call",
      "status": "success",
      "duration_ms": 150
    },
    {
      "id": "verify_created",
      "type": "assert",
      "status": "passed",
      "duration_ms": 5
    }
  ]
}
```

Reports are saved to `reports/` by default. Use `--report-dir` to customize.

---

## Examples

<details>
<summary>Test a remote HTTP server (DeepWiki)</summary>

```yaml
version: 1
name: "DeepWiki Test"

server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"

steps:
  - id: ask_about_react
    type: tool_call
    tool: "ask_question"
    input:
      repoName: "facebook/react"
      question: "What are React hooks?"
    save: "$"
    delay_ms: 2000

  - id: check_answer
    type: assert
    from: "ask_about_react"
    check:
      op: "jsonpath_contains"
      path: "$.content[0].text"
      value: "hook"
```
</details>

<details>
<summary>Test a local server with API key auth (Brave Search)</summary>

```yaml
version: 1
name: "Brave Search Test"

env:
  BRAVE_API_KEY: "your-api-key"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-brave-search"]
  env:
    BRAVE_API_KEY: "{{env.BRAVE_API_KEY}}"

steps:
  - id: search_python
    type: tool_call
    tool: "brave_web_search"
    input:
      query: "Python programming"
      count: 5
    save: "$"
    delay_ms: 2000

  - id: check_results
    type: assert
    from: "search_python"
    check:
      op: "jsonpath_len_gte"
      path: "$.content"
      value: 1
```
</details>

---

## CI/CD Integration

Orchestra exits with code `0` on pass and `1` on failure — drop it into any pipeline:

```yaml
# GitHub Actions
- name: Test MCP server
  run: orchestra run schemas/my_test.yaml --quiet
```

---

## Contributing

Contributions welcome. Clone the repo and run `pip install -e .` to develop locally.

---

## License

Proprietary — all rights reserved. See [LICENSE](LICENSE). Built on the [Model Context Protocol](https://modelcontextprotocol.io/) by Anthropic.