Metadata-Version: 2.4
Name: orchestra-mcp
Version: 0.1.7
Summary: MCP server testing and orchestration tool
Project-URL: Homepage, https://ahaan.world/orchestra
Project-URL: Documentation, https://ahaan.world/orchestra/docs
Project-URL: Changelog, https://ahaan.world/orchestra/docs/changelog
Author: Ahaan Chaudhuri
License: Proprietary
License-File: LICENSE
Keywords: automation,mcp,model-context-protocol,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: certifi>=2024.0.0
Requires-Dist: jsonpath-ng>=1.6.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.9.0
Provides-Extra: agent
Requires-Dist: mcp>=1.2; extra == 'agent'
Provides-Extra: chuk
Requires-Dist: chuk-mcp-client-oauth; extra == 'chuk'
Provides-Extra: dev
Requires-Dist: chuk-mcp-client-oauth; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: mcp>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: truststore>=0.9; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: server
Requires-Dist: fastapi>=0.135.3; extra == 'server'
Requires-Dist: truststore>=0.9; extra == 'server'
Requires-Dist: uvicorn>=0.44.0; extra == 'server'
Description-Content-Type: text/markdown

# 🎵 Orchestra

**Automated testing for MCP servers — declarative, fast, CI/CD-ready.**

[![License: Proprietary](https://img.shields.io/badge/License-Proprietary-red.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/)

The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is becoming the standard way to connect AI models to external tools and data. Orchestra is a CLI tool that lets you write declarative YAML test suites for any MCP server — whether you're building one or integrating one into your stack.

**Write tests once. Run them anywhere. Catch regressions before they reach production.**

---

## Why Orchestra?

MCP servers are proliferating fast, but tooling for testing them is still early. Orchestra fills that gap:

- **No code required** — define tests in readable YAML
- **Works with any MCP server** — STDIO (local subprocess), HTTP (Streamable HTTP), and SSE transports
- **Assertions that understand MCP** — JSONPath queries + first-class `isError` detection
- **CI/CD native** — exit codes, quiet mode, JSON reports

---

## Install

Orchestra is closed-source. The source is proprietary (see [LICENSE](LICENSE)),
but the CLI is distributed as a wheel on PyPI so it can be installed in CI
without the desktop app:

```bash
uv tool install orchestra-mcp     # or: pipx install orchestra-mcp
orchestra --help
```

Installing it as a *tool* rather than into the project environment keeps
Orchestra's dependencies from colliding with the MCP server under test.

The desktop app bundles the same CLI; **Get Started → Install the orchestra
command-line tool** links it onto your PATH.

For development in this repo:

```bash
uv sync --extra server --extra dev
uv run orchestra --help
```

---

## Quickstart

**New to Orchestra? Use the interactive builder:**

```bash
orchestra new schemas/my_test.yaml
```

The wizard guides you through transport, auth, and your first test step. Time to first passing test: ~3 minutes.

**Or write YAML directly:**

```yaml
# schemas/memory_test.yaml
version: 1
name: "Memory Server Test"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]

steps:
  - id: create_entity
    type: tool_call
    tool: "create_entities"
    input:
      entities:
        - name: "TestUser"
          entityType: "person"
          observations: ["Loves testing"]
    delay_ms: 1000

  - id: verify_created
    type: assert
    from: "create_entity"
    check:
      op: "jsonpath_exists"
      path: "$.content[0].text"
```

```bash
orchestra run schemas/memory_test.yaml
```

```
============================================================
 Running: Memory Server Test
 Server: stdio | Steps: 2
============================================================

📡 Connecting to MCP server...
 ✅ Connected to memory-server v0.6.3

▶ Step: create_entity (tool_call)
  ✅ Success

▶ Step: verify_created (assert)
  Check: jsonpath_exists at $.content[0].text
  ✅ Passed

═══════════════════════════════════════════════════════════
 Status: ✅ PASSED  |  Duration: 1247ms  |  2/2 steps
═══════════════════════════════════════════════════════════
📁 Report saved: reports/abc123.json
```

---

## Features

| Feature | Description |
|---|---|
| 🎯 **Interactive builder** | `orchestra new` walks through setup — no YAML knowledge needed |
| 🔍 **Server discovery** | `orchestra inspect` reveals all tools and their schemas |
| 🌐 **Multi-transport** | STDIO, Streamable HTTP, SSE |
| 🔒 **Auth support** | Bearer, API Key, Basic, **OAuth** (browser login + saved session) |
| ✅ **Rich assertions** | JSONPath queries, `is_error` / `no_error`, length checks |
| 📸 **Contract snapshots** | Record the tool surface, fail on drift with breaking/additive/cosmetic severities |
| 👀 **Watch mode** | `orchestra watch` rebuilds, re-inspects, and diffs the surface on every save |
| 🤖 **MCP server mode** | `orchestra mcp` exposes Orchestra to coding agents — including "did my edit break the contract?" |
| 🔎 **Zero-config connect** | Import from Claude Desktop / Cursor / Windsurf, paste a README config, or detect from a project folder |
| ⚡ **Rate limit handling** | Per-step `delay_ms` and `timeout_ms`, retries for transient transport failures |
| 📊 **JSON + JUnit reports** | Run IDs, per-step results, and a `summary.json` across a whole suite |
| 🚀 **CI/CD ready** | Exit codes for pass/fail, `--quiet` mode for automation |
| 🔐 **Secrets-safe** | `{{env.VAR}}` interpolation — no hardcoded credentials |

---

## CLI Reference

```bash
orchestra inspect <name|url|yaml>    # Discover tools, schemas, resources, prompts
orchestra call <name|url> <tool>     # Call one tool and print the result
orchestra run <collection|dir>       # Execute a collection, or every one in a directory
orchestra watch <name> --path src    # Rebuild + re-inspect on change, diff the surface
orchestra validate <collection.yaml> # Validate schema without connecting
orchestra new [output_file]          # Interactive collection builder
orchestra mcp                        # Serve Orchestra itself over MCP, for AI agents
orchestra playground                 # Run the built-in demo MCP server
orchestra info                       # Version and environment info

orchestra servers list                       # Named connections (global + workspace)
orchestra servers add <name> --url <url>     # Save a connection
orchestra servers add <name> --command <cmd> --args "..."
orchestra servers detect <dir>               # Find a running HTTP server from its project folder
orchestra servers import                     # Import from Claude Desktop / Cursor / Windsurf
orchestra servers remove <name>

orchestra auth login <mcp-url>       # OAuth sign-in (saves tokens for run/inspect)
orchestra auth logout <url|profile>  # Remove stored OAuth session
```

Arguments to `call` follow the httpie convention: `-i key=value` is a string,
`-i key:=value` is raw JSON, so `-i count:=3` sends a number and `-i count=3`
sends `"3"`.

### Run options

```bash
orchestra run tests/mcp --output junit --report-dir reports  # JUnit XML + summary.json for CI
orchestra run tests/mcp --environment staging               # Named env from orchestra.env.yaml
orchestra run tests/mcp --env API_KEY=$SECRET               # Override a variable (repeatable)
orchestra run tests/mcp --fail-on breaking                  # Only fail on breaking contract changes
orchestra run tests/mcp --update-baselines                  # Accept snapshot changes, jest-style
orchestra run tests/mcp --show-responses                    # Print full JSON responses
orchestra run tests/mcp --quiet                             # Errors only
orchestra run tests/mcp --no-report                         # Skip saving report files
```

---

## Transports

**Local (STDIO)** — runs the server as a subprocess:
```yaml
server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-memory"]
  env:
    API_KEY: "{{env.MY_API_KEY}}"
```

**Remote (HTTP)** — Streamable HTTP:
```yaml
server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"
```

**SSE:**
```yaml
server:
  transport: "sse"
  url: "http://localhost:3001"
```

---

## Authentication

**OAuth (MCP HTTP servers with authorization metadata)** — run once, then use `type: oauth` in YAML:

```bash
orchestra auth login https://mcp.example.com
# Optional: --profile myapp   # if set, add oauth_profile: "myapp" in YAML
# Optional: --scopes "scope1 scope2"
```

Tokens are stored under `~/.config/orchestra/oauth_sessions.json` (POSIX mode `600`). This file-backed store is used by default for both CLI and the desktop GUI so OAuth works consistently across processes. Set `ORCHESTRA_OAUTH_PROVIDER=chuk` only if you prefer chuk's platform credential store instead (requires the optional extra: `pip install "orchestra-mcp[chuk]"`). If **dynamic client registration** fails, set `ORCHESTRA_OAUTH_CLIENT_ID` (and `ORCHESTRA_OAUTH_CLIENT_SECRET` if required).

```yaml
server:
  transport: "http"
  url: "https://mcp.example.com"
  auth:
    type: "oauth"
    # oauth_profile: "myapp"  # only if you used --profile on login
```

```yaml
# Bearer token
auth:
  type: "bearer"
  token: "{{env.API_TOKEN}}"

# API key
auth:
  type: "api_key"
  key: "{{env.API_KEY}}"

# Basic auth
auth:
  type: "basic"
  username: "{{env.USERNAME}}"
  password: "{{env.PASSWORD}}"
```

---

## Assertions

| Operator | Description |
|---|---|
| `jsonpath_exists` | Path exists in response |
| `jsonpath_eq` | Value equals expected |
| `jsonpath_gt` | **Value** at the path > N |
| `jsonpath_gte` | **Value** at the path ≥ N |
| `jsonpath_lt` | **Value** at the path < N |
| `jsonpath_lte` | **Value** at the path ≤ N |
| `jsonpath_contains` | String or array contains value |
| `jsonpath_matches` | String at the path matches a regex |
| `jsonpath_len_eq` | **Length** of the array/string equals N |
| `jsonpath_len_gte` | **Length** of the array/string ≥ N |
| `jsonpath_len_lte` | **Length** of the array/string ≤ N |
| `is_error` | MCP response has `isError: true` |
| `no_error` | MCP response has no error |

```yaml
- id: check_result
  type: assert
  from: "my_step"
  check:
    op: "jsonpath_contains"
    path: "$.content[0].text"
    value: "expected string"
```

**`jsonpath_gt` vs `jsonpath_len_gte`** — the comparison operators compare the
*value* found at the path; the `len_` operators compare the *length* of the
array or string found there. `$.results` with three items satisfies
`jsonpath_len_gte: 3`; `$.total` holding `42` satisfies `jsonpath_gt: 41`.
Pointing a comparison at a non-number (a string, an object, `null`) fails the
step with "Cannot compare … numerically" rather than guessing, and booleans are
never treated as `1`/`0`.

```yaml
- id: enough_results
  type: assert
  from: "search"
  check:
    op: "jsonpath_gt"
    path: "$.structuredContent.total"
    value: 0
```

**Pattern matching** — `jsonpath_matches` uses `re.search`, so the pattern is
found *anywhere* in the string, consistent with `jsonpath_contains` being a
substring check. Anchor it with `^...$` when you want a full match. An invalid
regex is rejected by `orchestra validate` before the run starts.

```yaml
- id: looks_like_an_id
  type: assert
  from: "create"
  check:
    op: "jsonpath_matches"
    path: "$.structuredContent.id"
    value: "^user_[0-9a-f]{8}$"
```

**Negation** — add `not: true` to any check to invert it, instead of a separate
`jsonpath_not_*` operator for every operator:

```yaml
- id: no_stack_traces
  type: assert
  from: "search"
  check:
    op: "jsonpath_matches"
    path: "$.content[0].text"
    value: "Traceback"
    not: true
```

`not: true` flips pass and fail. It does *not* rescue a check that could not be
evaluated at all — an unparseable JSONPath or a string under a numeric
comparison stays an error, since reporting "not greater than: pass" for a value
that was never a number would hide the mistake behind a green check.

**MCP error detection** — Orchestra distinguishes between JSON-RPC transport errors and tool-level errors (`isError: true` in the response body), so you can assert on both:

```yaml
- id: expect_failure
  type: assert
  from: "bad_call"
  check:
    op: "is_error"   # Passes if the tool itself returned an error
```

---

## Reports

Every run saves a structured JSON report:

```json
{
  "run_id": "abc123-def456",
  "collection_name": "My Test",
  "status": "passed",
  "duration_ms": 1234,
  "server": { "name": "memory-server", "version": "0.6.3" },
  "steps": [
    {
      "id": "create_entity",
      "type": "tool_call",
      "status": "success",
      "duration_ms": 150
    },
    {
      "id": "verify_created",
      "type": "assert",
      "status": "passed",
      "duration_ms": 5
    }
  ]
}
```

Reports are saved to `reports/` by default. Use `--report-dir` to customize.

Running a **directory** of collections also writes `reports/summary.json` — one
entry per collection with its source path, status, step counts, and the
severity counts of any contract-snapshot changes. A single exit code cannot say
which of N servers moved; that file can.

---

## Examples

<details>
<summary>Test a remote HTTP server (DeepWiki)</summary>

```yaml
version: 1
name: "DeepWiki Test"

server:
  transport: "http"
  url: "https://mcp.deepwiki.com/mcp"

steps:
  - id: ask_about_react
    type: tool_call
    tool: "ask_question"
    input:
      repoName: "facebook/react"
      question: "What are React hooks?"
    delay_ms: 2000

  - id: check_answer
    type: assert
    from: "ask_about_react"
    check:
      op: "jsonpath_contains"
      path: "$.content[0].text"
      value: "hook"
```
</details>

<details>
<summary>Test a local server with API key auth (Brave Search)</summary>

```yaml
version: 1
name: "Brave Search Test"

env:
  BRAVE_API_KEY: "your-api-key"

server:
  transport: "stdio"
  command: "npx"
  args: ["-y", "@modelcontextprotocol/server-brave-search"]
  env:
    BRAVE_API_KEY: "{{env.BRAVE_API_KEY}}"

steps:
  - id: search_python
    type: tool_call
    tool: "brave_web_search"
    input:
      query: "Python programming"
      count: 5
    delay_ms: 2000

  - id: check_results
    type: assert
    from: "search_python"
    check:
      op: "jsonpath_len_gte"
      path: "$.content"
      value: 1
```
</details>

---

## CI/CD Integration

Orchestra exits with code `0` on pass and `1` on failure — drop it into any pipeline:

```yaml
# GitHub Actions
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5

# Install the server under test FROM THIS BRANCH, not from a registry.
- run: uv sync --frozen          # or: npm ci && npm run build

# uvx keeps Orchestra's dependencies isolated from the server's.
- run: uvx --from orchestra-mcp orchestra run tests/mcp --output junit --report-dir reports
```

> **The npx trap.** Every MCP server README documents setup as
> `npx -y @scope/server`, because that is the Claude Desktop config. Do not
> copy that into CI: it installs the *published* package, so your gate tests
> the last release instead of the pull request — a green check that proves
> nothing. Build from source and run the local artifact.

Prefer stdio for the server under test: a subprocess over pipes binds no port,
so there is nothing to background and no port to wait on.

---

## Support

Orchestra is proprietary software (see [LICENSE](LICENSE)); the source is not
public. Bug reports, feature requests, and questions are welcome at
[admin@ahaan.world](mailto:admin@ahaan.world).

Docs: [ahaan.world/orchestra/docs](https://ahaan.world/orchestra/docs)

---

## License

Proprietary — all rights reserved. See [LICENSE](LICENSE). Built on the [Model Context Protocol](https://modelcontextprotocol.io/) by Anthropic.