Metadata-Version: 2.4
Name: prompt-to-mcp
Version: 0.1.0
Summary: Describe the MCP server you want, point at an OpenAPI spec, and get a small curated FastMCP server (a few workflow-level tools, not hundreds of endpoint clones) plus a built-in eval harness that measures it.
Author: Oussama Knani
License: MIT
License-File: LICENSE
Keywords: agents,codegen,eval,fastmcp,llm,mcp,model-context-protocol,openapi,tools
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.12
Requires-Dist: anthropic>=0.117.0
Requires-Dist: fastmcp
Requires-Dist: httpx
Requires-Dist: jinja2>=3.1.6
Requires-Dist: jsonref
Requires-Dist: openai>=2.46.0
Requires-Dist: pydantic
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: pyyaml
Requires-Dist: rich
Requires-Dist: typer
Description-Content-Type: text/markdown

# prompt-to-mcp

Describe the MCP server you want, point at an OpenAPI spec, and get a small curated
FastMCP server plus an eval harness that measures it.

```
Stripe API (2026-06-24)     587 operations   →   6 described tools
Tool-definition cost         ~462,960 tokens  →   ~956 tokens
```

The same spec can become a different server depending on what you ask for. You choose
the shape; the model designs the tools; the output is editable code, not a black box.

## Hero example

One command turns the 587-operation Stripe spec into a server scoped to a single job:

```bash
prompt-to-mcp plan specs/spec3.json \
    "a server for support agents to manage customers, invoices and refunds" \
    -o stripe-support.toolplan.yaml --max-tools 8 \
    --provider ollama --model gpt-oss:120b-cloud
```

The description drove the curation. The model kept the customer, invoice, and refund
surface and left everything else (subscriptions, products, prices, balance, connect,
webhooks, reporting) out:

| Tool | Read-only | What it does |
|---|---|---|
| `customer_lookup` | yes | Find or retrieve customer information |
| `customer_manage` | no | Create, update, or delete a customer |
| `invoice_lookup` | yes | Find or retrieve invoice information |
| `invoice_manage` | no | Create, update, finalize, pay, void, or send an invoice |
| `refund_lookup` | yes | List or retrieve refunds |
| `refund_manage` | no | Create, update, or cancel a refund |

Six tools, about 956 tokens of tool definitions, against 587 operations and roughly
462,960 tokens for the naive one-tool-per-endpoint conversion. Full artifacts:
[`stripe-support.toolplan.yaml`](stripe-support.toolplan.yaml) and
[`generated/stripe-support/`](generated/stripe-support/).

For comparison, curating the *whole* Stripe API with `--auto` (no description) yields a
broader 11-tool general server, [`stripe.toolplan.yaml`](stripe.toolplan.yaml). Same
spec, two servers: one shaped to a task, one shaped to the API.

## Why context is the headline

Every request an agent makes carries the full tool list. Drawn to scale against a
32k-token model window, for the GitHub REST API (1,206 operations):

```
naive      1,206 tools │ ████████████████████████████████████████  414,263 tok   12.6× over ✗
32k window   (ceiling) │ ███                                         32,768 tok
distilled      9 tools │ ▏                                            1,758 tok   18.6× under ✓
```

The naive tool list alone is about 13 times larger than the entire window. It cannot
fit before the agent reads a single word of the task. Providers differ in how they
fail, and the harness records both:

| Provider behavior on an oversized tool list | What the harness does |
|---|---|
| Strict (Anthropic, OpenAI): reject the request | records `provider_error` with the provider's message |
| Lenient (Ollama): silently truncate the list and run anyway | a pre-flight `--model-context` guard refuses the run, so a mangled tool list is never passed off as a result |

## The `--auto` mode and the benchmarks

`--auto` curates the whole API without a description, which is the mode every benchmark
below used. It is a fair, description-free comparison: naive one-tool-per-endpoint
against an automatically curated server, same agent, same scenarios.

| API | Operations | Naive tool-def cost | Distilled | Ratio | Eval (qwen2.5:7b, live) |
|---|---|---|---|---|---|
| GitHub | 1,206 | ~414,263 tok | 9 tools / ~1,758 tok | 236× | 13/15 (87%) |
| Stripe | 587 | ~462,960 tok | 11 tools / ~1,793 tok | 258× | 12/15 (80%) |

The agent is `qwen2.5:7b`, local via Ollama. The naive side was refused by the context
guard in every case (its tool list does not fit a 32k model). The distilled side ran
live against api.github.com and api.stripe.com (test mode). Full artifacts:
[`eval_runs/headline/`](eval_runs/headline/) and [`eval_runs/stripe/`](eval_runs/stripe/).

For GitHub, the per-scenario picture:

```mermaid
xychart-beta
    title "Distilled GitHub server: trials passed per scenario (3 trials each)"
    x-axis ["repo-overview", "repo-metadata", "list-branches", "repo-language", "search-repo"]
    y-axis "trials passed" 0 --> 3
    bar [3, 3, 3, 3, 1]
```

The weak scenario is also the expensive one. `search-repo` passed 1 of 3 trials and
consumed about 2.7 times the input tokens of any other task. A small model that cannot
decide between similar options flails, and flailing costs tokens:

```mermaid
xychart-beta
    title "Mean input tokens per task, by scenario"
    x-axis ["repo-overview", "repo-metadata", "list-branches", "repo-language", "search-repo"]
    y-axis "input tokens" 0 --> 10000
    bar [3582, 4062, 3448, 3401, 9569]
```

Stripe tells the same story from the other side: the four read scenarios passed 3 of 3
each; the one write scenario (create a customer) failed all three trials. The 7B model
could not reliably drive a mutating call to completion. That is a model limit, not a
curation limit, and the report records it plainly.

### What the numbers show, and what they don't

The context result is decisive. 414k or 462k tokens of tool definitions do not fit a
32k model, no matter how capable the agent is. On small-context models, curation is
what makes the API usable at all.

The 87% and 80% show the pipeline works end to end: an LLM-designed tool surface,
compiled to code, drove live API calls well enough for a 7B model to answer correctly.
It is not yet proof that curation beats a naive server that *fits*. That comparison (cap
the naive baseline with `--naive-max-tools` on a large-context model, add a no-tools
control) is supported by the harness and is the next experiment worth running. See
[Methodology and limitations](#methodology-and-limitations).

## Coverage tradeoff

Curation drops long-tail endpoints on purpose. The hero server above cannot touch
subscriptions or payouts, because the description did not ask for them. That is the
point: with a description, you decide what is covered, and the tool surface stays small
enough to fit and cheap enough to reason over.

Nothing is locked in. The ToolPlan is YAML you can read and edit, and `build` is free
and deterministic. A missing capability is one YAML edit and a rebuild away: add the
endpoint to a tool (or add a tool), run `build`, and the server has it. The model
proposes the starting point; you own the final shape.

## How it works

Four commands. Two of them cost tokens.

```mermaid
flowchart LR
    S["OpenAPI spec<br/>(587 ops)"] --> I["inspect<br/><i>damage report</i>"]
    I --> P["plan<br/><i>1 LLM pass</i>"]
    P --> Y[["ToolPlan YAML<br/>editable, diff-friendly"]]
    Y --> B["build<br/><i>deterministic codegen</i>"]
    B --> M["FastMCP server<br/>(6 tools)"]
    M --> E["eval<br/><i>naive vs distilled</i>"]
    style Y fill:#fde68a,stroke:#b45309,color:#000
    style P fill:#bfdbfe,stroke:#1e40af,color:#000
```

| Command | Cost | What it does |
|---|---|---|
| `inspect` | free | Parse the spec, print the naive-conversion damage report. |
| `plan` | 1 LLM pass | Curate the spec (for your description, or `--auto`) into an editable ToolPlan YAML. |
| `build` | free | Compile the ToolPlan into a runnable FastMCP package. |
| `eval` | LLM per trial | Run scenarios against both servers, emit a report. |

The ToolPlan YAML is the design decision the rest hangs on. The model proposes the
curation; a human reviews and hand-edits the YAML; `build` turns it into code with no
model in the loop. Nothing the LLM produces is a black box, and iterating on a plan is
free because only `plan` and `eval` spend tokens.

```yaml
# one of the 6 tools from the hero server, as it appears in the editable plan
- name: refund_lookup
  intent: List or retrieve refunds
  read_only: true
  endpoints: [GET /v1/refunds, GET /v1/refunds/{refund}]
  dispatch: { mode: select, selector: mode }   # "list" or "get" picks the endpoint
  response_filter:
    fields: [id, amount, currency, status, reason, created, charge]
```

## Quickstart

```bash
git clone <this-repo> && cd prompt-to-mcp && uv sync
```

The petstore spec ships in the repo, so this runs end to end out of the box.
`inspect`, `plan`, and `build` also accept an `http(s)` URL in place of a file path.

```bash
# 1. what a naive conversion would cost you
prompt-to-mcp inspect tests/fixtures/petstore.yaml

# 2a. curate it for a described server (the one paid step) into an editable ToolPlan
prompt-to-mcp plan tests/fixtures/petstore.yaml \
    "a server for looking up pets and their orders" \
    -o petstore.toolplan.yaml --provider anthropic --model claude-sonnet-5
#     then open petstore.toolplan.yaml and edit anything you disagree with

# 2b. or curate the whole API automatically
prompt-to-mcp plan tests/fixtures/petstore.yaml --auto \
    -o petstore.toolplan.yaml --provider anthropic --model claude-sonnet-5

# 3. compile the plan into a server (free, deterministic)
prompt-to-mcp build petstore.toolplan.yaml --spec tests/fixtures/petstore.yaml \
    -o generated/petstore
```

`p2m` is a short alias for the same CLI (`p2m plan ...`, `p2m build ...`).

To reproduce the Stripe and GitHub numbers you need their OpenAPI descriptions (not
checked in), a running Ollama, and an API token:

```bash
# Stripe (test mode): distilled ran live, naive refused by the guard
PYTHONUTF8=1 prompt-to-mcp eval specs/spec3.json --server-dir generated/stripe \
    --scenarios scenarios/stripe.yaml -o eval_runs/stripe --only distilled --trials 3 \
    --provider ollama --model qwen2.5:7b --model-context 32768

PYTHONUTF8=1 prompt-to-mcp eval specs/spec3.json --server-dir generated/stripe \
    --scenarios scenarios/stripe.yaml --only naive --trials 1 \
    --provider ollama --model qwen2.5:7b --model-context 32768

# GitHub: same shape against generated/github and scenarios/github.yaml
mkdir -p specs && curl -L -o specs/api.github.com.json \
  https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json
```

## Any model, any provider

The LLM layer speaks Anthropic-native or the OpenAI chat protocol. The two ends of this
project ran on different vendors with zero code changes: curation on a hosted model, the
eval on a local 7B via Ollama.

| Preset (`--provider`) | Protocol | Base URL |
|---|---|---|
| `anthropic` | anthropic | SDK default |
| `glm` | anthropic | `api.z.ai/api/anthropic` |
| `kimi` | anthropic | `api.moonshot.ai/anthropic` |
| `openrouter` | openai | `openrouter.ai/api/v1` |
| `ollama` | openai | `localhost:11434/v1` |

Custom gateways work with `--base-url <url> --api-protocol openai|anthropic`.
Configuration is read from flags, then environment variables, then a `.env` in the CWD,
then preset defaults. The environment variables are `P2M_API_KEY`, `P2M_MODEL`,
`P2M_BASE_URL`, `P2M_API_PROTOCOL`, and `P2M_MAX_OUTPUT_TOKENS`; a bare
`ANTHROPIC_API_KEY` or `OPENAI_API_KEY` is picked up as a key fallback. Run
`prompt-to-mcp providers` for the full picture.

Running on real providers surfaced two quirks, both handled in
[`prompt_to_mcp/llm.py`](prompt_to_mcp/llm.py):

- Gemini 3 attaches an opaque `thought_signature` to each tool call and rejects the next
  turn unless it is echoed back verbatim. The neutral `ToolCallRequest.extra` field
  carries it through the agent loop.
- Some gateways count a model's hidden reasoning tokens against `max_tokens`, so a
  modest cap starves the visible answer. The output budget is configurable (32k default)
  with a truncation-aware retry.

## Methodology and limitations

For each `(server, scenario, trial)` the harness runs an agent loop and records the
outcome, task success, tool-call count, token usage, and wall time. Outcomes are typed
so a failed run can never masquerade as a real one:

- `completed`: the agent answered; `success` says whether it passed the checks.
- `provider_error`: the provider rejected the request, or the `--model-context` guard
  refused an oversized tool surface before running it.
- `harness_error`: an exception inside one trial. It never crashes the whole run.

Success is checked cheapest-first: substring and regex checks on facts that don't drift,
with an LLM-as-judge rubric as fallback for open-ended answers.

Known limits:

| Limitation | Effect |
|---|---|
| `chars/4` token estimator | context figures are order-of-magnitude, not a tokenizer |
| Small N, 1 local model | success rates are noisy and model-specific |
| No no-tools control | some scenarios are partly answerable from model memory |
| Naive never run *fitted* | the win leans on the context ceiling, not proven tool quality |
| One write scenario, missed by the 7B | mutating paths are generated but not yet passed by this agent |
| LLM-as-judge fallback | can misjudge open-ended answers |

## Repo map

| Path | What |
|---|---|
| `prompt_to_mcp/ingest.py`, `analyze.py` | Spec loading and the deterministic damage report |
| `prompt_to_mcp/curate.py`, `toolplan.py` | The LLM curation pass and the ToolPlan schema |
| `prompt_to_mcp/emit.py`, `templates/` | ToolPlan to FastMCP codegen (Jinja2) |
| `prompt_to_mcp/evalrun.py`, `evalspec.py`, `evalreport.py` | The eval harness |
| `prompt_to_mcp/llm.py` | Provider-pluggable LLM layer (both protocols) |
| `prompt_to_mcp/cli.py` | The `inspect / plan / build / eval` CLI |
| `stripe-support.toolplan.yaml`, `generated/stripe-support/` | The hero prompt-first server |
| `scenarios/`, `generated/`, `eval_runs/` | Scenarios, the curated servers, the runs behind the numbers |

## License

MIT. See [`LICENSE`](LICENSE).
