Metadata-Version: 2.4
Name: lap-score
Version: 0.5.0
Summary: lap - measure & improve the token-efficiency of agent-facing APIs (OpenAPI & MCP): scorer, linter, the LAP profile, and a token benchmark
License: MIT
Project-URL: Homepage, https://github.com/lCrazyblindl/lap
Project-URL: Roadmap, https://github.com/lCrazyblindl/lap/blob/main/ROADMAP.md
Project-URL: Changelog, https://github.com/lCrazyblindl/lap/blob/main/CHANGELOG.md
Project-URL: Leaderboard, https://github.com/lCrazyblindl/lap/blob/main/docs/LEADERBOARD.md
Keywords: llm,agents,openapi,mcp,tokens,api,token-efficiency
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx
Requires-Dist: tiktoken
Requires-Dist: pyyaml
Provides-Extra: mcp
Requires-Dist: fastmcp; extra == "mcp"
Provides-Extra: faithful
Requires-Dist: anthropic; extra == "faithful"
Provides-Extra: all
Requires-Dist: fastmcp; extra == "all"
Requires-Dist: anthropic; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# lap — token-efficiency scorer for agent-facing APIs

`lap` is the open, neutral, standalone toolkit (no pet-zoo dependency) that measures
how many **tokens** an API's definitions cost an LLM. It answers: *is my agent-API
menu efficient, and by how much could it shrink?* — an open, reproducible number to set
beside the fast-growing MCP/OpenAPI tooling (see [`../docs/LANDSCAPE.md`](../docs/LANDSCAPE.md)
for the neighbors LAP builds on and credits).

## Install

```bash
pip install -e .                  # from the repo root (or: pip install lap-score once published)
pip install -e ".[mcp]"           # + real-MCP baseline (fastmcp)
pip install -e ".[faithful]"      # + faithful Anthropic count_tokens
```

Core deps are just `httpx` + `tiktoken` + `pyyaml`; `fastmcp` and `anthropic` are optional extras.
Robust to real specs: `allOf`/`oneOf`/`anyOf`, `$ref` in params/requestBodies/responses,
path-item-level parameters, OpenAPI 3.1 `type` lists, external `$ref`s (left intact), YAML input,
**Swagger/OpenAPI 2.0** (response `schema`, `in: body` params, type-on-parameter, `#/definitions`),
and non-JSON media types (`*+json`, form, XML). Verified crash-free + non-degenerate across 175+
real APIs.guru specs — re-run with [`../experiments/fuzz_corpus.py`](../experiments/fuzz_corpus.py).

## Quickstart

```bash
lap score  https://petstore3.swagger.io/api/v3/openapi.json   # menu (bucket A) token cost
lap lint   https://petstore3.swagger.io/api/v3/openapi.json   # flag LAP rule violations
lap score  --mcp-url http://localhost:8080/mcp                # score a live MCP server's tools
lap lint   --mcp-url http://localhost:8080/mcp                # lint a live MCP server (M-rules + grade)
lap lint   --mcp "python -m mcp_server_git"                   # ...or over stdio
lap score  lap/examples/bookstore.openapi.json

# no install needed, from the repo root:
python -m lap.score lap/examples/bookstore.openapi.json
```

Example output:

```
LAP menu score - Bookstore API
operations: 6   referenced component schemas: 2

  variant       A tokens  saved vs full  form
  openapi_full  418       +0%            6 tool(s)
  compact_sig   205       +51%           manifest text
  numbered      168       +60%           manifest text

Menu efficiency: compact signatures are +51% vs naive OpenAPI->tools (418 -> 205 tokens).
```

When `fastmcp` is installed, the score also includes a **real-MCP baseline**
(`FastMCP.from_openapi`) — what an actual MCP generator emits — plus its
output-schema-inclusive figure (`--no-mcp` to skip). On a real public API
(**Swagger Petstore**, 19 ops) the real MCP server costs **2226** menu tokens
(**3844** with output schemas) vs **415** for compact signatures — an ~81%
reduction. The toy finding holds in the wild: a real MCP generator is *heavier*
than the naive baseline.

The score also includes a lazy **`tool_search`** form (the Anthropic Tool Search /
Cloudflare Code Mode pattern: a fixed 2-tool menu + a name index, schemas loaded on
demand). Because it doesn't preload schemas, its bucket A is ~flat in the number of
operations — on a 120-operation API it collapses the menu ~83% vs full schemas,
beating even compact signatures at scale (Petstore: 1835 → 207, −89%).

- **Faithful counts:** set `ANTHROPIC_API_KEY` (uses the free Anthropic `count_tokens`
  endpoint; tool defs counted via the real `tools=` parameter). Without it, a GPT-style
  `tiktoken` approximation — absolute numbers approximate, **relative ordering robust**.

## The LAP grade + badge

`lap score` also prints a composite **LAP grade** — one documented 0–100 number (letter
A–F) folding three sub-scores: **menu** (naive-menu tokens per operation, 0.45), **result**
(heaviest estimated response, 0.30) and **hygiene** (lint findings per operation, 0.25);
log-scaled, constants in [`grade.py`](grade.py), formula in the
[profile](../profile/llm-api-profile.md#the-lap-grade-composite-0100--letter). Calibration on
real APIs (post-M3, query params counted): LaunchDarkly **B**, Spotify/SQS **C**,
GitHub/Postman/DynamoDB **D**, Google Drive **F**.

`lap badge` turns it into a README badge (shields.io endpoint JSON):

```bash
lap badge api/openapi.json -o docs/lap-badge.json   # commit it, then embed:
# https://img.shields.io/endpoint?url=<raw URL of docs/lap-badge.json>
```

…or from the bundled Action, add `badge-path: docs/lap-badge.json`.

## Score your installed MCP stack

`lap stack` answers the 2026 headline question — *"how many tokens does my agent pay before I
type a word?"* — for **your** machine. It reads the agent's own MCP config (Claude Code project
`.mcp.json`, Claude Code `~/.claude.json`, Claude Desktop `claude_desktop_config.json`, or any
JSON with an `mcpServers` map), connects to every server it lists (stdio or HTTP), and totals
the advertised tool menus:

```bash
lap stack                        # auto-discover Claude Code / Claude Desktop configs
lap stack path/to/mcp.json       # or score an explicit config
lap stack --only github,jira     # subset; --json for machine-readable; --timeout N per server
```

```
LAP stack scan - tokenizer: tiktoken-approx

demo-mcp-config.json
  server        kind   tools  menu tokens  compact  note
  time          stdio      2          283       31
  git           stdio     12         1418      153
  needs-node    stdio      -            -        -  RuntimeError: Client failed to connect: ...
  TOTAL                    14         1701      184

Your agent pays ~1,701 tokens of tool menus at session start - before you type a word
(14 tools across 2 reachable server(s)).
Compact signatures of the same tools would cost 184 (+89% saved); one lazy tool_search menu
over the whole stack, 193 (+89% saved).
```

Unreachable servers (missing binary, no credentials) become annotated rows, not crashes. The
stack-level `tool_search` what-if is counted honestly: the fixed search/call tools are paid
**once** for the whole stack, plus a name index across all servers. Needs the `[mcp]` extra.

## Auto-fix as an OpenAPI Overlay

`lap fix` turns the *structurally fixable* lint findings into an
[OpenAPI Overlay 1.0.0](https://spec.openapis.org/overlay/v1.0.0) document — advice becomes an
applicable patch: R3 → a `limit` parameter, R1 → `fields`, R2 → `filter`, E1 → a declared
`4XX` error response. (D3/A1 stay advisory — renames and new endpoints are semantic
decisions.) The overlay declares the *contract*; your server still has to implement it.

```bash
lap fix api/openapi.json -o lap-overlay.yaml          # apply with any Overlay-aware tool
lap fix api/openapi.json --apply patched.json         # ...or the built-in merge
```

On the bundled Bookstore example: 15 lint findings → 3 after applying, and the LAP grade
jumps **B (72) → A (91)**. _(The menu forms currently count path+body parameters only, so the
added query params don't change bucket A — see the roadmap for the query-param menu fix.)_

## Diff mode

`lap score --diff <before> <after>` compares two versions of a spec instead of scoring one —
"did this PR make the API worse for agents?" Reports the menu token delta per form, plus which
LAP lint findings were newly introduced or fixed:

```bash
lap score --diff old_openapi.json new_openapi.json
lap score --diff old_openapi.json new_openapi.json --gate-form compact_sig --max-growth 500
```

## CI gate

`--json` makes both commands machine-readable; thresholds set the exit code so LAP can fail a build:

```bash
lap score openapi.json --gate-form compact_sig --max-menu-tokens 800   # exit 1 if the menu is too heavy
lap score --diff old.json new.json --gate-form compact_sig --max-growth 500  # exit 1 if the menu grew too much
lap lint  openapi.json --fail-on warn                                  # exit 1 on any warning
lap lint  openapi.json --ignore R2,A1                                  # suppress rules (or a ./.lapignore file)
```

GitHub Actions:

```yaml
- run: pip install lap-score          # or: pip install -e .
- run: lap score api/openapi.json --gate-form compact_sig --max-menu-tokens 800
- run: lap lint  api/openapi.json --fail-on warn
```

…or the bundled composite **Action** (one step, no manual install):

```yaml
- uses: lCrazyblindl/lap@v0.5.0
  with:
    spec: api/openapi.json
    max-menu-tokens: "800"     # gate the compact_sig menu (omit = report only)
    fail-on: warn              # fail on any lint warning (omit = report only)
```

Already lint OpenAPI with **Spectral**? The same LAP rules ship as a ruleset —
see [`../spectral/`](../spectral/README.md).

## What it measures (and what it doesn't)

It measures **bucket A** (the definitions/menu the model carries in context) and **estimates**
the other two from the schemas: **B** (the call the model emits — tool name + required args in
a minimal tool-use envelope; optional params omitted, real schema examples honored) and **C**
(result size, from each response schema + an assumed `--page-size`). Both estimates are
structural lower bounds that capture keys/nesting/types. For list responses the C table also
shows a **projected** figure — the same page with each item cut to its first 3 schema fields —
annotated with whether the API actually advertises a projection param (`fields=`/`$select`),
so rule R1's saving is a number per endpoint, not advice. For *measured* B/C on live tasks see
[`../experiments/token-bench`](../experiments/token-bench/README.md). The conventions
behind the compact form are the [LAP profile](../profile/llm-api-profile.md).

## Files

| file | role |
| --- | --- |
| `openapi_ir.py` | load any OpenAPI (file/URL) → normalized operations + `inline_refs` |
| `menu.py` | render the menu forms (openapi_full / compact_sig / numbered) from the IR |
| `mcp_form.py` | real-MCP baseline via `FastMCP.from_openapi` (optional; `--no-mcp` to skip) |
| `mcp_client.py` | scores a live MCP server's advertised tools (`lap score --mcp-url`) |
| `stack.py` | `lap stack` — score the user's installed MCP stack from their agent config |
| `grade.py` | the composite LAP grade (0–100 + letter) and `lap badge` (shields.io endpoint JSON) |
| `estimate.py` | estimates bucket C (result size) from response schemas (`--page-size`) |
| `tokens.py` | token counting (Anthropic endpoint, or tiktoken approx) |
| `score.py` | the `lap score` CLI |
| `lint.py` | the `lap lint` CLI — checks a spec against the LAP profile rules (D3/R1/R2/R3/W1/E1/A1), or a live MCP server's tools (`--mcp-url`/`--mcp`, rules D3/M1–M4 + grade) |
| `overlay.py` | `lap fix` — the fixable lint findings as an OpenAPI Overlay 1.0.0 (+ built-in `--apply`) |
| `examples/` | sample specs: a Bookstore API, a gnarly OpenAPI 3.1 (allOf / $ref-params / nullable / external-ref), and a Swagger 2.0 spec (`swagger2.json`) |
