Metadata-Version: 2.4
Name: llm-rate-governor
Version: 0.1.0
Summary: Client-side rate limiting for the Claude API. Never sees your API key.
Project-URL: Homepage, https://github.com/Aathan-Aweso/llm-governor
Project-URL: Issues, https://github.com/Aathan-Aweso/llm-governor/issues
Project-URL: Changelog, https://github.com/Aathan-Aweso/llm-governor/blob/main/CHANGELOG.md
Author-email: Aathan Aweso <techawesome007@gmail.com>
License: MIT License
        
        Copyright (c) 2026 YOUR NAME
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: anthropic,claude,llm,rate-limit,throttle,token-bucket
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: dashboard
Requires-Dist: rich>=13.0; extra == 'dashboard'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: rich>=13.0; extra == 'dev'
Description-Content-Type: text/markdown

# llm-governor

**Client-side rate limiting for the Claude API.** Stop before the server has to say no.

> **This library never sees your API key.** It wraps calls you make with your own
> client, and reads only `anthropic-ratelimit-*` response headers. No credentials
> are required, requested, logged, or stored.

```bash
pip install llm-governor
```

---

## Why

Claude's Messages API enforces three independent limits per model: requests per
minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM).
Blow any one of them and you get a `429`.

Most retry-on-429 code reacts *after* the failure. `llm-governor` mirrors the
server's own token bucket locally, corrects itself from the rate-limit headers on
every response, and paces your requests so the 429 never happens.

It also understands **cached input**. On most Claude models, `cache_read_input_tokens`
do not count toward ITPM. A naive limiter that counts every input token will
throttle you to a fraction of your real throughput; this one tracks your rolling
cache-hit rate and reserves only what actually counts.

## Quick start

```python
import anthropic
from llm_governor import Governor

client = anthropic.Anthropic()          # your key, your client
gov = Governor(model="claude-sonnet-5") # knows nothing about keys

with gov.slot(messages=messages, max_tokens=1024) as slot:
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=messages,
    )
    slot.observe(resp)   # feeds headers + usage back in
```

The first call runs against conservative defaults. From the second call onward,
the governor knows your real limits because the API told it.

## Async

```python
gov = Governor(model="claude-sonnet-5", max_concurrency=8)

async def worker(messages):
    async with gov.aslot(messages=messages, max_tokens=1024) as slot:
        resp = await client.messages.create(...)
        slot.observe(resp)
        return resp

await asyncio.gather(*(worker(m) for m in batch))
```

## Live dashboard

```bash
pip install "llm-governor[dashboard]"
llm-governor demo          # simulated workload, no API key needed
```

```python
from llm_governor.dashboard import Dashboard

with Dashboard(gov):
    run_my_batch()
```

```
╭─ llm-governor · claude-sonnet-5 ─────────────────────╮
│  RPM   ████████████████░░░░   812 / 1000             │
│  ITPM  ███████░░░░░░░░░░░░░   1.4M / 2.0M            │
│  OTPM  ██████████████████░░   361K / 400K            │
│                                                      │
│  cache  78.0%                        94 req/min      │
│  flight     7                        231 done        │
│  429s       0                        3.2s waited     │
╰────────────────────────────── live ──────────────────╯
```

Bars turn amber below 25% headroom and red below 10%.

## Dry run

See whether a workload *would* throttle, without waiting for it:

```python
gov = Governor(model="claude-sonnet-5", dry_run=True)
# ... replay your workload ...
print(gov.snapshot()["throttled_seconds"])
```

## Options

| Option | Default | Meaning |
|---|---|---|
| `model` | `"claude-sonnet-5"` | Limits are per-model; use one governor per model |
| `rpm` / `itpm` / `otpm` | conservative | Starting guesses, overridden by headers |
| `headroom` | `0.90` | Fraction of the real limit to use; leaves margin for bursts |
| `max_concurrency` | `None` | Cap simultaneous in-flight requests |
| `dry_run` | `False` | Compute waits but never sleep |

## On credentials

The header allowlist in `llm_governor.headers.ALLOWED_HEADERS` is explicit and
tested. `scrub()` is the only path by which headers enter this library, and a test
asserts that `x-api-key` and `authorization` never survive it. If you extend this
library, do not replace that with `dict(response.headers)`.

## Notes and limits

- State is per-process. For multiple workers sharing one org limit, divide
  `headroom` by your worker count, or wait for the planned Redis backend.
- Input estimation is character-based and approximate. Pass `estimated_input`
  explicitly if you have a real count from the token-counting endpoint.
- Batch API requests have separate limits and are not covered.
- `max_tokens` is reserved pessimistically and refunded once the real output
  size is known.

## License

MIT
