Metadata-Version: 2.5
Name: pytest-llm-vcr
Version: 0.6.0
Summary: Record and replay LLM HTTP traffic for deterministic pytest runs
Project-URL: Homepage, https://github.com/yashshah9/llm-vcr
Project-URL: Repository, https://github.com/yashshah9/llm-vcr
Project-URL: Issues, https://github.com/yashshah9/llm-vcr/issues
Author-email: Yash Shah <yash376351@gmail.com>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: pyyaml>=6.0
Requires-Dist: structlog>=24.1
Provides-Extra: dev
Requires-Dist: mypy>=1.9; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

# llm-vcr

Record and replay LLM HTTP traffic for **deterministic, key-free pytest runs**.

[![PyPI](https://img.shields.io/pypi/v/pytest-llm-vcr.svg)](https://pypi.org/project/pytest-llm-vcr/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![CI](https://github.com/yashshah9/llm-vcr/actions/workflows/ci.yml/badge.svg)](https://github.com/yashshah9/llm-vcr/actions/workflows/ci.yml)

> **Status:** v0.6 — SSE streaming replay, sequential tool-call cassettes, OpenAI + Anthropic semantic matching, model-date normalize, and `llm-vcr diff`.

## 60-second try

```bash
pip install pytest-llm-vcr
llm-vcr health
docker compose run --rm test    # cassette replay, no API key
```

## Why this vs alternatives

| Approach | Strength | Gap |
|----------|----------|-----|
| **llm-vcr** | LLM-aware matching, redaction, SSE + sequential cassettes | httpx-focused today |
| VCR.py / pytest-recording | Mature generic HTTP cassettes | No model-date normalize or LLM-shaped diffs |
| Hand-written mocks | Fast, no network | Drift from live provider payloads |
| Live API in CI | Highest fidelity | Flaky, keyed, and expensive |

## Problem

Testing code that calls LLMs is slow, flaky, and expensive. Hand-written mocks drift from reality. Generic HTTP cassettes (VCR.py) don't understand LLM request shapes or redact API keys well.

## Key features (v0.6)

- **pytest plugin** — `@llm_vcr` decorator, `llm_vcr_client` fixture, `--llm-vcr-record`
- **httpx transport** — sync + async, including `client.stream(...)`
- **YAML cassettes** — `streaming: true` + `chunks` for SSE
- **Matching** — exact (default) or `matcher="semantic"` (volatile keys, model aliases, messages by role+content, tools by name); Anthropic Messages API via `api.anthropic.com` URL detection
- **`llm-vcr diff`** — show normalized differences between JSON bodies or cassette interactions
- **Automatic redaction** — strips api_key, `x-api-key`, token, authorization fields

## Architecture

```
@pytest test
    └── @llm_vcr decorator
            └── VCRTransport (httpx)
                    ├── replay mode → read cassette YAML
                    └── record mode → live HTTP + save cassette
```

| Component | Technology | Why |
|-----------|------------|-----|
| HTTP | httpx | Modern, sync+async, transport hooks |
| Cassettes | YAML | Readable diffs in PRs |
| Tests | pytest entry point | Zero-config discovery |

## Installation

```bash
pip install pytest-llm-vcr
pip install -e ".[dev]"  # from source
```

## Local development

```bash
pip install -e ".[dev]"
pytest tests/ -v
llm-vcr health
llm-vcr diff left.json right.json
```

## Docker

```bash
docker compose run --rm test
docker compose run --rm health
```

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_VCR_CASSETTE_DIR` | `tests/cassettes` | Cassette storage directory |
| `LLM_VCR_RECORD` | `false` | Force record mode |

## Usage

### Replay (CI — no API key needed)

```python
import httpx
from llm_vcr.plugin import llm_vcr

@llm_vcr("my_test")
def test_chat(client: httpx.Client) -> None:
    resp = client.post(
        "https://api.openai.com/v1/chat/completions",
        json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello"}]},
    )
    assert resp.json()["choices"][0]["message"]["content"]
```

### Record new cassettes

```bash
LLM_VCR_RECORD=true pytest tests/ --llm-vcr-record
```

### Diff normalized request bodies

```bash
llm-vcr diff left.json right.json
llm-vcr diff cassette_a.yaml cassette_b.yaml --index 0
```

Model strings with a trailing `-YYYY-MM-DD` normalize equal so dated aliases do not show as diffs.

### Anthropic Messages API

Point httpx at `https://api.anthropic.com/v1/messages` the same way as OpenAI — record once, replay in CI without a key. Query strings (e.g. beta flags) are stripped for matching. Use `matcher="semantic"` to ignore `metadata` and `tool_use` / `tool_result` ids:

```python
@llm_vcr("anthropic_chat", matcher="semantic")
def test_anthropic(client: httpx.Client) -> None:
    resp = client.post(
        "https://api.anthropic.com/v1/messages",
        json={
            "model": "claude-3-5-sonnet",
            "max_tokens": 64,
            "messages": [{"role": "user", "content": "Say hello"}],
        },
        headers={"x-api-key": "sk-ant-...", "anthropic-version": "2023-06-01"},
    )
    assert resp.json()["content"][0]["text"]
```

`x-api-key` is redacted in cassette bodies when present.

## Running tests

```bash
pytest tests/ -v
```

## Roadmap

- [x] SSE streaming chunk replay
- [x] Tool-call multi-step loops (`@llm_vcr(..., sequential=True)`)
- [x] Async httpx transport (fixture + replay)
- [x] Model-date normalize + `llm-vcr diff`
- [x] OpenAI + Anthropic semantic request matching (`matcher="semantic"`)

## Known limitations (v0.6)

- Sequential matching is opt-in (`sequential=True`); default matching is still hash-based
- Matching: exact hash (default) or semantic (volatile keys, model aliases, message/tool shape; Anthropic when host is `api.anthropic.com`)
- Record mode for streaming stores chunks, not per-event timestamps

## License

MIT
