Metadata-Version: 2.4
Name: pytest-llmtest
Version: 0.1.0
Summary: The pytest for LLMs — fast, Pydantic-based assertions for AI applications
Project-URL: Homepage, https://llmtest.dev
Project-URL: Documentation, https://docs.llmtest.dev
Project-URL: Repository, https://github.com/bahadiraraz/llmtest
Project-URL: Bug Tracker, https://github.com/bahadiraraz/llmtest/issues
Project-URL: Changelog, https://github.com/bahadiraraz/llmtest/releases
Author-email: Bahadır Araz <bahadiraraz@protonmail.com>
Maintainer-email: Bahadır Araz <bahadiraraz@protonmail.com>
License: MIT
License-File: LICENSE
Keywords: ai,evaluation,llm,pydantic,pytest,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: llmtest-cli
Requires-Dist: llmtest-core
Requires-Dist: llmtest-pytest
Requires-Dist: pydantic>=2.0
Provides-Extra: all
Requires-Dist: llmtest-anthropic; extra == 'all'
Requires-Dist: llmtest-ollama; extra == 'all'
Requires-Dist: llmtest-openai; extra == 'all'
Requires-Dist: pydantic-ai>=1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: llmtest-anthropic; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: coverage>=7.0; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: ollama
Requires-Dist: llmtest-ollama; extra == 'ollama'
Provides-Extra: openai
Requires-Dist: llmtest-openai; extra == 'openai'
Provides-Extra: pydantic-ai
Requires-Dist: pydantic-ai>=1.0; extra == 'pydantic-ai'
Description-Content-Type: text/markdown

<p align="center">
  <img src=".github/icon.svg" alt="LLMTest" width="80" height="80" />
</p>

<h1 align="center">llmtest</h1>

<p align="center">
  <strong>The pytest for LLMs.</strong> Test your AI outputs like you test your code.
</p>

<p align="center">
  <a href="https://badge.fury.io/py/pytest-llmtest"><img src="https://badge.fury.io/py/pytest-llmtest.svg" alt="PyPI version" /></a>
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10+-blue.svg" alt="Python 3.10+" /></a>
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT" /></a>
</p>

## Quick Start

```bash
pip install pytest-llmtest[anthropic]
```

```python
from llmtest import expect, llm_test

@llm_test(
    expect.contains("Paris"),
    expect.latency_under(2000),
    expect.cost_under(0.001),
    model="claude-sonnet-4-20250514",
)
def test_capital(llm):
    output = llm("What is the capital of France?")
    assert "Paris" in output.content
```

```bash
pytest test_capitals.py -v
```

## Features

- **22+ assertions** — text, performance, agent, composable
- **Zero LLM calls** for most checks — deterministic and instant
- **Built on Pydantic** — auto-validation, JSON serialization, schema generation
- **Multi-provider** — OpenAI, Anthropic, Ollama out of the box
- **Agent testing** — tool calls, loop detection, call ordering
- **Retry support** — handle non-deterministic outputs
- **CI/CD native** — pytest markers, JUnit XML, JSON reporters
- **Pydantic AI integration** — test your existing agents

## Installation

```bash
pip install pytest-llmtest[anthropic]    # Anthropic
pip install pytest-llmtest[openai]       # OpenAI
pip install pytest-llmtest[ollama]       # Ollama (local)
pip install pytest-llmtest[all]          # Everything
```

## Documentation

Full docs at **[docs.llmtest.dev](https://docs.llmtest.dev)**

## Contributing

```bash
git clone https://github.com/bahadiraraz/llmtest
cd llmtest
uv sync --all-extras
uv run pytest tests/ -v
```

## License

MIT
