Metadata-Version: 2.4
Name: ghostrun
Version: 0.1.1
Summary: pytest for LLMs: deterministic HTTP record/replay, local LLM-as-judge semantic assertions, and prompt regression diffing for testing GenAI applications.
Project-URL: Homepage, https://github.com/parthmax2/ghostrun
Project-URL: Issues, https://github.com/parthmax2/ghostrun/issues
Project-URL: Documentation, https://github.com/parthmax2/ghostrun#documentation
Project-URL: Changelog, https://github.com/parthmax2/ghostrun/blob/main/CHANGELOG.md
Author: parthmax2
License: MIT
License-File: LICENSE
Keywords: agent-testing,ai-testing,anthropic,ci-cd,deterministic-testing,genai,generative-ai,http-cache,http-mocking,llm,llm-as-judge,llm-eval,llm-evaluation,llm-testing,ollama,openai,prompt-regression,prompt-testing,pytest,pytest-plugin,record-replay,regression-testing,semantic-assertions,testing,unit-testing,vcr
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Testing :: Unit
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx>=0.24
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# ghostrun

[![PyPI](https://img.shields.io/pypi/v/ghostrun.svg)](https://pypi.org/project/ghostrun/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue.svg)](pyproject.toml)
[![CI](https://github.com/parthmax2/ghostrun/actions/workflows/ci.yml/badge.svg)](https://github.com/parthmax2/ghostrun/actions/workflows/ci.yml)

**pytest for LLMs.** Deterministic record/replay and semantic assertions for GenAI apps — **local-first, privacy-first, zero SaaS lock-in.**

### The problem

```python
reply = generate_reply("Where is my refund?")
assert reply == "I'm sorry for the delay..."   # fails tomorrow: LLM never says the same thing twice
```

Every real test run also means a live API call — slow, costs money, and now your CI needs a secret API key just to run the test suite.

### What ghostrun does about it

1. **Deterministic replay** — the first run records real LLM HTTP calls to a local `.ghostrun_cache/`; every run after replays them instantly from disk. Zero API cost, zero latency, zero flakiness, no key needed in CI.
2. **Semantic assertions** — assert on *meaning*, not exact text:
   ```python
   ghostrun.expect(reply).contains_intent("apology")
   ghostrun.expect(reply).tone_is("empathetic")
   ```
   Graded by a **local Ollama model** by default — your prompts and data never leave your machine. That grading verdict gets cached too, so it's also free and deterministic after the first run.

No cloud dashboard. No custom CLI to learn. No dataset to author by hand. Just `pytest`.

---

## Install

```bash
pip install ghostrun
```

For the default (local, free, private) judge, install [Ollama](https://ollama.com) and pull a small model:

```bash
ollama pull llama3.2:3b
```

If anything doesn't work, `ghostrun doctor` diagnoses the setup — see
[Configuration](doc/guide/configuration.md#diagnosing-a-broken-setup-ghostrun-doctor).

**Fastest start:** `ghostrun init` scaffolds a working first test against
whatever LLM SDK it finds in your project (OpenAI, Anthropic, or a generic
HTTP fallback) plus a `.ghostrun.yaml` — no config to author by hand:

```bash
ghostrun init
pytest test_ghostrun_example.py
```

## Quickstart

```python
# test_customer_support.py
import ghostrun
from my_app import generate_reply

@ghostrun.record(model="gpt-4o-mini")
def test_reply_generation():
    reply = generate_reply("Where is my refund?")

    ghostrun.expect(reply).contains_intent("apology")
    ghostrun.expect(reply).contains_intent("refund policy")
    ghostrun.expect(reply).does_not_contain_intent("arguing")
    ghostrun.expect(reply).tone_is("empathetic")
```

```bash
$ pytest test_customer_support.py
================================ test session starts ================================
collected 1 item

test_customer_support.py .                                                     [100%]
================================ 1 passed in 0.04s =================================
```

The `0.04s` is the whole point — after the first record, calls replay from disk.

> **Note on the API:** the assertion entry point is `ghostrun.expect(...)`, not
> `ghostrun.assert(...)` — `assert` is a reserved Python keyword and cannot be a
> function name.

**Record/replay alone needs no Ollama at all** — the judge is only touched when
you call a judge-backed assertion (`contains_intent`, `tone_is`, `matches`).
Deterministic assertions (`contains`, `is_valid_json`) and tool-call assertions
never invoke it.

## Is this for you?

**Use ghostrun if** you're writing pytest tests around code that calls an LLM
(directly or via the OpenAI/Anthropic SDKs) and want that suite to run offline,
free, and deterministically after the first recording.

**Skip it if** you need a hosted dashboard/observability platform for
production traffic (see Langfuse/LangSmith/Braintrust instead), you're
building a red-team/adversarial test suite (see Giskard), or you want 50+
pre-built judge metrics out of the box today (see DeepEval — more mature, more
metrics, but doesn't intercept your app's own HTTP calls the way ghostrun
does). See [doc/comparison.md](doc/comparison.md) for the full, researched
breakdown of where ghostrun is ahead and where it's duplicating existing work.

## Documentation

Start here, in order:

| Guide | What's in it |
| :--- | :--- |
| [doc/guide/recording.md](doc/guide/recording.md) | How record/replay works, judge-verdict caching, supported providers, secret redaction, parallel test runs |
| [doc/guide/assertions.md](doc/guide/assertions.md) | Semantic assertions, judge reliability (benchmarked, not asserted), majority-vote verdicts, tool/function-call assertions |
| [doc/guide/configuration.md](doc/guide/configuration.md) | `.ghostrun.yaml`, environment variables, pytest flags, `ghostrun doctor`, `ghostrun init` |

Deeper reference, once you're past the basics:

| Guide | What's in it |
| :--- | :--- |
| [doc/guide/regression-tracking.md](doc/guide/regression-tracking.md) | Snapshotting runs, `ghostrun diff`, posting a regression as a PR comment, JUnit CI integration |
| [doc/guide/api-reference.md](doc/guide/api-reference.md) | Every public function, class, exception, and config field |
| [doc/guide/why-not-diy.md](doc/guide/why-not-diy.md) | The actual bugs found building this — the case for a maintained package over a five-minute prompt |
| [doc/judge-voting-benchmark.md](doc/judge-voting-benchmark.md) | Full methodology and results for the majority-vote judge-caching benchmark |
| [doc/comparison.md](doc/comparison.md) | Researched comparison against DeepEval, Promptfoo, Ragas, vcr-langchain, and 9 other tools |
| [CHANGELOG.md](CHANGELOG.md) | Release notes |

A hosted, searchable version of this documentation is planned at
[parthmax2.github.io/ghostrun](https://parthmax2.github.io/ghostrun/) (config
in `mkdocs.yml`, builds via `.github/workflows/docs.yml`).

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) — setup, test requirements, and where
things live in the codebase.

## Development

```bash
pip install -e ".[dev]"
pytest            # runs fully offline using the echo judge
```

## License

MIT
