# ghostrun

> CI-native LLM evals for real applications: record LLM API calls once, replay them deterministically in pytest, and catch semantic regressions before they ship.

ghostrun is a Python package, pytest plugin, and CLI for app-native LLM regression testing. It is built for developers who want LLM evals inside the software test suite instead of a separate dataset, dashboard, YAML workflow, or one-off script.

ghostrun tests the real Python code path your users hit. The first run records live OpenAI, Anthropic, Gemini, Bedrock, Ollama, or other LLM provider HTTP calls at the transport layer. Later runs replay those calls deterministically in pytest and CI, so evals are fast, repeatable, and cheap enough to run on every pull request.

Important search phrases: LLM evals in CI, LLM regression testing, pytest LLM evals, deterministic LLM tests, prompt regression testing, semantic assertions for LLM apps, testing OpenAI apps with pytest, testing Anthropic apps with pytest, app-native LLM evals, GenAI testing, AI testing, prompt testing, and record/replay LLM API calls.

Semantic assertions such as `ghostrun.expect(reply).contains_intent("apology")` grade meaning rather than exact text. Judge verdicts are cached, and optional majority-of-k voting is available for higher-signal semantic checks. `ghostrun diff` compares recorded runs to flag assertion regressions, fixes, stable checks, removed checks, and LLM output drift.

ghostrun is not a hosted SaaS, not an observability platform, and not a cross-model leaderboard. Compared with DeepEval, Promptfoo, Ragas, Langfuse, Braintrust, and LangSmith, ghostrun's focused wedge is app-native regression evals in normal pytest: real code, real LLM calls, deterministic replay, semantic assertions, and CI-friendly failure output.

## Core documentation

- [README](README.md): install, quickstart, positioning, and documentation index
- [LLM regression testing](site/guide/llm-regression-testing.html): CI-native LLM evals for catching semantic and prompt regressions in real app code
- [Pytest LLM evals](site/guide/pytest-llm-evals.html): write LLM evals as normal pytest tests
- [Test OpenAI apps offline](site/guide/test-openai-apps-offline.html): record/replay OpenAI and Anthropic API calls so CI does not repeat live model calls
- [Recording and replay](doc/guide/recording.md): HTTP transport interception, judge-verdict caching, supported providers, secret redaction, parallel test runs
- [Semantic assertions](doc/guide/assertions.md): `expect()` API, judge reliability, majority-vote verdicts, tool-call assertions
- [Prompt regression tracking](doc/guide/regression-tracking.md): run snapshots, `ghostrun diff`, CI/PR integration, GitHub Actions, JUnit XML
- [Configuration](doc/guide/configuration.md): `.ghostrun.yaml`, environment variables, `ghostrun doctor`, `ghostrun init`
- [API reference](doc/guide/api-reference.md): every public function, class, and config field

## Research and rationale

- [Why not just ask an LLM to write this?](doc/guide/why-not-diy.md): concrete bugs found in this project's own development
- [Judge-voting benchmark](doc/judge-voting-benchmark.md): methodology and results for majority-vote judge caching
- [Comparison with other tools](doc/comparison.md): sourced comparison against DeepEval, Promptfoo, Ragas, vcr-langchain, Langfuse, Braintrust, LangSmith, Giskard, and others

## Project links

- Website: https://ghostrun.parthmax.in
- Source: https://github.com/parthmax2/ghostrun
- PyPI: https://pypi.org/project/ghostrun/
- Changelog: CHANGELOG.md
- License: MIT
