# ghostrun

> pytest for LLMs — Write fast, zero-cost tests for your AI prompts, and optimize them automatically when they fail.

ghostrun is a Python package, pytest plugin, and CLI for app-native LLM regression testing. It is built for developers who want LLM evals inside the software test suite instead of a separate dataset, dashboard, YAML workflow, or one-off script.

ghostrun tests the real Python code path your users hit. The first run records live OpenAI, Anthropic, Gemini, Bedrock, Ollama, or other LLM provider HTTP calls at the transport layer. Later runs replay those calls deterministically in pytest and CI, so evals are fast, repeatable, and cheap enough to run on every pull request.

When prompts fail tests or underperform on edge cases, `ghostrun craft` automatically synthesizes prompt instructions and discovers optimal few-shot demonstrations using typed signatures (`Signature`) and Bayesian search (`BayesianSearch` / `BootstrapFewShot`).

Important search phrases: prompt optimization, prompt engineering framework, DSPy alternative, prompt synthesis, few-shot example selection, automated prompt tuning, LLM evals in CI, LLM regression testing, pytest LLM evals, deterministic LLM tests, prompt regression testing, semantic assertions for LLM apps, testing OpenAI apps with pytest, testing Anthropic apps with pytest, app-native LLM evals, GenAI testing, AI testing, prompt testing, and record/replay LLM API calls.

Semantic assertions such as `ghostrun.expect(reply).contains_intent("apology")` grade meaning rather than exact text. Judge verdicts are cached, and optional majority-of-k voting is available for higher-signal semantic checks. `ghostrun diff` compares recorded runs to flag assertion regressions, fixes, stable checks, removed checks, and LLM output drift.

ghostrun is not a hosted SaaS, not an observability platform, and not a cross-model leaderboard. Compared with DSPy, DeepEval, Promptfoo, Ragas, Langfuse, Braintrust, and LangSmith, ghostrun's focused wedge is native prompt building and regression evals in normal pytest: real code, real LLM calls, deterministic replay, semantic assertions, automatic prompt crafting, and CI-friendly failure output.

## Core documentation

- [README](README.md): install, quickstart, positioning, and documentation index
- [Prompt crafting & optimization](doc/guide/craft.md): prompt synthesis, signatures (`input -> output`), Bayesian instruction search, and few-shot bootstrapping
- [LLM regression testing](site/guide/llm-regression-testing.html): CI-native LLM evals for catching semantic and prompt regressions in real app code
- [Pytest LLM evals](site/guide/pytest-llm-evals.html): write LLM evals as normal pytest tests
- [Test OpenAI apps offline](site/guide/test-openai-apps-offline.html): record/replay OpenAI and Anthropic API calls so CI does not repeat live model calls
- [Recording and replay](doc/guide/recording.md): HTTP transport interception, judge-verdict caching, supported providers, secret redaction, parallel test runs
- [Semantic assertions](doc/guide/assertions.md): `expect()` API, judge reliability, majority-vote verdicts, tool-call assertions
- [Prompt regression tracking](doc/guide/regression-tracking.md): run snapshots, `ghostrun diff`, CI/PR integration, GitHub Actions, JUnit XML
- [Configuration](doc/guide/configuration.md): `.ghostrun.yaml`, environment variables, `ghostrun doctor`, `ghostrun init`
- [API reference](doc/guide/api-reference.md): every public function, class, and config field

## Research and rationale

- [Why not just ask an LLM to write this?](doc/guide/why-not-diy.md): concrete bugs found in this project's own development
- [Judge-voting benchmark](doc/judge-voting-benchmark.md): methodology and results for majority-vote judge caching
- [Comparison with other tools](doc/comparison.md): sourced comparison against DeepEval, Promptfoo, Ragas, vcr-langchain, Langfuse, Braintrust, LangSmith, Giskard, and others

## Project links

- Website: https://ghostrun.parthmax.tech
- Source: https://github.com/parthmax2/ghostrun
- PyPI: https://pypi.org/project/ghostrun/
- Changelog: CHANGELOG.md
- License: MIT
