pytest for LLMs

Write fast, zero-cost tests for your AI prompts, and optimize them automatically when they fail.

pip install ghostrun
ollama pull llama3.2:3b   # for the default local judge
ghostrun init              # scaffolds a working first test
ghostrun run               # runs tests with live animated mascot

Fast & free tests

Wrap your tests in @ghostrun.record. The first run saves real LLM calls; later runs replay instantly in 0.05 seconds offline.

No more flakiness

Assert on meaning and intent (tone, intent, output matching) instead of exact text, graded by a free, private local judge.

Self-healing prompts

If your prompt fails a test, `ghostrun craft` automatically searches for the best instructions and few-shots to pass the test.

Native to pytest

No dashboard migrations, no YAML workarounds, no playground exports. Evals live in normal Python and pytest.

Guide

Getting started
LLM regression testing
Pytest LLM evals
Test OpenAI apps offline
Recording and replay
Semantic assertions
Prompt regression tracking
Configuration
API reference
Why not just ask an LLM to write this?

Research

Judge-voting benchmark
Comparison with other tools

Source, issues, and the changelog live on GitHub.