Metadata-Version: 2.5
Name: culprit-ai
Version: 0.1.5
Summary: Finds which prompt commit broke your agent.
Project-URL: Homepage, https://github.com/abi6374/culprit
Project-URL: Documentation, https://abi6374.github.io/culprit/
Project-URL: Repository, https://github.com/abi6374/culprit
Project-URL: Issues, https://github.com/abi6374/culprit/issues
Author: Abinivas
License-Expression: MIT
License-File: LICENSE
Keywords: evals,git,llm,prompts,semantic-diff
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: anthropic>=0.40
Requires-Dist: openai>=1.50
Requires-Dist: pydantic>=2.8
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: hatch-vcs>=0.5; extra == 'dev'
Requires-Dist: mkdocs-material>=9.5; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pillow>=11.0; extra == 'dev'
Requires-Dist: pre-commit>=3.8; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

# culprit

> `git bisect` thinking for prompts: compare what changed in behavior, not only in text.

Not to be confused with the unrelated [`culprit` package on PyPI](https://pypi.org/project/culprit/) or the [Culprit Rust CLI](https://github.com/Yanis897349/culprit) — see [How this compares](#how-this-compares).

[![PyPI version](https://img.shields.io/pypi/v/culprit-ai.svg)](https://pypi.org/project/culprit-ai/)
[![CI](https://github.com/abi6374/culprit/actions/workflows/ci.yml/badge.svg)](https://github.com/abi6374/culprit/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/abi6374/culprit/branch/main/graph/badge.svg)](https://codecov.io/gh/abi6374/culprit)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

[![Real culprit bisect run finding a seeded regression](docs/assets/bisect-demo.gif)](docs/bisect.md)

## Features

- [Semantic diff](docs/semantic-diff.md): an LLM summary backed by per-case eval evidence.
- [Behavioral bisect](docs/bisect.md): binary-search prompt history to locate the first regression.
- [Semantic merge (experimental)](docs/merge.md): conservatively combine compatible branch edits and verify the result before staging it.
- [Git-native architecture](docs/architecture.md): prompt history stays in ordinary Git.
- Provider adapters for OpenAI, Anthropic, and arbitrary OpenAI-compatible endpoints.
- Content-addressed [evaluation cache](docs/architecture.md) to avoid repeat API spend.
- `culprit check` warnings for common secret and PII patterns.

## How this compares

Among these similarly named tools, `culprit-ai` is differentiated by verifying prompt diff, bisect, and automated merge decisions against real eval runs rather than relying on text similarity or an LLM's judgment alone.

| Tool | Primary focus | Verification model |
|---|---|---|
| [`culprit` on PyPI](https://github.com/noordeen123/culprit) | Local, read-only root-cause and change-risk analysis for PRs, branches, fixes, and traces | Deterministic static analysis, with optional Git bisect confirmation when a repro command is available |
| [Culprit Rust CLI](https://github.com/Yanis897349/culprit) | A local-first, guided Git bisect workflow with optional AI explanation | Runs the supplied test command at candidate commits and restores the checkout afterward |
| **`culprit-ai` (this project)** | Prompt-behavior diff, behavioral bisect, and experimental semantic merge in one Git-native toolkit | Runs the configured eval set; merge additionally rejects a candidate that loses behavior shared by both parents |

Other prompt tools cover adjacent workflows and can complement `culprit-ai`:

| Tool | Best fit | How this project's focus differs |
|---|---|---|
| Promptfoo | Broad prompt/model evaluation and red teaming | `culprit-ai` centers comparisons on Git revisions and commit history. |
| Confident AI / DeepEval | Evaluation frameworks and hosted workflows | `culprit-ai` is a local Git wrapper with no required hosted backend. |
| promptvc | Prompt versioning workflows | `culprit-ai` leaves version storage to Git and adds eval-backed behavioral operations. |

## Installation

```bash
pip install culprit-ai
```

## Quickstart

```bash
cp examples/quickstart/culprit.yaml .
cp -r examples/quickstart/evals .
export OPENAI_API_KEY="..."
culprit diff HEAD~1 HEAD --eval-set evals/summarizer.yaml
```

## Contributing

Semantic diff, behavioral bisect, and experimental semantic merge improvements are open for contribution. See [CONTRIBUTING.md](CONTRIBUTING.md), including adapter guidance and suggested first issues.

Released under the [MIT License](LICENSE).
