Metadata-Version: 2.4
Name: promptdiff-cli
Version: 0.1.1
Summary: Git for prompts - version, diff, and test your LLM prompts across any model
Author: Raguram R
License-Expression: MIT
Project-URL: Homepage, https://github.com/Ragu3175/promptdiff
Project-URL: Repository, https://github.com/Ragu3175/promptdiff
Project-URL: Issues, https://github.com/Ragu3175/promptdiff/issues
Keywords: llm,prompt-engineering,prompt-versioning,cli,litellm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Version Control
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.110.0
Requires-Dist: uvicorn>=0.28.0
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: alembic>=1.13.0
Requires-Dist: typer>=0.9.0
Requires-Dist: rich>=13.7.0
Requires-Dist: litellm>=1.30.0
Requires-Dist: python-dotenv>=1.0.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Dynamic: license-file

# PromptDiff

**Git for prompts.** Version your LLM prompts, diff them word-by-word, and run any version against Gemini, Groq, or any other LiteLLM-supported model — all from the terminal.

```bash
promptdiff commit summarizer "Summarize: {text}" -m "v1: baseline"
promptdiff commit summarizer "Summarize concisely in one sentence: {text}" -m "v2: tighter constraint"
promptdiff diff summarizer 1 2
promptdiff run summarizer 2 --model gemini/gemini-2.5-flash
```

## Why PromptDiff

If you've ever overwritten a prompt and lost the version that actually worked, or pasted two prompt drafts into a doc just to eyeball what changed — PromptDiff is the tool for that. It treats prompts as versioned, diffable artifacts the same way Git treats code, and adds one thing Git can't: running any version against a real LLM and recording exactly what it produced, so you can trace output back to the exact prompt and model that generated it.

## How it's different from other prompt tools

The prompt-tooling space has grown a lot — [Promptfoo](https://github.com/promptfoo/promptfoo) is a mature, YAML-config-based open-source tool with strong CI/eval integration, and platforms like PromptLayer, Langfuse, and Braintrust offer hosted, full-lifecycle prompt management with branching, RBAC, and observability.

PromptDiff isn't trying to compete with those. It's deliberately smaller: a local-first CLI for a single developer iterating on prompts, with zero config files, zero hosted accounts, and zero cost beyond free-tier LLM API usage. If you outgrow it — multiple collaborators, non-engineer prompt editors, compliance requirements — those other tools are the right next step. If you're a solo developer who just wants `git commit`-style version control for prompts without standing up infrastructure, PromptDiff is built for exactly that.

## Features

- **Version control for prompts** — every `commit` is an immutable, numbered version (v1, v2, v3...) scoped per prompt
- **Word-level diffs** — see exactly which words changed between versions, rendered with color in your terminal (green additions, red strikethrough removals)
- **Multi-model runs** — execute any prompt version against multiple LLMs in a single command and compare cost, latency, and output side by side
- **Output comparison** — `diff-output` shows what two versions actually *produced*, not just what their text looks like
- **Free-tier friendly** — built and tested against Gemini and Groq's free tiers; rate-limit and quota failures are recorded gracefully, never crash the tool
- **Local-first** — everything lives in a local SQLite database, no account or server required

## Installation

```bash
pip install promptdiff-cli
```

(Or, to run from source — see [Development](#development) below.)

No API key is required to install or to use `commit`, `log`, and `diff`. You'll only need one for `run` and `diff-output`, since those actually call a model.

## Quick start

`commit`, `log`, and `diff` work immediately after install — no API key needed. You'll only need a key for the `run` and `diff-output` commands, which actually call a model. See [API keys](#api-keys) below.

```bash
# Create a project and switch to it
promptdiff project create my-app
promptdiff use my-app

# Create a prompt and commit your first version
promptdiff add summarizer
promptdiff commit summarizer "Summarize this in one sentence: {text}" -m "v1: baseline"

# Edit and commit a new version
promptdiff commit summarizer "Summarize concisely, max 20 words: {text}" -m "v2: tighter constraint"

# See version history
promptdiff log summarizer

# See exactly what changed
promptdiff diff summarizer 1 2

# Run a version against a model (needs an API key — see below)
promptdiff run summarizer 2 --model gemini/gemini-2.5-flash

# Run the same version against multiple models at once
promptdiff run summarizer 2 --model gemini/gemini-2.5-flash --model groq/llama-3.3-70b-versatile

# Compare what two versions actually produced
promptdiff diff-output summarizer 1 2 --model gemini/gemini-2.5-flash
```

## API keys

PromptDiff uses [LiteLLM](https://github.com/BerriAI/litellm) under the hood, so it works with any LiteLLM-supported provider. It's built and tested primarily against free tiers:

- **Gemini** — get a free key at [aistudio.google.com/apikey](https://aistudio.google.com/apikey)
- **Groq** — get a free key at [console.groq.com/keys](https://console.groq.com/keys)

Copy `.env.example` to `.env` in your project directory and fill in the keys you have:

```
GEMINI_API_KEY=your_key_here
GROQ_API_KEY=your_key_here
```

## Commands

| Command | What it does |
|---|---|
| `promptdiff project create <name>` | Create a new project |
| `promptdiff project list` | List all projects |
| `promptdiff use <project>` | Set the current project (so you don't need `--project` on every command) |
| `promptdiff add <prompt>` | Add a new prompt to the current project |
| `promptdiff commit <prompt> <content_or_file> -m "<message>"` | Commit a new version. Accepts a literal string or a path to a file |
| `promptdiff log <prompt>` | Show version history |
| `promptdiff diff <prompt> <v1> <v2>` | Word-level diff between two versions |
| `promptdiff run <prompt> <version> --model <model>` | Run a version against one or more models (repeat `--model` to run against several at once) |
| `promptdiff diff-output <prompt> <v1> <v2> [--model <model>]` | Compare what two versions actually produced. If a version was run against multiple models, `--model` is required to avoid an ambiguous comparison |

Every command accepts `-p` / `--project` to override the current project for that one call.

## Architecture

- **Database**: SQLite, 5 tables (`projects`, `prompts`, `versions`, `runs`, `outputs`) via SQLAlchemy + Alembic migrations
- **Diff engine**: Python's `difflib`, line-level structure with word-level precision inside changed lines
- **Runner**: [LiteLLM](https://github.com/BerriAI/litellm) for provider-agnostic model calls; failures (rate limits, timeouts, bad keys) are recorded as failed runs rather than crashing the CLI
- **CLI**: [Typer](https://typer.tiangolo.com/) + [Rich](https://github.com/Textualize/rich)

## Development

```bash
git clone https://github.com/Ragu3175/promptdiff.git
cd promptdiff
python -m venv venv
venv\Scripts\activate   # Windows
# source venv/bin/activate   # macOS/Linux
pip install -r requirements.txt
pip install -e .
pytest -v
```

## Status

Core CLI is complete and tested: `commit`, `log`, `diff`, `run`, and `diff-output` all work end-to-end against real Gemini and Groq calls. A REST API layer and web UI are planned but not yet built — the CLI is fully usable on its own today.

## Contributing

Issues and PRs welcome. This is an early-stage solo project — if you run into something broken or have an idea, please open an issue.

## License

MIT — see [LICENSE](LICENSE).

## Author

Built by [Raguram R](https://github.com/Ragu3175).
