Metadata-Version: 2.4
Name: promptproof-cli
Version: 0.2.0
Summary: Snapshot & regression testing for LLM prompts — YAML test suites, CI-friendly assertions, any OpenAI-compatible endpoint.
Author: Dylan XM
License: MIT
Project-URL: Homepage, https://github.com/dylanXM/promptproof
Project-URL: Repository, https://github.com/dylanXM/promptproof
Project-URL: Issues, https://github.com/dylanXM/promptproof/issues
Keywords: llm,prompt,testing,regression,evaluation,ci
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# PromptProof

**Snapshot & regression testing for LLM prompts.**

Write your prompts and test cases as YAML, run one command against any
OpenAI-compatible endpoint, and get a CI-friendly pass/fail report. When a
case fails, your build goes red — before your users notice the model started
behaving differently.

```console
$ promptproof run examples/basic.yaml --mock

PromptProof — suite: sentiment-classifier
provider: mock   model: mock
------------------------------------------------------------------------
CASE            RESULT      TIME  TOKENS
----------------------------------------
glowing review  PASS         0ms      66
angry review    PASS         0ms      63
factual review  PASS         0ms      59
----------------------------------------
3 passed, 0 failed, 3 total — 188 tokens (estimated)
```

## Why PromptProof?

Prompts are code, but they fail like weather. You swap a model version, tweak
one sentence of a system prompt, or your provider silently updates a model —
and the output format your parser depends on quietly changes. Nobody notices
until production does.

PromptProof treats a prompt the way unit tests treat a function:

- **Suites are files.** Prompts, variables and expectations live in YAML/JSON
  next to your code, reviewed in pull requests like everything else.
- **Assertions, not vibes.** `contains`, `regex`, `json_schema`, `max_chars`
  — deterministic checks that mean the same thing on every run. For the
  things patterns can't judge (tone, helpfulness, "did it actually answer
  the question?"), an `llm_judge` assertion lets a second model grade the
  response against a criterion you state in plain language.
- **Built for CI.** Plain-text table for humans, JSON report for machines,
  a self-contained HTML report for sharing (`--html report.html`, no assets
  needed), exit code `1` when any case fails so GitHub Actions turns red.
- **No key needed to try it.** A built-in mock provider runs any suite
  offline, so a fresh clone works in 30 seconds and CI never needs secrets
  just to test the harness.

## Quick start (offline demo)

```bash
pip install promptproof-cli        # or: pip install -e . from a clone
promptproof run examples/basic.yaml --mock
promptproof run examples/basic.yaml --mock --json report.json --html report.html
```

That runs the bundled sentiment-classifier suite end-to-end — rendering,
"model" calls, assertions, report — with zero API keys and zero network.

## Running against a real endpoint

```bash
export OPENAI_API_KEY=sk-...
promptproof run examples/basic.yaml
```

PromptProof speaks the OpenAI chat-completions API, so `base_url` can point
at OpenAI, Azure OpenAI, OpenRouter, Together, Groq, or a local server
(Ollama, LM Studio, …) — anything compatible. The API key is read from the
environment variable named by `api_key_env` in the suite; keys are never
written into suite files or reports.

Exit codes: `0` all cases passed · `1` a case failed · `2` the suite or
configuration is invalid.

## Suite files

```yaml
name: sentiment-classifier

model:
  base_url: https://api.openai.com/v1   # any OpenAI-compatible endpoint
  name: gpt-4o-mini
  api_key_env: OPENAI_API_KEY           # env var holding the key
  temperature: 0

prompt: |
  Classify the sentiment of this review. Respond with JSON only.
  Review: {{review}}

# Only used by --mock runs: deterministic offline responses.
mock:
  default: '{"sentiment": "neutral"}'
  rules:
    - match: "absolutely loved"        # substring, or /regex/
      response: '{"sentiment": "positive"}'

cases:
  - name: glowing review
    vars: {review: "I absolutely loved this keyboard."}
    assertions:
      - contains: '"positive"'
      - max_chars: 100
      - json_schema:
          type: object
          required: [sentiment]
          properties:
            sentiment: {type: string, enum: [positive, negative, neutral]}
```

Templates support `{{variable}}` placeholders only — a suite is data, not
code. A case may set `mock_response` to override the mock rules for that
case. Suites can also be plain JSON (same structure).

## Assertions

| Assertion      | Example                                   | Passes when…                                             |
| -------------- | ----------------------------------------- | -------------------------------------------------------- |
| `contains`     | `contains: '"positive"'`                 | the response contains the substring                      |
| `not_contains` | `not_contains: "I don't know"`            | the response does *not* contain the substring            |
| `regex`        | `regex: '"id":\s*\d+'`                    | the pattern matches somewhere in the response            |
| `json_schema`  | `json_schema: {type: object, required: [sentiment]}` | the response parses as JSON and validates against the schema |
| `max_chars`    | `max_chars: 500`                          | the response is at most N characters                     |
| `min_chars`    | `min_chars: 50`                           | the response is at least N characters                    |
| `equals`       | `equals: "yes"`                           | the response matches exactly (whitespace-trimmed)        |
| `starts_with`  | `starts_with: "Summary:"`                 | the response begins with the literal prefix              |
| `ends_with`    | `ends_with: "done."`                      | the response ends with the literal suffix                |
| `llm_judge`    | `llm_judge: "the response is polite"`     | a judge model decides the response meets the criterion  |

`json_schema` uses a small built-in validator covering the subset prompt
tests actually use — `type`, `required`, `properties`, `items`, `enum`,
`minimum`/`maximum`, `minLength`/`maxLength`, `pattern` — so PromptProof
needs no heavy dependencies. Assertions may also be written in the explicit
form `{type: contains, value: "x"}`.

### `llm_judge`: grading with a second model

Some expectations can't be a pattern: *the tone is friendly*, *the answer
addresses the question that was asked*, *no medical advice is given*. State
them as a criterion and let a judge model grade the response:

```yaml
cases:
  - name: support reply
    assertions:
      - llm_judge: "the response is polite and actually answers the question"
      # mapping form, with a per-assertion model override:
      - llm_judge: {criterion: "the response gives no medical advice", model: gpt-4o}
```

The judge endpoint is configured with environment variables only (keys
never go in suite files):

```bash
export PROMPTPROOF_JUDGE_BASE_URL=https://api.openai.com/v1  # any compatible endpoint
export PROMPTPROOF_JUDGE_MODEL=gpt-4o-mini
export PROMPTPROOF_JUDGE_API_KEY=sk-...
```

The judge's verdict and its one-sentence reason land in the report like any
other assertion (`"judge: polite and on-topic"`). If the endpoint is **not**
configured, `llm_judge` assertions report as *skipped* instead of failing —
suites keep running offline, under `--mock`, and in CI without secrets. A
configured judge that errors or answers unparseably fails the assertion
with the reason shown; it never crashes the run.

## GitHub Actions

```yaml
- name: Prompt regression tests
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: |
    pip install promptproof-cli
    promptproof run tests/prompts/checkout.yaml --json prompt-report.json
```

A failing case exits `1` and fails the step. For pull requests that only
touch prompt files, run with `--mock` to validate the suite itself without
spending tokens or requiring secrets from forks.

## Roadmap

- Golden-file snapshots: record real responses once, diff against them later
- Multi-turn conversation suites
- Cost budgets per suite (`max_tokens` / `max_usd` gates)
- ~~HTML report output~~ — done: `promptproof run suite.yaml --html report.html`

## Contributing

Issues and pull requests are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
Please keep the dependency footprint tiny and the offline demo working.

## License

MIT © 2026 Dylan XM — see [LICENSE](LICENSE).
