Metadata-Version: 2.4
Name: promptreg
Version: 0.1.0
Summary: Automated LLM prompt regression testing using local models
Author-email: H S Jayanth <jayanth.hs.05@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/jayanthhs05/promptreg
Project-URL: Repository, https://github.com/jayanthhs05/promptreg
Project-URL: Bug Tracker, https://github.com/jayanthhs05/promptreg/issues
Keywords: llm,prompt-engineering,regression-testing,evaluation,ollama,openai,anthropic,mlops,ai-safety
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: ollama
Requires-Dist: typer
Requires-Dist: rich
Requires-Dist: sqlite-utils<4.0,>=3.0
Requires-Dist: pyyaml
Provides-Extra: postgres
Requires-Dist: psycopg2-binary; extra == "postgres"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"

# 🎯 Prompt Regression Tracker (`promptreg`)

[![CI/CD Pipeline](https://github.com/jayanthhs05/promptreg/actions/workflows/ci.yml/badge.svg)](https://github.com/jayanthhs05/promptreg/actions)
[![Coverage Status](https://img.shields.io/badge/coverage-84%25-green.svg)](https://github.com/jayanthhs05/promptreg)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)](https://www.python.org/)
[![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![Type Checked: Mypy](https://img.shields.io/badge/types-mypy-blue.svg)](https://github.com/python/mypy)

`promptreg` is an enterprise-grade automated CLI tool designed to prevent quality regressions in LLM prompts. By integrating with local models (via Ollama) and cloud APIs (OpenAI/Anthropic), it systematically runs evaluation suites against pre-configured rubrics scored by a **judge model** (e.g., `phi3` or custom choices), alerting you if a prompt optimization or model update degrades generation quality.

*📊 **Visual Demos**: Explore the generated [Interactive HTML Report](file:///root/promptreg/demo_report.html) or review the structured [JSON Report Output](file:///root/promptreg/demo_report.json).*

---

## 🏗️ Architecture

```mermaid
graph TD
    A[CLI / CI Trigger] --> B[Test Case Loader]
    B --> C[Runner - Ollama / OpenAI / Anthropic]
    C --> D[Judge - LLM-as-a-Judge]
    D --> E[Database Layer - SQLite or Postgres]
    E --> F[Differ - Regression Checker]
    F --> G[Reporter - Terminal / JSON / HTML]
```

---

## ✨ Features

* **LLM-as-a-Judge Evaluation**: Scores responses from 0 to 10 against semantic rubrics using local or cloud models.
* **Dual Database Adapter**: Saves execution history, baselines, and runs to SQLite (local) or PostgreSQL (shared team database).
* **Parallel Execution**: Speeds up prompt testing via concurrent thread workers.
* **Golden Baselines**: Lock in your high-performance outputs to compare future changes against.
* **CI/CD Integration**: Returns non-zero exit codes on regressions for pre-commit hooks and Github workflows.
* **Premium Reporting**: Generates interactive HTML dashboards, structured JSON files, and beautiful terminal reports.

---

## 🚀 Getting Started

### 1. Install & Pull Ollama (Local LLM Support)
Download Ollama from [ollama.com](https://ollama.com) and pull your evaluation models:
```bash
ollama pull phi3
```

### 2. Install the Package
```bash
python3 -m venv venv
source venv/bin/activate
pip install -e ".[dev]"
```

---

## 📝 Writing Test Cases

Create YAML test files in `tests/suite/` (e.g., `tests/suite/summarize.yaml`):

```yaml
id: summarize_basic
description: Single-sentence summary of a news paragraph
model: phi3
prompt: |
  Summarize the following in one sentence:
  "The Federal Reserve raised interest rates by 0.25% on Wednesday,
  citing persistent inflation concerns despite signs of cooling in
  the labour market."
rubric:
  - must_include: ["Federal Reserve", "interest rates"]
  - max_words: 30
  - tone: neutral
```

---

## 🛠️ CLI Reference

### 1. Run the Evaluation Suite
Evaluate all test cases in the directory, compare outputs to baselines, and check for regressions:
```bash
promptreg run --suite tests/suite --threshold 1.5
```
*💡 Exits with code `1` if any regression >= threshold is detected.*

#### Command Options:
* `--suite PATH`: Directory containing test files (Default: `tests/suite`)
* `--threshold FLOAT`: Score drop threshold to trigger a regression (Default: `1.5`)
* `--db-url URL`: Postgres or SQLite database connection URL (Default: `promptreg.db`)
* `--judge-model MODEL`: Model to use for judging (e.g. `phi3`, `openai/gpt-4o`)
* `--concurrency INT`: Number of concurrent workers (Default: `4`)
* `--output-json PATH` / `--output-html PATH`: Export report paths

### 2. Promote Run to baseline
Mark the scores of a run as the benchmark golden baselines:
```bash
promptreg baseline <run_id>
```

### 3. Review Historical Scores
Display a tabular history of previous evaluation runs:
```bash
promptreg history [--test-id <test_id>] [--last 10]
```

### 4. Diff Two Runs Side-by-Side
Compare the scores of two specific runs:
```bash
promptreg diff <run_a> <run_b>
```

---

## 🤝 Git Integration (Pre-commit Hook)

Automate tests on commit by adding this to `.pre-commit-config.yaml`:

```yaml
repos:
  - repo: local
    hooks:
      - id: prompt-regression
        name: Prompt regression check
        entry: venv/bin/promptreg run
        language: system
        pass_filenames: false
        always_run: true
```

Install the hook:
```bash
pre-commit install
```

---

## 🔒 Centralized Team Operations & Security

When deploying `promptreg` in a team environment with a shared PostgreSQL database, you can secure write operations and track attribution using the built-in authentication and auditing system.

### 1. Database Configuration
Configure `promptreg` to use a central PostgreSQL database. This can be defined via CLI option `--db-url` or the `PROMPTREG_DATABASE_URL` environment variable:
```bash
export PROMPTREG_DATABASE_URL="postgresql://postgres:password@shared-db-host:5432/promptreg"
```

Alternatively, configure it inside `promptreg.yaml`:
```yaml
db_url: "postgresql://postgres:password@shared-db-host:5432/promptreg"
```

### 2. Token-Based Authentication
To prevent unauthorized users from overriding baseline benchmarks or committing runs to the shared database, enable token authentication in `promptreg.yaml`:
```yaml
auth:
  enabled: true
  admin_token: "your-team-secret-admin-token"
```

Or configure the expected token via the `PROMPTREG_ADMIN_TOKEN` environment variable on your database server.

#### Running Commands with Auth:
Once authentication is active, write operations (`run` and `baseline` promotion) require providing the token. Specify it via the `--auth-token` flag:
```bash
promptreg run --auth-token "your-team-secret-admin-token"
```

Or by setting the `PROMPTREG_AUTH_TOKEN` environment variable in your developer terminal or CI/CD workflow:
```bash
export PROMPTREG_AUTH_TOKEN="your-team-secret-admin-token"
promptreg baseline 42
```

### 3. Username Auditing
To identify which developer or pipeline triggered an evaluation or promoted a baseline, you can configure username attribution. `promptreg` tracks:
- `created_by` for each run
- `set_by` for each baseline

Specify your username via the `--username` CLI option or the `PROMPTREG_USERNAME` environment variable:
```bash
promptreg run --username "alice"
# or
export PROMPTREG_USERNAME="bob"
promptreg baseline 12
```

If not provided, `promptreg` automatically falls back to your OS login name.

---

## 🧪 Pytest Integration

`promptreg` exposes a built-in pytest plugin so you can run regression evaluations directly within your existing Python unit/integration tests.

When you install `promptreg`, the plugin is automatically registered. You can immediately inject the `prompt_regression` fixture.

### Example:

```python
# tests/test_prompts.py

def test_summarization_prompt_quality(prompt_regression):
    prompt = "Summarize the article..."
    model_output = call_your_llm_function(prompt)
    
    prompt_regression.assert_no_regression(
        test_id="summarize_basic",
        prompt=prompt,
        output=model_output,
        rubric=[
            {"must_include": ["Federal Reserve"]},
            {"max_words": 30},
            "tone should be objective"
        ],
        model="gpt-4o"
    )
```

Running `pytest` will execute the evaluation, track the score history in the `promptreg` database, compare results against baseline, and trigger failures if a quality regression occurs.


