Metadata-Version: 2.4
Name: promptreg
Version: 0.1.1
Summary: Automated LLM prompt regression testing using local models
Author-email: H S Jayanth <jayanth.hs.05@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/jayanthhs05/promptreg
Project-URL: Repository, https://github.com/jayanthhs05/promptreg
Project-URL: Bug Tracker, https://github.com/jayanthhs05/promptreg/issues
Keywords: llm,prompt-engineering,regression-testing,evaluation,ollama,openai,anthropic,mlops,ai-safety
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: ollama
Requires-Dist: typer
Requires-Dist: rich
Requires-Dist: sqlite-utils<4.0,>=3.0
Requires-Dist: pyyaml
Provides-Extra: postgres
Requires-Dist: psycopg2-binary; extra == "postgres"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"

# Prompt Regression Tracker (promptreg)

[![CI/CD Pipeline](https://github.com/jayanthhs05/promptreg/actions/workflows/ci.yml/badge.svg)](https://github.com/jayanthhs05/promptreg/actions)
[![Coverage Status](https://img.shields.io/badge/coverage-84%25-green.svg)](https://github.com/jayanthhs05/promptreg)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)](https://www.python.org/)
[![Code Style: Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![Type Checked: Mypy](https://img.shields.io/badge/types-mypy-blue.svg)](https://github.com/python/mypy)

`promptreg` is an enterprise-grade automated CLI tool and testing framework designed to prevent quality regressions in Large Language Model (LLM) prompts. By integrating with local models (via Ollama) and cloud APIs (OpenAI/Anthropic), it systematically executes evaluation suites against pre-configured rubrics. Using an LLM-as-a-Judge architecture, `promptreg` alerts you immediately if a prompt optimization or model update degrades response quality.

**Visual Demos**: Explore the generated [Interactive HTML Report](file:///root/promptreg/demo_report.html) or review the structured [JSON Report Output](file:///root/promptreg/demo_report.json).

---

## Architecture

```mermaid
graph TD
    A[CLI / CI Trigger] --> B[Test Case Loader]
    B --> C[Runner - Ollama / OpenAI / Anthropic]
    C --> D[Judge - LLM-as-a-Judge]
    D --> E[Database Layer - SQLite or Postgres]
    E --> F[Differ - Regression Checker]
    F --> G[Reporter - Terminal / JSON / HTML]
```

---

## Key Features

- **LLM-as-a-Judge Evaluation**: Scores responses quantitatively (0 to 10) against semantic rubrics utilizing either local or cloud models.
- **Dual Database Adapter**: Persists execution history, benchmarks, and historical runs to SQLite for local development or PostgreSQL for shared team environments.
- **Parallel Execution**: Accelerates prompt testing via concurrent thread workers for scalable evaluation.
- **Golden Baselines**: Establishes high-performance output benchmarks to continually compare future iterations against.
- **CI/CD Integration**: Supports pre-commit hooks and GitHub Actions with non-zero exit codes upon regression detection.
- **Premium Reporting**: Generates interactive HTML dashboards, structured JSON files, and comprehensive terminal output.

---

## Getting Started

### 1. Model Provisioning (Local Execution)
To utilize local models, download and install Ollama from [ollama.com](https://ollama.com). Pull the necessary evaluation models before execution:
```bash
ollama pull phi3
```

### 2. Installation
Initialize a virtual environment and install the package with development dependencies:
```bash
python3 -m venv venv
source venv/bin/activate
pip install -e ".[dev]"
```

---

## Writing Test Cases

Define test cases using YAML syntax within the `tests/suite/` directory (e.g., `tests/suite/summarize.yaml`):

```yaml
id: summarize_basic
description: Single-sentence summary of a news paragraph
model: phi3
prompt: |
  Summarize the following in one sentence:
  "The Federal Reserve raised interest rates by 0.25% on Wednesday,
  citing persistent inflation concerns despite signs of cooling in
  the labour market."
rubric:
  - must_include: ["Federal Reserve", "interest rates"]
  - max_words: 30
  - tone: neutral
```

---

## Command Line Interface (CLI) Reference

### 1. Evaluation Suite Execution
Evaluate all test cases within a targeted directory, compare outputs to established baselines, and detect regressions:
```bash
promptreg run --suite tests/suite --threshold 1.5
```
*Note: Exits with code `1` if any regression exceeds or equals the defined threshold.*

**Options:**
- `--suite PATH`: Directory containing test files (Default: `tests/suite`)
- `--threshold FLOAT`: Score drop threshold to trigger a regression (Default: `1.5`)
- `--db-url URL`: Postgres or SQLite database connection URL (Default: `promptreg.db`)
- `--judge-model MODEL`: Model utilized for evaluation logic (e.g. `phi3`, `openai/gpt-4o`)
- `--concurrency INT`: Number of concurrent workers (Default: `4`)
- `--output-json PATH` / `--output-html PATH`: Export report destinations

### 2. Baseline Promotion
Promote the scores of a specific run to serve as the benchmark golden baseline:
```bash
promptreg baseline <run_id>
```

### 3. Historical Data Review
Display a tabular history of preceding evaluation runs:
```bash
promptreg history [--test-id <test_id>] [--last 10]
```

### 4. Differential Comparison
Perform a side-by-side comparison of the scores of two specific runs:
```bash
promptreg diff <run_a> <run_b>
```

---

## CI/CD and Pre-commit Integration

Automate testing pipelines by adding `promptreg` to your `.pre-commit-config.yaml`:

```yaml
repos:
  - repo: local
    hooks:
      - id: prompt-regression
        name: Prompt regression check
        entry: venv/bin/promptreg run
        language: system
        pass_filenames: false
        always_run: true
```

Install the hook locally:
```bash
pre-commit install
```

---

## Team Operations & Security

For enterprise deployments utilizing a shared PostgreSQL database, `promptreg` includes built-in authentication and auditing systems to secure write operations.

### 1. Database Configuration
Configure a centralized PostgreSQL database via the `--db-url` CLI option or the `PROMPTREG_DATABASE_URL` environment variable:
```bash
export PROMPTREG_DATABASE_URL="postgresql://postgres:password@shared-db-host:5432/promptreg"
```

Alternatively, specify it within `promptreg.yaml`:
```yaml
db_url: "postgresql://postgres:password@shared-db-host:5432/promptreg"
```

### 2. Token-Based Authentication
To restrict unauthorized modifications to baseline benchmarks or test results, enable token authentication in `promptreg.yaml`:
```yaml
auth:
  enabled: true
  admin_token: "your-team-secret-admin-token"
```

**Executing Authenticated Commands:**
Write operations (`run` and `baseline` promotion) mandate token validation when authentication is active. Provide the token via the `--auth-token` flag:
```bash
promptreg run --auth-token "your-team-secret-admin-token"
```
Or via the `PROMPTREG_AUTH_TOKEN` environment variable:
```bash
export PROMPTREG_AUTH_TOKEN="your-team-secret-admin-token"
promptreg baseline 42
```

### 3. User Auditing
Maintain auditability by tracking the identity of the developer or CI pipeline executing operations. `promptreg` tracks:
- `created_by` for each execution run
- `set_by` for baseline promotions

Specify the identity using the `--username` CLI option or the `PROMPTREG_USERNAME` environment variable:
```bash
promptreg run --username "alice"
# or
export PROMPTREG_USERNAME="bob"
promptreg baseline 12
```
If omitted, `promptreg` automatically falls back to the active operating system login name.

---

## Pytest Integration

`promptreg` exposes a native `pytest` plugin to seamlessly execute regression evaluations within standard Python test suites.

Installation automatically registers the plugin. The `prompt_regression` fixture can be immediately injected into test cases.

### Implementation Example:

```python
# tests/test_prompts.py

def test_summarization_prompt_quality(prompt_regression):
    prompt = "Summarize the article..."
    model_output = call_your_llm_function(prompt)
    
    prompt_regression.assert_no_regression(
        test_id="summarize_basic",
        prompt=prompt,
        output=model_output,
        rubric=[
            {"must_include": ["Federal Reserve"]},
            {"max_words": 30},
            "tone should be objective"
        ],
        model="gpt-4o"
    )
```

Executing `pytest` will run the evaluation, log historical scoring data, compare the results against the designated baseline, and force test failure upon quality regression detection.
