Metadata-Version: 2.4
Name: parametricbench
Version: 0.1.0
Summary: Catch LLM and VLM regressions with lightweight, parameterized benchmarks for any Python callable.
Author-email: "Eduardo J. Barrios" <edujbarrios@outlook.com>
License-Expression: MPL-2.0
Project-URL: Homepage, https://github.com/edujbarrios/parametricbench
Project-URL: Source, https://github.com/edujbarrios/parametricbench
Project-URL: Issues, https://github.com/edujbarrios/parametricbench/issues
Keywords: llm,vlm,benchmark,evaluation,testing,regression-testing,ai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# ParametricBench

[![PyPI](https://img.shields.io/pypi/v/parametricbench)](https://pypi.org/project/parametricbench/)
[![License: MPL 2.0](https://img.shields.io/badge/license-MPL%202.0-blue.svg)](https://github.com/edujbarrios/parametricbench/blob/main/LICENSE)

Catch LLM and VLM regressions by running the same evaluation cases across configurable prompts, models, and generation settings.

ParametricBench is a lightweight, provider-independent test runner for validating model behavior with plain Python. Use it to compare configurations, test prompt changes, validate text or multimodal responses, and run repeatable model checks locally or in CI.

## Installation

```bash
pip install parametricbench
```

## Usage

Run several checks in one benchmark while keeping case-specific expectations and settings together:

```python
from parametricbench import benchmark


def model(prompt, temperature=0):
    return prompt.upper()


def contains_expected(response, case):
    return case["expected"] in response


cases = [
    {"name": "uppercase-greeting", "input": "hello", "expected": "HELLO"},
    {
        "name": "uppercase-question",
        "input": "how are you?",
        "expected": "HOW ARE YOU?",
        "params": {"temperature": 0.2},
    },
    {
        "name": "contains-keyword",
        "input": "parametric benchmarks",
        "expected": "BENCHMARKS",
    },
]

report = benchmark(
    target=model,
    cases=cases,
    evaluators=[contains_expected],
    params={"temperature": 0},
)

print(f"Passed: {report['passed_count']}/{report['total']}")
for result in report["results"]:
    print(result["name"], result["passed"], result["score"])
```

## Why ParametricBench?

Model behavior can change when prompts, providers, models, retrieved context, or generation settings change. Manual checks are difficult to reproduce and easy to forget. ParametricBench turns them into reusable Python benchmarks for development, automated tests, production checks, and CI.

It is intentionally smaller than a complete evaluation platform:

* No provider lock-in or required model SDK
* No server, database, dashboard, or configuration language
* No runtime dependencies or infrastructure to deploy

Use it when an ad hoc script is too fragile but a full evaluation platform is unnecessary.

## Compare configurations

```python
report = benchmark(
    target=model,
    cases=cases,
    variants=[
        {"name": "deterministic", "params": {"temperature": 0}},
        {"name": "creative", "params": {"temperature": 0.8}},
    ],
    evaluators=[contains_expected],
)
```

Each case runs once with every named variant. Parameters are merged without mutation in this order: global, then variant, then case-specific parameters, with later values taking precedence.

## Custom evaluators

Evaluators receive the response and complete benchmark case:

```python
def exact_match(response, case):
    return response == case["expected"]
```

They may return a boolean or a numeric score from `0.0` to `1.0`. Multiple scores are averaged. Without evaluators, a successful target call scores `1.0`; a target error scores `0.0`. Target and evaluator errors are captured without stopping later cases.

## LLM and VLM inputs

Inputs can be any Python value:

```python
case = {
    "input": {"image": "traffic-light.png", "prompt": "Which object is shown?"},
    "expected": "traffic light",
}
```

ParametricBench passes the value to your target without loading or interpreting it.

## Use cases

* Detect regressions after prompt or model changes
* Compare model versions, providers, temperatures, or environments
* Validate structured, text, and multimodal responses
* Reuse the same checks in development, production, and CI

## Features

* Any Python callable and arbitrary inputs
* Global, variant, and case parameters
* Boolean or numeric evaluators and configurable thresholds
* Ordered dictionary reports with timing and concise errors
* No runtime dependencies

## Issues

Report issues at `https://github.com/edujbarrios/parametricbench`.

## Author

Eduardo J. Barrios — `edujbarrios@outlook.com`

## License

Mozilla Public License 2.0
