Metadata-Version: 2.4
Name: evalgate-cli
Version: 0.2.0
Summary: Decide whether an eval delta is a real regression or sampling noise.
Project-URL: Homepage, https://github.com/jmweb-org/evalgate
Project-URL: Repository, https://github.com/jmweb-org/evalgate
Project-URL: Issues, https://github.com/jmweb-org/evalgate/issues
Author: José del Río
License: MIT License
        
        Copyright (c) 2026 José del Río
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: ci,cli,evaluation,mcnemar,mlops,significance,statistics
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Description-Content-Type: text/markdown

# evalgate

[![CI](https://github.com/jmweb-org/evalgate/actions/workflows/ci.yml/badge.svg)](https://github.com/jmweb-org/evalgate/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/evalgate-cli.svg)](https://pypi.org/project/evalgate-cli/)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Decide whether an eval delta is a real regression or just sampling noise, and
fail CI only when it is real.

A model eval that drops from 90.0% to 89.4% on a 1,000-example set looks like a
regression, but on that sample size it is noise. Gating CI on the raw number
makes the build flap; ignoring it lets real regressions through. `evalgate`
runs the appropriate significance test and fails only when the candidate is
significantly worse.

```console
$ evalgate proportions \
    --baseline-score 0.900 --baseline-n 1000 \
    --candidate-score 0.894 --candidate-n 1000
verdict     worse, but within noise
difference  -0.0060
p-value     0.6232
alpha       0.05
# exit code 0 -> build passes
```

## Install

```console
$ pip install evalgate-cli                 # from PyPI, once released
$ pip install git+https://github.com/jmweb-org/evalgate   # latest, available now
```

Pure standard library plus typer and rich. No heavy dependencies.

## Usage

### From two aggregate accuracies

```console
$ evalgate proportions \
    --baseline-score 0.90 --baseline-n 2000 \
    --candidate-score 0.87 --candidate-n 2000 \
    --alpha 0.05
```

Uses a two-proportion z-test on the accuracies and their sample sizes.

### From paired per-example results

When both models were evaluated on the same examples, a paired test is more
powerful. Give a CSV with per-example correctness (or predictions plus a truth
column):

```console
$ evalgate paired results.csv --baseline base_correct --candidate cand_correct
$ evalgate paired results.csv --baseline pred_a --candidate pred_b --truth label
```

Uses McNemar's test: an exact binomial test on the discordant pairs, or the
continuity-corrected chi-squared approximation for large samples.

### In CI

```yaml
- run: evalgate proportions --baseline-score 0.90 --baseline-n 2000
        --candidate-score "$SCORE" --candidate-n 2000
```

## Verdicts and exit codes

| Verdict | Meaning | Exit |
| --- | --- | --- |
| `improvement` | Candidate is better | 0 |
| `unchanged` | No measurable difference | 0 |
| `noise` | Worse, but not significant at `alpha` | 0 |
| `regression` | Significantly worse | 1 |

A bad invocation (scores out of range, missing column, unreadable file) exits 2.
## JSON output

When using the `--json` flag, `evalgate` returns a JSON object with the following fields:

| Field                    | Type      | Description                                                                                             |
| ------------------------ | --------- | ------------------------------------------------------------------------------------------------------- |
| `verdict`                | `string`  | One of `improvement`, `unchanged`, `noise`, or `regression`.                                            |
| `p_value`                | `float`   | The p-value from the statistical test.                                                                  |
| `difference`             | `float`   | Signed difference between candidate and baseline. A negative value means the candidate performed worse. |
| `alpha`                  | `float`   | Significance threshold used for the test.                                                               |
| `is_regression`          | `boolean` | `true` if the result is a statistically significant regression; otherwise `false`.                      |
| `test`                   | `string`  | Statistical test used (`two_proportion_z` or `mcnemar`).                                                |
| `baseline_only_correct`  | `integer` | *(Paired mode only)* Number of examples only the baseline answered correctly.                           |
| `candidate_only_correct` | `integer` | *(Paired mode only)* Number of examples only the candidate answered correctly.                          |


## What it does and does not do

It answers one question: is this difference larger than sampling variation?
It does not correct for multiple comparisons across many evals, and a
`noise` verdict means "not proven", not "proven equal". For small evals,
collect more examples rather than trusting a borderline p-value.

## License

MIT. See [LICENSE](LICENSE).
