Metadata-Version: 2.4
Name: ridi-check
Version: 0.3.2
Summary: See what a model or prompt update changed in an LLM assistant's answers, and which changes hold on re-run.
Author: ridi-check contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/adeebnoor/ridi-check
Project-URL: Documentation, https://github.com/adeebnoor/ridi-check/blob/main/docs/USER_GUIDE.md
Project-URL: Changelog, https://github.com/adeebnoor/ridi-check/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/adeebnoor/ridi-check/issues
Keywords: llm,evals,regression-testing,ai-agents,prompt-testing,rag,ci
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# ridi-check

**See what a model or prompt update really changed in your assistant's answers, before you ship it.**

[![tests](https://github.com/adeebnoor/ridi-check/actions/workflows/tests.yml/badge.svg)](https://github.com/adeebnoor/ridi-check/actions/workflows/tests.yml)
[![licence: MIT](https://raw.githubusercontent.com/adeebnoor/ridi-check/main/docs/img/badge-licence.svg)](https://github.com/adeebnoor/ridi-check/blob/main/LICENSE)
[![Python 3.10+](https://raw.githubusercontent.com/adeebnoor/ridi-check/main/docs/img/badge-python.svg)](https://github.com/adeebnoor/ridi-check/blob/main/pyproject.toml)

A new model or an edited prompt can change what your assistant or agent answers without anyone noticing:
average accuracy barely moves while individual answers flip. ridi-check runs the old and the new version on
the same questions, more than once, and shows which answers changed and which of those changes hold on
re-run. You need no test set, no answer key and no model judge.

![The report: where each case landed, and the change between versions against re-run variation](https://raw.githubusercontent.com/adeebnoor/ridi-check/main/docs/img/report.png)

[What you will see](#what-you-will-see) · [Who it is for](#who-it-is-for) · [Start in one minute](#start-in-one-minute) ·
[Use it in CI](#use-it-in-ci) · [Connect your assistant](#connect-your-assistant) · [FAQ](#faq) ·
[Roadmap](#roadmap) · [Limitations](#limitations) · [Reference](https://github.com/adeebnoor/ridi-check/blob/main/docs/REFERENCE.md)

## What you will see

**In your terminal or CI log**, `ridi-check gate` lists every answer that was right and is now wrong in
every run, and exits 1 so the pipeline stops:

```text
ridi-check gate: ridi-demo/ridi-out/results.csv, 30 cases, answer-key mode
stable_regression  case q02
stable_regression  case q09
stable_regression  case q18
stable_regression  case q25
counts: stable_regression=4 stable_improvement=1 regression_unconfirmed=0 improvement_unconfirmed=0 unstable=4 wrong_both=4 correct_both=17 incomplete=0
stable regressions: 4 (limit 0) - over the limit
gate: a limit was exceeded
```

**In the report**, each changed case shows the question, both answers, what the second run gave and the
passages only one version used:

![One case to review: a stable regression, both answers, and the passages that differ](https://raw.githubusercontent.com/adeebnoor/ridi-check/main/docs/img/report-case.png)

Every question ends in one group:

| Group | Meaning |
|---|---|
| Stable regression | right before, wrong after, in every run: an answer your update broke |
| Stable improvement | wrong before, right after, in every run |
| Stable change | no answer key: a different answer in every run, for you to judge |
| Unstable on re-run | a version disagreed with itself, so the difference is not evidence about the update |
| Wrong in both | no change, still wrong; listed separately |

The report is one self-contained HTML file. It works offline, on a phone, in light and dark, in English and
Arabic. The numbers above come from `ridi-check demo`, which uses synthetic data.

## Who it is for

- **You are about to swap the model, edit the prompt or change retrieval** and want to know which answers
  move, not only whether an average moved.
- **You have questions but no answer key.** The report shows what changed; you judge only the changes that
  held on re-run.
- **You want a CI check** that fails when an answer that was right becomes wrong in every run.
- **You only have answers in a spreadsheet.** The [browser version](https://github.com/adeebnoor/ridi-check/blob/main/web/ridi-update-check.html) reads
  pasted cells and needs no install.

It is **not** for proving that answers are correct or that a system is ready to deploy, for agents that use
tools or keep state between turns (not tested), or for explaining why an answer changed. See
[Limitations](#limitations) and the [use cases](https://github.com/adeebnoor/ridi-check/blob/main/docs/USE_CASES.md).

## Start in one minute

Python 3.10 or later, no dependencies. The demo needs no network and no API key; its data is synthetic.

```bash
pip install git+https://github.com/adeebnoor/ridi-check
```

```bash
ridi-check demo --open
```

Then check your own assistant:

```bash
ridi-check init my-check
```

It writes `questions.csv`, `my_assistant.py` (one function to fill in), `ridi-check.json` and
`ASK_YOUR_AI.md`, a prompt that lets your AI coding assistant do the wiring. In that folder, try five
questions with `ridi-check run --limit 5`, then run everything with `ridi-check run`. The
[user guide](https://github.com/adeebnoor/ridi-check/blob/main/docs/USER_GUIDE.md) walks through each step and how to read the report.

**No install at all:** download [web/ridi-update-check.html](https://github.com/adeebnoor/ridi-check/blob/main/web/ridi-update-check.html), open it and
paste your results (CSV text or cells copied from a spreadsheet). The page makes no network request.

## Use it in CI

```yaml
- run: pip install git+https://github.com/adeebnoor/ridi-check
- run: ridi-check run --config ridi-check.json     # calls both versions, writes ridi-out/results.csv
- run: ridi-check gate ridi-out/results.csv        # exits 1 on a stable regression
```

A complete GitHub Actions workflow to copy: [examples/ci/ridi-check.yml](https://github.com/adeebnoor/ridi-check/blob/main/examples/ci/ridi-check.yml).
`gate` prints counts and the offending cases; it never prints a verdict about your system. Limits such as
`--max-stable-regressions 2` are in the [reference](https://github.com/adeebnoor/ridi-check/blob/main/docs/REFERENCE.md#ridi-check-gate-resultscsv).

## Connect your assistant

ridi-check has no model provider built in. Your assistant is the system under test, reached through one
function:

```python
def answer(question: str) -> dict:
    return {"answer": "text", "context": ["doc-12", "doc-40"]}   # context is optional
```

| Your assistant is… | Pass it as |
|---|---|
| importable from Python | `python:module:function` |
| a program in any language | `command:<shell command>` (question on stdin, JSON or text on stdout) |
| an HTTP service | `http://…`, or a JSON file that describes the request and the reply |

Ready-made examples in [examples/adapters](https://github.com/adeebnoor/ridi-check/blob/main/examples/adapters/README.md): an OpenAI-compatible HTTP
API (no code), the Claude API through its official SDK, and a shell command with `curl`. Each reads its key
from an environment variable and is tested against a local stand-in server.

The two versions should differ only in what the update changes. Details:
[developer guide](https://github.com/adeebnoor/ridi-check/blob/main/docs/DEVELOPER_GUIDE.md#connecting-an-assistant).

## FAQ

**Do I need a test set or an answer key?** No. Without an `expected` column the report shows which answers
differ and you judge only the stable changes. With one, it speaks of regressions and improvements.

**Does my data go anywhere?** No. ridi-check calls only your assistant and writes local files. The report
and the browser version make no network request.

**Does it use a model to judge answers?** No. A judge adds its own error. Scoring is exact match after
normalisation, or a function you write.

**How many questions and runs?** 30 to 100 real user questions is a good first set. Two runs per version is
the minimum for the stable label; use more when the decision is expensive. Calls = questions × 2 × runs.

**Almost everything is "unstable" or "changed". Why?** Your assistant words the same answer differently each
time. Add `expected` answers with a scoring function, or use `--similarity 0.9`.

**Which providers and frameworks does it support?** Any. It talks to a function, a command or an HTTP
endpoint, not to a provider.

**Can I use it on an agent?** For its question-in, answer-out behaviour, yes. Agents that use tools or keep
state between turns have not been tested.

## Roadmap

No dates and no promises; this is what is planned and what is not.

- **Next:** a release on PyPI and a hosted link for the browser version.
- **Being considered:** intervals for the rates, multi-turn conversations, tool-using agents, more report
  languages.
- **Not planned:** model-based judging, or any hosted service that receives your data.

Tell us what you need in an [issue](https://github.com/adeebnoor/ridi-check/issues).

## Limitations

- It finds change, not correctness. A version that is wrong the same way every time looks stable, and
  cases wrong in both versions are listed separately.
- Without an answer key or your judgment, a difference is a change, not a regression.
- Passages are shown as they differ and are not graded; an ungraded passage can still carry the evidence.
- Two runs is a thin estimate of re-run variation. More runs make the stable label firmer.
- It is not a readiness or safety certificate. No output says a system is fit to deploy; the gate prints
  counts and the decision stays yours.
- Not in this version: agents that use tools or keep state, model-based judging, statistical tests.

## Documentation

- [User guide](https://github.com/adeebnoor/ridi-check/blob/main/docs/USER_GUIDE.md): step by step, and how to read the report
- [Use cases](https://github.com/adeebnoor/ridi-check/blob/main/docs/USE_CASES.md): twelve situations and how to set each one up
- [Developer guide](https://github.com/adeebnoor/ridi-check/blob/main/docs/DEVELOPER_GUIDE.md): adapters, files, CI, library use
- [Reference](https://github.com/adeebnoor/ridi-check/blob/main/docs/REFERENCE.md): every command and option, `results.csv`, the labelling rules
- بالعربية: [دليل الاستخدام](https://github.com/adeebnoor/ridi-check/blob/main/docs/USER_GUIDE.ar.md) · [حالات الاستخدام](https://github.com/adeebnoor/ridi-check/blob/main/docs/USE_CASES.ar.md) ·
  [دليل المطورين](https://github.com/adeebnoor/ridi-check/blob/main/docs/DEVELOPER_GUIDE.ar.md)

ridi-check follows the RIDI method.

## Contributing

Bug reports, questions and pull requests are welcome: see [CONTRIBUTING.md](https://github.com/adeebnoor/ridi-check/blob/main/CONTRIBUTING.md) and the
[changelog](https://github.com/adeebnoor/ridi-check/blob/main/CHANGELOG.md). Report a vulnerability privately, as [SECURITY.md](https://github.com/adeebnoor/ridi-check/blob/main/SECURITY.md)
describes.

## Licence

MIT. See [LICENSE](https://github.com/adeebnoor/ridi-check/blob/main/LICENSE).
