Metadata-Version: 2.5
Name: flakerate
Version: 0.1.0
Summary: Measure how flaky your tests actually are, find out why, and quarantine them with an expiry date.
Project-URL: Homepage, https://github.com/aviseth/flakerate
Project-URL: Repository, https://github.com/aviseth/flakerate
Project-URL: Changelog, https://github.com/aviseth/flakerate/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/aviseth/flakerate/issues
Author-email: Avi Seth <avi@crispa.ai>
License-Expression: MIT
License-File: LICENSE
Keywords: ci,flakiness,flaky,pytest,quarantine,test-reliability,tests
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: tomli-w>=1.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Description-Content-Type: text/markdown

# flakerate

Measure how flaky a test actually is, find out why, and quarantine it with an expiry date.

[![PyPI](https://img.shields.io/pypi/v/flakerate.svg)](https://pypi.org/project/flakerate/)
[![Python](https://img.shields.io/pypi/pyversions/flakerate.svg)](https://pypi.org/project/flakerate/)
[![CI](https://github.com/aviseth/flakerate/actions/workflows/ci.yml/badge.svg)](https://github.com/aviseth/flakerate/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

The usual response to a flaky test is to retry it until it passes. That turns the build green and
throws away the only evidence there was. You never learn whether it fails one run in fifty or one
in three, and you never learn why, so it stays in the suite getting slowly worse.

flakerate runs the suite repeatedly, reports a failure rate with a confidence interval, diagnoses
the cause by experiment, and writes a quarantine file that expires.

## Installation

```shell
pip install flakerate
```

Requires Python 3.10+ and pytest 8+.

## Quick start

```shell
flakerate run -n 20 -- tests
```

```text
20 runs, 412 tests, 3 flaky

failed  rate  95% range  cause  test
------  ----  ---------  -----  ---------------------------------------------
  7/20   35%    18%-57%         tests/test_orders.py::test_cancel_refunds
  6/20   30%    15%-52%         tests/test_api.py::test_rate_limit_headers
  1/20    5%     1%-24%         tests/test_sync.py::test_backfill_idempotent
```

Every run gets a different shuffle seed, because a suite that always runs in the same order will
never show you an order-dependent failure, and that is the most common kind.

The range is a Wilson score interval and it is the number to make decisions against. Note the last
row: one failure in twenty runs is anywhere from 1% to 24%. That is not a measurement yet, it is a
signal to run it more times. Wilson rather than the textbook normal approximation because the
counts are small and the rates sit near zero, which is exactly where the normal approximation
returns a negative lower bound.

## Diagnosing the cause

```shell
flakerate run -n 30 --classify -- tests
```

```text
failed  rate  95% range  cause             test
------  ----  ---------  ----------------  ---------------------------------------------
  9/30   30%    17%-48%  order-dependent   tests/test_orders.py::test_cancel_refunds
  8/30   27%    14%-45%  environment       tests/test_api.py::test_rate_limit_headers

tests/test_orders.py::test_cancel_refunds
  in suite: 9/30 failed, alone: 0/5 failed, fixed order: 0/3 failed
  passes on its own, so something earlier in the suite leaves state behind. Look for module-level
  globals, a cached singleton, an unclosed database transaction, or a fixture with a wider scope
  than it needs.
```

Classification runs experiments rather than pattern-matching tracebacks.

| Experiment | What it answers |
| --- | --- |
| Run the test alone, several times | Is the rest of the suite involved at all? |
| Run the whole suite, several times, in one fixed order | Is the order the variable, or something else? |

| Cause | What it means |
| --- | --- |
| `order-dependent` | Passes alone. Something that runs before it leaves state behind. |
| `nondeterministic` | Fails alone too. An unseeded random source, a real clock, dict ordering, a thread. |
| `environment` | Fails alone, and the failures look like they come from outside the process. |
| `unclear` | The test could not be run on its own, so no conclusion is offered. |

Failure messages only ever refine a conclusion the experiments already reached. A traceback
mentioning a socket is suggestive, not proof, and it is never the sole basis for a verdict.

The isolation experiment is per test and cheap. The fixed-order runs are shared across every flaky
test in the report, so classifying ten of them costs the same suite runs as classifying one.

## Quarantine

```shell
flakerate run -n 30 --classify --quarantine --days 14 -- tests
```

Writes `flakerate.toml`:

```toml
[quarantine."tests/test_orders.py::test_cancel_refunds"]
reason = "tests/test_orders.py:88: AssertionError"
rate = 0.3
cause = "order-dependent"
added = "2026-08-24"
expires = "2026-09-07"
```

Commit it, then in CI:

```yaml
- run: pip install flakerate
- run: flakerate check
- run: FLAKERATE_QUARANTINE=flakerate.toml pytest
```

Quarantined tests still run and still report. They are marked non-strict xfail, so one that starts
passing again shows up as an xpass rather than silently becoming load-bearing.

An entry with no expiry date, or one that cannot be read, counts as expired. Failing the other
way would make `expires = "soon"` a quarantine nobody ever sees the end of, which is the thing the
date exists to prevent.

`flakerate check` fails the build once an entry is past its date:

```text
$ flakerate check
fail 2 quarantine entry/entries have expired

expired     cause             test
----------  ----------------  ---------------------------------------------
2026-09-07  order-dependent   tests/test_orders.py::test_cancel_refunds
2026-09-07  environment       tests/test_api.py::test_rate_limit_headers

Fix the test, or renew it on purpose with 'flakerate run --quarantine'.
```

That date is the point. Quarantine without one is deletion with extra steps, and every codebase
that has one has tests in it nobody has looked at in three years.

## Configuration

All keys live under `[tool.flakerate]` in `pyproject.toml`. All are optional.

| Key | Type | Default | Meaning |
| --- | --- | --- | --- |
| `runs` | integer | `20` | How many times `run` repeats the suite |
| `quarantine` | string | `"flakerate.toml"` | Path to the quarantine file |
| `pytest_args` | list of strings | `[]` | Arguments used when none are given on the command line |
| `max_flake_rate` | number, 0 to 1 | `0.0` | Reserved for a future rate-based gate |

Values are type-checked rather than coerced, so `max_flake_rate = true` is an error rather than
quietly becoming 1.0.

## Command reference

| Command | What it does |
| --- | --- |
| `flakerate run -n N -- <pytest args>` | Run the suite N times and report flake rates |
| `flakerate run --classify` | Also diagnose the cause of each flaky test |
| `flakerate run --quarantine --days N` | Write findings to the quarantine file |
| `flakerate run --no-shuffle` | Keep pytest's own order |
| `flakerate check` | Exit non-zero if a quarantine entry has expired |
| `flakerate list` | Show what is quarantined and how long it has left |

Every command takes `--json`.

## pytest options

| Option | Environment variable | Effect |
| --- | --- | --- |
| `--flakerate-report PATH` | `FLAKERATE_REPORT` | Write a JSON outcome report |
| `--flakerate-seed N` | `FLAKERATE_SEED` | Shuffle test order with a reproducible seed |
| `--flakerate-quarantine PATH` | `FLAKERATE_QUARANTINE` | Apply a quarantine file |

## How it compares

| Tool | Measures the rate | Confidence interval | Diagnoses cause | Quarantine expiry | Maintained |
| --- | --- | --- | --- | --- | --- |
| `pytest-rerunfailures` | no | no | no | no | yes |
| `flaky` | no | no | no | no | last release 2024 |
| `pytest-flakefinder` | reruns only | no | no | no | last release 2022 |
| flakerate | yes | yes | yes | yes | yes |

## Notes

How many runs you need depends on what you are trying to catch. Twenty finds anything failing more
than about 1 in 10. A 1-in-50 flake needs a few hundred, and `flakerate run` says so when the
sample is too small to conclude anything.

A test that fails on every run is broken, not flaky, and is reported separately.

Failures in setup and teardown count as failures. A fixture that intermittently explodes is a
flaky test to anyone waiting on the build, whatever pytest calls it internally.

A pytest run that exits 2 or higher (a collection error, a bad argument) is discarded rather than
counted as everything failing.

`--flakerate-seed` reproduces an exact order, so a failure found here can be re-run on its own.

The plugin is a pytest entry point, so it is imported by every pytest process on the machine. It
registers nothing and returns immediately from every hook unless given a report path or a
quarantine file.

`pytest-xdist` is not supported yet: outcomes from several worker processes are not merged.

## Contributing

Bug reports and pull requests are welcome. `uv sync` then `uv run pytest` to get started.

## License

MIT.
