Metadata-Version: 2.4
Name: flaky-repro
Version: 0.1.2
Summary: An empirical, frequency-based flaky test diagnosis and reproduction tool.
Author: Flaky-Repro Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Framework :: Pytest
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pytest>=7.0.0
Dynamic: license-file

# Flaky-Repro
**An empirical, frequency-based flaky test diagnosis and reproduction tool for pytest.**
Flaky-Repro investigates flaky tests by repeatedly executing them under different execution conditions, measuring failure frequencies, comparing PASS/FAIL executions, correlating conditions with captured evidence, confirming stronger candidates through repeated experiments, and attempting to reproduce the observed behavior.
---
##  Why Flaky-Repro?
Flaky tests are difficult to diagnose because they can pass and fail under seemingly identical conditions.
A simple approach is to repeatedly run a test:
```bash
pytest test_example.py::test_something
```
But repeated execution alone does not tell you which execution condition is associated with the failures.
Flaky-Repro takes an empirical approach.
Instead of only asking:
> Does this test fail sometimes?
Flaky-Repro investigates:
> Under which execution conditions does this test fail more frequently, and can that behavior be reproduced?
---
##  Installation
Install Flaky-Repro directly from PyPI:
```bash
pip install flaky-repro
```
Verify the installation:
```bash
flaky-repro --version
```
Expected:
```text
flaky-repro 0.1.2
```
---
##  Quick Start
Run Flaky-Repro against a pytest target:
```bash
flaky-repro <pytest-target>
```
For example:
```bash
flaky-repro flaky_repro/examples/functional_validation/test_worker_validation.py::test_worker_sensitive
```
Flaky-Repro will automatically run the target through its diagnosis and reproduction pipeline.
---
##  How It Works
Flaky-Repro follows a multi-stage empirical diagnosis process:
```text
                         Pytest Target
                              │
                              ▼
                    ┌──────────────────┐
                    │     Preflight    │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ Code Analysis    │
                    │  → experiment    │
                    │    shortlisting  │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │     Baseline     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Investigation  │
                    │ workers / mode   │
                    │ timing / timeout │
                    │ order / state    │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Correlation    │
                    │ condition ↔      │
                    │ captured evidence│
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ PASS ↔ FAIL      │
                    │   Comparison     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │    Structured    │
                    │     Findings     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Confirmation   │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Reproduction   │
                    └────────┬─────────┘
                             │
                             ▼
                       Diagnostic Result
```
The goal is not simply to rerun a test.
The goal is to determine whether specific execution conditions are associated with increased failure frequency and whether those conditions can be reproduced consistently.
---
##  What It Investigates
Flaky-Repro can investigate execution conditions and signals such as:
- Worker/concurrency levels
- Sequential vs parallel execution
- Timing delays
- Timeout conditions
- Test order and state-related signals
- Failure frequency under different conditions
- PASS ↔ FAIL differences
- Captured exceptions, tracebacks, stderr, and assertion evidence
- Candidate strength and consistency
- Reproduction consistency
Before running experiments, Flaky-Repro can analyze the target test source and shortlist relevant experiment types from code-level signals such as async code, mutable state, timing calls, concurrency primitives, file writes, and randomness.
The investigation remains empirical: code analysis selects hypotheses, while execution validates or rejects those hypotheses.
---
##  Example
The following output is representative of the included functional validation pipeline for a timing/concurrency-sensitive pytest target.
Result
```text
                    FLAKY-REPRO
============================================================
TEST
Target : flaky_repro/examples/functional_validation/test_mode_validation.py::test_mode_sensitive
RESULT
Status : FLAKY
```
Code Analysis
```text
CODE ANALYSIS
Signals detected:
- concurrency
- timing
- timeout
Shortlisted experiments:
- worker
- timing
- mode
- timeout
  ```
Code analysis is used to prioritize experiments; it does not claim that a detected code pattern is the root cause.
Baseline
```text
BASELINE
Runs         : 20
Passed       : 20
Failed       : 0
Failure Rate : 0%
```
Investigation
Each selected condition is evaluated repeatedly. In the validated V0.1.2 run, 100 repetitions were used per investigation condition:
```text
Type       Condition                 Failure Rate
Worker     2 workers                 37%
Worker     4 workers                 29%
Worker     8 workers                 20%
Timing     10 ms                     17%
Timing     20 ms                     11%
Timeout    30 s                       9%
Timeout    60 s                      12%
```
Condition Correlation
Flaky-Repro groups executions by condition fingerprint and associates observed failures with the evidence captured in those runs.
```text
CONDITION CORRELATION
Total known runs : 1260
Passed           : 1008
Failed           : 252
Unknown          : 0
Top observed condition:
Parallel / 4 workers / 30 s timeout
Observed failure rate : 27.14%
Signal strength       : MODERATE
```
The strength classification describes the observed signal in the collected runs. It is not statistical confidence and does not establish causation.
PASS ↔ FAIL Comparison
Flaky-Repro also compares passing and failing executions:
```text
PASS ↔ FAIL COMPARISON
Passed runs : 1008
Failed runs : 252
Unknown     : 0
Average duration
PASS : 1.02 s
FAIL : 1.17 s
Delta: +0.15 s
Failing exception : TestFailure
```
This helps preserve the distinction between what conditions were associated with failures and what differed between successful and failing executions.
Structured Findings
The final output turns the collected evidence into structured findings containing:
- condition and condition fingerprint
- known/pass/fail/unknown counts
- observed failure rate and signal strength
- baseline comparison
- captured failure evidence
- PASS ↔ FAIL differences
- confirmation and reproduction results
- replay context where available
- an observational interpretation
Findings explicitly avoid claiming that an observed condition is the definitive root cause.
Confirmation and Reproduction
Candidates are repeatedly evaluated before being treated as stronger findings, followed by higher-volume reproduction runs.
```text
CONFIRMATION
Candidate conditions are re-tested to determine
whether the observed signal persists.
REPRODUCTION
Stronger candidates are subjected to additional runs
to determine whether the observed behavior can be reproduced.
```
This demonstrates the V0.1.2 workflow:
preflight → code analysis → baseline → investigation → condition/evidence correlation → PASS/FAIL comparison → structured findings → confirmation → reproduction
---
Target : flaky_repro\examples\functional_validation\test_worker_validation.py::test_worker_sensitive
RESULT
------------------------------------------------------------
Status               : FLAKY
Observed Pattern     : Parallel Execution
Strongest Signal     : Worker Count
Condition            : 8 workers
Reproduction         : REPRODUCED
```
### Baseline
The test passed consistently under the sequential baseline:
```text
Baseline
  Runs         : 20
  Passed       : 20
  Failed       : 0
  Failure Rate : 0%
```
### Candidate
Under the strongest candidate condition:
```text
Candidate
  Condition    : Worker (8 workers)
  Runs         : 20
  Passed       : 13
  Failed       : 7
  Failure Rate : 35%
```
The observed failure-rate difference was:
```text
Effect
  Failure delta: +35 pp
  Direction    : Increased
```
### Investigation
```text
Type           Condition                 Failure Rate
------------------------------------------------------------
Worker         2 workers                 5%
Worker         4 workers                 30%
Worker         8 workers                 35%
Timing         10 ms                     5%
Timing         20 ms                     10%
Mode           Sequential                0%
Mode           Parallel / 4              30%
```
### Candidate Ranking
```text
Rank   Candidate                    Failure Rate   Signal
------------------------------------------------------------
#1     Worker (8 workers)           35%            STRONG
#2     Worker (4 workers)           30%            STRONG
#3     Timing (20 ms)               10%            STRONG
```
### Confirmation
Candidates are repeatedly evaluated to determine whether the observed effect persists:
```text
Candidate                      Classification        Consistency
-----------------------------------------------------------------
#1 Worker (8 workers)          REPEATEDLY OBSERVED   100.0%
#2 Worker (4 workers)          REPEATEDLY OBSERVED   100.0%
#3 Timing (20 ms)              REPEATEDLY OBSERVED   100.0%
```
### Reproduction
The strongest candidates are then subjected to higher-volume reproduction runs:
```text
Rank   Candidate                  Repetitions   Consistency    Result
-----------------------------------------------------------------------
#1     Worker (8 workers)         3             100.0%         REPRODUCED
#2     Worker (4 workers)         3             100.0%         REPRODUCED
#3     Timing (20 ms)             3             100.0%         REPRODUCED
```
### Interpretation
In this example:
- The test passed consistently under the sequential baseline.
- Parallel execution increased the observed failure rate.
- The 8-worker configuration produced the strongest observed signal.
- The candidate remained consistent during confirmation.
- The behavior was successfully reproduced during the reproduction stage.
This demonstrates how Flaky-Repro moves from **observation → investigation → candidate detection → confirmation → reproduction**.
---
##  CLI Usage
### Show help
```bash
flaky-repro --help
```
or:
```bash
flaky-repro -h
```
### Show version
```bash
flaky-repro --version
```
or:
```bash
flaky-repro -V
```
### Run a test
```bash
flaky-repro <pytest-target>
```
Example:
```bash
flaky-repro tests/test_concurrency.py::test_sensitive
```
---
##  Included Validation Examples
The repository includes functional validation tests designed to exercise different flaky-test behaviors, including:
- Worker/concurrency behavior
- Timing behavior
- Sequential vs parallel execution
- Stable tests
- Random/flaky behavior
- Multifactor conditions
These examples are useful for understanding and validating the diagnostic pipeline.
---
##  Safety & Limitations
Flaky-Repro repeatedly executes the target test under multiple configurations.
**Only run it against tests and environments where repeated execution is safe and side effects are acceptable.**
Be especially careful with tests that:
- Modify production data
- Send emails or messages
- Perform financial transactions
- Create irreversible external side effects
- Depend on shared external state
- Interact with rate-limited services
### Important
Flaky-Repro is an **empirical diagnostic tool**.
Its conclusions are based on observed execution frequencies and repeated experiments.
A strong signal means that a condition was associated with an increased failure rate in the observed experiments. It should not automatically be interpreted as definitive proof of the underlying root cause.
---
##  Project Status
Current version: 0.1.2
The 0.1.2 release provides:
- PyPI packaging
- Command-line interface
- Preflight target validation
- Structured execution runs
- Deterministic condition fingerprints
- Code analysis and experiment shortlisting
- Baseline execution
- Multi-condition investigation
- Candidate detection
- Condition ↔ evidence correlation
- PASS ↔ FAIL comparison
- Structured findings
- Candidate confirmation
- Reproduction testing
- Human-readable diagnostic output
- Functional validation examples
Flaky-Repro is still an early release. Diagnostic behavior, internal APIs, configuration options, and output formats may change in future versions.
Current limitations
V0.1.2 is intentionally focused on empirical investigation rather than full root-cause diagnosis.
- Observed correlations are not causal proof.
- Exact replay of a prior execution is not yet implemented.
- Rich passing-side evidence and more advanced statistical analysis are planned for later work.
- Environment-specific CI reproduction is not guaranteed.
- Baseline and experiment sample sizes affect the stability of observed failure rates.
- Some experiment types, such as order/state investigation, are not automatically wired into every pipeline path yet.
- Tests with unsafe or irreversible side effects should not be investigated casually.
---
##  Development
Clone the repository and enter the project directory:
```bash
git clone https://github.com/MrigankKumawat/Flaky-Repro.git
cd Flaky-Repro
```
Create a virtual environment:
```bash
python -m venv .venv
```
Activate it on Windows:
```bash
.venv\Scripts\activate
```
Install the project in editable mode:
```bash
pip install -e .
```
---
##  License
Flaky-Repro is released under the MIT License.
See the `LICENSE` file for the complete license text.
---
##  Contributing
Issues, bug reports, suggestions, and improvements are welcome.
If you discover a flaky-test pattern that Flaky-Repro handles incorrectly, opening an issue with a reproducible example can help improve the project.
---
##  Flaky-Repro
**Investigate the conditions behind flaky tests.**
