Metadata-Version: 2.4
Name: peekproof
Version: 0.1.0
Summary: A/B tests you can check early: group sequential boundaries, power, and peeking simulations.
Author: Zoheb Islam
License-Expression: MIT
Project-URL: Homepage, https://github.com/Zohebislam/cookie-cats-ab-testing
Project-URL: Demo, https://cookie-cats-ab-test1.streamlit.app/
Keywords: ab-testing,sequential-testing,experimentation,statistics,peeking
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Requires-Dist: scipy>=1.11
Requires-Dist: pandas>=2.1
Provides-Extra: test
Requires-Dist: pytest>=7.4; extra == "test"
Dynamic: license-file

# peekproof

A/B tests you can check early without fooling yourself.

If you check an A/B test every day and stop the first time p < 0.05, your false positive
rate is no longer 5%. With daily checks over two weeks it is about 22%. `peekproof` gives
you the standard fix, group sequential testing, plus the tools around it.

```bash
pip install peekproof
```

## Plan a test, then check it as often as you planned

```python
import peekproof

test = peekproof.plan(n_looks=5)      # five evenly spaced looks, 5% false positives overall
print(test.table())
```

```
 look  info_fraction  z_boundary  nominal_p  alpha_spent
    1            0.2      4.8769     0.0000       0.0000
    2            0.4      3.3570     0.0008       0.0008
    3            0.6      2.6803     0.0074       0.0076
    4            0.8      2.2898     0.0220       0.0244
    5            1.0      2.0310     0.0423       0.0500
```

The bar is very high early and close to an ordinary test at the end. At each look, pass the
counts so far:

```python
test.check(look=2, x_a=300, n_a=2000, x_b=360, n_b=2000)
# {'z': 2.556, 'boundary': 3.357, 'nominal_p': 0.0106, 'stop': False, 'decision': 'keep going', ...}

test.check(look=5, x_a=760, n_a=5000, x_b=880, n_b=5000)
# {'z': 3.241, 'boundary': 2.031, 'stop': True, 'decision': 'stop: B is higher than A', ...}
```

At look 2 an ordinary test would have called a winner (p = 0.011). The plan says keep going.

Options: `spending="obf"` (O'Brien-Fleming-type, the default) or `spending="pocock"`, any
`alpha`, and uneven looks with `peekproof.plan(info_fracs=[0.25, 0.5, 1.0])`.

## What else is in it

```python
peekproof.sample_size_per_arm(0.19, -0.01)        # 23665 users per group to detect a 1-point drop
peekproof.srm_test(44700, 45489)                  # sample ratio mismatch check
peekproof.two_prop_ztest(8502, 44700, 8279, 45489)  # difference, z, p-value, confidence interval

# See the peeking problem for yourself: false positives vs number of looks
peekproof.simulate.fpr_vs_looks(0.19, 45000, looks=(1, 5, 14))
#  looks  false_positive_rate
#      1                0.048
#      5                0.141
#     14                0.223
```

| Module | What it does |
|---|---|
| `design` | `plan` a sequential test and `check` it at each look |
| `sequential` | group sequential boundaries (Lan-DeMets alpha spending) and mSPRT always-valid p-values |
| `power` | minimum detectable effect, sample size, power |
| `frequentist` | sample ratio mismatch, two-proportion z-test, bootstrap, Holm correction |
| `bayesian` | Beta-Binomial posteriors and expected loss |
| `simulate` | simulations of peeking and of each fix, on simulated or shuffled real data |

## How it is checked

- Boundaries are computed by numerical integration, not simulation, and match the published
  O'Brien-Fleming values from gsDesign / ldbounds to three decimals.
- Simulations of 300,000 tests with no real effect confirm the overall false positive rate
  stays at 5% for both spending functions, including with uneven looks.
- 38 tests in the repository cover the statistics against scipy, published values and simulation.

## Limits

- Two groups and yes/no outcomes (conversion, retention). No revenue-style metrics yet.
- Looks are planned in advance. If you need to look at any time, use the mSPRT functions in
  `peekproof.sequential`, which trade some power for that freedom.

## Where it came from

The package grew out of an analysis of the Cookie Cats mobile game A/B test (90,189 players).
The full analysis and write-up are in the
[repository](https://github.com/Zohebislam/cookie-cats-ab-testing), and there is an
[interactive app](https://cookie-cats-ab-test1.streamlit.app/) with a peeking simulator.

## References

- Lan, DeMets (1983). Discrete sequential boundaries for clinical trials. Biometrika.
- Jennison, Turnbull (2000). Group Sequential Methods with Applications to Clinical Trials.
- Johari, Pekelis, Walsh (2017). Peeking at A/B Tests: Why it matters, and what to do about it. KDD.

MIT License.
