Metadata-Version: 2.4
Name: skxperiments
Version: 0.1.0.dev0
Summary: Randomization-based experimental design and causal inference, sklearn-style.
Project-URL: Homepage, https://github.com/gusbruschi13/skxperiments
Project-URL: Repository, https://github.com/gusbruschi13/skxperiments
Project-URL: Issues, https://github.com/gusbruschi13/skxperiments/issues
Author-email: Gustavo Bruschi <gustavo.bruschi@riachuelo.com.br>
License-Expression: MIT
Keywords: ab-testing,causal-inference,design-of-experiments,experiments,potential-outcomes,randomization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scipy
Provides-Extra: dev
Requires-Dist: black; extra == 'dev'
Requires-Dist: hypothesis; extra == 'dev'
Requires-Dist: ipykernel; extra == 'dev'
Requires-Dist: matplotlib; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: nbmake; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: viz
Requires-Dist: matplotlib; extra == 'viz'
Description-Content-Type: text/markdown

# skxperiments

> Randomization-based experimental design and causal inference, sklearn-style.

![CI](https://github.com/username/skxperiments/actions/workflows/ci.yml/badge.svg)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![Status](https://img.shields.io/badge/status-alpha-orange)

A Python library for designing randomized experiments and estimating causal effects under the
potential outcomes framework (Rubin Causal Model). Treatment assignment is the starting point;
statistical models come second.

## Status

The v1 feature set is complete: Phases 0–7 are done (sequential testing is
deferred to v2). See [Project status](#project-status) below for details.

## Installation

```bash
pip install skxperiments
```

Requires Python 3.10+. Dependencies: `numpy`, `pandas`, `scipy`.

## Quick start

```python
import numpy as np
import pandas as pd
from skxperiments.design.crd import CRD
from skxperiments.estimators.difference_in_means import DifferenceInMeans
from skxperiments.inference import RandomizationTest

# 1. Generate a synthetic dataset
rng = np.random.default_rng(42)
df = pd.DataFrame({
    "x": rng.normal(0.0, 1.0, 200),
    "y": rng.normal(0.0, 1.0, 200),
})

# 2. Design: completely randomized assignment, 50/50 split
design = CRD(p=0.5, seed=42)
assignment = design.randomize(df)

# 3. Point estimate of the ATE
estimator = DifferenceInMeans(outcome_col="y")
result = estimator.fit(assignment).estimate()
print(result.ate)

# 4. Randomization-based p-value (Fisher's sharp null)
rt = RandomizationTest(estimator=estimator, n_permutations=10_000, seed=0)
result = rt.fit(assignment).estimate()
print(result.ate, result.p_value)
```

For variance reduction with covariates, use `LinEstimator` (Lin 2013) or `CUPED` (Deng et al.
2013). For blocked or factorial designs, use `BlockedCRD` + `BlockedDifferenceInMeans` or
`FactorialDesign` + `FactorialEstimator`. For rerandomization on Mahalanobis distance,
use `ReRandomizedCRD` (Morgan & Rubin 2012). `RandomizationTest` works with all of these
(except `FactorialAssignment` in v1). To control the family-wise error rate or false
discovery rate when reporting multiple effects, wrap the result in
`MultipleTestingCorrection`.

## Documentation and tutorials

Learning-path notebooks (bilingual, Portuguese and English) live in
[`examples/for_starters/`](examples/for_starters/); conceptual docs (a glossary
and a "how to choose" guide) are in [`docs/`](docs/README.md).

## Design philosophy

1. **The assignment mechanism is primary**, not the statistical model.
2. **API in scikit-learn style**: parameters in `__init__`, data in `fit()`, learned attributes
   end with `_`.
3. **`Assignment` is the contract** between designs and estimators — estimators receive
   `Assignment` objects, not loose DataFrames.
4. **Randomization-based inference is the default**; classical t-tests are not.
5. **Finite-population vs. superpopulation inference are distinguished explicitly.**
6. **Fail fast** with clear messages when designs and estimators are incompatible.
7. **No side effects**: `fit()` and `randomize()` never mutate input DataFrames.

## Project status

| Phase | Module | Status |
|---|---|---|
| 0 | Scaffold, exceptions, CI | ✓ Complete |
| 1 | Core (`Assignment`, `Results`, base classes) | ✓ Complete |
| 2 | Designs (CRD, BlockedCRD, ReRandomizedCRD, FactorialDesign, balance, power) | ✓ Complete |
| 3 | Estimators (DIM, BlockedDIM, Factorial, Lin, CUPED) | ✓ Complete |
| 4 | Inference (RandomizationTest, MultipleTestingCorrection, NeymanCI, BootstrapCI) | ✓ Complete (4.1–4.4; sequential → v2) |
| 5 | Diagnostics (SRMTest, AATest, BalanceReport) | ✓ Complete |
| 6 | Pipeline (ExperimentPipeline, ExperimentComparison) | ✓ Complete |
| 7 | Visualization and reporting (plots, ExperimentReport) | ✓ Complete |

Test coverage: 720 tests, all passing on CI.

See [`ROADMAP.md`](ROADMAP.md) for deferred features and v2 plans, and `CHANGELOG.md` for the
full history of changes.

## What's implemented

### Designs (`skxperiments.design`)

- **`CRD`** — Completely randomized design.
- **`BlockedCRD`** — Independent randomization within blocks.
- **`ReRandomizedCRD`** — Mahalanobis acceptance criterion with cached covariance matrix; loop with `max_attempts`.
- **`FactorialDesign`** — 2^K factorial design with equal cell sizes; little-endian cell encoding.
- **`check_balance(assignment, covariates)`** — Standardized mean differences (SMD), pooled std with `ddof=1`.
- **`power_analysis(...)`** — Sample size, MDE, or power for two-sample mean comparisons.

### Estimators (`skxperiments.estimators`)

- **`DifferenceInMeans`** — Simple ATE for `CRDAssignment`.
- **`BlockedDifferenceInMeans`** — Size-weighted ATE for `BlockedAssignment`.
- **`FactorialEstimator`** — All 2^K − 1 effects (main effects and interactions of all orders) for `FactorialAssignment`. Returns `Results` in multi-effect mode.
- **`LinEstimator`** — Covariate-adjusted ATE via OLS with treatment-covariate interaction (Lin 2013).
- **`CUPED`** — Variance reduction with a pre-experiment covariate (Deng et al. 2013).

All estimators return `Results` with point estimates only; standard errors and confidence
intervals come from inference classes in `skxperiments.inference`.

### Inference (`skxperiments.inference`)

- **`RandomizationTest`** — Fisher's sharp null hypothesis test via Monte Carlo permutations.
  Uses `Assignment.draw()` to respect the original randomization mechanism (including
  rerandomization Mahalanobis criterion and within-block proportions). P-value via the
  Phipson & Smyth (2010) continuity correction. Three alternatives: `"two-sided"`,
  `"greater"`, `"less"`. Works with `DifferenceInMeans`, `BlockedDifferenceInMeans`,
  `LinEstimator`, and `CUPED`.
- **`MultipleTestingCorrection`** — Bonferroni, Holm (FWER) and Benjamini-Hochberg (FDR)
  correction over a family of p-values. Accepts a multi-effect `Results` (typical from
  `FactorialEstimator` after inference) or a list of scalar `Results` (for comparing
  independent experiments). Clips corrected p-values to `[0, 1]`; preserves originals
  in `Results.extra["original_p_values"]`. Default method is Holm.
- **`NeymanCI`** — Neyman variance-based two-sided Wald confidence interval and p-value
  for finite-population inference. Conservative variance for `CRDAssignment` (including
  rerandomized) and stratified variance for `BlockedAssignment`, consistent with the
  size-weighted ATE of `BlockedDifferenceInMeans`. Wraps `DifferenceInMeans` or
  `BlockedDifferenceInMeans`; rejects superpopulation mode (use `BootstrapCI`).
- **`BootstrapCI`** — Bootstrap confidence interval (percentile or BCa) for
  superpopulation inference. Resamples units within each arm (within each
  block-by-arm stratum for blocked designs) and refits the estimator, so it
  works with any scalar estimator (`DifferenceInMeans`, `BlockedDifferenceInMeans`,
  `LinEstimator`, `CUPED`).

### Diagnostics (`skxperiments.diagnostics`)

- **`SRMTest`** — Sample Ratio Mismatch via chi-squared: observed vs. intended arm/cell
  allocation, flagged below a threshold (default 0.001). Two-arm and factorial.
- **`BalanceReport`** — Standardized mean differences (SMD) per covariate, flagging
  `|SMD| > 0.1`. Consumes `check_balance`; `to_dataframe()` feeds the Phase 7 Love plot.
- **`AATest`** — A/A calibration: re-randomizes a design on fixed data, runs a wrapped
  inference, and checks the false-positive rate (exact binomial test) and p-value
  uniformity (KS).

Each returns a dedicated result with `to_diagnostics_report()` for pipeline aggregation.

### Pipeline (`skxperiments.pipeline`)

- **`ExperimentPipeline`** — Composes an inference (with its estimator) and diagnostics,
  runs them on an `Assignment`, and bundles the result. Runs `SRMTest` automatically;
  diagnostics are best-effort and flags are surfaced without halting (opt-in
  `raise_on_flag`).
- **`ExperimentComparison`** — Compares independent experiments, applying
  `MultipleTestingCorrection` across the family. Returns a comparison table ready for the
  forest plot. Subgroup comparison is deferred to v2.

### Reporting (`skxperiments.reporting`)

Requires the optional `viz` extra (`pip install skxperiments[viz]`).

- **Plots** — diagnostic (`plot_balance`, `plot_srm`, `plot_null_distribution`) and result
  (`plot_effect`, `plot_forest`, `plot_interaction`, `plot_power_curve`). Each returns a
  matplotlib `Axes` and accepts an optional `ax`.
- **`ExperimentReport`** — Renders a `PipelineResult` as a self-contained static HTML page
  (results table, diagnostics, embedded plots). `include_plots=False` skips the optional
  dependency.

## What's next (v2)

The v1 feature set is complete. Deferred items live in [`ROADMAP.md`](ROADMAP.md):
`SequentialTest` (mSPRT/always-valid), Benjamini-Yekutieli correction, covariate-adjusted
variance in `NeymanCI`, studentized and block-resampling bootstrap, subgroup comparison,
a plotly backend, and interactive dashboards.

## Contributing

Contributions are welcome. Please open an issue to discuss substantial changes before submitting
a pull request. The architecture has documented design decisions that should be respected — see
[`ROADMAP.md`](ROADMAP.md), the project notes in `CHANGELOG.md`, and the docstrings of base
classes (`BaseAssignment`, `BaseEstimator`, `Results`) for the contracts new code must follow.

Run the test suite with:

```bash
pytest tests/ -v
```

Skip slow statistical tests:

```bash
pytest tests/ -v -m "not slow"
```

## License

MIT.

## References

The implementations follow standard textbook formulations:

- Imbens, G. W., & Rubin, D. B. (2015). *Causal inference for statistics, social, and biomedical
  sciences: An introduction.* Cambridge University Press.
- Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining
  Freedman's critique. *Annals of Applied Statistics*, 7(1), 295–318.
- Morgan, K. L., & Rubin, D. B. (2012). Rerandomization to improve covariate balance in
  experiments. *Annals of Statistics*, 40(2), 1263–1282.
- Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the sensitivity of online
  controlled experiments by utilizing pre-experiment data. *WSDM 2013*.
- Box, G. E. P., Hunter, J. S., & Hunter, W. G. (2005). *Statistics for experimenters: Design,
  innovation, and discovery* (2nd ed.). Wiley.
- Cohen, J. (1988). *Statistical power analysis for the behavioral sciences* (2nd ed.). Routledge.
- Austin, P. C. (2009). Balance diagnostics for comparing the distribution of baseline covariates
  between treatment groups in propensity-score matched samples. *Statistics in Medicine*.
- Phipson, B., & Smyth, G. K. (2010). Permutation P-values should never be zero: calculating
  exact P-values when permutations are randomly drawn. *Statistical Applications in Genetics
  and Molecular Biology*, 9(1).
- Fisher, R. A. (1935). *The Design of Experiments*. Oliver and Boyd.
- Holm, S. (1979). A simple sequentially rejective multiple test procedure. *Scandinavian
  Journal of Statistics*, 6(2), 65–70.
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and
  powerful approach to multiple testing. *Journal of the Royal Statistical Society: Series B*,
  57(1), 289–300.