Metadata-Version: 2.4
Name: ab-analysis-kit
Version: 0.0.1.dev0
Summary: A/B experiment analysis as declarative YAML + SQL
Author: ab-analysis-kit authors
License: MIT
Project-URL: Homepage, https://abkit.pipelab.dev
Project-URL: Documentation, https://abkit.pipelab.dev
Project-URL: Repository, https://github.com/alexeiveselov92/ab-analysis-kit
Project-URL: Changelog, https://github.com/alexeiveselov92/ab-analysis-kit/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/alexeiveselov92/ab-analysis-kit/issues
Keywords: ab-testing,experimentation,statistics,cuped,bootstrap,causal-inference,clickhouse
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: statsmodels>=0.14.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: click>=8.0
Requires-Dist: jinja2>=3.0
Requires-Dist: orjson>=3.0
Requires-Dist: requests>=2.25.0
Provides-Extra: clickhouse
Requires-Dist: clickhouse-driver>=0.2.0; extra == "clickhouse"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
Provides-Extra: mysql
Requires-Dist: pymysql>=1.0.0; extra == "mysql"
Provides-Extra: all-db
Requires-Dist: clickhouse-driver>=0.2.0; extra == "all-db"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "all-db"
Requires-Dist: pymysql>=1.0.0; extra == "all-db"
Provides-Extra: orchestration
Requires-Dist: prefect>=2.0; extra == "orchestration"
Provides-Extra: all
Requires-Dist: clickhouse-driver>=0.2.0; extra == "all"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "all"
Requires-Dist: pymysql>=1.0.0; extra == "all"
Requires-Dist: prefect>=2.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pytest-mock>=3.0; extra == "dev"
Requires-Dist: requests-mock>=1.12; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Provides-Extra: integration
Requires-Dist: testcontainers[clickhouse,mysql,postgres]>=4.0; extra == "integration"
Requires-Dist: psycopg2-binary>=2.9.0; extra == "integration"
Requires-Dist: pymysql>=1.0.0; extra == "integration"
Dynamic: license-file

# ab-analysis-kit

**A/B experiment analysis as declarative YAML + SQL — with a chart-first cockpit.**

`ab-analysis-kit` (CLI `abk`) is an open-source, declarative
(dbt / [detectkit](https://dtk.pipelab.dev)-style), database-agnostic, numpy-first
Python library for analyzing A/B experiments. You define an **experiment** and its
**metrics** in YAML + SQL; abkit computes per-method effect + confidence interval +
p-value + MDE/power **cumulatively over the experiment's lifetime** (the
stabilization chart), writes them to a clean warehouse table any BI can read, and
gives you a local cockpit to tune the analysis and a harness to prove your method is
actually calibrated.

> **Status: pre-development.** This repo currently holds the **project-initiation
> specs** (the development contract). Start with the master plan (RU):
> [docs/ru/project-initiation-spec.md](docs/ru/project-initiation-spec.md), then the
> [specs index](docs/specs/00-overview.md).

## What it will do

- **Declarative experiments** — `experiments/*.yml` (assignment + variants +
  comparisons) referencing a reusable `metrics/*.yml` library (YAML + SQL).
- **A rigorous statistical engine** — t-test, two-proportion z-test, CUPED, ratio
  (delta-method), and a vectorised bootstrap family (plain/paired/Poisson/
  post-normed), with relative & absolute effects, MDE/power, and multiple-testing
  correction. Ported from a battle-tested legacy engine and improved deliberately.
- **The cumulative stabilization chart** — effect + CI per day from experiment
  start, so you see the estimate converge and call a winner only once it stabilizes.
- **`abk explore`** — a local, chart-first cockpit to turn method knobs (CUPED,
  stratification, alpha…) and watch the result recompute live, with A/A calibration
  always in view. *The priority interface.*
- **`abk validate`** — an A/A false-positive + power matrix that measures your
  method's **real** α (including the honest cumulative-peeking FPR), not the nominal.
- **BI-agnostic** — results land in one clean table; connect Grafana, Lightdash,
  Metabase, or Superset. Orchestrate with **Prefect**.
- **AI-native** — `abk init-claude` sets up assistant context + skills so an
  assistant can scaffold and tune experiments with (or for) you.

## Design at a glance

```
experiment (YAML)  ──▶ load exposures ──▶ SRM gate ──▶ compute (t/z/CUPED/bootstrap) ──▶ readout
  └ references reusable metrics (YAML + SQL)                                          └ _ab_results → your BI
```

abkit is the sibling of detectkit: same DNA (CLI-first, db-agnostic, numpy-first,
self-contained reports, a chart-first cockpit, `init-claude`), with the anomaly
`detect` stage replaced by a statistical `compute` stage and the primary entity
flipped from *metric* to *experiment*.

## Documentation

- **Master plan (RU):** [docs/ru/project-initiation-spec.md](docs/ru/project-initiation-spec.md)
- **Specs index:** [docs/specs/00-overview.md](docs/specs/00-overview.md)
- **Architecture:** [docs/specs/architecture.md](docs/specs/architecture.md)
- **Roadmap:** [ROADMAP.md](ROADMAP.md) · **Principles:** [PRINCIPLES.md](PRINCIPLES.md)
- **Contributor guide:** [CLAUDE.md](CLAUDE.md)

## License

MIT (planned).
