Metadata-Version: 2.4
Name: broom-sm
Version: 0.2.0
Summary: Tidy-style summaries for statsmodels fits
Home-page: https://github.com/ezraair555/broom-sm/
Author: EzraAir555
Author-email: ezraair555@gmail.com
License: MIT
Project-URL: Documentation, https://ezraair555.github.io/broom-sm/
Project-URL: Source, https://github.com/ezraair555/broom-sm/
Project-URL: Tracker, https://github.com/ezraair555/broom-sm/issues
Platform: any
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
License-File: LICENSE.txt
Requires-Dist: pandas>=1.5
Requires-Dist: numpy>=1.23
Requires-Dist: pandas_flavor>=0.6
Requires-Dist: statsmodels>=0.14
Requires-Dist: scipy>=1.9
Requires-Dist: importlib-metadata; python_version < "3.11"
Provides-Extra: docs
Requires-Dist: sphinx; extra == "docs"
Requires-Dist: myst-parser; extra == "docs"
Requires-Dist: linkify-it-py; extra == "docs"
Provides-Extra: viz
Requires-Dist: seaborn>=0.12; extra == "viz"
Requires-Dist: matplotlib>=3.6; extra == "viz"
Provides-Extra: bayes
Requires-Dist: bayesian_bootstrap>=0.1; extra == "bayes"
Provides-Extra: testing
Requires-Dist: setuptools; extra == "testing"
Requires-Dist: pytest; extra == "testing"
Requires-Dist: pytest-cov; extra == "testing"
Dynamic: license-file

# broom-sm <img src="README_files/broom_sm_hex_sticker.jpg" align="right" height="150" />

[![PyPI version](https://img.shields.io/pypi/v/broom-sm.svg)](https://pypi.org/project/broom-sm/)
[![Python versions](https://img.shields.io/pypi/pyversions/broom-sm.svg)](https://pypi.org/project/broom-sm/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE.txt)
[![CI](https://github.com/ezraair555/broom-sm/actions/workflows/ci.yml/badge.svg)](https://github.com/ezraair555/broom-sm/actions/workflows/ci.yml)

**Tidy-style statistical inference for Python with statsmodels**

broom-sm brings the ergonomic design of [broom](https://broom.tidymodels.org/) and the tidyverse to Python's statsmodels ecosystem. The package centers around three main verbs—`stats_tidy()`, `stats_glance()`, and `stats_augment()`—supplemented by bootstrapping utilities, diagnostic plots, and Bayesian helpers.

```python
import pandas as pd
import statsmodels.api as sm
from broom_sm import stats_report

# Load data
mtcars = sm.datasets.get_rdataset("mtcars").data

# Fit once, get everything
report = mtcars.stats_report(
    formula="mpg ~ wt + hp",
    stat_type="ols"
)

# Tidy coefficient table
print(report["tidy"])
#>       term  estimate  std.error  conf.low  conf.high   statistic    p.value
#> 0  Intercept  37.227270   1.877627  33.343267  41.111273  19.826764  1.27e-17
#> 1         wt  -3.877831   0.714968  -5.355789  -2.399873  -5.423781  1.19e-05
#> 2         hp  -0.031157   0.011436  -0.054812  -0.007501  -2.724389  1.12e-02

# Model-level statistics
print(report["glance"])
#>   stat_type  nobs      llf       aic       bic  df_model  df_resid  rsquared
#> 0       ols    32 -72.54928  153.09856  158.95938       2.0      29.0  0.826783
```

## Installation

```bash
# Core package (tidy verbs + diagnostics)
pip install broom-sm

# With visualization dependencies
pip install broom-sm[viz]

# With Bayesian bootstrap support
pip install broom-sm[bayes]
```

## The Tidy Workflow

broom-sm is built around three core verbs that convert statsmodels objects into tidy DataFrames:

| Verb | Purpose | Output |
|------|---------|--------|
| `stats_tidy()` | Coefficient tables | One row per term |
| `stats_glance()` | Model-level statistics | One row per model |
| `stats_augment()` | Add predictions & residuals | One row per observation |

### Example: Analysis of Variance

Test whether vehicle weight differs by cylinder count:

```python
import pandas as pd
import statsmodels.api as sm

mtcars = sm.datasets.get_rdataset("mtcars").data

# Calculate observed statistic
obs_stat = mtcars.stats_anova_tidy(
    formula="wt ~ factor(cyl)",
    anova_type=2
)

# Bootstrap the null distribution
null_dist = mtcars.boot_tidy(
    formula="wt ~ factor(cyl)",
    stat_type="ols",
    n_boot=1000,
    seed=42
)

# Visualize
from broom_sm import stats_residual_plot
figures = mtcars.stats_residual_plot(["cyl"], y="wt")
figures[0][1].show()

# Calculate p-value
from scipy import stats
f_stat = obs_stat["statistic"].iloc[0]
p_value = 1 - stats.f.cdf(f_stat, obs_stat["df"].iloc[0], obs_stat["df_resid"].iloc[0])
```

## Key Features

### 🔁 Tidy Verbs

All core verbs work with formulas or pre-fitted statsmodels results:

```python
# Formula interface
df.stats_tidy("y ~ x1 + x2", stat_type="ols")

# Pre-fitted model interface
import statsmodels.formula.api as smf
model = smf.ols("y ~ x1 + x2", data=df).fit()
df.stats_tidy(model=model)
```

### 🧱 Extensible Model Registry

Support for OLS, GLMs (Poisson, Gamma, Beta, Negative Binomial), GEE, MixedLM, PHReg/Survival, and Quantile Regression. Register custom models:

```python
from broom_sm.model_registry import ModelSpec, register_model
import statsmodels.formula.api as smf

register_model(
    "tobit",
    ModelSpec(
        fitter=lambda formula, data, **kwargs: smf.tobit(formula, data=data, **kwargs).fit(),
        stat_name="z_stat"
    )
)
```

### 📦 Bootstrapping

Built-in resampling with consistent logging:

```python
boot = mtcars.boot_tidy(
    formula="mpg ~ wt",
    stat_type="ols",
    n_boot=500,
    seed=11
)
boot.groupby("term")["estimate"].agg(["mean", "std"])
```

### 📊 Diagnostics & Visualization

All plot helpers return Matplotlib figures (no implicit `plt.show()`):

```python
# Residual diagnostics
figures = df.stats_residual_plot(["x1", "x2"], y="y")

# Influence plot
fig = df.stats_influence_plot("y ~ x1 + x2", stat_type="ols")

# Coefficient forest plot
tidy = df.stats_tidy("y ~ x1 + x2", stat_type="ols")
fig, ax = stats_coef_forest(tidy)
```

### 🧪 Robust Standard Errors

Pass `cov_type`, `cov_kwds`, `family`, `link`, or `weights` directly:

```python
df.stats_tidy(
    formula="mpg ~ wt",
    stat_type="glm",
    family="binomial",
    weights=df["weights"],
    cov_type="HC3"
)
```

### 🛠️ Command-Line Interface

Quick reports from the terminal:

```bash
# Single model report
broom-sm report --data data.csv --formula 'y ~ x1 + x2' --stat-type ols

# Compare multiple models
broom-sm compare --data data.csv --stat-type ols \
  --formulas "y ~ x1" "y ~ x1 + x2"
```

Output defaults to JSON; pass `--format csv` for tabular output.

### 🔗 `widyr` Integration (R parity)

`broom-sm` now includes a Python port of core
[`widyr`](https://github.com/juliasilge/widyr) verbs for tidy pairwise and
wide-matrix workflows:

- `pairwise_count`, `pairwise_cor`, `pairwise_dist`, `pairwise_similarity`
- `pairwise_pmi`, `pairwise_delta`
- `widely_svd`, `widely_kmeans`, `widely_hclust`, `widely`, `squarely`
- `cor_sparse`

```python
from broom_sm import pairwise_cor, widely_kmeans

# Pairwise country similarity by life expectancy trajectories
corr = pairwise_cor(gapminder, "country", "year", "lifeExp", method="pearson")

# Cluster countries in wide feature space
clusters = widely_kmeans(gapminder, "country", "year", "lifeExp", k=3, random_state=0)
```

All pairwise outputs use tidy columns (`item1`, `item2`, metric column), and
the module is fully exported from `broom_sm.__init__`.

## Model Coverage

| Model Type | `stat_type` | Robust SEs | Weights | Family/Link |
|------------|-------------|------------|---------|-------------|
| OLS | `"ols"` | ✅ | ✅ | — |
| GLM (Gaussian) | `"glm"` | ✅ | ✅ | ✅ |
| GLM (Poisson) | `"poisson"` | ✅ | ✅ | ✅ |
| GLM (Gamma) | `"gamma"` | ✅ | ✅ | ✅ |
| GLM (Beta) | `"beta"` | ✅ | ✅ | ✅ |
| Negative Binomial | `"negbin"` | ✅ | ✅ | — |
| Quantile Regression | `"quantreg"` | ✅ | ✅ | — |
| GEE | `"gee"` | ✅ | ✅ | ✅ |
| MixedLM | `"mixedlm"` | ✅ | ✅ | — |
| PHReg (Survival) | `"phreg"` | ✅ | ✅ | — |
| Logit / Binomial | `"logit"` | ✅ | ✅ | ✅ |

## Tidy Diagnostics (broom + broomExtra parity)

The package also exposes tidy wrappers for diagnostic, model-comparison, and
inference helpers — many of which are parity work for R's `broom` and
`broomExtra`:

| broom-sm function | Equivalent R helper | Purpose |
|-------------------|---------------------|---------|
| `stats_kendall_tidy` | `broom::tidy.Kendall` | Kendall's τ correlation matrix |
| `stats_coeftest` | `lmtest::coeftest` | Wald z-tests for any fitted model |
| `stats_manova_tidy` | `broom::tidy.manova` | One-way MANOVA (Wilks / Pillai / Hotelling-Lawley / Roy) |
| `stats_rmse` | `broomExtra::perf_rmse` | RMSE / MAE / R² per group |
| `stats_roc_tidy` | `broomExtra::perf_roc` | ROC curve + trapezoidal AUC |
| `stats_breusch_pagan` / `stats_white_test` | `lmtest::bptest` | Heteroskedasticity tests |
| `stats_dffits` / `stats_cooks_distance` / `stats_leverage` | `broom::augment.lm` columns | Influence diagnostics |
| `stats_crossv_kfold` / `stats_crossv_mc` | `broomExtra::crossv_*` | Tidy cross-validation splits |

See [`docs/audit_vs_r_broom.md`](docs/audit_vs_r_broom.md) for the full
parity matrix (which verbs are covered, partial, or out-of-scope due to a
missing statsmodels analogue).

## Changelog

### Version 0.2.0 — 2026-09-02

### Added

- New `widyr` parity module with tidy pairwise/wide verbs:
  `pairwise_count`, `pairwise_cor`, `pairwise_dist`, `pairwise_similarity`,
  `pairwise_pmi`, `pairwise_delta`, `widely_svd`, `widely_kmeans`,
  `widely_hclust`, `widely`, `squarely`, and `cor_sparse`.
- Public exports for the full `widyr` surface from `broom_sm.__init__`.
- New integration coverage in `tests/test_widyr.py` for pairwise outputs,
  upper-triangle filtering, metric validation, sparse correlation, and
  clustering/SVD behavior.
- New `phreg` (Cox PH) support in `stats_tidy`/`stats_glance`, including
  synthesized `nobs`/`aic`/`bic` when they are missing from fitted results.
- New parity helpers:
  `stats_kendall_tidy`, `stats_coeftest`, `stats_manova_tidy`, `stats_rmse`,
  `stats_roc_tidy`, `stats_breusch_pagan`, `stats_white_test`,
  `stats_dffits`, `stats_cooks_distance`, `stats_leverage`,
  `stats_crossv_kfold`, and `stats_crossv_mc`.
- New AI workflow guide:
  [`docs/howto/ai-assistant.md`](docs/howto/ai-assistant.md).

### Changed

- GitHub Actions CI is now multi-job with:
  - matrix tests on Python 3.10/3.11/3.12
  - explicit extras install (`testing,viz,bayes`)
  - Sphinx docs build with warnings treated as errors
  - advisory `ruff` and `mypy` checks
- Fixed CI dependency installation by removing invalid `.[dev]`.
- `stats_tidy` and `stats_glance` now use shared coercion/synthesis helpers so
  numpy-backed statsmodels results are converted to robust tidy/glance output.

### Documentation

- Added [`docs/audit_vs_r_broom.md`](docs/audit_vs_r_broom.md), a detailed
  parity audit against `broom`, `broomExtra`, and `broom.mixed`.
- Updated `README.md`, `docs/index.md`, `docs/howto/index.md`, and
  `CONTRIBUTING.md` for the AI assistant playbook and CI expectations.
- Added parity test summary to `tests/test_parity.py` (34 new tests; 96 passed,
  coverage 93%).

### Version 0.1.3 — 2026-07-13

Quality fixes, visual/plotting diagnostics testing, and coverage expansion to 96%:

- **Fixed OLS Weights:** Changed the direct Ordinary Least Squares fitter registration to use WLS when weights are supplied, making weights functional rather than silent placebos.
- **Fixed `stats_augment` NaN alignment:** Rewrote alignment logic to assign pandas Series directly (relying on index alignment rather than `.values`), avoiding length mismatches when rows are dropped. Used pre-transformed exog values for predictions.
- **Optimized `stats_tidy` merges:** Replaced consecutive DataFrame merges on the `"term"` column with direct coefficient construction.
- **Dependency cleanup:** Moved `seaborn`, `matplotlib`, and `bayesian_bootstrap` to optional package extras, adding guarded imports and descriptive import errors.
- **Coverage expansion:** Created extensive tests for visual diagnostics, CLI parameters (`--index-col`), fallback paths, and mocked import environments, raising line coverage to 96% with all 62 tests passing.

### Version 0.1.2 — 2026-06-20

P0 fixes from the 2026-06-20 code review:

- `stats_augment` now validates index uniqueness for `data` and `new_data`, rejects overlapping indices, and aligns residuals / influence diagnostics position-wise for the in-sample path. The `.in_sample` flag is now a single boolean rather than a `set`-based membership test.
- `prepare_fit` now passes `freq_weights` to GLM-family fitters (Poisson, Gamma, Negative Binomial, Beta, etc.) and keeps `weights` for OLS.
- `boot_tidy`, `boot_glance`, and `boot_augment` raise `RuntimeError` when every bootstrap replication fails, instead of returning an empty DataFrame.
- `stats_residual_plot` validates that the target column `y` is numeric before passing it to plotting / `probplot`.
- `stats_vif` now emits a clear warning that the intercept is omitted, and handles no-intercept formulas consistently without adding a constant.

Selected P1 fixes in the same release:

- `anova_type` is validated to be 1, 2, or 3 in `stats_anova_tidy`.
- `stats_kruskal_tidy` validates that `group_col` and `value_col` exist.
- `stats_correlation_tidy` validates that requested `columns` exist and are numeric.
- `stats_formula` now quotes non-syntactic column names with `Q('...')`.
- `bayes_boot` validates `target_column` / `n_samples` and warns when NaN values are dropped.
- `stats_chisquare_plot` drops NaN categories before building the contingency table.
- Repository URLs in `setup.cfg` updated from `jcvall/broom-sm` to `ezraair555/broom-sm`.
- Removed the unused `src/extra_sm` package.

## Documentation

Full documentation (API, how-to guides, tutorials, and plot gallery) lives in `docs/`:

- **[Tutorials](docs/tutorials/index.md)** — End-to-end walkthroughs
- **[How-to Guides](docs/howto/index.md)** — Task-oriented recipes
- **[AI Assistant Workflow](docs/howto/ai-assistant.md)** — Deterministic patterns for coding agents
- **[API Reference](docs/api-reference.md)** — Complete function documentation
- **[Quick Start](docs/quickstart.md)** — Get started in 5 minutes

The rendered site is at <https://ezraair555.github.io/broom-sm/>.

## Contributing

We welcome contributions! Please review our [contributing guidelines](https://github.com/ezraair555/broom-sm/blob/main/CONTRIBUTING.md) and [Python Software Foundation code of conduct](https://www.python.org/psf/codeofconduct/).

For questions and discussions, please [post on GitHub Discussions](https://github.com/ezraair555/broom-sm/discussions). If you think you've encountered a bug, please [submit an issue](https://github.com/ezraair555/broom-sm/issues).

## License

MIT License — see [LICENSE.txt](https://github.com/ezraair555/broom-sm/blob/main/LICENSE.txt) for details.

## Acknowledgments

broom-sm draws inspiration from:
- [broom](https://broom.tidymodels.org/) (R) — Tidy model outputs
- [infer](https://infer.tidymodels.org/) (R) — Tidy statistical inference
- [pandas_flavor](https://github.com/Zsailer/pandas_flavor) — DataFrame method registration
- [statsmodels](https://www.statsmodels.org/) — Statistical modeling in Python
