Metadata-Version: 2.4
Name: nullsweep
Version: 1.0.0
Summary: A comprehensive Python package for managing and analyzing missing data in pandas DataFrames, starting with detection and expanding to complete handling.
Author-email: Okan Yenigun <okanyenigun@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/okanyenigun/nullsweep
Project-URL: Repository, https://github.com/okanyenigun/nullsweep
Project-URL: Changelog, https://github.com/okanyenigun/nullsweep/blob/main/CHANGE.md
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENCE.md
Requires-Dist: pandas<3,>=2.0
Requires-Dist: polars<2,>=1.0
Requires-Dist: pyarrow>=14
Requires-Dist: numpy>=1.23
Requires-Dist: scipy>=1.10
Requires-Dist: scikit-learn>=1.3
Requires-Dist: statsmodels>=0.14
Requires-Dist: matplotlib>=3.6
Requires-Dist: seaborn>=0.12
Requires-Dist: missingno>=0.5
Requires-Dist: upsetplot>=0.8
Requires-Dist: wordcloud>=1.9
Provides-Extra: dev
Requires-Dist: pytest==8.2.2; extra == "dev"
Requires-Dist: build==1.2.2.post1; extra == "dev"
Requires-Dist: twine==6.2.0; extra == "dev"
Dynamic: license-file

# NullSweep

[![PyPI version](https://img.shields.io/pypi/v/nullsweep.svg)](https://pypi.org/project/nullsweep/)
[![Python versions](https://img.shields.io/pypi/pyversions/nullsweep.svg)](https://pypi.org/project/nullsweep/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENCE.md)

**Diagnose missing data before you impute it.** NullSweep reads the *shape* and the *mechanism* of the holes in your DataFrame, then gives you a single interface to fill them with a method the evidence actually justifies.

Works with **pandas** and **Polars** — both installed by default, no extras to remember.

## Why

`df.fillna(df.mean())` is a guess. Two questions, answerable straight from the data, turn it into a decision:

| Question | What it tells you | API |
|---|---|---|
| **Where** are the gaps? (pattern) | Which methods are computationally applicable | `detect_global_pattern` |
| **Why** are they there? (mechanism) | Whether a fill will be honest or biased | `detect_feature_pattern` |
| **What** now? | The smallest tool the evidence supports | `impute_nulls` |

Pattern is about tractability; mechanism is about correctness. You need both.

## Installation

```bash
pip install nullsweep
```

A single batteries-included install: pandas and Polars support, the KNN/MICE/regression imputers, the MAR/MCAR statistical tests, and the visualizations all come with it.

## Quickstart

```python
import nullsweep as ns

shape, details = ns.detect_global_pattern(df)              # "univariate" | "monotone" | "non-monotone"
mech, details = ns.detect_feature_pattern(df, "income")    # "MAR_evidence" | "MCAR_consistent" | "undetermined"

ns.plot_missing_values(df, "matrix")                       # look before you test

df = ns.impute_nulls(df, column="income", strategy="mice") # strategy chosen from the evidence
```

`impute_nulls` returns a **new** frame by default — your original is never mutated.

## API

| Function | Purpose |
|---|---|
| `detect_global_pattern(df)` | Classify the dataset's missingness arrangement |
| `detect_feature_pattern(df, column, ...)` | Classify one column's missingness mechanism (MAR screen + Little's MCAR test) |
| `impute_nulls(df, column, strategy, ...)` | One-shot imputation across all strategies |
| `NullSweepImputer(...)` | Stateful `fit`/`transform` transformer for train/test workflows |
| `plot_missing_values(df, plot_type, ...)` | Nine views of the missingness, returns a Matplotlib figure |

## Detecting patterns (where)

```python
pattern, details = ns.detect_global_pattern(df)
```

| Pattern | Structure | What it unlocks |
|---|---|---|
| `univariate` | One column missing, the rest complete | A single prediction problem — one model, done |
| `monotone` | Nested, orderable holes (study drop-out) | A sequence of ordinary regressions — one deterministic pass |
| `non-monotone` | Scattered, circular dependencies | Iterative methods — `mice`, `knn` |

For `monotone`, `details["matrix"]` shows the nesting: each cell answers *"whenever the row variable is missing, is the column variable also missing?"*

```
       A      B      C
A  False   True   True      # A missing ⇒ B and C missing
B  False  False   True      # A ⊆ B ⊆ C — that nesting is monotonicity
C  False  False  False
```

For `univariate`, `details["column"]` names the offending column.

## Detecting mechanisms (why)

```python
label, details = ns.detect_feature_pattern(df, "target")
```

The column's missingness indicator is screened against every other column with logistic regression. If nothing predicts it, Little's (1988) MCAR test runs on the numeric columns to separate "looks random" from "we can't tell."

| Label | What it means | What to do |
|---|---|---|
| `MAR_evidence` | An observed column predicts the missingness | Impute with a model that conditions on it — `regression`, `mice`, `knn` |
| `MCAR_consistent` | No predictor, and MCAR could not be rejected | Simple fills or deletion are safe; nothing fancy required |
| `undetermined` | No predictor, and MCAR was rejected or untestable | Don't assume — suspect MNAR; `flag` it, or run a sensitivity analysis |

`MCAR_consistent` means *failed to reject*, not *proven*. `details` carries `mar_predictors` (a per-predictor evidence map) and, when it ran, `mcar_test` (statistic, degrees of freedom, p-value, and a plain-language message).

Useful options:

```python
# Catch confounded MAR with a single joint model instead of marginal screens
ns.detect_feature_pattern(df, "target", multivariate=True)

# The pseudo R-squared gate (default 0.2) is deliberately strong and can hide
# weak-but-real signals. Lower it to increase sensitivity:
ns.detect_feature_pattern(df, "target", pseudo_r_squared_threshold=0.05)

# Numeric predictors only / skip the MCAR step
ns.detect_feature_pattern(df, "target", include_categorical=False, run_mcar_test=False)
```

Little's test is also available directly: `from nullsweep.patterns.mcar.little import little_mcar_test`.

## Imputing (what)

```python
df = ns.impute_nulls(df, column="age", strategy="mean")
df = ns.impute_nulls(df, column=["age", "income"], strategy="mice")
df = ns.impute_nulls(df)                                    # strategy="auto" on every column with gaps
df = ns.impute_nulls(df, strategy="listwise", threshold=2)  # drop rows with >= 2 missing values
df = ns.impute_nulls(df, column="city", strategy="constant", fill_value="Unknown")
```

### Strategy families

| Family | Strategies | For | Best when |
|---|---|---|---|
| Statistical | `mean`, `median` | numeric | MCAR — a central value is unbiased |
| Directional | `interpolate`, `forwardfill`, `backfill` | ordered numeric/categorical | time series, sorted data |
| Model-based | `knn`, `mice`, `regression` | numeric | MAR — condition on the other columns |
| Frequency | `most_frequent`, `least_frequent`, `constant` | categorical | filling labels |
| Structural | `flag`, `delete_column`, `listwise` | any | mark or remove instead of inventing |
| Automatic | `auto` | any | let each column's type and shape pick |

Notes worth knowing:

- **`knn` / `mice` / `regression`** operate on numeric columns and reject non-numeric targets. When a target column is given, complete numeric predictors are still used as context but only the target is written back. Encode categoricals first, or use a frequency strategy.
- **`flag`** adds a `<column>_missing` indicator per column. With `column=None` it flags only columns that have gaps; `include_all_columns=True` flags every column.
- **`delete_column` / `listwise`** take a `threshold`: floats are missing-value proportions, integers are counts, and equality is deleted.
- **Directional fills leave edge gaps** — `backfill` cannot fill a trailing NaN. Check the result rather than assuming it is complete.
- **`auto`** uses interpolation for continuous columns only when the index is datetime/timedelta (override with `strategy_params={"allow_ordered_interpolation": True}`), and treats integer/boolean-coded numeric columns as categorical (disable with `{"detect_numeric_categorical": False}`).

### Parameters

| Parameter | Type | Description |
|---|---|---|
| `df` | `pd.DataFrame` \| `pl.DataFrame` | Input frame. Must not be empty. |
| `column` | `str` \| `Iterable[str]` \| `None` | Target column(s). `None` selects every column with missing values. |
| `strategy` | `str` | One of the strategies above. Defaults to `"auto"`. |
| `fill_value` | `Any` | Constant used when `strategy="constant"`. |
| `strategy_params` | `dict` \| `None` | Strategy configuration, e.g. `{"method": "linear", "order": 2}` for `interpolate`. |
| `in_place` | `bool` | Mutate the input instead of returning a copy (pandas only; Polars warns and returns a new frame). Defaults to `False`. |
| `**kwargs` | `Any` | Handler-specific options, e.g. `n_neighbors` for `knn`, `threshold` for `listwise`. |

### Train/test workflows

`impute_nulls` recomputes its fill values from whatever frame it is handed — applying it to a test set leaks. Use the transformer instead:

```python
from nullsweep import NullSweepImputer

imputer = NullSweepImputer(column="income", strategy="mean").fit(train)
train_imputed = imputer.transform(train)
test_imputed = imputer.transform(test)   # filled with TRAIN's statistics
```

It learns means, medians, modes, the KNN/MICE/regression models, and the per-column choice made by `auto` at `fit` time, then applies them unchanged. Directional strategies carry no cross-frame state; `listwise` learns a per-row mask, so use `fit_transform` for it.

## Visualizing

```python
fig = ns.plot_missing_values(df, "heatmap", figsize=(8, 4), cmap="magma")
```

| Question | Plot types |
|---|---|
| Where are the gaps? | `heatmap`, `matrix` |
| How much is missing? | `percentage`, `histogram`, `wordcloud` |
| Do the gaps travel together? | `correlation`, `dendrogram`, `upset_plot` |
| Are the incomplete rows different? | `pair` |

Each returns a Matplotlib figure and accepts the underlying plot function's keyword arguments. None of them prove anything — they point your eye at the structure so you know which test to run first.

## pandas and Polars

Every public function accepts either a pandas or a Polars DataFrame, and the imputers return the same type they were given. Polars frames are immutable, so `in_place=True` warns and returns a new frame.

## Contributing

Contributions are welcome. Please submit pull requests, open issues, or suggest improvements.

## License

[MIT](LICENCE.md)
