Metadata-Version: 2.4
Name: missingly
Version: 1.0.0
Summary: A comprehensive package for missing data analysis and visualization.
Author-email: Ali Sadeghi Aghili <alisadeghiaghili@gmail.com>
License: MIT License
        
        Copyright (c) 2025 Ali Sadeghi Aghili
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/alisadeghiaghili/missingly
Project-URL: Bug Tracker, https://github.com/alisadeghiaghili/missingly/issues
Project-URL: Documentation, https://missingly.readthedocs.io
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: matplotlib>=3.7.0
Requires-Dist: seaborn>=0.12.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: scikit-learn>=1.2.0
Requires-Dist: jinja2>=3.1.0
Provides-Extra: docs
Requires-Dist: sphinx>=7.0.0; extra == "docs"
Requires-Dist: furo>=2023.9.10; extra == "docs"
Requires-Dist: nbsphinx>=0.9.0; extra == "docs"
Requires-Dist: numpydoc>=1.6.0; extra == "docs"
Requires-Dist: ipykernel; extra == "docs"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: upsetplot>=0.9.0; extra == "test"
Provides-Extra: interactive
Requires-Dist: plotly>=5.0.0; extra == "interactive"
Provides-Extra: rtl
Requires-Dist: arabic-reshaper>=3.0.0; extra == "rtl"
Requires-Dist: python-bidi>=0.4.2; extra == "rtl"
Provides-Extra: upset
Requires-Dist: upsetplot>=0.9.0; extra == "upset"
Provides-Extra: all
Requires-Dist: plotly>=5.0.0; extra == "all"
Requires-Dist: arabic-reshaper>=3.0.0; extra == "all"
Requires-Dist: python-bidi>=0.4.2; extra == "all"
Requires-Dist: upsetplot>=0.9.0; extra == "all"
Dynamic: license-file

# missingly

> **This README describes the v1.0.0+ public API. For historical experiments see the [`legacy-experiments`](https://github.com/alisadeghiaghili/missingly/tree/legacy-experiments) branch.**

> **Missing data analysis for pandas — batteries included.**

[![PyPI](https://img.shields.io/pypi/v/missingly)](https://pypi.org/project/missingly/)
[![Python](https://img.shields.io/pypi/pyversions/missingly)]()
[![CI](https://github.com/alisadeghiaghili/missingly/actions/workflows/ci.yml/badge.svg)](https://github.com/alisadeghiaghili/missingly/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/alisadeghiaghili/missingly/branch/main/graph/badge.svg)](https://codecov.io/gh/alisadeghiaghili/missingly)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![API stability](https://img.shields.io/badge/API-v1.0.0%20stable-brightgreen)](https://github.com/alisadeghiaghili/missingly/releases/tag/v1.0.0)

🌐 English | [Deutsch](README_DE.md) | [فارسی](README_FA.md)

missingly is a Python package for **diagnosing, visualising, and imputing
missing data** in pandas DataFrames. It provides:

- A fluent `df.miss.*` accessor that mirrors the ergonomics of the R `naniar` package.
- sklearn-compatible transformers (`MissinglyImputer`) for use inside `Pipeline`.
- One-shot HTML reports (`create_report`).
- Statistical tests for MCAR / MAR / MNAR mechanisms.
- Time-series-aware gap analysis and imputation.

### Multiple Imputation (advanced)

For statistically valid inference after imputation, generate *m* datasets
with `impute_mice(..., n_imputations=m)` and pool the model results using
Rubin's Rules via the utilities in `missingly.mi`:

```python
import numpy as np
import pandas as pd
from missingly import impute_mice
from missingly.mi import pool_scalar_estimates
from sklearn.linear_model import LinearRegression

# 1. Generate m imputed datasets
dfs = impute_mice(df, n_imputations=5)

# 2. Fit model on each imputed dataset
beta1_ests, beta1_vars = [], []
for d in dfs:
    reg = LinearRegression().fit(d[["x"]], d["y"])
    beta1_ests.append(float(reg.coef_[0]))
    resid = d["y"] - reg.predict(d[["x"]])
    ss_x = float(((d["x"] - d["x"].mean()) ** 2).sum())
    beta1_vars.append(float(np.var(resid, ddof=2)) / ss_x)

# 3. Pool with Rubin's Rules
result = pool_scalar_estimates(beta1_ests, beta1_vars)
print(f"Pooled beta1 = {result['q_bar']:.3f}  (total var = {result['t']:.4f})")
```

For multivariate models use `pool_linear_regression_results(coefs, covs)`
which accepts arrays of shape `(m, p)` and `(m, p, p)` and returns a
pooled coefficient vector plus a pooled covariance matrix.

---

## Public API (v1)

The symbols below are the **stable, supported API surface** for v1.
Breaking changes to these will be announced via a major-version bump.

### `df.miss.*` accessor

```python
import missingly  # registers df.miss automatically
import pandas as pd
import numpy as np

df = pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, 6]})

df.miss.n_miss()            # 3  — total missing count
df.miss.pct_miss()          # 50.0  — % missing across whole DataFrame
df.miss.miss_var_summary()  # per-column summary table
df.miss.vis_miss()          # missingness matrix visualisation
df.miss.impute(strategy="mean")  # returns imputed DataFrame
```

### Summary & Diagnosis

```python
import missingly as mi

mi.n_miss(df)             # int — total missing count
mi.pct_miss(df)           # float — overall % missing
mi.miss_var_summary(df)   # pd.DataFrame — per-column breakdown
mi.miss_case_summary(df)  # pd.DataFrame — per-row breakdown
mi.mcar_test(df)          # Little's MCAR test result
mi.mar_mnar_test(df)      # MAR vs MNAR indicator
mi.diagnose_missing(df)   # mechanism + recommendation dict
```

### Visualisation

The visualisation layer lives in `missingly.visualisation` and is re-exported
through `missingly.visualise` for backwards compatibility.

> **Module layout**
> | Module | Contents |
> |---|---|
> | `missingly.visualisation.static` | All matplotlib-based functions |
> | `missingly.visualisation.interactive` | All Plotly backends (called when `interactive=True`) |
> | `missingly.visualisation._base` | Shared helpers: `_rtl_safe`, `_safe_labels`, `_nullity`, `_pct_labels` |
> | `missingly.visualise` | Thin re-export facade — use this in application code |

#### Basic

```python
import missingly as mi
import pandas as pd, numpy as np

df = pd.DataFrame({
    "age":    [25, np.nan, 47, 33, np.nan],
    "income": [50000, 62000, np.nan, np.nan, 71000],
    "city":   ["A", "B", np.nan, "A", "C"],
})

mi.vis_miss(df)          # annotated tile matrix with per-column % labels
mi.matrix(df)            # raw presence/absence heatmap
mi.bar(df)               # bar chart: count of missing per column
mi.miss_case(df)         # bar chart: count of missing per row
mi.miss_var_pct(df)      # horizontal bars: % missing per variable, sorted
```

#### Patterns

```python
mi.miss_patterns(df)     # horizontal bars: top-N most frequent missingness patterns
mi.miss_cooccurrence(df) # symmetric heatmap: how often two columns miss together
mi.upset(df)             # UpSet plot of intersecting missingness sets
                         # returns dict of Axes: {"intersections", "matrix", "totals"}
```

#### Correlation / Clustering

```python
# Nullity-correlation heatmap (Pearson on binary missingness indicators).
# Delegates to data_quality_toolkit.visualization.correlation_heatmap when
# that package is installed; falls back to a pure-seaborn renderer otherwise.
mi.heatmap(df)
mi.heatmap(df, mask_insignificant=True)  # grey out non-significant cells

# Hierarchical clustering of rows by missingness pattern.
# Returns a single matplotlib.axes.Axes (not a dict).
ax = mi.miss_cluster(df)

# Dendrogram of variables clustered by nullity correlation.
mi.dendrogram(df)
```

#### Interactive

Pass `interactive=True` to any function below to get a Plotly figure
that can be panned, zoomed, and exported to HTML.
When Plotly is not installed the function silently falls back to the
static backend.

```python
mi.vis_miss(df, interactive=True)
mi.heatmap(df, interactive=True)
mi.matrix(df, interactive=True)
mi.bar(df, interactive=True)
mi.miss_var_pct(df, interactive=True)
mi.miss_cooccurrence(df, interactive=True)
mi.miss_case(df, interactive=True)
mi.upset(df, interactive=True)
mi.miss_patterns(df, interactive=True)
```

### Imputation

```python
mi.impute_mean(df)           # mean imputation
mi.impute_median(df)         # median imputation
mi.impute_mode(df)           # mode imputation
mi.impute_knn(df)            # k-NN imputation (Euclidean distance, numeric-safe)
mi.impute_mice(df)           # MICE (IterativeImputer + BayesianRidge)
mi.impute_rf(df)             # Random Forest imputation
mi.impute_gb(df)             # Gradient Boosting imputation

# Multiple Imputation — generate m datasets for Rubin pooling
dfs = mi.impute_mice(df, n_imputations=5)
```

#### KNN with Gower distance (mixed numeric + categorical)

By default `impute_knn` ordinal-encodes categorical columns and uses
Euclidean distance — fast and suitable for mostly-numeric datasets.

For datasets dominated by categorical columns, pass `metric="mixed"` to
use **Gower distance** instead.  Gower treats numeric and categorical
columns correctly: numeric columns are normalised by their range, nominal
columns are compared by exact match.

```python
df_mixed = pd.DataFrame({
    "age":    [25, np.nan, 35, 40],
    "city":   ["London", "Paris", None, "Berlin"],
    "grade":  ["A", "B", "A", None],
})

# Euclidean KNN (default) — fast, ordinal-encodes categoricals
result = mi.impute_knn(df_mixed, n_neighbors=3)

# Gower KNN — statistically sound for heavy-categorical data
result = mi.impute_knn(df_mixed, n_neighbors=3, metric="mixed")
```

> **Performance note:** Gower distance is **O(n²)** in both memory and
> runtime.  Avoid `metric="mixed"` for datasets with more than ~10 000 rows.

The same `metric` parameter is available on `MissinglyImputer`:

```python
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("impute", mi.MissinglyImputer(strategy="knn", metric="mixed", n_neighbors=5)),
    ("model",  LogisticRegression()),
])
pipe.fit(X_train, y_train)
```

### Time-series missingness

For time-indexed data, missingly provides gap-aware summary statistics,
visualisation helpers, and interpolation-based imputation.

```python
import missingly as mi
import pandas as pd
import numpy as np

# Build a temperature series with some gaps
index = pd.date_range("2024-01-01", periods=14, freq="D")
temp  = [5.1, 4.8, np.nan, np.nan, 6.2, 6.5, np.nan, 7.0,
         7.3, np.nan, np.nan, np.nan, 8.1, 8.4]
ts = pd.DataFrame({"temp": temp}, index=index)

# 1. Summarise gaps
summary = mi.miss_ts_summary(ts, col="temp")
print(summary)
# n_miss          5
# n_gaps          3
# mean_gap_len    1.67
# max_gap_len     3
# longest_gap_start  2024-01-10
# longest_gap_end    2024-01-12

# 2. Visualise missingness over the time axis
ax = mi.vis_ts_miss(ts)

# 3. Impute with linear interpolation
ts_filled = mi.impute_ts(ts, strategy="linear")
print(ts_filled.isnull().sum())  # temp    0
```

**Available strategies for `impute_ts`:** `ffill`, `bfill`, `linear`, `time`,
`spline`. Use `limit=n` to cap how many consecutive NaNs are filled.

```python
# Fill at most 2 consecutive NaNs, leave longer gaps as-is
ts_partial = mi.impute_ts(ts, strategy="linear", limit=2)
```

**Gap inspection with `gap_table`:**

```python
from missingly.timeseries import gap_table

gt = gap_table(ts)
print(gt)
#    column  gap_start   gap_end  gap_length
# 0    temp 2024-01-03 2024-01-04           2
# 1    temp 2024-01-07 2024-01-07           1
# 2    temp 2024-01-10 2024-01-12           3
```

### sklearn Pipeline integration

```python
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("impute", mi.MissinglyImputer(strategy="knn")),
    ("model",  LogisticRegression()),
])
pipe.fit(X_train, y_train)
```

---

## Installation

```bash
# Core package
pip install missingly

# With interactive Plotly charts
pip install missingly[interactive]

# With Persian / Arabic (RTL) support for static matplotlib plots
# Required when column names or labels contain Persian/Arabic characters
pip install missingly[rtl]

# Everything (interactive + RTL)
pip install missingly[all]
```

> **Persian/Arabic users:** static matplotlib plots require `missingly[rtl]`
> (installs `arabic-reshaper` and `python-bidi`) **plus** a compatible font
> such as [Vazirmatn](https://fonts.google.com/specimen/Vazirmatn) installed
> on your system.  Interactive Plotly charts (`interactive=True`) work
> correctly out of the box with no extra dependencies.

---

## License

MIT — see [LICENSE](LICENSE).
