Metadata-Version: 2.4
Name: mendr
Version: 0.4.0
Summary: Bounded, auditable, reversible data repair — the layer that runs after your validator.
Project-URL: Repository, https://github.com/Pi-Wi/mendr
Project-URL: Documentation, https://github.com/Pi-Wi/mendr/tree/main/docs
Author: Pi-Wi
License-Expression: MIT
License-File: LICENSE
Keywords: audit,data-quality,data-repair,pandas,polars,reversible
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: great-expectations>=1.0; extra == 'dev'
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pandera>=0.20; extra == 'dev'
Requires-Dist: polars>=1.0; extra == 'dev'
Requires-Dist: pyarrow>=15.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: gx
Requires-Dist: great-expectations>=1.0; extra == 'gx'
Requires-Dist: pandas>=2.0; extra == 'gx'
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == 'pandas'
Provides-Extra: pandera
Requires-Dist: pandas>=2.0; extra == 'pandera'
Requires-Dist: pandera>=0.20; extra == 'pandera'
Provides-Extra: polars
Requires-Dist: polars>=1.0; extra == 'polars'
Description-Content-Type: text/markdown

# mendr

Bounded, auditable, reversible data repair — the layer that runs *after* your validator and turns "here's what's wrong" into "here's the fix, with receipts, and an undo."

## Status

**v0.4.0 — release-ready, upload pending.** The full loop works: repair → ledger → report → revert, on pandas *and* polars, with pandera *and* Great Expectations adapters, cross-column rules, quarantine, a CLI (including `mendr diff` for run-to-run drift), CI repair budgets, and benchmarks ([benchmarks/](benchmarks/): ~200k repairs on a 1M-row frame in seconds, exact revert included). The ledger file format is a documented, stable contract ([docs/ledger-format.md](docs/ledger-format.md)), ledgers are tested to revert **across** backends (pandas-written ledger → polars frame and vice versa), and columns mendr can't faithfully record are refused loudly at ingest instead of silently misbehaving later. Docs live in [docs/](docs/), a ten-row walkthrough in [examples/one_ledger_two_uses.py](examples/one_ledger_two_uses.py), a 195k-row real-data showcase in [examples/nyc311_bounded_repair.py](examples/nyc311_bounded_repair.py), a 2.96M-row parquet showcase in [examples/nyc_taxi_bounded_repair.py](examples/nyc_taxi_bounded_repair.py) (480k repairs, a live budget refusal, byte-identical pandas/polars ledgers), the roadmap in `TODO.md`, and release history in [CHANGELOG.md](CHANGELOG.md).

## The gap

Validators (pandera, Great Expectations) are good at telling you what's wrong and then stopping. "Cleaning" tools are good at silently fixing things you can't inspect or undo. Almost nobody owns the middle: repair that is *declared*, *bounded*, and *fully auditable*.

That middle is the whole point of mendr. Every change it makes is a named action you asked for, constrained by a limit you set, recorded in a ledger you can read, diff, and reverse. No magic. Nothing changes that you didn't authorize.

mendr is explicitly **not** a validator. It doesn't compete with pandera or GE — it runs after them (or after its own minimal built-in checks) and does the one thing they don't: repair you can trust in production.

## The core idea: one ledger, two uses

Every repair mendr makes is recorded as an atomic entry — which rule fired, which row and column, the old value, the new value, and why.

That single artifact is both halves of the product:

- **Read it forward** → it's the audit report. Every change, explained.
- **Replay it inverted** → it's the undo log. `.revert()` round-trips the frame back to where it started.

Reversibility isn't a second bookkeeping system kept in sync with the report — it *falls out* of the report. The record is the product.

## Quick look

```python
import pandas as pd
import mendr as md

df = pd.read_csv("messy.csv")

policy = md.Policy([
    md.Rule(
        column="age",
        constraint=md.InRange(0, 120),
        remediation=md.Clip(),
        bound=md.MaxFraction(0.05),      # repair at most 5% of rows, else fail
    ),
    md.Rule(
        column="signup_date",
        constraint=md.IsType("datetime"),
        remediation=md.Cast(),
        bound=md.MaxRows(50),
    ),
    md.Rule(
        column="email",
        constraint=md.NotNull(),
        remediation=md.DropRow(),
        bound=md.MaxFraction(0.10),
        reason="email is required downstream",
    ),
])

result = md.repair(df, policy)

result.frame                     # the repaired dataframe
print(result.report())           # human-readable summary of every change
result.ledger.to_json("repairs.json")

original = result.revert()       # round-trips exactly back to df
```

If a rule would need to repair more than its bound allows, mendr **stops** — it does not silently over-repair. Bounded means bounded.

The same loop runs from the command line, budget and all:

```bash
mendr check  data.csv --policy policy.json           # exit 3 if violations exist
mendr repair data.csv --policy policy.json \
    --out repaired.csv --ledger repairs.json \
    --quarantine quarantined.csv --max-repairs 100
mendr revert repaired.csv --ledger repairs.json      # undo, from the ledger file alone
mendr diff   yesterday.json repairs.json             # exit 4 if repair activity drifted
```

## Concepts

**Rule** — a constraint + a remediation + a bound. The constraint says what must be true (`InRange`, `IsType`, `NotNull`, `OneOf`, `MatchesPattern`, `Unique`, `MaxLength`, cross-column `ColumnCompare`, …); the remediation is the single named action taken when it isn't (`Clip`, `Cast`, `Fill`, `Replace`, `Truncate`, `Round`, `Lowercase`, `NormalizeWhitespace`, `DropRow`, `Quarantine`); the bound caps how much repair is permitted before mendr refuses and raises.

**Policy** — an ordered set of rules applied to a frame.

**Ledger** — the append-only record of every atomic change. Cell edits store `(rule_id, row_id, column, old, new, reason)`. Row drops store the whole row so a revert can re-insert it — with a `destination` marker that distinguishes deleted rows from **quarantined** ones (moved to a side frame, `result.quarantine`, for human review instead of being nulled or deleted). This one structure is the audit report and the undo log — and `mendr diff` compares two of them to catch run-to-run drift in CI.

**Row identity** — mendr owns row identity. It assigns a stable row id at ingest (positional, or a key column you declare) and the ledger references *that*, never the native pandas index. This keeps the ledger portable and backend-agnostic — the same audit format works whether the frame came from pandas or polars.

**Backends** — the engine never touches a dataframe library directly. It talks to a small adapter interface (get column, cast, drop row, set cell, materialize). pandas and polars both ship; same data + same policy produce a byte-identical ledger on either. Your data stays in its native format — no forced conversion, no dtype surprises on ingest.

**Validators** — pluggable. The built-in constraints cover the basics; `Satisfies` / `SatisfiesRow` wrap any predicate; `mendr.integrations.pandera` turns a `pandera.Check` — or a whole `DataFrameSchema` — into mendr constraints, and `mendr.integrations.gx` does the same for Great Expectations (one expectation, or a whole `ExpectationSuite`). Your validator stays the authority on what's wrong; you declare what mendr may do about it.

## Design principles

**Deterministic.** Same frame and same policy produce the same changes every time.

**Bounded.** Every remediation is capped. Exceeding the cap is a hard stop, not a silent decision.

**Legible.** Every change is a named action with a reason, recorded in a format a human can read and a CI job can diff.

**Reversible.** Any repair run can be undone from its ledger alone.

## Non-goals (for the initial release, by design)

- **No LLM-based repair.** The trust wedge is that you can predict exactly what mendr will do.
- **No ML imputation models.** Fills are declared strategies (constant, forward-fill, median-with-flag) — not learned guesses.
- **No fuzzy / "smart" cleaning.** If it isn't a named, bounded action, mendr doesn't do it.
- **Not a validator.** mendr repairs; it leans on validators (built-in or pluggable) to decide what's wrong.

*LLM-assisted rule **suggestion** — proposing a policy for a human to approve — is a possible far-future feature, and would live strictly outside the repair path. Never in it.*

## Where it fits

mendr is a sibling to [agentrec](https://github.com/Pi-Wi/agent-rec) — same ethos, different domain. agentrec records LLM API traffic so CI can catch regressions; mendr records data repairs so humans and CI can trust them. Both make the *record* the product, both are git-diffable, and both are built to fail a build when something drifts past a declared budget.

## Install

Not on PyPI yet (release prep is done — see `TODO.md`). From source:

```bash
git clone https://github.com/Pi-Wi/mendr && cd mendr
pip install .[pandas]        # or .[polars], .[pandera], .[gx], .[dev]
```

The core has zero runtime dependencies; backends bring their own library.

## License

MIT.
