Metadata-Version: 2.4
Name: mirrorml
Version: 0.1.0
Summary: Static detection of training-serving skew in ML feature pipelines.
Author: MirrorML contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-engineering,feature-pipelines,machine-learning,mlops,static-analysis,training-serving-skew
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: libcst>=1.1
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: sqlglot>=23
Requires-Dist: typer>=0.12
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars>=0.20; extra == 'polars'
Description-Content-Type: text/markdown

# MirrorML

> Find the bugs that make a machine learning model behave differently in testing
> than it does in production.

MirrorML checks whether two data pipelines that are supposed to compute the same
thing actually do.

Here is the problem it solves. A machine learning model learns from "features":
the input numbers fed into it. Those numbers are usually computed twice, by two
separate pieces of code: once over saved historical data while the model is being
trained, and again over live data while the model is running for real users.
These two pieces of code are often written by different people, in different
languages, at different times. When they fall out of sync even slightly (one
rounds a number differently, one fills in missing values differently, one
averages over a different time range), the model quietly makes worse predictions
in production than it did in testing. This mismatch is called **training-serving
skew**. Because nothing crashes and no error is raised, it can go unnoticed for a
long time.

MirrorML reads both pieces of code and builds a short summary of what each one
does, called a *fingerprint*. It then compares the two summaries. If they match,
the pipelines are equivalent and there is no skew. If they do not, MirrorML points
to the exact step that differs and labels what kind of difference it is (for
example: a different time zone, a different rounding, adding numbers up where the
other averages them, or handling missing values differently). It works by reading
the code, so it does not need to run the pipelines or have any real data.

MirrorML checks only the feature code, the part that prepares a model's inputs.
Whatever you train the model with afterward (PyTorch, JAX, XGBoost, or anything
else) runs later and is outside what MirrorML looks at.

## Status

Early development (`v0.1.0`). The core works end to end. MirrorML can read
pipelines written in pandas, Polars, or SQL (three common ways to work with
tables of data in Python and databases), summarize each one, compare them, and
point to the step that differs. The same pipeline written in any of the three
produces the same summary, so a training pipeline written in one language can be
checked against a production pipeline written in another.

MirrorML recognizes all fifteen kinds of difference it sets out to catch; the
[category support matrix](docs/concepts/category_support.md) lists exactly which
pipeline patterns trigger each one. As a second, independent check, it can also
run both pipelines on a small batch of generated data and compare the actual
results. So far it has been tested mainly
on examples we created ourselves, plus four real-world bugs rebuilt from
published reports. Testing on pipelines collected from public projects is still
to come, so there are not yet accuracy numbers from real-world use.

## Install

MirrorML is not yet published to PyPI. From a clone:

```bash
uv sync --all-extras --dev
```

Once published:

```bash
uv add mirrorml              # core
uv add 'mirrorml[pandas]'    # pandas tracer
uv add 'mirrorml[polars]'    # Polars tracer
```

The core install does not require pandas or Polars. Each one is loaded only when
you actually use its tracer, so `import mirrorml` stays fast.

## Quickstart

The example below takes a training pipeline written in pandas and a production
pipeline written in SQL, and confirms they compute the same thing. Then it
changes the SQL side to add scores up instead of averaging them, and MirrorML
reports the difference and where it is.

```python
from mirrorml import diff, trace_pandas, trace_sql

EVENTS = (("uid", "int64"), ("score", "float64"))


def offline(df):
    return df[df["score"] > 0].groupby("uid").agg({"score": "mean"})


pandas_fp = trace_pandas(offline, input_schema=EVENTS, source_name="events")
sql_fp = trace_sql(
    "SELECT uid, AVG(score) AS score FROM events WHERE score > 0 GROUP BY uid",
    schemas={"events": EVENTS},
)

assert diff(pandas_fp, sql_fp) == ()  # the two pipelines are equivalent

# Now the online side sums instead of averaging:
sql_skewed = trace_sql(
    "SELECT uid, SUM(score) AS score FROM events WHERE score > 0 GROUP BY uid",
    schemas={"events": EVENTS},
)
for d in diff(pandas_fp, sql_skewed):
    print(d.category, "|", d.detail)
# aggregation_function | aggregation 'score': function 'mean' vs 'sum'
```

The Polars tracer takes the same shape; its pipeline receives the frame plus a
`pl` expression namespace:

```python
from mirrorml import trace_polars


def offline(lf, pl):
    return lf.filter(pl.col("score") > 0).group_by("uid").agg(pl.col("score").mean())


polars_fp = trace_polars(offline, input_schema=EVENTS, source_name="events")
```

## CLI

```bash
# Emit one side of a pair as canonical fingerprint JSON
mirrorml trace path/to/pair --side offline -o offline.json

# Diff two on-disk fingerprints (exit 1 if they diverge)
mirrorml diff offline.json online.json

# Trace both sides of a pair, diff them, and check the result against the
# differences the pair says to expect; exits with an error code on a
# mismatch, so it can run as an automated check
mirrorml verify path/to/pair
```

A "pair" is a directory containing a `meta.yaml` (which names each side's
language, source file, and schema) plus the source files themselves. The
MirrorBench pairs under `bench/pairs/` are runnable examples.

## Public API

The seven names exported from `mirrorml` are the entire stable surface:

| Name | Kind | Status |
|---|---|---|
| `Fingerprint` | Pydantic model | Stable |
| `fingerprint(...)` | constructor | Stable |
| `Divergence` | Pydantic model | Stable |
| `diff(...)` | function | Implemented |
| `trace_pandas(...)` | function | Implemented |
| `trace_sql(...)` | function | Implemented |
| `trace_polars(...)` | function | Implemented |

Anything not listed is internal and may change without notice.

## Development

```bash
uv sync --all-extras --dev
uv run ruff check .
uv run ruff format --check .
uv run mypy
uv run pytest -q
```

## License

Apache-2.0. See [LICENSE](./LICENSE).
