Metadata-Version: 2.4
Name: pandatrace
Version: 1.0.0.post1
Summary: Provenance tracking for pandas pipelines: what each operation did, which columns it created, and where the nulls came from
Keywords: pandas,provenance,lineage,data-quality,dataframe,debugging
Author: Ryushin Wells
Author-email: Ryushin Wells <ryushin.wells@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Debuggers
Classifier: Typing :: Typed
Requires-Dist: pandas>=2.2
Requires-Python: >=3.11
Project-URL: Repository, https://github.com/Pikaryu729/pandatrace
Project-URL: Documentation, https://pikaryu729.github.io/pandatrace/
Project-URL: Changelog, https://github.com/Pikaryu729/pandatrace/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/Pikaryu729/pandatrace/issues
Description-Content-Type: text/markdown

# pandatrace

Provenance tracking for pandas pipelines — what each operation did to your
frame, which columns it created, and where the nulls came from.

The goal is that after a long chain of transformations you can trace a frame
back through its whole history, even across however many intermediate
variables you bound along the way.

`TracedDataFrame` automatically records a whitelisted set of operations —
both `df = df.sort_index()` reassignment and `df.pipe(f).sort_index()...`
chaining are tracked correctly into one shared ledger. Indexing (`.loc`,
`.iloc`, `df[...]`) and direct mutation aren't covered — see
[Limitations](#limitations) for the honest list.

## Installation

Python **3.11+**, pandas **2.2+**. Fully typed.

```bash
pip install pandatrace
```

## Usage

`TracedDataFrame` is a `pd.DataFrame` subclass — use it like a normal
DataFrame, and a whitelisted set of operations get recorded automatically,
whichever style you write them in:

```python
import pandas as pd
from pandatrace import TracedDataFrame

orders = TracedDataFrame(
    {
        "id": [1, 2, 3, 4],
        "price": [10.0, 20.0, None, 40.0],
        "qty": [1, 2, 3, 4],
    },
    source="orders.csv",
)
customers = pd.DataFrame({"id": [1, 2, 3], "region": ["eu", "us", "eu"]})

result = orders.merge(customers, on="id", how="left").assign(
    total=lambda d: d["price"] * d["qty"]
)

for step in result.ledger.steps:
    print(
        f"{step.index}. {step.op:8} rows={step.row_delta} "
        f"new={step.new_columns} nulls={step.nulls_introduced()}"
    )
```

```
0. read     rows=None new=() nulls={}
1. merge    rows=0 new=('region',) nulls={'region': 1}
2. assign   rows=0 new=('total',) nulls={'total': 1}
```

The left join brought in a `region` that is null for the unmatched order, and
the `total` inherited the null from `price` — both attributed to the step that
introduced them, with no manual bookkeeping. This works identically whether
you write it as one chain (above), reassign at every step
(`orders = orders.merge(...); orders = orders.assign(...)`), or split the
chain across `.pipe()` calls — the same `Ledger` is shared across every frame
descended from the same original, however many intermediate variables you
bind or leave unbound. You can then trace a column exactly as before:

```python
result.ledger.parents_of("total")  # ('price', 'qty')
[s.op for s in result.ledger.origin_of("region")]  # ['merge']
```

`Ledger.report()` renders the whole history as a plain-text table — one line
per step, with row deltas, column churn, and the nulls each step introduced —
and in a notebook a bare `ledger` at the end of a cell renders the same table
as HTML. `Ledger.to_json()` exports it (plain ints only, nothing
numpy-shaped) for logging or diffing between runs:

```python
print(result.ledger.report())
result.ledger.to_json(indent=2)
```

`Ledger` and its supporting types are also usable directly, for anything
outside the tracked whitelist:

```python
from pandatrace import Ledger

ledger = Ledger(df, source="orders.csv")
ledger.record("custom_op", before_df, after_df, deps={"total": ("price", "qty")})
```

### The layers

| Type | Role |
| --- | --- |
| `Snapshot` | Frozen structural summary of a frame: row count, column labels, nulls per column. Holds **no reference** to the frame, so it stays cheap and cannot go stale when you mutate the original. |
| `Step` | Frozen record of one operation, with `before`/`after` snapshots, optional column `deps`, and a call-site `origin`. Deltas are derived from the snapshots on demand, never stored. |
| `Ledger` | The one mutable object. Append-only list of steps, seeded with a `read` step at index 0. |
| `TracedDataFrame` | A `pd.DataFrame` subclass. Wraps a whitelisted set of methods (`TRACKED_OPS` in `pandatrace.traced`) to call `Ledger.record()` automatically; everything else behaves like a normal DataFrame. |

## Limitations

Worth knowing before you reach for this:

- **Only a whitelisted set of operations is tracked.** `TRACKED_OPS` in
  `pandatrace.traced` covers `pipe`, `query`, `assign`, `rename`, `drop`,
  `dropna`, `fillna`, `drop_duplicates`, `sort_values`, `sort_index`,
  `reindex`, `reset_index`, `head`, `tail`, `sample`, `merge`, and `join`, plus
  a `pandatrace.concat` shim for the module-level `pd.concat`. Anything else
  that returns a new frame (`groupby`, `apply`, `pivot`, …) behaves like normal
  pandas but records nothing — the ledger's `Snapshot`/`Step` won't lie about
  what happened, but it will have a gap.
- **Indexing isn't tracked.** `.loc`, `.iloc`, `df[...]`, and boolean masking
  all fall outside the wrapped whitelist — column subsetting is a real
  provenance event this doesn't see yet.
- **Direct mutation is invisible.** `traced_df["x"] = 1` bypasses every hook;
  there's no `__setitem__` interception.
- **`inplace=True` is accepted but silently untracked**, not rejected. This is
  deliberate, not an oversight: pandas' own internals rely on calling tracked
  methods with `inplace=True` as plumbing (`merge()` strips duplicate
  join-key columns via an internal `drop(inplace=True)`), so rejecting it
  outright breaks `merge`/`join` from the inside. A step needs a `before`
  snapshot taken before the mutation happens, which an in-place call doesn't
  give a clean opportunity for — so it's recorded as nothing rather than
  guessed at.
- **`merge`/`concat` across two independently-ledgered frames keep only one
  history.** The first/left operand's ledger survives; the other operand's
  step count is noted in the recorded step's `detail` so the join stays
  auditable, but its full history isn't merged in.
- **Dependencies are declared, not inferred.** `deps={"total": ("price", "qty")}`
  is something you assert. Nothing verifies it against what the operation really
  did, so a stale `deps` will quietly misreport provenance.
- **Null counts are addressed by name, and duplicate labels defeat that.**
  `concat(axis=1)`, a colliding `rename`, and labels that differ to pandas but
  collide under `str()` all map several real columns onto one key. Counts are
  folded together rather than one silently overwriting another, and
  `Snapshot.duplicated_columns` tells you which labels that happened to — but
  the per-column figure is genuinely lost, not recoverable.
- **Snapshots are structural only.** Row counts, column labels, and null counts.
  No dtypes, no index information, no value-level diffing.

## Development

Managed with [uv](https://docs.astral.sh/uv/): `uv sync`, then

```bash
scripts/test.sh                     # pytest, args passed through
scripts/coverage.sh                 # suite under coverage (gate: 90%)
scripts/lint.sh                     # ruff check + ty check (both clean)
scripts/docs.sh                     # sphinx with -W; also executes the example notebook
scripts/check.sh                    # all of the above, the pre-merge gate
```

Docs build clean and are built with `-W` so warnings fail — keep it that way.
CI enforces the full test matrix (Python 3.11–3.14 × pandas 2.2/2.3/3.x) plus
all of the gates above on every push.

Longer prose docs, including the design invariants, live in `docs/` (`index.rst`
for the narrative, `api.rst` for the autodoc API reference).
