Metadata-Version: 2.5
Name: paneldx
Version: 0.3.1
Summary: Validate that a longitudinal dataset is what it claims to be, before you model it.
Project-URL: Homepage, https://github.com/stemmatics/paneldx
Project-URL: Repository, https://github.com/stemmatics/paneldx
Project-URL: Issues, https://github.com/stemmatics/paneldx/issues
Author: Umer Imran
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: data leakage,data validation,entity resolution,longitudinal,machine learning,panel data,reproducibility
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Requires-Dist: numpy>=1.21
Requires-Dist: pandas>=1.3
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.9; extra == 'dev'
Description-Content-Type: text/markdown

# paneldx

[![CI](https://github.com/stemmatics/paneldx/actions/workflows/ci.yml/badge.svg)](https://github.com/stemmatics/paneldx/actions/workflows/ci.yml)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE)

**Check that your longitudinal dataset is what it claims to be, before you model it.**

A panel dataset asserts that each row belongs to an entity observed repeatedly
over time. That assertion is usually taken on faith. When it is wrong, because
rows were joined by position or by an ID that is not actually stable, every
within-entity quantity computed afterwards is quietly meaningless: lags,
differences, trajectories, sequence models, grouped cross-validation splits.

Nothing errors. The numbers just stop meaning anything.

```bash
paneldx audit data.csv --time quarter --target risk_score --html report.html
```

Exits non-zero when it finds a defect that invalidates within-entity analysis,
so it can gate a pipeline or a CI job.

---

## The bug this was built from

A published physician-analytics pipeline built its entity ID from row position:

```python
physician_id = ((serial_number - 1) % rows_per_quarter) + 1
```

Each quarter was sorted by platform rank before this ran, so position *i* was a
different doctor every quarter. The ID linked strangers together, and that ID
then fed momentum features, a GRU over "trajectories", and the grouped CV splits.

`paneldx` on that dataset, given no hints:

```
key: physician_id                      key: Disease + Opening time
  columns explained  2 of 30  (7%)       columns explained  13 of 29  (45%)
  VERDICT  NOT SUPPORTED                 VERDICT  supported by the data
```

Blind search over every one- and two-column combination ranked the correct key
first. That was one of three defects. The other two turned up on the same
dataset without being told anything about it:

| Check | What it found |
|---|---|
| `detect_counters` | `Total patients` (lag-1 ρ = 0.962), `Total visits` (0.871), `Medical consultation records` (0.961). Lifetime totals that barely move, so a target built from them is autocorrelated by construction |
| `target_leakage` | R² = **0.923** reconstructing the target from its own features, naming `inv_rank`, `log_gifts` and `log_visits`, which were three of the four components the target was averaged from |
| `persistence_baseline` | Carry-forward MAE **0.0320** against the published model's **0.0942**. Doing nothing was 2.9x better |

### The defects hide each other

A broken key does not only corrupt features. It disarms the check that would
have caught it:

| Key used | Persistence MAE | R² | What a researcher concludes |
|---|---|---|---|
| Positional (as published) | 0.4717 | 0.191 | "carry-forward is useless, my 0.0942 is good" |
| Recovered | **0.0320** | **0.971** | carry-forward beats the model by 2.9x |

Under the fabricated key the naive forecast looks worthless, so nobody thinks to
compare against it. Under the real key it is close to unbeatable. This is why
`paneldx` establishes the key before it reports any baseline, and why running the
baseline alone would not have saved that paper.

---

## Install

Not on PyPI yet. Install from source:

```bash
pip install git+https://github.com/stemmatics/paneldx.git
```

Python 3.9+, `numpy` and `pandas`. Nothing else: no modelling framework, no
template engine.

---

## Usage

```python
from paneldx import audit, to_html
from paneldx import validate_key, discover_keys
from paneldx import detect_counters, target_leakage, persistence_baseline

# Is the entity key real?
print(validate_key(df, "patient_id", time_col="quarter"))
print(discover_keys(df, time_col="quarter")[0])       # no hints needed

# Are the numbers about to fool you?
print(detect_counters(df, "patient_id", "quarter"))   # lifetime totals
print(target_leakage(df, target="risk_score"))        # is y inside X?
print(persistence_baseline(df, "patient_id", "quarter", "risk_score"))

# Or all of it at once
result = audit(df, "quarter", key="patient_id", target="risk_score")
open("report.html", "w").write(to_html(result))
```

The CLI reads CSV, TSV, Excel, Parquet, Feather and JSON:

```bash
paneldx audit panel.xlsx --time t --key site_id patient_id --target outcome
```

---

## The three traps

**Cumulative features.** Lifetime totals barely move between periods, so a target
built from them is near-perfectly autocorrelated. Models trained on them report
excellent R² for restating what they were handed. Difference them into
per-period flows.

**Target composition.** When the target is computed from columns that are also
features, the model is not predicting, it is recovering arithmetic. The metrics
look superb and mean nothing. `target_leakage` fits a deliberately *linear*
model: the point is not to predict the target well, but to show no prediction was
ever required.

**Missing naive baseline.** On autocorrelated panels, carrying the last value
forward is often unbeatable. A model reported without it may be losing to a
one-line rule, and no reader could tell.

---

## How key validation works

A correct entity key reveals structure that an incorrect one cannot.

**Invariants.** Some attributes cannot change for a real entity: a birth date, an
account opening timestamp. Under the correct key they are constant within an
entity. Under a wrong key they flicker.

**Counters.** Some measures only accumulate. Under the correct key their
within-entity differences are non-negative. Under a wrong key they wander.

Neither is known in advance, so `paneldx` does not ask you to declare them. It
measures how much of this structure a candidate key reveals, and compares that
against a null: the same key with entity labels shuffled within each period. That
preserves the panel's exact shape while destroying any true correspondence. A
real key scores far above its null; a fabricated one scores at it.

The verdict uses the **share** of columns explained, not the count. An artifact of
row ordering explains only the handful of columns that drive the ordering, while
a genuine key explains most of the table. Counting raw columns passes the
fabricated key; counting the share rejects it.

| `evidence_frac` | Verdict |
|---|---|
| ≥ 0.40 | supported by the data |
| 0.15 to 0.40 | weak, inspect the listed columns by hand |
| < 0.15 | not supported, within-entity quantities are unsafe |

---

## Limitations

- **Near-valid keys are hard to separate from perfect ones.** The tolerances
  exist because real keys collide too, so a key that accidentally merges a couple
  of entities can score like a clean one. Ties break toward the finer partition,
  which mitigates but does not eliminate this.
- **Leakage detection is linear.** A target computed from its features by some
  non-linear rule can slip past `target_leakage`. A high score is strong
  evidence; a low one is not a clearance.
- **Two-column search is O(n²)** in columns. On a 24k x 31 table the full pair
  search takes about 23 seconds. Pass `--key` when you already know it.
- **A passing verdict is not a proof.** It says the data is consistent with the
  key, not that the key is correct. Domain knowledge still wins.
- Needs at least two periods, and enough entities to measure a rate against.

---

## Roadmap

- [x] Panel key discovery and validation
- [x] Cumulative-counter detection
- [x] Target-composition leakage
- [x] Naive-baseline harness
- [x] One-command HTML report
- [x] CLI
- [ ] Look-ahead detection in sequence construction
- [ ] Faster key search on wide tables

---

## Documentation

- [Getting started](docs/getting-started.md)
- [How it works](docs/how-it-works.md)
- [Interpreting results](docs/interpreting-results.md)
- [API reference](docs/api.md)

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). False positives are bugs: if `paneldx`
rejects a key you know is correct, that is worth an issue.

Release notes are in [CHANGELOG.md](CHANGELOG.md).

## Citation

If you use `paneldx` in research, see [CITATION.cff](CITATION.cff) or use the
"Cite this repository" button on GitHub.

## License

Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
