Metadata-Version: 2.4
Name: pipewhy
Version: 0.1.0.dev0
Summary: Array-based data integrity and lineage diagnostics
License-Expression: LicenseRef-Proprietary
Requires-Dist: numpy==2.4.6 ; extra == 'cpu'
Requires-Dist: numba>=0.68.1,<0.69 ; extra == 'cpu'
Requires-Dist: cuda-bindings>=12.9.10,<13 ; extra == 'cuda'
Requires-Dist: cuda-core>=1.2.2,<1.3 ; extra == 'cuda'
Requires-Dist: numba-cuda==0.30.4 ; extra == 'cuda'
Requires-Dist: numba>=0.68.1,<0.69 ; extra == 'cuda'
Requires-Dist: numpy==2.4.6 ; extra == 'cuda'
Requires-Dist: nvidia-cuda-nvcc-cu12==12.6.85 ; extra == 'cuda'
Requires-Dist: nvidia-cuda-runtime-cu12==12.6.77 ; extra == 'cuda'
Requires-Dist: nvidia-cuda-nvrtc-cu12==12.6.85 ; extra == 'cuda'
Requires-Dist: nvidia-nvjitlink-cu12==12.6.85 ; extra == 'cuda'
Requires-Dist: pyarrow==26.0.0 ; extra == 'enron'
Requires-Python: >=3.14, <3.15
Project-URL: Homepage, https://pipewhy-68315.web.app/
Project-URL: Documentation, https://pipewhy-68315.web.app/
Project-URL: Repository, https://github.com/oakplume/pipewhy
Project-URL: Issues, https://github.com/oakplume/pipewhy/issues
Project-URL: Changelog, https://github.com/oakplume/pipewhy/releases
Provides-Extra: cpu
Provides-Extra: cuda
Provides-Extra: enron
Description-Content-Type: text/markdown

<div align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://pipewhy-68315.web.app/_static/pipewhy-dark.svg">
    <img src="https://pipewhy-68315.web.app/_static/pipewhy.svg" alt="PipeWhy" width="280">
  </picture>
</div>

# PipeWhy: data integrity and lineage diagnostics

**Capture data. Detect drift. Follow the evidence.**

[![Python](https://pipewhy-68315.web.app/badges/python.svg)](https://pypi.org/project/pipewhy/)
[![Build](https://pipewhy-68315.web.app/badges/build.svg)](https://github.com/oakplume/pipewhy/actions)
[![Tests](https://pipewhy-68315.web.app/badges/tests.svg)](https://pipewhy-68315.web.app/quality/)
[![Coverage](https://pipewhy-68315.web.app/badges/coverage.svg)](https://pipewhy-68315.web.app/quality/)
[![Black](https://pipewhy-68315.web.app/badges/black.svg)](https://github.com/oakplume/pipewhy/blob/main/pyproject.toml)
[![Typing](https://pipewhy-68315.web.app/badges/typing.svg)](https://github.com/oakplume/pipewhy/blob/main/pyproject.toml)

[![Documentation](https://pipewhy-68315.web.app/badges/docs.svg)](https://pipewhy-68315.web.app/)
[![Package](https://pipewhy-68315.web.app/badges/package.svg)](https://pypi.org/project/pipewhy/)

| | |
| --- | --- |
| Package | `0.1.0.dev0` prerelease · Python 3.14 · Poetry · uv_build |
| Documentation | [Getting started](https://pipewhy-68315.web.app/getting_started/) · [User guide](https://pipewhy-68315.web.app/user_guide/) · [API reference](https://pipewhy-68315.web.app/reference/) · [Developer guide](https://pipewhy-68315.web.app/development/) |
| Project | [Source](https://github.com/oakplume/pipewhy) · [Issues](https://github.com/oakplume/pipewhy/issues) · [Releases](https://github.com/oakplume/pipewhy/releases) |

## What is it?

**PipeWhy** is a Python package for finding changes in versioned data and explaining
them using captured observations and declared lineage. It normalizes typed values,
builds composable fingerprints, stores immutable local snapshots and compares
historical captures with explicit uncertainty.

The first release targets **accidental drift and corruption detection**. Missing
reads, unknown upstream lineage and incomparable observations stay visible in
reports. Capture time alone does not establish an operation's timing or cause.
Fingerprints do not provide an adversarial integrity guarantee.

## Table of contents

- [Main features](#main-features)
- [Where to get it](#where-to-get-it)
- [Dependencies](#dependencies)
- [Installation from sources](#installation-from-sources)
- [A first snapshot](#a-first-snapshot)
- [Documentation](#documentation)
- [Background](#background)
- [Getting help](#getting-help)
- [Development and contributing](#development-and-contributing)
- [License](#license)

## Main features

- **Explicit equality rules:** strict representation by default, with a separate
  opt-in logical normalization profile.
- **Versioned fingerprints:** reference Python execution, optional native CPU
  execution and an experimental CUDA backend with recorded hardware qualification.
- **Durable local snapshots:** saved schema, identity, coverage and retention context
  support comparisons and queries after the source goes offline.
- **Qualified diagnostics:** historical comparison, declared lineage, incident
  analysis and integrity reports preserve the limits of their evidence.
- **Useful CLI output:** `scan`, `diff`, `lineage`, `why`, `report integrity` and
  `export`, with text, JSON, checksummed CSV and graph formats where supported.
- **Pinned Enron integration:** the public `corbt/enron-emails` dataset is read at
  an exact revision, with verified cache contents and explicit unknown upstream lineage.
- **Resumable local workers:** the Python execution API coordinates partitions,
  finite retries, durable attempts and replay-safe snapshot publication.

## Where to get it

Source and tagged releases are maintained in [oakplume/pipewhy](https://github.com/oakplume/pipewhy).
The source repository is private; repository and release downloads require access.

The first release series is a development prerelease. To install from PyPI:

```console
python -m pip install --pre pipewhy
```

See [PyPI](https://pypi.org/project/pipewhy/) for published versions and
[GitHub Releases](https://github.com/oakplume/pipewhy/releases) for wheel downloads
and checksums. The [installation guide](https://pipewhy-68315.web.app/getting_started/installation.html)
also describes building and installing a local wheel.

## Dependencies

The **core wheel has no runtime dependencies**. Optional extras enable specific
features; Poetry locks the development and documentation environments separately.

| Extra | Purpose | Main dependencies |
| --- | --- | --- |
| `cpu` | Native CPU fingerprints | NumPy, Numba |
| `cuda` | Experimental CUDA execution | NumPy, Numba, numba-cuda, CUDA bindings and compiler/runtime payloads |
| `enron` | Parquet-backed Enron source | PyArrow |

Python **3.14** and Poetry **2.5.x** are required for development. Current
qualification uses CPython 3.14.7. Exact dependency constraints live in
[pyproject.toml](pyproject.toml) and [poetry.lock](poetry.lock). Read the
[CUDA guide](docs/user_guide/cuda.md) before enabling that backend.

## Installation from sources

From the repository root:

```console
poetry config virtualenvs.in-project true --local
poetry sync
poetry run pipewhy --help
```

This creates the project `.venv`. Use the Poetry executable from your tool environment if `poetry` is not on your PATH.

For native CPU execution and the Enron source:

```console
poetry sync --extras cpu --extras enron
```

Include every extra you intend to keep when running `poetry sync`. Fixtures and
the Enron corpus are external source data; they are not bundled in the wheel.

## A first snapshot

Capture the fixture supplied with this checkout using the reference backend:

```console
poetry run pipewhy scan fixture --config examples/local-fixture.toml --source-version baseline-v1 --retain value --format json
```

The example writes to `.artifacts/local-snapshots` and uses repository test data.
Value retention here is explicit. The [quickstart](docs/getting_started/quickstart.md)
continues with changed and missing observations, comparisons and saved reports.
For a repeatable demonstration, use the PowerShell 7
[local slice script](examples/run-local-slice.ps1).

## Documentation

Read the **[PipeWhy documentation](https://pipewhy-68315.web.app/)** for getting
started, the user guide, API reference and developer guide. It includes search,
section navigation and light/dark modes.

To build the static HTML and offline ZIP from the source checkout:

```console
poetry sync --with docs
poetry run python tools/build_docs.py
```

Open `docs/_build/html/index.html`, or extract `dist/pipewhy-docs.zip` and open
`index.html`. The [documentation build guide](docs/development/documentation.md)
describes local previews and the generated files.

## Background

PipeWhy separates source identity, normalized representation, fingerprints,
observations and lineage evidence. Comparisons carry that context forward so a
report can distinguish detected drift from missing or incompatible information.
Saved queries work with the core package without reopening source data.

The [architecture guide](docs/development/architecture.rst) links the versioned
contracts and acceptance records. Local benchmark results retain their measured
hardware, source and dependency provenance; they are not universal performance claims.

## Getting help

Use the [issue tracker](https://github.com/oakplume/pipewhy/issues) for bugs,
questions and proposals. Include the package version, command or API call,
backend, expected result and a minimal reproducible example. Keep private data
and credentials out of reports. `poetry run pipewhy --help` lists CLI commands.

## Development and contributing

Follow the [contributor requirements](CONTRIBUTING.md)
and [developer guide](docs/development/contributing.md). Work in small, meaningful
commits tied to issues. Tests must check behavior, including failure paths.

```console
poetry run python tools/quality.py
poetry run python -m pytest tests/gates tests/integration
poetry run python tools/build_docs.py
poetry build
```

The quality gate enforces Black, `ty`, dependency checks and pytest under
Coverage.py, with at least **95% statement and combined coverage** and no coverage
exclusions. CI also builds the documentation with warnings treated as errors,
rebuilds the source distribution and checks clean installed packages for silent
runtime breaks. See the [packaging guide](docs/development/packaging.md).

## License

Current package metadata is **`LicenseRef-Proprietary`**. No public open-source
license has been selected. Distribution preparation does not grant redistribution
rights or authorize publication.
