Metadata-Version: 2.5
Name: veridelta
Version: 0.11.1
Summary: Semantic diffing for mission-critical data pipelines.
Project-URL: Homepage, https://veridelta.github.io/veridelta/
Project-URL: Repository, https://github.com/Veridelta/veridelta
Project-URL: Issues, https://github.com/Veridelta/veridelta/issues
Author-email: The Veridelta Contributors <veridelta.labs@gmail.com>
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: data-engineering,diff,polars,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: polars>=1.39.3
Requires-Dist: pydantic>=2.12.5
Requires-Dist: pyyaml>=6.0.3
Provides-Extra: all
Requires-Dist: connectorx>=0.4.5; extra == 'all'
Requires-Dist: databricks-sql-connector>=3.0.0; extra == 'all'
Requires-Dist: deltalake>=0.15.0; extra == 'all'
Requires-Dist: fastexcel>=0.11.0; extra == 'all'
Requires-Dist: google-cloud-bigquery>=3.11.0; extra == 'all'
Requires-Dist: pyarrow<24,>=14.0.1; (python_version >= '3.14') and extra == 'all'
Requires-Dist: pyarrow>=10.0.1; extra == 'all'
Requires-Dist: pyarrow>=10.0.1; (python_version < '3.14') and extra == 'all'
Requires-Dist: pyarrow>=14.0.1; extra == 'all'
Requires-Dist: pyiceberg>=0.6.0; extra == 'all'
Requires-Dist: rapidfuzz>=3.0.0; extra == 'all'
Requires-Dist: snowflake-connector-python>=3.7.0; extra == 'all'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery>=3.11.0; extra == 'bigquery'
Requires-Dist: pyarrow>=14.0.1; extra == 'bigquery'
Provides-Extra: database
Requires-Dist: connectorx>=0.4.5; extra == 'database'
Requires-Dist: pyarrow>=14.0.1; extra == 'database'
Provides-Extra: databricks
Requires-Dist: databricks-sql-connector>=3.0.0; extra == 'databricks'
Requires-Dist: pyarrow>=10.0.1; extra == 'databricks'
Provides-Extra: delta
Requires-Dist: deltalake>=0.15.0; extra == 'delta'
Provides-Extra: excel
Requires-Dist: fastexcel>=0.11.0; extra == 'excel'
Provides-Extra: fuzzy
Requires-Dist: rapidfuzz>=3.0.0; extra == 'fuzzy'
Provides-Extra: iceberg
Requires-Dist: pyiceberg>=0.6.0; extra == 'iceberg'
Provides-Extra: snowflake
Requires-Dist: pyarrow<24,>=14.0.1; (python_version >= '3.14') and extra == 'snowflake'
Requires-Dist: pyarrow>=10.0.1; (python_version < '3.14') and extra == 'snowflake'
Requires-Dist: snowflake-connector-python>=3.7.0; extra == 'snowflake'
Description-Content-Type: text/markdown

# Veridelta

[![CI Pipeline](https://github.com/veridelta/veridelta/actions/workflows/ci.yml/badge.svg)](https://github.com/veridelta/veridelta/actions)
[![codecov](https://codecov.io/gh/veridelta/veridelta/graph/badge.svg)](https://codecov.io/gh/veridelta/veridelta)
[![PyPI version](https://badge.fury.io/py/veridelta.svg)](https://pypi.org/project/veridelta/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

Veridelta compares two datasets on their keys and reports exactly what changed after applying the variance you declared as expected. It is built for system modernizations, model retrains, and pipeline migrations — anywhere "equal" has to be defined, not assumed.

Powered by [Polars](https://pola.rs/). **[Documentation](https://veridelta.github.io/veridelta)**

## Why

- **Deterministic verdicts.** Nine fixed transform stages. The same rule produces the same result locally and in a warehouse, verified by a differential harness.
- **Scale.** Lazy Polars scans. Warehouse pushdown compiles comparison SQL and never extracts full tables.
- **Exactness.** Nothing is forgiven unless a rule says so. `strict_types` treats type drift as a mismatch, not a cast.
- **CI/CD.** Exit codes 0 (match), 1 (drift or a failure), and 2 (invalid arguments). `--json` on stdout. `--html` writes a standalone report, `--markdown` a summary for pull requests. Artifacts for added, removed, and changed rows.
- **Schema evolution.** `schema_mode` is `intersection`, `exact`, `allow_additions`, or `allow_removals`.
- **Connectors.** Snowflake, Databricks, BigQuery, and opt-in Postgres SQL pushdown; Delta Lake and Iceberg scans; PostgreSQL, MySQL, SQL Server, Oracle, SQLite, and more through ConnectorX. Optional extras.

## Install

```bash
uv add veridelta
# or: pip install veridelta
uv add 'veridelta[snowflake]'   # extras: snowflake, databricks, bigquery, delta, iceberg, database, excel, fuzzy, all
```

Routing, YAML fields, and time travel: [configuration guide](https://veridelta.github.io/veridelta/configuration/).

## Architecture

```mermaid
flowchart LR
  subgraph sources [Sources]
    files[Files]
    lakehouse[Delta Iceberg]
    databases[Postgres MySQL SQLite]
    warehouse[Snowflake Databricks BigQuery]
  end
  files --> loader[LoaderFactory]
  lakehouse --> loader
  databases --> loader
  warehouse --> compiler[SQLPushdownCompiler]
  databases -. pushdown .-> compiler
  loader --> engine["DiffEngine"]
  engine --> result[DiffResult]
  compiler --> warehouseSql[Warehouse SQL]
  warehouseSql --> result
  result --> artifacts[Artifacts]
  result --> reports["HTML JSON"]
  result --> exitCode[Exit code]
```

File, lakehouse, and database sources load through `LoaderFactory` into a local `DiffEngine` run. Same-warehouse pairs, and two Postgres tables that set `pushdown`, compile to SQL and execute in place. Both paths return a `DiffResult`.

## Quick start

Python — `DiffEngine` consumes `LazyFrame`s:

```python
import polars as pl
from veridelta import DiffConfig, DiffEngine, DiffRule

result = DiffEngine(
    DiffConfig(
        primary_keys=["user_id"],
        rules=[DiffRule(pattern="^AMT_.*", absolute_tolerance=0.05)],
    ),
    pl.scan_parquet("legacy.parquet"),
    pl.scan_parquet("modern.parquet"),
).run()

if not result.summary.is_match:
    raise SystemExit(f"{result.summary.changed_count} rows differ")
```

YAML — the same comparison for CI:

```yaml
# veridelta.yaml
primary_keys: ["transaction_id"]
source:
  path: "legacy.parquet"
  format: "parquet"
target:
  path: "modern.parquet"
  format: "parquet"
rules:
  - column_names: ["grand_total"]
    relative_tolerance: 0.01
  - column_names: ["contact_number"]
    regex_replace: {"[^0-9]": ""}
```

```bash
veridelta validate -c veridelta.yaml   # what would stop a run, without reading any rows
veridelta run -c veridelta.yaml
```

## Where next

- [1. Core Concepts](https://veridelta.github.io/veridelta/examples/01_core_concepts/) — Python API, `DiffResult`, rules.
- [2. YAML and CLI](https://veridelta.github.io/veridelta/examples/02_yaml_and_cli/) — pipeline automation, `--json`, artifacts.
- [3. Advanced Rules](https://veridelta.github.io/veridelta/examples/03_advanced_rules/) — drift resolution on real data.
- [4. HTML Reports](https://veridelta.github.io/veridelta/examples/04_html_reports/) — audit and compliance hand-off.
- [CI integrations](https://veridelta.github.io/veridelta/ci/) — GitHub Action and GitLab CI template that comment on pull requests.
- [Configuration](https://veridelta.github.io/veridelta/configuration/) — fields, formats, extras, warehouse, lakehouse, and database routing.
- [API Reference](https://veridelta.github.io/veridelta/api/) — public Python surface.
- [Roadmap](https://veridelta.github.io/veridelta/roadmap/) — work that is not built yet.

## License

Apache 2.0.
