Metadata-Version: 2.4
Name: etlantic
Version: 0.21.0
Summary: Typed, contract-driven data pipeline modeling for Python.
Project-URL: Homepage, https://github.com/eddiethedean/etlantic
Project-URL: Documentation, https://etlantic.readthedocs.io/
Project-URL: Repository, https://github.com/eddiethedean/etlantic
Project-URL: Issues, https://github.com/eddiethedean/etlantic/issues
Project-URL: Changelog, https://github.com/eddiethedean/etlantic/blob/main/CHANGELOG.md
Author-email: Odo Matthews <odosmatthews@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: data-contracts,etl,pipelines,pydantic,typed
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anyio<5,>=4
Requires-Dist: click<9,>=8
Requires-Dist: contractmodel<0.3,>=0.2
Requires-Dist: dpcs<1,>=0.13
Requires-Dist: dtcs<1,>=0.13
Requires-Dist: jsonschema<5,>=4.23
Requires-Dist: packaging>=24
Requires-Dist: pydantic<3,>=2.12
Requires-Dist: typer<1,>=0.15
Provides-Extra: airflow
Requires-Dist: etlantic-airflow==0.21.0; extra == 'airflow'
Provides-Extra: arrow
Requires-Dist: pyarrow>=14; extra == 'arrow'
Provides-Extra: dataframes
Requires-Dist: etlantic-pandas==0.21.0; extra == 'dataframes'
Requires-Dist: etlantic-polars==0.21.0; extra == 'dataframes'
Provides-Extra: datafusion
Requires-Dist: etlantic-datafusion==0.21.0; extra == 'datafusion'
Provides-Extra: keyring
Requires-Dist: etlantic-keyring==0.21.0; extra == 'keyring'
Provides-Extra: observability
Requires-Dist: opentelemetry-api<2,>=1.36; extra == 'observability'
Provides-Extra: otel
Requires-Dist: opentelemetry-api<2,>=1.36; extra == 'otel'
Provides-Extra: pandas
Requires-Dist: etlantic-pandas==0.21.0; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: etlantic-polars==0.21.0; extra == 'polars'
Provides-Extra: postgresql
Requires-Dist: etlantic-sql==0.21.0; extra == 'postgresql'
Provides-Extra: prefect
Requires-Dist: etlantic-prefect==0.21.0; extra == 'prefect'
Provides-Extra: pyspark
Requires-Dist: etlantic-pyspark==0.21.0; extra == 'pyspark'
Provides-Extra: spark
Requires-Dist: etlantic-pyspark==0.21.0; extra == 'spark'
Provides-Extra: sparkforge
Requires-Dist: etlantic-sparkforge==0.21.0; extra == 'sparkforge'
Provides-Extra: sql
Requires-Dist: etlantic-sql==0.21.0; extra == 'sql'
Provides-Extra: sqlmodel
Requires-Dist: etlantic-sqlmodel==0.21.0; extra == 'sqlmodel'
Description-Content-Type: text/markdown

<p align="center">
  <img
    src="https://raw.githubusercontent.com/eddiethedean/etlantic/main/docs/theme/assets/etlantic-logo.svg"
    width="148"
    alt="ETLantic logo"
  >
</p>

<h1 align="center">ETLantic</h1>

<p align="center">
  <strong>Design once. Validate everywhere.</strong><br>
  Typed, contract-driven data pipelines for Python.
</p>

<p align="center">
  <a href="https://github.com/eddiethedean/etlantic/actions/workflows/ci.yml"><img src="https://github.com/eddiethedean/etlantic/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/etlantic/"><img src="https://img.shields.io/pypi/v/etlantic.svg" alt="PyPI"></a>
  <a href="https://pypi.org/project/etlantic/"><img src="https://img.shields.io/pypi/pyversions/etlantic.svg" alt="Python versions"></a>
  <a href="https://github.com/eddiethedean/etlantic/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-MIT-d6a84b.svg" alt="MIT license"></a>
  <a href="https://github.com/astral-sh/ruff"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff"></a>
</p>

<p align="center">
  <a href="https://etlantic.readthedocs.io/">Documentation</a> ·
  <a href="https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/QUICKSTART/">Quickstart</a> ·
  <a href="https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/CAPABILITIES/">Capabilities</a> ·
  <a href="https://etlantic.readthedocs.io/en/latest/11_DEVELOPMENT/ROADMAP_SUMMARY/">Roadmap</a>
</p>

---

ETLantic is a typed control layer for data pipelines. Define data,
transformations, and topology as Python contracts; validate them before work
begins; then run or compile the same logical pipeline for different backends.

```text
Typed contracts ──▶ Validation ──▶ Deterministic plan ──▶ Run or compile
```

The name describes the model: **ETL** is the data flow; **ETLantic** surrounds
it with typed contracts, validation, planning, and evidence.

```text
V(model) → Extract → V(input) → Transform → V(output) → Load → V/evidence
```

Validation is a control layer, not another business transformation. Runtime
checks are selected by policy and backend capability; publication evidence does
not imply an automatic sink reread.

## Why ETLantic?

- Catch invalid wiring, incompatible contracts, missing capabilities, and
  untrusted plugins before a write.
- Validate extracted inputs, transformation outputs, engine transitions, and
  publication boundaries against the same contracts.
- Keep one logical pipeline across local Python, Polars, Pandas, SQL, PySpark,
  Airflow, and Prefect.
- Review deterministic, secret-free plans and preserve structured diagnostics,
  lineage, schema observations, and run reports.
- Install a small core and add only the engines you need.

ETLantic does not replace dataframe engines, databases, Spark, schedulers,
storage systems, catalogs, or secret managers. It gives them one typed pipeline
model and one inspectable validation lifecycle.

> **Status:** ETLantic **0.21.0** ships a cohesive CLI journey with durable
> local workspaces. Stable for documented single-tenant reference deployments, not unrestricted enterprise production. Structured
> Streaming remains experimental. See [Capabilities](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/CAPABILITIES/)
> and [Production readiness](https://etlantic.readthedocs.io/en/latest/06_EXECUTION/PRODUCTION_READINESS/).

## Quickstart

ETLantic requires Python 3.11 or newer.

```bash
pip install etlantic==0.21.0
etlantic --version

mkdir my-pipeline && cd my-pipeline
etlantic init --with-toml
etlantic doctor --profile development
etlantic validate pipeline.py:SamplePipeline --profile development
etlantic plan pipeline.py:SamplePipeline --profile development
etlantic run pipeline.py:SamplePipeline --profile development
cat data/out.json
```

You should see run status `succeeded` and JSON rows for Ada and Grace. The CLI
defaults to `development` when `--profile` is omitted (or your project's
`default_profile`). Prefer an explicit profile in scripts and CI.

Full walkthrough: [Quickstart](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/QUICKSTART/).
Next: [First Pipeline](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/FIRST_PIPELINE/)
(evolve the generated project). From a checkout, an optional in-memory SDK demo
is [`examples/memory_customers.py`](https://github.com/eddiethedean/etlantic/blob/main/examples/memory_customers.py).

## Engines and integrations

| Integration | Install | Role |
|---|---|---|
| Polars | `etlantic-polars` | Eager/lazy dataframe execution and portable compilation |
| Pandas | `etlantic-pandas` | Eager dataframe execution and portable compilation |
| SQL | `etlantic-sql` | Parameterized relational execution and portable SQL compilation |
| PySpark | `etlantic-pyspark` | Spark execution and portable compilation |
| Airflow | `etlantic-airflow` | Compile plans into DAG artifacts |
| Prefect | `etlantic-prefect` | Direct-execution scheduler integration |
| Keyring | `etlantic-keyring` | OS keyring secret provider |
| SQLModel | `etlantic-sqlmodel` | SQLModel bridge helpers |
| SparkForge | `etlantic-sparkforge` | Medallion adapter (bronze/silver/gold stay out of core) |
| DataFusion | `etlantic-datafusion` | Experimental query engine stub (Gate B) |

See [Optional packages](https://etlantic.readthedocs.io/en/latest/10_REFERENCE/OPTIONAL_PACKAGES/)
for observability (`otel` / `observability` extras) and Arrow helpers.

Matching extras such as `etlantic[polars]` are equivalent. Pin matching minors
while ETLantic is pre-1.0. Airflow is compile-only and does not install Apache
Airflow itself.

## Architecture

ETLantic keeps logical meaning separate from physical execution:

```text
Data (ODCS/ContractModel) + Transformation (DTCS) + Pipeline (DPCS)
                              │
                       validate and plan
                              ▼
                    secret-free PipelinePlan
                              │
                  ┌───────────┼───────────┐
                  ▼           ▼           ▼
               execute      compile     generate
                  │           │           │
                  └──── plugins and external systems
```

Plans and reports contain secret references, never resolved secret values.
Production profiles require explicit plugin allowlists. Backend optimizations
may change the physical graph but must preserve contracts, validation
boundaries, security domains, and logical attribution.

## Capability boundary

| Capability | 0.21 |
|---|---|
| Cohesive CLI (`init`, `doctor`, durable reports) | Available |
| Typed contracts, graph validation, deterministic planning | Available |
| Local, Polars, Pandas, SQL, and PySpark execution paths | Available |
| Portable compilers for Polars, Pandas, SQL, and PySpark | Available |
| ODCS, DTCS, DPCS, schema drift, lineage, reports, and SARIF | Available |
| Airflow compilation and Prefect scheduling | Available |
| Versioned Polars↔Pandas tabular interchange | Available |
| Contract and configuration freeze (deep plans, security_mode) | Available |
| Trust, isolation, safe I/O, SBOM/attestations | Available |
| Structured Streaming | Experimental |
| `etlantic-datafusion` | Experimental |
| Full multi-tenant control plane | Not included |

See the full [Capabilities](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/CAPABILITIES/)
and [Validation Everywhere](https://etlantic.readthedocs.io/en/latest/02_FOUNDATIONS/VALIDATION_EVERYWHERE/)
guides for precise guarantees and limitations.

## Learn more

[Installation](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/INSTALLATION/)
· [Quickstart](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/QUICKSTART/)
· [Engine selection](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/ENGINE_SELECTION/)
· [Compare](https://etlantic.readthedocs.io/en/latest/01_GETTING_STARTED/COMPARE/)
· [Security](https://etlantic.readthedocs.io/en/latest/02_FOUNDATIONS/SECURITY/)
· [Roadmap](https://etlantic.readthedocs.io/en/latest/11_DEVELOPMENT/ROADMAP_SUMMARY/)
· [Contributing](https://github.com/eddiethedean/etlantic/blob/main/CONTRIBUTING.md)

MIT licensed.
