Metadata-Version: 2.4
Name: etlantic
Version: 0.10.0
Summary: Typed, contract-driven data pipeline modeling for Python.
Project-URL: Homepage, https://github.com/eddiethedean/etlantic
Project-URL: Documentation, https://github.com/eddiethedean/etlantic/tree/main/docs
Project-URL: Repository, https://github.com/eddiethedean/etlantic
Project-URL: Issues, https://github.com/eddiethedean/etlantic/issues
Project-URL: Changelog, https://github.com/eddiethedean/etlantic/blob/main/CHANGELOG.md
Author-email: Odo Matthews <odosmatthews@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: data-contracts,etl,pipelines,pydantic,typed
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: anyio<5,>=4
Requires-Dist: click<9,>=8
Requires-Dist: contractmodel>=0.1.2
Requires-Dist: dpcs<1,>=0.13
Requires-Dist: dtcs<1,>=0.11
Requires-Dist: packaging>=24
Requires-Dist: pydantic<3,>=2.12
Requires-Dist: typer<1,>=0.15
Provides-Extra: airflow
Requires-Dist: etlantic-airflow==0.10.0; extra == 'airflow'
Provides-Extra: arrow
Requires-Dist: pyarrow>=14; extra == 'arrow'
Provides-Extra: dataframes
Requires-Dist: etlantic-pandas==0.10.0; extra == 'dataframes'
Requires-Dist: etlantic-polars==0.10.0; extra == 'dataframes'
Provides-Extra: keyring
Requires-Dist: etlantic-keyring==0.10.0; extra == 'keyring'
Provides-Extra: observability
Requires-Dist: opentelemetry-api<2,>=1.36; extra == 'observability'
Provides-Extra: otel
Requires-Dist: opentelemetry-api<2,>=1.36; extra == 'otel'
Provides-Extra: pandas
Requires-Dist: etlantic-pandas==0.10.0; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: etlantic-polars==0.10.0; extra == 'polars'
Provides-Extra: postgresql
Requires-Dist: etlantic-sql==0.10.0; extra == 'postgresql'
Provides-Extra: pyspark
Requires-Dist: etlantic-pyspark==0.10.0; extra == 'pyspark'
Provides-Extra: spark
Requires-Dist: etlantic-pyspark==0.10.0; extra == 'spark'
Provides-Extra: sparkforge
Requires-Dist: etlantic-sparkforge==0.10.0; extra == 'sparkforge'
Provides-Extra: sql
Requires-Dist: etlantic-sql==0.10.0; extra == 'sql'
Provides-Extra: sqlmodel
Requires-Dist: etlantic-sqlmodel==0.10.0; extra == 'sqlmodel'
Description-Content-Type: text/markdown

![ETLantic banner](https://raw.githubusercontent.com/eddiethedean/etlantic/main/docs/theme/assets/etlantic-banner.png)

# ETLantic

Catch incompatible data-pipeline wiring before you process data.

Define datasets, transformations, and pipelines as typed Python classes.
Validate and plan them once. Run locally today; swap Polars, Pandas, or SQL
backends without rewriting the logical pipeline.

**Status:** Alpha **0.10.0** — local runtime + optional
Polars/Pandas/SQL/PySpark/Airflow plugins. Structured Streaming is
experimental.

> **Install note:** Until a matching `v0.10.0` tag is published to PyPI, prefer
> installing from source (below). Anonymous `pip install etlantic` may fail if
> wheels are not yet uploaded.

## Install (from source — works today)

Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/eddiethedean/etlantic.git
cd etlantic
uv sync
uv run python -c "import etlantic; print(etlantic.__version__)"
uv run python examples/quickstart.py
```

### Optional when wheels are on PyPI

```bash
pip install etlantic
# optional engines / compilers
pip install etlantic-polars etlantic-pandas etlantic-sql
pip install etlantic-pyspark etlantic-airflow
pip install etlantic-keyring etlantic-sqlmodel etlantic-sparkforge
```

## Quick example

```python
from etlantic import (
    Data,
    Input,
    Output,
    Pipeline,
    PipelineRuntime,
    Sink,
    Source,
    Transformation,
)


class RawCustomer(Data):
    customer_id: int
    first_name: str
    last_name: str


class Customer(Data):
    customer_id: int
    full_name: str


class NormalizeCustomers(Transformation):
    customers: Input[RawCustomer]
    result: Output[Customer]


class CustomerPipeline(Pipeline):
    raw: Source[RawCustomer] = Source(binding="customer_source")
    normalized = NormalizeCustomers.step(customers=raw)
    curated: Sink[Customer] = Sink(
        input=normalized.result,
        binding="customer_sink",
    )


@NormalizeCustomers.implementation("local")
def normalize(customers: list[RawCustomer]) -> list[Customer]:
    return [
        Customer(
            customer_id=row.customer_id,
            full_name=f"{row.first_name} {row.last_name}",
        )
        for row in customers
    ]


CustomerPipeline.validate(profile="development").raise_for_errors()

runtime = PipelineRuntime()
runtime.memory.seed(
    "customer_source",
    [RawCustomer(customer_id=1, first_name="Ada", last_name="Lovelace")],
)
run_report = CustomerPipeline.run(profile="development", runtime=runtime)
print(runtime.memory.get("customer_sink"))
```

Run the complete tested version at
[examples/quickstart.py](examples/quickstart.py).

## Current capability boundary

| Capability | 0.10 |
|---|---|
| Typed modeling, validation, contracts, and planning | Available |
| Local Python execution and run reports | Available |
| Memory, callable, JSON, CSV, and no-write storage | Available |
| Polars and Pandas dataframe plugins | Available (`etlantic-polars` / `etlantic-pandas`) |
| SQL plugin | Available (`etlantic-sql`) |
| PySpark plugin + local provider | Available (`etlantic-pyspark`) |
| Structured Streaming | Experimental |
| Airflow orchestrator compiler | Available (`etlantic-airflow`) |
| CLI compile / generate / schema / SARIF | Available |
| Plugin allowlists / keyring / SQLModel extras | Available |
| SparkForge migration adapter | Available (`etlantic-sparkforge`) |

**Next design line:** releases 0.11-0.15 are planned to add a PySpark-inspired
portable transformation language, followed by Polars, PySpark, Pandas, and safe
SQL compilers. This is documented future design, not part of the 0.10 API. See
the [portable transformation design](docs/04_TRANSFORMATIONS/PORTABLE_TRANSFORMATIONS.md)
and [roadmap](docs/11_DEVELOPMENT/ROADMAP.md).

## Documentation

- [Getting Started](docs/01_GETTING_STARTED/README.md) (start here)
- [Capabilities and Limitations](docs/01_GETTING_STARTED/CAPABILITIES.md)
- [Quickstart](docs/01_GETTING_STARTED/QUICKSTART.md)
- [Evaluator brief](docs/01_GETTING_STARTED/EVALUATOR.md)
- [Core Concepts](docs/02_FOUNDATIONS/CORE_CONCEPTS.md)
- [Architecture](docs/02_FOUNDATIONS/ARCHITECTURE.md)
- [Roadmap](docs/11_DEVELOPMENT/ROADMAP.md)

Build the docs locally with `uv run mkdocs serve` (hosted site TBD).

## Development

Requires [uv](https://docs.astral.sh/uv/).

```bash
uv sync
uv run pytest
uv run ruff check .
uv run ruff format .
```

`uv sync` creates `.venv`, installs the package in editable mode, and installs
the `dev` dependency group (pytest, ruff, mkdocs) by default.

## Release

The first upload of each new package name counts against PyPI’s new-project
rate limit (`429 Too many new projects created`). Release CI waits **10 minutes**
between package publishes. See
[Release Process](docs/11_DEVELOPMENT/RELEASE_PROCESS.md).

Tag a version that matches `src/etlantic/_version.py` (and every
`packages/*/pyproject.toml`), then push **only that tag**:

```bash
git tag -a v0.10.0 -m "ETLantic 0.10.0"
git push origin v0.10.0
```

Do not use `git push --tags`. GitHub Actions runs checks and publishes to PyPI
using the `PYPI_API_TOKEN` repository secret.

## License

MIT
