Metadata-Version: 2.4
Name: datacoolie
Version: 0.1.2
Summary: Metadata-driven ETL framework that unifies engines, stays cloud-agnostic, and is currently batch-first with a roadmap to micro-batch and streaming.
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Keywords: etl,etl-framework,data-engineering,metadata-driven,polars,pyspark,lakehouse
Author: DataCoolie Team
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Provides-Extra: all
Provides-Extra: api
Provides-Extra: aws
Provides-Extra: aws-polars
Provides-Extra: aws-spark
Provides-Extra: boto3
Provides-Extra: databricks
Provides-Extra: databricks-polars
Provides-Extra: databricks-spark
Provides-Extra: db
Provides-Extra: delta-spark
Provides-Extra: deltalake
Provides-Extra: excel
Provides-Extra: fabric
Provides-Extra: fabric-polars
Provides-Extra: fabric-spark
Provides-Extra: iceberg
Provides-Extra: polars
Provides-Extra: spark
Requires-Dist: boto3 (>=1.28) ; extra == "all"
Requires-Dist: boto3 (>=1.28) ; extra == "aws"
Requires-Dist: boto3 (>=1.28) ; extra == "aws-polars"
Requires-Dist: boto3 (>=1.28) ; extra == "aws-spark"
Requires-Dist: boto3 (>=1.28) ; extra == "boto3"
Requires-Dist: delta-spark (>=3.0) ; extra == "all"
Requires-Dist: delta-spark (>=3.0) ; extra == "databricks"
Requires-Dist: delta-spark (>=3.0) ; extra == "databricks-spark"
Requires-Dist: delta-spark (>=3.0) ; extra == "delta-spark"
Requires-Dist: delta-spark (>=3.0) ; extra == "fabric"
Requires-Dist: delta-spark (>=3.0) ; extra == "fabric-spark"
Requires-Dist: deltalake (>=0.15) ; extra == "all"
Requires-Dist: deltalake (>=0.15) ; extra == "aws"
Requires-Dist: deltalake (>=0.15) ; extra == "aws-polars"
Requires-Dist: deltalake (>=0.15) ; extra == "databricks"
Requires-Dist: deltalake (>=0.15) ; extra == "databricks-polars"
Requires-Dist: deltalake (>=0.15) ; extra == "deltalake"
Requires-Dist: deltalake (>=0.15) ; extra == "fabric"
Requires-Dist: deltalake (>=0.15) ; extra == "fabric-polars"
Requires-Dist: fastexcel (>=0.9) ; extra == "all"
Requires-Dist: fastexcel (>=0.9) ; extra == "excel"
Requires-Dist: httpx (>=0.24) ; extra == "all"
Requires-Dist: httpx (>=0.24) ; extra == "api"
Requires-Dist: openpyxl (>=3.1) ; extra == "all"
Requires-Dist: openpyxl (>=3.1) ; extra == "excel"
Requires-Dist: polars (>=1.0) ; extra == "all"
Requires-Dist: polars (>=1.0) ; extra == "aws"
Requires-Dist: polars (>=1.0) ; extra == "aws-polars"
Requires-Dist: polars (>=1.0) ; extra == "databricks"
Requires-Dist: polars (>=1.0) ; extra == "databricks-polars"
Requires-Dist: polars (>=1.0) ; extra == "fabric"
Requires-Dist: polars (>=1.0) ; extra == "fabric-polars"
Requires-Dist: polars (>=1.0) ; extra == "polars"
Requires-Dist: pyiceberg (>=0.6) ; extra == "all"
Requires-Dist: pyiceberg (>=0.6) ; extra == "aws"
Requires-Dist: pyiceberg (>=0.6) ; extra == "aws-polars"
Requires-Dist: pyiceberg (>=0.6) ; extra == "iceberg"
Requires-Dist: pyspark (>=3.5) ; extra == "all"
Requires-Dist: pyspark (>=3.5) ; extra == "databricks"
Requires-Dist: pyspark (>=3.5) ; extra == "databricks-spark"
Requires-Dist: pyspark (>=3.5) ; extra == "fabric"
Requires-Dist: pyspark (>=3.5) ; extra == "fabric-spark"
Requires-Dist: pyspark (>=3.5) ; extra == "spark"
Requires-Dist: sqlalchemy (>=2.0,<3.0) ; extra == "all"
Requires-Dist: sqlalchemy (>=2.0,<3.0) ; extra == "db"
Requires-Dist: tzdata (>=2024.1) ; sys_platform == "win32"
Project-URL: Documentation, https://datacoolie.github.io/datacoolie/
Project-URL: Homepage, https://github.com/datacoolie/datacoolie
Project-URL: Repository, https://github.com/datacoolie/datacoolie
Description-Content-Type: text/markdown

<p align="center">
    <img src="https://raw.githubusercontent.com/datacoolie/datacoolie/main/docs/images/banners/datacoolie-banner-dark.png" alt="DataCoolie banner" width="100%">
</p>

<p align="center">
  <a href="https://pypi.org/project/datacoolie/"><img src="https://img.shields.io/pypi/v/datacoolie" alt="PyPI version"></a>
  <a href="https://pypi.org/project/datacoolie/"><img src="https://img.shields.io/pypi/pyversions/datacoolie" alt="Python versions"></a>
  <a href="https://pypi.org/project/datacoolie/"><img src="https://img.shields.io/pypi/dm/datacoolie" alt="Downloads"></a>
  <a href="https://github.com/datacoolie/datacoolie/actions/workflows/ci.yml"><img src="https://github.com/datacoolie/datacoolie/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://datacoolie.github.io/datacoolie/"><img src="https://img.shields.io/badge/docs-datacoolie.github.io-blue" alt="Docs"></a>
  <a href="LICENSE"><img src="https://img.shields.io/github/license/datacoolie/datacoolie" alt="License"></a>
</p>

# DataCoolie — Metadata-driven ETL Framework

Metadata-driven ETL framework that unifies execution engines (Spark, Polars, and more in the future), remains cloud-agnostic (Fabric, AWS, Databricks, and more in the future), and currently focuses on batch workloads with a roadmap to micro-batch and streaming.

## What problem does it solve?

Data teams often prototype pipelines locally, then rewrite the same pipeline
for Spark and again for each cloud runtime. That duplicates ETL code and makes
operational behavior such as watermarks, schema hints, partitions, load
strategies, and maintenance drift across environments.

DataCoolie solves this by separating pipeline intent from execution details.
You define connections, dataflows, transforms, and operational controls as
metadata, then run the same intent on Polars or Spark and on local, Fabric,
Databricks, or AWS platforms.

## Why it helps

- **Metadata-driven** — pipeline behavior lives in metadata instead of being re-implemented in each job.
- **Right-sized compute** — small and medium jobs can stay on lighter runtimes like Polars or local execution instead of paying Spark or cluster overhead too early.
- **Portable** — the same metadata can move to Spark and cloud platforms when workloads grow.
- **Engine-unified** — the same metadata runs on Spark *and* Polars; swap at runtime.
- **Cloud-agnostic** — `local`, `aws`, `fabric`, `databricks` platforms abstract file I/O and secrets.
- **Lakehouse-native** — first-class Delta Lake and Apache Iceberg via `fmt="delta"` / `fmt="iceberg"`.
- **Operationally complete** — watermarks, schema hints, partitions, load strategies, logging, and maintenance are built in.
- **Plugin everything** — engines, platforms, sources, destinations, transformers, and secret resolvers are all entry-point plugins.

## Start here

If you are evaluating DataCoolie for the first time, use this order:

1. Install the smallest useful runtime: `pip install "datacoolie[polars,deltalake]"`
2. Run the quick start below
3. Then move to the docs for using your own input and building a multi-stage flow

If you already know your runtime will be Spark, swap the install to
`pip install "datacoolie[spark,delta-spark]"` and keep the same metadata pattern.

## Installation

```bash
# Most common first install
pip install "datacoolie[polars,deltalake]"

# Spark-first local validation
pip install "datacoolie[spark,delta-spark]"

# Core only (mainly useful for extension work)
pip install datacoolie

# All engines
pip install datacoolie[all]
```

## Quick Start

Install, then run two short scripts:

1. `prepare_quickstart.py` creates a sample CSV and `metadata.json`.
2. `run_quickstart.py` loads that metadata and runs the pipeline.

```bash
pip install "datacoolie[polars]"
```

### Part 1 — Prepare sample data and metadata

```python
# prepare_quickstart.py
import json
from pathlib import Path

root = Path("dc_quickstart")
(root / "input" / "orders").mkdir(parents=True, exist_ok=True)
(root / "output").mkdir(parents=True, exist_ok=True)

(root / "input/orders/orders.csv").write_text(
    "order_id,customer_id,amount\n1,100,19.99\n2,100,42.50\n3,101,7.25\n"
)

metadata = {
    "connections": [
        {"name": "csv_in", "connection_type": "file", "format": "csv",
         "configure": {"base_path": str(root / "input"),
                "read_options": {"header": "true", "inferSchema": "true"}}},
        {"name": "parquet_out", "connection_type": "file", "format": "parquet",
         "configure": {"base_path": str(root / "output")}},
    ],
    "dataflows": [
        {"name": "orders_csv_to_parquet", "stage": "bronze2silver",
         "processing_mode": "batch",
         "source": {"connection_name": "csv_in", "table": "orders"},
         "destination": {"connection_name": "parquet_out", "table": "orders",
                         "load_type": "full_load"},
         "transform": {}},
    ],
}
metadata_path = root / "metadata.json"
metadata_path.write_text(json.dumps(metadata, indent=2))
print(f"Created {metadata_path}")
```

```bash
python prepare_quickstart.py
```

### Part 2 — Run the pipeline

```python
# run_quickstart.py
from pathlib import Path

from datacoolie.engines.polars_engine import PolarsEngine
from datacoolie.platforms.local_platform import LocalPlatform
from datacoolie.metadata.file_provider import FileProvider
from datacoolie.orchestration.driver import DataCoolieDriver

root = Path("dc_quickstart")
metadata_path = root / "metadata.json"

platform = LocalPlatform()
engine = PolarsEngine(platform=platform)
provider = FileProvider(config_path=str(metadata_path), platform=platform)

with DataCoolieDriver(engine=engine, metadata_provider=provider) as driver:
    result = driver.run(stage="bronze2silver")
    print(f"Completed: {result.succeeded}/{result.total}")
```

```bash
python run_quickstart.py
```

Swap `PolarsEngine` for `SparkEngine(spark, ...)` or `LocalPlatform()` for
`AwsPlatform` / `FabricPlatform` / `DatabricksPlatform` — the metadata stays
the same.

## What to do next

- Use your own files while keeping the same runner pattern: <https://datacoolie.github.io/datacoolie/getting-started/use-your-own-data/>
- Build a multi-stage bronze→silver tutorial flow: <https://datacoolie.github.io/datacoolie/getting-started/first-dataflow/>
- Learn the metadata model field by field: <https://datacoolie.github.io/datacoolie/how-to/metadata-guide/>

## Testbed & scenarios

See [usecase-sim/README.md](usecase-sim/README.md) for a ready-made integration
testbed that exercises every `{polars,spark} × {file,database,api} × {local,aws}`
combination, plus lakehouse maintenance and a Docker-compose backend stack.

## License

[AGPL-3.0-or-later](LICENSE) — free and open source.

See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution terms.
