Metadata-Version: 2.4
Name: duckle
Version: 0.5.7
Summary: Local-first ETL/ELT pipelines that run on DuckDB. Headless CLI.
Author: Slothflow Labs
License: Apache-2.0
Project-URL: Homepage, https://duckle.org
Project-URL: Source, https://github.com/slothflowlabs/duckle
Project-URL: Issues, https://github.com/slothflowlabs/duckle/issues
Keywords: etl,elt,duckdb,data-engineering,pipeline,data-integration
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Rust
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Build Tools
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: duckdb-cli==1.5.4

# Duckle

**Local-first ETL/ELT pipelines that run on DuckDB.** Python builds the plan, DuckDB moves the rows. Your data never leaves the machine.

[![PyPI](https://img.shields.io/pypi/v/duckle.svg)](https://pypi.org/project/duckle/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/slothflowlabs/duckle/blob/main/LICENSE)
[![GitHub](https://img.shields.io/github/stars/slothflowlabs/duckle?style=social)](https://github.com/slothflowlabs/duckle)

```sh
pip install duckle
```

<img src="https://raw.githubusercontent.com/slothflowlabs/duckle/main/docs/assets/pypi-demo-install.svg" alt="Terminal: pip install duckle, then a job.py that reads a CSV, filters, derives columns and writes Parquet, running in 394 ms" width="660"/>

That installs a ~20 MB native binary plus the DuckDB CLI. No JVM, no Docker, no server, no account.

---

## No data passes through Python

Every method appends a node to a pipeline graph. Nothing executes until `.run()`, and then the whole graph is handed to the engine, compiled to SQL, and executed inside DuckDB.

That is the difference from a dataframe library: a billion-row job costs what the SQL costs, because it *is* SQL. There is no interpreter in the data path, and no `to_pandas()` escape hatch quietly pulling rows into memory.

```python
import duckle
from duckle import col

(duckle.read_csv("orders.csv")
    .where(col.amount >= 20)
    .derive(total="round(amount * 1.2, 2)", tag="f'{region}-{id}'")
    .write_parquet("out.parquet")
    .run())
```

## Python expressions, compiled to SQL

`amount * 1.2`, `f'{a}-{b}'`, `x if c else y`, `name.strip().upper()` and `email is None` are translated to vectorized DuckDB SQL **at plan time**:

```python
.derive(band="'high' if amount > 50 else 'low'")
# CASE WHEN ("amount" > 50) THEN 'high' ELSE 'low' END
```

An expression with no exact SQL equivalent is **rejected with the construct named**, never quietly executed somewhere slower. Comprehensions, `lambda`, indexing and `eval` are refused by name.

Prefer editor completion over strings? `col` builds the same tree with real operators, and fragments are reusable:

```python
eu_only = col.region.isin(["EU", "UK"])
p.where(eu_only & (col.amount >= 20))
```

See exactly what will run, before it runs:

```python
p.explain()   # prints the compiled SQL, one block per stage
```

## 359 components, not 10 file formats

Component ids map onto attribute paths, so anything in the catalog is reachable:

```python
duckle.src.salesforce(object="Account", authMode="clientCredentials")
duckle.xf.geo.reproject(geomColumn="geom", targetCrs="EPSG:3857")
duckle.snk.salesforce.bulk(operation="upsert", externalIdField="Ext__c")
```

104 sources, 66 sinks and 138 transforms: Postgres, MySQL, Oracle, SQL Server, Snowflake, Databricks, Teradata, SAP OData, Salesforce (including Bulk API 2.0), Kafka, WebSocket, S3, SFTP, IMAP, LanceDB and more.

```python
duckle.component_ids(contains="salesforce")
duckle.describe("snk.salesforce.bulk")   # settings, and which ones the engine ignores
```

## A CI gate that needs no engine

<img src="https://raw.githubusercontent.com/slothflowlabs/duckle/main/docs/assets/pypi-demo-validate.svg" alt="Terminal: duckle validate reports one failing and one passing pipeline, then exits 1" width="660"/>

`duckle validate` compiles every pipeline without opening a source or writing a sink, so it needs **no DuckDB, no credentials and no network**. Exit codes are stable and safe to gate on:

| code | meaning |
|---|---|
| `0` | clean |
| `1` | a real finding: a pipeline failed, or did not compile |
| `2` | the runner could not start: bad usage, missing engine |

```sh
duckle validate --json          # machine-readable, for a build step
duckle --pipeline my.json       # run one
```

Note: `validate` does not yet catch every missing required property, so a clean validate is not proof that a run will succeed.

## Code and canvas are the same file

Pipelines are the same JSON the [Duckle desktop studio](https://github.com/slothflowlabs/duckle) reads. A pipeline written in Python opens on the canvas, and one drawn on the canvas runs from Python:

```python
p.save("pipelines/orders.json")          # opens in the studio
duckle.from_json("pipelines/orders.json").run()
```

## Install notes

`duckle` depends on [`duckdb-cli`](https://pypi.org/project/duckdb-cli/), published by the DuckDB Foundation, so the engine arrives with the install and works offline. To pin your own build instead, set `DUCKLE_DUCKDB_BIN`.

Wheels ship for Linux, macOS and Windows on x86-64 and arm64. The wheel carries a compiled Rust binary and is Python-version independent (`py3-none-<platform>`).

---

Apache-2.0 &nbsp;·&nbsp; [GitHub](https://github.com/slothflowlabs/duckle) &nbsp;·&nbsp; [duckle.org](https://duckle.org) &nbsp;·&nbsp; [Issues](https://github.com/slothflowlabs/duckle/issues)
