Metadata-Version: 2.5
Name: dte-airflow
Version: 0.1.0
Summary: Batch data transformation engine that runs as an Airflow 3 app: a visual pipeline builder over DuckDB, ibis and dlt
Project-URL: Homepage, https://github.com/flowexperts/dte-airflow
Project-URL: Source, https://github.com/flowexperts/dte-airflow
Project-URL: Issues, https://github.com/flowexperts/dte-airflow/issues
Author: Bhavani Ravi
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: airflow,data-pipelines,dlt,duckdb,etl,ibis
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Apache Airflow
Classifier: Framework :: Apache Airflow :: Provider
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Requires-Python: >=3.12
Requires-Dist: alembic<2.0,>=1.13
Requires-Dist: asyncpg<1.0,>=0.29
Requires-Dist: croniter<7.0,>=2.0
Requires-Dist: dlt[duckdb]<2.0,>=1.4
Requires-Dist: duckdb<2,>=1.1
Requires-Dist: fastapi<1.0,>=0.115
Requires-Dist: httpx<1.0,>=0.27
Requires-Dist: ibis-framework[duckdb,postgres,sqlite]<13.0,>=12.0
Requires-Dist: jmespath
Requires-Dist: opentelemetry-api<2.0,>=1.27
Requires-Dist: opentelemetry-sdk<2.0,>=1.27
Requires-Dist: prometheus-client<1.0,>=0.20
Requires-Dist: pydantic-settings<3.0,>=2.4
Requires-Dist: pydantic<3.0,>=2.7
Requires-Dist: pymongo<5.0,>=4.6
Requires-Dist: python-dateutil<3.0,>=2.8
Requires-Dist: python-multipart<1.0,>=0.0.9
Requires-Dist: pyyaml
Requires-Dist: sqlalchemy[asyncio]<3.0,>=2.0
Requires-Dist: sqlglot>=23.4
Requires-Dist: sqlmodel<0.1,>=0.0.22
Requires-Dist: uvicorn[standard]<1.0,>=0.30
Provides-Extra: airflow
Requires-Dist: apache-airflow-providers-anthropic>=1.1; extra == 'airflow'
Requires-Dist: apache-airflow-providers-openai>=1.8; extra == 'airflow'
Requires-Dist: apache-airflow>=3.3; extra == 'airflow'
Provides-Extra: amazon
Requires-Dist: apache-airflow-providers-amazon[s3fs]; extra == 'amazon'
Provides-Extra: azure
Requires-Dist: apache-airflow-providers-microsoft-azure; extra == 'azure'
Provides-Extra: bigquery-source
Requires-Dist: ibis-framework[bigquery]<13.0,>=12.0; extra == 'bigquery-source'
Provides-Extra: bigquery-writer
Requires-Dist: dlt[bigquery]<2.0,>=1.4; extra == 'bigquery-writer'
Provides-Extra: dev
Requires-Dist: anthropic<2.0,>=1.0; extra == 'dev'
Requires-Dist: httpx<1.0,>=0.27; extra == 'dev'
Requires-Dist: hypothesis<7.0,>=6.100; extra == 'dev'
Requires-Dist: mongomock<5.0,>=4.1; extra == 'dev'
Requires-Dist: mutmut<4.0,>=2.5; extra == 'dev'
Requires-Dist: openai<3.0,>=1.50; extra == 'dev'
Requires-Dist: pdfplumber<1.0,>=0.11; extra == 'dev'
Requires-Dist: psycopg2-binary<3.0,>=2.9; extra == 'dev'
Requires-Dist: pyarrow>=15; extra == 'dev'
Requires-Dist: pypdf<7.0,>=5.0; extra == 'dev'
Requires-Dist: pytest-asyncio<1.0,>=0.23; extra == 'dev'
Requires-Dist: pytest-cov<7.0,>=5.0; extra == 'dev'
Requires-Dist: pytest<9.0,>=8.0; extra == 'dev'
Requires-Dist: requests-mock<2.0,>=1.11; extra == 'dev'
Requires-Dist: ruff<1.0,>=0.6; extra == 'dev'
Provides-Extra: google
Requires-Dist: apache-airflow-providers-google; extra == 'google'
Provides-Extra: models
Requires-Dist: apache-airflow-providers-anthropic>=1.1; extra == 'models'
Requires-Dist: apache-airflow-providers-openai>=1.8; extra == 'models'
Provides-Extra: mongo
Requires-Dist: apache-airflow-providers-mongo; extra == 'mongo'
Provides-Extra: mssql
Requires-Dist: apache-airflow-providers-microsoft-mssql; extra == 'mssql'
Requires-Dist: dlt[mssql]<2.0,>=1.4; extra == 'mssql'
Requires-Dist: ibis-framework[mssql]<13.0,>=12.0; extra == 'mssql'
Provides-Extra: mysql-source
Requires-Dist: ibis-framework[mysql]<13.0,>=12.0; extra == 'mysql-source'
Provides-Extra: mysql-writer
Requires-Dist: dlt[sqlalchemy]<2.0,>=1.4; extra == 'mysql-writer'
Requires-Dist: pymysql<2.0,>=1.1; extra == 'mysql-writer'
Provides-Extra: parquet
Requires-Dist: fsspec>=2024.6; extra == 'parquet'
Requires-Dist: pyarrow>=15; extra == 'parquet'
Provides-Extra: postgres-writer
Requires-Dist: dlt[postgres]<2.0,>=1.4; extra == 'postgres-writer'
Requires-Dist: psycopg[binary]<4.0,>=3.1; extra == 'postgres-writer'
Provides-Extra: sftp
Requires-Dist: apache-airflow-providers-sftp; extra == 'sftp'
Requires-Dist: sshfs; extra == 'sftp'
Provides-Extra: snowflake-source
Requires-Dist: ibis-framework[snowflake]<13.0,>=12.0; extra == 'snowflake-source'
Provides-Extra: snowflake-writer
Requires-Dist: dlt[snowflake]<2.0,>=1.4; extra == 'snowflake-writer'
Provides-Extra: trino
Requires-Dist: ibis-framework[trino]<13.0,>=12.0; extra == 'trino'
Requires-Dist: trino<1.0,>=0.329; extra == 'trino'
Provides-Extra: unstructured
Requires-Dist: pdfplumber<1.0,>=0.11; extra == 'unstructured'
Requires-Dist: pypdf<7.0,>=5.0; extra == 'unstructured'
Description-Content-Type: text/markdown

# dte — map it once, run it forever

> **Beta.** Interfaces and the `dte` schema may change between releases. Don't run it on production data without backups. Issues welcome; PRs from approved collaborators only (see [CONTRIBUTING](./CONTRIBUTING.md)).

Your domain expert maps the source — once, click by click. Airflow runs it every day after.

dte is a batch data transformation engine that runs as an Airflow 3 app: a visual pipeline builder on top of a typed backend (DuckDB + ibis + dlt) that turns hand-drawn field mappings into reproducible runs. Every pipeline is an Airflow DAG; Airflow owns login, connections, scheduling and run history.

## Quickstart

DTE is an Airflow plugin. Install it into the Airflow 3.3+ you already run, with
Airflow's own constraints file, unmodified:

```bash
AIRFLOW=3.3.2 PY=3.12
curl -fsSLo constraints.txt \
  https://raw.githubusercontent.com/apache/airflow/constraints-$AIRFLOW/constraints-$PY.txt
pip install dte-airflow --constraint constraints.txt
export DTE_DATABASE_URL=postgresql://user:pass@host:5432/dte   
dte db upgrade
```

Then drop the one-line DAG file into your DAGs folder and restart Airflow. Full
steps, extras and environment: [Install into your own Airflow](https://dte.flowexperts.ai/docs/self-hosting/install-plugin).

DTE is under **Plugins ▸ DTE Pipelines** in Airflow's left nav (Transform, Quality
Check); sources, destinations and lookup tables are under **Admin**. Eval and
Visualize pipelines are API-only for now.

1. Use any Airflow connection whose provider is installed (**Admin → Connections**,
   `AIRFLOW_CONN_*` variables or a secrets backend). See [Connections](#connections).
2. Under **Admin**, create a DTE source and destination that use it; then create a pipeline in **DTE Pipelines**.
3. Click **Run**. The run is a dag run of `dte__<pipeline id>`, visible in both UIs.

### No Airflow yet? Try it with Docker Compose

```bash
git clone https://github.com/flowexperts/dte-airflow.git
cd dte-airflow
scripts/bootstrap.sh          # writes .env: fresh secrets + admin password
docker compose up -d --build
```

Open http://localhost:8080 and log in as `admin` with the password `bootstrap.sh`
printed (it's also in `.env`). The stack listens on localhost only and is meant for
local use, not a server.

Ports:

- `8080` — Airflow (UI, REST API, DTE at `/dte`)
- `5432` — postgres (Airflow metadata; DTE tables in the `dte` schema)
- `5433` — demo warehouse

## What's inside

- `backend/` — Python package `dte` (FastAPI API, ibis compiler, DuckDB engine, alembic migrations, tests).
- `frontend/` — React app built by Vite into an Airflow `react_apps` bundle; renders natively inside Airflow at `/plugin/dte` with Airflow's theme.
- `backend/dte/airflow/` — the Airflow integration: plugin, DAG factory, operator, connection adapter.
- `dags/dte_dags.py` — one line that turns every DTE pipeline and schedule into a DAG.
- `plans/` — numbered feature plans documenting the architecture and each shipped change.

## Testing

Three tiers — run only what the change touches:

| Tier | Command | Requires |
|------|---------|----------|
| Unit | `cd backend && uv run pytest` | nothing |
| Integration (Postgres) | `cd backend && uv run pytest tests/integration -m integration` | `DTE_DATABASE_URL` |
| Airflow (stock image + emulators) | `scripts/test-pip-install.sh` | Docker, pnpm, uv |
| Scenario (end-to-end) | `cd backend && uv run pytest tests/scenarios -m scenario -v` | nothing |

### Airflow tier

`scripts/test-pip-install.sh` builds the wheel, installs it into a stock
`apache/airflow` image with Airflow's constraints file unmodified, and runs
pipelines through Airflow, each through its provider: MinIO (S3, amazon),
Azurite (Azure, microsoft-azure), fake-gcs-server (GCS, google), an SFTP server
(sftp) and Postgres (postgres). See
[Install into your own Airflow](https://dte.flowexperts.ai/docs/self-hosting/install-plugin).

## Features

> Legend: **Available** — shipped and working · **Hidden** — implemented, off by default (feature flag) · **Planned** — stubbed or in progress

### Sources

| Source | Type | Status |
|--------|------|--------|
| Local filesystem (JSON, JSONL, CSV, Parquet, Excel) | filesystem | Available |
| AWS S3 (JSON, CSV, Parquet, Excel) | s3 | Available |
| Google Cloud Storage | gcs | Available |
| Azure Blob Storage / ADLS | azure | Available |
| SFTP | sftp | Available |
| PostgreSQL | sql_database | Available |
| MySQL | sql_database | Available |
| SQL Server | sql_database | Available |
| SQLite | sql_database | Available |
| Snowflake | sql_database | Available |
| BigQuery | sql_database | Available |
| MongoDB | mongodb | Available |
| REST API (with pagination) | rest_api | Available |

### Destinations

| Destination | Status |
|-------------|--------|
| JSON, CSV, Excel or Parquet file — local, S3, GCS, Azure or SFTP | Available |
| PostgreSQL | Available |
| SQL database (MySQL, SQL Server, Snowflake, BigQuery) | Available |

### Transform Steps

| Step | Description | Status |
|------|-------------|--------|
| Filter — Basic | GUI-built SQL `WHERE` (no injection surface) | Available |
| Filter — Advanced | Raw SQL expression | Available |
| Rename | Rename one or more columns | Available |
| Retype | Cast a column to a target type | Available |
| Map · Lookup | Join a saved lookup table to replace values | Available |
| Map · Expression | Derive a new column from a raw SQL expression | Available |
| Map · Calculation | Derive a column from a typed formula (same backend as expression) | Available |
| Sort | Order rows by one or more fields, asc or desc | Available |
| Aggregate | Group-by with SUM / COUNT / AVG / MIN / MAX | Available |
| Join | Inner / left / right / outer join across two sources | Available |
| Pivot | Reshape rows to columns (DuckDB PIVOT) | Available |
| Select / Drop fields | Keep or drop a set of columns, with optional aliases | Available |
| Check | Data-quality assertion — fail or warn when the expression is false (Quality Check pipelines only) | Available |
| Profile columns | Per-column null count, distinct count, min/max stats (Quality Check pipelines only) | Available |

### Connections

DTE uses Airflow connections and the provider that owns them does the auth, so IAM
roles, service principals, managed identities and service-account keys work as they
do in your other DAGs. A connection type shows up in DTE only when its provider is
installed; otherwise the picker names the package to install.

| Airflow connection type | Provider | In DTE |
|---|---|---|
| `aws` | amazon (`[s3fs]`) | S3 files |
| `google_cloud_platform`, `gcpbigquery` | google | GCS files; BigQuery (`gcpbigquery`, or extra `dte_type: bigquery`) |
| `wasb`, `adls`, `azure_data_lake` | microsoft-azure | Azure Blob / ADLS files |
| `sftp` | sftp (+ `sshfs`) | SFTP files |
| `postgres`, `mysql`, `mssql`, `sqlite`, `snowflake` | the matching provider | SQL tables (via the hook's SQLAlchemy URL) |
| `mongo` | mongo | MongoDB collections |
| `http` | http | REST APIs |
| `fs` | standard | Local files |

Connections resolve like any task's: secrets backends, `AIRFLOW_CONN_*` variables,
then the metadata DB. **Test** calls the provider's own connection test (follows
Airflow's `[core] test_connection`).

### Runtime & Pipeline Features

| Feature | Status |
|---------|--------|
| Runs execute as Airflow dag runs (`dte__<pipeline>`) | Available |
| Schedules: cron, or after upstream pipelines (Airflow Assets) | Available |
| Pipeline lineage as Airflow Assets | Available |
| Pipelines as YAML files in git (`DTE_PIPELINE_STORE=file`), `dte validate` / `dte run` | Opt-in |
| Preview / dry run (sample rows, compile-time errors surfaced) | Available |
| Run history per pipeline | Available |
| Run output peek (`GET /api/runs/{id}/output`) | Available |
| Template parameters — `{{params.x}}` in step expressions | Available |
| System variables — `{{today}}` resolves at trigger time | Available |
| Schema inference from an in-flight source body | Available |
| Nested struct fields via dot syntax (`order.amount`) | Available |
| Lookup tables — CRUD + referenced by Map · Lookup step | Available |
| Per-step run timeline events, grouped and timed in the task log | Available |
| Quality contracts checked before the write | Available |

### Engines

| Engine | Status |
|--------|--------|
| DuckDB (default) | Available |
| Trino | Hidden — enable with `DTE_TRINO_ENABLED=true` |

### Auth & Access Control

| Feature | Status |
|---------|--------|
| Airflow login and auth manager | Available |
| DTE permissions derived from Airflow DAG access | Available |

## Docs

Full documentation: **[dte.flowexperts.ai/docs](https://dte.flowexperts.ai/docs/)**. The source lives in
[`docs/user/`](./docs/user/); edits there publish on merge to `main`.

- **[User guide](https://dte.flowexperts.ai/docs/)** — start here if you're using the app.
- **[Install into your Airflow](https://dte.flowexperts.ai/docs/self-hosting/install-plugin)** — the pip install.
- **[Try it with Docker Compose](https://dte.flowexperts.ai/docs/self-hosting/install)** — local stack.
- **[Architecture](https://dte.flowexperts.ai/docs/architecture/overview)** — how it fits together.

## Built on

DTE is a thin layer over open-source projects other people build and maintain:

- [Apache Airflow](https://airflow.apache.org) schedules, runs and records every pipeline.
- [DuckDB](https://duckdb.org) executes each step, in process, on the Airflow worker.
- [ibis](https://ibis-project.org) compiles steps to SQL and pushes work down to your database.
- [sqlglot](https://github.com/tobymao/sqlglot) translates SQL between dialects.
- [dlt](https://dlthub.com) loads REST APIs and MongoDB, and writes to SQL destinations.

Every project and its license: https://dte.flowexperts.ai/open-source/

## Contributing and security

See [CONTRIBUTING.md](./CONTRIBUTING.md), [SECURITY.md](./SECURITY.md) and the [Code of Conduct](./CODE_OF_CONDUCT.md).

## License

Apache-2.0 — see [LICENSE](./LICENSE).

Apache Airflow, Airflow and the Airflow logo are trademarks of the Apache Software Foundation. DTE is an independent project, not affiliated with or endorsed by the ASF.
