Metadata-Version: 2.4
Name: hydra-etl
Version: 0.12.1
Summary: Declarative ETL engine with a CLI, a REST API and a visual studio.
Author: Bechir Bejaoui
License-Expression: AGPL-3.0-or-later
Project-URL: Homepage, https://hydraetl.com
Project-URL: Documentation, https://hydraetl.com
Project-URL: Repository, https://github.com/bejaouibechir/Hydra
Project-URL: Issues, https://github.com/bejaouibechir/Hydra/issues
Project-URL: Changelog, https://github.com/bejaouibechir/Hydra/blob/main/CHANGELOG.md
Keywords: etl,data,pipeline,workflow,yaml,orchestration
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Requires-Dist: pydantic>=2.0
Requires-Dist: click>=8.1
Requires-Dist: pandas>=2.0
Requires-Dist: python-dotenv>=1.0
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == "server"
Requires-Dist: uvicorn[standard]>=0.29; extra == "server"
Requires-Dist: apscheduler>=3.10; extra == "server"
Provides-Extra: native
Requires-Dist: hydra-native<0.2,>=0.1; extra == "native"
Requires-Dist: pyarrow>=14.0; extra == "native"
Provides-Extra: duckdb
Requires-Dist: duckdb>=0.10; extra == "duckdb"
Provides-Extra: parquet
Requires-Dist: pyarrow>=14.0; extra == "parquet"
Provides-Extra: mysql
Requires-Dist: mysql-connector-python>=8.3; extra == "mysql"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == "postgres"
Provides-Extra: mssql
Requires-Dist: pymssql>=2.3; extra == "mssql"
Provides-Extra: mongodb
Requires-Dist: pymongo>=4.6; extra == "mongodb"
Provides-Extra: http
Requires-Dist: requests>=2.31; extra == "http"
Provides-Extra: mcp
Requires-Dist: mcp>=2.0; python_version >= "3.10" and extra == "mcp"
Provides-Extra: all
Requires-Dist: hydra-etl[server]; extra == "all"
Requires-Dist: hydra-etl[duckdb]; extra == "all"
Requires-Dist: hydra-etl[native]; extra == "all"
Requires-Dist: hydra-etl[parquet]; extra == "all"
Requires-Dist: hydra-etl[mysql]; extra == "all"
Requires-Dist: hydra-etl[postgres]; extra == "all"
Requires-Dist: hydra-etl[mssql]; extra == "all"
Requires-Dist: hydra-etl[mongodb]; extra == "all"
Requires-Dist: hydra-etl[http]; extra == "all"
Requires-Dist: hydra-etl[mcp]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: httpx>=0.24; extra == "dev"
Requires-Dist: requests-mock>=1.11; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: black>=24.0; extra == "dev"
Dynamic: license-file

# Hydra ETL

**Your data pipelines are described, not programmed.**

[![CI](https://github.com/bejaouibechir/Hydra/actions/workflows/ci.yml/badge.svg)](https://github.com/bejaouibechir/Hydra/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/hydra-etl.svg)](https://pypi.org/project/hydra-etl/)
[![Python](https://img.shields.io/pypi/pyversions/hydra-etl.svg)](https://pypi.org/project/hydra-etl/)
[![License: AGPL v3](https://img.shields.io/badge/license-AGPL--3.0--or--later-blue.svg)](LICENSE)
[![Status: beta](https://img.shields.io/badge/status-beta-orange.svg)](#project-status)
[![Glama quality score](https://glama.ai/mcp/servers/bejaouibechir/Hydra/badges/score.svg)](https://glama.ai/mcp/servers/bejaouibechir/Hydra)
<a href="https://www.saashub.com/hydra-etl?utm_source=badge&utm_campaign=badge&utm_content=hydra-etl&badge_variant=color&badge_kind=approved"><img src="https://cdn-b.saashub.com/img/badges/approved-color.png?v=1" alt="Hydra ETL on SaaSHub" height="20"></a>

**[Website](https://hydraetl.com)** · **[Video tutorials](https://hydraetl.com/academy/?utm_source=github&utm_medium=readme&utm_campaign=academy)** · **[Docs & playground](https://hydraetl.com/playground)** · **[MCP server](docs/MCP.md)** · **[VS Code extension](#vs-code-extension)** · **[Changelog](CHANGELOG.md)**

<p align="center">
  <a href="https://youtu.be/9cCMWptQEgE"><img src="assets/readme/teaser-cli2.jpg" alt="Video (4:35): create a Hydra ETL project with hdrctl init and explore its structure" width="820"></a>
  <br>
  <sub>▶ Watch (4:35): create a project with <code>hdrctl init</code> and explore its structure. More videos in <a href="https://hydraetl.com/academy/?utm_source=github&amp;utm_medium=readme&amp;utm_campaign=academy">Hydra Academy</a>.</sub>
</p>

Hydra is an open-source declarative ETL engine. You write what a pipeline *is* in YAML;
Hydra validates it before touching any data, then runs it. The same manifests run from
the terminal, a REST API, or a visual editor in your browser, with one `pip install` and nothing else to deploy.

## Try it without installing anything

Two sandboxes run Hydra in your browser. Nothing is installed on your machine,
and nothing is left behind when you close the tab.

### Guided labs — no account needed

**[killercoda.com/hydra-etl](https://killercoda.com/hydra-etl)**

Killercoda gives you a Linux terminal inside your browser, next to a short
lesson that tells you exactly what to type. Each lab takes ten to fifteen
minutes and checks your work as you go. There is nothing to sign up for.

Start with *Meet hdrctl* to see what a job is, then *Parameters and secrets* to
run one pipeline against two databases.

### A full environment, with the visual editor

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/bejaouibechir/hydra-demo)

GitHub Codespaces starts a real machine in the cloud and opens an editor in your
browser. It needs a free GitHub account, and it runs within GitHub's free
monthly allowance.

This one comes with a MySQL database already running beside it and a working
pipeline. **Once it opens, type `./tour`** — a five-minute walkthrough that
shows you each command before running it. It is the only thing you have to type.

Unlike the labs, this environment includes **Hydra Studio**, the visual editor,
so you can see the same pipeline as a diagram and run it from there.

<!-- TODO: replace with a short GIF: build a job in Studio → hdrctl validate → hdrctl run -->

<img src="assets/hydrastudio.png" alt="Hydra Studio: visual editor for jobs and workflows" width="720">

---

## Quick start

```bash
pip install "hydra-etl[server]"
hdrctl serve --open            # Studio + API on http://localhost:5678
```

Or stay in the terminal:

```bash
hdrctl init my_project                 # or `hdrctl init .` in an empty folder
hdrctl run my_project --dry-run        # validate the whole project, run nothing
hdrctl run my_project                  # run every workflow (or `hdrctl run .` from inside)
hdrctl validate my_project/jobs/job1   # strict validation of one job, no data touched
hdrctl run my_project/jobs/job1        # run one job alone
```

A project is a set of workflows; a workflow is a set of jobs and actions.

`init` scaffolds a full project:

```
my_project
├── .gitignore           keeps .hydra/secrets and .env out of git
├── .hydra/              project marker, Studio data, and secrets (name=value, git-ignored)
├── Data/                input data
├── environments/        parameter values per environment (dev.yaml, prod.yaml…)
├── jobs/
│   └── job1/
│       ├── README.md
│       ├── data/input.csv
│       ├── destinations.yaml
│       ├── pipeline.yaml
│       ├── sources.yaml
│       └── transformations.yaml
├── outputs/             files produced by the jobs
├── parameters.yaml      project parameters, used as {{ param:NAME }}
├── README.md
├── scripts/             scripts called by workflow actions
└── workflows/
    ├── thread.yaml      execution order of the workflows
    └── workflow1.yaml
```

Six job templates are available with `--template` (`basic`, `csv`, `mysql`, `postgres`, `api`,
`mongodb`). `workflows/thread.yaml` lists the workflows in stages, each `sequential` or `parallel`,
with an optional `delay_after` and an `active` flag per workflow:

```yaml
version: "1.0"
thread:
  - mode: sequential
    delay_after: 30s          # optional pause before the next stage
    workflows:
      - name: workflow1
      - name: workflow2
        active: false         # skipped, without removing it
  - mode: parallel
    workflows:
      - name: export-a
      - name: export-b
loops:                        # optional
  - from: export-b            # when export-b finishes…
    to: workflow1             # …go back to workflow1
    times: 3                  # 3 passes in total
    delay: 5m                 # optional pause before each repeat
```

A loop can link any two workflows of the thread (`to` comes before `from`, or is the same
workflow); loops must not overlap. In Studio, the project page edits the same file
(**Execution order** → **Define**), draws it (**Preview**) and has a **Run project** button.
`hdrctl run <project>` checks the thread, every active workflow and the job folders they use
before starting anything, then runs the stages in order. A failed workflow stops the project;
in a parallel stage the workflows already started finish first. Without `thread.yaml`, every
`workflows/*.yaml` runs one after the other, in alphabetical order. Parameters come from the
project's `parameters.yaml`, `environments/<env>.yaml` (`--env`) and `-P KEY=VALUE`, in that
order of precedence. Workflow triggers (`schedule`, `webhook`) are ignored by `hdrctl run`;
they apply under `hdrctl serve`.

No Node, no build step, no database, no message broker. Python 3.9+ on Linux, macOS or Windows.

---

## Why Hydra

- **Validate before you run.** `hdrctl validate` checks sources, steps, types and destinations
  deterministically, before any data is read or written.
- **Declarative, visual, zero deployment.** YAML manifests, a browser editor that outputs the same YAML,
  and a single Python process. No cluster, no JVM, no container required.
- **Built for CI/CD.** Manifests are versioned and reviewed like code; `hdrctl` scaffolds, validates,
  tests and runs from a terminal or a CI job.
- **Built-in scheduler.** DAG workflows with dependencies, parallel branches, delayed retries, cron triggers,
  conditional guards and eleven actions (webhook, email, Bash, PowerShell, SSH, Python…).
- **One job, many environments.** `{{ param: }}`, `{{ env: }}` and `${SECRET:}` keep the manifest identical
  across dev, staging and production.
- **Two engines per operation.** pandas or DuckDB, chosen step by step, with optional Rust acceleration
  (see [Native acceleration](#native-acceleration-optional)).
- **AI-ready.** An MCP server lets Claude, Cursor or VS Code write pipelines that Hydra validates.

<p>
  <img src="assets/3/validate-before-run.svg" alt="Validation before execution" width="49%">
  <img src="assets/2/declarative-visual-zero-deploy.svg" alt="Declarative, visual, zero deployment" width="49%">
</p>

### How it compares

|                      | Hydra                    | Airflow                           | dbt                        | Airbyte             | NiFi / Apache Hop        |
| -------------------- | ------------------------ | --------------------------------- | -------------------------- | ------------------- | ------------------------ |
| Pipelines defined in | YAML                     | Python                            | SQL + YAML                 | UI / config         | UI flows                 |
| Scope                | Extract, transform, load | Orchestration                     | Transform in the warehouse | Extract & load      | Extract, transform, load |
| Visual editor        | Included                 | Monitoring UI                     | Not in dbt Core            | Included            | Included                 |
| To get started       | `pip install`            | Scheduler, webserver, metadata DB | A data warehouse           | Docker / Kubernetes | JVM                      |

Hydra is not a replacement for all of these. It targets file-and-database pipelines that should stay
readable and run without extra infrastructure.

---

## What you get

| Component     | Description                                                              |
| ------------- | ------------------------------------------------------------------------ |
| **Engine**    | Declarative jobs: one source, N transformations, one destination         |
| **CLI**       | `hdrctl`: scaffold, validate, run, inspect. English and Spanish          |
| **API**       | FastAPI, with interactive docs at `/docs`                                |
| **Studio**    | Visual editor for jobs and workflows; opens an existing project folder   |
| **Workflows** | Multi-job DAG with dependencies, actions, retries and runtime parameters |
| **MCP**       | Natural-language pipeline authoring, validated by Hydra                  |

**Connectors** (sources and destinations):

| <img src="assets/connectors/csv.png" alt="" width="40"><br>CSV | <img src="assets/connectors/json.png" alt="" width="40"><br>JSON | <img src="assets/connectors/parquet.png" alt="" width="40"><br>Parquet | <img src="assets/connectors/mysql.png" alt="" width="40"><br>MySQL / MariaDB | <img src="assets/connectors/postgresql.png" alt="" width="40"><br>PostgreSQL | <img src="assets/connectors/sqlserver.png" alt="" width="40"><br>SQL Server | <img src="assets/connectors/mongo.png" alt="" width="40"><br>MongoDB | <img src="assets/connectors/api.png" alt="" width="40"><br>Web API |
|:--------------------------------------------------------------:|:----------------------------------------------------------------:|:----------------------------------------------------------------------:|:----------------------------------------------------------------------------:|:----------------------------------------------------------------------------:|:---------------------------------------------------------------------------:|:--------------------------------------------------------------------:|:------------------------------------------------------------------:|

**Transformation engines:** pandas and DuckDB.

---

## Who it is for

- **Data engineers and analysts** who need repeatable file-and-database pipelines without standing up
  infrastructure for them.
- **Teams where pipelines must stay readable** by people who do not write Python: a YAML manifest
  reviewed in a pull request, not a script.
- **Developers embedding ETL in a product**, who want a CLI and a REST API over the same engine.

---

## A job in four files

A job is a folder. Four manifests describe it, and each one answers a single question.

**`sources.yaml`**: where the data comes from

```yaml
version: "1.0"
sources:
  src_input:
    type: csv
    extract:
      table: ./input.csv
```

**`transformations.yaml`**: how it is reshaped

```yaml
version: "1.0"
steps:
  - cast:
      mapping:
        amount: float
  - filter:
      expr: "amount > 0"
  - aggregate:
      by: [name]
      agg:
        total: { func: sum, col: amount }
```

A CSV carries no types, so `cast` comes before any numeric comparison.

**`destinations.yaml`**: where it goes

```yaml
version: "1.0"
destinations:
  dest_output:
    type: csv
    load:
      table: ./output.csv
      mode: replace          # append | replace | upsert
```

**`pipeline.yaml`**: which source feeds which destination

```yaml
version: "1.0"
pipeline:
  from: src_input
  to: dest_output
```

```bash
hdrctl validate my_job && hdrctl run my_job
```

---

## Workflows: order several jobs

```yaml
version: "1.0"
workflow:
  name: daily_etl
  trigger:
    type: schedule
    cron: "0 8 * * *"
  steps:
    - name: extract
      type: job
      job: ./jobs/extract
      depends_on: []

    - name: transform
      type: job
      job: ./jobs/transform
      depends_on: ["extract"]     # always a list, supports fan-in

    - name: notify
      type: action
      action: webhook
      params: { url: "{{ env:WEBHOOK_URL }}" }
      depends_on: ["transform"]
      on_failure: skip
```

```bash
hdrctl workflow add daily_sales        # new empty workflow, added at the end of thread.yaml
hdrctl workflow list                   # workflows in thread order: stage, active/inactive, steps
hdrctl workflow validate daily_sales   # by name inside a project, or by file path
hdrctl workflow run daily_sales        # one workflow
hdrctl workflow remove daily_sales     # deletes the file and its thread entry; jobs are kept
hdrctl run .                           # every workflow, in the order of thread.yaml
```

Jobs are added to a workflow as a chain: each step runs after the previous one.

```bash
hdrctl job add load_orders                    # new job jobs/load_orders/, attached nowhere
hdrctl job add load_orders daily_sales        # new job, appended to the workflow
hdrctl job attach clean_orders daily_sales -p 1   # existing job, as step 1: the old step 1 now runs after it
hdrctl job detach clean_orders daily_sales    # the next step inherits its dependencies; the folder is kept
hdrctl job detach clean_orders                # from every workflow
hdrctl job list                               # each job: free, or the workflows it is in
```

`job add` uses the same templates as `init` (`--template mysql`, `postgres`…). These commands rewrite
the workflow file (comments in it are not kept) and refuse any change that would leave it invalid.

The `workflow` commands work on the project that contains the current folder, or on the one given
with `--project <path>`; outside a project they stop with an error. Studio keeps `thread.yaml` in
step too: a workflow created or deleted in Studio is added to or removed from it.

An edge is a **dependency**, not a pipe: it decides *when* a job runs, never what data reaches it.
Steps that share no dependency run in parallel.

<img src="assets/5/dag-workflows-built-in-scheduler.svg" alt="DAG workflows with a built-in scheduler" width="720">

---

## Parameters

Values can be declared once and reused, or created while the workflow runs.

```yaml
- filter:
    expr: "region == '{{ param:region }}'"
```

`{{ param:NAME }}` reads a parameter and `{{ env:NAME }}` an environment variable.
`${SECRET:NAME}` reads a secret. It is looked up first in the environment, under the name upper-cased with
dots and dashes turned into underscores — so `${SECRET:db.password}` reads `DB_PASSWORD` (CI variables, a
secret manager, your shell, a `.env` file) — then in the project's `.hydra/secrets` file:

```
# .hydra/secrets — one name=value per line, listed in .gitignore by hdrctl init
db.password=change-me
api.token=abc123
```

In Studio, the **Secrets** tab of the workflow editor manages the same file (names only; values are never
sent back to the browser). To keep that file outside the project, point to it with `HYDRA_SECRETS_FILE=<path>` or
`hdrctl run . --secrets-file <path>`. The environment always wins over the file, so production can supply
the values without touching the project.
The `set_param` and `assign_param` actions create and change parameters mid-run, so two jobs can share a
placeholder and produce different results.

---

## Install what you need

The base install is the engine and the CLI. Everything else is opt-in.

```bash
pip install hydra-etl                  # engine + CLI
pip install "hydra-etl[server]"        # + API + Studio
pip install "hydra-etl[postgres]"      # + PostgreSQL driver
pip install "hydra-etl[mssql]"         # + Microsoft SQL Server driver
pip install "hydra-etl[all]"           # everything
```

Available extras: `server`, `native`, `duckdb`, `parquet`, `mysql`, `postgres`, `mssql`, `mongodb`, `http`, `mcp`, `all`.

### Or run it in a container — optional

Hydra needs no container: it stays one `pip install` and one Python process.
If your team standardises on containers, or you would rather keep Python off
the host, the image carries the engine, the CLI, the API and the Studio:

```bash
docker run --rm -p 5678:5678 -v hydra-workspace:/workspace \
  ghcr.io/bejaouibechir/hydra:latest
```

Building it yourself, per system: [Linux](GHCR/README.md#linux) ·
[Windows](GHCR/README.md#windows) · [macOS](GHCR/README.md#macos).

---

## Serving

```bash
hdrctl serve                 # Studio and API on port 5678
hdrctl serve --open          # and open the browser
hdrctl serve --no-studio     # API only, for a headless server
hdrctl serve --port 8080
```

The server writes projects into the directory you launch it from. Interactive API docs are at `/docs`.

---

## Your AI assistant, connected (MCP)

Hydra ships an MCP server. Point Claude Desktop, Cursor, VS Code or any MCP client at it, and ask for a
pipeline in plain language.

```bash
pip install "hydra-etl[mcp]"
hydra-mcp
```

Your assistant writes the manifests; **Hydra validates them before anything is written**, and nothing runs
until you ask. Requires Python 3.10+. See [docs/MCP.md](docs/MCP.md) for client configuration.

Graded by [Glama](https://glama.ai/mcp/servers/bejaouibechir/Hydra), which builds the server in a
sandbox, runs security checks and scores the tool definitions:

[![hydra-etl MCP server – quality and maintenance score on Glama](https://glama.ai/mcp/servers/bejaouibechir/Hydra/badges/card.svg)](https://glama.ai/mcp/servers/bejaouibechir/Hydra)

---

## Native acceleration (optional)

Parts of the engine have a Rust implementation. It is optional and off by default: without it, Hydra
behaves exactly the same.

```bash
pip install "hydra-etl[native]"
HYDRA_BACKEND=rust hdrctl run ./jobs/sales
```

Or per operation, with a `hydra.backends.yaml` file next to the job:

```yaml
default: python
overrides:
  csv.read: rust      # currently the only accelerated operation
```

On one million rows, CSV reading is about four times faster, with byte-for-byte identical results
(parity checked on 26,000 CSV files and one million floats against CPython `repr()`). Hydra falls back to
Python automatically whenever that guarantee cannot be kept. Benchmarks: [`bench/`](bench/).

---

## VS Code extension

Completion and validation for Hydra manifests inside the editor, without installing Hydra.
Download the `.vsix` from the [latest release](https://github.com/bejaouibechir/Hydra/releases/latest),
or build it yourself from [`vscode-extension/`](vscode-extension/) (`python build_vsix.py`, no Node required).

<img src="assets/hydravs.png" alt="Hydra VS Code extension" width="720">

---

## Project status

**Beta.** The documented connectors and steps are implemented and covered by tests. The shape of the YAML
DSL is settled; any breaking change will be announced in the release notes before 1.0.
Use it on real work, and pin your version.

---

## Contributing

Issues, ideas and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).
If Hydra is useful to you, a ⭐ on GitHub helps others find it.

---

# License

Hydra ETL is open source under the **GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later)**.

## You can, for free

- Use Hydra ETL for any purpose, including inside a company and for commercial work.
- Run your own pipelines with it, internally or for clients.
- Modify it and redistribute it.

Using Hydra ETL to move **your** data does not make your data, your YAML pipelines or your other applications subject to the AGPL.

## The one obligation

If you **modify Hydra ETL itself** and make it available to others — including over a network, for example as a hosted service — you must publish your modified source code under the same license.

## Commercial license

If that obligation does not fit your case — typically, you embed a modified Hydra ETL in a proprietary product or offer it as a service without publishing your changes — a commercial license is available.

**Contact:** admin@hydraetl.com — reply within 2 business days.

## Support and services

- Support with response time commitments
- Help migrating existing scripts or tools to Hydra ETL
