Metadata-Version: 2.4
Name: kedro_dagster
Version: 0.7.0
Summary: Kedro-Dagster makes it easy to deploy Kedro projects to Dagster
Project-URL: Homepage, https://github.com/stateful-y/kedro-dagster
Project-URL: Documentation, https://kedro-dagster.readthedocs.io
Project-URL: Repository, https://github.com/stateful-y/kedro-dagster
Project-URL: Bug Tracker, https://github.com/stateful-y/kedro-dagster/issues
Author-email: Guillaume Tauzin <gtauzin@stateful-y.io>
Maintainer-email: Guillaume Tauzin <gtauzin@stateful-y.io>
License: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <3.15,>=3.11
Requires-Dist: cachetools>=4.1
Requires-Dist: dagster-dg-cli
Requires-Dist: dagster-webserver
Requires-Dist: dagster<2.0,>=1.10
Requires-Dist: kedro-datasets>=1.0.0
Requires-Dist: kedro>=1.0.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: mlflow
Requires-Dist: dagster-mlflow; extra == 'mlflow'
Requires-Dist: kedro-mlflow>=0.13; extra == 'mlflow'
Requires-Dist: mlflow>=2.20; extra == 'mlflow'
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/logo_light.png">
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/logo_dark.png">
    <img src="https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/logo_light.png" alt="Kedro-Dagster">
  </picture>
</p>

[![Python Version](https://img.shields.io/pypi/pyversions/kedro-dagster)](https://pypi.org/project/kedro-dagster/)
[![License](https://img.shields.io/github/license/stateful-y/kedro-dagster)](https://github.com/stateful-y/kedro-dagster/blob/main/LICENSE)
[![PyPI Version](https://img.shields.io/pypi/v/kedro-dagster)](https://pypi.org/project/kedro-dagster/)
[![Conda Version](https://img.shields.io/conda/vn/conda-forge/kedro-dagster.svg)](https://anaconda.org/conda-forge/kedro-dagster)
[![CodeCov](https://codecov.io/gh/stateful-y/kedro-dagster/branch/main/graph/badge.svg)](https://codecov.io/gh/stateful-y/kedro-dagster/branch/main)

[![Powered by Kedro](https://img.shields.io/badge/powered_by-kedro-ffc900?logo=kedro)](https://kedro.org)
[![Slack Organisation](https://img.shields.io/badge/slack-chat-blueviolet.svg?label=Kedro%20Slack&logo=slack)](https://slack.kedro.org)

## What is Kedro-Dagster?

The Kedro-Dagster plugin enables seamless integration between [Kedro](https://kedro.readthedocs.io/), a framework for creating reproducible and maintainable data science code, and [Dagster](https://dagster.io/), a data orchestrator for machine learning and data pipelines. This plugin makes use of Dagster's orchestration capabilities to automate and monitor Kedro pipelines effectively.

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/example/local_asset_graph_light.png">
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/example/local_asset_graph_dark.png">
    <img src="https://raw.githubusercontent.com/stateful-y/kedro-dagster/main/docs/assets/example/local_asset_graph_light.png" alt="Kedro-Dagster Asset Graph">
  </picture>
</p>

Currently, Kedro-Dagster supports Kedro versions 0.19.x and 1.x, and Dagster versions 1.10.x, 1.11.x, and 1.12.x.

## What are the features of Kedro-Dagster?

- **Configuration‑driven workflows:** Centralize orchestration settings in a `dagster.yml` file for each Kedro environment. Define jobs from filtered Kedro pipelines, assign executors, schedules.
- **Customization:** The core integration lives in the auto‑generated Dagster `definitions.py`. For advanced use cases, you can extend or override these definitions.
- **Kedro hooks preservation:** Kedro hooks are preserved and called at the appropriate time during pipeline execution, so custom logic (e.g., data validation, logging) continues to work seamlessly.
- **MLflow compatibility:** Use [Kedro-MLflow](https://github.com/Galileo-Galilei/kedro-mlflow) with Dagster’s [MLflow integration](https://dagster.io/integrations/dagster-mlflow) to track experiments, log models, and register artifacts.
- **Logger integration:** Unifies Kedro and Dagster logging so logs from Kedro nodes appear in the Dagster UI and are easy to trace and debug. Additionally, provides configuration to customize Dagster run loggers.
- **(Experimental) Dagster partition support:** Make use of Dagster's partitions to fan-out Kedro nodes acting on partitioned data.

## How to install Kedro-Dagster?

Install the Kedro-Dagster plugin using `pip`:

```bash
pip install kedro-dagster
```

or using `uv`:

```bash
uv pip install kedro-dagster
```

or using `conda`:

```bash
conda install -c conda-forge kedro-dagster
```

or using `mamba`:

```bash
mamba install -c conda-forge kedro-dagster
```

or alternatively, add `kedro-dagster` to your `requirements.txt` or `pyproject.toml` file.

## How to get started with Kedro-Dagster?

1. **Initialize the plugin in your Kedro project**

Use the following command to generate a `definitions.py` file, where all translated Kedro objects are available as Dagster objects, and a `dagster.yml` configuration file:

```bash
kedro dagster init --env <ENV_NAME>
```

2. **Configure jobs, executors, and schedules**

Define your job executors and schedules in the `dagster.yml` configuration file located in your Kedro project's `conf/<ENV_NAME>` directory. This file allows you to filter Kedro pipelines and assign specific executors and schedules to them.

```yaml
# conf/local/dagster.yml
schedules:
  daily: # Schedule name
    cron_schedule: "0 0 * * *" # Schedule parameters

executors: # Executor name
  sequential: # Executor parameters
    in_process:

  multiprocess:
    multiprocess:
      max_concurrent: 2

jobs:
  default: # Job name
    pipeline: # Pipeline filter parameters
      pipeline_name: __default__
    executor: sequential

  parallel_data_processing:
    pipeline:
      pipeline_name: data_processing
      node_names:
      - preprocess_companies_node
      - preprocess_shuttles_node
    schedule: daily
    executor: multiprocess

  data_science:
    pipeline:
      pipeline_name: data_science
    schedule: daily
    executor: sequential
```

3. **Launch the Dagster UI**

Start the Dagster UI to monitor and manage your pipelines using the following command:

```bash
kedro dagster dev --env <ENV_NAME>
```

The Dagster UI will be available at http://127.0.0.1:3000.

For a concrete use-case, see the [Kedro-Dagster example repository](https://github.com/stateful-y/kedro-dagster-example).

## How do I use Kedro-Dagster?

Full documentation is available at [https://kedro-dagster.readthedocs.io/](https://kedro-dagster.readthedocs.io/).

## Can I contribute?

We welcome contributions, feedback, and questions:

- **Report issues or request features:** [GitHub Issues](https://github.com/stateful-y/kedro-dagster/issues)
- **Join the discussion:** [GitHub Discussions](https://github.com/stateful-y/kedro-dagster/discussions)
- **Contributing Guide:** [CONTRIBUTING.md](https://github.com/stateful-y/kedro-dagster/blob/main/CONTRIBUTING.md)

If you are interested in becoming a maintainer or taking a more active role, please reach out to Guillaume Tauzin on [GitHub Discussions](https://github.com/stateful-y/kedro-dagster/discussions).

## Where can I learn more?

- Full documentation: [https://kedro-dagster.readthedocs.io/](https://kedro-dagster.readthedocs.io/)
- GitHub Discussions: [https://github.com/stateful-y/kedro-dagster/discussions](https://github.com/stateful-y/kedro-dagster/discussions)

For questions and discussions, you can also open a [discussion](https://github.com/stateful-y/kedro-dagster/discussions).

## License

This project is licensed under the terms of the [Apache-2.0 License](https://github.com/stateful-y/kedro-dagster/blob/main/LICENSE).

## Acknowledgements

This project is maintained by [stateful-y](https://stateful-y.io), an ML consultancy specializing in MLOps and data science & engineering. If you're interested in collaborating or learning more about our services, please visit our website.

<p align="center">
  <a href="https://stateful-y.io">
    <img src="docs/assets/made_by_stateful-y.png" alt="Made by stateful-y" width="200">
  </a>
</p>
