Metadata-Version: 2.3
Name: dagster-ray
Version: 0.4.3
Summary: Dagster integration library for Ray
Keywords: dagster,ray,ETL,distributed
Author: Daniel Gafni
Author-email: Daniel Gafni <danielgafni16@gmail.com>
License: Apache-2.0
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: dagster>=1.10.0
Requires-Dist: eval-type-backport>=0.2.2 ; python_full_version < '3.10'
Requires-Dist: tenacity>=8.0.0
Requires-Dist: ray[client] ; extra == 'executor'
Requires-Dist: pyyaml ; extra == 'kuberay'
Requires-Dist: kubernetes>=20.0.0 ; extra == 'kuberay'
Requires-Dist: dagster-k8s>=0.19.0 ; extra == 'kuberay'
Requires-Dist: pyyaml>=4.0.0 ; extra == 'kuberay'
Requires-Dist: ray[client] ; extra == 'pipes'
Requires-Dist: ray[client] ; extra == 'run-launcher'
Requires-Python: >=3.10, <3.13
Project-URL: GitHub, https://github.com/danielgafni/dagster-ray
Project-URL: Documentation, https://danielgafni.github.io/dagster-ray/
Project-URL: Changelog, https://github.com/danielgafni/dagster-ray/blob/main/CHANGELOG.md
Provides-Extra: executor
Provides-Extra: kuberay
Provides-Extra: pipes
Provides-Extra: run-launcher
Description-Content-Type: text/markdown

# `dagster-ray`

[![PyPI version](https://img.shields.io/pypi/v/dagster-ray.svg)](https://pypi.python.org/pypi/dagster-ray)
[![License](https://img.shields.io/pypi/l/dagster-ray.svg)](https://pypi.python.org/pypi/dagster-ray)
[![Python versions](https://img.shields.io/pypi/pyversions/dagster-ray.svg)](https://pypi.python.org/pypi/dagster-ray)
[![CI](https://github.com/danielgafni/dagster-ray/actions/workflows/CI.yml/badge.svg)](https://github.com/danielgafni/dagster-ray/actions/workflows/CI.yml)
[![prek](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/j178/prek/master/docs/assets/badge-v0.json)](https://github.com/j178/prek)
[![basedpyright - checked](https://img.shields.io/badge/basedpyright-checked-42b983)](https://docs.basedpyright.com)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

---

**Ray integration for Dagster.**

`dagster-ray` enables working with distributed Ray compute from Dagster pipelines, combining Dagster's excellent orchestration capabilities and Ray's distributed computing power together.

> [!NOTE]
> This project is ready for production use, but some APIs may change between minor releases.

Learn more in the [docs](https://danielgafni.github.io/dagster-ray)

## 🚀 Key Features

- **🎯 Run Launchers & Executors**: Submit Dagster runs or individual steps by submitting Ray jobs
- **🔧 Ray Resources**: Automatically create and destroy ephemeral Ray clusters and connect to them in client mode
- **📡 Dagster Pipes Integration**: Submit external scripts as Ray jobs, stream back logs and rich Dagster metadata
- **☸️ KubeRay Support**: Utilize `RayJob` and `RayCluster` custom resources in client or job submission mode ([tutorial](https://danielgafni.github.io/dagster-ray/tutorial/kuberay))
- **🏭 Production Ready**: Tested against a matrix of core dependencies, integrated with Dagster+
- **⚡ Instant Startup**: Leverage `RayCluster` with [Cluster Sharing](https://danielgafni.github.io/dagster-ray/tutorial/kuberay/#cluster-sharing) for lightning-fast development cycles with zero cold start times (intended for development environments)

## 📦 Installation

```shell
pip install dagster-ray
```

## 📚 Docs

**📖 [Full Documentation](https://danielgafni.github.io/dagster-ray)**

- **[Tutorial](https://danielgafni.github.io/dagster-ray/tutorial/)**: Step-by-step guide with examples
- **[API Reference](https://danielgafni.github.io/dagster-ray/api/)**: Complete API reference
- **[Changelog](https://danielgafni.github.io/dagster-ray/changelog)**: History of changes and links to docs for older `dagster-ray` releases.

## 👥 Who is using `dagster-ray`?

These happy users are running `dagster-ray` in production:

- [Anam](https://anam.ai)
- [kaiko](https://kaiko.ai/)
- [Sanas](https://www.sanas.ai/)

## 🤝 Contributing

Contributions are very welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for details.
