Metadata-Version: 2.5
Name: airflow-provider-pysparkonk8s
Version: 1.0.0
Summary: Airflow provider that runs @task.pyspark tasks as client-mode Spark drivers on Kubernetes
Project-URL: Homepage, https://sdaberdaku.github.io/pysparkonk8s/
Project-URL: Source, https://github.com/sdaberdaku/pysparkonk8s
Project-URL: Changelog, https://github.com/sdaberdaku/pysparkonk8s/blob/main/CHANGELOG.md
Author: Sebastian Daberdaku
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Framework :: Apache Airflow
Classifier: Framework :: Apache Airflow :: Provider
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <3.15,>=3.10
Requires-Dist: apache-airflow-providers-apache-spark>=6.3
Requires-Dist: apache-airflow-providers-cncf-kubernetes>=10
Requires-Dist: apache-airflow>=2.11
Requires-Dist: pyspark>=4.0
Requires-Dist: pyyaml>=6
Description-Content-Type: text/markdown

# PySparkOnK8s

[![PyPI](https://img.shields.io/pypi/v/airflow-provider-pysparkonk8s)](https://pypi.org/project/airflow-provider-pysparkonk8s/)
[![CI](https://github.com/sdaberdaku/pysparkonk8s/actions/workflows/ci.yml/badge.svg)](https://github.com/sdaberdaku/pysparkonk8s/actions/workflows/ci.yml)

An Airflow provider that runs `@task.pyspark` tasks as client-mode Spark drivers on Kubernetes. The Airflow worker pod that executes the task becomes the Spark driver, and Spark creates the executors as pods. When the task ends, the executors are deleted, so every task gets its own short-lived Spark cluster.

It supports Airflow 2.11 and 3.3 with the KubernetesExecutor, Spark 4 and Python 3.10 to 3.14.

## Quick start

Give the worker service account the permissions of a Spark driver:

```bash
helm install pysparkonk8s-rbac oci://ghcr.io/sdaberdaku/charts/pysparkonk8s-rbac \
  --version 1.0.0 -n airflow --set serviceAccountName=<worker service account>
```

Use `ghcr.io/sdaberdaku/pysparkonk8s/driver:1.0.0-airflow3.3.2-python3.12` as the Airflow worker image (it contains Java, PySpark and this package), then write a task:

```python
from airflow.sdk import task  # airflow.decorators on Airflow 2
from pysparkonk8s import SparkDriverConf, SparkExecutorConf, pyspark_on_k8s

driver_conf = SparkDriverConf(image="ghcr.io/sdaberdaku/pysparkonk8s/driver:1.0.0-airflow3.3.2-python3.12")
executor_conf = SparkExecutorConf(image="ghcr.io/sdaberdaku/spark:4.2.0-python3.12")


@task.pyspark
def count(spark=None):
    return spark.range(1000).count()


count.override(**pyspark_on_k8s(driver_conf=driver_conf, executor_conf=executor_conf))()
```

To install the package in your own image, run `pip install airflow-provider-pysparkonk8s`. The image also needs a JRE, `pyspark==4.2.0` and `pyspark-client==4.2.0` (install both at the same version).

## Documentation

The full documentation is at https://sdaberdaku.github.io/pysparkonk8s/: configuration reference, examples, how it works, operations and limitations.

## License

Apache-2.0.
