Metadata-Version: 2.4
Name: dpone-airflow-pack
Version: 0.66.0
Summary: Lightweight Airflow scheduler-side pack provider for dpone GitOps workloads
Keywords: airflow,gitops,data-engineering,scheduler,kubernetes
Author: PaulKov
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Maintainer: PaulKov
Requires-Python: >=3.11, <3.13
Project-URL: Homepage, https://github.com/PaulKov/dpone
Project-URL: Repository, https://github.com/PaulKov/dpone
Project-URL: Issues, https://github.com/PaulKov/dpone/issues
Project-URL: Documentation, https://paulkov.github.io/dpone/
Description-Content-Type: text/markdown

# dpone-airflow-pack

`dpone-airflow-pack` is the lightweight Airflow scheduler/webserver provider for dpone GitOps packs.

It only reads a static `airflow-pack.json` and builds visible Airflow/Kubernetes tasks. It does not import the full
`dpone` runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL,
`pyodbc`, pandas, polars, or ConnectorX.

Recommended Airflow DAG import:

```python
from dpone_airflow_pack import build_dpone_gitops_task_group_from_pack
```

The full `dpone[full,accel]` package belongs in the KPO runtime image, not in the scheduler image.

## Remote pack cache

Remote `airflow-pack.json` artifacts can be published to object storage and
mirrored into a bounded local scheduler cache.

Bootstrap one generation:

```bash
dpone-airflow-pack-sync \
  --once \
  --index-uri s3://do-dwh/dpone-artifacts/prod/airflow-dags-dev/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow
```

Refresh without restarting Airflow pods:

```bash
dpone-airflow-pack-sync \
  --watch \
  --interval-seconds 60 \
  --jitter-seconds 15 \
  --index-uri s3://do-dwh/dpone-artifacts/prod/airflow-dags-dev/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow
```

The sync loop is fail-open. On S3, connection, index, or hash errors it writes a
warning status and preserves the previous `current` generation. DAG parse stays
local and cache-only.

Inspect the current cache status:

```bash
dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow
```

## Airflow execution policy

`airflow-pack.json` can carry a scheduler-side execution policy:

```json
{
  "airflow": {
    "execution": {
      "task_executor": "KubernetesExecutor",
      "deferrable": true,
      "on_finish_action": "delete_succeeded_pod"
    }
  }
}
```

The lightweight provider maps it to KubernetesPodOperator kwargs
(`executor`, `deferrable`, `on_finish_action`) and treats those fields as
pack-owned. DAG-local overrides may still set DAG-owned values such as
`retries`, `pool`, `get_logs`, or `in_cluster`.
