Metadata-Version: 2.4
Name: dpone-airflow-pack
Version: 0.68.0
Summary: Lightweight Airflow scheduler-side pack provider for dpone GitOps workloads
Keywords: airflow,gitops,data-engineering,scheduler,kubernetes
Author: PaulKov
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Dist: pyyaml>=6.0.1
Maintainer: PaulKov
Requires-Python: >=3.11, <3.13
Project-URL: Homepage, https://github.com/PaulKov/dpone
Project-URL: Repository, https://github.com/PaulKov/dpone
Project-URL: Issues, https://github.com/PaulKov/dpone/issues
Project-URL: Documentation, https://paulkov.github.io/dpone/
Description-Content-Type: text/markdown

# dpone-airflow-pack

`dpone-airflow-pack` is the lightweight Airflow scheduler/webserver provider for dpone GitOps packs.

It only reads a static `airflow-pack.json` and builds visible Airflow/Kubernetes tasks. It does not import the full
`dpone` runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL,
`pyodbc`, pandas, polars, or ConnectorX.

Recommended Airflow DAG import:

```python
from dpone_airflow_pack import build_dpone_gitops_task_group_from_pack
```

For remote immutable packs, keep DAG parsing cache-only:

```bash
dpone-airflow-pack-sync \
  --once \
  --index-uri s3://bucket/dpone-artifacts/prod/repo/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --json
```

Use `load_dpone_airflow_pack_with_provenance("cached://<workload_id>")` when a DAG wrapper needs to expose which
cache generation and pack checksum were used during parse. A KubernetesExecutor runtime worker is not required to mount
the scheduler or DAG processor cache.

For DAGs that should run a whole GitOps domain or a named workflow group, keep the grouping in the domain catalog and
resolve it through the lightweight provider:

```yaml
workflow_groups:
  daily:
    workload_ids:
      - dim_customer
      - fact_order
```

```python
from dpone_airflow_pack import workload_ids_from_gitops_domain

workload_ids = workload_ids_from_gitops_domain(
    repo_root,
    "sales",
    group="daily",
)
```

The resolver only reads YAML files and validates that every group member exists in the same domain catalog.

The full `dpone[full,accel]` package belongs in the KPO runtime image, not in the scheduler image.
