Metadata-Version: 2.4
Name: dpone-airflow-pack
Version: 0.73.14
Summary: Lightweight Airflow scheduler-side pack provider for dpone GitOps workloads
Keywords: airflow,gitops,data-engineering,scheduler,kubernetes
Author: PaulKov
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Dist: pyyaml>=6.0.1
Maintainer: PaulKov
Requires-Python: >=3.11, <3.13
Project-URL: Homepage, https://github.com/PaulKov/dpone
Project-URL: Repository, https://github.com/PaulKov/dpone
Project-URL: Issues, https://github.com/PaulKov/dpone/issues
Project-URL: Documentation, https://paulkov.github.io/dpone/
Description-Content-Type: text/markdown

# dpone-airflow-pack

`dpone-airflow-pack` is the lightweight, Airflow-independent reader and DAG
construction library for dpone GitOps packs. The formal
`apache-airflow-providers-dpone` distribution owns provider discovery and the
canonical `airflow.providers.dpone` namespace.

It only reads a static `airflow-pack.json` and builds visible Airflow/Kubernetes tasks. It does not import the full
`dpone` runtime and intentionally contains no source/sink/native transfer dependencies such as ClickHouse, MSSQL,
`pyodbc`, pandas, polars, or ConnectorX.

Recommended zero-boilerplate Airflow DAG import:

```python
from airflow.providers.dpone import load_dpone_dags

load_report = load_dpone_dags(
    globals(),
    index_path="/opt/airflow/dags/.dpone-cache/current/airflow-index.json",
)
if load_report.fatal:
    error_code = (
        load_report.errors[0].get("code", "DPONE_AIRFLOW_INDEX_INVALID")
        if load_report.errors
        else "DPONE_AIRFLOW_INDEX_INVALID"
    )
    raise RuntimeError(
        f"{error_code}: dpone Airflow deployment index could not be loaded"
    )
```

The formal provider's canonical namespace is PEP 561 typed.
`DponeDag.from_spec(...)` and
`DponeTaskGroup.from_pack(...)` remain typed escape hatches for hybrid DAGs.
The old `dpone_airflow_pack` imports remain compatibility shims and emit one
deprecation warning per process for provider-facade objects.

For the complete import, generated-loader, and cache-layout cutover, follow the
[provider and cache migration guide](../../docs/airflow-provider-cache-migration.md).

Legacy installations that still consume a mutable remote
`latest/pack-index.json` may keep the compatibility sync temporarily:

```bash
dpone-airflow-pack-sync \
  --once \
  --index-uri s3://bucket/dpone-artifacts/prod/repo/latest/pack-index.json \
  --reader-connection-id s3_dpone_artifacts_reader \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow

dpone-airflow-pack-cache-status \
  --cache-dir /opt/airflow/dags/.dpone-cache/airflow \
  --json
```

The loader reads only the local composite deployment index and its listed
checksum-verified DAG specs/packs. It does not scan authoring sources, refresh a
remote cache, or read Airflow Variables, Connections, Vault, Kubernetes, or a
metadata database during parse.

For DAGs that should run a whole GitOps domain or a named workflow group, keep the grouping in the domain catalog and
resolve it through the lightweight provider:

```yaml
workflow_groups:
  daily:
    workload_ids:
      - dim_customer
      - fact_order
```

```python
from dpone_airflow_pack import workload_ids_from_gitops_domain

workload_ids = workload_ids_from_gitops_domain(
    repo_root,
    "sales",
    group="daily",
)
```

The resolver only reads YAML files and validates that every group member exists in the same domain catalog.

The full `dpone[full,accel]` package belongs in the KPO runtime image, not in the scheduler image.
