Metadata-Version: 2.4
Name: prizm-airflow-cli
Version: 0.1.1
Summary: Prizm Airflow metadata extraction CLI (DAG/run/task metadata only, no Airflow execution)
Author-email: Prizm Team <team@prizm.com>
Maintainer-email: Prizm Team <team@prizm.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/prizm/prizm-common
Project-URL: Repository, https://github.com/prizm/prizm-common.git
Project-URL: Documentation, https://prizm-common.readthedocs.io/
Project-URL: Bug Tracker, https://github.com/prizm/prizm-common/issues
Keywords: prizm,airflow,cli,data-lineage,observability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31.0
Provides-Extra: dev
Requires-Dist: pre-commit==2.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest==8.2.2; extra == "test"
Requires-Dist: pytest-cov==2.0; extra == "test"
Requires-Dist: pytest-mock==3.0; extra == "test"

# prizm-airflow

Prizm Airflow metadata extraction CLI for reading Airflow REST API metadata, packaging it into a JSON bundle, and pushing it to Prizm.

**Location:** `prizm-cli/prizm-airflow-cli` (under Server root)

**Key Design Principles:**
- ✅ Does **NOT** execute Airflow DAGs or tasks
- ✅ Does **NOT** process metadata locally — reads Airflow's REST API and forwards the bundle as-is
- ✅ Safe to use in CI/CD, cron, Airflow itself, GitHub Actions, Jenkins, etc.
- ✅ Explicit flags, no magic, deterministic behavior

Prizm stages the uploaded bundle and runs TECHNICAL followed by OPERATIONAL processing server-side.

## Installation

### From PyPI

```bash
pip install prizm-airflow-cli
```

### From Wheel File

```bash
pip install dist/prizm_airflow_cli-*.whl
```

The wheel is self-contained; only `requests` is required at runtime.

### From Source (Development)

```bash
pip install -e .
```

For development, you can install the package in editable mode. See the [Development](#development) section for more details.

## Quick Start

### 1. Set Environment Variables (Optional)

You can pass credentials as flags, or export them and reference via your shell:

```bash
export PRIZM_API_TOKEN=prizm_xxx
```

### 2. Extract and Push Airflow Metadata to Prizm

```bash
prizm-airflow extract \
  --prizm-url https://<prizm-host> \
  --api-token "$PRIZM_API_TOKEN" \
  --connection-name "airflow_cli" \
  --hostname http://localhost:8080 \
  --username admin \
  --password admin \
  --runs 10 \
  --extract-source-code true
```

## Commands

### `prizm-airflow extract`

Read DAGs, runs, tasks, and task instances from the Airflow REST API, build a JSON bundle, and upload it to Prizm.

**Required Flags:**
- `--prizm-url`: Prizm base URL
- `--connection-name`: Prizm connection/source name (must match an existing Source in Prizm)

**Authentication Flags (one of):**
- `--api-token`: Prizm API access token
- `--client-secret`: Alias for `--api-token`

**Airflow Connection Flags:**
- `--hostname`: Airflow base URL (default: `http://localhost:8080`)
- `--username` / `--password`: Airflow basic-auth credentials
- `--basic-auth`: Base64-encoded `username:password`, alternative to `--username`/`--password`

**Optional Flags:**
- `--endpoint-url`: Full ingest URL; overrides `--prizm-url` + default path
- `--runs`: Number of recent DAG runs to fetch per DAG (default: 30)
- `--extract-source-code`: Fetch DAG file source (default: true)
- `--include-full-log`: Fetch and include the full task log for failed task instances (default: false)
- `--dry-run`: Build the bundle and write it to `--out` locally without uploading
- `--out`: Output path for `--dry-run` (default: `airflow_bundle.json`)

**Example:**

```bash
prizm-airflow extract \
  --prizm-url https://api.prizm.ai \
  --api-token "$PRIZM_API_TOKEN" \
  --connection-name "My Airflow Source" \
  --hostname https://airflow.internal.example.com \
  --basic-auth "$AIRFLOW_BASIC_AUTH_B64" \
  --runs 50 \
  --extract-source-code true \
  --include-full-log false
```

**Bundle Contents:**
- DAGs (id, metadata)
- DAG runs (up to `--runs` most recent per DAG)
- Tasks per DAG
- Task instances per DAG run
- DAG source code (when `--extract-source-code true`)
- Full failed-task logs (when `--include-full-log true`)

**Dry run (`--dry-run`):** builds the bundle and writes it to `--out` instead of uploading — useful for inspecting bundle contents or debugging Airflow connectivity before sending anything to Prizm.

```bash
prizm-airflow extract \
  --prizm-url https://api.prizm.ai \
  --connection-name "My Airflow Source" \
  --hostname http://localhost:8080 \
  --username admin \
  --password admin \
  --dry-run \
  --out /tmp/airflow_bundle.json
```

The CLI authenticates to Prizm's ingest endpoint with `Authorization: Bearer <token>` and uploads the bundle as a multipart file (`airflow_bundle.json`) alongside the `connection_name` field.

## CI/CD Integration

### GitHub Actions

```yaml
- name: Push Airflow metadata to Prizm
  env:
    PRIZM_API_TOKEN: ${{ secrets.PRIZM_API_TOKEN }}
  run: |
    prizm-airflow extract \
      --prizm-url https://api.prizm.ai \
      --api-token "$PRIZM_API_TOKEN" \
      --connection-name "My Airflow Source" \
      --hostname https://airflow.internal.example.com \
      --basic-auth "${{ secrets.AIRFLOW_BASIC_AUTH_B64 }}" \
      --runs 30
```

### Airflow (self-monitoring)

```python
from airflow.operators.bash import BashOperator

push_prizm_metadata = BashOperator(
    task_id="push_prizm_airflow_metadata",
    bash_command="""
    prizm-airflow extract \
      --prizm-url https://api.prizm.ai \
      --connection-name "My Airflow Source" \
      --hostname http://localhost:8080 \
      --username "{{ var.value.AIRFLOW_ADMIN_USER }}" \
      --password "{{ var.value.AIRFLOW_ADMIN_PASSWORD }}" \
      --runs 30
    """,
    env={
        "PRIZM_API_TOKEN": "{{ var.value.PRIZM_API_TOKEN }}",
    },
)
```

### Jenkins

```groovy
stage('Push Airflow metadata to Prizm') {
    steps {
        sh '''
            prizm-airflow extract \
              --prizm-url https://api.prizm.ai \
              --api-token "${PRIZM_API_TOKEN}" \
              --connection-name "My Airflow Source" \
              --hostname https://airflow.internal.example.com \
              --basic-auth "${AIRFLOW_BASIC_AUTH_B64}" \
              --runs 30
        '''
    }
}
```

## Development

### Development Setup

```bash
pip install -e ".[dev,test]"
```

### Running Tests

```bash
make test
```

## Build Wheel

```bash
make build
```

For offline/local environments where build dependencies are already installed:

```bash
make build-local
```

This outputs a wheel and sdist to `dist/`.

### Clean Build Artifacts

```bash
make clean
```

This removes `build/` and `dist/` directories, `egg-info`, and Python cache files.

## Publish

Build both source and wheel distributions:

```bash
python -m build --sdist --wheel
```

Upload to TestPyPI first:

```bash
python -m twine upload --repository testpypi dist/*
```

Upload to PyPI:

```bash
python -m twine upload dist/*
```

## Security

- Token-based authentication via `--api-token`/`--client-secret` or environment variable
- Tokens never logged or printed
- TLS enforced for all API calls
- No secrets written to the dry-run output bundle

## License

MIT License

## Support

For issues and questions, please visit:
https://github.com/DQLabs-Inc/prizm-cli/issues
