Metadata-Version: 2.4
Name: prizm-dbt-cli
Version: 0.1.0
Summary: Prizm dbt artifact utility CLI (artifact-only, no dbt execution)
Author-email: Prizm Team <team@prizm.com>
Maintainer-email: Prizm Team <team@prizm.com>
License: MIT
Project-URL: Homepage, https://github.com/prizm/prizm-common
Project-URL: Repository, https://github.com/prizm/prizm-common.git
Project-URL: Documentation, https://prizm-common.readthedocs.io/
Project-URL: Bug Tracker, https://github.com/prizm/prizm-common/issues
Keywords: prizm,dbt,cli,data-lineage,observability
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: click>=8.2.0
Requires-Dist: requests>=2.31.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pre-commit==2.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest==8.2.2; extra == "test"
Requires-Dist: pytest-cov==2.0; extra == "test"
Requires-Dist: pytest-mock==3.0; extra == "test"

# prizm-dbt

Prizm dbt artifact utility CLI for collecting dbt artifacts, packaging them into a compressed artifact bundle, and pushing it to Prizm.

**Location:** `prizm-cli/prizm-dbt-cli` (under Server root)

**Key Design Principles:**
- ✅ Does **NOT** execute dbt
- ✅ Operates purely on filesystem artifacts and sends gzip-compressed artifact bundles
- ✅ Safe to use in CI/CD, Airflow, GitHub Actions, Jenkins, etc.
- ✅ Explicit flags, no magic, deterministic behavior

## Installation

### From Wheel File

```bash
pip install dist/prizm_dbt_cli-*.whl
```

The wheel is self-contained and has no dependency on prizm-dbt; only click, requests, and PyYAML are required.

### From Source (Development)

```bash
pip install -e .
```

For development, you can install the package in editable mode. See the [Development](#development) section for more details.

## Quick Start

### 1. Set Environment Variables

Set your Prizm API credentials (see [Configuration](#configuration) section for detailed setup instructions):

```bash
export PRIZM_API_TOKEN=prizm_xxx
export PRIZM_API_ENDPOINT=https://api.prizm.ai
```

### 2. Verify Configuration (Optional)

Check that your environment is configured correctly:

```bash
prizm-dbt doctor
```

### 3. Run dbt (using your existing orchestration)

```bash
dbt run --target prod
```

### 4. Push Artifacts to Prizm

```bash
prizm-dbt push-artifacts \
  --prizm-connection "My Prizm Source" \
  --project-dir . \
  --target-path target \
  --env prod \
  --dbt-target prod \
  --adapter snowflake
```

## Commands

### `prizm-dbt push-artifacts`

Collect dbt artifacts, build a gzip-compressed artifact bundle, and push it to Prizm.

**Required Flags:**
- `--prizm-connection`: Prizm connection/source name (must match an existing Source in Prizm)
- `--project-dir`: dbt project root (where dbt_project.yml exists)
- `--target-path`: Directory containing dbt artifacts
- `--env`: Logical environment (dev, staging, prod)
- `--dbt-target`: dbt target name used during execution
- `--adapter`: Warehouse adapter (snowflake, bigquery, databricks, etc.)

**Optional Flags:**
- `--execution-status`: success | failed | partial (default: success)
- `--invocation-id`: CI/CD run ID (GitHub Actions, Airflow DAG run, etc.)
- `--timeout`: API response timeout in seconds (default: 1800 / 30 minutes)
- `--retries`: Retry attempts for transient upload failures (default: 3)
- `--dry-run`: Validate artifacts without pushing

**Example:**

```bash
prizm-dbt push-artifacts \
  --prizm-connection "My Prizm Source" \
  --project-dir /workspace/dbt \
  --target-path /workspace/dbt/target \
  --env production \
  --dbt-target prod \
  --adapter snowflake \
  --execution-status success \
  --invocation-id "$GITHUB_RUN_ID"
```

**Artifacts Collected:**
- `manifest.json` (required when `--artifacts` is omitted)
- `run_results.json` (optional)
- `semantic_manifest.json` (optional)
- `catalog.json` (optional)

**Selective push (`--artifacts`):** omit for the normal full discovery flow. Pass a comma-separated list to package only those files, e.g. `--artifacts run_results,catalog` or `--artifacts manifest,run_results`. Names may be bare (`run_results`) or filenames (`run_results.json`).

```bash
prizm-dbt push-artifacts \
  --prizm-connection "My Warehouse" \
  --project-dir . \
  --target-path target \
  --env prod \
  --dbt-target prod \
  --adapter snowflake \
  --artifacts run_results,catalog
```

The CLI sends these artifacts as a gzip-compressed tar bundle, not as multipart file uploads. The bundle contains `metadata.json` plus the original dbt artifact JSON files under `artifacts/`, with per-file SHA-256 checksums and a whole-bundle checksum header. Dry runs show artifact names, paths, sizes, checksums, and compressed bundle size without printing artifact contents.

For large dbt projects, schedule-side parsing and persistence can take several minutes after the upload finishes. The default response timeout is 30 minutes. If the CLI still reports a response timeout, check schedule logs for the printed `upload_id` before retrying, or rerun with a larger `--timeout`.

### `prizm-dbt validate`

Preflight validation to ensure the environment is correctly configured.

**Required Flags:**
- `--project-dir`: dbt project root
- `--target-path`: Directory containing dbt artifacts

**Example:**

```bash
prizm-dbt validate \
  --project-dir . \
  --target-path target
```

**Validates:**
- ✔ Prizm auth token
- ✔ Endpoint connectivity
- ✔ Read permissions
- ✔ Artifact presence
- ✔ Project structure

### `prizm-dbt doctor`

Diagnostics and troubleshooting for enterprise support and debugging.

**No flags required** - reads config + environment

**Example:**

```bash
prizm-dbt doctor
```

**Outputs:**
- Resolved project_dir and target_path
- Detected dbt artifacts and their sizes
- Environment context (env, target, adapter)
- Connectivity check to Prizm
- Result of the most recent artifact push (if available)

## CI/CD Integration

### GitHub Actions

```yaml
- name: Run dbt
  run: dbt run --target prod

- name: Push dbt metadata to Prizm
  env:
    PRIZM_API_TOKEN: ${{ secrets.PRIZM_API_TOKEN }}
    PRIZM_API_ENDPOINT: ${{ secrets.PRIZM_API_ENDPOINT }}
  run: |
    prizm-dbt push-artifacts \
      --prizm-connection "My Prizm Source" \
      --project-dir . \
      --target-path target \
      --env prod \
      --dbt-target prod \
      --adapter snowflake \
      --invocation-id "${{ github.run_id }}"
```

### Airflow

```python
from airflow.operators.bash import BashOperator

push_prizm_metadata = BashOperator(
    task_id="push_prizm_metadata",
    bash_command="""
    prizm-dbt push-artifacts \
      --prizm-connection "My Prizm Source" \
      --project-dir /usr/local/airflow/dbt \
      --target-path /usr/local/airflow/dbt/target \
      --env prod \
      --dbt-target prod \
      --adapter snowflake
    """,
    env={
        "PRIZM_API_TOKEN": "{{ var.value.PRIZM_API_TOKEN }}",
        "PRIZM_API_ENDPOINT": "{{ var.value.PRIZM_API_ENDPOINT }}",
    },
)
```

### Jenkins

```groovy
stage('Push to Prizm') {
    steps {
        sh '''
            export PRIZM_API_TOKEN="${PRIZM_API_TOKEN}"
            export PRIZM_API_ENDPOINT="${PRIZM_API_ENDPOINT}"
            prizm-dbt push-artifacts \
              --prizm-connection "My Prizm Source" \
              --project-dir . \
              --target-path target \
              --env prod \
              --dbt-target prod \
              --adapter snowflake \
              --invocation-id "${BUILD_NUMBER}"
        '''
    }
}
```

## Development

### Development Setup

For local development and testing:

```bash
# Install the CLI package in editable mode
pip install -e .

# Install connector for testing (optional, only needed for running tests)
# The connector is built from prizm-dbt-mcp/dbt, not prizm-common
make install-connector

# Install test dependencies
pip install -e ".[test]"
```

**Connector source:** The dbt connector used by this CLI is built from `prizm-dbt-mcp/dbt`. To build the connector wheel separately: `cd prizm-dbt-mcp/dbt && python -m build --wheel -o ../dist/`

### Running Tests

```bash
# Run tests (requires connector to be installed)
make test

# Run tests with coverage
make test-cov
```

**Note:** The `install-connector` step installs the connector from `prizm-dbt-mcp/dbt` and is only needed for development/testing when running tests. For production builds, the connector is automatically bundled from `prizm-dbt-mcp/dbt` into the wheel during `make build`.

### Development Workflow

1. **Make changes** to CLI code or connector library
2. **Run tests** using `make test` (requires `install-connector` for now)
3. **Build wheel** using `make build` (automatically bundles connector)
4. **Test wheel** by installing it in a clean environment

## Building Wheels

The CLI package uses a bundling approach where the dbt connector from `prizm-dbt-mcp/dbt` is automatically included in the wheel during build. This creates a single, self-contained wheel file that includes everything needed. The connector is **not** sourced from prizm-common.

### Build Wheel Package

From the Server root:

```bash
cd prizm-cli/prizm-dbt-cli
make build
```

This command builds a wheel file containing the CLI and its dependencies (click, requests, PyYAML) and outputs it to the `dist/` directory. The CLI is standalone and does not require the prizm-dbt package.

### Clean Build Artifacts

```bash
make clean
```

This removes build and dist directories, egg-info, and Python cache files.

## Configuration

The `prizm-dbt` CLI requires authentication credentials to communicate with the Prizm API. These are configured via environment variables.

### Environment Variables

| Variable | Description | Required | Default |
|----------|-------------|----------|---------|
| `PRIZM_API_TOKEN` | Authentication token for Prizm API | Yes | - |
| `PRIZM_API_ENDPOINT` | Prizm API endpoint URL | No | `https://api.prizm.ai` |

### Setting Environment Variables

There are several ways to set these environment variables depending on your use case:

#### Method 1: Temporary (Current Shell Session)

**Linux/macOS (bash/zsh):**
```bash
export PRIZM_API_TOKEN=prizm_xxx
export PRIZM_API_ENDPOINT=https://api.prizm.ai
```

**Windows (PowerShell):**
```powershell
$env:PRIZM_API_TOKEN="prizm_xxx"
$env:PRIZM_API_ENDPOINT="https://api.prizm.ai"
```

**Windows (CMD):**
```cmd
set PRIZM_API_TOKEN=prizm_xxx
set PRIZM_API_ENDPOINT=https://api.prizm.ai
```

These settings only last for the current terminal session and are lost when you close the terminal.

#### Method 2: Permanent Setup

**Linux/macOS - Add to Shell Profile:**

For bash (`~/.bashrc` or `~/.bash_profile`):
```bash
echo 'export PRIZM_API_TOKEN=prizm_xxx' >> ~/.bashrc
echo 'export PRIZM_API_ENDPOINT=https://api.prizm.ai' >> ~/.bashrc
source ~/.bashrc
```

For zsh (`~/.zshrc`):
```bash
echo 'export PRIZM_API_TOKEN=prizm_xxx' >> ~/.zshrc
echo 'export PRIZM_API_ENDPOINT=https://api.prizm.ai' >> ~/.zshrc
source ~/.zshrc
```

For all shells (`~/.profile`):
```bash
echo 'export PRIZM_API_TOKEN=prizm_xxx' >> ~/.profile
echo 'export PRIZM_API_ENDPOINT=https://api.prizm.ai' >> ~/.profile
source ~/.profile
```

**Windows - System Environment Variables:**

1. Open Settings → System → About → Advanced system settings
2. Click "Environment Variables"
3. Under "User variables" or "System variables", click "New"
4. Add `PRIZM_API_TOKEN` with your token value
5. Add `PRIZM_API_ENDPOINT` with your endpoint URL (optional, defaults to `https://api.prizm.ai`)
6. Restart your terminal/command prompt

#### Method 3: Using .env Files (Development)

For local development, you can use a `.env` file in your project directory:

**Create `.env` file:**
```bash
# .env
PRIZM_API_TOKEN=prizm_xxx
PRIZM_API_ENDPOINT=https://api.prizm.ai
```

**Load before running commands:**

Linux/macOS:
```bash
export $(cat .env | xargs)
prizm-dbt push-artifacts ...
```

Or use tools like `direnv` or `python-dotenv` to automatically load `.env` files.

**Important:** Never commit `.env` files to version control. Add `.env` to your `.gitignore`.

### Verifying Configuration

After setting environment variables, verify your configuration:

```bash
prizm-dbt doctor
```

This command will show:
- Whether `PRIZM_API_TOKEN` is set
- The configured `PRIZM_API_ENDPOINT`
- Connectivity status to Prizm

### Security Best Practices

- **Never commit tokens to version control**: Always use `.gitignore` for `.env` files and never hardcode tokens in scripts
- **Use secrets management in CI/CD**: Store tokens as encrypted secrets in your CI/CD platform (GitHub Secrets, GitLab Variables, etc.)
- **Rotate tokens regularly**: Update your API tokens periodically for better security
- **Use different tokens per environment**: Use separate tokens for dev, staging, and production environments
- **Limit token permissions**: Create tokens with only the minimum required permissions
- **Monitor token usage**: Regularly review token access logs in your Prizm dashboard

## Supported Adapters

- snowflake
- bigquery
- databricks
- postgres
- redshift
- spark
- athena
- trino
- duckdb

## Error Handling

The CLI provides clear, actionable error messages:

- **Missing artifacts**: Lists checked paths and next steps
- **Authentication failures**: Clear token validation errors
- **Network errors**: Connection and timeout handling
- **Permission errors**: File access validation

## Security

- Token-based authentication via environment variables
- Tokens never logged or printed
- TLS enforced for all API calls
- No secrets in CLI arguments

## License

MIT License

## Support

For issues and questions, please visit:
https://github.com/DQLabs-Inc/prizm-dbt-mcp/issues
