Metadata-Version: 2.4
Name: prefect-nomad
Version: 0.2.0
Summary: Prefect integration for running workflows on HashiCorp Nomad
Keywords: prefect,nomad,workflow,orchestration,hashicorp
Author: nas
Author-email: nas <nasdenkov@gmail.com>
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Distributed Computing
Classifier: Operating System :: OS Independent
Requires-Dist: prefect>=3.6.21
Requires-Dist: pydantic>=2.12.5
Requires-Dist: python-nomad>=2.1.0
Requires-Dist: pytest>=7.0 ; extra == 'dev'
Requires-Dist: pytest-asyncio ; extra == 'dev'
Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
Requires-Dist: ruff ; extra == 'dev'
Requires-Python: >=3.13
Project-URL: Homepage, https://github.com/ubiquitousbyte/prefect-nomad
Project-URL: Repository, https://github.com/ubiquitousbyte/prefect-nomad
Project-URL: Issues, https://github.com/ubiquitousbyte/prefect-nomad/issues
Provides-Extra: dev
Description-Content-Type: text/markdown

# prefect-nomad

[![Pre-commit](https://github.com/ubiquitousbyte/prefect-nomad/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/ubiquitousbyte/prefect-nomad/actions/workflows/pre-commit.yml)
[![Tests](https://github.com/ubiquitousbyte/prefect-nomad/actions/workflows/test.yml/badge.svg)](https://github.com/ubiquitousbyte/prefect-nomad/actions/workflows/test.yml)
[![codecov](https://codecov.io/gh/ubiquitousbyte/prefect-nomad/branch/main/graph/badge.svg)](https://codecov.io/gh/ubiquitousbyte/prefect-nomad)
[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

Prefect integration for running workflows on HashiCorp Nomad.

## Overview

`prefect-nomad` is a Prefect collection that enables you to run Prefect flows as batch jobs on HashiCorp Nomad clusters. It provides a `NomadWorker` that submits flow runs as Nomad jobs using the Docker task driver.

## Features

- **Native Nomad Integration**: Submit Prefect flow runs as Nomad batch jobs
- **Docker Support**: Run flows in Docker containers with configurable images
- **Resource Management**: Configure CPU, memory, and other resources for job execution
- **TLS & Authentication**: Secure connections with ACL tokens and mutual TLS
- **Job Monitoring**: Real-time log streaming and status tracking
- **Flexible Configuration**: Customize Nomad job specifications, task configurations, and scheduling

## Installation

```bash
pip install prefect-nomad
```

## Quick Start

### 1. Set up Nomad Credentials

Create a Prefect block to store your Nomad cluster credentials:

```python
from prefect_nomad.credentials import NomadCredentials

# Create and save credentials
credentials = NomadCredentials(
    address="http://nomad.example.com:4646",
    token="your-nomad-acl-token",  # Optional
    namespace="default",            # Optional
    region="global"                 # Optional
)
credentials.save("my-nomad-cluster")
```

### 2. Start a Nomad Worker

Start a worker that will poll for flow runs and submit them to Nomad:

```bash
prefect worker start --pool nomad-pool --type nomad
```

### 3. Create a Deployment

Deploy your flow to run on the Nomad worker pool:

```python
from prefect import flow

@flow
def my_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    my_flow.deploy(
        name="my-nomad-deployment",
        work_pool_name="nomad-pool",
        image="prefecthq/prefect:3-python3.13",
    )
```

### 4. Run Your Flow

```bash
prefect deployment run my-flow/my-nomad-deployment
```

## Docker Usage

Pre-built Docker images with the `prefect-nomad` integration are available on GitHub Container Registry (GHCR). These images are built on top of the official Prefect base image and include the integration pre-installed.

### Using Pre-built Images (Recommended)

**Pull the latest image**:

```bash
docker pull ghcr.io/ubiquitousbyte/prefect-nomad:latest
```

**Available image variants**:

```bash
# Latest version with Prefect 3 and Python 3.13 (recommended)
ghcr.io/ubiquitousbyte/prefect-nomad:latest

# Specific version with default Prefect
ghcr.io/ubiquitousbyte/prefect-nomad:0.1.0

# Specific version with explicit Python and Prefect versions
ghcr.io/ubiquitousbyte/prefect-nomad:0.1.0-py3.13-prefect3
ghcr.io/ubiquitousbyte/prefect-nomad:0.1.0-py3.13-prefect3.6.20

# Just Python version specified (uses default Prefect)
ghcr.io/ubiquitousbyte/prefect-nomad:0.1.0-py3.13
```

**Run a Nomad worker**:

```bash
docker run -it --rm \
  -e PREFECT_API_URL=https://your-prefect-server.com/api \
  ghcr.io/ubiquitousbyte/prefect-nomad:latest \
  prefect worker start --pool nomad-pool --type nomad
```

**Run Prefect server with the integration**:

```bash
docker run -d -p 4200:4200 \
  ghcr.io/ubiquitousbyte/prefect-nomad:latest \
  prefect server start
```

**Use in deployments**:

```python
my_flow.deploy(
    name="my-nomad-deployment",
    work_pool_name="nomad-pool",
    image="ghcr.io/ubiquitousbyte/prefect-nomad:0.1.0-py3.13-prefect3",
)
```

### Building from Source

If you need to build a custom image or test local changes:

Build the image with the default Prefect version (3-python3.13):

```bash
docker build -t prefect-nomad:latest .
```

Or specify a custom Prefect version:

```bash
docker build --build-arg PREFECT_VERSION=3.1-python3.13 -t prefect-nomad:3.1 .
```

### Image Details

- **Base**: Official Prefect image (`prefecthq/prefect`)
- **Integration**: `prefect-nomad` installed from source
- **Verification**: Runs `verify_integration.py` during build to ensure proper installation
- **Entrypoint**: Inherits from base Prefect image (supports all Prefect CLI commands)
- **Registry**: [GitHub Container Registry](https://github.com/ubiquitousbyte/prefect-nomad/pkgs/container/prefect-nomad)

## Configuration

### Worker Configuration

The Nomad worker can be configured with various options through the work pool:

```python
from prefect.deployments import DeployConfig

my_flow.deploy(
    name="my-nomad-deployment",
    work_pool_name="nomad-pool",
    job_variables={
        "image": "my-registry/my-image:latest",
        "cpu": 500,                    # MHz
        "memory": 512,                 # MB
        "namespace": "production",
        "region": "us-east-1",
        "job_priority": 50,
        "job_timeout": 3600,          # seconds
        "env": {
            "MY_VAR": "value"
        }
    }
)
```

### TLS Configuration

For secure connections to Nomad:

```python
from prefect_nomad.credentials import NomadCredentials

credentials = NomadCredentials(
    address="https://nomad.example.com:4646",
    token="your-acl-token",
    tls_ca_cert="/path/to/ca.crt",
    tls_client_cert="/path/to/client.crt",
    tls_client_key="/path/to/client.key",
    tls_skip_verify=False
)
credentials.save("secure-nomad-cluster")
```

## Architecture

The Nomad worker uses a two-phase monitoring approach:

1. **Evaluation Phase**: Monitors job registration and scheduling decisions
   - Validates that Nomad can schedule the job
   - Detects resource constraints or placement failures
   - Raises errors for failed evaluations

2. **Allocation Phase**: Monitors job execution
   - Tracks task allocation and execution
   - Streams logs in real-time
   - Reports task exit codes

## Requirements

- Python 3.13+
- Prefect 3.6.21+
- HashiCorp Nomad cluster with Docker task driver enabled
- Docker images available to Nomad clients

## Resources

- [Prefect Documentation](https://docs.prefect.io/)
- [HashiCorp Nomad Documentation](https://www.nomadproject.io/docs)
- [python-nomad Client](https://github.com/jrxFive/python-nomad)

## Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

## License

This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.

## Disclaimer

This is an independent project and is not officially associated with or endorsed by HashiCorp or Prefect Technologies, Inc.
