Metadata-Version: 2.5
Name: dagster-uc
Version: 0.6.7
Summary: CLI for managing user code deployments of a Dagster instance that was deployed on kubernetes.
Project-URL: Homepage, https://github.com/ASML-Labs/dagster-uc
Project-URL: Issues, https://github.com/ASML-Labs/dagster-uc/issues
Author: Stefan Verbruggen, Ion Koutsouris
License-File: LICENSE
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.9
Requires-Dist: kr8s<1.0,>=0.20.1
Requires-Dist: packaging>=25.0
Requires-Dist: pydantic-settings>=2
Requires-Dist: pydantic>=2
Requires-Dist: pytz
Requires-Dist: pyyaml
Requires-Dist: tomli
Requires-Dist: typer>=0.15.1
Description-Content-Type: text/markdown

# Introduction

This experimental CLI allows you to manage user code deployments for a Dagster instance deployed on Kubernetes. It packages your code branch into a Docker container, uploads it to your container registry, and updates your existing Dagster instance to enable your user code deployment.

# Pre-requisites

* kubectl with a valid config
* Helm 3
* Podman
* Python 3.10+
* Azure CLI (if you are using Azure container registry and `use_az_login`)

# Installation

* Install from PyPI:
  ```
  pip install dagster-uc
  ```

* Create a configuration file named `.config_user_code_deployments.yaml` in the root of your repository or in your home directory.
  You can also create one by running:
  ```
  dagster-uc init-config -f '.config_user_code_deployments.yaml'
  ```

# Configuration (nested structure)

The configuration format is now nested and grouped by concerns (e.g. `docker`, `kubernetes`, `helm`). The top-level keys you will commonly use are:

* `defaults` — values applied to every environment unless overridden.
* Per-environment sections (e.g. `dev`, `acc`, `prd`) — environment-specific overrides.
* `docker` — Docker/build related settings (dockerfile, registry, image prefix, build env vars, etc).
* `kubernetes` — Kubernetes-specific settings (context, namespace, resource requests/limits, env and secret lists).
* `helm` — Helm-related settings (for example, skip schema validation).
* `chart` — chart-specific values you may want to supply into the user-deployments Helm chart.
* `use_latest_chart_version`, `use_project_name`, and `project_name_override` — deployment behavior flags.

Order of loading configuration:
1. `defaults`
2. environment-specific keys (e.g. `dev`)
3. environment variable overrides

Below is an example config that mirrors the new nested structure:

```yaml
defaults:
  repository_root: "."
  code_path: example/repo.py
  dagster_version: 1.11.16
  cicd: false
  use_project_name: True
  use_latest_chart_version: True
  # Docker configuration (grouped under `docker`)
  docker:
    docker_root: "."
    docker_env_vars:
      - FOO
      - FOO='string'
    image_prefix: "example"
    use_az_login: True
    container_registry_chart_path: "helm/dagster/dagster-user-deployments"

  # Helm configuration (grouped under `helm`)
  helm:
    skip_schema_validation: True

  # Kubernetes configuration (grouped under `kubernetes`)
  kubernetes:
    namespace: dagster
    node: cpunode
    requests:
      cpu: 150m
      memory: 750Mi
    limits:
      cpu: 4000m
      memory: 2000Mi
    user_code_deployment_env_secrets:
      - name: dagster-storage-secret
    pull_policy: Always

  chart: {}

dev:
  environment: dev
  dagster_gui_url: "http://dagster.dev"

  docker:
    dockerfile: "docker/dev.Dockerfile"
    container_registry: dagster-uc.dev.acr.io

  kubernetes:
    context: "aks-dev"
    user_code_deployment_env:
      - name: ON_K8S
        value: '1'
      - name: ENVIRONMENT
        value: dev
    pull_policy: IfNotPresent
acc:
  environment: acc
  dagster_gui_url: "http://dagster.acc"
  project_name_override: 'example-acc'

  docker:
    dockerfile: "docker/acc.Dockerfile"
    container_registry: dagster-uc.acc.acr.io

  kubernetes:
    context: "aks-acc"
    user_code_deployment_env:
      - name: ON_K8S
        value: '1'
      - name: ENVIRONMENT
        value: acc
```

Notes on common config keys (now nested):
* `code_path` — path to the Python module that starts your Dagster definitions (used to start the user-code gRPC server).
* `docker.dockerfile` — path to the Dockerfile to build the image.
* `docker.container_registry` — target container registry for the built image (per-environment override).
* `docker.image_prefix` — prefix used when naming images.
* `docker.docker_env_vars` — list of environment variables to pass into the build process.
* `docker.use_az_login` — set to `True` if you need to log in to Azure before pushing images.
* `kubernetes.context` — the kube context to use for deployments in that environment.
* `kubernetes.namespace` — the namespace where Dagster and user deployments live.
* `kubernetes.requests` / `kubernetes.limits` — resource requests and limits for the user code deployment pod.
* `kubernetes.user_code_deployment_env` — a list of name/value env objects to inject into the user-code deployment container.
* `kubernetes.user_code_deployment_env_secrets` — a list of secrets to be mounted/injected as environment variables.
* `kubernetes.pull_policy` - Standard Kubernetes Pull Policy for images, can either be 'IfNotPresent' or 'Always'
* `helm.skip_schema_validation` — useful for older Helm chart setups or when schema validation causes issues.
* `use_project_name` — when True, the project name from `pyproject.toml` is prefixed to the deployment name.
* `project_name_override` - When set, the project name from `pyproject.toml` is overridden with this value

### Overriding Config with Environment Variables

Environment variables can still be used to override configuration at load time but you need to scaffold them in the yaml using `cicd: ${CICD}`. You can export common flags (for example `CICD=TRUE` or `VERBOSE=TRUE`) to affect behavior — top-level boolean or string fields are typically overridden this way. If you need to override nested values in automation, set the environment variables your automation expects before running the CLI.

# Usage

* Deploy the currently checked out Git branch:
  ```
  dagster-uc deployment deploy
  ```

* See all available commands:
  ```
  dagster-uc --help
  ```
# Command Line Interface

## Global Options

```bash
dagster-uc [GLOBAL_OPTIONS] <command> [SUBCOMMAND] [OPTIONS]
```

Global options:

* -e, --environment TEXT — Target environment (default: dev). Available options are derived from your YAML config file
* -c, --config-file PATH — Path to your YAML config file. (default: '.')
* -v, --verbose — Enable DEBUG logs.

## Show Config

```bash
dagster-uc [GLOBAL_OPTIONS] show-config
```

Pretty print the effective configuration for the selected environment.

## Deployment List

```bash
dagster-uc [GLOBAL_OPTIONS] deployment list 
```

List active user code deployments registered in the ConfigMap.
## Deploy Deployment

```bash
dagster-uc [GLOBAL_OPTIONS] deployment deploy [OPTIONS]
```

Using the configuration file provided, build a dagster-user-deployment image using podman/buildah, push it to a container registry, then adds/creates (if not present) a code location deployment for Dagster to read.

Options:

* -f, --force — Always do a full redeploy (reapply manifests; useful  for clean updates).
* -b, --skip-build — Skip building/pushing the container image.
* -s, --deployment-name-suffix TEXT — Append a suffix to the default name (branch-based).
* -n, --deployment-name TEXT — Override the name entirely (ignores suffix).
* -r, --reset-lock — Reset the deployment semaphore if a previous deploy is stuck.
* -u, --use-sudo — Run the build tool with sudo.
* --ignore-check — Skip the podman presence check (helpful in opinionated CI).
* --extra-env TEXT (repeatable) — Inject extra environment variables into the user code pod (must be in format '{key}={value}').

## Deployment Check

```bash
dagster-uc [GLOBAL_OPTIONS] deployment check [-n name] [-t timeout]
```
Options:

* -n, --name TEXT — Deployment name; if omitted, uses the default for the current branch.
* -t, --timeout INTEGER — Seconds to follow logs (default: 60).

## Deployment Revive

```bash
dagster-uc [GLOBAL_OPTIONS] deployment revive -t tag -n name

```

* -n, --name TEXT — Name of the deployment to revive (UI : will be normalized to --).
* -t, --tag TEXT — Existing image tag to use.

## Deployment Delete

```bash
dagster-uc [GLOBAL_OPTIONS] deployment delete [OPTIONS]
```

Options:

* -a, --all — Delete all deployment.
* -n, --name TEXT — Delete a single deployment by name (UI : normalized to --).
* -b, --branch TEXT — Use a branch name to compute the deployment name (respects project-name rules). Defaults to current git branch.

# Branch naming and deployments

* When `cicd: true` is set, the deployment name is derived from the `environment` value.
* When `cicd: false`, the deployment name is derived from the Git branch name. The branch name is normalized by replacing non-alphanumeric characters with hyphens and stripping leading/trailing hyphens.
  Example: `feat: my amazing feature` -> `feat-my-amazing-feature`

* You can deploy the same branch multiple times by supplying `--deployment-name-suffix`, which appends a suffix to the deployment name.

* When `use_project_name` is enabled, the internal deployment name will be prefixed by a project slug derived from your `pyproject.toml`. Internally the prefix separator is `--` so an example name may be `my-project--feat-a`, which appears in the Dagster UI as `project:branch`.

# Building and container behavior

* The build process passes a `BRANCH_NAME` build-arg so your code can behave differently per branch (e.g. selecting secrets, configuration).
* Images are versioned: the CLI will check the registry for existing tags and increment a version to avoid reusing tags that could break running jobs.
* Use a registry lifecycle/garbage-collection policy to keep old images from accumulating.

Example Dockerfile pattern:

```
FROM python:3.11-slim
ARG BRANCH_NAME
ARG DIR="APP"
WORKDIR $DIR
COPY my_project my_project
COPY pyproject.toml uv.lock README.md ./
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --no-dev --link-mode=copy
ENV PATH="/$DIR/.venv/bin:$PATH"
ENV BRANCH_NAME=${BRANCH_NAME}
```

# Kubernetes / Helm

* Make sure `kubernetes.context` can access the `kubernetes.namespace`.
* Configure `kubernetes.requests` and `kubernetes.limits` for the user-code deployment pod appropriately.
* Pass environment variables via `kubernetes.user_code_deployment_env` and secrets via `kubernetes.user_code_deployment_env_secrets`.
* Helm values and chart overrides can be supplied under `chart` in the config file — these are passed to the user-deployments Helm chart.

# Tips

* Keep a `defaults` section in your config file to reduce duplication between environments.
* Use environment-specific overrides for registry, kube context, and secrets.
* If you use Azure, set `docker.use_az_login: True` and ensure your environment has access to `az` and the appropriate credentials.

# Troubleshooting

* If Helm chart upgrades fail due to schema validation, try setting `helm.skip_schema_validation: True` in your `defaults` or environment override.
* Check that `kubernetes.context` points to the correct cluster and that your kube credentials have permission to modify the target namespace.
* Ensure the `code_path` points to an importable Python module that starts your Dagster definitions.

# Example test config

A small, real example is checked into the repo under `tests/config/.config_user_code_deployments.yaml` and demonstrates the new nested structure used by the CLI tests.
