Slurm Backend

CADDForge can submit workflows to Slurm through the core SlurmBackend. The backend writes a payload, submits an sbatch script, and uses CADDForge's Slurm runner on the compute node to execute the workflow.

Use Slurm for CADDForge-native workflows that can run in a normal Python environment on a compute node. Check each public tool's runtime requirements before selecting a backend.

Basic Usage

Replace the partition and account placeholders with values from your cluster administrator.

Set backend: slurm in the tool config or pass --backend slurm from the CLI:

caddforge run md plain-md --config plain_md.yaml --backend slurm --slurm-time 24:00:00

Common Slurm config fields:

backend: slurm
slurm_partition: your-partition
slurm_time: "24:00:00"
slurm_account: your-account
slurm_nodes: 1
slurm_ntasks: 1
slurm_cpus_per_task: 8
sbatch_options:
  gres: gpu:1

Environment-related shared fields include slurm_env_pack_path, slurm_env_cache_enabled, slurm_env_cache_dir, slurm_wheelhouse_path, slurm_requirements_path, and slurm_setup_commands.

Environment Requirement

Slurm jobs run on compute nodes, not in the interactive shell where caddforge was launched. The compute node must have access to a compatible Python environment containing:

• caddforge
• the plugin package being run
• all Python dependencies
• all scientific runtime dependencies needed by the workflow

For heavy scientific stacks, install conda-managed packages first, then install CADDForge packages with pip.

Option A: Packed Conda/Micromamba Environment

Use this when the workflow depends on conda-managed scientific packages such as OpenMM, RDKit, OpenFE, or OpenFF. Dependency resolution happens once before submission, and compute nodes reuse a checksum-addressed unpacked environment cache by default.

Create the environment, install CADDForge and the required plugins, then use caddforge pack-env. The core package already depends on conda-pack:

micromamba create -n caddforge -c conda-forge python=3.10
micromamba activate caddforge
micromamba install -c conda-forge openmm rdkit
# Requires the selected MD version to be published; see the version matrix.
python -m pip install caddforge-core caddforge-md

# Pack the active environment.
caddforge pack-env --output /shared/envs/caddforge.tar.gz

# Alternatively, select a named environment or explicit prefix.
caddforge pack-env --name caddforge --output /shared/envs/caddforge.tar.gz
caddforge pack-env --prefix /path/to/caddforge \
  --output /shared/envs/caddforge.tar.gz

The command creates:

/shared/envs/caddforge.tar.gz
/shared/envs/caddforge.tar.gz.sha256

The sidecar is written in standard sha256sum format. During a long build, pack-env keeps the conda-pack percentage bar on one updating terminal line, streams normal messages, reports elapsed time and the growing archive size during quiet periods, and displays hashing progress.

Existing output is protected by default. Use --force for a deliberate fixed-name replacement:

caddforge pack-env --output /shared/envs/caddforge-dev.tar.gz --force

Use the packed environment in the workflow config:

backend: slurm
slurm_env_pack_path: /shared/envs/caddforge.tar.gz

No sidecar path is configured. CADDForge discovers <slurm_env_pack_path>.sha256 automatically.

Sidecar and fallback behavior

With caching enabled, submission uses the following compatibility order:

1. A valid sidecar supplies the trusted SHA-256 immediately, so the large archive is not hashed by the submission command.
2. If the sidecar is missing, malformed, or names another archive, CADDForge warns and falls back to computing the full archive SHA-256. Manually created packs therefore remain supported.
3. If slurm_env_cache_enabled: false, CADDForge skips sidecar lookup and submission-time hashing, then unpacks the environment separately in each job's private scratch directory. This preserves the behavior used before environment caching was introduced.

The first job requiring a digest on a compute node copies the archive to job-private scratch, verifies it against the selected SHA-256, and installs it into the node-local cache. Later jobs on that node reuse the completed cache entry without copying or unpacking the archive again.

Node-local cache

The default per-user cache root is:

$SCRATCH_BASE/caddforge_environments/$USER

It contains separate environment, lock, and staging directories. Cache locks are node-local and never lock or modify the source archive. Jobs retain a shared usage lock while running, preventing cleanup from deleting an active environment. Entries idle for more than 14 days are removed opportunistically; the lock timeout is 30 minutes. These are core policies, not per-job options.

Sites can override the root with an absolute compute-node path:

slurm_env_cache_dir: /scratch/caddforge-environments/my-user

Node-local caches may disappear during node cleanup or replacement. CADDForge rebuilds missing entries from the shared archive.

Replacing a managed pack

pack-env --force creates and hashes the replacement before briefly acquiring a hidden advisory publication lock next to the archive. CADDForge submissions coordinate with this lock when it is present. The source archive itself is never locked.

An archive and sidecar are separate files, so direct external replacement bypasses this coordination and is unsupported for managed packs. If sidecar publication fails, CADDForge removes the old sidecar when possible, causing submissions to use the safe full-hash fallback.

A job records the digest selected at submission. If a fixed-name archive is replaced before an uncached queued job starts, that job fails checksum verification rather than running a different environment. A job whose matching digest is already cached on its compute node can still use that cache entry.

Option B: Wheelhouse and Per-Job Virtualenv

Use this when all required packages are available as wheels from a package index or a prepared wheelhouse.

Build a wheelhouse:

python -m pip freeze > requirements.txt
python -m pip download -r requirements.txt -d /shared/wheelhouse

Use it in the workflow config:

backend: slurm
slurm_wheelhouse_path: /shared/wheelhouse
slurm_requirements_path: /shared/requirements.txt

The Slurm job creates a temporary virtualenv, installs from the wheelhouse, and runs the workflow.

Option C: Site-Specific Setup Commands

Use slurm_setup_commands only when your cluster already provides a supported module or activation workflow:

backend: slurm
slurm_setup_commands:
  - module purge
  - module load caddforge/0.2
  - source /shared/envs/caddforge/bin/activate

If you use setup commands, do not also set slurm_env_pack_path, slurm_wheelhouse_path, or slurm_requirements_path. The setup commands are then fully responsible for making the correct Python executable available.

Practical Rules

• Ensure all referenced environment paths are readable from compute nodes.
• Use exactly one environment strategy per job.
• Keep packed environments on shared storage, not in a login-node-only path.
• Prefer caddforge pack-env so submission can use its managed SHA-256 sidecar.
• Use --force only when replacing a fixed-name archive is intentional, and avoid replacement while uncached jobs using that archive are queued.
• Rebuild the environment when plugin versions or scientific dependencies change.
• Prefer packed conda environments for workflows with compiled scientific dependencies.
• Prefer wheelhouses for pure Python or locally packaged stacks.