Configuration¶
CADDForge separates static configuration from runtime execution.
Tool configs are validated Python objects, usually Pydantic models. They should be:
- constructible from Python code
- loadable from YAML or JSON
- explicit about required inputs and defaults
- responsible for validation, not execution
Example config:
export_results is a shared core option. It defaults to false, which keeps outputs under the
standard run directory. Set it to true when a workflow should automatically copy its results/
directory back to the submission working directory after completion.
Use the CLI to inspect templates exposed by installed plugins:
Workflows receive validated config objects and use them to prepare and run jobs.
Transparent compressed inputs¶
At declared plugin boundaries, .csv.gz, .tsv.gz, and .sdf.gz inputs remain compressed on
shared storage and are expanded only when the tool starts. Local jobs use job-scoped temporary
storage, single-node Slurm jobs use node-local scratch, and multi-node jobs use a private shared
staging directory. Saved configuration and results retain the original compressed source path, and
runtime materializations are removed after the job.
Only paths declared by a plugin input contract are staged. Formats that natively consume gzip, such as selected large-space inputs or tool-specific compressed formats, remain compressed when the plugin marks them as native. Plain inputs keep their existing behavior.
Pipeline splitting can stream .csv.gz, .tsv.gz, and .sdf.gz sources into compressed shards;
each shard job expands only its own input. A compressed output is finalized atomically from a plain
tool output when the output port declares its configuration field.
Shared Slurm environment options¶
Packed environments use the following shared fields:
backend: slurm
slurm_env_pack_path: /shared/envs/caddforge.tar.gz
slurm_env_cache_enabled: true
# Optional absolute path on compute nodes:
# slurm_env_cache_dir: /scratch/caddforge_environments/my-user
slurm_env_cache_enabled defaults to true. With caching enabled, CADDForge reuses unpacked
environments in per-user node-local storage. Set it to false to use the original behavior where
every job unpacks its own environment under job-private scratch.
slurm_env_cache_dir overrides the automatically selected node-local cache root and must be an
absolute compute-node path. Cache retention and lock timeouts are core policies and are not
configured per job.
slurm_env_pack_path is mutually exclusive with slurm_wheelhouse_path and
slurm_requirements_path. See the Slurm backend guide for complete environment setup.