Skip to content

Artifacts

Pipeline stages exchange ArtifactRef objects. Downstream stages should not guess tool-specific directory layouts.

Important fields:

  • path: primary path or manifest path
  • paths: multi-path collection, such as one SDF file per retained pose
  • metadata: provenance hints, result path keys, scatter manifests, and path-collection markers

Input and output ports

Tool plugins can expose pipeline contracts:

  • InputPort(name, kind, config_field) maps a pipeline input to a config field.
  • OutputPort(name, kind, result_path_key) maps a pipeline output to a key in Result.paths.
  • OutputPort(..., required=False) marks outputs that are produced only for some configurations. Missing optional result path keys are skipped instead of failing the stage.
  • OutputPort(..., path_collection_column="pose_file") treats the result path as a CSV manifest and exposes the named column as ArtifactRef.paths. The artifact path remains the manifest, so downstream stages can keep both provenance and exact file paths.

Scatter provenance

Scatter over file inputs supports SDF, CSV, line-based, and auto splitting. Scatter over path collections supports one job per path or chunked path lists depending on the downstream config field type.

When a scattered tool stage receives multiple path-collection inputs and non-scatter inputs have the same number of paths as the scatter input, CADDForge binds those paths by row order for each shard. Use this for paired artifacts selected from the same manifest row, such as an ABFE ligand SDF plus its matching protein PDB. If the paired config field is typed as list[Path], each shard receives a one-item list containing the matching path. Use this for tools such as plain MD where the complex PDB is a single Path but the matching ligand SDF is passed through a list field. The scatter field is still the only field listed under scatter.by.

For path-collection scatter, the executor writes scatter_manifest.csv so downstream stages can join results back to ligand, pose, molecule, or source path metadata.

When scattered CSV, TSV, SDF, JSONL, or manifest outputs are reduced to one canonical file, downstream stages bind that merged file and split it again according to their own scatter contract. The reduced artifact can still retain individual shard paths for provenance. Genuine path collections continue to bind their member paths directly.

Compressed boundaries

Pipeline tools can consume and produce .csv.gz, .tsv.gz, and .sdf.gz through explicit port contracts. Inputs stay compressed between stages and are materialized only at tool runtime. Core atomically compresses declared outputs after the tool succeeds and persists the finalized path in the canonical result. A scattered stage must use a unique compressed output destination per shard.