Skip to content

Pipeline Examples

These examples show complete pipeline YAML files and the artifact flow between stages. Replace input paths and config filenames with project-specific files.

Run any example with:

caddforge pipeline run --config pipeline.yaml

These are advanced configuration templates. They require prepared molecular/data inputs and the runtimes listed in the example library. Start with the small runnable pipeline. The supplied configs are schema-checked; full external-engine execution is a separate validation step. Replace all scientific paths and box coordinates before running.

Download configurations

Docking Screen and Rescore

This pipeline screens a multi-record SDF with UniDock, filters by score, retains the selected poses, and rescores them with Gnina.

version: 1
name: docking-screen-rescore

defaults:
  backend: local
  registry_path: runs/job_registry.sqlite
  job_name: docking-screen-rescore

stages:
  - id: unidock_screen
    tool: docking/unidock
    config: configs/unidock.yaml
    scatter:
      # Split the ligand_multisdf_path field from configs/unidock.yaml into independent shards.
      by: ligand_multisdf_path
      chunk_size: 50000
    outputs:
      # Expose Result.paths["best_scores_csv"] as unidock_screen.scores.
      scores: best_scores_csv
      # Expose Result.paths["poses_dir"] as unidock_screen.poses.
      poses: poses_dir

  - id: top_unidock_scores
    type: filter.tabular
    inputs:
      # Read the score table produced by the previous stage.
      table: unidock_screen.scores
    config:
      sort_by: best_score
      ascending: true
      top_n: 1000
      output_name: top_unidock_scores.csv

  - id: retain_unidock_poses
    type: filter.molecules
    inputs:
      # Filter the pose directory/path collection from UniDock.
      molecules: unidock_screen.poses
      # Use the filtered table from filter.tabular; this built-in always emits filtered_table.
      ids: top_unidock_scores.filtered_table
    config:
      # The values in this column are matched against pose metadata or filenames.
      id_column: ligand

  - id: gnina_rescore
    tool: docking/gnina
    config: configs/gnina_rescore.yaml
    inputs:
      # Bind the retained pose path collection to GninaConfig.ligand_paths.
      ligand_paths: retain_unidock_poses.molecules
    outputs:
      # Expose Gnina result paths for downstream stages.
      scores: best_scores_csv
      poses: poses_dir

Artifact flow:

  • unidock_screen.scores points to the reduced UniDock best_scores.csv.
  • unidock_screen.poses points to the UniDock pose directory or path collection.
  • top_unidock_scores.filtered_table is the retained top-N table.
  • retain_unidock_poses.molecules is a retained pose path collection.
  • gnina_rescore.scores is the rescoring table for downstream ranking.

Gnina Flexible Complex Artifacts

This pipeline runs flexible Gnina docking and uses the default Gnina output ports. The flexible artifact ports are optional: they exist only when flexible docking, write_flex_receptor_output, and write_flexible_complexes are enabled in configs/gnina_flexible.yaml.

version: 1
name: gnina-flexible-artifacts

defaults:
  backend: local
  registry_path: runs/job_registry.sqlite
  job_name: gnina-flexible-artifacts

stages:
  - id: gnina_flexible
    tool: docking/gnina
    config: configs/gnina_flexible.yaml

  - id: first_flexible_pose_per_ligand
    type: filter.tabular
    inputs:
      # This is the flexible_complexes_manifest.csv emitted by Gnina.
      table: gnina_flexible.flexible_complexes_manifest
    config:
      group_by: ligand
      sort_by: pose_rank
      ascending: true
      top_n_per_group: 1
      output_name: first_flexible_pose_per_ligand.csv

  - id: retain_ligand_pose_sdfs
    type: filter.molecules
    inputs:
      # Manifest-backed path collection from the ligand_pose_file column.
      molecules: gnina_flexible.flexible_ligand_poses
      ids: first_flexible_pose_per_ligand.filtered_table
    config:
      path_column: ligand_pose_file

  - id: retain_protein_pdbs
    type: filter.molecules
    inputs:
      # Manifest-backed path collection from the protein_file column.
      molecules: gnina_flexible.flexible_proteins
      ids: first_flexible_pose_per_ligand.filtered_table
    config:
      path_column: protein_file

  - id: retain_complex_pdbs
    type: filter.molecules
    inputs:
      # Manifest-backed path collection from the complex_file column.
      molecules: gnina_flexible.flexible_complexes
      ids: first_flexible_pose_per_ligand.filtered_table
    config:
      path_column: complex_file

Artifact flow:

  • gnina_flexible.scores is the normal best_scores.csv.
  • gnina_flexible.flexible_complexes_manifest is the per-pose flexible artifact manifest.
  • gnina_flexible.flexible_ligand_poses, gnina_flexible.flexible_proteins, and gnina_flexible.flexible_complexes are manifest-backed path collections.
  • The retained artifacts contain matched ligand-only SDFs, protein-only PDBs, and complex PDBs for the selected flexible pose per ligand.

Flexible Gnina to OpenBPMD to MD to ABFE

This pipeline keeps ligands whose best Gnina score is <= -12, selects the top 4 flexible poses per retained ligand by CNNscore (larger is better), runs OpenBPMD on those protein-ligand complexes, keeps the pose with the best OpenBPMD comp_score, runs 10 ns plain MD for that complex, centers and aligns the MD trajectory, extracts a whole-frame cluster representative, and runs OpenFE ABFE on the representative protein-only PDB plus representative ligand-only SDF.

version: 1
name: flexible-gnina-openbpmd-md-abfe

defaults:
  backend: slurm
  registry_path: runs/job_registry.sqlite
  job_name: flexible-gnina-openbpmd-md-abfe

stages:
  - id: gnina_flexible
    tool: docking/gnina
    config: configs/gnina_flexible.yaml

  - id: retained_ligands
    type: filter.tabular
    inputs:
      # One row per ligand; best_score is lower-is-better affinity.
      table: gnina_flexible.scores
    config:
      filters:
        - column: best_score
          op: "<="
          value: -12.0
      columns: [ligand, best_score]
      output_name: retained_ligands.csv

  - id: top4_flexible_poses
    type: filter.tabular
    inputs:
      # One row per generated flexible pose, with complex_file/protein_file/ligand_pose_file.
      table: gnina_flexible.flexible_complexes_manifest
      # Retain all poses belonging to ligands that passed the best-score cutoff.
      ids: retained_ligands.filtered_table
    config:
      id_column: ligand
      ids_column: ligand
      group_by: ligand
      sort_by: CNNscore
      ascending: false
      top_n_per_group: 4
      columns:
        - ligand
        - pose_rank
        - minimizedAffinity
        - CNNscore
        - complex_file
        - protein_file
        - ligand_pose_file
      output_name: top4_flexible_poses.csv

  - id: retain_top4_complexes
    type: filter.molecules
    inputs:
      # OpenBPMD should receive the full protein-ligand complex PDB.
      molecules: gnina_flexible.flexible_complexes
      ids: top4_flexible_poses.filtered_table
    config:
      path_column: complex_file

  - id: openbpmd_top4
    tool: md/openbpmd
    config: configs/openbpmd_pose.yaml
    inputs:
      # Bind each selected complex PDB to OpenBPMDConfig.complex_pdb_path.
      complex_pdb_path: retain_top4_complexes.molecules
    scatter:
      by: complex_pdb_path
      chunk_size: 1
    outputs:
      # results.collect resolves the sibling result_record.json from this path.
      result_record: results_json

  - id: collect_openbpmd
    type: results.collect
    inputs:
      results: openbpmd_top4.result_record
      # Preserve ligand, pose_rank, and the exact flexible artifact paths.
      table: top4_flexible_poses.filtered_table
    config:
      metrics:
        comp_score: metrics.comp_score
        pose_score: metrics.pose_score
      output_name: openbpmd_top4_metrics.csv

  - id: best_openbpmd_pose
    type: filter.tabular
    inputs:
      table: collect_openbpmd.table
    config:
      group_by: ligand
      sort_by: comp_score
      ascending: true
      top_n_per_group: 1
      columns:
        - ligand
        - pose_rank
        - comp_score
        - pose_score
        - complex_file
        - protein_file
        - ligand_pose_file
        - result_file
      output_name: best_openbpmd_pose_per_ligand.csv

  - id: retain_best_complexes
    type: filter.molecules
    inputs:
      # Plain MD receives the full protein-ligand complex PDB for the selected pose.
      molecules: gnina_flexible.flexible_complexes
      ids: best_openbpmd_pose.filtered_table
    config:
      path_column: complex_file

  - id: retain_best_ligands
    type: filter.molecules
    inputs:
      # Plain MD needs the ligand-only SDF for non-standard ligand parameterization.
      molecules: gnina_flexible.flexible_ligand_poses
      ids: best_openbpmd_pose.filtered_table
    config:
      path_column: ligand_pose_file

  - id: plain_md_10ns
    tool: md/plain-md
    config:
      simulation_time_ns: 10.0
      trajectory_filename: traj.xtc
      output_dir_name: output
    inputs:
      system_pdb_path: retain_best_complexes.molecules
      molecule_sdf_paths: retain_best_ligands.molecules
    scatter:
      # Scatter over complex PDBs. The matching ligand SDF is bound as a one-item
      # molecule_sdf_paths list for the same shard.
      by: system_pdb_path
      chunk_size: 1

  - id: center_align_md
    tool: md/center-and-align
    config:
      center_selection: protein or resname LIG
      keep_selection: protein or resname LIG
      align_selection: protein and backbone
      output_trajectory_filename: traj-centered.xtc
      output_structure_filename: traj-centered.pdb
    inputs:
      # The plain-md trajectory uses the prepared solvated system topology.
      topology_path: plain_md_10ns.prepared_structure
      trajectory_path: plain_md_10ns.trajectory
    scatter:
      by: trajectory_path
      chunk_size: 1
    outputs:
      centered_structure: output_structure
      centered_trajectory: output_trajectory

  - id: cluster_representative
    tool: md/cluster-representative
    config:
      ligand_resname: LIG
      cluster_selection: all
      extract_selection: all
      last_ns: 10.0
      output_complex_filename: representative_complex.pdb
      output_protein_filename: representative_protein.pdb
      output_ligand_sdf_filename: representative_ligand.sdf
    inputs:
      topology: center_align_md.centered_structure
      trajectory: center_align_md.centered_trajectory
      # The ligand template preserves bond orders, hydrogens, and formal charges in the exported SDF.
      ligand_template: retain_best_ligands.molecules
    scatter:
      by: trajectory_path
      chunk_size: 1

  - id: openfe_abfe
    tool: fep/openfe-abfe
    config: configs/openfe_abfe.yaml
    inputs:
      # ABFE receives separate files from the representative frame, not the original docked pose.
      ligand_sdf: cluster_representative.ligand
      protein_pdb: cluster_representative.protein
    scatter:
      # Scatter over ligand_sdf; the matching representative protein_pdb is bound by row order.
      by: ligand_sdf
      chunk_size: 1
    outputs:
      results: results_json

Input choices:

  • Flexible Gnina must be configured with a PDB receptor, output_format: sdf, write_flex_receptor_output: true, and write_flexible_complexes: true.
  • OpenBPMD receives complex_pdb_path from gnina_flexible.flexible_complexes because it operates on a full protein-ligand complex.
  • Plain MD receives the selected complex PDB as system_pdb_path and the matching ligand-only SDF as a one-item molecule_sdf_paths list for ligand parameterization.
  • The LIG residue name in center_align_md and cluster_representative must match flexible_complex_ligand_resname from the Gnina config.
  • center_align_md uses plain_md_10ns.prepared_structure as topology and plain_md_10ns.trajectory as trajectory.
  • cluster_representative uses the centered structure/trajectory and the matching ligand template SDF. cluster_selection: all makes the representative frame come from the whole retained centered system rather than ligand-only clustering.
  • ABFE receives ligand_sdf and protein_pdb from the cluster representative. It does not receive the original docked ligand SDF, the original flexible protein PDB, or the representative complex PDB.

Minimize a Multi-Pose SDF

Use md/minimize to clean up already prepared poses against one receptor snapshot. Install a plugin revision containing minimize (see the version matrix). Each SDF record must contain one explicitly hydrogenated 3D molecule with authoritative chemistry, already positioned in the receptor coordinate frame. The receptor PDB must have complete protein heavy atoms and no target ligand. A separate ligand file is not aligned automatically.

Download the SDF scatter pipeline. Save it as pipeline_scatter.yaml, update the paths, and run:

caddforge pipeline run --config pipeline_scatter.yaml

# Poses must share receptor coordinates and contain explicit hydrogens.
# Paths resolve relative to the submission working directory.
version: 1
name: pose-cleanup

defaults:
  backend: local
  run_directory: runs/pose-cleanup
  registry_path: runs/job_registry.sqlite
  export_results: false

stages:
  - id: minimize
    tool: md/minimize
    config:
      protein_pdb_path: receptor.pdb
      ligand_path: poses.sdf
      mode: ligand_only
    scatter:
      by: ligand_path
      chunk_size: 1
    outputs:
      results: result_record
      ligands: minimized_ligand

  - id: collect
    type: results.collect
    inputs:
      results: minimize.results
    config:
      metrics:
        quality_passed: metrics.quality_passed
        convergence: metrics.convergence_status
        ligand_rmsd_angstrom: metrics.ligand_heavy_atom_rmsd_angstrom
        minimized_ligand_file: paths.minimized_ligand

  - id: accepted
    type: filter.tabular
    inputs:
      table: collect.table
    config:
      filters:
        - column: quality_passed
          op: '=='
          value: 1

minimize.results is the collection of canonical result records; minimize.ligands is the merged minimized SDF. collect.table has one row per job, while accepted.filtered_table retains only rows without quality flags. Its minimized_ligand_file column points to each final ligand SDF. No accepted rows is a valid result: inspect the rejected diagnostics before changing review thresholds. The table filter does not itself filter the merged SDF.

SDF titles and properties survive minimization. For explicit ligand and pose_id columns joined by core scatter provenance, use the extracted path collection in the next example. Total minimization energy is local cleanup information, not a binding score or a basis for ranking unrelated ligands.

Dock, Extract, and Minimize Poses

This example runs rigid-receptor GNINA docking, extracts one SDF per pose, restores explicit hydrogens, and scatters minimization over that path collection. It uses the same receptor PDB for docking and cleanup. Supply a GNINA executable and the GNINA requirements, plus the minimize runtime.

Download the docking/minimization pipeline. Run it with caddforge pipeline run --config pipeline_docking.yaml after replacing the data/ paths.

# Rigid-receptor GNINA docking followed by local OpenMM pose cleanup.
# Supply GNINA on PATH and authoritative ligand chemistry in the input SDF.
version: 1
name: dock-extract-minimize

defaults:
  backend: local
  run_directory: runs/dock-extract-minimize
  registry_path: runs/job_registry.sqlite
  export_results: false

stages:
  - id: dock
    tool: docking/gnina
    config:
      receptor_path: data/receptor.pdb
      ligand_path: data/ligands.sdf
      autobox_ligand_path: data/reference_ligand.sdf
      output_format: sdf
    outputs:
      poses: poses_dir
      scores: best_scores_csv

  - id: extract
    type: poses.extract
    inputs:
      poses: dock.poses
      table: dock.scores
    config:
      ligand_column: ligand
      path_column: pose_file
      output_name: pose_manifest.csv

  - id: hydrogenate
    type: molecules.hydrogenate
    inputs:
      molecules: extract.poses
      table: extract.manifest
    config:
      on_template_failure: fail
      on_failure: fail
      add_coords: true

  - id: minimize
    tool: md/minimize
    config:
      protein_pdb_path: data/receptor.pdb
      mode: sidechains_free_backbone_restrained
      backbone_restraint_strength: moderate
      strict_quality: false
    inputs:
      ligand: hydrogenate.molecules
    scatter:
      by: ligand_path
      chunk_size: 1
    outputs:
      results: result_record
      ligands: minimized_ligand

  - id: collect
    type: results.collect
    inputs:
      results: minimize.results
    config:
      metrics:
        quality_passed: metrics.quality_passed
        quality_flag_count: metrics.quality_flag_count
        convergence: metrics.convergence_status
        ligand_rmsd_angstrom: metrics.ligand_heavy_atom_rmsd_angstrom
        minimized_complex_file: paths.minimized_complex
        minimized_ligand_file: paths.minimized_ligand
        diagnostics_file: paths.diagnostics
      output_name: minimized_pose_metrics.csv

  - id: accepted
    type: filter.tabular
    inputs:
      table: collect.table
    config:
      filters:
        - column: quality_passed
          op: '=='
          value: 1
      output_name: accepted_minimized_poses.csv
  • extract.poses contains one SDF per pose; extract.manifest records ligand and pose IDs. Hydrogenation preserves those IDs in its output manifest.
  • The minimize input port is ligand; the scatter config field is ligand_path. Each shard receives exactly one pose.
  • results.collect automatically joins the path-collection scatter manifest, preserving ligand and pose_id alongside minimization metrics.
  • accepted.filtered_table points to the minimized ligand and complex files. Original provenance columns such as pose_file or output_file can still identify the input poses; consume the explicit minimized_*_file columns for final structures, including any relaxed receptor coordinates.

Direct hydrogen addition relies on the docking SDF's chemistry and cannot recover missing bond orders or an intended protonation state. Confirm those upstream. When an upstream stage provides authoritative templates, bind its artifact as templates on hydrogenate (see template-based repair). on_template_failure: fail prevents fallback if a configured template repair fails; it does not require a template to be present.

For flexible-receptor docking, carry the matching per-pose receptor/complex through paired scatter; do not combine all ligand poses with the original rigid receptor. Keep strict_quality: false while collecting finite flagged poses for review. Strict mode deliberately fails rejected jobs and may stop the pipeline before collection. Hard chemistry, parameterization, and nonfinite-state errors always fail. A quality pass alone does not establish pose accuracy.

For Slurm, change defaults.backend to slurm and supply the environment/resources from the Slurm guide. Choose resources per tool: the minimize stage defaults to CPU even when the upstream docking stage uses a GPU.

Pose-Level MD Selection

This pipeline extracts one file per docking pose, repairs explicit hydrogens after docking, scatters OpenBPMD over each repaired pose, collects per-pose metrics, selects the best pose per ligand, and retains exact pose files.

version: 1
name: pose-level-openbpmd-selection

defaults:
  backend: slurm
  registry_path: runs/job_registry.sqlite
  job_name: pose-level-openbpmd-selection

stages:
  - id: gnina_docking
    tool: docking/gnina
    config: configs/gnina_docking.yaml
    outputs:
      # Expose the score CSV and pose directory from Result.paths.
      scores: best_scores_csv
      poses: poses_dir

  - id: extract_poses
    type: poses.extract
    inputs:
      # Split every SDF/SDFGZ pose from Gnina into a normalized one-pose SDF file.
      poses: gnina_docking.poses
      # Optional but useful: carries score-table metadata into the pose manifest.
      table: gnina_docking.scores
    config:
      output_name: pose_manifest.csv
      ligand_column: ligand
      path_column: pose_file

  - id: repair_pose_hydrogens
    type: molecules.hydrogenate
    inputs:
      # Hydrogenate the extracted pose path collection before handing poses to OpenBPMD.
      molecules: extract_poses.poses
      # Reuse the pose manifest so ligand, pose_id, and pose_file provenance are preserved.
      table: extract_poses.manifest
    config:
      ligand_column: ligand
      pose_id_column: pose_id
      # Without a templates input, hydrogens are added directly to each pose.
      # See the built-in stage docs for template-based bond-order repair.
      on_template_failure: warn
      on_failure: fail
      add_coords: true
      output_name: hydrogenated_pose_manifest.csv

  - id: openbpmd_per_pose
    tool: md/openbpmd
    config: configs/openbpmd_pose.yaml
    inputs:
      # Bind each hydrogenated pose file to OpenBPMDConfig.ligand_path.
      ligand_path: repair_pose_hydrogens.molecules
    scatter:
      # Path-collection scatter submits each ligand_path shard independently.
      by: ligand_path
      chunk_size: 1
    outputs:
      # Expose canonical per-shard result records for collection.
      result_record: result_record_json

  - id: collect_pose_metrics
    type: results.collect
    inputs:
      # Collect metrics from every scattered OpenBPMD result record.
      results: openbpmd_per_pose.result_record
      # Join explicit hydrogenation metadata to the collected metrics.
      table: repair_pose_hydrogens.manifest
    config:
      metrics:
        comp_score: metrics.comp_score
        pose_score: metrics.pose_score
      output_name: openbpmd_pose_metrics.csv

  - id: best_pose_per_ligand
    type: filter.tabular
    inputs:
      # results.collect always emits collect_pose_metrics.table.
      table: collect_pose_metrics.table
    config:
      group_by: ligand
      sort_by: comp_score
      ascending: true
      top_n_per_group: 1
      output_name: best_pose_per_ligand.csv

  - id: retain_best_poses
    type: filter.molecules
    inputs:
      # Filter the hydrogenated pose files, not the pre-repair Gnina poses.
      molecules: repair_pose_hydrogens.molecules
      ids: best_pose_per_ligand.filtered_table
    config:
      # Prefer exact file paths from the filtered table over filename matching.
      path_column: output_file

Artifact flow:

  • extract_poses.poses is a path collection with one SDF per pose.
  • repair_pose_hydrogens.molecules is the repaired path collection used by OpenBPMD.
  • Path-collection scatter creates one OpenBPMD shard per repaired pose.
  • results.collect joins per-pose metrics to pose/hydrogenation provenance.
  • filter.tabular uses group_by: ligand; this selects the best pose per ligand rather than the globally best poses.
  • retain_best_poses.molecules contains exact selected repaired pose files.

Descriptor, Model, and Candidate Filter

This pipeline computes descriptors, trains an ML model, and filters candidates by predicted value. The example assumes the descriptor and ML configs agree on the generated table columns.

version: 1
name: descriptor-ml-filter

defaults:
  backend: local
  registry_path: runs/job_registry.sqlite

stages:
  - id: psa_descriptors
    tool: descriptors/ani-psa
    config: configs/ani_psa_batch.yaml
    outputs:
      # Expose Result.paths["results_csv"] as psa_descriptors.table.
      table: results_csv

  - id: train_gp
    tool: ml/simple-gp
    config:
      train_path: data/training_descriptors.csv
      target_column: measured_property
      # Use the table artifact from the descriptor stage as the GP test/prediction set.
      test_path: psa_descriptors.table
      feature_columns: [ani_psa, heavy_atom_count]
      kernel: rbf
      mode: train
      model_path: artifacts/simple_gp_model.joblib
    outputs:
      # Expose Result.paths["predictions_csv"] as train_gp.predictions.
      predictions: predictions_csv

  - id: promising_candidates
    type: filter.tabular
    inputs:
      table: train_gp.predictions
    config:
      filters:
        - column: y_pred
          op: ">="
          value: 7.0
      sort_by: y_pred
      ascending: false
      top_n: 250
      output_name: promising_candidates.csv

Artifact flow:

  • psa_descriptors.table is the descriptor table.
  • train_gp.predictions is the model prediction table.
  • promising_candidates.filtered_table is the ranked filtered candidate list.

Result Collection and Diagnostics

This pipeline collects scattered JSON results, filters the collected metrics, and produces a distribution diagnostic report.

version: 1
name: collect-and-diagnose

defaults:
  backend: local
  registry_path: runs/job_registry.sqlite

stages:
  - id: scattered_transfer
    tool: qm/transfer-free-energy
    config: configs/transfer_free_energy.yaml
    scatter:
      # Split the ligand SDF into one-molecule transfer calculations.
      by: ligand_sdf
      chunk_size: 1
    outputs:
      # Expose per-shard canonical result records.
      result_record: result_record_json

  - id: collect_transfer
    type: results.collect
    inputs:
      results: scattered_transfer.result_record
    config:
      metrics:
        # Copy payload["metrics"]["delta_g_kcal_mol"] into a table column.
        delta_g_kcal_mol: metrics.delta_g_kcal_mol
      output_name: transfer_metrics.csv

  - id: filter_transfer
    type: filter.tabular
    inputs:
      # results.collect emits collect_solvation.table.
      table: collect_transfer.table
    config:
      filters:
        - column: delta_g_kcal_mol
          op: "<="
          value: -3.0
      sort_by: delta_g_kcal_mol
      ascending: true
      output_name: favorable_transfer.csv

  - id: distribution_report
    tool: datascience/distribution-diagnostics
    config:
      # Built-in stage outputs can be bound into inline tool config fields.
      csv_path: filter_transfer.filtered_table
      column: delta_g_kcal_mol
      bins: 50
      include_log: false
    outputs:
      report: report_json
      plot: plot_png

Artifact flow:

  • scattered_transfer.result_record is a reduced collection of result records.
  • collect_transfer.table joins JSON metrics to scatter provenance.
  • filter_transfer.filtered_table is a subset table.
  • distribution_report.report and distribution_report.plot summarize the filtered distribution.