Scatter Work and Reduce Results

Use scatter when independent input shards can run as separate jobs. Start with a small pipeline, then adapt a scientific pipeline template.

# Template: replace tool, config, field, and result key with declared public ports.
version: 1
name: batch-analysis
defaults:
  backend: local
stages:
  - id: batch
    tool: section/tool
    config: tool.yaml
    scatter:
      by: input_path
      strategy: csv
      chunk_size: 100
    outputs:
      table: output_csv

This is a contract template, not a runnable tool configuration. Supply exactly one of chunk_size and chunks. Check the installed plugin's input field and reducer before applying scatter. Independent shards are submitted before waiting on Slurm.

File scatter supports SDF, CSV, TSV, lines, and automatic detection. For path collections, a Path input receives one path per shard; a list[Path] input can receive chunks. Keep scatter_manifest.csv and reduced artifact metadata to join results back to inputs. Do not align independently sorted collections by row number.

Molecular workflows often require pose-level execution followed by ligand-level selection. Use poses.extract, scatter the pose collection, collect metrics with results.collect, then apply group_by: ligand and top_n_per_group. Inspect manifests and failures before treating a reduced table as complete. See artifacts and built-ins.