Metadata-Version: 2.5
Name: caddforge-core
Version: 0.22.0
Summary: Core abstractions, CLI, and plugin system for the CADDForge toolbox.
Author-email: Manuel Sellner <manuel.sellner@idorsia.com>, Christoph Sager <christoph.sager@idorsia.com>
License: MIT
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: conda-pack>=0.9.1
Requires-Dist: petname>=2.6
Requires-Dist: pydantic>=2.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.9
Provides-Extra: test
Requires-Dist: hypothesis>=6.0; extra == 'test'
Requires-Dist: pytest>=7.0; extra == 'test'
Requires-Dist: rdkit; extra == 'test'
Requires-Dist: rich>=13.8; extra == 'test'
Description-Content-Type: text/markdown

# CADDForge Core

CADDForge Core provides the **shared abstractions, config models, and CLI** that all CADDForge
plugin tools build on. It is intentionally lightweight and backend-agnostic so that tool-specific
packages (e.g., `cadd_qm`, `cadd_md`, `cadd_fep`, `cadd_descriptors`) can focus on scientific logic
while still benefiting from standardized execution, logging, and results handling.

This repo is designed to be **open-source friendly**: a clean API surface, simple configuration,
and a CLI that is discoverable and reproducible.

---

## Table of Contents
- [Key Features](#key-features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Python API](#python-api)
- [Pipelines](#pipelines)
- [Core Concepts](#core-concepts)
- [Configuration](#configuration)
- [CLI Reference](#cli-reference)
- [Job Layout & Registry](#job-layout--registry)
- [Backends](#backends)
- [Plugin Architecture](#plugin-architecture)
- [Development & Testing](#development--testing)
- [License](#license)

---

## Key Features
- **Stable core abstractions**: `Dataset`, `MoleculeSet`, `Workflow`, `Job`, `Backend`, `Result`.
- **Config validation** via Pydantic models, with YAML/JSON support.
- **Backend-agnostic execution** (local + Slurm built in).
- **Standardized run directories**, metadata, and result serialization.
- **Plugin discovery** through Python entry points or dev overrides.
- **CLI-first UX** with `run`, `show-config`, `status`, `logs`, and more.

---

## Installation

Requirements: **Python 3.10+**

Install only core infrastructure and the CLI:

```bash
python -m pip install caddforge-core
```

`pip install caddforge` installs the complete public toolbox through a metadata-only
bundle, including GUI and user-facing extras. Individual plugins install core automatically.
The core distribution remains independent of scientific plugins; Python imports and the
`caddforge` command retain their names.

For an existing environment using the former `caddforge` core distribution, prefer a fresh
environment. For in-place migration, uninstall `caddforge` first, then install `caddforge-core`
and upgrade all retained plugins to their migrated releases (or install the complete bundle).
The old and renamed core distributions own the same Python files and must not coexist.
If both were already installed, uninstall the old distribution and force-reinstall core.

Runtime version queries now use `importlib.metadata.version("caddforge-core")`.
`version("caddforge")` reports the optional bundle version. Configuration formats,
plugin entry-point groups, job registries, and historical results remain compatible.

From source (editable):
```bash
cd caddforge-core
python -m pip install -e .
```

If you are using multiple tool plugins, install them in the same environment so the CLI can
discover their entry points.

---

## Quick Start

List installed tools:
```bash
caddforge list-tools
```

Show a tool's default config template:
```bash
caddforge show-config descriptors ani-psa
```

Run a tool from config:
```bash
caddforge run descriptors ani-psa --config path/to/config.yaml
```

Check status and logs:
```bash
caddforge status <job-id>
caddforge logs <job-id>
caddforge logs <job-id> --follow
```

---

## Python API

Use `submit_workflow` when Python code should follow the same backend selection,
run-directory, metadata, and registry path as the CLI:

```python
from caddforge.core.execution import load_result_for_job, submit_workflow
from caddforge.plugins import get_tool

spec = get_tool("descriptors", "ani-psa")
cfg = spec.config_cls.from_yaml("ani_psa.yaml")
workflow = spec.workflow_cls(cfg)

job = submit_workflow(workflow, cfg, backend_name=cfg.backend)
result = load_result_for_job(job)
if result is not None:
    print(result.status.value, result.metrics)
```

Calling `workflow.prepare_job()` followed by `workflow.run(job)` is still useful for
tests or custom inline execution, but it does not submit to a backend. Backend
submission is handled by `submit_workflow`, `submit_prepared_workflow_job`, or a
`Backend` instance such as `LocalBackend` / `SlurmBackend`.

---

## Pipelines

Pipelines chain multiple CADDForge workflows through file-based artifact references. This keeps
large data on disk, works with local and Slurm backends, and lets downstream stages receive paths
produced by upstream stages.

Create a pipeline YAML file, for example `pipeline.yaml`:
```yaml
version: 1
name: docking-screen

defaults:
  backend: local
  registry_path: runs/job_registry.sqlite
  job_name: unidock-screen-test

stages:
  - id: unidock_screen
    tool: docking/unidock
    config: unidock.yaml
    scatter:
      by: ligand_multisdf_path
      chunk_size: 50000
    outputs:
      scores: best_scores_csv
      poses: poses_dir

  - id: top_score_filter
    type: filter.tabular
    inputs:
      table: unidock_screen.scores
    config:
      sort_by: best_score
      ascending: true
      top_n: 1000
      output_name: filtered_scores.csv

  - id: retain_unidock_poses
    type: filter.molecules
    inputs:
      molecules: unidock_screen.poses
      ids: top_score_filter.filtered_table
    config:
      id_column: ligand

  - id: gnina_rescore
    tool: docking/gnina
    config: gnina.yaml
    inputs:
      ligand_paths: retain_unidock_poses.molecules
    outputs:
      scores: best_scores_csv
      poses: poses_dir
```

Run it:
```bash
caddforge pipeline run --config pipeline.yaml
```

Inspect the pipeline and its artifacts:
```bash
caddforge pipeline status <pipeline-id>
caddforge pipeline show-artifacts <pipeline-id>
```

Important details:
- `tool` uses the same `section/name` pair shown by `caddforge list-tools`.
- `config` can be an inline mapping or a YAML/JSON config file for that tool.
- `defaults.job_name` names the parent pipeline job. If omitted, CADDForge generates one with
  the same pet-name helper used by normal workflow configs.
- `outputs` maps pipeline artifact names to keys in the workflow `Result.paths`, such as
  `best_scores_csv` or `poses_dir`.
- Tool plugins can also declare default output ports. Optional ports are skipped when the workflow
  did not produce that result path, and manifest-backed ports can expose a CSV path column as a
  multi-path artifact for scatter.
- `inputs` maps downstream config fields to upstream artifacts using `stage_id.artifact_name`.
- `scatter` splits a large file input into shard jobs. Supported split strategies are `sdf`,
  `csv`, `tsv`, `lines`, or `auto`.
- Tool plugins may declare a `SplitContract` for structured collection fields. The contract selects
  the path member to split, preserves the other entry fields in every shard, and declares how shard
  outputs are reduced.
- `scatter` can also fan out a multi-path artifact, such as `poses.extract.poses`. If the target
  config field is a single `Path`, each path becomes one shard job. If it is `list[Path]`, paths are
  chunked according to `chunk_size` or `chunks`.
- When a scattered stage receives additional path-collection inputs with the same number of paths as
  the scatter input, those paths are bound by row order for each shard. This supports paired files
  selected from the same provenance table, such as ABFE `ligand_sdf` and matching `protein_pdb`
  inputs. If the paired config field is typed as `list[Path]`, each shard receives a one-item list
  containing the matching path, for example plain MD `molecule_sdf_paths`.
- In the current implementation, scatter shards are submitted as separate child jobs and reduced
  before downstream stages run.
- Pipeline tool stages support transparent `.csv.gz`, `.tsv.gz`, and `.sdf.gz` boundaries. Plugins
  declare gzip-capable inputs with `InputPort.config_field` and gzip-capable outputs by setting
  `OutputPort(..., config_field="output_path")`. Core stream-decompresses declared inputs only when
  the tool starts, lets the tool write a plain staging output, and atomically creates the requested
  gzip result. On single-node Slurm jobs, plain inputs live under `$SLURM_TMPDIR`; local jobs use
  job-scoped system temporary storage. Multi-node jobs use a private shared staging directory because
  one node's local scratch is not visible to the other nodes.
  This explicit contract prevents existing outputs, unrelated archives such as Conda `.tar.gz`
  packs, and same-named files from being misclassified.
- Original compressed paths remain in saved configs and results; runtime materializations are
  reused within a job and removed after it finishes. If local scratch fills, the job fails clearly
  instead of silently expanding the input on shared storage. Plugins mark native compressed inputs
  with `InputCompressionPolicy.NATIVE`; the Glide `.sdfgz` convention remains separate and unchanged.
- Plugin contracts may select nested paths with dotted collection syntax, for example
  `InputPort("inputs", ArtifactKind.FILE, "input_files", runtime_selector="input_files[].path")`.
  `config_field` remains the pipeline binding target, while `runtime_selector` only identifies paths
  to stage. The default compression policy is transparent gzip for backward compatibility. Use
  `compression=InputCompressionPolicy.NATIVE` when a tool consumes the gzip stream itself or when
  `.gz` is part of its native file format.
- Scattering a gzip source with `chunk_size` reads the data in one streaming pass and writes gzip
  shards on shared storage. Each shard job expands only its own shard in runtime scratch. Using
  `chunks` still requires a count pass to determine balanced shard sizes. Scattered stages may
  provide one unique gzip output path per shard; one shared output path is rejected because
  concurrent shard writes would corrupt it.
- When scatter fans out a multi-path artifact, CADDForge writes a `scatter_manifest.csv` for the
  scattered stage. This preserves row-level provenance such as `ligand`, `pose_id`, and `pose_file`
  so later stages can collect per-shard results and join them back to the input pose.
- When a retained artifact contains multiple paths and the downstream config field is typed as
  `list[Path]`, CADDForge passes the path list. This is how `retain_unidock_poses.molecules` is
  routed into Gnina's `ligand_paths`.
- The example above sorts UniDock's `best_scores_csv`, which currently contains `ligand`,
  `pose_file`, and `best_score`. If you want ligand efficiency, add or join a descriptor column such
  as `heavy_atom_count` upstream, then use `derive: {ligand_efficiency: best_score / heavy_atom_count}`.

### Built-in filter stages

Built-in stages use `type:` instead of `tool:`. They have fixed output artifact names, so those
names do not need to be listed in an `outputs:` block. You reference them downstream as
`stage_id.output_name`.

#### `filter.tabular`

Filters, ranks, and rewrites delimited tabular files. CSV and TSV inputs are supported.

Required source (choose exactly one):

- `table`: a tabular artifact from an upstream stage, for example `unidock_screen.scores`.
- `input_path`: a file path in `config`, for a source stage that reads a raw table directly. A
  relative path is resolved against the pipeline specification directory.

Optional inputs:

- `ids`: a tabular artifact containing identifiers to retain before filtering and ranking. By
  default, `filter.tabular` matches the `ligand` column in both tables.

Output artifacts:

- `filtered_table`: the filtered CSV or TSV file. Downstream reference:
  `top_score_filter.filtered_table`.

Config keys:

- `input_path`: optional raw CSV or TSV input. This cannot be used together with the `table` input.
- `input_delimiter`: optional one-character delimiter override. Otherwise, `.csv` uses a comma,
  `.tsv` and `.tab` use a tab, and other extensions are detected from the file contents.
- `derive`: optional mapping of new column names to restricted arithmetic/comparison expressions.
- `filters`: optional list of column filters. Supported operators are `==`, `!=`, `<`, `<=`, `>`,
  `>=`, `in`, and `not in`.
- `where`: optional restricted boolean expression evaluated per row.
- `sort_by`: optional column used for sorting.
- `ascending`: sort direction. Set to `true` for ascending order or `false` for descending order.
- `top_n`: optional number of rows to retain after filtering/sorting.
- `group_by`: optional column used with `top_n_per_group`.
- `top_n_per_group`: optional number of rows to retain per group after filtering/sorting.
- `columns`: optional output column order/subset.
- `id_column`: column in `table` used with the optional `ids` input. Defaults to `ligand`.
- `ids_column`: column in `ids` used with the optional `ids` input. Defaults to `id_column`.
- `output_name`: output filename under the stage results directory; defaults to
  `filtered_scores.csv`. A `.tsv` or `.tab` extension selects tab-separated output; other
  extensions default to comma-separated output.
- `output_delimiter`: optional one-character output delimiter override.

Example using an upstream artifact:

```yaml
- id: top_score_filter
  type: filter.tabular
  inputs:
    table: unidock_screen.scores
  config:
    sort_by: best_score
    ascending: true
    top_n: 1000
```

Example using a raw TSV source next to the pipeline specification:

```yaml
- id: reinvent_prefilter
  type: filter.tabular
  config:
    input_path: reinvent_export.tsv
    output_name: filtered_reinvent.tsv
    filters:
      - column: Score
        op: ">="
        value: 0.75
```

The output is referenced as:

```yaml
top_score_filter.filtered_table
```

#### `poses.extract`

Splits docking pose artifacts into one normalized SDF file per pose and writes a pose manifest.
Use this when a docking tool produces multiple poses per ligand and downstream stages should
receive individual pose files.

Required inputs:
- `poses`: a pose artifact, for example `unidock_screen.poses`.

Optional inputs:
- `table`: a CSV artifact with score/metadata rows, for example `unidock_screen.scores`.

Output artifacts:
- `manifest`: a CSV pose manifest. Downstream reference: `extract_poses.manifest`.
- `poses`: a multi-path artifact containing the extracted single-pose SDF files. Downstream
  reference: `extract_poses.poses`.

Manifest columns:
- `ligand`: ligand identifier.
- `pose_id`: stable pose identifier generated from ligand and rank.
- `pose_rank`: rank/order among poses for that ligand.
- `pose_file`: extracted single-pose SDF path.
- `source_pose_file`: original SDF/SDFGZ file.
- `source_record_index`: record index inside the original file.
- Any matching columns from the optional `table`, plus an optional SDF property selected with
  `score_column`.

Config keys:
- `ligand_column`: ligand column in the optional table. Defaults to `ligand`.
- `path_column`: pose path column in the optional table. If omitted, CADDForge auto-detects common
  path columns such as `pose_file`.
- `score_column`: optional SDF property to copy into the manifest when it is not already present
  in the table.
- `sort_by`: optional manifest column used for sorting.
- `ascending`: sort direction. Set to `true` for ascending order or `false` for descending order.
- `top_n_per_ligand`: optional number of poses to retain per ligand after sorting.
- `output_name`: manifest filename; defaults to `pose_manifest.csv`.

Example: extract the best three poses per ligand, then fan them out one pose per downstream job:
```yaml
- id: extract_unidock_poses
  type: poses.extract
  inputs:
    poses: unidock_screen.poses
    table: unidock_screen.scores
  config:
    ligand_column: ligand
    score_column: dock_score
    sort_by: dock_score
    ascending: true
    top_n_per_ligand: 3

- id: downstream_per_pose
  tool: my-section/my-tool
  config: downstream.yaml
  inputs:
    ligand_path: extract_unidock_poses.poses
  scatter:
    by: ligand_path
    chunk_size: 1
```

#### `molecules.hydrogenate`

Adds missing explicit hydrogens to retained molecules or poses. This is useful after docking tools
that remove non-polar hydrogens and before downstream tools that expect complete ligand structures.
Place this stage after filtering and pose extraction so only retained poses are repaired.

Required inputs:
- `molecules`: molecule/pose artifact. This can be a single SDF/SDFGZ, multi-record SDF/SDFGZ,
  directory, or multi-path artifact such as `extract_gnina_poses.poses`.

Optional inputs:
- `templates`: fully protonated template molecules used to preserve protonation/formal charges and
  bond orders.
- `table`: CSV/manifest used to find template paths or merge metadata.

Output artifacts:
- `molecules`: repaired single-record SDF path collection.
- `manifest`: CSV with input/output paths, hydrogen counts, template status, and messages.

Config keys:
- `ligand_column`: ligand identifier column. Defaults to `ligand`.
- `pose_id_column`: pose identifier column. Defaults to `pose_id`.
- `template_path_column`: template path column. Defaults to `auto`, which detects columns such as
  `template_file`, `original_ligand_file`, `source_molecule_file`, `ligand_file`, or
  `molecule_file`.
- `on_template_failure`: `warn`, `fail`, or `ignore`. Defaults to `warn`.
- `on_failure`: `fail` or `warn`. Defaults to `fail`.
- `add_coords`: generate coordinates for added hydrogens. Defaults to `true`.
- `output_name`: manifest filename; defaults to `hydrogenated_manifest.csv`.

Template behavior:
- If a template path column is available, CADDForge loads only those template files for the retained
  poses.
- If no template path column is available but a `templates` artifact is provided, CADDForge indexes
  templates by SDF title and ligand ID. This is convenient for small retained sets, but explicit
  template paths are preferred for very large libraries.
- If no matching template is found and `on_template_failure: warn`, CADDForge falls back to direct
  RDKit hydrogen addition and records that fallback in the manifest.

Example:
```yaml
- id: repair_gnina_pose_hydrogens
  type: molecules.hydrogenate
  inputs:
    molecules: extract_gnina_poses.poses
    templates: retain_unidock_poses.molecules
  config:
    ligand_column: ligand
    on_template_failure: warn
    on_failure: fail

- id: openbpmd
  tool: md/openbpmd
  config: openbpmd.yaml
  inputs:
    ligand_path: repair_gnina_pose_hydrogens.molecules
  scatter:
    by: ligand_path
    chunk_size: 1
```

#### `results.collect`

Collects JSON result files from a scattered stage into one CSV table and joins each result back to
the scattered input provenance. Use this after per-pose tools such as OpenBPMD when you need to
rank or filter poses by a result metric.

Required inputs:
- `results`: a result artifact from a scattered tool stage, for example `openbpmd.results`.

Optional inputs:
- `table`: provenance CSV. If omitted, CADDForge uses the scattered stage's automatically generated
  `scatter_manifest.csv`.

Output artifacts:
- `table`: collected CSV table. Downstream reference: `collect_openbpmd.table`.

Config keys:
- `metrics`: mapping of output CSV column names to JSON paths inside each result record. For a
  standard CADDForge `Result`, use paths such as `metrics.comp_score`.
- `output_name`: output filename; defaults to `collected_results.csv`.

Behavior:
- Rows are matched to scattered inputs by shard order. For path-collection scatter, the automatic
  provenance includes pose metadata from upstream manifests, commonly `ligand`, `pose_id`,
  `pose_rank`, and `pose_file`.
- If a tool writes both a human-readable `results.json` and a canonical `result_record.json` in the
  same results directory, CADDForge reads the canonical record for metric extraction.

Example: keep the most stable OpenBPMD pose per ligand, where lower `comp_score` is better:
```yaml
- id: openbpmd
  tool: md/openbpmd
  config: openbpmd.yaml
  inputs:
    ligand_path: repair_gnina_pose_hydrogens.molecules
  scatter:
    by: ligand_path
    chunk_size: 1
  outputs:
    results: results_json

- id: collect_openbpmd
  type: results.collect
  inputs:
    results: openbpmd.results
  config:
    metrics:
      comp_score: metrics.comp_score

- id: most_stable_pose_per_ligand
  type: filter.tabular
  inputs:
    table: collect_openbpmd.table
  config:
    sort_by: comp_score
    ascending: true
    group_by: ligand
    top_n_per_group: 1
    output_name: most_stable_openbpmd_poses.csv

- id: retain_stable_openbpmd_poses
  type: filter.molecules
  inputs:
    molecules: repair_gnina_pose_hydrogens.molecules
    ids: most_stable_pose_per_ligand.filtered_table
  config:
    id_column: pose_id
    path_column: output_file
```

#### `filter.molecules`

Retains molecule or pose files whose identifiers occur in a filtered table.

Required inputs:
- `molecules`: a molecule/pose artifact. This can be a file, a directory, or a multi-path artifact.
- `ids`: a CSV artifact containing identifiers to retain, usually from `filter.tabular`.

Output artifacts:
- `molecules`: the retained molecule/pose artifact. Downstream reference:
  `retain_unidock_poses.molecules`.

Config keys:
- `id_column`: column in the `ids` table containing identifiers. Defaults to `ligand_id`.
- `path_column`: optional column containing exact molecule/pose file paths. If omitted,
  CADDForge auto-detects common columns such as `pose_file`, `molecule_file`, `ligand_file`,
  `path`, or `file`.
- `output_name`: output filename for SDF filtering; defaults to `filtered_molecules.sdf`.

Behavior:
- If a path column is present, CADDForge uses those exact files first. For SDF files, it filters
  records inside the files and writes a filtered SDF. This is the preferred mode for sharded
  docking outputs where each shard has its own pose directory.
- If no path column is available and `molecules` points to a directory, files are retained when an
  identifier occurs in the file stem. The output artifact contains the retained file paths.
- If `molecules` points to an SDF-like file, records are retained when an identifier occurs in the
  record text. The output artifact is a filtered SDF file.
- If the downstream config field is typed as `list[Path]`, CADDForge passes retained paths as a
  Python list. Gnina uses this through `ligand_paths`.
- Otherwise, CADDForge passes the artifact's primary path.

Example:
```yaml
- id: retain_unidock_poses
  type: filter.molecules
  inputs:
    molecules: unidock_screen.poses
    ids: top_score_filter.filtered_table
  config:
    id_column: ligand

- id: gnina_rescore
  tool: docking/gnina
  config: gnina.yaml
  inputs:
    ligand_paths: retain_unidock_poses.molecules
```

Here, `retain_unidock_poses.molecules` exists because `filter.molecules` always emits an artifact
named `molecules`.

To add a downstream workflow, append another stage. The input key must be a real config field on
that tool, or an input port declared by the plugin:
```yaml
  - id: downstream_analysis
    tool: my-section/my-downstream-tool
    config: downstream.yaml
    inputs:
      score_table_path: top_score_filter.filtered_table
    outputs:
      report: results_json
```

For Slurm, set the pipeline default backend and normal Slurm options:
```yaml
defaults:
  backend: slurm
  registry_path: runs/job_registry.sqlite
```

Then pass optional CLI overrides as needed:
```bash
caddforge pipeline run --config pipeline.yaml \
  --slurm-partition short \
  --slurm-time 02:00:00 \
  --sbatch-option=--gres=gpu:1
```

Pipeline runs create a parent pipeline job under `runs/YYYY-MM-DD/<pipeline-id>/`. Child stage jobs,
shards, reduced outputs, and `artifact_manifest.json` are stored under that parent run directory.

### Pipeline Python API

The same pipeline engine can be used directly from Python. Build a `PipelineSpec`, then pass it to
`PipelineExecutor`:
```python
from pathlib import Path

from caddforge.pipeline import PipelineExecutor, PipelineSpec

spec = PipelineSpec.model_validate(
    {
        "version": 1,
        "name": "docking-screen",
        "defaults": {
            "backend": "local",
            "registry_path": Path("runs/job_registry.sqlite"),
            "job_name": "unidock-screen-test",
        },
        "stages": [
            {
                "id": "unidock_screen",
                "tool": "docking/unidock",
                "config": "unidock.yaml",
                "scatter": {
                    "by": "ligand_multisdf_path",
                    "chunk_size": 50000,
                },
                "outputs": {
                    "scores": "best_scores_csv",
                    "poses": "poses_dir",
                },
            },
            {
                "id": "top_score_filter",
                "type": "filter.tabular",
                "inputs": {
                    "table": "unidock_screen.scores",
                },
                "config": {
                    "sort_by": "best_score",
                    "ascending": True,
                    "top_n": 1000,
                    "output_name": "filtered_scores.csv",
                },
            },
            {
                "id": "retain_unidock_poses",
                "type": "filter.molecules",
                "inputs": {
                    "molecules": "unidock_screen.poses",
                    "ids": "top_score_filter.filtered_table",
                },
                "config": {
                    "id_column": "ligand",
                },
            },
            {
                "id": "gnina_rescore",
                "tool": "docking/gnina",
                "config": "gnina.yaml",
                "inputs": {
                    "ligand_paths": "retain_unidock_poses.molecules",
                },
                "outputs": {
                    "scores": "best_scores_csv",
                    "poses": "poses_dir",
                },
            },
        ],
    }
)

result = PipelineExecutor(spec).run()

print(result.pipeline_job.id)
print(result.pipeline_job.run_directory)
print(result.artifacts["top_score_filter.filtered_table"].primary_path())
```

If you already have a YAML/JSON pipeline file, use `run_pipeline`:
```python
from caddforge.pipeline import run_pipeline

result = run_pipeline("pipeline.yaml")
print(result.pipeline_job.status.value)
```

---

## Core Concepts

### Dataset & MoleculeSet
- `Dataset` represents a reproducible, labeled collection of data (molecules + labels).
- `MoleculeSet` represents an immutable set of molecules.

### Workflow
A `Workflow` orchestrates a protocol. It is backend-agnostic and delegates execution to a
`Backend`.

### Job
A `Job` is the canonical record of a workflow execution. It captures:
- identity (`job_id`), timestamps, and status
- run directory and metadata
- backend used

### Backend
A `Backend` executes workflows. Core provides:
- `LocalBackend` (in-process execution)
- `SlurmBackend` (cluster submission via `sbatch`)

### Result
A `Result` is a standardized, serializable outcome (status, metrics, paths, extra payloads).

### Scratch synchronization
`LogSync` periodically mirrors live log files from node-local scratch without allowing transient
copy failures to interrupt a workflow. `ArtifactSyncQueue` asynchronously publishes completed,
immutable files through atomic replacement, retries unresolved copies on a fixed interval, and
returns any files still pending after its final copy pass.

---

## Configuration

All tools use Pydantic config models derived from `BaseConfig`. Configs can be loaded from YAML
or JSON and are validated on load.

Example YAML (tool-specific fields omitted):
```yaml
backend: local
run_directory: null
registry_path: runs/job_registry.sqlite
```

Base config also supports Slurm execution:
```yaml
backend: slurm
slurm_partition: short
slurm_time: 01:00:00
slurm_account: my-account
sbatch_options:
  - --gres=gpu:1
slurm_nodes: 1
slurm_ntasks: 1
slurm_cpus_per_task: 1
```

---

## CLI Reference

Open the documentation shipped by the optional `caddforge-docs` package:

```bash
python -m pip install caddforge-docs
caddforge docs
caddforge docs --host 0.0.0.0 --port 8001 --no-browser
```

This command appears only when the docs package is installed. It opens a browser and serves
until Ctrl+C, using `127.0.0.1` and an automatically assigned port by default. `--host` supports
hostnames, IPv4, and IPv6; `--no-browser` suppresses automatic opening. Documentation rendering
tools are needed only to build the docs package.

List tools grouped by section:
```bash
caddforge list-tools
```

Show config template:
```bash
caddforge show-config <section> <tool>
```

Run a workflow:
```bash
caddforge run <section> <tool> --config path/to/config.yaml
```

Backend test:
```bash
caddforge backend-check --backend slurm --slurm-partition short
```

Status, logs, and job listing:
```bash
caddforge status <job-id>
caddforge logs <job-id>
caddforge logs <job-id> --follow
caddforge list-jobs --status running --status pending
```

Follow mode prints newly synchronized `stdout.log` content until the job reaches a terminal state.
Transient log-file or job-registry read failures are reported and retried, which is useful when run
directories reside on an intermittently available shared filesystem.

Export results manually (copies `results/` back to the submission directory):
```bash
caddforge export <job-id>
```

---

## Job Layout & Registry

Run directories follow a standardized layout:
```text
runs/
  YYYY-MM-DD/
    <job-id>/
      config.yaml
      metadata.json
      stdout.log
      stderr.log
      results/
        results.json
        tool_specific/...
```

A SQLite registry tracks jobs and powers `status`/`logs`/`list-jobs`:
```
runs/job_registry.sqlite
```

By default, CADDForge leaves workflow outputs in the run directory. To also copy the
`results/` directory back to the submission directory automatically when a job finishes, opt in
from the workflow config:
```yaml
export_results: true
```

Manual export remains available through `caddforge export <job-id>` even when automatic export is
disabled.

The registry uses short SQLite transactions with bounded retries for transient locking and I/O
errors. For large Slurm fan-out runs, prefer placing `registry_path` on POSIX storage with working
advisory locks. CIFS/SMB/NFS mounts with unreliable locking can leave the registry temporarily
stale under high concurrency; per-job `metadata.json` and `results/results.json` remain the
fallback source of truth. Do not use SQLite WAL mode for a registry shared across network-mounted
hosts.

---

## Backends

### LocalBackend
- Executes workflows in-process.
- Captures stdout/stderr to log files.
- Saves `results.json` and updates metadata.

### SlurmBackend
- Writes a payload and submits an `sbatch` script.
- Tracks status using `sacct` and `squeue`.
- Uses `slurm_runner.py` inside the job to execute the workflow callable.
- Requires the compute node to have access to a compatible Python environment; the login-node
  environment is usually not enough by itself.

Slurm options can be provided in config or via CLI flags:
- `slurm_partition`
- `slurm_time`
- `slurm_account`
- `sbatch_options`
- `slurm_setup_commands`
- `slurm_env_pack_path` / `slurm_wheelhouse_path` / `slurm_requirements_path`
- `slurm_env_cache_enabled` / `slurm_env_cache_dir`
- `slurm_nodes`
- `slurm_ntasks`
- `slurm_cpus_per_task`

#### Sharing the Python environment with Slurm
Slurm jobs execute on compute nodes, not in the interactive shell where you launched
`caddforge`. In most cluster setups the compute node either cannot see the login-node virtualenv or
cannot safely reuse it. For reproducible Slurm runs, provide **one** environment strategy in the
workflow config.

Use `slurm_env_pack_path` when you want one prebuilt, relocatable environment. This is the preferred
strategy for heavy scientific stacks because dependency resolution happens once, before submission.
Use `slurm_wheelhouse_path` plus `slurm_requirements_path` when you want the Slurm job to create a
fresh per-job virtualenv from a shared wheelhouse.

**Option A: conda-pack / micromamba pack (single tarball)**
1. Create an environment that contains:
   - `caddforge-core`
   - the tool plugin(s) you want to run
   - all scientific dependencies needed by those tools
   Install conda-managed scientific packages first, then install CADDForge packages with pip.
2. Use the core `pack-env` command to create the archive and managed SHA-256 sidecar. `conda-pack`
   is installed as a `caddforge-core` dependency:
   ```bash
   # Example environment creation with micromamba
   micromamba create -p /scratch/$USER/envs/caddforge -c conda-forge python=3.10
   micromamba install -p /scratch/$USER/envs/caddforge -c conda-forge openmm rdkit
   micromamba run -p /scratch/$USER/envs/caddforge python -m pip install caddforge-core <tool-packages>

   # Pack an explicit prefix
   caddforge pack-env --prefix /scratch/$USER/envs/caddforge \
     --output /shared/envs/caddforge.tar.gz

   # Or pack the active environment
   caddforge pack-env --output /shared/envs/caddforge-dev.tar.gz

   # Rebuild a fixed-name development pack
   caddforge pack-env --output /shared/envs/caddforge-dev.tar.gz --force

   # Pack an intentionally modified environment with missing Conda-managed files
   caddforge pack-env --output /shared/envs/caddforge-dev.tar.gz \
     --ignore-missing-files
   ```
   A named environment can be selected with `--name`. The command preserves `conda-pack`'s progress
   bar as a single updating terminal line, streams normal output, reports quiet-period heartbeats
   with the growing archive size plus hashing progress, and writes
   `<archive>.sha256` automatically.
3. Point CADDForge to the tarball:
   ```yaml
   backend: slurm
   slurm_env_pack_path: /shared/envs/caddforge.tar.gz
   ```
   By default, CADDForge caches the unpacked environment in node-local scratch and reuses it for
   later jobs on that node. Set `slurm_env_cache_enabled: false` to retain job-private unpacking.

   The default per-user cache root is
   `$SCRATCH_BASE/caddforge_environments/$USER`, where `SCRATCH_BASE` is the same node-local root
   selected for Slurm job scratch. Sites with a different layout can provide an absolute
   compute-node path with `slurm_env_cache_dir`.

**Option B: wheelhouse + per-job venv**
1. Build a wheelhouse on a machine with internet access or your internal package index:
   ```bash
   # In a clean venv that has your deps installed:
   python -m pip freeze > requirements.txt
   python -m pip download -r requirements.txt -d /shared/wheelhouse
   ```
2. Provide the wheelhouse and requirements file:
   ```yaml
   backend: slurm
   slurm_wheelhouse_path: /shared/wheelhouse
   slurm_requirements_path: /shared/requirements.txt
   ```
   The Slurm job will create a temporary venv, install from the wheelhouse,
   and execute the workflow with that Python.

**Notes**
- `slurm_env_pack_path` is mutually exclusive with `slurm_wheelhouse_path` /
  `slurm_requirements_path`.
- If you use `slurm_setup_commands`, do not set env-pack or wheelhouse options; those commands are
  then responsible for loading or creating the Python environment.
- CADDForge determines the environment-pack SHA-256 before submission. Cache entries include the
  archive name and digest. When `<archive>.sha256` exists and is valid, submission reads that small
  sidecar instead of hashing the archive. Missing or invalid sidecars fall back to full hashing with
  a warning, preserving compatibility with manually created packs.
- Only the job installing a missing cache entry copies and verifies the archive; cache hits do not
  reread it. With `slurm_env_cache_enabled: false`, CADDForge skips sidecar lookup and submission
  hashing and restores the original job-private unpacking behavior.
- `pack-env --force` prepares and hashes the replacement before acquiring the publication lock, so
  submissions may continue using the previous valid pair while packing runs. Replacing an archive
  after a job was submitted can make an uncached queued job fail checksum verification.
- `pack-env --ignore-missing-files` passes the corresponding option to `conda-pack`. Use it only for
  intentionally modified environments: files recorded in Conda package metadata but missing from
  the source environment are omitted, so the resulting archive may be incomplete.
- The packaging lock is a separate hidden advisory file next to the archive. It never locks or
  modifies the source archive itself. Direct external archive replacement bypasses sidecar
  coordination and is therefore unsupported for managed packs.
- Cache locks are advisory files under the node-local cache root, never beside or on the source
  archive. Jobs and other compute nodes can continue reading the original pack concurrently.
- Installation uses a temporary directory and an atomic move. Active jobs retain a shared lock so
  opportunistic cleanup cannot remove their environment.
- Cache entries idle for more than 14 days are removed opportunistically. The retention period and
  30-minute lock timeout are internal core policies rather than per-job configuration.
- Node-local caches may disappear during cluster cleanup or node replacement; CADDForge rebuilds a
  missing entry from the source pack.
- Ensure the shared paths are readable from compute nodes.

---

## Plugin Architecture

Installed packages can also register optional top-level CLI commands in the
`caddforge.cli_commands` entry-point group. The entry-point name becomes the command name and
its target is a callable with a Typer-compatible signature and docstring. Core discovers these
after registering built-in commands. Conflicting names, duplicate extensions, and broken imports
or signatures are logged and skipped. This interface is separate from scientific tool discovery;
command providers should keep imports lightweight and load expensive dependencies on invocation.

Tools register themselves as plugins via entry points:
```
[project.entry-points."caddforge.tools"]
ani-psa = "cadd_descriptors.tool:DescriptorsTool"
```

During development you can skip installation by setting:
```bash
export CADD_FORGE_DEV_PLUGINS=cadd_descriptors.tool:DescriptorsTool
```

The CLI will then discover tools on `PYTHONPATH`.

---

## Development & Testing

Run tests:
```bash
pytest
```

Formatting/linting (if configured in your environment):
```bash
black .
ruff .
```

---

## License

This project is intended for open-source use. Add your preferred license text here.

## Continuous integration and releases

`.github/workflows/release.yml` builds a wheel and source archive, validates
metadata, and tests the installed wheel on Linux with Python 3.10 and 3.12.
The lightweight test selection is recorded in `scripts/ci_tests.json`;
external scientific engines are validated separately. Core CI installs the
wheel with its `[test]` extra, including RDKit for molecular hydrogenation
coverage. RDKit remains optional for runtime installations. CLI help tests
check both plain and styled output.

Pushing a matching `v*` release tag publishes the tested artifacts to PyPI.
Branch pushes, pull requests and manual runs never publish. Configure a
PyPI Trusted Publisher for the `idorsia` repository, `release.yml` workflow
and `pypi` environment before pushing release tags. Core is published before the plugins; its runtime dependencies are resolved
from public PyPI.
