Metadata-Version: 2.4
Name: nexus-snakemake-utils
Version: 0.2.0
Summary: Shared workflow infrastructure and image-cache commands for Snakemake 8.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: snakemake>=8.14
Requires-Dist: pandas<3,>=2
Requires-Dist: matplotlib<4,>=3.7
Requires-Dist: Jinja2<4,>=3.1
Requires-Dist: PyYAML<7,>=6
Requires-Dist: dpath<3,>=2

# Nexus Snakemake utilities

Shared workflow infrastructure for Python 3.11+ on Linux/macOS, targeting
Snakemake 8.14. Newer Snakemake versions are allowed but remain unverified.
Install into the Conda environment that runs Snakemake:

```bash
python -m pip install 'nexus-snakemake-utils @ git+https://github.com/ETH-NEXUS/nexus_snakemake_utils.git@v0.2.0'
# Local development, from a neighboring workflow checkout:
python -m pip install -e ../nexus_snakemake_utils
```

Publish the `v0.2.0` tag before using the pinned installation command. Earlier
tags contain the old package. Until the tag is available remotely, use the local
editable installation. Packaging declares the
controller dependencies. This does not install analysis or R dependencies.

## Portability

Nexus/Leomed access and Slurm are not required. The supported environment is
Linux/macOS with Python 3.11+ and Snakemake 8.14 as the tested target. Packaging
requires `snakemake>=8.14` without an upper bound; this permits Snakemake 9 but
does not establish compatibility. Validation of newer versions is deferred.
Core utilities and benchmark reports work with the local executor. Slurm
accounting is queried only for Slurm runs; missing accounting tools or records
produce warnings without failing the workflow. Container usage registration
skips runtimes without the proxy interface.

The optional container proxy uses Unix file locking and requires a real
Singularity/Apptainer runtime for container execution. Its `singularity` launcher
is installed with this package and must find the real runtime later on PATH or
through `SNAKEMAKE_REAL_SINGULARITY`. Set an image directory with `--image-dir`
or `LEOMED_SNAKEMAKE_IMAGE_DIR` when using its image-cache commands. The variable
name does not require Leomed storage; mirror mappings are optional. Keep cluster
paths, accounts and binds in site profiles, and publish a local profile too.

Reporting libraries are installed as package dependencies even if reporting is
unused. Analysis tools and report-rendering software remain workflow dependencies.

## Workflow interface

```python
import nexus_snakemake_utils as utils
# Top level of the Snakefile: runs during parsing, including dry runs.
# Preserve an explicit container workflow label, otherwise use the repository
# name; then log time, user, host, Git revision/checkout state and Snakemake version.
utils.initialize(repository_dir)

onstart:
    utils.register_container_usage(workflow)
onsuccess:
    utils.write_resource_report(workflow)
onerror:
    utils.write_resource_report(workflow, partial=True)
```

`initialize` preserves an explicit `SNAKEMAKE_CONTAINER_WORKFLOW` label and
otherwise sets it to the repository name, then calls `log_run_info`. It does not
register container usage, execute jobs, or write resource reports. Keep it at the
top level of the Snakefile so logging also runs for dry runs.

Select optional inputs or targets with `optional(val, on, dpath=None)`:

```python
rule all:
    input:
        utils.optional(rules.report.output.html, config, dpath="optional/report"),
```

This returns `val` unchanged when the flag is truthy, otherwise `[]`. Omit `dpath`
to pass a flag directly, for example `utils.optional(targets, enabled)`. Config
paths use dpath syntax; missing or ambiguous paths raise errors. Define flags
explicitly in the base configuration (for example `optional: {report: false}`)
and enable them through overlays. Values are evaluated before calling the helper;
use lazy input functions when resolving a value itself requires optional files.

Use callable factories directly in a rule's `resources` block:

```python
resources:
    mem_mb=utils.memory_by_input(
        base_mem_mb=2000,
        mem_per_input_mb=2,
        maximum=lookup(dpath="resources/max_mem_mb", within=config),
    ),
    runtime=utils.runtime_by_input(
        base_minutes=5,
        minutes_per_gib=1,
        maximum=lookup(dpath="resources/max_minutes", within=config),
    ),
```

Both factories require an explicit maximum. Keep limits in workflow configuration
and use `lookup` without defaults so missing required settings fail clearly.
The factory captures the coefficients and limit during parsing, then returns a
function accepting `wildcards` and `input`. Snakemake evaluates that function when
the job's inputs are available; missing inputs remain `<TBD>` during dry runs.
Memory is `ceil(base_mem_mb + mem_per_input_mb * input.size_mb)`; runtime is
`ceil(base_minutes + minutes_per_gib * input.size_mb / 1024)`, each capped at its
maximum. Estimates apply to the whole job, not per thread.

Snakemake 8.14 resource functions receive no `params` or other resource values.
Keep resource coefficients here or in workflow configuration, rather than
introducing artificial scheduler resources. The lower-level `dynamic_mem_mb` and
`dynamic_runtime` functions remain available for direct numeric estimates.
Metadata loading and scientific validation stay in the workflow.

`external_path(value, label, repository_dir)` resolves a path against the current
run directory and rejects the repository itself and its descendants, including
paths that resolve into it through symlinks. It returns an absolute `Path` without
requiring that the file exists. Pass the repository boundary explicitly:

```python
sample_sheet = utils.external_path(
    config["sample_sheet"], "Sample sheet", repository_dir=repository_dir
)
```

Enable `benchmark-extended: true` in execution profiles to record job metadata
and historical input sizes. The resource report uses those values when available,
compares requested and measured resources, and exports `measurements.tsv`;
ordinary benchmarks remain supported. `nexus-resource-report --help` describes
standalone inspection of existing benchmark outputs. See
[resource reporting](docs/resource_report.md). The success and failure hooks
retain existing outputs and accounting semantics; missing site
services are reported as warnings. All Snakemake private API access belongs in
`hooks/resource_report.py`.

## Singularity image proxy

Installation exposes `singularity` in the environment. Ensure that environment's
`bin` precedes the real runtime on PATH. Discovery skips this launcher; set
`SNAKEMAKE_REAL_SINGULARITY=/absolute/path/to/real/singularity` when needed. Do not
point it at the proxy. Native Apptainer can be supplied through this variable too.

```bash
export LEOMED_SNAKEMAKE_IMAGE_DIR=/path/to/shared/images
export SNAKEMAKE_CONTAINER_PROXY_CONFIG=/path/to/site/container-proxies.json
singularity list
singularity list '*samtools*'
singularity locate docker://python:3.14
singularity pull docker://python:3.14
singularity exec docker://python:3.14 python --version
singularity --image-dir /another/cache list
```

Use the same variable in a Snakemake profile:

```yaml
apptainer-prefix: "$LEOMED_SNAKEMAKE_IMAGE_DIR"
```

An explicit `--image-dir` before the command or among its leading options wins
over the environment. Listing and URL resolution require a location; local image
paths and unrelated native commands do not. Quote URL globs to prevent shell
expansion. `list` prints downloaded URLs, paths and usage metadata as TSV, omitting
missing files and showing unmapped `.simg` files with URL `unknown`. It does not
modify the manifest. `locate` prints only an existing absolute path.

`exec`, `run` and `shell` resolve the image operand only. URLs in runtime option
values or in the container's arguments are preserved. Runtime help supplies option
arity, supporting the installed runtime's flags. Local paths and `instance://`
references pass through unchanged. Missing URL images fail with an explicit pull
command; execution never downloads implicitly. Usage denotes an execution attempt,
not successful completion. Mutable tags reuse their existing image until an
explicit forced pull.

Unnamed pulls write `<md5(exact-original-url)>.simg` under the selected directory.
Explicit filenames and Snakemake's `pull --name ...` retain their original output
location. Mirror rewriting never changes the hash. Keep URLs exactly consistent;
registry aliases are not normalized. Use an explicit destination for sandbox pulls.
The proxy records file-backed images in `container_list.tsv`, retaining the existing
schema, per-image locks, atomic manifest updates and per-workflow usage timestamps.

Proxy mappings are an optional JSON object of source prefixes to replacements.
Longest prefix wins. The generic `docker://` mapping applies to Docker Hub while
preserving explicit other registries. Without configured mappings, URLs pass
through unchanged. Keep site addresses in site configuration, outside this package.

Native commands, including `cache list`, pass through. This workflow image store
is separate from the runtime's layer cache. Existing `--proxy-capabilities`,
`--proxy-register`, `--proxy-real-runtime` and `--proxy-audit DIRECTORY` interfaces
remain supported. Audit refreshes presence information and discovers unmapped
images; it does not recover their original URLs.

## Maintenance and validation

Use Black and Flake8 (88 columns, ignoring E203 for Black compatibility). Keep
Snakemake 8.14 compatibility and private APIs confined to the resource-report
module. No automatic test suite is introduced. Verify changed branches with
focused local execution and temporary caches; never use a production image cache
for development checks.

On Leomed, manually verify dry runs, prefetching, cache listing and a known URL
execution against the configured cache. Check successful and failed resource
reports and repeat delayed accounting inspection using existing outputs. Suggest
these checks without executing cluster jobs automatically. Publish packages,
Git tags and site updates separately; no push is part of local maintenance.
