Metadata-Version: 2.4
Name: nexus-snakemake-utils
Version: 0.3.0
Summary: Shared workflow infrastructure and image-cache commands for Snakemake 8.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: snakemake>=8.14
Requires-Dist: pandas<3,>=2
Requires-Dist: matplotlib<4,>=3.7
Requires-Dist: Jinja2<4,>=3.1
Requires-Dist: PyYAML<7,>=6
Requires-Dist: dpath<3,>=2

# Nexus Snakemake utilities

Shared workflow infrastructure for Python 3.11+ on Linux/macOS, targeting
Snakemake 8.14. Newer Snakemake versions are allowed but remain unverified.
Install the [PyPI release](https://pypi.org/project/nexus-snakemake-utils/0.2.0/)
into the Python/Conda environment that runs Snakemake:

```bash
python -m pip install "nexus-snakemake-utils==0.2.0"
```

For an existing conda environment, installation also works without activation:

```bash
snakemake_env_dir="/absolute/path/to/snakemake_v8.14.0"
conda run --prefix "$snakemake_env_dir" python -m pip install \
  "nexus-snakemake-utils==0.2.0" "snakemake==8.14.0"
conda run --prefix "$snakemake_env_dir" python -m pip check
```

Include `pip` when creating the conda environment; install the conda packages
first and the utils afterwards. The explicit Snakemake constraint in this example
retains the tested version; the utils package itself allows Snakemake >=8.14.

To install directly from the GitHub release tag instead:

```bash
python -m pip install 'nexus-snakemake-utils @ git+https://github.com/ETH-NEXUS/nexus_snakemake_utils.git@v0.2.0'
```

The package installs controller dependencies. Analysis tools and R dependencies
must be supplied separately by your workflow.

## Portability

Nexus/Leomed access and Slurm are not required. The supported environment is
Linux/macOS with Python 3.11+ and Snakemake 8.14 as the tested target. Packaging
requires `snakemake>=8.14` without an upper bound; this permits Snakemake 9 but
does not establish compatibility. Validation of newer versions is deferred.
Core utilities and benchmark reports work with the local executor. Slurm
accounting is queried only for Slurm runs; missing accounting tools or records
produce warnings without failing the workflow. Container usage registration
skips runtimes without the proxy interface.

The optional container proxy uses Unix file locking and requires a real
Singularity/Apptainer runtime for container execution. Its `singularity` launcher
is installed with this package and must find the real runtime later on PATH or
through `SNAKEMAKE_REAL_SINGULARITY`. Set an image directory with `--image-dir`
or `LEOMED_SNAKEMAKE_IMAGE_DIR` when using its image-cache commands. The variable
name does not require Leomed storage; mirror mappings are optional. Keep cluster
paths, accounts and binds in site profiles, and publish a local profile too.

Reporting libraries are installed as package dependencies even if reporting is
unused. Analysis tools and report-rendering software remain workflow dependencies.

## Workflow interface

```python
import nexus_snakemake_utils as utils
# Top level of the Snakefile: runs during parsing, including dry runs.
# Preserve an explicit container workflow label, otherwise use the repository
# name; then log time, user, host, Git revision/checkout state and Snakemake version.
utils.initialize(repository_dir)

onstart:
    utils.register_container_usage(workflow)
onsuccess:
    utils.write_resource_report(workflow)
onerror:
    utils.write_resource_report(workflow, partial=True)
```

`initialize` preserves an explicit `SNAKEMAKE_CONTAINER_WORKFLOW` label and
otherwise sets it to the repository name, then calls `log_run_info`. It does not
register container usage, execute jobs, or write resource reports. Keep it at the
top level of the Snakefile so logging also runs for dry runs.
Parsing-time messages are buffered until Snakemake opens its execution log, so
provenance appears once in both the terminal and `.snakemake/log/`. Dry runs
retain terminal output without creating an execution log.

Select optional inputs or targets with `optional(val, on, dpath=None)`:

```python
rule all:
    input:
        utils.optional(rules.report.output.html, config, dpath="optional/report"),
```

This returns `val` unchanged when the flag is truthy, otherwise `[]`. Omit `dpath`
to pass a flag directly, for example `utils.optional(targets, enabled)`. Config
paths use dpath syntax; missing or ambiguous paths raise errors. Define flags
explicitly in the base configuration (for example `optional: {report: false}`)
and enable them through overlays. Values are evaluated before calling the helper;
use lazy input functions when resolving a value itself requires optional files.

Derive a directory from a resolved output without a lambda:

```python
params:
    outdir=utils.dirname("events", index=0),  # output.events[0]
```

`dirname("quant")` selects a single named output; `dirname()` uses `output[0]`.
It preserves `os.path.dirname` semantics, including `""` for a bare filename.

Plain `lookup(...)` runs while loading rules unless its query/path needs
wildcards. Use `lazylookup` in `input` or `params` so unused rules do not require
their settings:

```python
params:
    extra=utils.lazylookup(dpath="rule_specific/tool/extra", within=config),
```

It accepts Snakemake's `lookup` arguments, including defaults and wildcard paths.
Missing required keys still fail when the job is evaluated, including during dry
runs. `transform=shlex.split` can convert an option string after lookup. Arguments
are still evaluated when calling the factory: pass `within=config`, not an eager
lookup of an optional subsection. Use ordinary `lookup` for load-time setup,
output paths and Python conditions; a callable itself is always truthy.

Pass a configuration branch to a script as JSON and extend it with other settings:

```python
params:
    report_config=utils.json_dumps(
        within=config,
        base="rule_specific/de_analysis",
        tx2gene="resources/tx2gene",
        gene_info="resources/gene_info",
    ),
```

`json_dumps` returns a callable. It copies the base mapping, applies output-path →
input-path mappings in keyword order, then serializes the result. Omit `base` to
start with `{}`. Additional mappings override existing values; nested output
paths use `**{"references/tx2gene": "resources/tx2gene"}`. Source paths support
Snakemake wildcards. All lookups happen at job evaluation; missing paths and a
non-mapping base raise errors. The original configuration is never modified.

Register system limits once in `workflow/rules/misc.smk`:

```python
limits = utils.ResourceLimits(
    memory_mb=lookup(dpath="resources/max_mem_mb", within=config),
    runtime_minutes=lookup(dpath="resources/max_minutes", within=config),
)
```

Both limits must be positive integers. Each instance holds its own immutable
limits. Use its callable factories in each rule's `resources` block:

```python
resources:
    mem_mb=limits.memory_by_input(
        base_mem_mb=2000,
        mem_per_input_mb=2,
    ),
    runtime=limits.runtime_by_input(
        base_minutes=5,
        minutes_per_gib=1,
    ),
```

Keep system limits in configuration and use `lookup` without defaults so missing
required settings fail clearly. These limits cap estimates made through the
object; fixed resource requests in rules and scheduler limits are separate.
The standalone `utils.memory_by_input` and `utils.runtime_by_input` factories
remain available with an explicit `maximum`.
The factory captures the coefficients and limit during parsing, then returns a
function accepting `wildcards` and `input`. Snakemake evaluates that function when
the job's inputs are available; missing inputs remain `<TBD>` during dry runs.
Memory is `ceil(base_mem_mb + mem_per_input_mb * input.size_mb)`; runtime is
`ceil(base_minutes + minutes_per_gib * input.size_mb / 1024)`, each capped at its
maximum. Estimates apply to the whole job, not per thread.

Snakemake 8.14 resource functions receive no `params` or other resource values.
Keep resource coefficients here or in workflow configuration, rather than
introducing artificial scheduler resources. The lower-level `dynamic_mem_mb` and
`dynamic_runtime` functions remain available for direct numeric estimates.
Metadata loading and scientific validation stay in the workflow.

`external_path(value, label, repository_dir)` resolves a path against the current
run directory and rejects the repository itself and its descendants, including
paths that resolve into it through symlinks. It returns an absolute `Path` without
requiring that the file exists. Pass the repository boundary explicitly:

```python
sample_sheet = utils.external_path(
    config["sample_sheet"], "Sample sheet", repository_dir=repository_dir
)
```

Enable `benchmark-extended: true` in execution profiles to record job metadata
and historical input sizes. The resource report uses those values when available,
compares requested and measured resources, and exports `measurements.tsv`;
ordinary benchmarks remain supported. `nexus-resource-report --help` describes
standalone inspection of existing benchmark outputs. See
[resource reporting](https://github.com/ETH-NEXUS/nexus_snakemake_utils/blob/main/docs/resource_report.md). The success and failure hooks
retain existing outputs and accounting semantics; missing site
services are reported as warnings.

## Singularity image proxy

Installation exposes `singularity` in the environment. Ensure that environment's
`bin` precedes the real runtime on PATH. Discovery skips this launcher; set
`SNAKEMAKE_REAL_SINGULARITY=/absolute/path/to/real/singularity` when needed. Do not
point it at the proxy. Native Apptainer can be supplied through this variable too.

```bash
export LEOMED_SNAKEMAKE_IMAGE_DIR=/path/to/shared/images
export SNAKEMAKE_CONTAINER_PROXY_CONFIG=/path/to/site/container-proxies.json
singularity list
singularity list '*samtools*'
singularity locate docker://python:3.14
singularity rm 'docker://python:3.14*'
singularity pull docker://python:3.14
singularity exec docker://python:3.14 python --version
singularity --image-dir /another/cache list
```

To persist the proxy configuration path for a Conda environment:

```bash
conda env config vars set --prefix /path/to/env \
  SNAKEMAKE_CONTAINER_PROXY_CONFIG=/path/to/site/container-proxies.json
```

Reactivate the environment afterward. Image pulls warn on stderr when the proxy
configuration variable is unset or its JSON mapping is empty; direct pulls remain
allowed. Version checks and other native commands stay quiet.

Use the same image directory variable in a Snakemake profile:

```yaml
apptainer-prefix: "$LEOMED_SNAKEMAKE_IMAGE_DIR"
```

An explicit `--image-dir` before the command or among its leading options wins
over the environment. Listing and URL resolution require a location; local image
paths and unrelated native commands do not. Quote URL globs to prevent shell
expansion. `list` prints downloaded URLs, paths and usage metadata as TSV, omitting
missing files and showing unmapped `.simg` files with URL `unknown`. It does not
modify the manifest. `locate` prints only an existing absolute path.

`rm URL-GLOB` removes cached images matching the same original URL patterns as
`list`, using the same per-image locks as pulls. Quote patterns, for example
`singularity rm '*python*'`. An explicit URL or pattern is required; no matches
return an error. Inventory history is retained and deleted images are marked
missing. As with `list`, `'*'` includes images whose URL is `unknown`. Remove shared images only when no jobs are using them. This command does
not clean the runtime's separate layer cache. Other native commands, including
`cache`, `instance` and `help`, continue to pass through unchanged; `rm` inside
container commands is also unaffected.

`exec`, `run` and `shell` resolve the image operand only. URLs in runtime option
values or in the container's arguments are preserved. Runtime help supplies option
arity, supporting the installed runtime's flags. Local paths and `instance://`
references pass through unchanged. Missing URL images fail with an explicit pull
command; execution never downloads implicitly. Usage denotes an execution attempt,
not successful completion. Mutable tags reuse their existing image until an
explicit forced pull.

Unnamed pulls write `<md5(exact-original-url)>.simg` under the selected directory.
Explicit filenames and Snakemake's `pull --name ...` retain their original output
location. Mirror rewriting never changes the hash. Keep URLs exactly consistent;
registry aliases are not normalized. Use an explicit destination for sandbox pulls.
The proxy records file-backed images in `container_list.tsv`, retaining the existing
schema, per-image locks, atomic manifest updates and per-workflow usage timestamps.

Proxy mappings are an optional JSON object of source prefixes to replacements.
Longest prefix wins. The generic `docker://` mapping applies to Docker Hub while
preserving explicit other registries. Without configured mappings, URLs pass
through unchanged. Keep site addresses in site configuration, outside this package.

Native commands, including `cache list`, pass through. This workflow image store
is separate from the runtime's layer cache. Existing `--proxy-capabilities`,
`--proxy-register`, `--proxy-real-runtime` and `--proxy-audit DIRECTORY` interfaces
remain supported. Audit refreshes presence information and discovers unmapped
images; it does not recover their original URLs.

For contributing, validation and releases, see the separate
[developer guide](https://github.com/ETH-NEXUS/nexus_snakemake_utils/blob/main/DEVELOPMENT.md).
