Metadata-Version: 2.3
Name: libpipe
Version: 0.8
Summary: Pipeline orchestration and node management utilities
License: GPL-3.0-or-later
Author: Florent Mertens
Author-email: florent.mertens@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Provides-Extra: dev
Requires-Dist: asyncssh (>=2.0)
Requires-Dist: click (>=8.0)
Requires-Dist: progressbar2 (>=3.40)
Requires-Dist: pytest (>=8.0) ; extra == "dev"
Requires-Dist: pytest-cov (>=5.0) ; extra == "dev"
Requires-Dist: ruff (==0.16.1) ; extra == "dev"
Requires-Dist: toml (>=0.10)
Project-URL: Homepage, https://gitlab.com/flomertens/libpipe
Project-URL: Issues, https://gitlab.com/flomertens/libpipe/-/issues
Project-URL: Repository, https://gitlab.com/flomertens/libpipe
Description-Content-Type: text/markdown

Libpipe
=======

Common pipeline framework and library

Installation
------------

libpipe can be installed via pip:

    $ pip install libpipe

and requires Python 3.10 or higher.

Worker pools on a Slurm cluster
-------------------------------

A `WorkerPool` runs its tasks locally, on other hosts over SSH, or through Slurm with the
host `slurm`: each task is then started with `srun`, as a job of its own when the pool runs
on a login node, as a job step inside an allocation (`salloc`, `sbatch`).

    pool = worker.WorkerPool(['slurm'], name='Imaging', max_tasks_per_worker=8,
                             slurm_options='-p regular --mem=200G --time=24:00:00')

`max_tasks_per_worker` is the number of jobs in flight. The srun options (partition,
account, resources) come from `slurm_options`, or from the `LIBPIPE_SLURM_OPTIONS`
environment variable; `max_time` becomes the job time limit when the options set none.
Host constraints (`run_on_host`) do not apply: Slurm places the tasks, so the data must be
on a file system shared with the compute nodes. Give the memory (`--mem`) in the options:
inside an allocation, a step without it can claim all the memory of its node.

