Metadata-Version: 2.4
Name: vivarium-dagger
Version: 0.1.0
Summary: Build and run multi-step data pipelines.
Author-email: The vivarium developers <vivarium.dev@gmail.com>
License: BSD-3-Clause
Project-URL: Homepage, https://github.com/ihmeuw/vivarium-suite
Project-URL: Source, https://github.com/ihmeuw/vivarium-suite/tree/main/libs/dagger
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Natural Language :: English
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: BSD
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Education
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.11
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: vivarium-dependencies[click,loguru,pyyaml]
Requires-Dist: vivarium-build-utils<5.0.0,>=4.0.0
Requires-Dist: vivarium-cluster-tools>=5.0.0
Provides-Extra: cluster
Requires-Dist: jobmon_installer_ihme; extra == "cluster"
Provides-Extra: simulation
Requires-Dist: vivarium-psimulate>=1.0.0; extra == "simulation"
Provides-Extra: test
Requires-Dist: vivarium-dependencies[pytest]; extra == "test"
Requires-Dist: pytest-vivarium; extra == "test"
Provides-Extra: docs
Requires-Dist: vivarium-dependencies[docutils,sphinx,sphinx-click]; extra == "docs"
Provides-Extra: interactive
Requires-Dist: vivarium-dependencies[interactive]; extra == "interactive"
Provides-Extra: lint
Requires-Dist: vivarium-dependencies[lint]; extra == "lint"
Provides-Extra: dev
Requires-Dist: vivarium-dagger[cluster,docs,interactive,lint,simulation,test]; extra == "dev"
Provides-Extra: ci-github
Requires-Dist: vivarium-dagger[docs,lint,simulation,test]; extra == "ci-github"
Provides-Extra: ci-jenkins
Requires-Dist: vivarium-dagger[ci_github,cluster]; extra == "ci-jenkins"
Dynamic: license-file

===============
Vivarium Dagger
===============

.. image:: https://badge.fury.io/py/vivarium-dagger.svg
    :target: https://badge.fury.io/py/vivarium-dagger

.. image:: https://readthedocs.org/projects/vivarium-dagger/badge/?version=latest
    :target: https://vivarium-dagger.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status

This library builds and runs multi-step data pipelines. It provides ``dagger``,
a command that runs a pipeline described in a YAML file as one Jobmon workflow
on a Slurm cluster.

**Supported Python versions: 3.11, 3.12, 3.13**

You can install ``vivarium-dagger`` from PyPI with pip:

.. code-block:: bash

   pip install vivarium-dagger

or build it from source by cloning the monorepo and installing this package:

.. code-block:: bash

   git clone https://github.com/ihmeuw/vivarium-suite.git
   cd vivarium-suite
   pip install libs/dagger

.. note::

    The ``[cluster]`` extra (``pip install vivarium-dagger[cluster]``) pulls in
    ``jobmon_installer_ihme`` from IHME's internal PyPI mirror. ``dagger`` needs
    it to submit jobs. The ``[simulation]`` extra installs ``vivarium-psimulate``,
    which a workflow with a ``simulation`` step needs.

Multi-step workflows with ``dagger``
------------------------------------

A ``dagger`` workflow file lists steps that run in order. Each step gives its
command and its compute resources:

.. code-block:: yaml

    workflow:
      name: my_pipeline
      project: proj_simscience
      queue: all.q
      output_directory: /path/to/output
      steps:
        - name: run_sims
          type: simulation
          resources:
            memory_gb: 3
            runtime: "24:00:00"
          args:
            model_specification: /path/to/model_specification.yaml
            branch_configuration: /path/to/branches.yaml
        - name: post_process
          command: my_post_processing_script /path/to/output
          resources:
            memory_gb: 8
            runtime: "00:30:00"

Launch the workflow with

.. code-block:: console

    dagger run -c /path/to/workflow.yaml

If a run fails partway through, resume it from its output directory, skipping
steps that already completed, with

.. code-block:: console

    dagger restart /path/to/output

`Check out the docs! <https://vivarium-dagger.readthedocs.io/en/latest/>`_
--------------------------------------------------------------------------
