Metadata-Version: 2.4
Name: kwdagger
Version: 0.2.4
Summary: The kwdagger module
Author-email: Jon Crall <jon.crall@kitware.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://gitlab.kitware.com/computer-vision/kwdagger
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/x-rst
Requires-Dist: numpy>=2.3.4; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: numpy>=2.1.0; python_version < "3.14" and python_version >= "3.13"
Requires-Dist: numpy>=1.26.0; python_version < "3.13" and python_version >= "3.12"
Requires-Dist: numpy>=1.23.2; python_version < "3.12" and python_version >= "3.11"
Requires-Dist: numpy>=1.21.6; python_version < "3.11" and python_version >= "3.10"
Requires-Dist: scipy>=1.16.2; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: scipy>=1.14.1; python_version < "3.14" and python_version >= "3.13"
Requires-Dist: scipy>=1.11.2; python_version < "3.13" and python_version >= "3.12"
Requires-Dist: scipy>=1.9.2; python_version < "3.12" and python_version >= "3.11"
Requires-Dist: scipy>=1.7.2; python_version < "3.11" and python_version >= "3.10"
Requires-Dist: ubelt>=1.3.6
Requires-Dist: kwarray>=0.6.19
Requires-Dist: kwutil>=0.3.8
Requires-Dist: networkx>=3.6; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: networkx>=3.2; python_version < "3.14"
Requires-Dist: rich>=12.5.1
Requires-Dist: pandas>=2.3.3; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: pandas>=2.2.3; python_version < "3.14" and python_version >= "3.13"
Requires-Dist: pandas>=2.1.1; python_version < "3.13" and python_version >= "3.12"
Requires-Dist: pandas>=1.5.3; python_version < "3.12" and python_version >= "3.11"
Requires-Dist: pandas>=1.5.3; python_version < "3.11" and python_version >= "3.10"
Requires-Dist: scriptconfig>=0.8.4
Requires-Dist: psutil>=6.1.0; python_version < "4.0" and python_version >= "3.9"
Requires-Dist: psutil>=5.9.1; python_version < "3.9" and python_version >= "3.7"
Requires-Dist: ruamel.yaml>=0.17.22
Requires-Dist: PyYAML>=6.0.1; python_version < "4.0" and python_version >= "3.12"
Requires-Dist: PyYAML>=6.0; python_version < "3.12"
Requires-Dist: pygtrie>=2.5.0
Requires-Dist: fasteners>=0.17.3
Requires-Dist: wrapt>=1.17.3
Requires-Dist: cmd_queue>=0.1.20
Requires-Dist: pint>=0.24.4; python_version < "4.0" and python_version >= "3.13"
Requires-Dist: pint>=0.23; python_version < "3.13"
Requires-Dist: safer>=4.4.1
Requires-Dist: parse>=1.19.0
Provides-Extra: docs
Requires-Dist: sphinx>=5.0.1; extra == "docs"
Requires-Dist: sphinx-autobuild>=2021.3.14; extra == "docs"
Requires-Dist: sphinx_rtd_theme>=1.2.1; extra == "docs"
Requires-Dist: sphinxcontrib-napoleon>=0.7; extra == "docs"
Requires-Dist: sphinx-autoapi>=1.8.4; extra == "docs"
Requires-Dist: Pygments>=2.9.0; extra == "docs"
Requires-Dist: myst_parser>=0.18.0; extra == "docs"
Requires-Dist: sphinx-reredirects>=0.0.1; extra == "docs"
Requires-Dist: xdoctest>=1.1.2; extra == "docs"
Provides-Extra: linting
Requires-Dist: flake8>=5.0.0; extra == "linting"
Provides-Extra: optional
Requires-Dist: kwplot>=0.4.14; extra == "optional"
Requires-Dist: matplotlib>=3.10.5; (python_version < "4.0" and python_version >= "3.14") and extra == "optional"
Requires-Dist: matplotlib>=3.9.2; (python_version < "3.14" and python_version >= "3.13") and extra == "optional"
Requires-Dist: matplotlib>=3.8.2; (python_version < "3.13" and python_version >= "3.12") and extra == "optional"
Requires-Dist: matplotlib>=3.8.2; (python_version < "3.12" and python_version >= "3.11") and extra == "optional"
Requires-Dist: matplotlib>=3.8.2; (python_version < "3.11" and python_version >= "3.10") and extra == "optional"
Provides-Extra: tests
Requires-Dist: xdoctest>=1.1.5; extra == "tests"
Requires-Dist: pytest>=8.1.1; (python_version < "4.0" and python_version >= "3.13") and extra == "tests"
Requires-Dist: pytest>=8.1.1; (python_version < "3.13" and python_version >= "3.12") and extra == "tests"
Requires-Dist: pytest>=8.1.1; (python_version < "3.12" and python_version >= "3.11") and extra == "tests"
Requires-Dist: pytest>=6.2.5; (python_version < "3.11" and python_version >= "3.10") and extra == "tests"
Requires-Dist: pytest>=6.2.5; (python_version < "3.10" and python_version >= "3.8") and extra == "tests"
Requires-Dist: pytest-cov>=3.0.0; python_version >= "3.6.0" and extra == "tests"
Requires-Dist: pytest-cov>=2.9.0; (python_version < "3.6.0" and python_version >= "3.5.0") and extra == "tests"
Requires-Dist: pytest-cov>=2.8.1; (python_version < "3.5.0" and python_version >= "3.4.0") and extra == "tests"
Requires-Dist: pytest-cov>=2.8.1; (python_version < "2.8.0" and python_version >= "2.7.0") and extra == "tests"
Requires-Dist: pytest_timeout>=2.3.1; (python_version < "4.0" and python_version >= "3.12") and extra == "tests"
Requires-Dist: pytest_timeout>=1.4.2; python_version < "3.12" and extra == "tests"
Requires-Dist: coverage>=7.3.0; (python_version < "4.0" and python_version >= "3.12") and extra == "tests"
Requires-Dist: coverage>=6.1.1; (python_version < "3.12" and python_version >= "3.10") and extra == "tests"
Requires-Dist: coverage>=5.3.1; (python_version < "3.10" and python_version >= "3.9") and extra == "tests"
Requires-Dist: coverage>=6.1.1; (python_version < "3.9" and python_version >= "3.8") and extra == "tests"

Kwdagger
========

|Pypi| |PypiDownloads| |GitlabCIPipeline| |GitlabCICoverage| |ReadTheDocs|

+-----------------+-----------------------------------------------------+
| Read the Docs   | http://kwdagger.readthedocs.io/en/latest/           |
+-----------------+-----------------------------------------------------+
| Gitlab (main)   | https://gitlab.kitware.com/computer-vision/kwdagger |
+-----------------+-----------------------------------------------------+
| Github (mirror) | https://github.com/Kitware/kwdagger                 |
+-----------------+-----------------------------------------------------+
| Pypi            | https://pypi.org/project/kwdagger                   |
+-----------------+-----------------------------------------------------+

Overview
--------
KWDagger is a lightweight framework for defining bash-centric DAGs and running
large parameter sweeps. It builds on top of
`cmd_queue <https://gitlab.kitware.com/computer-vision/cmd_queue>`_ and
`scriptconfig <https://gitlab.kitware.com/utils/scriptconfig>`_ to provide:

* Reusable ``kwdagger.pipeline.Pipeline`` and ``kwdagger.pipeline.ProcessNode``
  abstractions for wiring inputs / outputs together.
* A scheduling CLI (``kwdagger.schedule``) that materializes pipeline
  definitions over a parameter grid and executes them via Slurm, tmux, or a
  serial backend.
* An aggregation CLI (``kwdagger.aggregate``) that loads job outputs, computes
  metrics, and optionally plots parameter/metric relationships.
* A self-contained demo pipeline in ``kwdagger.demo.demodata`` that is used
  in CI and serves as a reference implementation.

Repository layout
-----------------
* ``kwdagger/pipeline.py`` – core pipeline and process node definitions, networkx
  graph construction, and configuration utilities.
* ``kwdagger/schedule.py`` – ``ScheduleEvaluationConfig`` CLI for expanding
  parameter grids into runnable jobs and dispatching them through cmd_queue
  backends.
* ``kwdagger/aggregate.py`` – ``AggregateEvluationConfig`` CLI for loading job
  outputs, computing parameter hash IDs, and generating text/plot reports.
* ``kwdagger/demo/demodata.py`` – end-to-end demo pipeline with prediction and
  evaluation stages plus CLI entry points for each node.
* ``docs/`` – Sphinx sources, including an example user module under
  ``docs/source/manual/tutorials/twostage_pipeline``.
* ``tests/`` – unit and functional coverage for pipeline wiring, scheduler
  behavior, aggregation, and import sanity checks.

Quickstart
----------
Run the demo pipeline locally to see the CLI workflow end-to-end:

.. code:: bash

    TMP_DPATH=$(mktemp -d --suffix "-kwdagger-demo")
    cd "$TMP_DPATH"
    echo "demo" > input.txt

    EVAL_DPATH=$PWD/pipeline_output
    python -m kwdagger.schedule \
        --params="
            pipeline: 'kwdagger.demo.demodata.my_demo_pipeline()'
            matrix:
                stage1_predict.src_fpath:
                    - input.txt
                stage1_predict.param1:
                    - 123
                stage1_evaluate.workers: 2
        " \
        --root_dpath="${EVAL_DPATH}" \
        --backend=serial --skip_existing=1 --run=1

    python -m kwdagger.aggregate \
        --pipeline='kwdagger.demo.demodata.my_demo_pipeline()' \
        --target "
            - $EVAL_DPATH
        " \
        --output_dpath="$EVAL_DPATH/full_aggregate" \
        --eval_nodes="
            - stage1_evaluate
        " \
        --stdout_report="
            top_k: 10
            concise: 1
        "

The scheduler will generate per-node job directories with ``invoke.sh`` and
``job_config.json`` metadata. The aggregator then consolidates results,
computes parameter hash IDs, and prints a concise report.

A novel graph based symlink structure allows for navigation of dependencies
within a node. The ``.succ`` folder holds symlinks to successors (i.e. results
that depend on the current results), and ``.pred`` holds symlinks to folders of
results that the current folder depends on.

For more in-depth information see tutorials:

* `Tutorial #1: twostage_pipeline <docs/source/manual/tutorials/twostage_pipeline/README.rst>`_
* `Tutorial #2: ollama_benchmark <docs/source/manual/tutorials/ollama_benchmark/README.rst>`_

Command line entry points
-------------------------
* ``python -m kwdagger.schedule`` or ``kwdagger schedule`` – build and run a
  pipeline over a parameter matrix (see ``kwdagger.schedule.ScheduleEvaluationConfig``).
* ``python -m kwdagger.aggregate`` or ``kwdagger aggregate`` – load completed
  runs and generate tabular and plotted summaries
  (``kwdagger.aggregate.AggregateEvluationConfig``).
* ``python -m kwdagger`` – modal CLI that exposes the ``schedule`` and
  ``aggregate`` commands via ``kwdagger.__main__.KWDaggerModal``.

.. |Pypi| image:: https://img.shields.io/pypi/v/kwdagger.svg
    :target: https://pypi.python.org/pypi/kwdagger
.. |PypiDownloads| image:: https://img.shields.io/pypi/dm/kwdagger.svg
    :target: https://pypistats.org/packages/kwdagger
.. |ReadTheDocs| image:: https://readthedocs.org/projects/kwdagger/badge/?version=latest
    :target: http://kwdagger.readthedocs.io/en/latest/
.. |GitlabCIPipeline| image:: https://gitlab.kitware.com/computer-vision/kwdagger/badges/main/pipeline.svg
    :target: https://gitlab.kitware.com/computer-vision/kwdagger/-/jobs
.. |GitlabCICoverage| image:: https://gitlab.kitware.com/computer-vision/kwdagger/badges/main/coverage.svg
    :target: https://gitlab.kitware.com/computer-vision/kwdagger/commits/main

