Metadata-Version: 2.4
Name: kwdagger
Version: 0.4.1
Summary: The kwdagger module
Author-email: Jon Crall <jon.crall@kitware.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://gitlab.kitware.com/computer-vision/kwdagger
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: numpy>=2.3.4; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: numpy>=2.1.0; python_version < "3.14" and python_version >= "3.13"
Requires-Dist: numpy>=1.26.0; python_version < "3.13" and python_version >= "3.12"
Requires-Dist: numpy>=1.23.2; python_version < "3.12" and python_version >= "3.11"
Requires-Dist: numpy>=1.21.6; python_version < "3.11" and python_version >= "3.10"
Requires-Dist: scipy>=1.16.2; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: scipy>=1.14.1; python_version < "3.14" and python_version >= "3.13"
Requires-Dist: scipy>=1.11.2; python_version < "3.13" and python_version >= "3.12"
Requires-Dist: scipy>=1.9.2; python_version < "3.12" and python_version >= "3.11"
Requires-Dist: scipy>=1.7.2; python_version < "3.11" and python_version >= "3.10"
Requires-Dist: ubelt>=1.3.6
Requires-Dist: kwarray>=0.6.19
Requires-Dist: kwutil>=0.3.8
Requires-Dist: networkx>=3.6; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: networkx>=3.2; python_version < "3.14"
Requires-Dist: rich>=12.5.1
Requires-Dist: pandas>=2.3.3; python_version < "4.0" and python_version >= "3.14"
Requires-Dist: pandas>=2.2.3; python_version < "3.14" and python_version >= "3.13"
Requires-Dist: pandas>=2.1.1; python_version < "3.13" and python_version >= "3.12"
Requires-Dist: pandas>=1.5.3; python_version < "3.12" and python_version >= "3.11"
Requires-Dist: pandas>=1.5.3; python_version < "3.11" and python_version >= "3.10"
Requires-Dist: psutil>=6.1.0; python_version < "4.0" and python_version >= "3.9"
Requires-Dist: psutil>=5.9.1; python_version < "3.9" and python_version >= "3.7"
Requires-Dist: ruamel.yaml>=0.17.22
Requires-Dist: PyYAML>=6.0.1; python_version < "4.0" and python_version >= "3.12"
Requires-Dist: PyYAML>=6.0; python_version < "3.12"
Requires-Dist: pygtrie>=2.5.0
Requires-Dist: fasteners>=0.17.3
Requires-Dist: wrapt>=1.17.3
Requires-Dist: cmd_queue>=0.3.2
Requires-Dist: kwconf>=0.10.0
Requires-Dist: pint>=0.24.4; python_version < "4.0" and python_version >= "3.13"
Requires-Dist: pint>=0.23; python_version < "3.13"
Requires-Dist: safer>=4.4.1
Requires-Dist: parse>=1.19.0
Provides-Extra: docs
Requires-Dist: sphinx>=5.0.1; extra == "docs"
Requires-Dist: sphinx-autobuild>=2021.3.14; extra == "docs"
Requires-Dist: sphinx_rtd_theme>=1.2.1; extra == "docs"
Requires-Dist: sphinxcontrib-napoleon>=0.7; extra == "docs"
Requires-Dist: sphinx-autoapi>=1.8.4; extra == "docs"
Requires-Dist: Pygments>=2.9.0; extra == "docs"
Requires-Dist: myst_parser>=0.18.0; extra == "docs"
Requires-Dist: sphinx-reredirects>=0.0.1; extra == "docs"
Requires-Dist: xdoctest>=1.1.2; extra == "docs"
Provides-Extra: linting
Requires-Dist: flake8>=5.0.0; extra == "linting"
Provides-Extra: optional
Requires-Dist: kwplot>=0.4.14; extra == "optional"
Requires-Dist: matplotlib>=3.10.5; (python_version < "4.0" and python_version >= "3.14") and extra == "optional"
Requires-Dist: matplotlib>=3.9.2; (python_version < "3.14" and python_version >= "3.13") and extra == "optional"
Requires-Dist: matplotlib>=3.8.2; (python_version < "3.13" and python_version >= "3.12") and extra == "optional"
Requires-Dist: matplotlib>=3.8.2; (python_version < "3.12" and python_version >= "3.11") and extra == "optional"
Requires-Dist: matplotlib>=3.8.2; (python_version < "3.11" and python_version >= "3.10") and extra == "optional"
Provides-Extra: tests
Requires-Dist: xdoctest>=1.1.5; extra == "tests"
Requires-Dist: pytest>=8.1.1; (python_version < "4.0" and python_version >= "3.13") and extra == "tests"
Requires-Dist: pytest>=8.1.1; (python_version < "3.13" and python_version >= "3.12") and extra == "tests"
Requires-Dist: pytest>=8.1.1; (python_version < "3.12" and python_version >= "3.11") and extra == "tests"
Requires-Dist: pytest>=6.2.5; (python_version < "3.11" and python_version >= "3.10") and extra == "tests"
Requires-Dist: pytest-cov>=3.0.0; python_version >= "3.6.0" and extra == "tests"
Requires-Dist: pytest_timeout>=2.3.1; (python_version < "4.0" and python_version >= "3.12") and extra == "tests"
Requires-Dist: pytest_timeout>=1.4.2; python_version < "3.12" and extra == "tests"
Requires-Dist: coverage>=7.3.0; (python_version < "4.0" and python_version >= "3.12") and extra == "tests"
Requires-Dist: coverage>=6.1.1; (python_version < "3.12" and python_version >= "3.10") and extra == "tests"
Dynamic: license-file

Kwdagger
========

|Pypi| |PypiDownloads| |GitlabCIPipeline| |GitlabCICoverage| |ReadTheDocs|

+-----------------+-----------------------------------------------------+
| Read the Docs   | http://kwdagger.readthedocs.io/en/latest/           |
+-----------------+-----------------------------------------------------+
| Gitlab (main)   | https://gitlab.kitware.com/computer-vision/kwdagger |
+-----------------+-----------------------------------------------------+
| Github (mirror) | https://github.com/Kitware/kwdagger                 |
+-----------------+-----------------------------------------------------+
| Pypi            | https://pypi.org/project/kwdagger                   |
+-----------------+-----------------------------------------------------+

Overview
--------
KWDagger turns parameterized definitions of existing command-line programs into
static, inspectable graphs of shell commands and hashed result directories. It
builds on
`cmd_queue <https://gitlab.kitware.com/computer-vision/cmd_queue>`_ and
`kwconf <https://gitlab.kitware.com/utils/kwconf>`_ to provide:

* Reusable ``kwdagger.pipeline.Pipeline`` and ``kwdagger.pipeline.ProcessNode``
  abstractions for constructing commands and wiring produced artifacts.
* Parameter-matrix expansion with operational deduplication of equivalent
  requested work.
* Per-process ``invoke.sh`` files and a navigable ``.pred`` / ``.succ`` symlink
  graph, so generated work can be inspected, rerun, or invalidated without
  remaining inside the kwdagger runtime.
* A scheduling CLI (``kwdagger.schedule``) that hands the static command graph
  to serial, tmux, or Slurm cmd_queue backends. The tmux backend is the most
  commonly used interactive runner.
* An optional aggregation CLI (``kwdagger.aggregate``) that loads completed
  results, requested and resolved parameters, and metrics into analytical
  tables and reports.
* A self-contained demo pipeline in ``kwdagger.demo.demodata`` that is used in
  CI and serves as a reference implementation.
* ``examples/yaml_introspection_example.py`` is a single-file modal CLI with
  three tiny pipeline stages, an ``introspect`` command that queries their
  declarative and compiled graphs, and a ``run`` command that executes the
  same YAML pipeline locally with the serial backend.

Kwdagger wraps ordinary scripts rather than replacing them. A node may use the
default named-argument command convention or subclass ``ProcessNode`` to support
an existing positional or otherwise specialized CLI.

Repository layout
-----------------
* ``kwdagger/pipeline/`` – core pipeline and process node definitions, networkx
  graph construction, and configuration utilities. Import from
  ``kwdagger.pipeline``; the submodules inside it are private.
* ``kwdagger/schedule.py`` – ``ScheduleEvaluationConfig`` CLI for expanding
  parameter grids into runnable jobs and dispatching them through cmd_queue
  backends.
* ``kwdagger/aggregate.py`` – ``AggregateEvaluationConfig`` CLI for loading job
  outputs, computing parameter hash IDs, and generating text/plot reports.
* ``kwdagger/demo/demodata.py`` – end-to-end demo pipeline with prediction and
  evaluation stages plus CLI entry points for each node.
* ``docs/`` – Sphinx sources, including an example user module under
  ``docs/source/manual/tutorials/twostage_pipeline``.
* ``tests/`` – unit and functional coverage for pipeline wiring, scheduler
  behavior, aggregation, and import sanity checks.

Quickstart
----------
Run the demo pipeline locally to see the CLI workflow end-to-end:

.. code:: bash

    TMP_DPATH=$(mktemp -d --suffix "-kwdagger-demo")
    cd "$TMP_DPATH"
    echo "demo" > input.txt

    EVAL_DPATH=$PWD/pipeline_output
    python -m kwdagger.schedule \
        --params="
            pipeline: 'kwdagger.demo.demodata.my_demo_pipeline()'
            matrix:
                stage1_predict.src_fpath:
                    - input.txt
                stage1_predict.param1:
                    - 123
                stage1_evaluate.workers: 2
        " \
        --root_dpath="${EVAL_DPATH}" \
        --backend=serial --skip_existing=1 --run=1

    python -m kwdagger.aggregate \
        --pipeline='kwdagger.demo.demodata.my_demo_pipeline()' \
        --target "
            - $EVAL_DPATH
        " \
        --output_dpath="$EVAL_DPATH/full_aggregate" \
        --eval_nodes="
            - stage1_evaluate
        " \
        --stdout_report="
            top_k: 10
            concise: 1
        "

The scheduler generates per-node job directories with ``invoke.sh`` and
``job_config.json`` metadata. Each ``invoke.sh`` is intended to be a usable
recomputation command even when kwdagger is not involved in the rerun. The
aggregator is then one optional way to consolidate results and print a report.

The hashed result tree contains a graph-based symlink structure for navigating
produced-artifact dependencies. The ``.succ`` folder links to results that
depend on the current result, and ``.pred`` links to results consumed by the
current process. This makes downstream inspection and invalidation possible
after the original scheduling command has finished.

For more in-depth information see tutorials:

* `Tutorial #1: twostage_pipeline <docs/source/manual/tutorials/twostage_pipeline/README.rst>`_
* `Tutorial #2: ollama_benchmark <docs/source/manual/tutorials/ollama_benchmark/README.rst>`_

Command line entry points
-------------------------
* ``python -m kwdagger.schedule`` or ``kwdagger schedule`` – build and run a
  pipeline over a parameter matrix (see ``kwdagger.schedule.ScheduleEvaluationConfig``).
* ``python -m kwdagger.aggregate`` or ``kwdagger aggregate`` – load completed
  runs and generate tabular and plotted summaries
  (``kwdagger.aggregate.AggregateEvaluationConfig``).
* ``python -m kwdagger`` – modal CLI that exposes the ``schedule`` and
  ``aggregate`` commands via ``kwdagger.__main__.KWDaggerModal``.

.. |Pypi| image:: https://img.shields.io/pypi/v/kwdagger.svg
    :target: https://pypi.python.org/pypi/kwdagger
.. |PypiDownloads| image:: https://img.shields.io/pypi/dm/kwdagger.svg
    :target: https://pypistats.org/packages/kwdagger
.. |ReadTheDocs| image:: https://readthedocs.org/projects/kwdagger/badge/?version=latest
    :target: http://kwdagger.readthedocs.io/en/latest/
.. |GitlabCIPipeline| image:: https://gitlab.kitware.com/computer-vision/kwdagger/badges/main/pipeline.svg
    :target: https://gitlab.kitware.com/computer-vision/kwdagger/-/jobs
.. |GitlabCICoverage| image:: https://gitlab.kitware.com/computer-vision/kwdagger/badges/main/coverage.svg
    :target: https://gitlab.kitware.com/computer-vision/kwdagger/commits/main

