Metadata-Version: 2.4
Name: shexec
Version: 2.3.0
Summary: Run a shell-type command over every file of an input tree: a ChRIS DS plugin on pfdo_run
Home-page: https://github.com/FNNDSC/pl-shexec
Author: FNNDSC
Author-email: dev@babyMRI.org
License: MIT
Project-URL: Homepage, https://github.com/FNNDSC/pl-shexec
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX
Requires-Python: >=3.11
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: chris_plugin>=0.4.0
Requires-Dist: pfdo>=6.2.0
Requires-Dist: pfdo_run>=4.2.0
Requires-Dist: hostlib>=1.2.0
Requires-Dist: pflog>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Dynamic: license-file

pl-shexec
=========

**Run a command over every file of a tree, as a ChRIS plugin; and the pattern for turning any command-line program into one.**

``pl-shexec`` is a `ChRIS <https://chrisproject.org/>`_ DS plugin that does two things. In a pipeline it is the plugin for the jobs too small to deserve one of their own: convert every image, zip every directory, rename, copy one file out of each subject, count lines. It takes a command template, runs it once per selected file of every directory in its input tree, and writes the results, and a record of each run, into an output tree that mirrors the input.

.. code-block:: bash

    shexec --fileFilter 'glob:*.jpg'                                            \
        --exec "convert %{inputWorkingDir}/%{inputWorkingFile}
                        %{outputWorkingDir}/%{inputWorkingFile|rmext}.png"      \
        /incoming /outgoing

It is also the design for wrapping a large existing program as a plugin. Put the program in the image, set the default command, and the plugin is done: the program's whole command line stays available through the template rather than a hand-picked subset of flags, and the tree walk, the records, the parallelism and the exit status come from the stack. With for example FreeSurfer in the image, one ``recon-all`` per subject directory is:

.. code-block:: bash

    shexec --analyzeFileIndex f --fileFilter 'glob:*.nii.gz'                     \
        --exec "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile}
                          -s %{inputWorkingDir|basename} -sd %{outputWorkingDir}" \
        /incoming /outgoing

pl-freesurfer is an example of the pattern: a plugin that depends on this one the way this one depends on pfdo_run, contributing the FreeSurfer image, that template as the default ``--exec``, and nothing else. See `Building a plugin on pl-shexec`_.

The engine is `pfdo_run <https://github.com/FNNDSC/pfdo_run>`_; the plugin is pfdo_run with its directories supplied by ChRIS and its own run timed.

The template
------------

``--exec`` is not a shell command. It is a template whose ``%{...}`` tags name the file and directories in hand, resolved once per file, and run directly without a shell, so a path with a space in it stays one argument and nothing is interpreted that was not written. Three tags describe the job:

==========================  ==============================================
``%{inputWorkingDir}``      the input directory being processed
``%{outputWorkingDir}``     its counterpart in the output tree
``%{inputWorkingFile}``     the file being processed
==========================  ==============================================

Functions derive one name from another: ``%{inputWorkingFile|rmext}.png`` turns ``scan.jpg`` into ``scan.png``; ``%{inputWorkingDir|basename}`` is the directory's own name. Every flag of the plugin is a tag too, and so are ``%{timestamp}``, ``%{hostname}`` and ``%{user}``. The grammar is `pftag <https://github.com/FNNDSC/pftag>`_'s; ``shexec --man`` lists it under TAGS. A template with no tag draws a warning, since the same command would then run for every file.

What each run leaves behind
---------------------------

Each directory of the output tree gets, beside its results, one file, ``jobs.jsonl``, with one line per job that ran there:

.. code-block:: text

    scan.png
    jobs.jsonl

The line is the whole record of the job as JSON: the command as executed, everything it printed and everything it complained, its exit status, the host it ran on, and when it started and ended. ``pflog read out/**/jobs.jsonl`` prints a day's jobs as a table, ``--failed`` keeps the ones that went wrong, and a log collector indexes the lines by field.

.. code-block:: text

    STARTED                     DURATION  STATUS  NAME
    2026-10-14T16:02:07-04:00      4.213  ok      convert /incoming/a/scan.jpg /outgoing/a/scan.png
    2026-10-14T16:02:12-04:00      0.004  failed  convert /incoming/b/bad.jpg /outgoing/b/bad.png

``--jobFiles`` also writes each job as four files beside its result, ``scan.jpg-cmd``, ``-stdout``, ``-stderr`` and ``-returncode``, the stream files rendered as a terminal would have shown them, for the reader who opens them in an editor. ``--noJobLogging`` keeps no record at all. ``--json`` prints the whole run's record on standard output.

The plugin's own run is timed by `pflog <https://github.com/FNNDSC/pflog>`_: one line of JSON on standard error, when it began, when it ended, the seconds, and whether it succeeded. CUBE keeps that line with the rest of the plugin's output, and any log collector indexes it.

Selecting files and directories
-------------------------------

The filters are pftree's. ``--fileFilter`` and ``--dirFilter`` take substrings, or ``glob:`` and ``re:`` patterns, joined by ``--fileFilterLogic`` and ``--dirFilterLogic``; ``--maxDepth`` bounds the descent. ``--analyzeFileIndex`` chooses which of a directory's selected files get the command: ``a`` for all, ``f`` for the first, ``l`` for the last, ``m`` for the middle, or an index. ``f`` is how one command runs per directory, started on the first file, as for a DICOM series.

Directories run in parallel under ``--threads``. At ``--verbosity 2`` each command's output streams to the console as it runs.

Exit status
-----------

0 when at least one directory's jobs all succeeded; 1 when every directory failed or the run could not start; 2 when ``--exec`` is missing or a flag's value is unusable. A failed template, a command that cannot be found, or a non-zero exit fails its directory and the run continues; the record says why.

Wrapping a program
------------------

A plugin for a program with its own command line, FreeSurfer, ANTs, FSL, DCMTK, is pl-shexec with three changes and no new code:

- the image carries the program, with whatever licence or data it needs;
- the default ``--exec`` is the program's usual invocation as a template, so the common case needs no command at all and the unusual case overrides it;
- the default ``--analyzeFileIndex`` chooses the unit, ``f`` for one run per directory, as for a subject or a DICOM series, ``a`` for one run per file.

The user of such a plugin gets the program's entire interface, because anything the program accepts can be written into the template, and gets the same records, the same filters, the same threads and the same truthful exit status as every other plugin on the stack. This is the reason pl-shexec exists in the form it has.

Building a plugin on pl-shexec
------------------------------

pl-shexec is a library as well as a plugin, in the same way pfdo_run is a library to pl-shexec. A downstream plugin does not copy it; it depends on the ``shexec`` package and calls ``plugin_build`` with a spec of what differs:

.. code-block:: python

    # freesurfer/__main__.py, the whole of pl-freesurfer's code
    from shexec import PluginSpec, plugin_build

    from freesurfer import __version__

    RECON_ALL: str = (
        "recon-all -all -i %{inputWorkingDir}/%{inputWorkingFile} "
        "-s %{inputWorkingDir|basename} -sd %{outputWorkingDir}"
    )

    PLUGIN = plugin_build(PluginSpec(
        name="freesurfer",
        version=__version__,
        title="FreeSurfer recon-all over every subject",
        minMemory="8Gi",
        minCpu="4000m",
        defaults={"exec": RECON_ALL, "analyzeFileIndex": "f", "fileFilter": "glob:*.nii.gz"},
        bannerTitle="FreeSurfer recon-all over an input tree",
        description=DESCRIPTION,
        examples=EXAMPLES,
    ))
    main = PLUGIN.main

``PluginSpec`` holds everything a plugin changes: its name and version, which become its command, its span and its ``--version``; its title, category and resource limits for ChRIS; the defaults of its flags, above all ``--exec``, which ChRIS shows as the defaults and the user can override; and its words on the manual page, over the sections every plugin shares. A plugin may also decline engine flags it should not offer: ``omit=("threads", "threadsSplit")`` removes them from its parser, its ChRIS description and its manual, and fixes their values, as pl-freesurfer does because two ``recon-all`` runs in one container exhaust its memory. ``plugin_build`` returns the parser with those defaults applied, the manual, and ``main`` already wrapped in pflog's ``timed`` and ``chris_plugin``, ready to be the console script. The engine, the flags, the records and the exit status are not in the spec, because they are the same for every plugin on the stack, and a fix to them reaches every plugin on its next build.

The rest of pl-freesurfer is a ``pyproject.toml`` naming ``chris_plugin`` and ``shexec`` as its dependencies, the first so that ``chris_plugin_info`` finds the plugin, a ``setup.cfg``, a Dockerfile that starts from the FreeSurfer image and installs the package, tests, and docs. pl-shexec itself is the first plugin built this way; its ``__main__`` is the same call with an empty default command. The pf* stack's ``docs/plugin.adoc`` has the template.

Running it
----------

From ChRIS, add the plugin to a pipeline and give it ``--exec`` and filters as parameters.

From the command line, with the image:

.. code-block:: bash

    docker run --rm -v $PWD/in:/incoming:ro -v $PWD/out:/outgoing:rw            \
        fnndsc/pl-shexec shexec --fileFilter txt                                \
        --exec "wc -l %{inputWorkingDir}/%{inputWorkingFile}" /incoming /outgoing

    docker run --rm fnndsc/pl-shexec shexec --man /tmp /tmp

With Apptainer, ``apptainer exec docker://fnndsc/pl-shexec shexec ...``. The image carries the coreutils, zip and unzip, bc, perl, tar, ImageMagick and the DCMTK tools.

As a Python package, ``pip install .`` in this repository gives the ``shexec`` command and the ``shexec`` module, whose ``plugin_run(options, inputdir, outputdir)`` is the body of the plugin and whose ``shexec.models`` re-exports every record pfdo_run, pfdo, pftree, hostlib and pflog produce.

Development
-----------

.. code-block:: bash

    uv venv --python /usr/bin/python3.14 .venv
    uv pip install --python .venv/bin/python -e ".[dev]"
    .venv/bin/ruff check . && .venv/bin/pytest -q
    PATH=$PWD/.venv/bin:$PATH chris_plugin_info --dock-image docker.io/fnndsc/pl-shexec:2.1.0 shexec

The CI workflow lints and tests on Python 3.11 to 3.14, builds the image on every push to ``main``, pushes it to Docker Hub and ghcr.io, and on a ``v*`` tag uploads the plugin's description to cube.chrisproject.org and publishes the ``shexec`` package to PyPI, so that a plugin built on pl-shexec can depend on it. The tag must match ``__version__`` in ``shexec/__init__.py``.

Layout
------

.. code-block:: text

    shexec/
    ├── __main__.py   the plugin: parser, manual, @chris_plugin @timed main
    ├── shexec.py     ShExec, options_build, plugin_run
    └── models.py     ShExecOptions, and every record below re-exported

``docs/purpose.adoc`` says why the plugin exists, ``docs/architecture.adoc`` how it is built, ``docs/history.adoc`` where it came from.
