Metadata-Version: 2.4
Name: adagio-cli
Version: 0.1.2
Summary: Command-line runner for Adagio pipeline files.
Keywords: adagio,bioinformatics,cli,pipelines,qiime2,workflow
Author: Cymis
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: cyclopts>=4.5.3
Requires-Dist: pydantic>=2.12.5
Requires-Dist: rich>=14.1.0
Requires-Dist: tomli>=2.2.1 ; python_full_version < '3.11'
Requires-Python: >=3.10
Project-URL: Changelog, https://github.com/cymis/adagio-cli/blob/dev/CHANGELOG.md
Project-URL: Homepage, https://github.com/cymis/adagio-cli
Project-URL: Issues, https://github.com/cymis/adagio-cli/issues
Project-URL: Repository, https://github.com/cymis/adagio-cli
Description-Content-Type: text/markdown

# Adagio CLI

`adagio-cli` is the Python command-line interface for Adagio pipeline execution.

For user-facing documentation and product guides, please reference the docs:

- [Adagio Docs](https://docs.adagiodata.com)

The adagio frontend is used to build pipelines that can be run with this package on the command line
It can be found here:

- [Adagio](https://adagio.run)

## Development

Set up the project and run the test suite with:

```bash
uv sync --group dev
uv run pytest
```

## Runtime environments

Every plugin action requires an explicit execution environment. A runtime config
passed with `--config` can define one default and override it per plugin or task;
the CLI never guesses an image from a plugin name.

Conda environments are supported with `kind = "conda"`:

```toml
version = 1

[defaults]
kind = "conda"
prefix = "/opt/conda/envs/qiime2-2026.1"

[plugins]
dada2 = { kind = "conda", prefix = "/opt/conda/envs/q2-dada2" }
```

The environment must already exist and contain QIIME 2 plus the plugins needed
by the pipeline. Adagio enters it with `conda run`; it does not create or manage
the environment.

## Task resource requests

A runtime config can also declare the requested shape of each task execution:

```toml
[resources.tasks."denoise-node"]
cpus = 4
memory = "8 GiB"
```

`cpus` is a positive whole number and `memory` is a positive, unit-bearing
quantity. Each entry describes one independently schedulable execution of that
task, not an aggregate pipeline allocation. If a task is later expanded into
parallel subtasks, each subtask requests this shape and the scheduler determines
the total concurrent allocation.

The current serial executor parses and validates these fields but intentionally
does not apply them yet. Omitting either field leaves that resource at the
executor's default.

## Plugin submission defaults

QAPI submissions can persist the environment that Adagio should use for the
submitted plugins:

```bash
adagio qapi build --plugin my-plugin \
  --default-conda-prefix /opt/conda/envs/my-plugin

adagio qapi build --plugin my-plugin \
  --default-docker-image registry.example.org/my-plugin:2026.1
```

These options are explicit and mutually exclusive. Omitting both leaves the
plugin without a default; the Adagio app will flag that plugin until an
environment is selected for a run.

## Remote artifact inputs

An artifact input accepts a local path or an HTTPS URL, both on the command
line and in an arguments JSON file:

```bash
adagio run pipeline.adg --config runtime.toml --cache-dir ./cache \
  --input-seqs 'https://example.org/data/seqs.qza' --log-dir ./logs
```

```json
{"version": 1, "inputs": {"seqs": "https://example.org/data/seqs.qza"}}
```

Pass that file with `--arguments arguments.json`. The flag name (`seqs` here)
comes from the pipeline's input name.

The consuming task downloads the artifact inside its execution environment;
the orchestration process does not fetch it. The original archive is loaded
without re-importing, preserving its UUID and QIIME provenance. The downloaded
type must match the declared input type and consuming action. Collections may
contain both local paths and URLs; artifact URLs can also supply metadata when
the artifact supports a metadata view. Metadata inputs also accept HTTPS URLs
to TSV files, which QIIME loads directly without importing them as artifacts.

Downloads stream to disk with bounded retries and share a locked staging
directory for the run. Every new run fetches again before QIIME result-cache
lookup, including runs with reuse enabled. To verify a specific version, append
`#sha256=<64 hexadecimal digits>` to the URL. A mismatch fails the input.
Transfer resumption and persistent download caching are not implemented yet.

The staging directory lives in the run's temporary work directory, which is
under the system temporary directory (`TMPDIR` on macOS and Linux). Point
`TMPDIR` at a larger volume for large inputs. A download whose reported size
exceeds the free space there fails before transfer; disk errors are not
retried. Download time appears as `input_download_seconds` in the task timings.

Raw data imports, remote manifests, visualizations, login flows, and embedded
URL credentials are not supported.
The URL need not end in `.qza`; its contents are validated. Workers need network
access to the source. No `wget` or `curl` installation is needed.

Task logs contain transfer progress. `--log-dir` also saves per-task
`*_inputs.json` receipts with the source, final URL, byte count, SHA-256, type,
and original UUID for artifacts (metadata TSVs have no artifact UUID).
Connected runs include these receipts in their task-finished events. URL query strings are omitted from receipts and download logs; the full
source is represented by a hash and remains in the supplied run arguments.

## Catalog pipelines

Run a pipeline from the Adagio pipeline catalog:

```bash
adagio pipeline show @adagio/microbial-diversity
adagio run @adagio/microbial-diversity --cache-dir /path/to/cache --arguments run-arguments.json
```

`@adagio/<slug>` first resolves against a nearby local `adagio-pipelines`
checkout when one is available. If no local catalog is found, Adagio fetches
`pipeline.adg` from `cymis/adagio-pipelines` on GitHub, checking `official`
before `community`.

During `adagio run`, remote catalog pipelines are downloaded under the selected
`--cache-dir` and reused by source name and slug on later runs. `adagio pipeline
show` uses a temporary download when it fetches from GitHub because it does not
take a cache directory.

Private GitHub access is explicit: set `GITHUB_TOKEN` or `GH_TOKEN` to a token
that can read `cymis/adagio-pipelines`; with a token, the CLI fetches through
the GitHub contents API. The CLI does not read browser, git, or `gh` credentials
automatically.

## Cache progress and diagnostics

A task reports **Preparing task** during host-side setup, including input
hashing, then the environment's download/start phases. Inside the environment,
it reports **Preparing cache lookup** while resolving and loading inputs, then
**Checking cache**. A hit reports **Using cached result** while restoring
and saving that result, and finishes as **Reused cached result**. Only a miss
reports **Running action**. Tasks with reuse disabled report **Preparing task**
then **Running action**. Environment startup remains visible separately.

Each successful task records monotonic timings in the `task_finished` event's
`timings` metadata and in its node log (persisted by `--log-dir`). The app's node
log viewer exposes the same breakdown. Useful fields, in seconds:

| Field | Work measured |
| --- | --- |
| `input_signature_seconds` | Host-side file hashing for telemetry; does not determine reuse |
| `pull_seconds` | Docker image availability check/pull |
| `environment_overhead_seconds` | Launcher's process wall time minus worker time; includes startup and shutdown |
| `plugin_setup_seconds` | Framework imports, plugin discovery, action resolution and cache construction |
| `input_loading_seconds` | Loading/importing inputs, metadata, parameter coercion and defaults |
| `cache_pool_seconds` | Creating/reopening the recycle pool |
| `cache_index_seconds` | Building the framework cache index |
| `cache_match_seconds` | Constructing the resolved invocation identity |
| `cache_load_seconds` | Loading matching cached artifacts |
| `cache_lookup_seconds` | Total pool setup, indexing, matching and cached-result loading |
| `action_seconds` | Scientific action execution; zero on a cache hit |
| `output_save_seconds` | Saving worker artifacts and requested metadata views |
| `output_publish_seconds` | Publishing pipeline outputs after the task finishes |
| `worker_seconds` / `run_seconds` | Worker total / encompassing environment process wall time |

Totals overlap their component stages: do not add every field together. Missing
fields mean the stage was not measured (including older worker manifests).
Tasks with reuse disabled omit the cache-stage timing fields.
These are timings of successful tasks, not a profiler for failed actions.
No cache identity, validation, or reuse policy is changed.

To profile a representative pipeline, run it twice against a new isolated cache
with identical inputs, parameters and environment. Preserve both log directories:

```sh
profile_dir=$(mktemp -d)
adagio runtime --spec pipeline.adg --arguments arguments.json --config runtime.toml \
  --cache-dir "$profile_dir/cache" --output-dir "$profile_dir/first/outputs" \
  --log-dir "$profile_dir/first/logs"
adagio runtime --spec pipeline.adg --arguments arguments.json --config runtime.toml \
  --cache-dir "$profile_dir/cache" --output-dir "$profile_dir/repeat/outputs" \
  --log-dir "$profile_dir/repeat/logs"
```

Confirm `Reused cached result: true` before interpreting the repeat as a cache
hit; mutable plugins, changing inputs, or random defaults can still require an
action. Compare repeated runs with the same cache size and input sizes as the
reported slow workload before deciding which stage to optimize.

An opt-in Docker integration test exercises a two-step pipeline with synthetic
feature tables, three cache-hit repeats, changed parameters, changed inputs,
and reuse disabled. Use an installed QIIME image containing `feature-table`:

```sh
ADAGIO_DOCKER_TEST_IMAGE=ghcr.io/cymis/qiime2-plugin-feature-table:2026.1 \
  PYTHONPATH=src python -m pytest -q -s tests/test_docker_cache_integration.py
```

The test uses an isolated temporary cache and prints the path to
`cache-profile.json`, containing the image ID, phases, timings, and output
artifact IDs for each run. Cache hits must preserve artifact IDs and never
report `running`; misses must execute the actions.

Connected deployments require the backend and runtime server with
`adapter-schemas` 0.5.0. Deploy backend support for the new phases before the
runtime update. Progress travels through a mounted JSON-lines file; workers do
not receive runtime credentials or need network access to report phases.
