Metadata-Version: 2.4
Name: miniwdl-fovus
Version: 1.0.2
Summary: miniwdl container backend that runs WDL tasks as jobs on the Fovus Serverless HPC Platform, grouped under one Fovus pipeline per workflow.
Author: Fovus
License: MIT
Keywords: miniwdl,wdl,fovus,hpc,workflow,pipeline
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: miniwdl-fovus-runtime==1.14.2.post1
Requires-Dist: fovus>=2.0.0
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

# miniwdl-fovus

Run [WDL](https://openwdl.org/) workflows on the [Fovus Serverless HPC Platform](https://help.fovus.co/cli/get_started.html) using [miniwdl](https://github.com/chanzuckerberg/miniwdl).

`miniwdl-fovus` registers a miniwdl **container backend** (`fovus_job`) that submits
each WDL task as a containerized **Fovus job**, grouping every task of a run under a
single **Fovus pipeline** so the whole workflow appears as one pipeline on
[app.fovus.co](https://app.fovus.co) with unified status / stop / resume.

It talks to Fovus by shelling out to the official [`fovus`](https://pypi.org/project/fovus/)
CLI — the same approach as Fovus's Nextflow plugin
[`nf-fovus`](https://github.com/Fovus/nf-fovus).

## How it works

```
miniwdl run (cfg: container_backend = fovus_job)
        │
        ├─ one Fovus pipeline per run (fovus pipeline create)
        │
        └─ per WDL task:  FovusJob(TaskContainer)
                            ├─ build job-config JSON from runtime{} Fovus attrs
                            ├─ fovus job create <cfg> <task_dir> --pipeline-id ...
                            ├─ poll: fovus job status --job-id ...
                            └─ read rc / stdout / stderr from the shared mount
```

### Shared-storage I/O model

Fovus storage is mounted (`fovus storage mount`) at one path that exists on the
miniwdl host **and** on every Fovus compute node. The miniwdl run directory lives on
that mount, so — like miniwdl-aws's EFS backend — `container_dir == host_dir`. There
is **no per-task upload, no `fovus job download`**: a task's outputs are visible to
the host the moment its Fovus job finishes. Inputs produced by upstream tasks are
staged into each task's directory on the same mount (a copy within Fovus storage).

## Installation

```bash
pip install miniwdl-fovus    # pulls in miniwdl-fovus-runtime and the fovus CLI
fovus auth login             # authenticate once
```

> `miniwdl-fovus` depends on **`miniwdl-fovus-runtime`**, a Fovus-patched build of
> miniwdl (import package still `WDL`) that adds non-POSIX shared-storage support.
> Do not also install stock `miniwdl` in the same environment. Compatibility:
> `miniwdl-fovus` 1.0.x <-> `miniwdl-fovus-runtime` 1.14.2.post1.

## Usage

The simplest path is the bundled wrapper, which verifies auth, mounts the shared
storage, creates the pipeline, and runs miniwdl with the right config and run dir
(default `--dir`: `/fovus-storage/pipelines/<pipeline_id>/fovus-work`):

```bash
miniwdl-fovus run workflow.wdl -i inputs.json --pipeline-name my-run \
    --storage-mount-point /fovus-storage
```

Or drive `miniwdl run` directly once storage is mounted and auth is set up:

```bash
fovus storage mount /fovus-storage
miniwdl run workflow.wdl -i inputs.json \
    --cfg $(python -c "import miniwdl_fovus,os;print(os.path.join(os.path.dirname(miniwdl_fovus.__file__),'fovus.cfg'))") \
    --dir /fovus-storage/miniwdl_run
```

> The run directory (`--dir`) **must** be under the storage mount point.

## Declaring Fovus resources in WDL

Tasks declare Fovus-native resources directly in `runtime{}`, using the Fovus
job-config field names verbatim (no prefix). Anything omitted falls back to the
`[fovus]` defaults in the config.

```wdl
task align {
    command <<< ... >>>
    runtime {
        docker: "biocontainers/bwa:latest"   # task image (required)
        benchmarkingProfileName: "Default CPU"
        minvCpu: 4
        maxvCpu: 16
        minvCpuMemGiB: 8
        minGpu: 0
        storageGiB: 100
        walltimeHours: 3
        supportedCpuArchitectures: ["x86-64", "arm-64"]
        timeToCostPriorityRatio: "0.5/0.5"
        allowPreemptible: false
    }
}
```

| `runtime` key | Fovus job-config field |
|---|---|
| `benchmarkingProfileName` | `constraints.jobConstraints.benchmarkingProfileName` |
| `supportedCpuArchitectures` | `constraints.jobConstraints.supportedCpuArchitectures` |
| `allowPreemptible` | `constraints.jobConstraints.allowPreemptible` |
| `minvCpu` / `maxvCpu` | `constraints.taskConstraints.minvCpu` / `maxvCpu` |
| `minvCpuMemGiB` | `constraints.taskConstraints.minvCpuMemGiB` |
| `minGpu` / `maxGpu` | `constraints.taskConstraints.minGpu` / `maxGpu` |
| `storageGiB` | `constraints.taskConstraints.storageGiB` |
| `walltimeHours` | `constraints.taskConstraints.walltimeHours` |
| `timeToCostPriorityRatio` | `objective.timeToCostPriorityRatio` |

Power users can bypass the builder entirely with
`configJson: "/path/to/job_config.json"` — the file is used verbatim, with the
generated `runCommand` injected.

## Configuration

All `[fovus]` keys (see [`miniwdl_fovus/fovus.cfg`](miniwdl_fovus/fovus.cfg)) can be
overridden by environment variable, e.g.
`MINIWDL__FOVUS__BENCHMARKING_PROFILE_NAME="Default CPU"`.

Key settings: `storage_mount_point`, `poll_interval_seconds`,
`auto_stop_pipeline_on_exit` (stop the pipeline on Ctrl-C / crash), and the resource
defaults applied when a task omits the corresponding runtime attribute.

## Development

```bash
# The patched runtime lives in a separate repo (Fovus/miniwdl). For local dev,
# clone it under vendor/ (git-ignored) and install it before the plugin:
git clone https://github.com/Fovus/miniwdl.git vendor/miniwdl   # once
pip install ./vendor/miniwdl   # miniwdl-fovus-runtime (satisfies the runtime pin)
pip install -e ".[test]"       # the plugin
pytest                         # unit + simulated-backend integration tests
```

Alternatively, once `miniwdl-fovus-runtime` is published, `pip install -e ".[test]"`
alone resolves the runtime from PyPI. CI and released builds always install it from
the index (see the `Release` GitHub Actions workflow).

`tests/e2e/` contains fake `fovus` and `docker` executables used to exercise the full
`miniwdl run` integration locally without a Fovus account or real Docker.

## Status / scope

v1.0.0 supports **containerized** Fovus jobs grouped under one pipeline per run, with
the shared-storage I/O model. Out of scope for now: monolithic (license-bound) jobs,
auto-provisioning the storage mount, per-task live log streaming, and miniwdl call-cache
integration.

## License

MIT
