Metadata-Version: 2.4
Name: bio-pype-runtime-models
Version: 0.1.0
Summary: Learn per-workflow resource, timeline and power-curve models from bio-pype runtime dumps.
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.23
Provides-Extra: pype
Requires-Dist: bio-pype; extra == "pype"

# bio-pype-runtime-models

Learn per-workflow **resource**, **timeline** and **power-curve** models from
[Bio-Pype](https://bio-pype.readthedocs.io) runtime data, then use them to
predict a new run's cost from a single input size, to grade a finished run
against its own baseline, and to suggest better default requirements.

Built to be imported into other projects ([compute.bio](https://compute.bio)
trains and serves these models from its database). Pure `numpy`; models
serialise to plain JSON — no pickle.

> **Status: alpha.** The modelling choices and the JSON model format will
> change between releases, and estimates from few runs are rough. Retrain
> after upgrading rather than loading models written by an older version.

The distribution is `bio-pype-runtime-models`; the import name and the command
are `pype_runtime_models` / `pype-runtime-models`.

## Install

```bash
pip install bio-pype-runtime-models
```

Requires Python >= 3.9 and `numpy`. Bio-Pype itself is not needed: the package
only reads the runtime data Bio-Pype writes.

## Input data

One *run* is the content of a Bio-Pype `pipeline_runtime.yaml`: a mapping of
job id to job record (command, status, timings, `requirements`,
`resource_consumption`, `data_sizes`), plus run metadata and environment. It
can come from:

- a directory of `*.json` files, one per run ("dumps");
- parsed `Run` objects, built with `Run.from_dict(mapping)`, when the runs live
  somewhere else, such as a database.

## What it does (mapped to the original goals)

| Goal | Where |
|------|-------|
| Detect which workflow a runtime object is | `workflow.detect_workflow` — `pipeline_name`, else a snippet-set signature match |
| One model per workflow, improving as more runs land | `train.train_models` rebuilds the whole registry each time; safe to schedule |
| Per-step resource consumption, spot deviations | `StepModel` per snippet; `evaluate_run` flags per-step anomalies |
| Predict timeline + resources of each step from input size | `predict.predict_run` |
| Flag steps whose memory/CPU is driven by input size | `StepModel.input_sensitive` / `sensitivity_report` |
| Recommend raising/lowering the default requirement | `StepModel.recommend` → increase / decrease / ok |
| Post-hoc performance eval, overall and per-step | `evaluate.evaluate_run` |
| Predict the power curve for a workflow | `predict.predict_power_curve` (experimental, see below) |

## Workflow identity & provenance

Snippets live in `$PYPE_MODULES/snippets` and are **shared by every pipeline in a
module**, so the natural owner of a snippet's resource model is the module, not
the individual pipeline. Identity is therefore pluggable (`--identity` on train,
`identity_mode=` in the API):

- **`module`** (default) — key models by the `PYPE_MODULES` basename, pooling
  every pipeline in the module. Shared snippets like `mosdepth` train on all
  their samples at once (e.g. n=116 across the 5 `ensemble_sv` pipelines instead
  of split five ways). Resource models are pooled; each pipeline keeps its own
  timeline view (snippet set + start offsets), so `predict <module> --pipeline X`
  gives X's schedule against the pooled models.
- **`pipeline`** — the older one-model-per-`pipeline_name` behaviour.
- **`registry`** — key by a versioned `namespace/name:version` parsed from a
  registry-style module path (e.g. `.../registry/sci-bioinformatics/somatic:20260701`).
  This is the collision-proof, version-aware identity the module registry will
  provide; it falls back to the module basename when no registry info is present.

Every model records its provenance (`module_paths`, `namespaces`, `versions`,
`pipeline_names`, `profiles`), so the day the registry lands, switching to
version-aware keying — or sharing/federating models at the app level — is a
config change, not a rewrite. `predict`/`evaluate` accept either an identity key
or a `pipeline_name` (auto-resolved to its module).

Provenance includes module paths, which often contain user or group names.
Strip it before showing a model to anyone other than the people whose runs
trained it.

Caveat for `module` mode: two *unrelated* modules that happen to share a basename
(your `somatic` vs a third party's) would be merged — the registry namespace is
what ultimately disambiguates them.

## Key modelling choices

- **Canonical step identity** is the *snippet name* parsed from the job command
  (`ingest.snippet_name_from_command`), not the `name` field — the latter carries
  per-sample suffixes (`mosdepth_EGAD2585`) that would fragment the models.
- **Driver size.** Each step is regressed against its own *driver size*:
  `input_size_bytes`, or `output_size_bytes` when the input is 0 (generators /
  aggregators). Because a caller supplies only one number at prediction time, we
  also learn how each step's driver scales with the **run-level** driver
  (`driver_scale`), so the whole chain is predictable from a single input size.
  The run driver is the largest input among the earliest-submitted jobs — the
  recorded `dependencies` are often partial in dumps, so we don't rely on a
  strict topological root.
- **Per-step fit** picks the best of `constant` / `linear` / `power` (power fit
  and scored in log space for numerical stability) by **leave-one-out R²**, and
  only beats the constant baseline if it generalises — noisy/sparse steps stay
  constant instead of overfitting.
- **Few samples means no size dependence.** A step needs at least 5 samples
  with differing driver sizes before a size-dependent fit is even tried. Below
  that the prediction is the median of what was seen, whatever the input size:
  three runs of a workflow give the same expected duration for a small and a
  large input.
- **Robust throughout**: medians, MAD, and Spearman rank correlation, because
  runtime telemetry is heavy-tailed. Predictions are capped at 50× the largest
  observed value so queries far outside the training range can't blow up.

## How a run is graded

`evaluate_run` compares each completed step with the prediction for that
step's own driver size:

- A step is **flagged** when duration or peak memory is at least 3 robust
  standard deviations above the prediction, or at least 2x the prediction.
  Average CPU is reported for context and never flagged.
- The spread used for the z-score has a floor of 5% of the predicted value, so
  a step whose training runs happened to be almost identical does not flag
  every normal variation.
- Steps with no model are reported as `unmodelled`, not dropped.
- The overall `performance_score` (0-100) and `verdict` come from the duration
  z-scores. The verdict is `unknown` when no step's duration could be graded.

## Requirement recommendations

`StepModel.recommend` compares the default requirement recorded with the jobs
to what the step used: predicted peak memory with 1.3x headroom, predicted
duration with 1.5x headroom. It suggests lowering a requirement that is more
than 3x above the 95th percentile of use. Check a suggestion against the
underlying measurements before applying it; a bad measurement in the training
runs produces a bad suggestion.

## GPU steps

Steps are also grouped by a vendor-neutral GPU class. Once a class has at
least 3 samples the model learns a duration factor for it relative to the
size-based prediction; `--gpu-class` / `gpu_class=` applies it, and omitting it
predicts for the mix of cards seen in training.

## Data hygiene (split runs & mis-measured inputs)

These matter because of how runs are actually dumped, and are handled generally
— **no per-snippet logic is hardcoded anywhere**:

- **Split / partial runs are kept, not discarded.** A logical run is often
  spread across several dump files: a failed run is dropped, a resumed run
  completes the same file, and a *relaunched* run starts a fresh file missing
  the steps that already succeeded. Every completed step in every file is a
  valid per-step sample, so all of them train the per-step models. The
  `quality` report shows how many runs per workflow are partial.
- **Run-level input size is only trusted when it's actually present.** Each
  workflow's input-tier ("entry") snippets are learned from the data (the steps
  whose driver size is largest by a wide margin). A partial file that lacks an
  entry step contributes its per-step samples but no run-level driver, so it
  can't feed the chain-scaling model a misleading input size.
- **Implausible driver sizes are dropped per step.** Some inputs are folders
  historically mis-measured as a few bytes, or are overwritten in place with a
  dummy value by convert/cleanup steps. A driver is dropped only when it is
  *both* < 1 MB *and* > 10,000× smaller than that step's own largest driver —
  a general statistical test that removes the bogus cluster while leaving
  genuinely small-input steps (all a few KB) untouched. Steps left with no
  size signal fall back to a constant (median) model, which is correct for
  cleanup-style steps whose cost isn't driven by input size. Counts surface in
  the `quality` report and per-step `n_driver_outliers`.

```bash
python -m pype_runtime_models quality models/     # partial runs, dropped outliers, entry snippets
```

## CLI

```bash
# train / retrain from a dumps directory
pype-runtime-models train dumps/ --out models/

# list what got trained
pype-runtime-models workflows models/

# predict per-step resources + timeline for a 50 GB input
pype-runtime-models predict models/ ensemble_sv_TO --input-size 50gb --full

# grade a finished run
pype-runtime-models evaluate models/ dumps/<hash>.json

# requirement changes worth making
pype-runtime-models recommend models/ ensemble_sv_TO --changes-only

# learned per-GPU-class duration factors
pype-runtime-models gpu models/ ensemble_sv_TO

# predict the power curve (30-min bins by default; experimental)
pype-runtime-models power models/ ensemble_sv_TO --input-size 50gb --bin-minutes 60
```

`train` options: `--identity module|pipeline|registry` (default `module`),
`--min-workflow-runs N` (skip identities seen in fewer runs, default 1) and
`--min-step-samples N` (skip steps with fewer completed jobs, default 3).
`python -m pype_runtime_models ...` is equivalent to the `pype-runtime-models`
command.

## Library

```python
from pype_runtime_models import (
    Run, train_models, load_models, predict_run, evaluate_run,
)

reg = train_models("dumps/", "models/")          # or load_models("models/")
pred = predict_run(reg, "ensemble_sv_TO", input_size_bytes=50 * 1024**3)
report = evaluate_run(reg, "dumps/somehash.json")

print(report["verdict"], report["performance_score"])
for step in report["flagged_steps"]:
    print(step["name"], step["flags"])
```

Without files, for runs held in a database or loaded from
`pipeline_runtime.yaml`:

```python
runs = [Run.from_dict(mapping) for mapping in runtime_mappings]
reg = train_models(runs, min_workflow_runs=3)    # nothing written to disk
report = evaluate_run(reg, runs[0])

for snippet, recs in reg.get("ensemble_sv").all_recommendations().items():
    ...
```

`train_models` only writes to disk when given a models directory. A workflow
model round-trips through `WorkflowModel.to_json()`.

## Power curve (experimental)

`predict_power_curve` turns a predicted run into a binned power curve with a
copy of Bio-Pype's power estimate that is bundled with this package.

**Known issue:** when Bio-Pype is importable in the same environment, the
package tries to use Bio-Pype's own `pipeline_power_curve` instead, and with
current Bio-Pype that call fails with a `TypeError`. Training, prediction of
resources and timeline, evaluation and recommendations are not affected.

## Recurrent retraining

`train_models` is idempotent — point it at the growing dumps directory and it
rebuilds every workflow model from the full corpus, so new runs simply sharpen
the estimates. Schedule it every few hours, e.g. cron:

```cron
0 */4 * * * pype-runtime-models train /path/to/dumps --out /path/to/models \
    >> /var/log/pype_models.log 2>&1
```

## Package layout

```
pype_runtime_models/
  ingest.py      Run/Job parsing, snippet name parsing, driver sizes
  units.py       memory / time requirement parsing
  provenance.py  module/registry provenance + pluggable identity key
  gpu.py         vendor-neutral GPU-class key + display name guess
  workflow.py    identity detection (provenance + signature fallback)
  features.py    per-step training samples, metric list
  regression.py  numpy constant/linear/power fitting + driver cleaning
  model.py       StepModel / WorkflowModel / PipelineView / ModelRegistry
  train.py       train_models / load_models (module-level pooling)
  predict.py     predict_run / predict_power_curve (GPU band, pipeline view)
  evaluate.py    evaluate_run (per-step + overall, GPU-adjusted grading)
  power.py       power estimate (bundled copy of Bio-Pype's, see Power curve)
  cli.py         argparse entrypoint
  tests/         self-contained smoke test (no pytest needed)
```

The smoke test is not part of the published package. From a source checkout,
with a directory of runtime dumps:
`python -m pype_runtime_models.tests.test_smoke dumps/`

Source and issues: <https://codeberg.org/bio-pype/bio-pype-runtime-models>
