Metadata-Version: 2.4
Name: nv-sflow
Version: 0.3.1
Summary: A Python CLI tool designed to automate and manage benchmarking workflows on multiple backends.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jinja2>=3.1.6
Requires-Dist: mermaid-py>=0.8.0
Requires-Dist: parse<2,>=1.16
Requires-Dist: pydantic>=2.11.10
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: rich>=13.7.0
Requires-Dist: textual>=8.2.7
Requires-Dist: typer>=0.19.2
Provides-Extra: dev
Requires-Dist: ipython>=8.37.0; extra == "dev"
Requires-Dist: matplotlib>=3.8.0; extra == "dev"
Requires-Dist: moto>=5.0.0; extra == "dev"
Requires-Dist: pytest>=8.4.2; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
Requires-Dist: pytest-cov>=7.0.0; extra == "dev"
Requires-Dist: pytest-subprocess>=1.5.3; extra == "dev"
Requires-Dist: pytest-xdist>=3.8.0; extra == "dev"
Provides-Extra: s3
Requires-Dist: boto3>=1.34.0; extra == "s3"
Provides-Extra: monitor
Requires-Dist: matplotlib>=3.8.0; extra == "monitor"
Dynamic: license-file

# sflow

## Install

```bash
uv venv --python python3 && source .venv/bin/activate
uv pip install "nv-sflow @ git+https://github.com/NVIDIA/nv-sflow.git@main"
sflow --version
```

Already on **sflow v0.3.0+**? Upgrade in place — no incantation to remember:

```bash
sflow upgrade                 # latest main; --branch <ref> / --dry-run also supported
```

No `uv`? `pip install uv` (or `curl -LsSf https://astral.sh/uv/install.sh | sh`).
Optional extras: `sflow[s3]` (S3 uploads), `sflow[monitor]` (PNG hardware charts).
Contributing? Skip to [Development Setup](#development-setup) for the editable install.

## What is sflow

A **declarative workflow descriptor for massive GPU clusters** that separates _what to deploy_ from _where to deploy it_.

**One semantic across every platform — backend agnostic by design.** A deployment's logic never changes: start etcd and NATS, launch a frontend, spin up workers, run the benchmark. Only the infrastructure glue does — and today that glue is rewritten from scratch for every platform. `sflow` consolidates it into a single portable YAML: tasks, dependencies, resources, and launch methods. The **same `sflow.yaml` runs on Docker, Slurm, and Kubernetes** — swap the backend fragment, rewrite nothing. Backends delegate to each platform's native ecosystem (`srun`/MPI, `docker run`, pods and MPI jobs) rather than reimplementing it.

**Cluster-level orchestration at scale.** Topology-aware node and GPU placement, multi-node replicas and sweeps, readiness/failure probes, and batch submission — so one descriptor drives hundreds of GPUs instead of a pile of hand-written bash.

All four backends ship today: `local`, `docker`, `slurm`, and `kubernetes` (`k8s` / `k8s_mpi`). It is light enough to write and debug a recipe on your laptop and submit that same file to the cluster.

![sflow TUI](docs-site/static/img/sflow_tui.gif)

Define _what to run_ in a `sflow.yaml` — tasks, dependencies, how to launch each task, and required resources. `sflow` executes the DAG in order, collects logs, and organizes outputs into a consistent directory structure. Example of a dynamo PD disaggregation LLM inference service workflow:

<p align="center">
  <img src="docs-site/static/img/workflow-dag.png" alt="Workflow DAG Example" width="700">
</p>

## What it's for

Any workload that runs in **steps that depend on each other**: build → train → evaluate,
start a service → wait for it → drive load against it → collect results, stage data →
process shards in parallel → merge. If you would otherwise write a shell script that
launches things in order, polls until something is ready, and tidies up afterwards, that
script is a workflow — and sflow already does the tedious parts.

Distributed LLM inference is the *hardest* case it is designed around, not the only one: a
dozen processes across several nodes that must come up in the right order, find each other,
share GPUs correctly, and be torn down cleanly when any one of them dies. Whatever survives
that tends to handle simpler pipelines comfortably.

### The parts you don't write again

Each of these is a YAML field, not a page of bash:

| Instead of hand-rolling… | You declare |
| --- | --- |
| `until curl -sf …; do sleep 5; done` | `probes.readiness` (HTTP / TCP / log-watch) with a timeout |
| retry loops and backoff arithmetic | `retries: {count, interval, backoff}` |
| stray-process cleanup, `trap` handlers | teardown runs on success, failure and Ctrl-C |
| `mkdir -p logs/$(date +%s)`, `tee`, scattered output | one run directory per workflow, a log per task, a run summary |
| `grep`/`awk` over logs into a CSV | `result:` parsing (regex, named patterns, or JSON) → `result.json` |
| `aws s3 cp` sprinkled through the script | `uploads:` per task on `COMPLETED`, credentials from boto3's chain |
| copying config/data files onto nodes by hand | `artifacts:` staged and mounted at the same path on every backend |
| `for cfg in …; do sbatch …; done` | `replicas:`, or `batch` over a CSV of parameter rows |
| hard-coding hosts, ports and device ids | `${{ }}` expressions over variables, backend nodes and task services |

Plus a live TUI, `--dry-run` for the whole plan, and per-task GPU/CPU/network sampling —
none of which you have to build.

### One recipe, laptop to cluster

**At cluster scale:**

- **Multi-node placement** — `resources.nodes` (`indices` / `count` / `exclude`) and
  `resources.gpus` (`count`, or `indices` to pin exact device ids) pack tasks onto specific
  nodes and devices, with conflict detection at plan time rather than a failure ten minutes
  into the run. On Slurm the plan is re-applied *inside* the step by GPU UUID, so
  `slurmstepd` and container renumbering cannot silently land every rank on device 0.
- **Real ordering** — `depends_on` waits for a process to *start*; readiness **probes**
  (HTTP, TCP, log-watch) make dependents wait for a service to actually be up. `required_by`
  expresses the reverse edge so a workload fragment can attach itself to a benchmark.
- **Distributed launchers** — `srun` with MPI/pmix on Slurm, `k8s` and `k8s_mpi` on
  Kubernetes, where a multi-node task becomes one pod per reserved node (GPUs via the
  device plugin or DRA) and co-located GPU tasks can be merged into a single pod.
- **Fleet-scale runs** — `replicas` for fan-out, `release_after` so a later task can reuse a
  still-running server's GPUs, plus `batch` / bulk-input CSV sweeps over a parameter grid.
- **Results you can compare** — `result:` parsing, `uploads:` to S3, and a passive hardware
  monitor that samples GPU/CPU/network per task, with `monitor.window` log markers to clip a
  report to the phase you actually measured.
- **Run output that explains itself** — `sflow_summary.log` records planned vs. observed GPU
  assignment, node topology, and external command health (`kubectl`/`srun`/`docker` latency
  and failures); a stalled event loop dumps every thread's stack to `loop_stalls.txt`.

**On one machine:**

- **`local` backend, no scheduler** — the Quickstart below runs a DAG in a few seconds.
- **`--dry-run`** renders the full plan — placement, GPU slices, and the exact commands —
  without touching a cluster, so most recipe mistakes surface before you queue anything.
- **`docker` backend** runs the same containerized tasks on one box, reserving GPUs per task
  so several `sflow run` sessions can share a workstation without colliding.
- **Composable fragments** — a workload YAML pairs with a backend fragment, so the recipe you
  proved locally is the one you submit to the cluster. See
  [`examples/modular/backend_agnostic/`](examples/modular/backend_agnostic/).

**The same workload, three platforms.** [`examples/modular/backend_agnostic/`](examples/modular/backend_agnostic/)
composes a backend fragment + a workload + a benchmark. Moving platforms means swapping one `-f`:

```bash
# Kubernetes
sflow run -f backends/k8s.yaml    -f workloads/dynamo_common.yaml -f workloads/dynamo_trtllm.yaml -f benchmark.yaml
# Slurm — same workload, same benchmark
sflow run -f backends/slurm.yaml  -f workloads/dynamo_common.yaml -f workloads/dynamo_trtllm.yaml -f benchmark.yaml
# One local box, containers
sflow run -f backends/docker.yaml -f workloads/sglang_serve.yaml -f benchmark.yaml
```

## CLI at a Glance

| Command | Purpose | Key Flags |
|---------|---------|-----------|
| `sflow run` | Execute a workflow | `--dry-run` `--tui` `--set` `-f` (multi-file) |
| `sflow batch` | Generate / submit sbatch scripts | `--submit` `--bulk-input` `--row` |
| `sflow compose` | Merge multiple YAMLs | `--resolve` `--missable-tasks` `-o` |
| `sflow visualize` | Render DAG graph | `--format png/svg/mermaid` |
| `sflow sample` | List / copy examples | `--list` `-o` |
| `sflow skill` | Export AI agent skills | `--list` `-o` |
| `sflow upgrade` | Reinstall sflow in place (alias: `sflow update`) | `--branch` `--dry-run` `--force` |

## Introduction

Start here:

- **User docs**: `docs/user/intro.md`
- **Release notes**: `docs/release_notes/` — start with
  [v0.3.1](docs/release_notes/RELEASE_NOTES_v0.3.1.md) for the current breaking changes and
  upgrade checklist
- **PLC docs**:
  - SPP: `docs/plc/sflow_spp.md`
  - SRD: `docs/plc/sflow_srd.md`
  - SADD: `docs/plc/sflow_sadd.md`

## Quickstart

If you just want to validate the workflow engine locally (no Slurm required):

```bash
uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"

sflow run --file examples/self_contained/local/hello_world.yaml --tui
```

That example is the whole file — no backend block, no operator, no boilerplate:

```yaml
version: "0.1"

variables:
  WHO:
    description: "who to greet"
    value: Nvidia

workflow:
  name: local_hello_world
  tasks:
    - name: hello
      script:
        - echo "Hello ${WHO}"
```

With nothing declared, sflow runs on the `local` backend with its default operator. Add a
`backends:` block later and the same tasks run under Docker, Slurm or Kubernetes.

## Development Setup

This guide will help you set up the development environment for contributing to `sflow`.

### Prerequisites

- **Python 3.10 or higher**

  ```bash
  python --version  # Check your Python version
  ```

- **uv** (Python package installer and resolver)

  If you don't have `uv` installed, you can install it using:

  ```bash
  curl -LsSf https://astral.sh/uv/install.sh | sh
  ```

  Or via pip:

  ```bash
  pip install uv
  ```

### Getting Started

1. **Clone the repository**

   ```bash
   git clone <repository-url>
   cd sflow
   ```

2. **Create a virtual environment**

   ```bash
   uv venv
   ```

3. **Activate the virtual environment**

   ```bash
   source .venv/bin/activate
   ```

4. **Install the project with development dependencies**

   ```bash
   uv pip install -e ".[dev]"
   ```

   This will install:

   - The `sflow` package in editable mode
   - All runtime dependencies (typer, pydantic, pyyaml, etc.)
   - Development tools (pytest, pytest-cov, ipython, etc.)

5. **Run unit tests to validate your setup**

   ```bash
   pytest
   ```

## Contributing to the project

Please follow the Git Flow process defined here: https://coda.io/d/_d0pJwz5IXhP/Manage-Git-branches-and-workflows_suLlBLYx
