Metadata-Version: 2.5
Name: domyn-swarm
Version: 0.35.0
Summary: CLI and programmatic API to launch LLM inference endpoints on Slurm, using simple configuration files
Project-URL: Homepage, https://www.domyn.com/
Project-URL: Repository, https://github.com/DomynAI/domyn-swarm
Project-URL: Issues, https://github.com/DomynAI/domyn-swarm/issues
Project-URL: Changelog, https://github.com/DomynAI/domyn-swarm/blob/main/CHANGELOG.md
Author-email: Federico D'Ambrosio <federico.dambrosio@domyn.com>
Maintainer-email: Federico D'Ambrosio <federico.dambrosio@domyn.com>, Alessandro Rognoni <alessandro.rognoni@domyn.com>
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: <3.15,>=3.11
Requires-Dist: alembic>=1.17.1
Requires-Dist: blake3>=1.0.5
Requires-Dist: deprecated>=1.2.18
Requires-Dist: fsspec>=2025.3.0
Requires-Dist: huggingface-hub>=1.16.0
Requires-Dist: jinja2>=3.1.6
Requires-Dist: openai>=1.88.0
Requires-Dist: pandas>=1.5.3
Requires-Dist: pyarrow>=22.0.0
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pydantic<3,>=2.11.7
Requires-Dist: python-ulid[pydantic]>=3.1.0
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: requests>=2.34.2
Requires-Dist: rich<15,>=14.0.0
Requires-Dist: sqlalchemy>=2.0.44
Requires-Dist: tenacity<10,>=9.0.0
Requires-Dist: typer>=0.16.0
Provides-Extra: all
Requires-Dist: leptonai>=0.27.2; (python_version < '3.14') and extra == 'all'
Requires-Dist: polars>=1.31.0; extra == 'all'
Requires-Dist: ray[data]>=2.53.0; extra == 'all'
Provides-Extra: lepton
Requires-Dist: leptonai>=0.27.2; (python_version < '3.14') and extra == 'lepton'
Provides-Extra: polars
Requires-Dist: polars>=1.31.0; extra == 'polars'
Provides-Extra: ray
Requires-Dist: ray[data]>=2.55.0; extra == 'ray'
Description-Content-Type: text/markdown

<div align="center">

<img src="https://raw.githubusercontent.com/DomynAI/domyn-swarm/main/static/domyn-swarm-logo-primary.svg" alt="Domyn Swarm" height="100">

**Deploy LLMs and run resumable batch inference on Slurm and NVIDIA DGX Cloud Lepton.**

domyn-swarm is a CLI and Python library that combines model deployment and batch processing in one
workflow. vLLM handles inference. domyn-swarm manages replicas, load balancing, and job execution.

[![Documentation](https://img.shields.io/badge/docs-latest-blue)](https://domynswarm.domyn.com/)
[![CI](https://github.com/DomynAI/domyn-swarm/actions/workflows/ci.yaml/badge.svg)](https://github.com/DomynAI/domyn-swarm/actions/workflows/ci.yaml)
[![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/DomynAI/domyn-swarm/badges/coverage.json)](https://github.com/DomynAI/domyn-swarm/actions/workflows/ci.yaml)
[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-brightgreen)](https://github.com/DomynAI/domyn-swarm/blob/main/pyproject.toml)
[![License - Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue)](https://github.com/DomynAI/domyn-swarm/blob/main/LICENSE)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Pyright](https://microsoft.github.io/pyright/img/pyright_badge.svg)](https://github.com/microsoft/pyright)

<a href="#quick-start">Quick start</a> •
<a href="#write-your-own-job">Custom jobs</a> •
<a href="#backends">Backends</a> •
<a href="#documentation">Documentation</a> •
<a href="https://github.com/DomynAI/domyn-swarm/blob/main/CONTRIBUTING.md">Contributing</a>

</div>

[![Domyn Swarm CLI walkthrough: configure, launch, inspect, submit a batch job, monitor, and shut down a swarm](https://raw.githubusercontent.com/DomynAI/domyn-swarm/main/static/domyn-swarm-cli-demo.gif)](https://github.com/DomynAI/domyn-swarm/blob/main/static/domyn-swarm-cli-demo.mp4)

<p align="center"><sub>CLI walkthrough — click the demo to open the MP4.</sub></p>

---

<table>
<tr>
<td width="50%" valign="top">

### ⚡ Serve

- **YAML configuration** for model and replica settings
- **OpenAI-compatible endpoints** backed by vLLM
- **Multiple replicas** behind a single load-balanced URL
- **Lifecycle commands** to launch, inspect, and stop named swarms

</td>
<td width="50%" valign="top">

### 🚀 Run

- **Batch inference** over Parquet datasets
- **Checkpointing and resume** for interrupted jobs
- **Retries with backoff** and **bounded concurrency**
- **Input sharding** for parallel job execution

</td>
</tr>
</table>

On Slurm, optional [Prometheus metrics and dashboards](https://domynswarm.domyn.com/latest/guides/metrics.html)
show inference throughput, queue depth, and GPU utilization.

---

## Quick start

```bash
uv tool install --python 3.12 domyn-swarm
```

<details>
<summary>Extras and other install methods</summary>

```bash
pip install domyn-swarm             # into the current environment

# Optional extras
pip install 'domyn-swarm[lepton]'   # NVIDIA DGX Cloud Lepton backend
pip install 'domyn-swarm[polars]'   # Polars data backend
pip install 'domyn-swarm[ray]'      # Ray Data backend
pip install 'domyn-swarm[all]'      # everything
```

`uv add`, installs from source and the Lepton `lep login` step are covered in
[Installation](https://domynswarm.domyn.com/latest/getting-started/installation.html).

</details>

**1. Describe the swarm**

The [`examples/configs/`](https://github.com/DomynAI/domyn-swarm/blob/main/examples/configs/) directory holds ready-made configurations. This one
serves NVIDIA Nemotron-3-Super-120B-A12B across two replicas of four GPUs each, following the
[vLLM recipe](https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16)
([`nemotron_3_super.yaml`](https://github.com/DomynAI/domyn-swarm/blob/main/examples/configs/nemotron_3_super.yaml)). Set the image paths,
partition, account, and QoS for your cluster:

```yaml
# NVIDIA Nemotron-3-Super-120B-A12B (BF16) — a ~120B latent-MoE with ~12B active
# parameters. Serving flags follow https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
# The recipe's `--served-model-name` is deliberately omitted: jobs address the
# endpoint by the `model` value below, so renaming the served model breaks them.
model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
name: nemotron-3-super-120b
revision: main
gpus_per_replica: 4       # tensor-parallel size 4, one node on most clusters
replicas: 2               # two independent replicas behind the load balancer
image: "/path/to/vllm.sif"   # needs vLLM >= 0.17.1
args: >-
  --tensor-parallel-size 4
  --kv-cache-dtype fp8
  --max-model-len 262144
  --trust-remote-code
  --enable-auto-tool-choice
  --tool-call-parser qwen3_xml
  --reasoning-parser nemotron_v3
env:
  HF_HOME: /path/to/shared_hf_cache
wait_endpoint_s: 3600
backend:
  type: slurm
  partition: partition_name
  account: account_name
  qos: qos_name
  endpoint:
    port: 9001
    cpus_per_task: 4
    wall_time: "36:00:00"
    nginx_image: "/path/to/nginx.sif"
```

### 2. Launch the swarm

`up` submits the replica and load-balancer jobs and returns once the endpoint answers a health check.
Every swarm name combines the configured name with a short suffix. `up` writes
this name to standard output so that a script can capture it:

```console
$ SWARM=$(domyn-swarm up -c examples/configs/nemotron_3_super.yaml)
[08/31/26 11:56:44] INFO     Creating deployment nemotron-3-super-120b on slurm...
                    INFO     Submitted replicas job 15427318 with command: sbatch --parsable …
                    INFO     Submitted load balancer job 15427319 with command: sbatch --parsable …
                    INFO     LB healthy → http://lrdn0431:9001/v1

$ echo $SWARM
nemotron-3-super-120b-01m1bq26m7
```

### 3. Run a batch job

```console
$ domyn-swarm job submit \
    --name $SWARM \
    --input examples/data/chat_completion.parquet \
    --output results.parquet
[MainThread] Checkpoint flushed 16 rows, new total: 16
[MainThread] Checkpoint flushed 16 rows, new total: 32
[MainThread] Checkpoint flushed 16 rows, new total: 48
[MainThread] Checkpoint flushed 16 rows, new total: 64
[MainThread] Checkpoint flushed 1 rows, new total: 65
[MainThread] Processing all items in worker: 100%|██████████| 65/65 [01:12<00:00,  1.11s/sample]
```

This is the built-in chat-completion job reading the `messages` column. The example dataset has 65
rows. By default, the job writes a checkpoint after every 16 rows. An interrupted
run continues from the last checkpoint. A higher `replicas` value distributes
requests across more replicas.

### 4. Inspect and stop the swarm

```console
$ domyn-swarm status $SWARM      # live panel: replica states, endpoint, recorded jobs

$ domyn-swarm down -y $SWARM
✅ Swarm nemotron-3-super-120b-01m1bq26m7 shutdown request sent.
```

The [Quickstart](https://domynswarm.domyn.com/latest/getting-started/quickstart.html) covers the same
run step by step.

---

## Write your own job

Subclass `SwarmJob` and implement `transform_items` to define how inputs are processed.
The framework handles concurrency, retries, and checkpointing.

```python
from typing import Any

from domyn_swarm.jobs import SwarmJob


class SentimentJob(SwarmJob):
    """Classify each prompt as positive, negative, or neutral."""

    async def transform_items(self, items: list[Any]) -> list[str]:
        results = []
        for prompt in items:
            resp = await self.client.chat.completions.create(
                model=self.model,
                messages=[
                    {"role": "system", "content": "Reply only: positive, negative, or neutral."},
                    {"role": "user", "content": prompt},
                ],
            )
            results.append(resp.choices[0].message.content or "")
        return results
```

This example uses the swarm's configured client and writes labels to the default `results` column.
See [Your first custom job](https://domynswarm.domyn.com/latest/getting-started/first-custom-job.html)
for input and output requirements, import setup, and CLI submission.

Built-in jobs cover chat completion, multi-turn conversations, perplexity, and translation.
See the [Jobs API reference](https://domynswarm.domyn.com/latest/reference/api/jobs.html).

---

## Backends

The same core commands are available on both platforms: `up`, `job submit`,
`status`, and `down`.

- Slurm is included in the base installation. See [Slurm setup](https://domynswarm.domyn.com/latest/guides/slurm.html).
- NVIDIA DGX Cloud Lepton requires the `lepton` extra and workspace credentials. See [Lepton setup](https://domynswarm.domyn.com/latest/guides/lepton.html).

Job input and output support pandas (default), polars, and Ray Data.
See [Choosing a data backend](https://domynswarm.domyn.com/latest/guides/data-backends.html).

To add support for another platform, see [Implementing a backend](https://domynswarm.domyn.com/latest/guides/implementing-a-backend.html).

---

## Documentation

The full documentation is published at **[domynswarm.domyn.com](https://domynswarm.domyn.com/)**.

| | |
| --- | --- |
| [Getting started](https://domynswarm.domyn.com/latest/getting-started/index.html) | install, launch a swarm, write your first custom job |
| [Guides](https://domynswarm.domyn.com/latest/guides/index.html) | Slurm and Lepton, checkpointing, sharding, data backends, monitoring |
| [Concepts](https://domynswarm.domyn.com/latest/concepts/index.html) | architecture, backend protocols, the `SwarmJob` lifecycle, configuration |
| [Reference](https://domynswarm.domyn.com/latest/reference/index.html) | CLI, configuration, environment variables, and Python API generated from source |

---

## Contributing

Issues and pull requests are welcome. [CONTRIBUTING.md](https://github.com/DomynAI/domyn-swarm/blob/main/CONTRIBUTING.md) covers the development
setup, coding style, and commit conventions. Participation is governed by our
[Code of Conduct](https://github.com/DomynAI/domyn-swarm/blob/main/CODE_OF_CONDUCT.md). To report a vulnerability, follow our
[security policy](https://github.com/DomynAI/domyn-swarm/blob/main/SECURITY.md).

## License

The project uses the Apache License 2.0. See [LICENSE](https://github.com/DomynAI/domyn-swarm/blob/main/LICENSE) and [NOTICE](https://github.com/DomynAI/domyn-swarm/blob/main/NOTICE).

## Citation

```bibtex
@software{domyn_swarm,
  author  = {D'Ambrosio, Federico and Rognoni, Alessandro},
  title   = {domyn-swarm: LLM serving endpoints and high-throughput batch jobs on Slurm and DGX Cloud Lepton},
  year     = {2026},
  url     = {https://github.com/DomynAI/domyn-swarm},
  license = {Apache-2.0}
}
```

## Acknowledgements

Built on [vLLM](https://github.com/vllm-project/vllm) and [Ray](https://github.com/ray-project/ray),
with optional support for [NVIDIA DGX Cloud Lepton](https://www.nvidia.com/en-us/data-center/dgx-cloud-lepton/)
through the Lepton SDK.
