Metadata-Version: 2.4
Name: scitex-hpc
Version: 0.8.1
Summary: Generic SLURM dispatch (srun, sbatch, sync, poll, fetch) for the SciTeX ecosystem — login nodes never run compute
Author-email: Yusuke Watanabe <ywatanabe@scitex.ai>
License-Expression: AGPL-3.0-only
Project-URL: Homepage, https://github.com/ywatanabe1989/scitex-hpc
Project-URL: Repository, https://github.com/ywatanabe1989/scitex-hpc
Project-URL: Documentation, https://scitex-hpc.readthedocs.io
Keywords: scitex,hpc,slurm,srun,sbatch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Clustering
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=5.1
Requires-Dist: click>=8.0
Requires-Dist: scitex-config>=0.3.0
Requires-Dist: scitex-ssh>=1.0.0
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytest-timeout; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: scitex-dev>=0.11.7; extra == "dev"
Provides-Extra: mcp
Requires-Dist: fastmcp>=0.1.0; extra == "mcp"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=2.0; extra == "docs"
Requires-Dist: myst-parser>=2.0; extra == "docs"
Requires-Dist: sphinx-copybutton>=0.5; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.25; extra == "docs"
Requires-Dist: sphinx-click>=5.0; extra == "docs"
Requires-Dist: scitex-dev>=0.11.7; extra == "docs"
Provides-Extra: all
Requires-Dist: scitex-hpc[dev,docs,mcp]; extra == "all"
Dynamic: license-file

# scitex-hpc

<p align="center">
  <a href="https://scitex.ai">
    <img src="docs/scitex-logo-blue-cropped.png" alt="SciTeX" width="400">
  </a>
</p>

<p align="center"><b>Generic SLURM dispatch — `srun` / `sbatch` / reservations / sync / poll / fetch — for any HPC cluster.</b></p>

<p align="center">
  <a href="https://scitex-hpc.readthedocs.io/">Full Documentation</a> · <code>uv pip install scitex-hpc[all]</code>
</p>

<!-- scitex-badges:start -->
<p align="center">
  <a href="https://pypi.org/project/scitex-hpc/"><img src="https://img.shields.io/pypi/v/scitex-hpc.svg" alt="PyPI"></a>
  <a href="https://pypi.org/project/scitex-hpc/"><img src="https://img.shields.io/pypi/pyversions/scitex-hpc.svg" alt="Python"></a>
  <a href="https://github.com/ywatanabe1989/scitex-hpc/actions/workflows/test.yml"><img src="https://github.com/ywatanabe1989/scitex-hpc/actions/workflows/test.yml/badge.svg" alt="Tests"></a>
  <a href="https://codecov.io/gh/ywatanabe1989/scitex-hpc"><img src="https://codecov.io/gh/ywatanabe1989/scitex-hpc/graph/badge.svg" alt="Coverage"></a>
  <a href="https://scitex-hpc.readthedocs.io/en/latest/"><img src="https://readthedocs.org/projects/scitex-hpc/badge/?version=latest" alt="Docs"></a>
  <a href="https://www.gnu.org/licenses/agpl-3.0"><img src="https://img.shields.io/badge/license-AGPL_v3-blue.svg" alt="License: AGPL v3"></a>
</p>
<!-- scitex-badges:end -->

---

## Problem and Solution

| # | Problem | Solution |
|---|---------|----------|
| 1 | **Login nodes silently run compute** — sysadmins kill stray processes, jobs die unannounced | **Every command wrapped in `srun` / `sbatch`** via login-shell SSH so SLURM modules load correctly |
| 2 | **Queue wait dominates iteration** for short multi-agent / dev workflows | **`Reservation.book(..., persistent=True)`** — book a node once, `exec()` many short commands inside the same allocation; SIGUSR1 trap auto-resubmits at walltime |
| 3 | **Per-cluster knob soup** (partition, cpus, time, mem) repeated in every job script | **`SCITEX_HPC_*` env overrides** + `JobConfig` defaults for spartan/sapphire — script once, deploy anywhere |

## Installation

```bash
pip install scitex-hpc
```

## Demo

```bash
# Submit + watch a SLURM job (via the local scheduler config)
scitex-hpc submit-sbatch examples/00_minimal.sbatch
scitex-hpc poll-job <jobid>
scitex-hpc fetch-result <jobid> --out ./results/

# Or interactive: book a chunk, attach, work
scitex-hpc dispatch-srun --time 1:00:00 --gpus 1
```

```mermaid
graph LR
    Local["Local laptop\n(scitex-hpc CLI / Python API)"] -->|submit-sbatch| Login["HPC login node"]
    Local -->|dispatch-srun| Login
    Login -->|salloc/sbatch| Compute["Compute / GPU nodes"]
    Compute -->|stdout/err + artefacts| Sync["sync-project rsync"]
    Sync --> Local
```

## Architecture

`scitex-hpc` is a thin client over the SLURM CLI (`squeue`, `sbatch`,
`srun`, `salloc`). Site-specific config lives in
`~/.scitex/hpc/config.yaml`; nothing else is host-aware. The package is
fleet-agnostic: it does not know about scitex-orochi or any specific
HPC site beyond its config.

```
scitex_hpc/
├── _cli/                       ← `scitex-hpc` Click commands
│   ├── submit_sbatch.py        ← submit-sbatch verb
│   ├── dispatch_srun.py        ← dispatch-srun verb
│   ├── poll_job.py             ← poll-job verb
│   ├── fetch_result.py         ← fetch-result verb
│   └── sync_project.py         ← sync-project verb
├── _slurm/                     ← thin wrappers over squeue / sbatch / srun / salloc
├── _state/                     ← job-state cache (`~/.scitex/hpc/runtime/`)
└── config/                     ← YAML schema for site config

~/.scitex/hpc/                  ← user state (gitignored, host-specific)
├── config.yaml                 ← partitions, walltimes, module loads, GPU types
└── runtime/                    ← per-job caches, result-rsync metadata
```

## 1 Interfaces

<details open>
<summary><strong>Python API ⭐⭐⭐ (primary)</strong></summary>

<br>

```python
from scitex_hpc import JobConfig, srun, sbatch, sync, poll_job, fetch_result

cfg = JobConfig(
    project="scitex-dsp",
    command="pip install -e '.[dev]' -q && python -m pytest tests/ -n 16",
    host="spartan", partition="sapphire",
    cpus=16, time="00:30:00", mem="64G",
)

sync(cfg)                           # 1. push local sources to the cluster
exit_code = srun(cfg)               # 2a. blocking interactive run
job_id = sbatch(cfg)                # 2b. async batch submission
print(poll_job(cfg, job_id))        # {'state': 'COMPLETED', 'exit_code': '0:0', ...}
fetch_result(cfg, job_id)           # downloads the .out file
```

### Reservations (book once, exec many)

```python
from scitex_hpc import JobConfig, Reservation

res = Reservation.book(
    JobConfig(project="dev-pool", host="spartan", partition="cascade",
              cpus=8, mem="32G", time="7-0"),
    persistent=True,                # walltime auto-resubmit via SIGUSR1
)

res.exec("hostname")
res.exec(["python", "-m", "unittest", "discover"])
res.attach(cmd="bash")              # interactive shell on the compute node

# Look up later by friendly name (state in ~/.scitex/hpc/leases/)
res = Reservation.get("dev-pool")
res.release()                       # scancel + clear state
```

</details>

<details>
<summary><strong>CLI ⭐⭐</strong></summary>

<br>

```bash
scitex-hpc reservations book dev-pool --host spartan --cpus 8 --mem 32G --time 7-0 --persistent
scitex-hpc reservations list
scitex-hpc reservations exec dev-pool 'hostname'
scitex-hpc reservations attach dev-pool
scitex-hpc reservations cancel dev-pool
```

</details>

## Defaults & overrides

Every `JobConfig` field has a `SCITEX_HPC_*` env-var override:

| Field | Default | Env override |
|---|---|---|
| `host` | `spartan` | `SCITEX_HPC_HOST` |
| `partition` | `sapphire` | `SCITEX_HPC_PARTITION` |
| `cpus` | `16` | `SCITEX_HPC_CPUS` |
| `time` | `00:20:00` | `SCITEX_HPC_TIME` |
| `mem` | `128G` | `SCITEX_HPC_MEM` |
| `remote_base` | `~/proj` | `SCITEX_HPC_REMOTE_BASE` |

Resolution priority: explicit `JobConfig` field → env var → built-in default.

## Walltime auto-resubmit (`persistent=True`)

When `persistent=True`, scitex-hpc:

1. Adds `#SBATCH --signal=B:USR1@3600` so SLURM signals the script 1 h before walltime.
2. Wraps the sbatch body with a SIGUSR1 trap that calls `sbatch "$0"` to resubmit itself.
3. The friendly name (`dev-pool`) stays stable across resubmits; the SLURM `job_id` changes.

```python
res = Reservation.get("dev-pool")
res.refresh()                       # squeue --user --name=dev-pool
res.exec("...")                     # uses the new job_id
```

SLURM's documented signaling mechanism — no custom daemon. Compatible with HPC policies that ban persistent user-space daemons. SSH ControlMaster pooling on the calling host amortizes per-`exec()` handshake cost.

## Part of SciTeX

`scitex-hpc` is part of [**SciTeX**](https://scitex.ai). Install via
the umbrella with `pip install scitex[hpc]` to use as
`scitex.hpc` (Python) or `scitex hpc ...` (CLI).

>Four Freedoms for Research
>
>0. The freedom to **run** your research anywhere — your machine, your terms.
>1. The freedom to **study** how every step works — from raw data to final manuscript.
>2. The freedom to **redistribute** your workflows, not just your papers.
>3. The freedom to **modify** any module and share improvements with the community.
>
>AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.

---

<p align="center">
  <a href="https://scitex.ai" target="_blank"><img src="docs/scitex-icon-navy-inverted.png" alt="SciTeX" width="40"/></a>
</p>
