Metadata-Version: 2.4
Name: slurmclust
Version: 0.3.0
Summary: Show SLURM cluster usage: capacity, QOS quotas, and who holds what
Home-page: 
Author: Badr Youbi Idrissi
Author-email: badryoubiidrissi@gmail.com
License: MIT
Keywords: slurm cluster gpu monitoring hpc qos
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Clustering
Requires-Python: >=3.7.0
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: requires-python
Dynamic: summary


# clust — see who is using the SLURM cluster, and how much of your quota is left

`clust` reads SLURM directly and prints a dashboard: how much of the cluster is
busy, how each QOS stands against its GPU quota, and which users are holding
what. A full refresh takes well under a second on a cluster with thousands of
nodes and jobs.

All examples below are rendered from a **fictional** cluster.

## Install

```bash
pip install slurmclust     # installs the `clust` command
```

No third-party Python dependencies. Needs `scontrol` and `squeue` on `PATH`;
`sacctmgr` too if you want quotas. Tested against SLURM 25.05; the `squeue`
`--Format` fields it relies on (`tres-alloc`, `PendingTime`, `Account`) have been
available since SLURM 20.11.

## Usage

```bash
clust                     # your QOS and accounts: quota, usage, top consumer
clust -q gpu_vision_high  # per-user breakdown of one QOS — who do I ping?
clust -a                  # whole cluster: partitions, free capacity, top users, queue
clust -p gpu,gpu-big      # cluster view restricted to some partitions
clust -A vision           # focus on an account instead of your own
clust -u someone          # resolve QOS access as if you were another user
clust -g a100             # restrict to one GPU type (case-insensitive substring)
clust -f bigmem           # restrict to nodes with given features
clust -w 10               # refresh every 10s
clust --ascii             # bars without block characters
clust --debug-timing      # how long each SLURM call took
```

### Default view — your QOS

With no arguments, `clust` resolves the QOS you can actually submit to (via
`sacctmgr show assoc user=$USER`) and shows each against the GPU quota SLURM
enforces on it (`GrpTRES gres/gpu`):

```
CLUSTER  mycluster  96 nodes · 672 GPUs · 9 running · 4 queued                                12:50:10
  ███████████████████████████▓▒░······   79.6%   535 used · 8 blocked · 18 drain · 8 down · 103 free

YOUR QOS   user alice · quota is the QOS GrpTRES gres/gpu limit SLURM enforces
  QOS               PART      QUOTA   USED   %QUOTA   PEND   USERS   TOP CONSUMER
  gpu_vision_high   gpu-big     256    224      88%     96       3   alice (128)    ██████████████████··
  gpu_shared        gpu         512    128      25%     16       2   dave (80)      █████···············
  gpu_lowest        gpu         672     16       2%      -       1   dave (16)      ····················

ACCOUNTS
  ACCOUNT   QUOTA   USED   %QUOTA   PEND   USERS
  vision      320    272      85%     96       3   ███████████████████████████████·····
```

QOS rows are sorted by how close they are to their quota, so whatever is
contended shows up first. Usage can exceed 100% where SLURM lets a QOS overflow
its group limit.

### Drill-down — who is holding the GPUs

```
$ clust -q gpu_vision_high

gpu_vision_high  ████████████████████████████████····  224 / 256 GPUs  (88%)   +96 pending
  USER    GPUS   SHARE   JOBS   NODES   OLDEST   PEND   WAITING
  alice    128     57%      1      16    2d06h     32       30m   ███████████████···········
  bob       64     29%      1       8   18h22m     64     2h35m   ███████···················
  carol     32     14%      1       4    3d01h      -         -   ████······················
```

`OLDEST` is the age of the user's longest-running job, `PEND` the GPUs they have
queued behind it, and `WAITING` how long that queued work has been sitting.

### Cluster view

```
$ clust -a

PARTITIONS
  PARTITION   NODES   GPUS   USED   FREE   BLOCKED   DRAIN   DOWN   UTIL
  gpu-big        60    480    387     67         0      18      8    81%   ██████████████████████▒░····
  gpu            24    192    148     36         8       0      0    77%   ██████████████████████▓·····
  cpu            12      –      –      –         –       –      –      –   cpu only

FREE CAPACITY   GPUs obtainable in whole chunks of N per node
  PARTITION   ≥1 GPU   ≥2 GPU   ≥4 GPU   ≥8 GPU
  gpu-big         67       60       60       32
  gpu             36       36       24        0

TOP USERS   by allocated GPUs · 6 active · bars relative to the leader
  USER    ACCOUNT   GPUS   SHARE   JOBS   NODES   OLDEST   PEND
  alice   vision     176   36.1%      2      22    2d06h     32   ██████████████████████████████
  dave    speech      96   19.7%      2      12    1d04h     16   ████████████████··············
  erin    nlp         96   19.7%      1      12   11h48m    128   ████████████████··············
  bob     vision      64   13.1%      1       8   18h22m     64   ███████████···················

QUEUE   4 pending jobs
  PARTITION   JOBS   GPUS REQ   USERS   LONGEST WAIT   TOP WAITER
  gpu-big        3        224       3          3d03h   erin (128)
  gpu            1         16       1            10m   dave (16)
```

Columns that are zero everywhere (`BLOCKED`, `DRAIN`, `DOWN`, `PEND`) are hidden,
so a healthy cluster shows a narrower table. Bars shrink and then drop out
entirely on narrow terminals — you lose the decoration, never the numbers.

## How capacity is counted

Every GPU falls into exactly one bucket, so the columns always add up:

| bucket    | meaning                                              |
|-----------|------------------------------------------------------|
| `used`    | allocated to a running job                            |
| `free`    | actually obtainable right now                         |
| `blocked` | idle GPU on a healthy node whose CPUs are exhausted   |
| `drain`   | idle GPU on a draining node                           |
| `down`    | GPU on a node that is down or unreachable             |

`blocked` is the "virtual GPU" idea this tool was built around: a job that takes
many CPUs and no GPUs still makes the rest of the node unusable. A node can only
offer `floor(free_cpus / cpus_per_gpu)` GPUs, whatever its idle GPU count says.

`FREE CAPACITY` reports GPUs obtainable in whole chunks per node — `≥8 GPU` is
what a job asking for a full node can actually get. The chunk sizes shown are
derived from the largest node in the selection.

Capacity comes from node state only; per-user, per-QOS and per-account
attribution comes from jobs only. QOS and account usage is reported in **raw
allocated GPUs**, because that is what SLURM's `GrpTRES` limits are enforced
against.

## Performance

`clust` deliberately avoids SLURM's `--json` output, which serialises the entire
object graph. Measured on a 3,464-node / 4,265-job cluster:

| data                         | via `--json` | what clust uses                      |
|------------------------------|--------------|--------------------------------------|
| node state                   | 20 s, 166 MB | `scontrol show node -o` — **0.07 s** |
| job list                     | 41 s, 45 MB  | `squeue -O` — **0.4 s**              |
| node list expansion per job  | 16 ms/job    | pure Python — **0 forks**            |

All four SLURM calls are issued in parallel, so wall clock is the slowest one.
Use `--debug-timing` for the breakdown.

## Handy commands

```bash
scontrol show partition -a | grep PartitionName=      # list partitions
sinfo -hNo '%f' | tr -d ' ' | tr ',' '\n' | sort -u   # list node features
sacctmgr show qos format=Name,GrpTRES                 # list QOS quotas
```

## Development

```bash
pip install -e .
python -m pytest tests/ -q        # 84 tests, no SLURM required
python -m flake8 clust.py tests/
```
