Metadata-Version: 2.4
Name: slurm-job-analyzer
Version: 0.1.1
Summary: Parse Slurm sacct exports: GPU waste reports, utilization summaries, and queue analytics for HPC clusters
Author-email: Naman Adep <adepnaman@gmail.com>
License: MIT License
        
        Copyright (c) 2024 Naman
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/namanadep/slurm-job-analyzer
Project-URL: Repository, https://github.com/namanadep/slurm-job-analyzer
Project-URL: Bug Tracker, https://github.com/namanadep/slurm-job-analyzer/issues
Keywords: slurm,hpc,gpu,scheduler,sacct,utilization,benchmarking,nvidia,rocm,cluster
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# slurm-job-analyzer

[![PyPI version](https://img.shields.io/pypi/v/slurm-job-analyzer)](https://pypi.org/project/slurm-job-analyzer/)
[![Python versions](https://img.shields.io/pypi/pyversions/slurm-job-analyzer)](https://pypi.org/project/slurm-job-analyzer/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![CI](https://github.com/namanadep/slurm-job-analyzer/actions/workflows/publish.yml/badge.svg)](https://github.com/namanadep/slurm-job-analyzer/actions)

> Parse Slurm `sacct` exports to identify GPU waste, measure utilization efficiency,
> and produce per-partition queue analytics — without touching the scheduler.

Works with any Slurm cluster: NVIDIA (H100/H200/A100), AMD (MI300X), on-prem or cloud.

---

## Why this exists

HPC clusters routinely see 30–60 % of allocated GPU-hours go unused — jobs request
8 GPUs but run at 5 % utilization. Standard `sacct` output is hard to parse at scale.
`slurm-analyze` turns raw sacct TSV exports into actionable reports in one command.

---

## Install

```bash
pip install slurm-job-analyzer
```

Requires Python 3.10+. No CUDA, no cluster access needed — works on any machine
with a sacct export file.

---

## Quick start

### 1. Export from Slurm

```bash
sacct \
  --starttime=2024-01-01 \
  --endtime=2024-12-31 \
  --format=JobID,JobName,Partition,State,Elapsed,AllocCPUS,AllocTRES,ReqMem,MaxRSS,NodeList \
  --parsable2 \
  > jobs.tsv
```

### 2. Partition utilization summary

```bash
slurm-analyze sacct --input jobs.tsv --out results/summary.csv
```

```
Partition summary (1423 jobs parsed):
  partition   jobs  total_wall_h  avg_wall_min  completed_pct  total_gpu_hours  gpu_nodes
  ---------   ----  ------------  ------------  -------------  ---------------  ---------
  gpu          847        4821.3          34.2         0.8900          9821.44         12
  gpu-debug    312         183.1           3.6         0.9400           124.80          4
  preempt      264        1204.7          27.4         0.7100          2340.12          8
```

### 3. GPU waste report

Combine sacct with `nvidia-smi` snapshots for per-job efficiency verdicts:

```bash
slurm-analyze gpu-waste \
  --input jobs.tsv \
  --gpu-log snapshots.csv \
  --idle-threshold 20 \
  --out results/gpu_waste.csv
```

```
GPU waste report (847 GPU jobs):
  job_id  job_name      alloc_gpus  elapsed_h  gpu_hours  avg_util_pct  verdict
  ------  --------      ----------  ---------  ---------  ------------  -------
  10042   train_llama   8           12.3        98.4       87.2          EFFICIENT
  10051   eval_run      4            2.1         8.4        6.1          IDLE_WASTE
  10063   finetune      8            5.0        40.0       52.3          UNDERUTILIZED

Summary:
  Total GPU-hours allocated:   9821.44 h
  EFFICIENT (>=80% util):      312
  UNDERUTILIZED (20-79%):      401
  IDLE_WASTE (<20% util):       98
  MEM_OVERALLOC (mem <50%):     36
```

### 4. Queue / partition resource summary

```bash
slurm-analyze queue-wait --input jobs.tsv
```

### 5. GPU utilization timeline

```bash
slurm-analyze timeline --job 10042 --snapshots snapshots.csv
```

---

## GPU snapshot format

Collect snapshots per node with:

```bash
nvidia-smi \
  --query-gpu=timestamp,name,utilization.gpu,temperature.gpu,power.draw,memory.used,memory.total \
  --format=csv,noheader,nounits \
  --loop=10
```

Then transform into the expected CSV with columns:
`timestamp_s, node, gpu_index, utilization_pct, temperature_c, power_w, memory_used_mib, memory_total_mib`

---

## Python API

```python
from slurm_analyze import read_sacct_tsv, gpu_waste_report, summarize_by_partition
from pathlib import Path

rows = read_sacct_tsv(Path("jobs.tsv"))
summary = summarize_by_partition(rows)

for partition in summary:
    print(f"{partition['partition']}: {partition['total_gpu_hours']:.1f} GPU-hours")

reports = gpu_waste_report(rows, snapshots=[], idle_threshold_pct=20.0)
wasted = [r for r in reports if r.verdict == "IDLE_WASTE"]
print(f"{len(wasted)} jobs wasted GPU time")
```

---

## CLI reference

```
slurm-analyze --version

slurm-analyze sacct       --input FILE [--out FILE]
slurm-analyze gpu-waste   --input FILE [--gpu-log FILE] [--out FILE] [--idle-threshold N]
slurm-analyze queue-wait  --input FILE [--out FILE]
slurm-analyze timeline    --job JOB_ID --snapshots FILE
```

---

## Development

```bash
git clone https://github.com/namanadep/slurm-job-analyzer
cd slurm-job-analyzer
pip install -e ".[dev]"
pytest
```

---

## License

MIT — see [LICENSE](LICENSE).
