Metadata-Version: 2.4
Name: slurmscope
Version: 1.3.0
Summary: Live terminal dashboard for Slurm clusters: queue, GPUs, QoS, nodes, memory and storage.
Author: Fabio Bozzoli
License-Expression: MIT
Project-URL: Homepage, https://github.com/FabioBozzoli/slurmscope
Project-URL: Source, https://github.com/FabioBozzoli/slurmscope
Project-URL: Issues, https://github.com/FabioBozzoli/slurmscope/issues
Keywords: slurm,hpc,cluster,gpu,dashboard,tui,monitoring
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console :: Curses
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Clustering
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# slurmscope

Interactive terminal dashboard for any Slurm cluster: Python >= 3.8, standard library only, no dependencies.

## Install

    pipx install slurmscope      # recommended: isolated, puts `slurmscope` on your PATH
    pip install --user slurmscope
    python -m slurmscope         # equivalent to the `slurmscope` command

It needs the Slurm client commands (`squeue`, `scontrol`) in `PATH` on the machine where you run it,
typically a login node.

Version 1.3 adds master/follower sharing of `squeue` polling (see *Cost and data*).
Version 1.2 added an AImageLab-HPC profile based on all six **Using the Cluster** documentation pages.
Live Slurm configuration, `squota`, and `susage` remain the source of operational values.

    slurmscope                 # live, refresh every 1 s
    slurmscope -p <partition>  # start filtered on one partition
    slurmscope -u <user>       # whose jobs are highlighted / listed in the overview (default: you)
    slurmscope --page 4        # start on memory / storage
    slurmscope --once          # single overview snapshot (pipe-friendly); --no-color, --ascii, --no-mouse
    slurmscope --once --page 4 --scan-usage  # RAM, quota, and directory sizes

## Views (1 2 3 4 / Tab, or click the tab bar)
- **Overview** - KPIs + history sparklines, per-partition load, GPU models (free GPUs), your jobs
  (approximate priority rank, wait, backfill start estimate), pending reasons, top users, and a QoS summary.
  The overview adapts to the terminal height (side-by-side panels from 150 columns, lists trimmed to fit);
  it scrolls only when the terminal is too small, and the footer then shows the current line range.
  Press **Q** or **Enter** to browse **all configured QoS**, including ones with no jobs.
  Each row shows running/pending experiments, distinct users, allocated GPUs/CPUs, and
  (on wider terminals) priority and maximum wall time. Search by name, description, or user;
  sort or scroll through the complete list. **i** shows resource/job limits, policy flags,
  user activity, and pending reasons. **Enter** on a QoS opens its experiments across all
  visible users, clearing job-specific user/state/search filters; **Esc** returns to the browser.
  The partition filter still applies to experiment counts/lists; **P** cycles back to all partitions.
  QoS details distinguish shared GPU caps from individual caps and calculate visible GPU headroom
  across all partitions. Partition policy adds wall-time bounds and billing weights; associations,
  partition QoS, access rules and flags can further change the effective limits.
- **Jobs** - every job in the queue, scrollable. Running jobs of everybody or only yours.
  QoS is searchable, sortable, shown in job details and, on wide terminals, in its own column.
  Project accounts are searchable/sortable, shown on very wide terminals, and filterable with
  **a** or `-A PROJECT`. **v** opens live `sprio` factors and the job's `scontrol` SiteFactor.
  A displayed priority rank is not a guaranteed execution order or start-time estimate.
- **Nodes** - every node: state, CPU / GPU / memory bars, load, partitions.
  **v** cycles minimum GPU VRAM; `--min-vram GB` sets it initially. Values come from live GPU
  feature tags, not hardware assumptions or GPU-memory-utilization probes. Exact feature tokens
  are searchable; node detail includes a quoted `--constraint` suggestion and raw GRES/tmpfs data.
- **Memory** - cluster configured/allocated RAM, capacity on nodes that are up and not drained,
  unallocated RAM on those nodes, OS-reported free RAM, allocations by user (selected user first),
  and RAM by node. Also includes the selected user's disk quota and on-demand directory sizes. The page is compact: users/nodes are packed into columns and trimmed to fit the terminal height;
  it scrolls only when the terminal is too small.

## AImageLab-HPC integration

```bash
slurmscope --profile aimagelab
slurmscope --profile aimagelab --page 4
slurmscope --profile aimagelab --page 3 --min-vram 24 -p all_usr_prod
slurmscope --profile aimagelab --page 2 -A my_project
```

The default `--profile auto` recognizes an AImageLab/AILB cluster name, an `ailb-*` hostname,
or the presence of both `squota` and `susage`. `--profile generic` explicitly disables site integration.
You can force `aimagelab` when using a workstation with Slurm client tools.

- **Storage:** absolute `/homes/USER`, `/work`, and `/dres` paths; personal/project usage from
  `squota`; capacity rows for associated project paths. Project counters include collaborators.
  Recursive scans under `/work` and `/dres`, including aliases, are skipped in this profile.
- **B — Budgets:** all projects reported by `susage`, current-year remaining standard hours,
  low/exhausted balances, and original counters. The overview also shows a compact summary.
  Current-year columns are selected by their actual header; missing values stay unknown.
- **v on a job — Priority:** live component values, with the separate site factor. Recent-use
  fairness changes priority; exhausted annual budgets can reject paid submissions. This follows
  the newer Budget page where its annual-budget restriction qualifies the older Priority page.
- **E — Session:** loaded modules, local Python, job/session context and CUDA visibility.
  On `ailb-login-02`, an active CUDA 12+ module produces the documented CUDA 11.8 compatibility hint.
  No modules are loaded or unloaded, and another job's environment is never inferred.
- **D — Guide:** brief usage guidance, storage lifecycle, execution/transfer hints and links to
  every source page. The reference was reviewed on 2026-11-01; live data takes precedence.

`squota` and `susage` run as the **logged-in Unix account**. `-u OTHER_USER` only changes job
highlighting and RAM summaries; it does not impersonate that user for these site commands.
Their identity is labelled in the UI. Missing commands, changed table formats and query errors
are shown explicitly; cached counters remain visible and marked stale after a failed refresh.
No job submissions, module changes, transfers or filesystem writes are performed by this integration.

| Documentation page | Integration |
|---|---|
| [Environment and Customization](https://ailb-web.ing.unimore.it/coldfront/documentation/LrU3q) | Session/module inspection and CUDA compatibility hint |
| [Scheduler and Job Submission](https://ailb-web.ing.unimore.it/coldfront/documentation/UKN8n) | Project accounts, QoS caps and partition policy |
| [Node Features and Constraints](https://ailb-web.ing.unimore.it/coldfront/documentation/flOS4) | Minimum VRAM filters and constraint tokens |
| [Budget and Accounting](https://ailb-web.ing.unimore.it/coldfront/documentation/GSOa2) | Live project budgets and balance summaries |
| [Job Priority](https://ailb-web.ing.unimore.it/coldfront/documentation/cHTQV) | Priority components, SiteFactor and approximate-rank labels |
| [File Systems and Data Management](https://ailb-web.ing.unimore.it/coldfront/documentation/UJMLD) | Site quotas, project scope and BeeGFS scan guard |

### QoS visibility

QoS definitions come from a read-only `sacctmgr show qos` query, cached for 60 seconds.
The experiment list uses jobs currently visible to `squeue`, not completed-job history.
Pending array ranges count as one queue row, matching the existing Jobs view.
Site privacy settings may hide other users' jobs. Definitions describe the accounting
database's QoS; access to submit jobs under them can still depend on your associations
and partition policies. If `sacctmgr`/SlurmDBD is unavailable or access is denied, a warning
appears and QoS found in the queue remain browsable; cached definitions are retained on failure.

### Memory and disk scope

RAM figures are **Slurm allocations, not measured process RSS**. Per-user RAM comes from
allocated TRES for active jobs; pending requests are not counted. Missing allocation data
(including the legacy `squeue -o` fallback) is marked **unknown**, not zero. Cluster allocation
is read from nodes, so it can differ from the sum of visible jobs. OS free memory is informational,
can be stale, and is reported with the number of nodes providing a value. Nodes are counted
once even when they belong to several partitions.

Disk data describes filesystems **mounted on the host running slurmscope**. Slurm does not
provide a cluster-wide filesystem inventory. By default the selected user's home and the
working directory are reported. Add shared storage paths explicitly:

```bash
slurmscope --page 4 --storage-path /scratch --storage-path /projects
slurmscope -u alice --page 4 --usage-path /scratch/alice --usage-path /projects/alice
```

`--storage-path` adds filesystem-capacity rows. `--usage-path` selects directory scan roots
(repeatable; replaces the default selected-user home). Filesystem **used** includes all users;
**available** excludes reserved blocks. Paths on the same filesystem can show the same capacity,
so those rows must not be added. The **P** filter affects RAM/jobs, not storage scope.

In the generic profile, user ownership and limits are shown using `quota -w -u USER` when available
(the standard block unit is KiB). Quota support/permissions depend on the site; some shared
filesystems require their own quota tools. If quota is unavailable, press **d** to measure your
home or configured directories. Scans use `du -skx`, stay on one filesystem, run in the background,
and time out after 180 seconds per path (a home on a network filesystem can take over 30 s); each path
shows `scanning Ns` / `queued` and its result as soon as it is ready. A directory footprint includes files of all owners in that
directory, and is not a complete ownership-based quota. Nested paths must not be summed.
Permission failures and partial scans are labelled; previous samples remain visible during a rescan.
Directory scans run only on demand, or with `--once --page 4 --scan-usage`.

## Keys
| key | action |
|---|---|
| `↑ ↓ j k`, `PgUp PgDn`, `g G`, wheel | move / page / top / bottom |
| `Enter`, double-click | details: job (resources, files, queue position, ...) or node (+ its jobs) |
| `Q`, `Enter` on overview | full QoS browser; `Q` / `Esc` returns to the overview |
| `Enter`, `i` in QoS browser | experiments of all visible users / QoS limits and user activity |
| `4`, `d`, `r` | memory/storage page / scan directories / refresh RAM and storage |
| `B`, `E`, `D` | project budgets / current session and modules / AImageLab guide |
| `a`, `v` in Jobs | selected project filter / priority components |
| `v` in Nodes | cycle minimum GPU VRAM filter |
| `l` | tail the job's stdout (follows; `↑` scrolls back, `End` follows again, `e` switches to stderr) |
| `/` | search (words are ANDed: `bob a100 pending`) |
| `m` `u` `s` `f` | only mine / only the selected user / cycle state / GPU jobs (nodes: free GPUs) |
| `o` `O`, click header | next sort column / reverse / sort by the clicked column |
| `P` | cycle partition filter (all views) |
| `c` | cancel the selected job - own jobs only, asks `y/N` |
| `x` | clear filters |
| `p`/`Space`, `+ -` | pause, refresh interval (0.5-60 s) |
| `?`, `q` | scrollable help, quit (closes the current screen first) |

Mouse capture hides terminal text selection; hold Shift to select, or use `--no-mouse`.

## Cost and data
**Master/follower polling.** Many instances polling every second would load `slurmctld`, so
the `squeue` / `scontrol show node` queries are shared: for each command one process at a time
(the *master*, elected with a file lock) runs it and publishes the output; every other
slurmscope on the same host (*followers*) reads that snapshot while it is younger than
`--share-interval` seconds (default 2; `0` disables sharing). There is no daemon: if the master
dies, the next process takes over; failures are shared too, so a sick controller is not hammered.
The status line shows `master` / `shared`. Snapshots live in a private `0700` directory per
user in the local temp dir (`--cache-dir` to change it), never shared between users, so jobs
hidden by site privacy settings cannot leak. Detail screens (`scontrol show job`) are not cached.

Per refresh: one `squeue -O ...tres-alloc...` and one `scontrol show node` (plus `squeue --start`
every 15 s if you have pending jobs; `scontrol show job` only while a detail/log screen is open).
QoS definitions refresh at most once per 60 s in a background worker. While page 4 is active,
user quota also refresh at most once per 60 s in the background.
Partition policies are cached for 60 s (`scontrol show partition -o`). The AImageLab profile
also refreshes `susage` at most once per 60 s. A priority report queries only the selected job
on demand, caches the result for 60 s, and supports explicit **r** refresh.
Only one query/scan of each kind can be in flight. Optional failures never stop queue polling.
`--once --page 1` queries definitions once; `--once --page 4` queries storage once.
The AImageLab overview snapshot also queries its project budget once. Site quota/budget tables
retain the original output when their column layout cannot be recognized; optional tools are
not required for core queue/node monitoring.
GPU counts come from the *allocated TRES*, so GPUs assigned by site defaults are counted too
(falls back to `squeue -o %b` on Slurm versions without `tres-alloc`).

Implementation references: [squeue](https://slurm.schedmd.com/squeue.html),
[QoS fields](https://slurm.schedmd.com/sacctmgr.html#SECTION_LIST/SHOW-QOS-FORMAT-OPTIONS),
and [scontrol](https://slurm.schedmd.com/scontrol.html).

## Development

    git clone https://github.com/FabioBozzoli/slurmscope && cd slurmscope
    pip install -e ".[test]"
    python -m pytest

Layout: `src/slurmscope/cli.py` (dashboard and CLI), `tests/test_slurmscope.py`.
Released under the MIT license (see `LICENSE`).
