Metadata-Version: 2.4
Name: gpusched
Version: 0.3.0
Summary: VRAM-aware single-node GPU job scheduler: queue shell commands, place them by free GPU memory, and verify declared-vs-actual VRAM per job.
License: MIT
Project-URL: Homepage, https://github.com/ceruleane/gpusched
Project-URL: Issues, https://github.com/ceruleane/gpusched/issues
Project-URL: Changelog, https://github.com/ceruleane/gpusched/blob/main/CHANGELOG.md
Keywords: gpu,scheduler,vram,nvidia,queue,cuda
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# gpusched

A VRAM-aware GPU job scheduler for a single machine. You write shell commands
in a text file, optionally declaring how much GPU memory each needs; gpusched
runs them, placing each job on a GPU with enough free memory the moment one
opens up, measures how much VRAM each job actually used, and tells you when
your declarations were wrong.

```
$ cat jobs.txt
[vram=18G] python train.py --config a.yaml
[vram=18G] python train.py --config b.yaml
python preprocess.py

$ gpusched jobs.txt --watch -v
[14:02:10] job 1 started on gpu [0] (1/3 dispatched, declared 18432 MiB/gpu) — ...
[14:02:10] job 2 started on gpu [1] (2/3 dispatched, declared 18432 MiB/gpu) — ...
[14:31:44] job 1 finished [OK] in 1774s — peak vram gpu0:17910 MiB | declared 18432 MiB → within ±10% | avg gpu util 96%
```

## What this is, and what it is not

gpusched is a **single-node, single-user research tool**: roughly a thousand
lines you can read in an afternoon, with no daemon, no database, and zero
Python dependencies. It exists for one workflow — you have a box with a few
GPUs, a pile of training/eval commands, and you want them to run unattended
with minimal GPU idle time and without OOM-ing each other.

It is **not** a cluster scheduler, and several of its guarantees are honest
best-effort rather than enforcement:

- **Declarations are advisory.** gpusched places jobs based on what you
  declare and warns loudly when reality diverges, but it cannot cap a
  process's VRAM (nothing can, short of MIG partitions). A job that blows
  through its declaration can still OOM a neighbor — you get a warning, not
  protection. `--exclusive` (one scheduled job per GPU) is the zero-risk mode.
- **It polls.** GPU state is sampled every `--poll` seconds (default 5) via
  `nvidia-smi`. VRAM spikes shorter than the poll interval are invisible to
  both placement and peak reporting.
- **Co-locating compute-heavy jobs can be slower than serializing them.**
  Packing optimizes VRAM occupancy, not throughput. Two trainers that each
  saturate the SMs will roughly halve each other's speed; packing pays off
  for memory-light or compute-light neighbors (eval scripts, dataloader-bound
  jobs, inference). The per-job utilization report helps you see which case
  you're in.
- **Per-job VRAM attribution can fail in some container setups** where
  `nvidia-smi` reports host-namespace PIDs that don't match the container's.
  gpusched then reports the peak as `n/a` rather than a wrong number;
  placement still works because it uses device-level totals.
- **Testing honesty:** the test suite (65 tests) exercises the full scheduler
  against simulated GPU backends and real subprocesses; the `nvidia-smi`
  parsing layer is straightforward but thin. Do a small smoke run on your
  hardware before trusting it with a week of compute.

## If your needs are bigger than this

Use the right tool instead of stretching this one:

| Need | Look at |
|---|---|
| Multiple machines, multiple users, fairness, accounting | Slurm (or PBS/LSF) |
| Hyperparameter optimization with early stopping / ASHA / PBT | Ray Tune, Optuna, W&B Sweeps |
| Distributed training orchestration | Ray, torchrun + Slurm, Kubernetes + device plugins |
| Workflow DAGs (job B consumes job A's output) | Snakemake, Makefile, Airflow |
| Hard VRAM/compute isolation between jobs | NVIDIA MIG (partitioning), MPS (limits) |
| Just a per-GPU FIFO queue, even simpler than this | task-spooler (`ts`), simple_gpu_scheduler |

For sweeps specifically, the intended pattern is: a 10-line script (or
Optuna in ask-and-tell mode) *generates* the jobs file; gpusched stays a dumb
command queue. The moment you want trials stopped early based on metrics, you
have outgrown this tool — that requires bidirectional communication with
running jobs, which is deliberately out of scope.

## Install

```
uv tool install gpusched        # isolated env, `gpusched` on PATH
# or: pip install gpusched
# or, from a clone: uv tool install .
```

Requires Python ≥ 3.10 and Linux with `nvidia-smi` (the scheduler itself has
zero Python dependencies). `uvx gpusched jobs.txt` runs it without installing.

## Sixty-second tour (no GPU required)

```
echo '[vram=8G] python3 -m gpusched.simjob --vram 8000 --ramp 2 --hold 5' > jobs.txt
gpusched jobs.txt --sim 2 --poll 0.5 -v
```

`--sim N` runs against N simulated 24 GiB GPUs; `gpusched.simjob` is a fake
GPU job that ramps and holds a declared amount of fake VRAM. Everything below
behaves identically in sim and on real hardware.

## The jobs file

One shell command per line; blank lines and `#` comments are skipped. An
optional leading `[...]` block declares per-job attributes:

```
# no declaration -> runs only on a fully idle GPU, alone
python preprocess.py

# declared max VRAM (per GPU): may share a GPU when the declared amount fits
[vram=18000] python train.py --config a.yaml
[vram=22G]   bash run_eval.sh

# multi-GPU: 2 GPUs, EACH with >= 30 GiB free; CUDA_VISIBLE_DEVICES gets both
[vram=30G gpus=2] torchrun --nproc_per_node=2 train_big.py

# auto-retry on CUDA OOM, declaration bumped ~1.25x of observed peak per retry
[vram=8G retries=2] python sweep.py --seed 3

# opt-in walltime: SIGTERM at 2h, SIGKILL +10s. No timeout attribute = runs
# forever; the scheduler never guesses which long-running jobs are hung.
[timeout=2h] python flaky_eval.py
```

`vram` accepts MiB integers or `G`/`GiB` suffixes and is always per GPU.
Declare your honest worst case; the completion report tells you how close you
were, so declarations converge after a run or two.

## How placement works

A job **without** a declaration gets a GPU only when it is fully idle (below
`--idle-threshold`, default 200 MiB, with no other scheduled job) — the safe
default when you don't know what a job needs.

A job **with** a declaration of E MiB can be placed wherever *effective
headroom* ≥ E + `--margin` (default 512 MiB). Effective headroom accounts for
three things. First, a just-launched job that hasn't allocated its CUDA
context yet still reserves everything it declared — this closes the classic
double-booking race in poll-based schedulers, where a GPU looks empty for the
few seconds before a process materializes. Second, every process gpusched did
not launch is tracked to its observed per-GPU peak and held to
`peak × (1 + --spike-buffer)` until it exits, so a fluctuating external
process's momentary trough is not treated as packable space. Third, a
scheduled job that exceeds its own declaration stops being trusted: its
budget escalates from the declaration to its buffered observed peak.

Queue order is file order with backfill: if the next job can't fit right now,
smaller jobs behind it run first. A job that could not fit even on a
completely empty GPU fails immediately as `INFEASIBLE` rather than stalling
the queue. Multi-GPU jobs take N distinct GPUs, each meeting the per-GPU
requirement, chosen best-fit to preserve large contiguous headroom.

(One physical reality worth knowing: PyTorch's caching allocator rarely
returns VRAM to the driver, so for torch jobs `nvidia-smi` already reads near
the high-water mark — the spike-buffer machinery matters most for processes
that genuinely release memory between phases.)

## Monitoring: declared vs actual

Each scheduled job runs in its own session (`setsid`), so all its descendant
processes share one process-group id; each poll, gpusched maps `nvidia-smi`'s
per-process VRAM onto jobs by pgid and tracks per-GPU peaks. Two asymmetric
notifications, tuned by `--tolerance` (default ±10%):

**Under-declaration warns immediately** — the first poll where actual exceeds
declared, you get `WARN job N EXCEEDS declared VRAM ... neighbors may OOM`,
because at that moment the packing math other jobs were placed under is
already violated. **Over-declaration is reported at completion** — a
fluctuating job may legitimately peak late, so it can only be judged once it
exits: `declared 12288 MiB → over-declared (-59%); lowering it frees packing
headroom`.

Every completion line is streamed the moment the job finishes and includes
its per-GPU peak and average device-level GPU utilization (device-level: when
two jobs share a GPU the number is confounded — treat it as a diagnostic for
spotting dataloader-bound runs, not a per-process metric). `--verbose` adds
live usage lines as a job's peak grows.

## Live queue, resume, and the status board

The jobs file is **user-owned and never written by the scheduler**; it is
re-read every poll. Each line has a stable identity (hash of its command text
plus an occurrence counter for duplicate lines), and an append-only journal
(`<log_dir>/journal.jsonl`) records attempts and outcomes per identity.
"Pending" is defined as: lines in the file that are neither running nor
terminal in the journal. Everything follows from that one definition —
append a line from any terminal and it is dispatched within a poll; delete a
pending line and it is dequeued; reorder pending lines and you have reordered
the queue (file order among pending IS the priority; there is no separate
priority mechanism); edit a pending line and you have replaced it. Edits to
running or completed lines do nothing. A malformed mid-edit save is rejected
with a warning and the last good queue is kept; in-flight jobs are never
affected.

Re-running the same command after a crash or Ctrl-C skips everything the
journal marks done — that is resume. `--fresh` wipes the journal to re-run
all; to re-run one job, change its line trivially (new identity). `--watch`
keeps the scheduler alive after the queue drains, waiting for appended lines.

A live board is rendered to `<log_dir>/status.txt` every poll
(`▶` running, `·` pending, `↻` retrying after OOM, `✓`/`✗` done):

```
watch -n2 cat gpusched_logs/status.txt
```

Known limitation: with continuous submission, a large blocked job can be
starved by a stream of small backfilled ones. There is no aging policy — you
are the priority mechanism. Move the big job's line up and hold small
submissions, or run it under `--exclusive`.

## OOM retry and timeouts

A failed job whose log tail matches CUDA OOM signatures and that declared
`[retries=N]` (or ran under `--oom-retries N`) is requeued instead of
terminal-failed: its declaration is bumped to ~1.25× of max(observed peak,
old declaration), recorded in the journal (so it survives scheduler
restarts), and applied on the next attempt — the retry is scheduled with
honest requirements instead of repeating the same collision. Non-OOM failures
never consume retries, so retry loops cannot mask code bugs.

Timeouts are strictly per-job opt-in (`[timeout=90s|15m|2h|1d]`).
Distinguishing a hung process from a legitimate three-day run is your
declaration, never a heuristic — heuristics (e.g. "0% util for 10 minutes")
kill legitimate CPU phases like preprocessing and checkpoint serialization,
so none are included.

## Running detached (tmux)

The scheduler is an ordinary foreground process — run it inside tmux and
disconnect freely:

```
tmux new -s sched
gpusched jobs.txt --watch -v            # pane 1
# Ctrl-b c -> pane 2:
watch -n2 cat gpusched_logs/status.txt
# Ctrl-b d to detach; later: tmux attach -t sched
```

You can also append jobs over ssh without attaching at all — the file is
re-read every poll. Failure-mode hierarchy, honestly: tmux protects
everything from SSH drops; the journal protects queue state from scheduler
death; nothing recovers *tracking* of jobs orphaned by a dead scheduler (they
keep running in their own sessions, but their exit codes are lost and a
restarted scheduler treats them as not-done).

## CLI reference

```
gpusched jobs.txt
  --gpus 0,1,3          restrict to these GPU indices (default: all visible)
  --idle-threshold 200  MiB below which a GPU counts as idle (undeclared jobs)
  --margin 512          MiB safety margin added to every declaration
  --tolerance 0.10      band before flagging over/under-declaration
  --spike-buffer 0.10   buffer over observed VRAM maxima of fluctuating processes
  --poll 5              seconds between scheduling rounds
  --exclusive           one scheduled job per GPU, even when declarations fit
  --watch               keep running after drain; pick up appended lines
  --oom-retries N       default CUDA-OOM auto-retries ([retries=N] overrides)
  --fresh               ignore + remove the journal: re-run everything
  --log-dir DIR         per-job logs, journal, status board (default: gpusched_logs)
  -v / --verbose        stream live per-job VRAM as peaks grow
  --sim N               dry-run on N simulated 24 GiB GPUs (no hardware)
```

Exit code: 0 if every job succeeded, otherwise the max failing job exit code.

## Architecture and extending

```
src/gpusched/
  jobspec.py     parsing: [vram=.. gpus=.. timeout=.. retries=..] cmd -> JobSpec
  backend.py     GpuBackend protocol; NvidiaSmiBackend (2 queries/poll); pgid attribution
  allocation.py  PURE placement function: headroom, reservations, best-fit
  journal.py     append-only JSONL: attempts + terminal outcomes per job identity
  scheduler.py   tick loop: snapshot -> attribute -> warn -> timeouts -> reap -> dispatch
  testing.py     FakeBackend (unit tests), SimBackend (integration / --sim)
  simjob.py      simulated GPU job for tests and dry runs
  cli.py         argparse front end
```

The deliberate seams: `allocation.find_allocation` is pure (snapshot +
occupants in, GPU list out), so new placement rules — GPU-type constraints,
NVLink-aware pairing — are filters there plus an attribute in the jobspec
parser, with nothing else touched. Alternative monitors (pynvml, DCGM)
implement the two-method `GpuBackend` protocol. The journal is the only
persistent state.

## Development

```
git clone <repo> && cd gpusched
uv venv && uv pip install -e ".[dev]"
uv run pytest -q          # 65 tests, ~20s, no GPU required
```

Tests drive the scheduler against fake/simulated backends with real
subprocesses; several are timing-based (sub-second sim jobs with fast polls),
so a heavily loaded machine can occasionally need a re-run.

## License

MIT.
