Metadata-Version: 2.5
Name: jobgpumonitor
Version: 0.4.1
Summary: Zero-dependency emitter that watches a running (Slurm/OAR/local) GPU job and streams structured events.
Project-URL: Homepage, https://github.com/marcpinet/jobgpumonitor
Author: Marc Pinet
License-Expression: MIT
License-File: LICENSE
Keywords: deep-learning,gpu,hpc,monitoring,oar,slurm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: jsonschema>=4; extra == 'dev'
Requires-Dist: psutil>=5.9; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: tqdm>=4.66; extra == 'dev'
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py>=12; extra == 'gpu'
Requires-Dist: psutil>=5.9; extra == 'gpu'
Description-Content-Type: text/markdown

<p align="center">
  <img src="docs/assets/logo.svg" alt="jgm" width="220">
</p>

<h1 align="center">jobgpumonitor</h1>

<p align="center">
  <b>Your GPU job tells you what it is doing. You stop ssh-ing in to check.</b><br>
  One import &nbsp;·&nbsp; zero dependencies &nbsp;·&nbsp; Slurm, OAR, PBS, LSF or none
</p>

<p align="center">
  <a href="https://github.com/marcpinet/jobgpumonitor/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/marcpinet/jobgpumonitor/actions/workflows/ci.yml/badge.svg"></a>
  <a href="https://pypi.org/project/jobgpumonitor/"><img alt="PyPI" src="https://img.shields.io/pypi/v/jobgpumonitor?color=blue"></a>
  <img alt="Python" src="https://img.shields.io/badge/python-3.9%2B-blue">
  <img alt="Dependencies" src="https://img.shields.io/badge/dependencies-0-success">
  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-lightgrey"></a>
</p>

---

Your job waited four hours in the queue, ran for two, and died at 3 a.m.
You found out at 9, from a `.err` file.

`jobgpumonitor` makes the job report on itself while it runs: where it is, how fast it goes,
whether the GPU is actually busy, how close it is to the memory limit, what the traceback was,
and how it ended. Everything is streamed as plain JSONL events to your shared filesystem,
so a notifier, a dashboard or a one-line `grep` can read it. No network from the compute node.

## 30-second start

```bash
pip install jobgpumonitor
```

```python
import jobgpumonitor.auto
```

That is the whole integration. No line at all, nothing installed in the job's
environment? Wrap the command: every Python it starts then monitors itself (tqdm,
metrics, traceback, GPU, memory), and any other language gets exit code and stderr.

```bash
jgm run -- python train.py
jgm run -- bash -c "python prep.py && python train.py"
```

With the package installed in the job's Python, `export JGM_AUTO=1` in the batch
script does the same without the wrapper. Helper interpreters spawned by a monitored
program (`python -c` probes, tooling) stay quiet; distributed ranks still report.

Events land in `~/.jobgpumonitor/runs/<cluster>/<job>/<restart>/`. Run `jgm doctor` on a node
to see what gets detected.

## What the job tells you

| Event | The question it answers |
|---|---|
| `run.start` | Which commit, which GPUs, which node, and *when the scheduler will kill it* |
| `progress.update` | tqdm ETA compared to the job deadline, hours before a time-out |
| `resource.sample` | Is the GPU idle? How close to the cgroup memory limit, before the OOM kill |
| `run.exception` | The full traceback, secrets masked, OOM flagged |
| `run.end` | Status, exit code, GPU utilisation and peak memory summary |
| `run.heartbeat` | Still alive, latest metrics, latest progress |

Add your own with `jobgpumonitor.log(loss=0.12, step=10)` or `jgm emit stage name=eval` from bash.

## Designed for real clusters

- **No network needed from the job.** Compute nodes rarely have internet; the shared filesystem always works. `jgm forward` on the login node ships the files to a server over HTTPS, through the site proxy, and resumes where it left off.
- **Never touches your program.** Hooks are chained, not replaced. Nothing raises. Forked workers and DDP ranks other than 0 stay quiet. Writes happen on a background thread, so a slow NFS never stalls a training step.
- **Works inside containers.** Reads environment variables and `/proc` only, no `squeue` required. Point `JGM_DIR` at a mounted path and you are done.
- **Honest about what it cannot see.** A SIGKILL leaves no trace from inside. Run `jgm scheduler` on the login node: it watches `squeue`/`sacct` and writes the scheduler's verdict (OOM, time-out, preemption, cancel), the queue state and the real `.out` path into the same run directories.

## Slurm in one screen

```bash
#!/bin/bash
#SBATCH --gpus-per-node=1 --time=04:00:00
export PYTHONUNBUFFERED=1
srun jgm run -- python train.py
```

Inside an enroot or Apptainer container, mount the event directory and export `JGM_DIR`:

```bash
mkdir -p "$HOME/.jobgpumonitor" && export JGM_DIR=/jgm
srun --container-mounts="$HOME/.jobgpumonitor:/jgm" jgm run -- python train.py
```

## Command line

`jgm` ships with the package (short for **j**ob**g**pu**m**onitor); `python -m jobgpumonitor.cli` is equivalent.

```
jgm run [--name N] -- CMD...   run a command under monitoring, forward signals, keep the stderr tail
jgm scheduler [--once]         login-node probe: squeue/sacct -> scheduler.state, tails .out/.err -> log.chunk
jgm forward --url U --token T  login-node relay: ship the event files to a jobgpumonitor-server (POST /ingest)
jgm emit TYPE k=v ...          emit one event from a shell script
jgm doctor [--json]            show what is detected on this node
jgm ls [--dir D]               list runs found in the event directory
```

Keep the probe alive on the login node with `tmux` or `systemd --user`, or run
`jgm scheduler --once` from cron.

## Restricted HPC login nodes

Some login nodes ship Python but no `pip`, no `ensurepip`, no root, and a per-process
memory cap small enough to kill `pip` or `uv` during dependency resolution. The emitter
has no dependencies, so it does not need any of that. Two supported ways to run it:

```bash
# a) one file, any Python >= 3.9, no install: from the GitHub release assets
curl -L https://github.com/marcpinet/jobgpumonitor/releases/latest/download/jgm.pyz -o ~/jgm.pyz
python3 ~/jgm.pyz scheduler
python3 ~/jgm.pyz forward --url https://your-server/jgm/ingest --token ...
```

```bash
# b) straight from a checkout
git clone https://github.com/marcpinet/jobgpumonitor && cd jobgpumonitor
PYTHONPATH=src python3 -m jobgpumonitor scheduler
```

The zipapp runs in well under 30 MB of RSS. Keep `jobgpumonitor-server` on a less
restricted machine and let `jgm forward` ship the events to it: only the zero-dependency
client has to live on the cluster.

## Configuration

Every knob is a `JGM_*` environment variable, all optional.
`JGM_DIR`, `JGM_SINKS`, `JGM_HEARTBEAT_S`, `JGM_SAMPLE_S`, `JGM_RANK_MODE`, `JGM_LOGGING_LEVEL`,
`JGM_CAPTURE_LOCALS`, `JGM_DISABLED`. The full list is in [docs/CONFIG.md](docs/CONFIG.md).

`pip install "jobgpumonitor[gpu]"` adds `nvidia-ml-py` and `psutil` for cheaper, richer samples.
Without them the same data comes from `nvidia-smi` and `/proc`.

## Ecosystem

This package only **emits**. Storage, API, notifications and dashboards live in a separate
server that consumes the [event protocol](docs/PROTOCOL.md) (JSON Schema in
[`schema/`](schema/)). Write your own consumer in an afternoon, or wait for ours.

## Contributing

```bash
pip install -e ".[dev]" && pytest
```

Design notes in [docs/DESIGN.md](docs/DESIGN.md). Issues and pull requests welcome, especially
reports from clusters and schedulers we have not seen.
