Metadata-Version: 2.4
Name: torchwatch-tui
Version: 0.1.0
Summary: A btop for your PyTorch GPU jobs: live terminal dashboard for training runs
Project-URL: Homepage, https://github.com/JayPatel095/TorchWatch
Project-URL: Repository, https://github.com/JayPatel095/TorchWatch
Project-URL: Issues, https://github.com/JayPatel095/TorchWatch/issues
Author-email: Jay Patel <jayashish.patel@mail.utoronto.ca>
License-Expression: MIT
License-File: LICENSE
Keywords: gpu,monitoring,pytorch,terminal,training,tui
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: click
Requires-Dist: nvidia-ml-py
Requires-Dist: psutil
Requires-Dist: textual
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# torchwatch

A btop for your PyTorch GPU jobs with live per-GPU utilization, VRAM pressure, training
throughput, loss curve, and ETA, all in your terminal, with zero code changes.

![torchwatch demo](https://raw.githubusercontent.com/JayPatel095/TorchWatch/main/docs/demo.gif)

## Install

```
pip install torchwatch-tui
```

(The command and package are `torchwatch`; only the PyPI name carries the suffix.)

## Run

The easiest way is to launch your training command under torchwatch:

```
torchwatch run -- python train.py --epochs 10
```

Your script runs as a child of torchwatch under a pseudo-terminal. Its output is
parsed live, and quitting the dashboard (`q`) terminates it.

Attach to a job that's already running:

```
torchwatch list           # list running PyTorch processes
torchwatch --pid 12345    # attach to one of them
```

Attaching reads the process's stdout via `/proc` (Linux only), and only works when
stdout is redirected to a file (`python train.py > train.log`). If it isn't,
torchwatch tells you why and suggests alternatives instead of guessing.

Smoke-test the dashboard on synthetic data

```
torchwatch --demo
```

## Current Features

- **Per-GPU panels** — utilization, VRAM (used/total and a pressure-colored gauge),
  temperature, and power with one tile per device
- **Loss sparkline** — scrolling curve of recent loss values
- **Throughput + ETA** — steps/sec over a rolling window, elapsed time, and a
  time-to-completion estimate through epoch restarts
- **Alerts area** — appears only when something needs attention, and lingers long
  enough to read:
  - VRAM above 95% on a specific GPU, with concrete fixes (smaller batch size,
    mixed precision, gradient checkpointing)
  - loss stalled (no meaningful change over the last 100 updates)
  - loss spike (latest value far above the recent average)

Keys: `q` quit · `p` pause

## Supported log formats

torchwatch parses training progress straight from stdout with no imports or callbacks.
Detected automatically:

| format | example line |
|---|---|
| tqdm | ` 42/500 [00:12<02:15, 3.4it/s, loss=0.693]` |
| PyTorch Lightning | `Epoch 3/10: 42/500 [... loss=0.693 ...]` |
| HF Trainer | `{'loss': 0.693, 'learning_rate': 5e-05, 'epoch': 1.2}` |
| plain | `step 42/500 loss: 0.693` |

The header shows which format was detected. Lines that match nothing are ignored.

## Notes

- GPU stats come from NVML (NVIDIA). Without a usable NVML (e.g. macOS, no driver),
  torchwatch falls back to clearly-labeled mock data so the dashboard still runs.
- `--pid` attach and process detection use `/proc` and process memory maps, so they
  are Linux-only. `torchwatch run` works everywhere.

## Development

```
pip install -e ".[dev]"
pytest
ruff check .
```

## Planned

Roughly in order of likelihood:

- **More alert rules** — NaN/inf loss (the run is dead, say so immediately),
  GPU temperature, sustained throughput drops (often a dataloader bottleneck)
- **Alert logging** — opt-in `--alert-log alerts.log`: a timestamped record of
  every alert, so a spike at 3am is still explainable at 9am
- **Configurable thresholds** — `--vram-warn` / `--vram-alert` instead of the
  built-in 85/95
- **Generic metric sparklines** — track any `key=value` metric your logs already
  print (accuracy, val_loss, grad_norm, …) and chart the ones you pick
- **Appearance** — theme selection, optional extra GPU stats
- **Config file** — persistent defaults once the flag count justifies it
- **Raw output pane** — the wrapped process's actual stdout, scrolling under
  the dashboard
- **Multi-job dashboard** — one dashboard, several training processes
