Metadata-Version: 2.4
Name: xpustat
Version: 0.3.0
Summary: A unified stat tool for NVIDIA, AMD, Intel, Huawei, Hygon, Cambricon, Moore Threads, Apple Silicon, AWS Neuron, Google TPU and Tenstorrent accelerators
Author: Salah Eddine Bekhouche, Hichem Telli
Author-email: bekhouchesalah@gmail.com
License: MIT
Project-URL: Homepage, https://github.com/Bekhouche/xpustat
Keywords: gpu,npu,accelerator,monitoring,nvidia,amd,intel,huawei,ascend,apple,macos,metal,neuron,trainium,tpu,tenstorrent,prometheus,pytorch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psutil>=5.9
Provides-Extra: nvidia
Requires-Dist: nvidia-ml-py>=12.535.108; extra == "nvidia"
Provides-Extra: amd
Requires-Dist: amdsmi; extra == "amd"
Provides-Extra: fast
Requires-Dist: nvidia-ml-py>=12.535.108; extra == "fast"
Requires-Dist: amdsmi; extra == "fast"
Dynamic: license-file

# xpustat

[![GitHub](https://img.shields.io/badge/GitHub-Bekhouche%2Fxpustat-blue?logo=github)](https://github.com/Bekhouche/xpustat)
**Authors:** Salah Eddine Bekhouche, Hichem Telli

A unified accelerator stat tool: `gpustat`, but for every vendor.

## Supported backends

| Status | Vendor | Device | CLI tool | Filter key |
|:---:|---|---|---|---|
| ✅ Tested | AMD | Radeon RX 7900 XTX (gfx1100) | `amd-smi` | `amd` |
| ✅ Tested | NVIDIA | NVIDIA L4 (cc8.9), TITAN V (cc7.0) | `nvidia-smi` | `nvidia` |
| 🔲 Untested | Intel | Gaudi / Gaudi 2 / Gaudi 3 | `hl-smi` | `gaudi` |
| 🔲 Untested | Intel | Arc / Data Center GPU | `xpu-smi` | `intel` |
| ✅ Tested | Huawei | Ascend 910B2 | `ascend-dmi` / `npu-smi` | `huawei` |
| 🔲 Untested | Hygon | DCU | `hy-smi` | `hygon` |
| 🔲 Untested | Cambricon | MLU | `cnmon` | `cambricon` |
| 🔲 Untested | Moore Threads | MTT GPU | `mthreads-gmi` | `moorethreads` |
| ✅ Tested | Apple | Apple M5 (10-core GPU) | IOKit (built into macOS) | `apple` |
| 🔲 Untested | AWS | Neuron (Inferentia / Trainium) | `neuron-ls` | `neuron` |
| 🔲 Untested | Google | Cloud TPU (v4 – v6e) | `tpu-info` | `tpu` |
| 🔲 Untested | Tenstorrent | Wormhole / Blackhole | `tt-smi` | `tenstorrent` |

> **Legend:** ✅ tested on real hardware; 🔲 implemented, awaiting a hardware test
>
> Backends whose management tool is not installed are silently skipped.
>
> **Apple Silicon note:** the GPU shares unified memory with the CPU, so `mem_total_mb` is the
> machine's total RAM and `mem_used_mb` is the slice currently held by the GPU driver.
> macOS exposes GPU temperature, power, and per-process usage only to root (`powermetrics`),
> so those columns show `N/A` on Macs. Discrete GPUs in Intel Macs report real VRAM.
>
> **Neuron / Tenstorrent note:** `neuron-ls` and `tt-smi` report inventory, processes and telemetry
> but not live memory usage, so those backends show memory as `0` / `N/A`.

### Running Vendor Hardware Tests

If you have access to specific hardware, you can help us verify the implementation by running the standalone test scripts found in the `vendor_devices_tests/` directory:

```bash
# Example: Testing the Huawei backend on a machine with Ascend NPUs
python vendor_devices_tests/test_huawei.py

# Example: Testing the NVIDIA backend
python vendor_devices_tests/test_nvidia.py
```
If the test successfully outputs your hardware metrics, please let us know so we can update the backend status to **✅ Tested**!

## Documentation

The full documentation is in [`docs/`](docs/index.md): [installation](docs/installation.md), [command line](docs/cli.md), [Python API](docs/python-api.md), [device selection](docs/device-selection.md), [Prometheus](docs/prometheus.md), [backends](docs/backends.md) and [contributing](docs/contributing.md). `mkdocs serve` renders it as a site.

## Install

**pip**
```bash
pip install xpustat
```

**conda**
```bash
conda install -c conda-forge xpustat
```

## CLI usage

```
xpustat                      # coloured table of all detected devices
xpustat -pui                 # watch mode, showing processes (-p), users (-u), refreshing every 1.0s (-i)
xpustat -i 2                 # watch mode, refresh every 2 seconds
xpustat -i 1 -n 10           # 10 refreshes then exit
xpustat -a                   # show everything (util, temp, power, arch, user, pid, cmd)
xpustat -f nvidia amd        # show only NVIDIA and AMD
xpustat --no-color           # plain text, no ANSI codes
xpustat --json               # raw JSON output
xpustat --prometheus         # Prometheus text format, once
xpustat --serve-prometheus 9400   # serve http://0.0.0.0:9400/metrics (fresh query per scrape)
xpustat --version
```

Example output:

```
xpustat 0.1.0  ·  myhost  ·  Fri May 22  11:24:00  2026
──────────────────────────────────────────────────────────────────────────
[NVIDIA] #0    Tesla V100                          │   1,024 / 32,768 MB   ███░░░░░░░░░░░░░    3%
[AMD   ] #0    Radeon RX 7900 XTX                  │      26 / 24,560 MB   ░░░░░░░░░░░░░░░░    0%
[APPLE ] #0    Apple M5                            │     375 / 32,768 MB   ░░░░░░░░░░░░░░░░    1%
[HUAWEI] #0:0  Ascend 310                          │   2,703 /  8,192 MB   █████░░░░░░░░░░░   33%
```

Example output with processes, users, and watch mode (`xpustat -pui`):

```
xpustat 0.1.0  ·  salah-amd  ·  Fri May 29  10:07:57  2026
──────────────────────────────────────────────────────────
[AMD   ] #0    Radeon RX 7900 XTX              │  17,918 / 24,560 MB  ████████████░░░░   72%
              └─ salah/348504(17480M)
  refreshing every 1.0s  ·  Ctrl+C to quit
```

## Python API

```python
import xpustat

# Query everything at once (parallel by default)
stats = xpustat.query_all()

print(stats)                      # XPUStatCollection(nvidia=0, amd=1, ...)
print(stats.amd[0].name)          # "Radeon RX 7900 XTX"
print(stats.amd[0].mem_used_mb)   # 26

# As a dict (JSON-serialisable)
import json
print(json.dumps(stats.to_dict(), indent=2))

# Query a single vendor
from xpustat import NvidiaGPUStatCollection
gpus = NvidiaGPUStatCollection.new_query()
```

### Picking a device for training / inference

Every device object carries vendor-neutral fields on top of the vendor ones:

| Field | Meaning |
|---|---|
| `vendor` | vendor key, e.g. `"nvidia"` |
| `device_index` | physical index as the vendor tool reports it |
| `mem_free_mb` | free memory (for Apple unified memory: what the OS can still give the GPU) |
| `framework_index` | index the ML framework will see after `CUDA_VISIBLE_DEVICES` / `HIP_VISIBLE_DEVICES` / `ASCEND_RT_VISIBLE_DEVICES` / … (`None` if hidden) |
| `torch_device` | `torch.device` string: `cuda:N` (NVIDIA, AMD, Hygon), `mps`, `npu:N`, `hpu:N`, `xpu:N`, `mlu:N`, `musa:N`, `xla:N` |

```python
import xpustat

dev = xpustat.pick_device()[0]            # most free memory first, honours *_VISIBLE_DEVICES
model.to(dev.torch_device)                # "cuda:2", "npu:0", "mps", ...

xpustat.pick_device(n=2, by="util")       # two least-busy devices
xpustat.pick_device(vendor="amd", min_free_mb=8000)
xpustat.visible_indices("nvidia")         # parsed CUDA_VISIBLE_DEVICES, or None when unset
```

### Prometheus / Grafana

```bash
xpustat --serve-prometheus 9400           # then scrape http://host:9400/metrics
```

Exposes `xpustat_memory_{used,total,free}_bytes`, `xpustat_utilization_percent`,
`xpustat_temperature_celsius`, `xpustat_power_watts`, `xpustat_process_memory_bytes` and
`xpustat_device_info` (with `arch` and `torch_device` labels) for every vendor from one process.
`xpustat.to_prometheus(stats)` returns the same text for embedding in your own exporter.

## Version history

### 0.3.0
- Apple Silicon GPU support (`apple` backend) read in-process from IOKit (~2 ms, `ioreg` fallback), no root required; tested on Apple M5
- `-A` shows the GPU core count for Apple devices (e.g. `10-core`)
- Fix: vendor collections now support index access (`stats.amd[0]`), as documented
- Faster subprocess fallbacks: the `nvidia-smi` and `amd-smi` invocations now run concurrently
- Watch mode no longer swallows errors silently
- Fix: process lines now render as `user:command/pid` instead of running user and command together
- The `nvidia` / `fast` extras now pull `nvidia-ml-py` (the official NVML bindings) instead of the deprecated `pynvml` package
- New backends: AWS Neuron (`neuron-ls`), Google Cloud TPU (`tpu-info`), Tenstorrent (`tt-smi`)
- Vendor-neutral device fields: `vendor`, `device_index`, `mem_free_mb`, `framework_index`, `torch_device`
- `xpustat.pick_device()` picks the least-loaded device(s), honouring `*_VISIBLE_DEVICES`
- Prometheus export: `--prometheus`, `--serve-prometheus PORT`, and `xpustat.to_prometheus()`

### 0.2.0
- Fix AMD GPUs not being detected when `amd-smi` reports `"N/A"` sections or idle GPUs (e.g. ROCm 10 with an iGPU present)
- Flicker-free watch mode: frames are redrawn in place on the alternate screen

### 0.1.0
- Initial release
- Support for NVIDIA, AMD, Intel Gaudi, Intel Arc/GPU, Huawei Ascend, Hygon DCU, Cambricon MLU, Moore Threads
