# GPU Host Report

Generated: 2026-03-02T12:23:12

Checked hosts (SSH key login success): `16-3 16-4 16-5 16-6 16-7 16-8 16-11 16-12 16-13 16-21 16-22`

## Summary Table

| Host | GPU Count | Busy | Idle | Note |
|---|---:|---:|---:|---|
| 16-3 | 6 | 6 | 0 | idle_idx=[] |
| 16-4 | 8 | 8 | 0 | idle_idx=[] |
| 16-5 | 7 | 0 | 7 | idle_idx=[0, 1, 2, 3, 4, 5, 6] |
| 16-6 | 8 | 8 | 0 | idle_idx=[] |
| 16-7 | 3 | 3 | 0 | idle_idx=[] |
| 16-8 | 3 | 0 | 3 | idle_idx=[0, 1, 2] |
| 16-11 | 2 | 2 | 0 | idle_idx=[] |
| 16-12 | 8 | 6 | 2 | idle_idx=[5, 7] |
| 16-13 | 8 | 8 | 0 | idle_idx=[] |
| 16-21 | 0 | 0 | 0 | nvidia-smi failed: 127 |
| 16-22 | 0 | 0 | 0 | nvidia-smi failed: 127 |

## Per-host Details

### 16-3
- gpu_count: 6, busy: 6, idle: 0
- gpus:
  - gpu0: busy, mem=12682/24564 MiB, util=99%
  - gpu1: busy, mem=12394/24564 MiB, util=98%
  - gpu2: busy, mem=12682/24564 MiB, util=99%
  - gpu3: busy, mem=13118/24564 MiB, util=99%
  - gpu4: busy, mem=11392/24564 MiB, util=97%
  - gpu5: busy, mem=11198/24564 MiB, util=100%
- processes:
  - 1763286, ray::RayTrainWorker, 12672 MiB
  - 1763287, ray::RayTrainWorker, 12384 MiB
  - 1763288, ray::RayTrainWorker, 12672 MiB
  - 1763289, ray::RayTrainWorker, 13108 MiB
  - 1763290, ray::RayTrainWorker, 11382 MiB
  - 1763291, ray::RayTrainWorker, 11188 MiB

### 16-4
- gpu_count: 8, busy: 8, idle: 0
- gpus:
  - gpu0: busy, mem=47347/49140 MiB, util=0%
  - gpu1: busy, mem=47349/49140 MiB, util=100%
  - gpu2: busy, mem=48219/49140 MiB, util=100%
  - gpu3: busy, mem=48219/49140 MiB, util=100%
  - gpu4: busy, mem=48219/49140 MiB, util=100%
  - gpu5: busy, mem=48219/49140 MiB, util=100%
  - gpu6: busy, mem=48219/49140 MiB, util=100%
  - gpu7: busy, mem=48219/49140 MiB, util=100%
- processes:
  - 1062247, ray::RayTrainWorker, 47334 MiB
  - 1062249, ray::RayTrainWorker, 47336 MiB
  - 1062248, ray::RayTrainWorker, 48206 MiB
  - 1062250, ray::RayTrainWorker, 48206 MiB
  - 1062251, ray::RayTrainWorker, 48206 MiB
  - 1062253, ray::RayTrainWorker, 48206 MiB
  - 1062254, ray::RayTrainWorker, 48206 MiB
  - 1062252, ray::RayTrainWorker, 48206 MiB

### 16-5
- gpu_count: 7, busy: 0, idle: 7
- gpus:
  - gpu0: idle, mem=0/24564 MiB, util=0%
  - gpu1: idle, mem=0/24564 MiB, util=0%
  - gpu2: idle, mem=0/24564 MiB, util=0%
  - gpu3: idle, mem=0/24564 MiB, util=0%
  - gpu4: idle, mem=0/24564 MiB, util=0%
  - gpu5: idle, mem=0/24564 MiB, util=0%
  - gpu6: idle, mem=0/24564 MiB, util=0%
- processes: none

### 16-6
- gpu_count: 8, busy: 8, idle: 0
- gpus:
  - gpu0: busy, mem=46016/49140 MiB, util=0%
  - gpu1: busy, mem=46016/49140 MiB, util=0%
  - gpu2: busy, mem=46016/49140 MiB, util=0%
  - gpu3: busy, mem=46016/49140 MiB, util=0%
  - gpu4: busy, mem=46016/49140 MiB, util=0%
  - gpu5: busy, mem=46016/49140 MiB, util=0%
  - gpu6: busy, mem=46016/49140 MiB, util=0%
  - gpu7: busy, mem=46016/49140 MiB, util=0%
- processes:
  - 783293, cross sectional model training, 46010 MiB
  - 783294, cross sectional model training, 46010 MiB
  - 783295, cross sectional model training, 46010 MiB
  - 783296, cross sectional model training, 46010 MiB
  - 783297, cross sectional model training, 46010 MiB
  - 783298, cross sectional model training, 46010 MiB
  - 783299, cross sectional model training, 46010 MiB
  - 783300, cross sectional model training, 46010 MiB

### 16-7
- gpu_count: 3, busy: 3, idle: 0
- gpus:
  - gpu0: busy, mem=16546/24564 MiB, util=61%
  - gpu1: busy, mem=16546/24564 MiB, util=77%
  - gpu2: busy, mem=16546/24564 MiB, util=62%
- processes:
  - 2444870, ray::_RayTrainWorker__execute.get_next, 16540 MiB
  - 2444872, ray::_RayTrainWorker__execute.get_next, 16540 MiB
  - 2444871, ray::_RayTrainWorker__execute.get_next, 16540 MiB

### 16-8
- gpu_count: 3, busy: 0, idle: 3
- gpus:
  - gpu0: idle, mem=0/24564 MiB, util=0%
  - gpu1: idle, mem=0/24564 MiB, util=0%
  - gpu2: idle, mem=0/24564 MiB, util=0%
- processes: none

### 16-11
- gpu_count: 2, busy: 2, idle: 0
- gpus:
  - gpu0: busy, mem=16499/49140 MiB, util=51%
  - gpu1: busy, mem=16499/49140 MiB, util=60%
- processes:
  - 2395226, ray::_RayTrainWorker__execute.get_next, 16490 MiB
  - 2395225, ray::_RayTrainWorker__execute.get_next, 16490 MiB

### 16-12
- gpu_count: 8, busy: 6, idle: 2
- gpus:
  - gpu0: busy, mem=12685/24564 MiB, util=98%
  - gpu1: busy, mem=12979/24564 MiB, util=100%
  - gpu2: busy, mem=11829/24564 MiB, util=100%
  - gpu3: busy, mem=12619/24564 MiB, util=99%
  - gpu4: busy, mem=12571/24564 MiB, util=100%
  - gpu5: idle, mem=0/24564 MiB, util=0%
  - gpu6: busy, mem=12685/24564 MiB, util=100%
  - gpu7: idle, mem=0/24564 MiB, util=0%
- processes:
  - 2243657, ray::RayTrainWorker, 12672 MiB
  - 2243656, ray::RayTrainWorker, 12966 MiB
  - 2243659, ray::RayTrainWorker, 11816 MiB
  - 2243660, ray::RayTrainWorker, 12606 MiB
  - 2243661, ray::RayTrainWorker, 12558 MiB
  - 2243658, ray::RayTrainWorker, 12672 MiB

### 16-13
- gpu_count: 8, busy: 8, idle: 0
- gpus:
  - gpu0: busy, mem=36092/49140 MiB, util=100%
  - gpu1: busy, mem=34476/49140 MiB, util=100%
  - gpu2: busy, mem=34476/49140 MiB, util=100%
  - gpu3: busy, mem=34476/49140 MiB, util=100%
  - gpu4: busy, mem=34476/49140 MiB, util=100%
  - gpu5: busy, mem=34476/49140 MiB, util=100%
  - gpu6: busy, mem=34476/49140 MiB, util=100%
  - gpu7: busy, mem=34476/49140 MiB, util=100%
- processes:
  - 3845003, cross sectional model training, 29732 MiB
  - 4177054, python, 6330 MiB
  - 3845004, cross sectional model training, 29732 MiB
  - 4177054, python, 4714 MiB
  - 3845005, cross sectional model training, 29732 MiB
  - 4177054, python, 4714 MiB
  - 3845006, cross sectional model training, 29732 MiB
  - 4177054, python, 4714 MiB
  - 3845007, cross sectional model training, 29732 MiB
  - 4177054, python, 4714 MiB
  - 3845008, cross sectional model training, 29732 MiB
  - 4177054, python, 4714 MiB
  - ... total 16 process entries

### 16-21
- status: unavailable (`nvidia-smi failed: 127`)

### 16-22
- status: unavailable (`nvidia-smi failed: 127`)

