Memory Timeline
Memory Breakdown by Component
Summary Statistics
Peak Allocated
17614.47MB
Avg Allocated
11289.88MB
Memory Efficiency
27.6%
Headroom
21569.64MB
Peak Reserved
19390.36MB
Memory Leak
⚠ DETECTED
Phase Averages (Allocated MB)
forward9153 MB
backward13503 MB
optimizer13998 MB
step8506 MB
Recommendations
[HIGH]
Possible memory leak detected
Allocated memory shows a consistent monotonic upward trend across steps. This suggests tensors are being accumulated and not released.
Action: Check for: (1) loss accumulation without .detach() — use loss.detach().item() not loss.item() when storing; (2) tensors appended to lists inside the loop; (3) call torch.cuda.empty_cache() between epochs. Profile with: torch.cuda.memory_snapshot()
[MEDIUM]
Low memory utilization — increase batch size
GPU memory utilization is only 27.6%. You're leaving significant VRAM unused, which reduces throughput.
Action: Increase batch size by ~2.7x (estimated safe multiplier) to improve hardware utilization.
[LOW]
Large headroom (21570 MB free) — optimize gradient accumulation
There is 21570 MB of unused VRAM. If you're using gradient accumulation, you may be able to reduce the number of accumulation steps and increase effective batch size.
Action: If using gradient accumulation, consider reducing gradient_accumulation_steps and increasing per-device batch size to fill available VRAM and improve throughput.
🤖 AI Analysis
Not configured
Add an Anthropic API key to get AI-powered recommendations:
vramwatch analyze run.vram --api-key sk-ant-...
Raw Event Data
Show raw events (800 total)
| step ⇅ | epoch ⇅ | phase ⇅ | loss ⇅ | alloc MB ⇅ | reserved MB ⇅ | peak MB ⇅ | free MB ⇅ | params MB ⇅ | grads MB ⇅ | acts MB ⇅ | opt MB ⇅ |
|---|