Server Status
Loading servers...
Recent Events
Loading events...
{% if fleet_home_url %}
Fleet Home
|
{% endif %}
 }})
{{ app_name }}
{{ version_string }}
⬆️ Update available
Total Servers
-
Online
-
Critical Events (24h)
-
Warnings (24h)
-
🤖
AI Insights
FREELoading AI features...
🚀 Start Your Free Trial
Create an account and verify your email to access IPMI Monitor AI features.
✨ Free Trial includes:
- 1 month of full AI features
- 1 million AI tokens
- Up to 50 servers with AI monitoring
- AI summaries, tasks & predictions
📋 How to get started:
- Click "Start Free Trial" below
- Create account or sign in
- You'll be redirected back automatically with AI enabled!
💡 Already have an account? Sign in, then return here.
Questions? support@cryptolabs.co.za
Syncing...
Last sync: checking...
Syncing data...
0%
🤖 AI Agent - Autonomous Recovery
Agent Status
Active
Analyzing fleet for issues...
Allow AI to automatically fix issues
⚠️
EXPERIMENTAL FEATURE
Autonomous recovery actions are experimental. By enabling these features, you accept that:
- Recovery actions may affect running workloads
- Reboots and power cycles will cause downtime
- Use at your own risk - no guarantees provided
📋 Prerequisites
🔑 SSH Access:
Checking...
🖥️ IPMI Access:
✓ Available
ℹ️ SSH access is required for GPU recovery actions. IPMI is required for reboot/power cycle.
These actions only affect the specific GPU or its workload. Other GPUs and workloads continue running.
⏹️ Stop Workload
SSH Required
Gracefully stop container/VM using the affected GPU
🔍 When this triggers:
- GPU stopped responding (hung/frozen)
- GPU memory errors detected
- Graphics processing exceptions
⚙️ What it does:
- Identifies which container/VM is using the affected GPU
- Sends graceful stop signal (docker stop / virsh shutdown)
- Waits up to 30 seconds for clean shutdown
- Container/VM can be restarted after GPU recovery
✅ Safe: Only the affected workload is stopped, not removed. Data is preserved.
🔄 GPU Soft Reset
SSH Required
Reset GPU via NVIDIA driver (nvidia-smi --gpu-reset)
🔍 When this triggers:
- GPU stopped responding to commands
- Driver reports GPU needs recovery
- GPU exceptions during compute operations
⚙️ What it does:
- Workload must be stopped first (auto or manual)
- Runs
nvidia-smi --gpu-reset -i <gpu> - Driver reinitializes the GPU
- GPU should be usable again within ~15 seconds
⚠️ Note: Requires no active processes on the GPU. Stop Workload runs first.
📉 GPU Clock Limit
SSH Required
Reduce GPU clocks by 20% to improve stability
🔍 When this triggers:
- Recurring GPU errors (same error multiple times)
- High rate of correctable memory errors (ECC)
- GPU showing signs of instability or overheating
⚙️ What it does:
- Gets current max GPU clock speed
- Sets new limit at 80% of max (20% reduction)
- Makes it persistent across reboots (via crontab)
- Flags GPU for maintenance review
⚠️ Trade-off: Reduces performance by ~15-20% but improves stability. Good for aging GPUs.
These actions may cause brief interruption to other workloads on the same system.
🔌 PCI Bus Reset
SSH Required
Remove and rescan GPU on PCI bus (hardware-level reset)
🔍 When this triggers:
- GPU "fell off the bus" - disconnected from system
- Soft reset failed to recover the GPU
- GPU not responding to any driver commands
⚙️ What it does:
- Stops container/VM using the GPU
- Removes GPU from PCI bus (
echo 1 > .../remove) - Rescans PCI bus to re-detect GPU (
echo 1 > .../rescan) - Waits ~30 seconds for GPU to reinitialize
- If successful, container/VM can be restarted
🔄 Recovery Flow:
Stop Workload → PCI Reset → ✓ Success → Restart Workload
↓ Failed
System Reboot (if enabled)
↓ Failed
System Reboot (if enabled)
⚠️ These actions will stop ALL workloads and cause system downtime!
🔁 System Reboot
IPMI Required
Warm reboot via IPMI when soft recovery fails
🔍 When this triggers:
- All soft recovery actions failed
- GPU requires driver reload (only via reboot)
- Uncorrectable memory errors (ECC)
- GPU firmware errors
⚙️ What it does:
- Sends IPMI reset command to BMC
- System performs warm reboot (~2-5 min)
- All services restart automatically
- GPU driver reinitializes fresh
🔄 Recovery Flow:
Soft Reset Failed → PCI Reset Failed → System Reboot
↓ Failed/Stuck
Power Cycle (if enabled)
↓ Failed/Stuck
Power Cycle (if enabled)
❗ Downtime: ~2-5 minutes. All containers/VMs will be stopped.
⚡ Power Cycle
IPMI Required
Hard power off/on - last resort when reboot fails or system is stuck
🔍 When this triggers:
- System is stuck during shutdown/reboot
- GPU firmware halted (microcontroller error)
- Reboot command didn't complete after timeout
- System completely unresponsive
⚙️ What it does:
- Sends IPMI power cycle command to BMC
- Server powers off completely
- BMC waits ~5 seconds
- Server powers back on and boots (~3-8 min total)
❗ Downtime: ~3-8 minutes. Equivalent to pulling the power cord. No graceful shutdown.
💾 Disk Cleanup
SSH Required
Auto-clean when disk usage exceeds 90%
🔍 What it cleans (safe items only):
- Docker build cache (
docker builder prune) - Dangling images (unused, untagged)
- Old container logs (>7 days)
- Systemd journal logs (>7 days)
✅ Safe: Does NOT delete stopped containers, volumes, or running workload data.
🏷️ Auto Maintenance Flag
Create maintenance task when recovery fails repeatedly
🔍 When this triggers:
- All enabled recovery actions have been tried and failed
- Same GPU error recurring multiple times (>3 in 24h)
- Multiple power cycles needed in short period
⚙️ What it does:
- Creates a maintenance task in the dashboard
- Records all recovery attempts and outcomes
- Suggests likely hardware issue (GPU replacement)
🤖 Agent Analysis
📜 Recovery History
No recent recovery actions
Summary Options
Ctrl+click to select multiple
📊 Fleet Summary
Scan for Issues
🔧 Maintenance Tasks
Predictive Failure Analytics
⚠️ Data Requirement: Predictions require at least 30 days of historical data for accuracy. Results improve with more data (90+ days recommended).
📊 Predictions are best-effort estimates similar to weather forecasts - use as guidelines for proactive maintenance, not guarantees.
🔮 Failure Predictions
Root Cause Analysis
Select a device to see its events
Tip: Describe issues not captured in SEL, like PCI errors or GPU issues
🔍 Root Cause Analysis
Try asking:
Which servers need maintenance?
Show ECC errors
NVIDIA server health
Why memory issues on brickbox-06?
👋 Hi! I'm your AI assistant. I have full context of your server fleet (devices, IPs, 72h of events and sensors). Ask me anything about maintenance, health, or issues.
📈 Token Usage & Subscription
Subscription
ENTERPRISE
Token Balance
Loading...
0 used this month
/ 0 monthly
AI Server Slots
0 / 0
⚠️ Low token balance. Top up now
Recent Usage
Loading usage history...
AI Server Selection
Choose which servers get AI monitoring
Loading servers...
⚠️ You've reached your server limit. Upgrade to add more.