# Azure Operations Dashboard Specification

**Status**: Design Document
**Created**: 2026-01-19
**Purpose**: Provide real-time browser-based visibility into Azure VM operations

---

## Problem Statement

When background agents run Azure operations (like WAA Docker rebuilds, benchmark evaluations), users have **NO visibility** into:

1. What step/phase the operation is on
2. How long it's been running
3. Expected time remaining
4. Cost accumulating so far
5. Expected total cost
6. Live logs/output from the VM

This is frustrating because Azure operations can take 10-60+ minutes and cost real money.

---

## Current Infrastructure Audit

### What Exists

| Component | Location | What It Does | Gaps |
|-----------|----------|--------------|------|
| **`vm monitor` command** | `cli.py:4947-5289` | Shows VM status, activity, costs, Azure ML jobs in terminal. Starts dashboard server and SSH tunnels. Polls status every 30s. | **Terminal only**, not browser-based. No granular progress on Docker builds. |
| **Dashboard server** | `local.py:489-1159` | HTTP server with many `/api/*` endpoints for benchmark status, costs, workers, VM probing, tunnels, etc. | Endpoints exist but **no unified UI** that ties them together for Azure ops monitoring. |
| **`benchmark.html`** | Generated by `benchmark_viewer.py` | Shows benchmark results, task replay, success rates. Has "Background Tasks" panel that polls `/api/tasks`. | Focused on **completed** results, not **live operations**. Background tasks panel has limited info. |
| **`live_tracker.py`** | `benchmarks/live_tracker.py` | Writes `benchmark_live.json` for real-time task evaluation progress. | Only tracks **benchmark evaluation** (tasks/steps), not Docker builds or VM setup. |
| **`vm_monitor.py`** | `benchmarks/vm_monitor.py` | `VMMonitor`, `VMActivity`, `calculate_vm_costs()`, `detect_vm_activity()` classes for checking VM state. | Good data collection, but **no continuous streaming** to browser. |
| **`ssh_tunnel.py`** | `cloud/ssh_tunnel.py` | Manages SSH tunnels for VNC (8006) and WAA (5000) access. | Works well for tunnels, but no **log streaming**. |
| **`write_live_status()`** | `cli.py:3403-3443` | Writes status to `benchmark_live.json` during `run-waa` command with phase/detail. | Only used during benchmark run, not during `setup-waa` or Docker builds. |

### API Endpoints Available

```
GET /api/benchmark-progress     # Polls benchmark_progress.json
GET /api/benchmark-live         # Polls benchmark_live.json (live task)
GET /api/tasks                  # Background tasks (limited)
GET /api/azure-jobs             # Azure ML job list (7 days)
GET /api/azure-job-logs?job_id= # Job logs from Azure
GET /api/benchmark/status       # Current benchmark job ETA
GET /api/benchmark/costs        # Cost breakdown
GET /api/benchmark/metrics      # Success rates
GET /api/benchmark/workers      # Worker utilization
GET /api/vms                    # VM registry with status
GET /api/probe-vm               # Probe WAA server
GET /api/tunnels                # SSH tunnel status
GET /api/current-run            # Running benchmark info
GET /api/benchmark-sse          # Server-Sent Events for updates
```

### What's Missing

1. **Unified Azure Operations View**: No single page showing all operations (Docker build, VM setup, benchmark) in one place.

2. **Docker Build Progress Streaming**: When `docker build` runs on VM, we only see "Building..." with no step-by-step output in browser.

3. **Granular Phase Tracking**: `write_live_status()` exists but only covers benchmark phases, not:
   - VM creation/startup
   - Docker installation
   - Docker image build (with step numbers)
   - Windows VM boot/setup inside Docker

4. **Cost-as-you-go Display**: Costs are calculable but not continuously displayed during operations.

5. **Time Estimation**: No ETA based on historical data or progress.

6. **Auto-opening Dashboard**: User must manually open browser; should auto-open when ops start.

---

## Proposed Solution

### Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    Browser Dashboard                            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐  │
│  │ VM Status   │  │ Cost Panel  │  │ Live Logs               │  │
│  │ ● Running   │  │ $0.42/hr    │  │ > Step 5/12: Installing │  │
│  │ 1h 23m up   │  │ Total: $1.82│  │ > Downloading Python... │  │
│  └─────────────┘  └─────────────┘  │ > ...                   │  │
│                                     └─────────────────────────┘  │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │ Progress Bar: ████████████░░░░░░░░ 60% (ETA: 12 min)        │ │
│  └─────────────────────────────────────────────────────────────┘ │
│  ┌─────────────┐  ┌─────────────────────────────────────────────┐│
│  │ VNC Preview │  │ Phase: Docker Build                        ││
│  │ [Thumbnail] │  │ Current Step: Installing WAA dependencies  ││
│  │ [Open VNC]  │  │ Started: 10:23 AM | Duration: 23m 15s      ││
│  └─────────────┘  └─────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
                              │
                     Polls every 2-5s
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Dashboard Server (local.py)                   │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │ GET /api/azure-ops-status                                    ││
│  │   - VM state (running/stopped/booting)                       ││
│  │   - Current operation (setup/build/benchmark)                ││
│  │   - Phase and step within operation                          ││
│  │   - Progress percentage and ETA                              ││
│  │   - Live log tail (last 50 lines)                            ││
│  │   - Running cost                                             ││
│  │   - VNC URL                                                  ││
│  └─────────────────────────────────────────────────────────────┘│
│                              │                                   │
│                     Reads from                                   │
│                              │                                   │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │ azure_ops_status.json (written by CLI commands)              ││
│  │ {                                                            ││
│  │   "operation": "docker_build",                               ││
│  │   "phase": "installing_dependencies",                        ││
│  │   "step": 5, "total_steps": 12,                              ││
│  │   "log_tail": ["...", "..."],                                ││
│  │   "started_at": "2026-01-19T10:23:00Z",                      ││
│  │   "cost_usd": 1.82,                                          ││
│  │   "eta_seconds": 720                                         ││
│  │ }                                                            ││
│  └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
                              │
                     SSH / az CLI
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                       Azure VM                                   │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │ Docker building waa-auto image                               ││
│  │ Output piped to ~/build.log                                  ││
│  └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘
```

### Implementation Plan

#### Phase 1: Unified Status File (1-2 hours)

**Goal**: Create a single source of truth for Azure operations status.

1. Create `/Users/abrichr/oa/src/openadapt-ml/openadapt_ml/benchmarks/azure_ops_tracker.py`:

```python
"""Azure operations status tracker.

Writes real-time status to azure_ops_status.json for dashboard consumption.
"""
from dataclasses import dataclass, asdict
from pathlib import Path
import json
from datetime import datetime

@dataclass
class AzureOpsStatus:
    operation: str  # "idle" | "vm_create" | "docker_install" | "docker_build" | "benchmark"
    phase: str  # e.g., "pulling_base_image", "installing_deps", "running_task_5"
    step: int
    total_steps: int
    progress_pct: float
    log_tail: list[str]  # Last 50 lines
    started_at: str
    elapsed_seconds: float
    eta_seconds: float | None
    cost_usd: float
    hourly_rate_usd: float
    vm_ip: str | None
    vm_state: str  # "running" | "starting" | "stopped" | "deallocated"
    vnc_url: str | None
    error: str | None

class AzureOpsTracker:
    OUTPUT_FILE = Path("benchmark_results/azure_ops_status.json")

    def __init__(self, vm_size: str = "Standard_D4ds_v5"):
        self.vm_size = vm_size
        self.start_time = datetime.now()
        self._status = AzureOpsStatus(...)

    def update(self, operation: str, phase: str, step: int, total_steps: int,
               log_lines: list[str] = None):
        """Update status and write to file."""
        ...

    def append_log(self, line: str):
        """Append a log line (keeps last 50)."""
        ...
```

2. Update CLI commands (`setup-waa`, `run-waa`, `vm monitor`) to use `AzureOpsTracker`.

3. Add `/api/azure-ops-status` endpoint to `local.py` that reads this file.

#### Phase 2: Dashboard HTML (2-3 hours)

**Goal**: Create `azure_ops.html` with real-time polling UI.

1. Create template with:
   - **Header**: Operation name, started time, elapsed time
   - **Progress bar**: Visual progress with percentage and ETA
   - **Cost panel**: Running cost, hourly rate, projected total
   - **VM Status**: State, IP, size
   - **Live logs**: Scrollable terminal-style panel with auto-scroll
   - **VNC button**: Links to `localhost:8006` when available
   - **Controls**: Stop/Cancel button, Open VNC button

2. JavaScript polling:
   - Fetch `/api/azure-ops-status` every 2 seconds
   - Update DOM with new data
   - Auto-scroll logs if user hasn't scrolled up
   - Calculate ETA from progress + elapsed time

3. Add to dashboard server's file serving.

#### Phase 3: Auto-open Browser (30 min)

**Goal**: Automatically open dashboard when Azure ops start.

1. In `setup-waa`, `run-waa` commands:
   - Start dashboard server (existing code)
   - Open `http://localhost:8765/azure_ops.html`

2. Add `--no-open` flag to skip auto-open.

#### Phase 4: Docker Build Progress Parsing (1-2 hours)

**Goal**: Parse Docker build output for step-by-step progress.

Docker build output looks like:
```
Step 1/12 : FROM dockurr/windows:latest
Step 2/12 : COPY oem /oem
Step 3/12 : RUN apt-get update
...
```

1. Parse "Step X/Y" pattern from build output.
2. Map to progress percentage.
3. Stream parsed progress to `AzureOpsTracker`.

Current code already streams build output:
```python
# cli.py:3716-3757
build_process = subprocess.Popen(...)
for line in iter(build_process.stdout.readline, ''):
    print(f"      {line.rstrip()}")
```

Add tracking:
```python
if "Step " in line and "/" in line:
    # Parse "Step 5/12"
    tracker.update_from_docker_line(line)
```

#### Phase 5: Historical ETA (1 hour)

**Goal**: Estimate time remaining based on past runs.

1. Save operation durations to `azure_ops_history.json`:
```json
{
  "docker_build": {"avg_seconds": 1200, "samples": 5},
  "benchmark_per_task": {"avg_seconds": 180, "samples": 100}
}
```

2. Use history for ETA calculation:
```python
eta = (total_steps - current_step) * avg_step_time
```

---

## Cost Tracking Details

VM pricing from `vm_monitor.py`:
```python
VM_HOURLY_RATES = {
    "Standard_D4ds_v5": 0.422,  # 4 vCPU, 16 GB RAM
    "Standard_D8ds_v5": 0.844,  # 8 vCPU, 32 GB RAM
}
```

Cost calculation:
```python
elapsed_hours = (datetime.now() - start_time).total_seconds() / 3600
cost_usd = elapsed_hours * hourly_rate
```

Display format:
- **Current**: "$1.82"
- **Rate**: "$0.42/hr"
- **Projected** (if ETA known): "~$2.50 total"

---

## Operations to Track

| Operation | Phases | Typical Duration | Steps Detectable |
|-----------|--------|------------------|------------------|
| VM Create | Creating, Starting, Configuring | 5-10 min | az CLI output |
| Docker Install | apt update, install, start | 2-5 min | ssh output |
| Docker Build | 12 Dockerfile steps | 10-20 min | "Step X/Y" |
| Windows Boot | QEMU starting, Windows installing | 15-25 min | QMP state |
| Benchmark Run | Tasks 1-N | 30-60 min | `/probe` response |

---

## Files to Create/Modify

### New Files

1. `openadapt_ml/benchmarks/azure_ops_tracker.py` - Status tracking class
2. `openadapt_ml/training/azure_ops_viewer.py` - HTML generation
3. `training_output/azure_ops.html` - Generated dashboard

### Modified Files

1. `openadapt_ml/benchmarks/cli.py`:
   - Import `AzureOpsTracker`
   - Add tracking calls in `setup-waa`, `run-waa`, `vm monitor`
   - Auto-open browser

2. `openadapt_ml/cloud/local.py`:
   - Add `GET /api/azure-ops-status` endpoint
   - Serve `azure_ops.html`

---

## User Experience

### Before (Current)

1. User runs `vm setup-waa` or `run-waa`
2. Terminal shows some output
3. User has no idea how long it will take
4. User can't see what's happening on Windows VM
5. User asks "is it still running?" repeatedly

### After (Proposed)

1. User runs `vm setup-waa` or `run-waa`
2. Browser automatically opens to dashboard
3. User sees:
   - "Phase: Building Docker Image (Step 5/12)"
   - Progress bar at 42%
   - "ETA: 8 minutes"
   - "Cost so far: $0.85"
   - Live logs scrolling
   - "VNC" button to see Windows VM
4. User knows exactly what's happening and can walk away

---

## Implementation Priority

| Priority | Item | Why |
|----------|------|-----|
| P0 | `azure_ops_tracker.py` | Foundation for everything |
| P0 | Basic HTML dashboard | Immediate user value |
| P1 | Docker build step parsing | Most common long operation |
| P1 | Auto-open browser | Reduces friction |
| P2 | Historical ETA | Nice-to-have improvement |
| P2 | VNC thumbnail preview | Cool but not essential |

---

## Testing

1. **Unit tests**: `AzureOpsTracker` writes correct JSON
2. **Integration**: CLI commands update status file
3. **Manual**: Run `vm setup-waa --rebuild` and watch dashboard

---

## Acceptance Criteria

- [ ] Running `vm setup-waa` auto-opens browser dashboard
- [ ] Dashboard shows current operation and phase
- [ ] Docker build shows "Step X/Y" progress
- [ ] Running cost updates every few seconds
- [ ] ETA is displayed and reasonably accurate
- [ ] Live logs scroll automatically
- [ ] VNC button works when available
- [ ] Stop button cancels operation (if safe)

---

## Notes

- Keep it simple: Single HTML file with inline JS, no build step
- Use existing CSS from `benchmark_viewer.py` for consistency
- Polling is fine (no need for WebSockets for MVP)
- Status file approach means multiple viewers can watch same operation
