# Run Benchmark UI Design

## Overview

Add a "Run Benchmark" panel to the Benchmarks tab in the dashboard (http://localhost:8765/benchmark.html) to allow users to configure and launch WAA benchmark runs directly from the UI instead of using the CLI.

## Current State

- Dashboard served from `openadapt_ml/cloud/local.py` via `cmd_serve()`
- Existing API endpoint: `POST /api/run-benchmark` (basic, only supports `provider` and `tasks`)
- CLI command: `uv run python -m openadapt_ml.benchmarks.cli vm run-waa --num-tasks 5`
- Progress tracking via SSE at `/api/benchmark-sse` and polling at `/api/benchmark-progress`
- Benchmark viewer generated by `openadapt_ml/training/benchmark_viewer.py`

## UI Design

### 1. Placement and Layout

Add a new "Run Benchmark" panel at the top of the Benchmarks tab, above the Background Tasks panel.

```
+------------------------------------------------------------------+
|  Benchmarks Tab                                                   |
+------------------------------------------------------------------+
|                                                                   |
|  +------------------------------------------------------------+  |
|  |  Run Benchmark                              [Start Run]     |  |
|  +------------------------------------------------------------+  |
|  |  Model: [GPT-4o ▼]     Tasks: [5 ▼]     Agent: [navi ▼]    |  |
|  |                                                             |  |
|  |  Task Selection: ( ) All tasks (154)                        |  |
|  |                  ( ) Domain: [General ▼]                    |  |
|  |                  ( ) Task IDs: [________________]           |  |
|  +------------------------------------------------------------+  |
|                                                                   |
|  +------------------------------------------------------------+  |
|  |  Background Tasks                          [Refresh]        |  |
|  +------------------------------------------------------------+  |
|  |  ...existing task cards...                                  |  |
|  +------------------------------------------------------------+  |
|                                                                   |
```

### 2. Configuration Options

#### Model Selection (dropdown)
```javascript
const MODELS = [
  { id: 'gpt-4o', label: 'GPT-4o', provider: 'openai' },
  { id: 'gpt-4o-mini', label: 'GPT-4o Mini', provider: 'openai' },
  { id: 'claude-sonnet-4-5-20250929', label: 'Claude Sonnet 4.5', provider: 'anthropic' },
  { id: 'claude-opus-4-5-20251101', label: 'Claude Opus 4.5', provider: 'anthropic' },
  { id: 'custom', label: 'Custom...', provider: null }
];
```
- Default: `gpt-4o`
- When "Custom..." selected, show text input for model ID

#### Number of Tasks (dropdown or input)
```javascript
const TASK_COUNTS = [1, 5, 10, 20, 30, 50, 100, 154];
```
- Default: `5`
- Max: `154` (total WAA tasks)
- Can type custom number (1-154)

#### Task Selection (radio group)
- **All tasks**: Run random selection from all 154 tasks
- **Domain**: Dropdown with WAA domains:
  ```javascript
  const DOMAINS = [
    'general', 'office', 'web', 'coding', 'system',
    'creative', 'data', 'communication', 'media', 'gaming', 'utility'
  ];
  ```
- **Task IDs**: Comma-separated list of specific task IDs (e.g., `task_001,task_015,task_042`)

#### Agent Type (dropdown)
```javascript
const AGENTS = [
  { id: 'navi', label: 'Navi (default)' },
  { id: 'som', label: 'Set-of-Marks' },
  { id: 'random', label: 'Random (baseline)' }
];
```
- Default: `navi`

### 3. API Endpoint

Extend existing `/api/run-benchmark` endpoint or create new `/api/benchmark/start`:

```python
# In local.py StopHandler.do_POST()
elif self.path == '/api/benchmark/start':
    content_length = int(self.headers.get('Content-Length', 0))
    body = self.rfile.read(content_length).decode('utf-8')
    params = json.loads(body)

    # Expected params:
    # {
    #   "model": "gpt-4o",
    #   "num_tasks": 5,
    #   "agent": "navi",
    #   "task_selection": "all" | "domain" | "task_ids",
    #   "domain": "general",  // if task_selection == "domain"
    #   "task_ids": ["task_001", "task_015"]  // if task_selection == "task_ids"
    # }

    result = start_benchmark_run(params)
    self.send_json_response(result)
```

#### Backend Implementation

```python
def start_benchmark_run(params: dict) -> dict:
    """Start a benchmark run with the given parameters.

    Runs the benchmark in a background thread and returns immediately.
    Progress is tracked via benchmark_progress.json.
    """
    import subprocess
    import threading

    # Build CLI command
    cmd = [
        "uv", "run", "python", "-m", "openadapt_ml.benchmarks.cli",
        "vm", "run-waa",
        "--num-tasks", str(params.get("num_tasks", 5)),
        "--model", params.get("model", "gpt-4o"),
        "--agent", params.get("agent", "navi"),
        "--no-open"  # Don't open viewer (already open)
    ]

    # Add task selection args
    if params.get("task_selection") == "domain":
        cmd.extend(["--domain", params.get("domain", "general")])
    elif params.get("task_selection") == "task_ids":
        task_ids = params.get("task_ids", [])
        cmd.extend(["--task-ids", ",".join(task_ids)])

    # Write initial progress
    progress_file = Path("benchmark_progress.json")
    progress_file.write_text(json.dumps({
        "status": "starting",
        "model": params.get("model"),
        "num_tasks": params.get("num_tasks"),
        "tasks_complete": 0,
        "message": "Starting benchmark run..."
    }))

    # Run in background thread
    def run():
        env = os.environ.copy()
        result = subprocess.run(cmd, capture_output=True, text=True, env=env)

        if result.returncode == 0:
            progress_file.write_text(json.dumps({
                "status": "complete",
                "message": "Benchmark complete. Refresh to see results."
            }))
            # Regenerate benchmark viewer
            _regenerate_benchmark_viewer_if_available(serve_dir)
        else:
            progress_file.write_text(json.dumps({
                "status": "error",
                "message": f"Benchmark failed: {result.stderr[:200]}"
            }))

    threading.Thread(target=run, daemon=True).start()

    return {"status": "started", "params": params}
```

### 4. Progress Tracking

Reuse existing SSE infrastructure (`/api/benchmark-sse`) which already:
- Sends `status` events with VM readiness
- Sends `progress` events with tasks_completed/total_tasks
- Sends `task_complete` events when tasks finish

The UI should:
1. Disable "Start Run" button when benchmark is running
2. Show progress bar in the panel header
3. Update task counts in real-time via SSE
4. Show "View Results" button when complete

### 5. Result Display

When benchmark completes:
1. Auto-refresh the results table below
2. Show success/failure summary in the Run panel
3. Enable "View Details" to scroll to the full results

```javascript
// After benchmark complete SSE event
function onBenchmarkComplete(result) {
    // Update Run panel
    showCompletionSummary(result);

    // Refresh results table
    refreshBenchmarkResults();

    // Show notification
    showNotification(`Benchmark complete: ${result.success_rate}% success`);
}
```

## Implementation Files

### Files to Modify

1. **`openadapt_ml/training/benchmark_viewer.py`**
   - Add `_get_run_benchmark_panel_css()` function
   - Add `_get_run_benchmark_panel_html()` function
   - Call from `generate_multi_run_benchmark_viewer()` or `generate_empty_benchmark_viewer()`

2. **`openadapt_ml/cloud/local.py`**
   - Add `POST /api/benchmark/start` handler in `StopHandler.do_POST()`
   - Add `start_benchmark_run()` helper function
   - Update progress file path handling

3. **`openadapt_ml/benchmarks/cli.py`**
   - Add `--domain` flag to `vm run-waa` subcommand
   - Add `--task-ids` flag to filter specific tasks

### CSS Styling Pattern

Follow existing patterns from `benchmark_viewer.py`:

```python
def _get_run_benchmark_panel_css() -> str:
    return '''
        .run-benchmark-panel {
            background: linear-gradient(135deg, rgba(16, 185, 129, 0.1) 0%, rgba(16, 185, 129, 0.05) 100%);
            border: 1px solid rgba(16, 185, 129, 0.3);
            border-radius: 12px;
            padding: 20px 24px;
            margin-bottom: 24px;
        }
        .run-benchmark-header {
            display: flex;
            align-items: center;
            justify-content: space-between;
            margin-bottom: 16px;
        }
        .run-benchmark-title {
            display: flex;
            align-items: center;
            gap: 10px;
            font-size: 1rem;
            font-weight: 600;
            color: #10b981;
        }
        .run-benchmark-form {
            display: grid;
            grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
            gap: 16px;
            margin-bottom: 16px;
        }
        .form-group {
            display: flex;
            flex-direction: column;
            gap: 6px;
        }
        .form-group label {
            font-size: 0.8rem;
            color: var(--text-secondary);
            font-weight: 500;
        }
        .form-group select,
        .form-group input {
            padding: 8px 12px;
            background: rgba(0, 0, 0, 0.3);
            border: 1px solid var(--border-color);
            border-radius: 6px;
            color: var(--text-primary);
            font-size: 0.9rem;
        }
        .start-btn {
            padding: 10px 20px;
            background: linear-gradient(135deg, #10b981, #059669);
            border: none;
            border-radius: 8px;
            color: white;
            font-weight: 600;
            cursor: pointer;
            transition: all 0.2s;
        }
        .start-btn:hover:not(:disabled) {
            transform: translateY(-2px);
            box-shadow: 0 4px 12px rgba(16, 185, 129, 0.3);
        }
        .start-btn:disabled {
            opacity: 0.5;
            cursor: not-allowed;
        }
    '''
```

### JavaScript Form Handling

```javascript
async function startBenchmarkRun() {
    const params = {
        model: document.getElementById('benchmark-model').value,
        num_tasks: parseInt(document.getElementById('benchmark-tasks').value),
        agent: document.getElementById('benchmark-agent').value,
        task_selection: document.querySelector('input[name="task-selection"]:checked').value
    };

    if (params.task_selection === 'domain') {
        params.domain = document.getElementById('benchmark-domain').value;
    } else if (params.task_selection === 'task_ids') {
        params.task_ids = document.getElementById('benchmark-task-ids').value
            .split(',')
            .map(id => id.trim())
            .filter(id => id);
    }

    // Disable button
    const btn = document.getElementById('start-benchmark-btn');
    btn.disabled = true;
    btn.textContent = 'Starting...';

    try {
        const response = await fetch('/api/benchmark/start', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify(params)
        });

        const result = await response.json();

        if (result.status === 'started') {
            btn.textContent = 'Running...';
            // SSE will handle progress updates
        } else {
            throw new Error(result.error || 'Failed to start');
        }
    } catch (e) {
        alert('Failed to start benchmark: ' + e.message);
        btn.disabled = false;
        btn.textContent = 'Start Run';
    }
}
```

## Testing

1. **Manual testing**: Open http://localhost:8765/benchmark.html, configure options, click Start
2. **API testing**: `curl -X POST localhost:8765/api/benchmark/start -d '{"model":"gpt-4o","num_tasks":1}'`
3. **Progress tracking**: Watch SSE updates in browser DevTools Network tab
4. **Error handling**: Test with invalid model, no VM, etc.

## Future Enhancements

1. **Model comparison mode**: Run same tasks with multiple models and show side-by-side results
2. **Scheduled runs**: Allow scheduling benchmark runs for off-peak hours
3. **Presets**: Save/load benchmark configurations
4. **Cost estimation**: Show estimated API costs before starting run
