---
name: infra-perf-setup
description: >
  Bootstrap Datadog monitors for API performance alerting — P99 latency, error
  rate, pod restarts, CPU, RAM. Creates monitors via Datadog API with Slack
  notification to #alerts-api-perf. Idempotent (skips existing monitors).
  Use when the user says /infra-perf-setup, 'setup monitors',
  'create monitors', 'configure alerting', 'bootstrap monitors', or needs
  to create Datadog monitors.
allowed-tools:
  - Bash
  - AskUserQuestion
  - mcp__datadog__list_monitors
---

One-shot skill to create Datadog monitors for API performance alerting.

## Required tools

- `aws`
- `datadog-mcp`

Run `pysae-ai-tools tools require <tools>` before starting; if it exits non-zero, abort and report the install command from stderr to the user.

## Reference files

- **[references/monitors.md](references/monitors.md)** — monitor templates with exact queries, thresholds, and messages
- **[../infra-perf/references/services.md](../infra-perf/references/services.md)** — service alias mapping

## Prerequisites

1. **Slack integration** must be configured in Datadog for `#alerts-api-perf` channel.
2. Channel `#alerts-api-perf` must exist in Slack.

## Execution flow

### Step 1: Load credentials

```bash
eval "$(pysae-ai-tools env resolve --set DD_API_KEY DD_APP_KEY)" || exit 1
```

### Step 2: List existing monitors

Use Datadog MCP to list all monitors with tag `source:perf-setup`. Record their names to avoid duplicates.

### Step 3: Create monitors

For each monitor template in `references/monitors.md`:

1. Check if a monitor with the same `name` already exists (from Step 2)
2. If exists → skip and log "already exists"
3. If not → create via Datadog API:

```bash
curl -s -X POST "https://api.datadoghq.eu/api/v1/monitor" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -d '<monitor JSON from template>'
```

For per-service monitors (P99 latency, error rate), iterate over all services from `services.md` and replace `<SERVICE>` placeholder.

### Step 4: Report results

Output a summary table:

```
Datadog Monitors — Setup Report

Monitor                                    Status
---------------------------------------------------------------
[API Perf] P99 latency — api               Created
[API Perf] P99 latency — api-fast          Created
[API Perf] P99 latency — api-ndp           Already exists
...
[API Perf] Error rate — api                 Created
...
[API Perf] Pod restarts — prod              Created
[API Perf] CPU high — prod                  Created
[API Perf] RAM high — prod                  Created

Total: X created, Y already existed, Z failed
```

### Step 5: Verify

After creation, list monitors again via MCP and confirm the count matches expectations.

## Critical rules

- **Idempotent**: Never create duplicate monitors. Always check first.
- **Ask before creating**: Show the user what will be created and ask for confirmation before running any curl commands.
- **No deletion**: This skill creates monitors only. To delete or modify, use Datadog UI.
- **Language**: Explanations in French, monitor names/messages in English.
