---
name: infra-perf-setup
description: >
  Bootstrap the Datadog monitors that power API performance alerting — P99 latency, error
  rate, pod restarts, CPU and RAM — creating each one via the Datadog API with a Slack
  notification to #alerts-api-perf, idempotently (existing monitors are skipped, never duplicated).
  Use when the user says /infra-perf-setup, 'setup monitors', 'crée les monitors', 'crée les monitors Datadog',
  'configure les alertes', 'configure l alerting', 'mets en place les monitors', 'bootstrap les monitors',
  'initialise le monitoring Datadog', 'create monitors', 'create the Datadog monitors', 'configure alerting',
  'set up the alerting', 'bootstrap the perf monitors', or needs to provision the Datadog monitors for the API.
  Distinct from /infra-perf (which reads live API latency/error metrics) — /infra-perf-setup
  only creates the alerting monitors, it does not diagnose performance.
allowed-tools:
  - Bash
  - AskUserQuestion
  - mcp__datadog__list_monitors
---

One-shot skill to create Datadog monitors for API performance alerting.

## Required tools

- `aws`
- `datadog-mcp`

Run `pysae-ai-tools tools require <tools>` before starting; if it exits non-zero, abort and report the install command from stderr to the user.

## Reference files

- **[references/monitors.md](references/monitors.md)** — monitor templates with exact queries, thresholds, and messages
- **[../infra-perf/references/services.md](../infra-perf/references/services.md)** — service alias mapping

## Prerequisites

1. **Slack integration** must be configured in Datadog for `#alerts-api-perf` channel.
2. Channel `#alerts-api-perf` must exist in Slack.

## Execution flow

### Step 1: Load credentials

```bash
eval "$(pysae-ai-tools env resolve --set DD_API_KEY DD_APP_KEY)" || exit 1
```

### Step 2: List existing monitors

Use Datadog MCP to list all monitors with tag `source:perf-setup`. Record their names to avoid duplicates.

### Step 3: Create monitors

For each monitor template in `references/monitors.md`:

1. Check if a monitor with the same `name` already exists (from Step 2)
2. If exists → skip and log "already exists"
3. If not → create via Datadog API:

```bash
curl -s -X POST "https://api.datadoghq.eu/api/v1/monitor" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -d '<monitor JSON from template>'
```

For per-service monitors (P99 latency, error rate), iterate over all services from `services.md` and replace `<SERVICE>` placeholder.

### Step 4: Report results

Output a summary table:

```
Datadog Monitors — Setup Report

Monitor                                    Status
---------------------------------------------------------------
[API Perf] P99 latency — api               Created
[API Perf] P99 latency — api-fast          Created
[API Perf] P99 latency — api-ndp           Already exists
...
[API Perf] Error rate — api                 Created
...
[API Perf] Pod restarts — prod              Created
[API Perf] CPU high — prod                  Created
[API Perf] RAM high — prod                  Created

Total: X created, Y already existed, Z failed
```

### Step 5: Verify

After creation, list monitors again via MCP and confirm the count matches expectations.

## Critical rules

- **Idempotent**: Never create duplicate monitors. Always check first.
- **Ask before creating**: Show the user what will be created and ask for confirmation before running any curl commands.
- **No deletion**: This skill creates monitors only. To delete or modify, use Datadog UI.
- **Language**: Explanations in French, monitor names/messages in English.
