# Datadog Monitor Templates

Templates for API performance monitors. Each monitor targets `env:prod` and notifies `#alerts-api-perf` via Slack.

## Monitor definitions

### 1. P99 Latency per service

```json
{
  "name": "[API Perf] P99 latency high — {service}",
  "type": "metric alert",
  "query": "avg(last_5m):avg:trace.fastapi.request.duration.by.service.99p{env:prod,service:<SERVICE>} > 2",
  "message": "P99 latency for {{service.name}} is {{value}}s (threshold: {{threshold}}s).\n\nCheck `/infra-perf <alias>` for details.\n\n@slack-alerts-api-perf",
  "options": {
    "thresholds": {
      "critical": 2,
      "warning": 0.8
    },
    "notify_no_data": false,
    "renotify_interval": 30,
    "evaluation_delay": 60
  },
  "tags": ["team:api", "source:perf-setup"],
  "priority": 3
}
```

Apply for each service in [../infra-perf/references/services.md](../infra-perf/references/services.md): replace `<SERVICE>` with the Datadog service name.

### 2. Error rate per service

```json
{
  "name": "[API Perf] Error rate high — {service}",
  "type": "metric alert",
  "query": "avg(last_5m):sum:trace.fastapi.request.errors{env:prod,service:<SERVICE>}.as_rate() / sum:trace.fastapi.request.hits{env:prod,service:<SERVICE>}.as_rate() * 100 > 5",
  "message": "Error rate for {{service.name}} is {{value}}% (threshold: {{threshold}}%).\n\nCheck `/infra-perf <alias>` for details.\n\n@slack-alerts-api-perf",
  "options": {
    "thresholds": {
      "critical": 5,
      "warning": 1
    },
    "notify_no_data": false,
    "renotify_interval": 30
  },
  "tags": ["team:api", "source:perf-setup"],
  "priority": 3
}
```

### 3. Pod restarts

```json
{
  "name": "[API Perf] Pod restarts — prod",
  "type": "metric alert",
  "query": "max(last_10m):kubernetes.containers.restarts{kube_namespace:prod} by {kube_deployment} > 5",
  "message": "Deployment {{kube_deployment.name}} has {{value}} restarts in the last 10 minutes.\n\n@slack-alerts-api-perf",
  "options": {
    "thresholds": {
      "critical": 5,
      "warning": 2
    },
    "notify_no_data": false,
    "renotify_interval": 60
  },
  "tags": ["team:api", "source:perf-setup"],
  "priority": 2
}
```

### 4. CPU per deployment

```json
{
  "name": "[API Perf] CPU high — prod",
  "type": "metric alert",
  "query": "avg(last_5m):avg:kubernetes.cpu.usage.total{kube_namespace:prod} by {kube_deployment} / avg:kubernetes.cpu.limits{kube_namespace:prod} by {kube_deployment} * 100 > 95",
  "message": "CPU for {{kube_deployment.name}} is {{value}}% of limit.\n\n@slack-alerts-api-perf",
  "options": {
    "thresholds": {
      "critical": 95,
      "warning": 80
    },
    "notify_no_data": false,
    "renotify_interval": 30
  },
  "tags": ["team:api", "source:perf-setup"],
  "priority": 3
}
```

### 5. RAM per deployment

```json
{
  "name": "[API Perf] RAM high — prod",
  "type": "metric alert",
  "query": "avg(last_5m):avg:kubernetes.memory.usage{kube_namespace:prod} by {kube_deployment} / avg:kubernetes.memory.limits{kube_namespace:prod} by {kube_deployment} * 100 > 95",
  "message": "RAM for {{kube_deployment.name}} is {{value}}% of limit.\n\n@slack-alerts-api-perf",
  "options": {
    "thresholds": {
      "critical": 95,
      "warning": 85
    },
    "notify_no_data": false,
    "renotify_interval": 30
  },
  "tags": ["team:api", "source:perf-setup"],
  "priority": 3
}
```

## Naming convention

All monitors are prefixed with `[API Perf]` for easy filtering in Datadog.

## Idempotence

Before creating a monitor, search for existing monitors with the same name using Datadog MCP `list_monitors`. If found, skip creation and report "already exists".
