---
name: infra-perf
description: >
  API performance diagnostics — latency, error rates, pods, CPU/RAM across all
  API services. Supports global check, per-service focus, custom time windows,
  and comparison against a previous period.
  Use when the user says /infra-perf, 'api performance', 'latence prod',
  'perfs api', 'latency', 'error rate', 'health check api', 'comment va l api',
  'l api rame', 'la prod est lente', 'le service est lent', or needs
  diagnostics on the live API in production. Distinct from /code-profile,
  which profiles a specific code path / test / app run with a real profiler
  (cProfile, py-spy, clinic, chrome-devtools) — use /code-profile instead
  whenever the user points at a file, function, test, or page rather than
  a deployed service.
argument-hint: "[service: ndp|api-fast|api|gtfsrt|stats|evt|simu|docs] [window: 15m|30m|1h|2h|4h|12h|24h] [compare: yesterday|-Nd]"
allowed-tools:
  - Bash
  - Agent
  - mcp__datadog__get_logs
  - mcp__datadog__list_monitors
  - mcp__datadog__get_metrics
  - mcp__datadog__list_metrics
  - mcp__kubernetes__kubectl_get
---

API performance diagnostics skill. Queries Datadog APM and Kubernetes to produce a unified health report.

## Required tools

- `aws`
- `kubectl`
- `atlas`
- `datadog-mcp`
- `jq`

Run `pysae-ai-tools tools require <tools>` before starting; if it exits non-zero, abort and report the install command from stderr to the user.

## Reference files

- **[references/services.md](references/services.md)** — service alias mapping (alias → Datadog service → K8S deployment)
- **[references/thresholds.md](references/thresholds.md)** — warning/critical thresholds
- **`/project-configuration`** — per-repo mappings (Datadog service, k8s deployment) from each repo's config

## Argument parsing

Parse `$ARGUMENTS` using these rules:

1. **Service filter**: If a token matches an alias from `services.md` (e.g. `ndp`, `api-fast`), scope to that service only. If absent, check all services.
2. **Time window**: If a token matches `\d+[mhd]` pattern (e.g. `2h`, `30m`, `24h`), use as lookback window. Default: `1h`.
3. **Compare mode**: If `compare` keyword is present, the next token is the reference period:
   - `yesterday` → same time window, shifted -24h
   - `-Nd` (e.g. `-7d`) → same time window, shifted -N days

Examples:
- `/infra-perf` → all services, 1h window, no compare
- `/infra-perf ndp` → NDP only, 1h window
- `/infra-perf api-fast 2h` → api-fast, 2h window
- `/infra-perf ndp compare yesterday` → NDP, 1h, vs yesterday
- `/infra-perf ndp compare -7d` → NDP, 1h, vs 7 days ago

## Execution flow

### Step 1: Resolve parameters

Parse arguments per rules above. Resolve service alias(es) to Datadog service name(s) and K8S deployment name(s) using `services.md`.

### Step 2: Collect data in parallel

Launch parallel queries using Agent sub-agents:

**Agent A — Datadog metrics** (one per service):

For each service, query Datadog MCP for:
- P50, P95, P99 latency
- Error rate
- Request throughput (req/s)

Use the Datadog metric query patterns from `services.md`. Time window per parsed arguments.

If compare mode is active, also query the reference period (same metrics, shifted time window).

**Agent B — Kubernetes status**:

```bash
# Pods status for each service
kubectl get pods -n prod -l app=<k8s_deployment> --no-headers 2>/dev/null | \
  awk '{ready=$2; restarts=$4; status=$3; print ready, status, restarts}'

# Resource usage for each service
kubectl top pods -n prod -l app=<k8s_deployment> --no-headers 2>/dev/null
```

If checking all services, run a single command per type:
```bash
kubectl get pods -n prod --no-headers 2>/dev/null
kubectl top pods -n prod --no-headers 2>/dev/null
```

Then filter results by deployment name.

**Agent C — MongoDB Atlas** (only when a service-level metric crossed the warning threshold, or the run is global):

The API is MongoDB-bound; when latency or error rate spikes, the cause is often DB-side (slow ops piling up, IOPS saturation, missing index). Pull the cheap signals first:

```bash
# Find the cluster's primary process for the prod profile
atlas -P pysae-prod processes list --output json | jq -r '.results[] | select(.typeName=="REPLICA_PRIMARY") | .id'

# Top slow ops in the last hour (use the host id from above as --processName)
atlas -P pysae-prod performanceAdvisor slowQueryLogs list --processName <host:port> --output json

# Index suggestions surfaced by Atlas's Performance Advisor
atlas -P pysae-prod performanceAdvisor suggestedIndexes list --output json
```

If the API window is `2h`, scope Atlas queries with `--since`/`--duration` to match. Skip this agent in compare mode — Atlas Performance Advisor is not period-windowed and would add noise to the comparison.

### Step 3: Aggregate and assess

For each service, build a row with:
- P50, P95, P99 (format: ms if <1s, e.g. `340ms`; seconds if >=1s, e.g. `1.2s`)
- Error rate (format: `0.2%`, `1.8%`)
- Throughput (format: `84 req/s`)
- Pods (format: `ready/total`, e.g. `3/3` or `4/5`)
- CPU (format: `42%`)
- RAM (format: `61%`)

Apply thresholds from `thresholds.md`:
- Add `!` for warning, `!!` for critical
- In compare mode, also compute delta % and flag per relative thresholds

### Step 4: Format output

```
API Performance — last <window> [vs <compare_ref>]

Service          P50    P95    P99    Err%   Req/s  Pods  CPU   RAM
----------------------------------------------------------------------
<service>        <val>  <val>  <val>  <val>  <val>  <val> <val> <val>  [!|!!]
...

[! <service>: <detail of each warning/critical>]
```

If in compare mode, add delta in parentheses for flagged metrics:
```
api-ndp          89ms   450ms  1.2s   1.8%   126    4/5   78%   73%  !
  ! P99 1.2s (+340% vs yesterday), error rate 1.8% (+200%)
```

### Step 5: Recommendations (if issues found)

If any metric is at critical level, add a short recommendations section:
- High latency → if Atlas Performance Advisor surfaced slow ops, list the top 3 with their collection + suggested index. Otherwise suggest checking recent deploys.
- High error rate → suggest checking logs via `/infra-ops` or Datadog
- High CPU/RAM → suggest scaling or checking for memory leaks
- Pod restarts → suggest checking crash logs via `/infra-ops`

## Critical rules

- **Read-only**: This skill NEVER modifies infrastructure. No scaling, no restarts, no deployments.
- **Fail gracefully**: when only Kubernetes data is unavailable, skip K8S rows and show Datadog-only results.
- **Language**: Explanations in French, commands in English.
- **No guessing**: If a metric query returns no data, show `-` not a made-up value.
