---
name: aws-eks-optimize
description: >
  Optimize EKS node groups — SPOT and ON_DEMAND — by crossing AWS's Spot Advisor
  (predictive interruption rates), our own CloudTrail eviction history, spot &
  on-demand prices, effective CPU performance (clock × IPC per microarchitecture),
  and the real per-node CPU usage from Datadog (to surface cheaper burstable `t*` /
  flex `-flex` types on lightly-loaded pools where the load fits under their
  baseline). For each environment and node group it proposes several
  instance_types mixes (same vCPU/RAM shape and arch) and shows the evolution of
  four factors (cost, performance, cost/perf, eviction risk) plus a Terraform
  list. A greenfield mode (--cpu/--ram) designs a brand-new node group. Use when
  the user says /aws-eks-optimize, 'optimise les node groups', 'réduis les
  évictions spot', 'réduis le coût', 'quelles instances mettre', 'analyse les
  préemptions par environnement', 'diversifier dev-bo', 'dimensionne un nouveau
  node group', or wants to tune EKS instance types for cost, performance or
  reliability.
argument-hint: "[--region eu-west-3] [--period 2w | --from … --to …] [--goal cost|evictions|balanced] [--no-cpu] [--exclude-throttled <ng>] [--force-throttled <ng>] | --cpu N --ram G --capacity spot|on-demand"
allowed-tools:
  - Bash
---

Optimize EKS node groups (SPOT **and** ON_DEMAND) by crossing reliability, cost
and effective performance, proposing concrete `instance_types` mixes per
environment and node group.

## Required tools

- `aws`

Run `pysae-ai-tools tools require aws || exit 1` before starting; the command exits non-zero with the install command on stderr if any tool is missing or out of date.

**Optional — Datadog CPU head-room analysis.** To let the skill check each node group's real sustained CPU load (and unlock cheaper throttled `t*`/`-flex` types where the load fits — see below), make `DD_API_KEY` / `DD_APP_KEY` resolvable: `eval "$(pysae-ai-tools env resolve --set DD_API_KEY DD_APP_KEY)"` before running. Without them the analysis is silently skipped (static baseline penalty applies) — use `--no-cpu` to skip it explicitly.

## What it does

For every managed node group of every EKS cluster in the region, it takes the
set of **compatible** instance types (same vCPU/RAM shape and architecture;
**GPU/accelerated families — `g/p/inf/trn/dl/vt/f` — are dropped outright**, they
are never a fit for a general-purpose node group) and gives each a single
**global score 0-100** that weighs three axes:

- **Reliability** — three signals, **weighted, never used to exclude**:
  1. **observed** interruption rate = evictions / launches from CloudTrail
     (`BidEvictedEvent` × `RunInstances`, attributed via the `eks:nodegroup-name`
     tag, keyed on the node group **prefix** so a recreated group keeps its
     history) — our real history; **weight 0.50**. The rate is **region-wide per
     instance type** (pooled across node groups): spot reclamation is a pool
     property (type × AZ × region), so a type's rate is the same whichever node
     group runs it — the per-node-group total is shown separately in the group
     header. The `Obs.` column shows both the rate **and the absolute counts** as
     `rate% (evictions/launches)` so a `100% (1/1)` sample is not read like a
     heavily-evicted `42% (60/142)` pool;
  2. **Spot Advisor predictive** rate (`<5%`…`>20%`) — global, static, less
     reliable (e.g. m6a was predicted `<5%` yet observed ~40% in eu-west-3);
     **weight 0.15**. Weights renormalize over the signals available — a
     never-run type (no observed rate) leans on the predictive rate.

  The **Spot Placement Score** (1-10, `get-spot-placement-scores`) is **not** a
  per-type signal: AWS scores a whole diversified *request* (a set of ≥3 types)
  as one configuration and returns a single pooled score — one or two types are
  always scored low. It is therefore evaluated **per mix**. For each SPOT node
  group the skill builds up to **three candidate mixes** by composition —
  **score**-led (top global score), **reliability**-led (lowest observed +
  predictive risk), **cost**-led (cheapest spot $/h) — and scores each (plus the
  current mix) by placement. The **chosen** mix is the best mean global score,
  **except** that a candidate within a small score band that places clearly
  better wins (placement is the tie-breaker, never overrides a real score gap).
- **Cost** — average spot $/h (`describe-spot-price-history`) or on-demand $/h
  (Price List API) over the window.
- **Performance** — `vCPU × clock × IPC` (clock from `describe-instance-types`,
  IPC curated per CPU microarchitecture: Intel Skylake→Granite Rapids, AMD
  Naples→Turin, Graviton 2-4), i.e. effective compute → cost per unit of work.

Global weights default to **50 % reliability / 30 % cost / 20 % perf**
(`--goal cost` or `evictions` re-weights). **The recommended mix = the
best-scoring types** of the preferred families. The realistic remainder
(general/compute/memory families) is shown ranked, for comparison, as *other
available instances*. **Burstable (`t*`) is hidden** from that listing unless a
node group is currently running it — in which case the in-use type stays visible
(flagged `off-fam`).

**The whole report is written in English** (tables, labels, recap, and the
synthesis you append to the file) — keep it English-only.

CloudTrail history is 90 days → window capped at 90 days. Default arch `x86_64`
(`--arch arm64` for Graviton — needs arm64 images).

## Run (existing node groups)

```bash
pysae-ai-tools aws eks-optimize [--region <region>] [--period <window>] [--goal <goal>]
```

- `--region` / `-r`: AWS code or alias (default `eu-west-3`).
- **Time range** (capped at 90 days), two ways:
  - relative: `--period 7d` / `24h` / `2w` / `1m` (month = 30d), default `2w`;
  - absolute: `--from 2026-05-01 [--to 2026-05-31]` (`--to` defaults to now).
- `--goal balanced|cost|evictions` (default `balanced`): score weighting
  (reliability/cost/perf = 0.65/0.20/0.15 · 0.40/0.45/0.15 · 0.80/0.12/0.08).
- `--family <prefix>` (repeatable) / `--all-families`: which families are
  **recommendable** (the recommended mix; burstable + accelerated excluded by
  default). The compared field keeps the realistic CPU families (GPU/accelerated
  are always dropped; burstable + flex shown only when currently in use). Pass
  `--all-families` to let burstable (`t*`) **and flex (`*-flex`)** compete in the
  recommended mix — useful for CPU-tolerant pools (CI runners, batch). They are not
  banned, just penalised: their **perf is the sustained (baseline) figure** (see below).
- **Throttled-type perf is baseline-adjusted.** For `t*` and `*-flex` types the
  **perf, cost/perf and score use the *sustained* compute** (`vCPU × clock × IPC ×
  baseline_ratio`) — `t*` at its per-size credit baseline, `*-flex` at its 40 %
  baseline (full clock only ~95 % of the time over 24 h) — not the burst peak. So
  they are honestly downgraded on perf but can still win on cost/reliability for a
  tolerant pool. Rows are tagged `burst` / `flex`, and any pool with such a type
  (current or proposed) gets a CPU-steal warning. Read it by pool: fine for CI
  runners / batch (slower jobs occasionally, few evictions, cheap), to avoid for
  latency-sensitive services (API).
- **Datadog CPU head-room (data-driven).** When `DD_API_KEY`/`DD_APP_KEY` are
  resolvable, the skill reads host-level system metrics (one series per node) over
  the window and takes the **p95 real-demand utilisation per node** as the sustained
  load: `(100 − system.cpu.idle − system.cpu.stolen)/100`. `stolen` is subtracted
  because hypervisor-stolen CPU is **not** workload usage — on an already-throttled
  burstable/flex node it would otherwise inflate the demand and reject the very types
  we are sizing for. Host metrics are used rather than `kubernetes.cpu.usage.total`
  because the latter is emitted at several cardinalities (container/pod/node) —
  `avg by host` under-counts and `sum by host` double-counts (can exceed the node's
  physical vCPU). If that sustained utilisation
  fits **under a throttled type's baseline fraction** (`baseline_ratio`, with a 20 %
  safety margin — fractions compare directly since the candidates share the shape),
  the throttle never fires → the type becomes **recommendable at full (un-penalised)
  perf**, and the pool gets a green "fits" note instead of the warning. This is what
  makes lightly-loaded dev pools surface `t*`/`-flex` as the cheap, correct choice.
  Metrics are matched on the node group **name prefix + wildcard** (`eks_nodegroup-name:<prefix>-*`)
  so a recreated node group (new Terraform suffix) still resolves to its history.
  The node group header also shows the **p99 utilisation** for context (peak amplitude)
  — it is **informational only** and does not gate the decision: short peaks above the
  p95 are exactly what a burstable/flex burst absorbs.
- **No CPU signal ≠ a recommendation.** Some pools have no Datadog metric at all
  (ephemeral / agent-less nodes — typically CI runners). There the head-room can't be
  computed, so throttled types stay non-recommended **by static penalty**, and the
  pool gets an explicit `ℹ️ Head-room CPU non calculable` note — do **not** read its
  fixed-perf mix as a load-based call. For a throttle-tolerant pool like a CI runner,
  re-run with `--force-throttled <ng-prefix>`.
- `--no-cpu`: skip the Datadog CPU lookup (offline runs, or no DD keys).
- `--exclude-throttled <ng-prefix>` (repeatable): never recommend `t*`/`-flex` on
  matching node groups, regardless of head-room — e.g. `--exclude-throttled prod-api`.
- `--force-throttled <ng-prefix>` (repeatable): force `t*`/`-flex` recommendable at
  full perf on matching node groups — for pools you *know* tolerate throttling
  (CI runners) even without a confirming CPU signal.
- `--arch x86_64|arm64` (default `x86_64`).
- `--no-placement`: skip the Spot Placement Score lookups entirely. Lookups are
  already quota-frugal by default — placement is scored **per mix** (the current
  set plus up to three candidate mixes per node group, each ≥3 types, the way AWS
  means the score to be used), not one per type. Identical mixes across node
  groups dedupe to a single call, and a **24h on-disk cache** replays a
  recently-seen score for free (AWS itself reuses a config requested within 24h
  without charging the daily configuration budget). On a `MaxConfigLimitExceeded`
  the loop **stops early** and the quota hit is reported (see below) — no need to
  disable unless you want zero placement calls.
- `--refresh-cache`: bypass **all** local caches and force fresh lookups; the
  stores are still refreshed for later runs. Three operations are cached on disk
  with their own TTL — placement scores (**24h**, mirrors AWS's free-replay
  window), on-demand prices (**7d**, list prices move rarely), instance specs
  (**30d**, immutable hardware facts). Spot prices and the CloudTrail history are
  never cached (must stay fresh). Use this flag only when you suspect a cached
  value is stale — note it re-spends the placement quota.
- `--clear-cache`: delete all local cache files (placement scores, on-demand
  prices, instance specs) and **exit** without running an analysis.
- `--output` / `-o <file>`: write the **Markdown report to a file** instead of
  stdout (opt-in; the confirmation goes to stderr). The file holds the
  deterministic **detail** (the per-node-group tables). **When the user asked for
  a file, append your synthesis to that same file** (see below) so it is a
  complete standalone report. `--json` emits the full structured plan
  (`current`, `other_available`, `recommended_*`).
- `--os Linux|Windows` (default Linux), `--profile`, `--run-buffer-days`,
  `--max-items`.

Default output is **Markdown** (English), one **table per node group** (env →
cluster → node group): each shown type ranked by **score**, current types marked
`◀ current`, an in-use off-family type tagged `off-fam`; the `Obs.` rate is
**region-wide per type** (same for a type across every node group), while the
group header carries that node group's **own** eviction total; columns Type /
Status / Predicted / Obs. (`rate% (ev/launch)`) / \$/h / µarch / \$/work /
**Score**
(placement is **not** a per-type
column — it is a per-mix line below). The node group name in the header drops its
trailing Terraform id (`dev-bo-20260401…013` → `dev-bo`). Below: the
**recommended mix** (top scores) as a `Factor | Current | Proposed | Δ` table +
absolute period cost + a **placement score** line (current → proposed, the mix
scored as a diversified request) + a **candidate mixes** table (the score- /
reliability- / cost-led candidates with their mean score and placement, ✓ on the
chosen one) + Terraform line. When the recommended mix already equals
the current one, both the proposal line and the recap are tagged
**`✓ already the current mix`**. Just before the recap the report prints an
**`## AWS API calls`** section — one row per AWS operation (`ec2
get-spot-placement-scores`, `ec2 describe-spot-price-history`, `ec2
describe-instance-types`, `pricing get-products`, `eks …`, `cloudtrail
lookup-events`) with **played / OK / failed / quota-blocked / cached / cache
hit / cache TTL** counts and a totals line (`cached`/`cache hit`/`cache TTL`
show `-` for the operations that are not cache-backed). When any request was
quota-blocked it appends a
callout explaining `MaxConfigLimitExceeded` (the daily configuration budget),
that affected types show `n/a` and fall back to observed + predictive, and that
the config replays for free within 24h. The report ends with a **`## Recommended
mixes`** recap — one line per node group with the advised instance list and, for
SPOT groups, the recommended mix's **Spot Placement Score** (`· Plc N/10`, or
`Plc n/a` when unscored/quota-blocked; absent for ON_DEMAND).

## Run (greenfield — design a NEW node group)

When sizing a brand-new node group, skip the existing-groups discovery and pass
the target shape directly:

```bash
pysae-ai-tools aws eks-optimize --cpu 4 --ram 16 --capacity spot      # or on-demand
```

`--cpu`/`--ram` switch to greenfield mode (no EKS discovery, no eviction
history). `--capacity spot|on-demand` (default spot) and `--desired N` (for the
monthly cost figure) tune it. Output shows each candidate mix with its
**absolute** four factors (no current mix to compare against).

## Building blocks (use directly when you only need one signal)

- `pysae-ai-tools aws spot-advisor -r <region> --cpu N --ram G` — predictive
  ranking + savings for a sizing (no AWS history needed).
- `pysae-ai-tools aws spot-evictions -r <region> --period <w>` — observed
  interruptions per environment per day.
- `pysae-ai-tools aws spot-prices -r <region> --types a,b,c` (or `--cpu/--ram`)
  — average spot $/h per type.

## Ask before running (local mode) — MANDATORY

In **local mode**, always ask the user these **three** questions before running
(skip any the user already answered explicitly in their request):

1. **Output** — a Markdown file, or the report directly in chat?
2. **Goal** — score weighting: `balanced` (default), `cost`, or `evictions`?
3. **Window** — history to analyze (CloudTrail, max 90 days): default `2w`,
   e.g. `7d` / `1m`, or an absolute `--from/--to`.

Use a single `AskUserQuestion` with the three. In **CI mode (`$CI` set)** ask
nothing — use the defaults (chat output, `balanced`, `2w`) or any explicit flags.

When generating a file, **build the filename in the shell with `date`** so the
format is guaranteed (current directory, timestamp prefix, `eks-optimize.md`
suffix) — never hand-write the timestamp:

```bash
OUT="$(date +%Y%m%d-%H%M%S)-eks-optimize.md"
pysae-ai-tools aws eks-optimize --region <region> --period <window> --output "$OUT"
echo "$OUT"   # then append your synthesis to this same file
```

In CI mode (`$CI` set), do not ask — emit to stdout (or honor an explicit path).

## Presenting to the user — MANDATORY order

**Always** answer in two parts, in this order:

1. **Full detail first — never collapsed.** Reproduce, node group by node group
   and every environment, the **current mix** (every type with predictive +
   observed + $/work + **score**), the **other available instances**
   table (the realistic CPU field, ranked by score), and the **recommended
   mix** with its 4-factor evolution + period cost + the per-mix **placement
   score** (current → proposed). Do **not** reduce it to a
   one-line-per-node-group "culprit" summary — this raw detail is the bulk of
   the answer.
2. **Then your synthesis** — a concise analysis on top of that detail, **written
   in English** (the document is English-only). **If the run produced a file
   (`--output`), append this synthesis to that same file** (a `## Synthesis`
   section after the `## Recommended mixes` recap) so the `.md` is a complete,
   self-contained report. Trust the **score** and the **observed** rate over
   AWS's predictive rate (proven unreliable: a type predicted `<5%` can be
   observed >40%). Flag any type whose observed rate contradicts a rosy
   predictive bucket, and read the per-mix **placement score** as a pool-level
   go/no-go (a low score on the current mix often means it is under-diversified).
   **When you cite an observed figure, be explicit whether it is a rate or a
   count**: the `Obs.` column gives both (`rate% (evictions/launches)`), so say
   "60 of 142 launches reclaimed (42%)", never a bare "42%" that reads like a
   count. A high rate on a tiny sample (`100% (1/1)`) is weak evidence — weight
   it by the launch denominator:

   1. **Per environment**, where the pain is (most observed evictions) and the cost.
   2. **Per node group**, validate the proposal: does it drop the high-interruption
      pool the group pins (the usual root cause) and widen the pool count?
      Quantify the cost delta.
   3. **Is the whole shape saturated?** If *every* type in the node group's shape
      shows a similar high observed rate (e.g. all `m*.xlarge` at 30-48%), the
      capacity crunch is on the **shape**, not one type — switching types inside
      the same shape will **not** help. Say so, and steer toward a different
      shape (e.g. `2xlarge`), an on-demand base, or `capacity-optimized` instead
      of a same-shape reshuffle that looks like progress but isn't.
   4. **Don't move off a pool that is actually holding.** Compare observed rates
      before recommending a family switch: a burstable pool reclaiming at ~1%
      (`t3.2xlarge` 1/79) is *more* reliable than an `m*` pool at 30%+ — moving
      it to a "proper" family would raise evictions. The default family policy
      hides burstable from the recommended mix; **override it in your synthesis**
      when the data shows burstable is the reliable pool. But weigh the flip side:
      a `t*` **or `*-flex`** trades eviction risk for **CPU-starvation risk** — its
      perf is shown as the sustained baseline, and under sustained load it throttles
      (`%steal`) once `t*` credits run out / a `*-flex` exceeds its 40 % baseline.
      That's a good deal **only for CPU-tolerant pools** (CI runners, batch:
      occasional slowdowns are fine) — never for a latency-sensitive service (API).
      State which side of that line the pool is on.
   5. Warn on **single-pool** SPOT groups, or a critical/prod group with **no
      on-demand safety net**.
   6. Offer the Terraform `instance_types` change. Remind that a managed node
      group must stay a **homogeneous shape** (don't mix `xlarge` + `2xlarge`)
      and should use the `capacity-optimized` allocation strategy.

In CI mode (`$CI` set), emit the full table + JSON and a short factual summary.

## Notes

- Reads CloudTrail at the **account + region** level — the `eks:nodegroup-name`
  / `eks:cluster-name` cross-reference is what separates dev from prod.
- Evictions and launches are attributed on the node group **prefix** (the name
  minus its trailing Terraform numeric id). Changing instance types recreates a
  node group with a new suffix but a stable prefix, so the history of the old
  and new groups is **merged** instead of split.
- A single `BidEvictedEvent` can reclaim several instances, so the observed
  count is per reclaimed instance.
- The observed **rate** is `evictions in [from, to]` / `launches in [from, to]`,
  both over the **same** analysis window. ``RunInstances`` is fetched with an
  extra ``--run-buffer-days`` lookback **only to attribute** evictions of
  instances launched just before the window (resolve their type/cluster) — those
  pre-window launches are **excluded from the denominator**, otherwise a bursty
  pool's rate gets diluted (it once read `m6i.xlarge` at ~8% when the in-window
  truth was ~40%). The rate is therefore **window-sensitive**: a long
  ``--period`` blends a recent eviction spike into many calm-day launches and
  understates current pressure — match the window to the horizon you care about
  (e.g. `--period 3d` for "what's evicting right now").
- The Spot Placement Score **daily configuration budget** (the one that raises
  `MaxConfigLimitExceeded`) is **not readable via any AWS API** — it has no
  Service Quotas entry and no usage metric. Only the *rate* token bucket
  (`L-71F2A72B` / `L-E18E92FB`, non-adjustable) and a CloudWatch
  `AWS/Usage` *call count* are exposed, and neither tells you the remaining
  config budget. The only honest "where are we" signal is the per-run accounting
  in the `## AWS API calls` section (configs played vs quota-blocked). When the
  budget is hit, advise the user to re-run later (the 24h cache makes it free) or
  request a quota increase — there is nothing to query for the remaining count.
