---
name: clawd-route-eval
description: >
  Measure how reliably the plugin's skills trigger from natural language: route a versioned
  corpus of phrasings through sub-agents of THIS session (no API key — uses the active OAuth
  session), then score recall@1 and the collisions (which wrong skill stole each phrasing) so
  descriptions can be disambiguated where routing actually fails. Use when the user says
  /clawd-route-eval, /route-eval, 'évalue le routing des skills', 'mesure le déclenchement',
  'est-ce que les skills se déclenchent bien', 'baseline du routing', 'quels skills se marchent
  dessus', 'teste le routing en langage naturel', 'evaluate skill routing', 'measure trigger
  recall', 'which skills collide', 'run the routing eval', 'baseline the skill routing', or wants
  to check natural-language triggering. Distinct from /clawd-skill-review, which audits one
  skill's structure and description form — this routes the whole corpus and measures behaviour.
argument-hint: "[--corpus <path> to use a custom corpus] [--batch <N> utterances per sub-agent (default 12)] [--min-recall <0..1> to fail below a threshold]"
allowed-tools:
  - Read
  - Write
  - Bash
  - Agent
---

Measure the plugin's **natural-language triggering**: every skill is routed only by its
`description`, so this harness checks — behaviourally — whether a real user phrasing reaches the
right skill. It routes a versioned corpus through **sub-agents of the active session** (so it
needs **no API key**), then scores recall@1 and surfaces the **collisions** (which wrong skill
stole each phrasing) — the actionable list of description pairs to disambiguate.

This is the *behavioural* oracle; the deterministic `scan-skill --check` lint is the *form*
oracle. Use them together: the lint blocks thin descriptions, this finds the ones that route wrong.

## Step 1 — Emit the routing tasks

```bash
SCRATCH="${TMPDIR:-/tmp}/route-eval"
mkdir -p "$SCRATCH"
pysae-ai-tools skills route-eval prompts ${ARG_CORPUS:+--corpus "$ARG_CORPUS"} --out "$SCRATCH/prompts.json"
```

`prompts.json` carries `{ "system": "<the skill catalogue prompt>", "utterances": [ {"id", "text"} … ] }`.
It deliberately carries **no expected answer** — the routing agents must stay blind. Read it.

## Step 2 — Route each utterance via blind sub-agents of this session

Fan out `Agent` calls (general-purpose) to route the utterances. **Batch** them — one sub-agent
routes ~`${BATCH:-12}` utterances — to keep the number of agents reasonable; send the batches in
parallel (multiple `Agent` calls in one turn).

Each sub-agent gets **only** the `system` prompt (the catalogue) + its batch of utterances, and is
told to decide **each utterance independently** and return strict JSON. Never give a sub-agent the
expected skill (you don't have it anyway) and never let it see the other agents' answers — that is
what makes the measurement honest.

Sub-agent prompt template (fill `<SYSTEM>` and the batch):

```
<SYSTEM>

Route each message below to exactly one skill name from the catalogue above. Decide each
independently. Return ONLY a JSON array, one object per message, no prose:
[{"id": <id>, "predicted": "<skill-name>"}, …]

Messages:
- id <id>: <text>
- id <id>: <text>
…
```

Collect every batch's array into one combined list covering all ids, and write it:

```bash
# PREDICTIONS_JSON = the merged [{"id":…, "predicted":…}, …] from all sub-agents
printf '%s' "$PREDICTIONS_JSON" > "$SCRATCH/predictions.json"
```

## Step 3 — Score

```bash
pysae-ai-tools skills route-eval score --predictions "$SCRATCH/predictions.json" \
  ${ARG_CORPUS:+--corpus "$ARG_CORPUS"} ${ARG_MIN_RECALL:+--min-recall "$ARG_MIN_RECALL"}
```

This prints **recall@1** and the **collisions** (`expected → predicted ×N`) plus each misrouted
utterance. `score` holds the expected answers and compares; the routing agents never saw them.

## Step 4 — Report and propose fixes

Present recall@1, then for **each collision** name the two competing skills and propose the
concrete disambiguation: a one-line `Distinct from /<other>, which …` to add to the losing skill's
`description` (and often the reciprocal line on the other), following
[../references/skill-description.md](../references/skill-description.md). A clean run (recall 100%,
no collisions) is the expected good outcome on a healthy corpus — say so plainly.

Note the self-eval caveat: a stronger orchestrating model yields a near-ceiling recall, so treat a
high score as an upper bound; the **collisions** are the durable signal regardless of model.

## Rules

- **No API key.** Routing runs on the active session's sub-agents, never a direct API call. (The
  `route-eval api` subcommand exists only for an automated CI run with a real key — not this skill.)
- **Blind routing.** Sub-agents see only the catalogue + utterances, never the expected skill nor
  each other's answers. The `prompts` output enforces this by omitting expectations.
- **French** for communication, English for commands. Apply [../references/french-writing-rules.md](../references/french-writing-rules.md).
- **Grow the corpus** when a real miss is found: add the phrasing to `route_corpus.yaml` so the
  regression is captured, then re-run.
