---
name: atk-find-failures-by-rule
description: Execute the reusable failure_rule.py script to find failing Agent tuning cases in the current results version.
---

# Agent Tuning — Find Failures by Rule

## Purpose

Execute `.atk/runner/failure_rule.py` to find failing rows in the current version's `eval_results.csv` and write them to `failure_cases.csv`. This Skill maps to `docs/codex_agent_tuning_prd.md` sections 2.4, 4, 5, and 7.

This Skill only runs an existing rule script. It does not create or update `.atk/runner/failure_rule.py`; if the script is missing, tell the user to run `atk-init-failure-rule` first.

Traceability note: section 2.4 defines rule-based failure-case discovery, section 4 defines current-version behavior, and section 7 defines delivery requirements.

## Inputs

- Current version directory resolved from `.atk/results/vN`.
- Required current file: `eval_results.csv`.
- Required shared rule script: `.atk/runner/failure_rule.py`.
- Shared rules in `docs/shared-versioning-and-confirmation.md`.

## Outputs

- Current `.atk/results/vN/failure_cases.csv` generated by `.atk/runner/failure_rule.py`.
- The output file name is always `failure_cases.csv` and it overwrites the current version's existing file after any required confirmation.

## Workflow

1. Resolve current version with `resolve_current_version()` using `RESULTS_DIR = Path(".atk/results")`.
2. Require `eval_results.csv` with `require_current_file(current_dir, "eval_results.csv")`.
3. Require `.atk/runner/failure_rule.py`.
4. If the script is missing, stop and tell the user to run `atk-init-failure-rule` to create `.atk/runner/failure_rule.py`.
5. If the script still contains `TODO_AGENT_TUNING`, stop and tell the user to finish or rerun `atk-init-failure-rule` with a clear rule.
6. If current `failure_cases.csv` already exists and may be overwritten, ask for confirmation before executing the script.
7. Execute the script with the target repository's Python runtime when available (`uv run python`, `.venv/bin/python`, Poetry, then `python3`): `<python-runtime> .atk/runner/failure_rule.py`.
8. Confirm that current `.atk/results/vN/failure_cases.csv` exists after execution and summarize the result. If the script exits nonzero, report stderr/stdout and do not invent results.

## Required rule behavior

- Use the current version directory, not a user-supplied version argument.
- Read current `eval_results.csv` and write current `failure_cases.csv`.
- Preserve original result columns in `failure_cases.csv`.
- Default to overwriting `failure_cases.csv`; do not backup or merge.
- Keep `failure_rule.py` under `.atk/runner/` and reuse it across versions.
- Keep dependencies stdlib-first unless the target project already requires more.

## Shared version rules

Use the canonical helper names and semantics from `docs/shared-versioning-and-confirmation.md`:

- `RESULTS_DIR = Path(".atk/results")`
- `resolve_current_version(results_dir=RESULTS_DIR)`
- `require_current_file(current_dir, filename)`

Do not fall back to an older version if current `eval_results.csv` is missing.

## Confirmation triggers

Ask before executing the script when:

- current `failure_cases.csv` already exists and execution would overwrite it;
- the Python runtime choice is ambiguous enough to change imports or behavior;
- the script appears hand-written and its side effects are unclear;
- running the script would do more than read current `eval_results.csv` and write current `failure_cases.csv`.

## Failure behavior

- Require current `vN/eval_results.csv`; if no current version or missing `eval_results.csv`, stop with repair/rerun guidance.
- If `.atk/runner/failure_rule.py` is missing, stop with: run `atk-init-failure-rule` first to create `.atk/runner/failure_rule.py`.
- If `.atk/runner/failure_rule.py` contains `TODO_AGENT_TUNING`, stop and ask the user to finish the rule via `atk-init-failure-rule` or manual editing.
- If script execution fails, report the command, exit status, stdout/stderr, and whether `failure_cases.csv` was produced or left unchanged.
- If no current `failure_cases.csv` exists after a successful script exit, report that as a script contract failure.

## Handoff message

After executing the script, summarize:

- current version read path;
- script path `.atk/runner/failure_rule.py`;
- output path `.atk/results/vN/failure_cases.csv`;
- overwrite behavior;
- failure row count if it can be read safely;
- next command: `atk-report` to generate `.atk/results/vN/report.md`.
