Appropriate use
Use for Playwright, Cypress, API, unit, integration, or mixed automation suites before expansion, migration, or reliability work.
Scope and intent
A controlled workflow specification
This page defines the behaviour, controls, evidence, and outputs required from the skill. It does not claim that a language model can enforce security or correctness by wording alone.
Non-goals
Do not use to judge tests only by pass rate, delete tests automatically, weaken assertions, or modify the repository without explicit approval.
- 8
- versioned workflow phases
- 4
- explicit approval gates
- 6
- defined result states
- 9
- versioned output artifacts
Material changes to the source, environment, identity, approved actions, or expected behaviour invalidate dependent approvals and require revalidation.
Workflow
The 8 phases
Each phase has explicit inputs and outputs. Approval gates are hard stops bound to exact versions and hashes.
Phase 1: Repository and scope preflight Gate 1 · approve audit scope
Resolve repository, branch, commit, test framework, configuration, CI workflow, target environments, ownership, and review scope. Ensure the working tree and write permissions are controlled.
repo · commit · framework · suites · CI · owner
Phase 2: Static test inventory
Map tests, tags, fixtures, helpers, selectors, assertions, retries, timeouts, mocks, data sources, cleanup, and suite dependencies.
test IDs · files · fixtures · shared state · runtime config
Phase 3: Quality-rule analysis
Detect weak or missing assertions, arbitrary waits, retry masking, brittle selectors, over-mocking, hidden dependencies, broad catches, ignored errors, and tests that cannot fail meaningfully.
correctness · determinism · observability · maintainability
Phase 4: Coverage-value analysis
Map tests to risks and product behaviour, identify duplicates, gaps, inappropriate test layers, dead tests, and expensive low-value paths.
risk trace · overlap · missing behaviour · layer fit
Phase 5: Controlled execution sampling Gate 2 · approve execution
Run approved suites or samples with retries exposed, order randomisation, concurrency variation, trace capture, and resource monitoring where safe.
attempts · duration · variance · order dependence · resource use
Phase 6: Finding validation
Reproduce suspected test flaws, distinguish product failures from automation failures, and avoid labelling a test flaky from one failed run.
reproduction · evidence · confidence · affected tests
Phase 7: Remediation plan Gate 3 · approve recommendations
Prioritise fixes, merges, deletions, layer moves, data isolation, selector changes, fixture redesign, and CI improvements. Show expected benefit and migration risk.
keep · fix · merge · move · remove · add
Phase 8: Publish audit and optional patches Gate 4 · publish
Publish the audit, machine-readable findings, and approved patch set. Any code change remains reviewable and is validated against the original failure mode.
audit report · patch diff · validation · ownership
Approval contract
Every gate approves an exact decision object
An approval cannot be reused after its environment, source, action plan, or approved document changes.
| Gate | Decision | Required binding |
|---|---|---|
| 1 | Gate 1 · approve audit scope | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
| 2 | Gate 2 · approve execution | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
| 3 | Gate 3 · approve recommendations | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
| 4 | Gate 4 · publish | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
Result model
Defined states prevent vague conclusions
Attempt outcomes remain separate from these workflow results. Errors, cancellations, and invalid context are never silently converted into a pass.
HEALTHY
The test provides reliable, meaningful coverage with appropriate cost and maintainability.
NEEDS IMPROVEMENT
The test has material quality debt but still provides useful signal.
FLAKY
Controlled evidence demonstrates inconsistent automation behaviour independent of product variation.
FALSE CONFIDENCE
The test can pass without proving the intended product behaviour.
DUPLICATE OR LOW VALUE
Coverage is redundant or its cost materially exceeds its distinct risk value.
BLOCKED
The test cannot be evaluated because required environment, ownership, or execution evidence is unavailable.
Evidence contract
Evidence must directly support the claim
Evidence has provenance, capture context, privacy classification, retention, and integrity metadata. A convenient artifact is not automatically sufficient proof.
| Claim | Primary evidence | Required validation |
|---|---|---|
| Test identity | Repository commit and test ID | The reviewed source is immutable for the audit. |
| Assertion quality | Code excerpt and behaviour trace | The assertion directly proves the intended result. |
| Flakiness | Multi-attempt controlled runs | Product, environment, order, and runner variation are separated. |
| Isolation | Resource and fixture map | Tests do not depend on shared mutable state without control. |
| Coverage value | Risk traceability | The test protects a distinct behaviour or defect class. |
| Runtime cost | CI timing and resource samples | Cost is measured across representative runs. |
Trust and safety
Controls that must be enforced outside the prompt
These controls belong in the runtime, connector permissions, sandbox, renderer, storage layer, and review process.
Runtime control
Read-only by default
Audit cannot modify tests, config, snapshots, or dependencies without approval.
Runtime control
No pass-rate bias
Frequently passing tests can still provide false confidence.
Runtime control
No retry hiding
Configured retries are reported and first-attempt outcomes remain visible.
Runtime control
No assertion weakening
Recommended fixes preserve or improve behavioural proof.
Runtime control
Repository safety
Commands, package installs, and scripts are allowlisted and sandboxed.
Runtime control
Patch review
Generated changes require code review and validation before merge.
Decision policy
Ordered outcomes, not averaged pass counts
The first applicable high-severity outcome takes precedence. A manual override is recorded separately and never rewrites the calculated result.
AUDIT INVALID
Repository identity or execution evidence is unreliable.
CRITICAL REMEDIATION
False-confidence or security-sensitive tests undermine release decisions.
REMEDIATION REQUIRED
Material reliability or isolation issues need planned work.
ACCEPTABLE WITH DEBT
Useful coverage remains with owned improvement work.
HEALTHY
The reviewed scope provides reliable and maintainable signal.
Known limitations
Uncertainty and coverage limits remain visible
The workflow documents what it cannot prove and what further evidence would change confidence.
- Limit 1. Static review cannot prove runtime reliability by itself.
- Limit 2. A flaky product can make a deterministic test appear flaky.
- Limit 3. Suite runtime varies with CI capacity and environment health.
- Limit 4. Coverage mapping depends on current requirements and architecture.
- Limit 5. Removing duplicates can reduce diagnostic value if distinctions are not understood.
- Limit 6. Generated patches may not match team conventions without human review.
Outputs
Versioned records for review and continuation
Human summaries are generated from validated structured records. They are not maintained as separate, drifting sources of truth.
audit-context.jsonVersioned output produced by the workflow.
test-inventory.jsonVersioned output produced by the workflow.
rule-findings.jsonlVersioned output produced by the workflow.
coverage-map.yamlVersioned output produced by the workflow.
execution-samples/Versioned output produced by the workflow.
flaky-analysis.mdVersioned output produced by the workflow.
remediation-plan.mdVersioned output produced by the workflow.
patches/Versioned output produced by the workflow.
automation-audit.htmlVersioned output produced by the workflow.
artifacts/qa/automation-review/<run-id>/ ├── audit-context.json ├── test-inventory.json ├── rule-findings.jsonl ├── coverage-map.yaml ├── execution-samples/ ├── flaky-analysis.md ├── remediation-plan.md ├── patches/ └── automation-audit.html
Example
Example request and controlled output
Input
Audit our Playwright regression suite for arbitrary waits, weak assertions, shared data, retries, duplicate coverage, and tests that pass without checking the UI result.
Output excerpt
Automation audit: CRITICAL REMEDIATION Critical finding: 14 tests call the API, wait for a fixed timeout, and pass without asserting that the expected UI state appears. Other findings: - 23 arbitrary waits - 8 tests share one organisation - 6 selectors depend on generated CSS classes - CI retries hide first-attempt failures in 11 tests Recommendation: Fix false-confidence tests first, then isolate data and remove fixed waits.
Production readiness
Required before operational use
The written skill is only one layer. The repository, runtime, connectors, evidence storage, and publication path must implement these requirements.
- Required. Tool, filesystem, repository, command, secret, and network permissions are enforced outside the model.
- Required. Every source, environment, build, plan, and approval has a version or content hash.
- Required. Untrusted values are escaped, length-bounded, path-safe, and separated from executable instructions.
- Required. Attempt outcomes, scenario verdicts, workflow decisions, and publication status remain separate.
- Required. Raw evidence, publishable evidence, access control, retention, redaction, and deletion are defined.
- Required. Concurrent runs use unique IDs, locks, idempotency keys, and conflict-aware updates.
- Required. Material changes invalidate dependent approvals and outcomes.
- Required. The repository validator passes before the skill is considered production-ready.
In one line
/automation-review
Audits an automated test suite for correctness, reliability, isolation, maintainability, coverage value, security, runtime cost, and false-confidence patterns.