/flaky-test-triageQA-1234

Flaky automated test diagnosis

Diagnoses inconsistent test results by separating product nondeterminism, automation defects, shared data, timing, infrastructure, environment, order, and concurrency effects.

  1. Flake evidence preflight
  2. Failure-signature clustering
  3. Hypothesis design
  4. Controlled reproduction plan
  5. Experiment execution
  6. Causal isolation
  7. Disposition recommendation
  8. Publish
Scope and intent

A controlled workflow specification

This page defines the behaviour, controls, evidence, and outputs required from the skill. It does not claim that a language model can enforce security or correctness by wording alone.

Use when

Appropriate use

Use when a test passes and fails across comparable runs, consumes CI time, or is repeatedly retried or quarantined.

Do not use when

Non-goals

Do not use to mark a test flaky after one failure, quarantine indefinitely, or rerun until green without preserving first-attempt evidence.

8
versioned workflow phases
4
explicit approval gates
6
defined result states
10
versioned output artifacts
Material changes to the source, environment, identity, approved actions, or expected behaviour invalidate dependent approvals and require revalidation.
Workflow

The 8 phases

Each phase has explicit inputs and outputs. Approval gates are hard stops bound to exact versions and hashes.

Phase 1: Flake evidence preflight Gate 1 · approve evidence

Resolve test identity, repository commit, application build, CI job, runner image, browser, retries, historical attempts, quarantine status, and failure signatures.

test ID · commit · build · runner · history · signature

Phase 2: Failure-signature clustering

Group failures by assertion, timeout, selector, network, data, crash, environment, setup, teardown, and product symptom rather than treating every red run as one flake.

signature · frequency · first seen · affected contexts

Phase 3: Hypothesis design

Create testable hypotheses for product race, automation timing, selector instability, data collision, order dependence, concurrency, infrastructure, dependency, and environment drift.

hypothesis · predicted evidence · controlled variable

Phase 4: Controlled reproduction plan Gate 2 · approve experiment

Define bounded repetitions, retry-disabled runs, order randomisation, worker counts, isolated data, network capture, trace, and resource sampling.

run budget · variables · stop conditions · no harmful load

Phase 5: Experiment execution

Run the matrix while preserving every attempt, seed, worker, timing, trace, logs, resource state, and environment health.

attempt ledger · seeds · concurrency · order · evidence

Phase 6: Causal isolation

Evaluate which variable consistently changes the outcome and distinguish correlation from causal evidence.

supports · contradicts · confidence · remaining alternatives

Phase 7: Disposition recommendation Gate 3 · confirm disposition

Recommend product defect, automation fix, data isolation, infrastructure work, environment fix, temporary quarantine with expiry, or more evidence.

owner · fix · validation · quarantine expiry

Phase 8: Publish and verify remediation Gate 4 · publish

Publish the analysis, apply approved changes, and prove improvement with retry-disabled validation across the relevant conditions.

before/after rate · first-attempt pass · residual risk
Approval contract

Every gate approves an exact decision object

An approval cannot be reused after its environment, source, action plan, or approved document changes.

Approval gates and binding requirements
GateDecisionRequired binding
1Gate 1 · approve evidenceApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
2Gate 2 · approve experimentApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
3Gate 3 · confirm dispositionApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
4Gate 4 · publishApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
Result model

Defined states prevent vague conclusions

Attempt outcomes remain separate from these workflow results. Errors, cancellations, and invalid context are never silently converted into a pass.

PRODUCT FLAKE

The product produces inconsistent outcomes under controlled equivalent conditions.

TEST FLAKE

Automation logic, selectors, timing, fixtures, or cleanup cause inconsistent results.

DATA COLLISION

Shared or stale data changes the test outcome.

INFRASTRUCTURE

Runner, browser process, network, service, or resource conditions cause the inconsistency.

ORDER OR CONCURRENCY

Outcome depends on execution order, workers, or simultaneous operations.

UNPROVEN

Available evidence does not establish genuine flakiness or a leading cause.

Evidence contract

Evidence must directly support the claim

Evidence has provenance, capture context, privacy classification, retention, and integrity metadata. A convenient artifact is not automatically sufficient proof.

Required evidence by claim type
ClaimPrimary evidenceRequired validation
HistoryFirst-attempt run recordsRetries and final green results do not hide original failures.
SignatureTrace, error, screenshot, and log clusterDifferent failures are analysed separately.
EnvironmentRunner and application healthResource exhaustion and service outages are visible.
DataResource IDs and fixture stateCollisions and stale state can be reconstructed.
Order/concurrencySeed, order, and worker countThe exact schedule is recorded.
RemediationRetry-disabled before/after runsImprovement is measured on comparable conditions.
Trust and safety

Controls that must be enforced outside the prompt

These controls belong in the runtime, connector permissions, sandbox, renderer, storage layer, and review process.

Runtime control

No one-failure label

A single failure is investigated but not automatically called flaky.

Runtime control

Retries exposed

Configured and manual retries remain visible in evidence and metrics.

Runtime control

Bounded experiments

Repetition has a fixed budget and cannot overload shared systems.

Runtime control

Quarantine expiry

Temporary quarantine requires owner, reason, review date, and release impact.

Runtime control

Product-test separation

A real intermittent product bug is not fixed by weakening the test.

Runtime control

Change validation

Remediation proves first-attempt reliability, not only final pass rate.

Decision policy

Ordered outcomes, not averaged pass counts

The first applicable high-severity outcome takes precedence. A manual override is recorded separately and never rewrites the calculated result.

UNPROVEN

Evidence is insufficient to confirm flakiness or cause.

PRODUCT FLAKE

Route as a product defect and keep coverage visible.

TEST OR DATA FLAKE

Fix automation or isolation and validate retry-disabled.

INFRASTRUCTURE ACTION

Route runner, network, service, or environment work.

RESOLVED

Controlled first-attempt reliability meets the approved threshold.

Known limitations

Uncertainty and coverage limits remain visible

The workflow documents what it cannot prove and what further evidence would change confidence.

  • Limit 1. Rare flakes may need many runs and still remain statistically uncertain.
  • Limit 2. CI and local environments can produce different failure classes.
  • Limit 3. Quarantine can protect signal temporarily but also hides coverage.
  • Limit 4. A test may contain more than one independent flake cause.
  • Limit 5. Infrastructure and product timing can interact.
  • Limit 6. Passing repeated runs reduces but does not eliminate future flake probability.
Outputs

Versioned records for review and continuation

Human summaries are generated from validated structured records. They are not maintained as separate, drifting sources of truth.

flake-context.json

Versioned output produced by the workflow.

history-snapshot.json

Versioned output produced by the workflow.

signature-clusters.yaml

Versioned output produced by the workflow.

hypothesis-plan.md

Versioned output produced by the workflow.

experiment-ledger.jsonl

Versioned output produced by the workflow.

evidence-manifest.json

Versioned output produced by the workflow.

causal-analysis.md

Versioned output produced by the workflow.

disposition.json

Versioned output produced by the workflow.

validation-results.md

Versioned output produced by the workflow.

flaky-triage.html

Versioned output produced by the workflow.

artifacts/qa/flaky-test-triage/<run-id>/
├── flake-context.json
├── history-snapshot.json
├── signature-clusters.yaml
├── hypothesis-plan.md
├── experiment-ledger.jsonl
├── evidence-manifest.json
├── causal-analysis.md
├── disposition.json
├── validation-results.md
└── flaky-triage.html
Example

Example request and controlled output

Request

Input

Investigate why the playlist drag-and-drop Playwright test fails about 8% of CI runs and passes on retry.
Result

Output excerpt

Flake classification: TEST FLAKE

Cause:
The test releases the mouse before the drop target finishes its layout transition. The final assertion checks API state but not the visible order.

Evidence:
- Failure follows reduced CPU allocation.
- Network and API responses are successful.
- Adding a fixed wait changes frequency but does not remove the race.
- Waiting for stable target geometry removes the failure in 100 retry-disabled runs.

Remediation:
Use a stable drag completion signal and assert both UI order and persisted API order.
Production readiness

Required before operational use

The written skill is only one layer. The repository, runtime, connectors, evidence storage, and publication path must implement these requirements.

  • Required. Tool, filesystem, repository, command, secret, and network permissions are enforced outside the model.
  • Required. Every source, environment, build, plan, and approval has a version or content hash.
  • Required. Untrusted values are escaped, length-bounded, path-safe, and separated from executable instructions.
  • Required. Attempt outcomes, scenario verdicts, workflow decisions, and publication status remain separate.
  • Required. Raw evidence, publishable evidence, access control, retention, redaction, and deletion are defined.
  • Required. Concurrent runs use unique IDs, locks, idempotency keys, and conflict-aware updates.
  • Required. Material changes invalidate dependent approvals and outcomes.
  • Required. The repository validator passes before the skill is considered production-ready.
In one line

/flaky-test-triage

Diagnoses inconsistent test results by separating product nondeterminism, automation defects, shared data, timing, infrastructure, environment, order, and concurrency effects.