/api-qaQA-1234

Contract, behaviour, and resilience API QA

Plans and executes API validation across contracts, authentication, authorisation, data integrity, errors, pagination, rate limits, idempotency, concurrency, and compatibility.

  1. API context preflight
  2. Contract
  3. Risk-based case generation
  4. Data
  5. Execution
  6. Resilience
  7. Finding
  8. Publish report
Scope and intent

A controlled workflow specification

This page defines the behaviour, controls, evidence, and outputs required from the skill. It does not claim that a language model can enforce security or correctness by wording alone.

Use when

Appropriate use

Use for new or changed endpoints, service integrations, backend regressions, API releases, or contract verification.

Do not use when

Non-goals

Do not use against production without explicit authorisation, as an unrestricted load test, or with secrets and real customer data in reports.

8
versioned workflow phases
4
explicit approval gates
6
defined result states
10
versioned output artifacts
Material changes to the source, environment, identity, approved actions, or expected behaviour invalidate dependent approvals and require revalidation.
Workflow

The 8 phases

Each phase has explicit inputs and outputs. Approval gates are hard stops bound to exact versions and hashes.

Phase 1: API context preflight Gate 1 · approve target

Resolve service, environment, deployment manifest, OpenAPI or contract revision, authentication method, consumer versions, data stores, and rate-limit policy.

base URL · contract version · source commits · auth · consumers

Phase 2: Contract and expectation analysis

Map methods, paths, schemas, required fields, defaults, validation, status codes, error shapes, headers, pagination, versioning, and deprecation.

request contract · response contract · errors · compatibility

Phase 3: Risk-based case generation

Create positive, negative, boundary, malformed, unauthorised, forbidden, missing, duplicate, idempotent, concurrent, and migration-sensitive cases.

data partitions · state transitions · consumer risk

Phase 4: Data and traffic authorisation Gate 2 · approve execution

Define isolated resources, idempotency keys, concurrency limits, rate-limit budget, external effects, cleanup, and prohibited production-like operations.

resource namespace · request cap · sinks · cleanup

Phase 5: Execution and protocol evidence

Run requests with exact method, URL, headers, body, timing, correlation ID, response, and resulting state recorded. Filter secrets before storage.

attempt ledger · request/response · server correlation · state

Phase 6: Resilience and compatibility checks

Test retries, timeouts, partial failure, pagination continuity, idempotency, concurrent updates, older consumers, schema evolution, and dependency failure where safe.

retry semantics · race handling · backward compatibility

Phase 7: Finding and contract assessment Gate 3 · confirm findings

Classify contract mismatch, behaviour defect, authorisation defect, data-integrity issue, performance concern, environment failure, or documentation gap.

severity · consumer impact · reproducibility · scope

Phase 8: Publish report and test recommendations Gate 4 · publish

Publish redacted findings, contract coverage, generated test proposals, cleanup results, and unresolved compatibility risk.

contract report · defect route · test layer · cleanup
Approval contract

Every gate approves an exact decision object

An approval cannot be reused after its environment, source, action plan, or approved document changes.

Approval gates and binding requirements
GateDecisionRequired binding
1Gate 1 · approve targetApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
2Gate 2 · approve executionApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
3Gate 3 · confirm findingsApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
4Gate 4 · publishApproval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state.
Result model

Defined states prevent vague conclusions

Attempt outcomes remain separate from these workflow results. Errors, cancellations, and invalid context are never silently converted into a pass.

PASS

The endpoint satisfies the approved contract and behaviour for the tested case.

CONTRACT FAIL

Request or response shape, status, header, or documented compatibility is wrong.

BEHAVIOUR FAIL

The protocol is valid but resulting application state or rule is incorrect.

SECURITY FAIL

Authentication, authorisation, or sensitive-data handling is incorrect.

RESILIENCE FAIL

Retry, timeout, idempotency, concurrency, or dependency behaviour is unsafe.

BLOCKED

The target, data, contract, or safe execution path is unavailable.

Evidence contract

Evidence must directly support the claim

Evidence has provenance, capture context, privacy classification, retention, and integrity metadata. A convenient artifact is not automatically sufficient proof.

Required evidence by claim type
ClaimPrimary evidenceRequired validation
RequestRedacted request recordMethod, path, query, safe headers, body hash, and correlation ID are recorded.
ResponseRedacted response recordStatus, safe headers, schema, body hash, and timing are captured.
ContractVersioned schema referenceThe expected rule applies to the tested consumer and version.
StateApproved read-back or event evidenceThe API result matches persisted or emitted behaviour.
ConcurrencyOrdered request and state timelineConflicts, lost updates, and idempotency are visible.
Rate limitBounded request sequenceLimit, reset behaviour, and client guidance are checked without harmful load.
Trust and safety

Controls that must be enforced outside the prompt

These controls belong in the runtime, connector permissions, sandbox, renderer, storage layer, and review process.

Runtime control

Secret filtering before write

Authorization, cookies, tokens, and sensitive payload fields are removed before evidence storage.

Runtime control

Traffic budget

The runtime enforces request rate and concurrency limits.

Runtime control

Production deny by default

Production hosts require explicit, scoped, read-only or otherwise approved access.

Runtime control

Contract provenance

Generated expectations cannot override the approved API contract.

Runtime control

Idempotent cleanup

Created resources are tracked by returned IDs and cleaned safely.

Runtime control

Security routing

Auth, data exposure, and cross-tenant findings use restricted channels.

Decision policy

Ordered outcomes, not averaged pass counts

The first applicable high-severity outcome takes precedence. A manual override is recorded separately and never rewrites the calculated result.

INVALID TARGET

Service identity, contract, or environment cannot be trusted.

REJECT

A release-blocking security, integrity, contract, or behaviour failure exists.

BLOCKED

Required API coverage cannot be completed safely.

PASS WITH GAPS

Required cases pass with accepted optional compatibility or resilience gaps.

PASS

All required API cases pass for the verified contract and deployment.

Known limitations

Uncertainty and coverage limits remain visible

The workflow documents what it cannot prove and what further evidence would change confidence.

  • Limit 1. A contract pass does not prove downstream business behaviour without state verification.
  • Limit 2. Safe concurrency tests may not reproduce production scale.
  • Limit 3. Rate-limit testing must avoid disrupting shared services.
  • Limit 4. Third-party APIs may forbid automated negative testing.
  • Limit 5. Backward compatibility depends on real consumer behaviour as well as schema shape.
  • Limit 6. Observability and correlation IDs may be unavailable in some environments.
Outputs

Versioned records for review and continuation

Human summaries are generated from validated structured records. They are not maintained as separate, drifting sources of truth.

api-context.json

Versioned output produced by the workflow.

contract-map.yaml

Versioned output produced by the workflow.

case-catalog.yaml

Versioned output produced by the workflow.

traffic-plan.md

Versioned output produced by the workflow.

attempt-ledger.jsonl

Versioned output produced by the workflow.

redacted-http/

Versioned output produced by the workflow.

state-evidence/

Versioned output produced by the workflow.

findings/

Versioned output produced by the workflow.

cleanup-report.md

Versioned output produced by the workflow.

api-report.html

Versioned output produced by the workflow.

artifacts/qa/api-qa/<run-id>/
├── api-context.json
├── contract-map.yaml
├── case-catalog.yaml
├── traffic-plan.md
├── attempt-ledger.jsonl
├── redacted-http/
├── state-evidence/
├── findings/
├── cleanup-report.md
└── api-report.html
Example

Example request and controlled output

Request

Input

Test the new playlist API for validation, Member permissions, pagination, idempotent create requests, and two concurrent updates.
Result

Output excerpt

API result: REJECT

Release-blocking finding:
Two concurrent updates with the same version both return 200, and the earlier write silently overwrites the later one.

Other results:
- Validation: PASS
- Member read permission: PASS
- Member update permission: PASS
- Pagination continuity: PASS
- Idempotent create: PASS

Classification:
Data-integrity and concurrency failure.
Production readiness

Required before operational use

The written skill is only one layer. The repository, runtime, connectors, evidence storage, and publication path must implement these requirements.

  • Required. Tool, filesystem, repository, command, secret, and network permissions are enforced outside the model.
  • Required. Every source, environment, build, plan, and approval has a version or content hash.
  • Required. Untrusted values are escaped, length-bounded, path-safe, and separated from executable instructions.
  • Required. Attempt outcomes, scenario verdicts, workflow decisions, and publication status remain separate.
  • Required. Raw evidence, publishable evidence, access control, retention, redaction, and deletion are defined.
  • Required. Concurrent runs use unique IDs, locks, idempotency keys, and conflict-aware updates.
  • Required. Material changes invalidate dependent approvals and outcomes.
  • Required. The repository validator passes before the skill is considered production-ready.
In one line

/api-qa

Plans and executes API validation across contracts, authentication, authorisation, data integrity, errors, pagination, rate limits, idempotency, concurrency, and compatibility.