Appropriate use
Use after deployment, before a release stage, after environment recovery, or as a rapid first check of a candidate build.
Scope and intent
A controlled workflow specification
This page defines the behaviour, controls, evidence, and outputs required from the skill. It does not claim that a language model can enforce security or correctness by wording alone.
Non-goals
Do not use as a substitute for full regression, long-duration stability, compatibility coverage, or security validation.
- 8
- versioned workflow phases
- 4
- explicit approval gates
- 6
- defined result states
- 8
- versioned output artifacts
Material changes to the source, environment, identity, approved actions, or expected behaviour invalidate dependent approvals and require revalidation.
Workflow
The 8 phases
Each phase has explicit inputs and outputs. Approval gates are hard stops bound to exact versions and hashes.
Phase 1: Candidate preflight Gate 1 · approve candidate
Verify environment identity, deployment manifest, service health, test role, tenant, critical integrations, and known incidents.
candidate build · environment · dependency status · role
Phase 2: Critical-path selection
Load the versioned smoke manifest and confirm which paths are required for this product, platform, and release type.
startup · authentication · navigation · core action · persistence · playback
Phase 3: Data and side-effect approval Gate 2 · approve actions
Confirm isolated test data, external notification sinks, cleanup, and any non-reversible action before execution.
resource IDs · test inbox · webhook sink · cleanup
Phase 4: Startup and reachability
Check application launch, service reachability, configuration loading, health indicators, and absence of release-wide errors.
startup time · health endpoints · console · crash signals
Phase 5: Core user journeys
Run the smallest required end-to-end paths with assertion-specific evidence and no retry-until-green.
login · primary navigation · create/read/update · publish/play
Phase 6: Critical negative and permission checks
Confirm at least one high-risk denial, validation, recovery, or error path required by the smoke manifest.
forbidden action · invalid input · dependency failure · recovery
Phase 7: Outcome calculation Gate 3 · confirm outcome
Apply fixed required-path rules and separate product failures from environment or automation errors.
required pass set · blocked paths · flaky paths · candidate validity
Phase 8: Publish and trigger next workflow Gate 4 · publish
Publish one concise result and link the next action: proceed to regression, hold, redeploy, incident investigation, or environment recovery.
summary · evidence bundle · next workflow
Approval contract
Every gate approves an exact decision object
An approval cannot be reused after its environment, source, action plan, or approved document changes.
| Gate | Decision | Required binding |
|---|---|---|
| 1 | Gate 1 · approve candidate | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
| 2 | Gate 2 · approve actions | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
| 3 | Gate 3 · confirm outcome | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
| 4 | Gate 4 · publish | Approval stores approver, role, timestamp, context hash, approved document hash, expiry, and invalidation state. |
Result model
Defined states prevent vague conclusions
Attempt outcomes remain separate from these workflow results. Errors, cancellations, and invalid context are never silently converted into a pass.
PASS
Every required smoke path passed in the verified candidate.
FAIL
A required core path produced a confirmed product failure.
BLOCKED
A required path could not run because of environment or dependency conditions.
FLAKY
A required path produced inconsistent product outcomes.
INVALID CANDIDATE
The tested deployment identity or configuration changed or could not be verified.
ABORTED
Execution ended early and completed coverage is recorded.
Evidence contract
Evidence must directly support the claim
Evidence has provenance, capture context, privacy classification, retention, and integrity metadata. A convenient artifact is not automatically sufficient proof.
| Claim | Primary evidence | Required validation |
|---|---|---|
| Candidate identity | Deployment attestation | The exact tested artifact and configuration are recorded. |
| Startup | Timing and runtime logs | Launch completes within the defined threshold without new fatal errors. |
| Core journey | Trace, screenshot, and API evidence | The user-visible outcome and backend result agree. |
| Permission denial | Role and API evidence | The forbidden action is denied for the verified actor. |
| Persistence | Reload or new-session check | Critical created state remains valid. |
| Playback or processing | Observation trace and runtime logs | The primary output begins and remains healthy for the defined smoke duration. |
Trust and safety
Controls that must be enforced outside the prompt
These controls belong in the runtime, connector permissions, sandbox, renderer, storage layer, and review process.
Runtime control
Versioned manifest
The smoke set cannot change silently between release stages.
Runtime control
Required-path policy
A required failure cannot be averaged out by optional passes.
Runtime control
Fast does not mean vague
Every path still has a direct expected result and evidence.
Runtime control
Environment classification
Shared outages do not become product defects, but they still block release confidence.
Runtime control
No hidden retries
Any attempt after an assertion failure keeps the scenario flaky or failed.
Runtime control
Stage binding
The result applies only to the verified release candidate and environment.
Decision policy
Ordered outcomes, not averaged pass counts
The first applicable high-severity outcome takes precedence. A manual override is recorded separately and never rewrites the calculated result.
INVALID CANDIDATE
Candidate identity or configuration cannot be trusted.
FAIL
A required core product path fails.
BLOCKED
A required path cannot be assessed.
PASS WITH NOTE
All required paths pass, but optional warnings or accepted limitations remain.
PASS
All required paths pass with valid evidence.
Known limitations
Uncertainty and coverage limits remain visible
The workflow documents what it cannot prove and what further evidence would change confidence.
- Limit 1. Smoke testing gives breadth across critical paths but little depth.
- Limit 2. A smoke pass cannot establish long-term stability.
- Limit 3. Optional integrations may be excluded only when the release policy permits it.
- Limit 4. Fast shared-environment tests are sensitive to unrelated outages.
- Limit 5. Hardware platforms may require separate smoke manifests.
- Limit 6. A passed smoke run does not override unresolved known release blockers.
Outputs
Versioned records for review and continuation
Human summaries are generated from validated structured records. They are not maintained as separate, drifting sources of truth.
smoke-context.jsonVersioned output produced by the workflow.
smoke-manifest.yamlVersioned output produced by the workflow.
data-plan.mdVersioned output produced by the workflow.
attempt-ledger.jsonlVersioned output produced by the workflow.
evidence-manifest.jsonVersioned output produced by the workflow.
smoke-results.yamlVersioned output produced by the workflow.
release-note.mdVersioned output produced by the workflow.
smoke-report.htmlVersioned output produced by the workflow.
artifacts/qa/smoke-test/<run-id>/ ├── smoke-context.json ├── smoke-manifest.yaml ├── data-plan.md ├── attempt-ledger.jsonl ├── evidence-manifest.json ├── smoke-results.yaml ├── release-note.md └── smoke-report.html
Example
Example request and controlled output
Input
Run the production-staging smoke test for Android Player 4.11.5. Include pairing, content download, playback, screenshot, restart, and Member permission denial.
Output excerpt
Smoke result: PASS WITH NOTE Required paths: 6 of 6 passed. Note: First screenshot command took 42 seconds, above the usual 15-second baseline but below the 60-second smoke threshold. Next: Proceed to targeted regression and monitor screenshot latency during staged rollout.
Production readiness
Required before operational use
The written skill is only one layer. The repository, runtime, connectors, evidence storage, and publication path must implement these requirements.
- Required. Tool, filesystem, repository, command, secret, and network permissions are enforced outside the model.
- Required. Every source, environment, build, plan, and approval has a version or content hash.
- Required. Untrusted values are escaped, length-bounded, path-safe, and separated from executable instructions.
- Required. Attempt outcomes, scenario verdicts, workflow decisions, and publication status remain separate.
- Required. Raw evidence, publishable evidence, access control, retention, redaction, and deletion are defined.
- Required. Concurrent runs use unique IDs, locks, idempotency keys, and conflict-aware updates.
- Required. Material changes invalidate dependent approvals and outcomes.
- Required. The repository validator passes before the skill is considered production-ready.
In one line
/smoke-test
Runs a short, version-bound health check across the most critical user and system paths to decide whether deeper testing or rollout should continue.