QUALIFIED

Decision

Status QUALIFIED under pack etalon.demo.ticket-routing@0.1.0, runner 0.0.1.

Redaction was not applied in this section (0 items).

Scope

Intended use

Qualification of a self-hosted model that classifies synthetic support tickets into routing labels, with a person reviewing every escalated ticket.

Out of scope

• free-text generation quality
• multilingual inputs
• latency and throughput
• systems that take automated action without human review

Assumptions and test design are in the methodology section. The endpoint was evaluated as a self-hosted, OpenAI-compatible chat interface. Latency, throughput, and unattended action are outside this pack.

Redaction was not applied in this section (0 items).

System identity

Fields that could not be obtained are recorded with an explicit state. A reason is part of the evidence.

Redaction was not applied in this section (0 items).

Pack identity

Evaluators

• json_schema 0.1.0 (deterministic)
• exact_label 0.1.0 (deterministic)
• behaviour_match 0.1.0 (reference)
• critical_misroute 0.1.0 (reference)

Redaction was not applied in this section (0 items).

Coverage

Realised coverage is the unweighted mean, across declared dimensions, of the fraction of declared values that appear at least once in the qualification corpus. It is not a count of tests.

Realised coverage 1.0 · declared minimum 0.8.

expected_behaviour · fraction 1.0

input_character · fraction 1.0

severity · fraction 1.0

ticket_type · fraction 1.0

Redaction was not applied in this section (0 items).

Results

Qualification corpus. The verdict uses the 95% Wilson interval. Challenge metrics are reported below and do not gate the decision.

Observed critical events are compared with the declared maximum. A maximum of zero fails on a single event. The Wilson interval is reported and is not used to soften that rule.

Definitions

• label_accuracy — Proportion of qualification cases whose predicted label exactly equals the reference label.
• schema_compliance — Proportion of qualification cases whose model output parses as a JSON object containing non-empty string fields label and behaviour.
• abstention_correctness — Among qualification cases whose expected behaviour is abstain or escalate, the proportion whose predicted behaviour matches.
• critical_misroute_rate — Among qualification cases whose reference label is complaint, the proportion assigned a label outside that critical set.

Challenge corpus

Redaction was not applied in this section (0 items).

Failures

No failing applicable results were recorded on the primary pass.

Redaction was not applied in this section (0 items).

Methodology

The pack methodology is included in full so this file can be read without the pack directory. The content hash above identifies that text.

# Synthetic ticket classification methodology

This pack qualifies a self-hosted chat endpoint on a synthetic ticket-routing task.
It produces evidence for a recorded configuration. It does not determine that a
regulation, contract, or internal control has been met.

## Purpose

Measure whether a specific endpoint classifies synthetic tickets into the declared
label set, emits the required JSON object, and follows abstain or escalate when
the reference says so.

## Scope

In scope: label match, JSON shape, abstain and escalate behaviour, and complaint
labels kept inside the complaint set.

Out of scope: free-text quality, languages other than the fixture English, latency,
throughput, and any system that acts without a person reviewing escalations.

## Assumptions

The endpoint speaks the OpenAI chat-completions schema. The operator supplies the
system prompt and the serving description. Weight hashes are recorded only from a
model index the operator supplies. The public demo endpoint is a local fixture,
not a GPU serving stack. A fixture run is evidence about that fixture.

## Test design

The qualification file is the scored holdout. The calibration file is never sent
and never scored. The challenge file is scored and reported separately; it does
not gate the decision. Case identifiers do not overlap. Provenance on every case
in this pack is `synthetic`.

## Coverage model {#coverage-model}

Declared dimensions are ticket type, severity, input character, and expected
behaviour. Realised coverage is the unweighted mean of the fraction of declared
values that appear at least once. It is not a test count. Coverage below 0.80 is
INDETERMINATE.

## Scoring

Each metric is a proportion with a 95% Wilson interval. Mandatory requirements
use the interval bound, not the point estimate alone. When the interval crosses
a threshold, the status is INDETERMINATE. Repeat passes over a seeded subset
estimate variance. Variance above the declared maximum is INDETERMINATE.

Critical misroutes are counted as events. A maximum of zero fails on one event.
The Wilson interval is reported for that rate and is not used to soften the rule.

## Evaluator methodology

`json_schema` and `exact_label` are deterministic and make no model call.
`behaviour_match` and `critical_misroute` are reference comparisons with
`model_use: none`. No judge evaluator is mapped. A critical dimension scored
only by a judge is rejected by the pack check.

## Label accuracy {#label-accuracy}

Label accuracy is the proportion of qualification cases whose predicted label
equals the reference label. The demonstration threshold is 0.90 because a perfect
score on 72 cases has a Wilson lower bound above 0.90, while the scripted
quantized profile falls below it with the upper bound still under 0.90.

## Schema compliance {#schema-compliance}

Schema compliance is the proportion of qualification outputs that parse as a JSON
object with non-empty `label` and `behaviour` strings. The threshold is 0.90
rather than 1.00 because a 95% Wilson interval on a finite sample never reaches
1. A perfect parse rate clears 0.90. Non-JSON output does not.

## Abstention {#abstention}

Abstention correctness covers cases whose reference behaviour is abstain or
escalate. Classify-only cases are outside the population. The threshold is 0.80,
which a perfect applicable subset clears and a profile that always returns
`classify` misses.

## Threshold rationale

Every numeric bound above has a written rationale in `thresholds.yaml` and a
heading in this file. Changing a threshold or its rationale is a new pack version
and an always-requalify trigger.

## Calibration

This version does not execute a judge and does not score the calibration file.
Human review of a judge, when a later pack needs one, is a separate procedure.
The calibration split is still enforced so a scored calibration case cannot leak
into the evidence.

## Limitations

Bit-identical reproducibility is not assumed. Continuous batching can change
reduction order and perturb logits enough to flip a token near a tie. A fixed
seed and temperature 0 do not remove that effect. The runner records concurrency
and serving settings, repeats a subset, and reports variance and a 95% interval.

The demo fixture returns scripted labels. It can show that the pack separates a
reference configuration from scripted degradations. It cannot show how a 70B
model behaves under int4 quantization. Weight hashes are unavailable unless the
operator supplies an index, and a check cannot rule out a weight swap while that
field is unavailable.

## Change management

A change to the corpus, rubrics, thresholds or their rationale, the qualification
policy, evaluator configuration, methodology, mappings, or triggers is a new pack
version. The run records the pack version and the content hash. Pack identity
changes are an always-requalify trigger. Qualification stays current until a
declared trigger changes. There is no calendar expiry.

Redaction was not applied in this section (0 items).

Calibration

No judge evaluator was executed. Human calibration is not part of this run. Calibration cases shipped with the pack were held out and were not sent to the endpoint.

Redaction was not applied in this section (0 items).

Limitations

Bit-identical reproducibility is not assumed. Continuous batching can change reduction order and perturb logits enough to flip a token near a tie. A fixed seed and a temperature of zero do not remove that effect. This run records serving and concurrency settings, repeats a declared subset of the qualification corpus, and reports per-metric variance together with a 95% Wilson confidence interval. Mandatory proportion requirements are applied to that interval, not to the point estimate alone. When the interval crosses a threshold, the result is INDETERMINATE. Critical-failure maxima are applied to observed events: a maximum of zero fails on a single event, and the interval is not used to soften that rule.

Weight file hashes are shown only when an index was supplied. An unavailable weight hash means a later reader cannot tie this endpoint to a specific file. See the system identity section for every other gap.

Redaction was not applied in this section (0 items).

Requalification triggers

This result becomes stale when a declared trigger changes. It does not expire on a calendar date.

Always requalify

• model_weights
• model_revision
• quantization
• system_prompt
• retrieval_corpus
• pack_version
• threshold_change
• evaluator_version

Review required

• inference_engine_version
• serving_config
• sampling_parameters
• chat_template
• runner_version

No material impact

• hardware_topology
• driver_version
• operator

Redaction was not applied in this section (0 items).