# Quality tiers

`orthography2ipa.types.QualityTier` classifies the data maturity of a
language spec. Every spec JSON declares its tier explicitly (guarded by
[`tests/test_data_quality.py`](../tests/test_data_quality.py)). This
page defines the measurable criteria a spec must meet for each tier, so
that a tier claim is verifiable against the repository rather than a
subjective label.

## `stub`

- Placeholder entry: `code`, `name`, `script` and a `parent` (so the clade
  chain, and with it `family`, resolves) are populated.
- `graphemes` and `allophones` may be empty.
- No gold benchmark required.
- Reserved for languages the library has not yet started encoding, or
  extinct languages with metadata-only entries (see
  `test_non_stub_resolves_to_content`).

## `skeleton`

- Non-empty `graphemes` inventory (and therefore a non-empty derived or
  explicit `allophones` map: see `LanguageSpec.__post_init__`).
- Passes every guard in `tests/test_data_quality.py`, in particular
  `test_non_stub_resolves_to_content` (non-extinct languages above stub
  must resolve to usable grapheme and allophone data).
- No positional grapheme rules or stress rules required.
- No gold benchmark required.

## `research`

All `skeleton` criteria, plus:

- At least one entry in `sources` (cited phonological reference).
- A `stress` block when the language has lexical/predictable word
  stress (languages without lexical stress, or whose stress is
  unencoded because it is not orthography-predictable, are exempt: document the exemption in the spec's `notes`).
- A gold benchmark registered for at least one of the spec's language
  tags in `scripts/benchmark.py`'s `DATASETS` registry, with a PER
  recorded for it in [`benchmarks/results.json`](../benchmarks/results.json).

## `production`

All `research` criteria, plus:

- The registered gold has **at least 500 evaluated entries** (`n` in
  `benchmarks/results.json`) for at least one dataset/language row.
- That row's provenance tier **can gate a promotion**: i.e. it is not
  `espeak-derived`, `epitran-derived` or `llm-generated`
  (`can_gate_promotion()` in `scripts/benchmark.py` is the single source of
  truth, and a test enforces this rule):
  - A gold generated by a **competing G2P system** (`espeak-derived`,
    `epitran-derived`) measures *agreement with that competitor*, not
    correctness. Both espeak and epitran are systems this project benchmarks
    itself against ([comparison](comparison.md)), so their output cannot also
    serve as our truth.
  - A gold generated by an **LLM** (`llm-generated`) has no lexicon, no rules
    and therefore **no error model**: a disagreement with it is not even
    diagnostic, so it can certify nothing.
  A language whose only ≥500-entry gold sits in one of these tiers has no
  usable gold and stays at `research`; a language that clears the bar on a
  trustworthy gold is **not** disqualified by a poor score on one of them: diverging from espeak (or epitran, or an LLM) may be exactly what a cited
  source demands. See [benchmarks: provenance](benchmarks.md).
- PER at or below a documented per-language threshold:
  - **≤ 0.15** for shallow/phonemic orthographies: orthographies where
    grapheme-to-phoneme correspondence is largely regular (e.g.
    Spanish, Finnish, Esperanto). A rule-driven engine should approach
    near-perfect accuracy on these; a higher PER signals a spec gap,
    not an inherent orthographic limit.
  - **≤ 0.25** for deep orthographies: orthographies with significant
    historical, morphophonemic, or dialectal irregularity (e.g.
    English, French, Portuguese, Danish). The threshold is looser
    because even human-curated gold sets disagree on regional/optional
    variants at this rate; PER above it means the engine is missing
    encodable rules, not just irregularity noise.
  - The applicable threshold and the shallow/deep classification for
    the language must be stated in its `docs/languages/` page (see
    below); silence defaults to the deep-orthography threshold.
  - The `≤ 0.15` shallow-orthography target is a reviewer-enforced
    convention, not a machine-checked one: `LanguageSpec` carries no
    orthographic-depth field, so `test_production_tier_has_qualifying_benchmark`
    cannot look up a per-language threshold and applies the `≤ 0.25`
    deep-orthography ceiling uniformly. A promotion PR for a
    shallow-orthography language must be reviewed against the `0.15`
    target by hand before merge: the guard test alone is not
    sufficient proof for those languages.
- A per-language documentation page exists under
  [`docs/languages/`](languages/) covering the spec's phonology,
  sources, and known limitations.
- Any known engine-limit exceptions (documented cases the current
  transcription engine cannot resolve, e.g. the positional-trie or
  hyphen-boundary-stress gaps noted in `AGENTS.md`) are listed on that
  page rather than silently lowering the measured PER's meaning.

## Guard test

[`tests/test_data_quality.py`](../tests/test_data_quality.py) asserts
that any spec whose `quality` field is `production` has a corresponding
row in `benchmarks/results.json` meeting the tier's threshold (`n` ≥
500 and `per` ≤ `0.25`). This is the deep-orthography ceiling applied
uniformly to every language, because the guard has no per-language
orthographic-depth signal to key off. Shallow-orthography languages
(e.g. Spanish, Finnish, Esperanto) are still expected to meet the
tighter `≤ 0.15` target documented above, but that expectation is
enforced by reviewers reading the spec's `docs/languages/` page during
promotion review, not by this test. No spec is currently at
`production` tier, so the guard is presently vacuous: promoting a spec
to `production` is a test-gated act: the benchmark row must exist and
pass the `0.25` guard, and for shallow-orthography languages the `0.15`
target must also be confirmed by the reviewer before the tier
claim is legitimate.

---
[← Linguistic Accuracy Guide](linguistic_accuracy.md) · [Home](index.md) · [IPA Reference →](ipa_reference.md)
