# orthography2ipa — Agent Guide

Pure-data Python package: linguistically motivated grapheme→IPA and allophone mappings for 350+ language codes (356 JSON specs), plus a maximal-munch IPA tokenizer, phonological/script distance metrics, dialect transforms, and a pluggable G2P plugin system (e.g. algorithmic Arabic).

Read **[SKILLS.md](SKILLS.md)** for the task playbooks (add a language, benchmark, error-analyse).

## Design constraints (read before touching `data/*.json`)

These are the rules that keep the specs a linguistic description rather than a
lookup table. They are not style preferences — violating them silently degrades
every downstream consumer.

### 1. The engine is language-agnostic; the data holds the language

The golden rule. No language name, script, or code may drive a branch in
`orthography2ipa/*.py`. Conversely, a spec may not carry logic. If a language
needs behaviour the engine cannot express, **add a generic engine capability**
(a new context class, a new rule field) and use it from the data. Never
special-case the language in code, and never simulate the missing capability by
enumerating data.

### 2. A `graphemes` key is an orthographic unit, not an n-gram

A key must be something the writing system actually treats as a symbol:

- a letter (`b`, `ä`, `ц`), or
- a **multigraph the orthography treats as one unit** — ⟨sch⟩ /ʃ/, ⟨szcz⟩ /ʂtʂ/,
  ⟨tch⟩ /tʃ/, ⟨ng⟩ /ŋ/, or a unit of a designed romanisation (pinyin finals
  ⟨iang⟩, ⟨uai⟩).

A multigraph earns its key if **any** of these three tests passes:

1. **Its IPA is not merely the concatenation of its parts' IPA.** ⟨sch⟩→/ʃ/ earns
   its key because s+c+h would give /sx/.
2. **It spells a single phonological unit** — one segment or one syllable
   nucleus — even where the IPA string happens to read as a concatenation:
   diphthongs and triphthongs (Finnish ⟨ai⟩, Limburgish ⟨aaj⟩ — one nucleus),
   affricates (⟨dž⟩ /dʒ/, ⟨ts⟩), prenasalized stops (Bantu ⟨mb⟩ /mb/), clicks
   (⟨gc⟩ /ɡǀ/), aspirates (Urdu ⟨پھ⟩ /pʰ/), geminates (⟨tt⟩ /tː/), and the
   letter+diacritic units of an abjad or abugida. Collapsing these into one slot
   is a *claim about the phonology*, and it is the right one.

3. **It blocks a wrong maximal-munch match.** The tokenizer is greedy, so a key
   can earn its place by defeating a longer-but-wrong neighbour. Mirandese
   ⟨anh⟩ /aɲ/ exists because without it ⟨an⟩ (the nasal-vowel digraph) wins and
   ⟨danho⟩ comes out /dɐ̃u/ instead of /daɲu/. Say so in `notes`.

Fail all three and the key is noise. ⟨bl⟩→/bl/ fails: b+l already yields /bl/,
and a consonant cluster is not a unit. Russian ⟨бь⟩→/bʲ/ fails: ь is a diacritic
the engine already composes as /ʲ/, so the key never disambiguates anything.

Before deleting a key the audit flags, **check all three** — then delete it and
run the suite. A key that turns out to be load-bearing is telling you it encodes
a fact; find the fact and state it properly (usually as a rule), or keep the key
and record which test it passes.

Forbidden as keys:

- **Cartesian products.** Every C×C pair (`bd`, `bf`, `bk`, `bp`, `fb`, `dp`…),
  every C×V pair (`дзе`, `дзё`, `дзю`, `дзя`, `дзі`). If a block of keys looks
  like it was generated by a nested loop, it was, and it is wrong.
- **Morpheme and suffix chunks.** `tion`, `cious`, `ssion`, `tial` are
  morphology, not graphemes. A closed irregular set belongs in
  `word_exceptions`; a productive pattern belongs in the rule layer.
- **Statistical substrings.** Anything justified by "it occurs often" rather than
  "the orthography encodes it as one symbol".

### 3. Never encode context by enumeration

If a mapping depends on its neighbours, on stress, or on position, express that
**in the rule layer** — that is what the rule layer is for:

| The context | The mechanism |
|---|---|
| neighbouring sound / letter class | `allophone_rules` with `preceded_by` / `followed_by` (`vowel`, `consonant`, `front_vowel`, `back_vowel`, `palatal`, `word_boundary`) |
| word / syllable position | `positional_graphemes`, `word_initial` / `word_final` |
| stress | `stress` schema, `nucleus_stressed` |
| across a word boundary | `sandhi_rules` |
| a closed irregular set | `word_exceptions` |

Spelling out every context as its own grapheme key produces a combinatorial
explosion, asserts spellings the language does not have, and — because the
tokenizer is maximal-munch — **changes how neighbouring rules see the word**,
so the damage is not merely cosmetic.

If the rule layer genuinely cannot express the context, that is a **missing
engine predicate**. Add it (constraint 1). Do not enumerate around it.

### 4. Design purity and linguistic accuracy outrank the score — HARD RULE

A benchmark number is evidence, never a justification. If an enumeration, a hack,
or a special case improves a score, it is **still wrong and still gets removed**,
and the score is allowed to drop. Removing a violation is expected to change
output — that is the violation being undone, not a regression.

Never reach for data or code that exists to move a metric. The questions are "is
this how the language works?" and "is this expressible in the architecture?" — not
"does this raise WER/PER?".

Anything that is not a linguistic description of the language — text normalisation,
orthographic reform, lexicon lookup, diacritic restoration, morphology,
application-specific post-processing — **belongs in a downstream consumer that
interfaces with the lattice**, not in this repo. This library emits the candidate
lattice; consumers (arbtok, tugaphone, bifonia, …) decide what to do with it.

### 5. Every mapping is a cited claim

A spec asserts facts about a language. Back them: fill `sources`, `wikipedia`,
`urls`, and keep `quality` honest (`stub` → `production`). Cited accuracy beats
byte-identical agreement with a gold set or with espeak — gold sets are not
trusted, and a disagreement that you can cite is a **win**, not a regression.
Retain Wikipedia sources alongside primary literature; never strip them.

### 6. No corpora — the payload is the description, not the data

The package ships language *descriptions*: grapheme maps, rules, inventories,
ancestry. It ships **no word lists, no lexicons, no gold sets, no weights**. A
lexicon is a corpus, not a description of a language; bundling one bloats the
wheel and freezes a corpus into a release.

The lexicon overlay is caller-supplied — a local path, a URL, or a Hugging Face
id:

```python
orthography2ipa.register_lexicon("en-GB", "hf://TigreGotico/en-lexicon/en-GB.tsv")
orthography2ipa.set_lexicon_dir("~/lexicons")   # or $ORTHOGRAPHY2IPA_LEXICON_DIR
```

Resolution is lazy and cached; nothing is fetched unless the caller asked for it.
English is the illustration: without a lexicon ⟨nation⟩ is /nætɪɒn/, and that is
the *honest* output of English grapheme rules. The fix is a lexicon the caller
supplies, or a morphological analyser downstream — never a ⟨tion⟩ pseudo-grapheme
smuggled into the spec.

### 7. The library does not normalise text

Specs assume already-normalised input. Orthographic reform handling, diacritic
restoration, and script conversion belong downstream (bifonia, arbtok, tugaphone,
scriptconv), never here.

### 8. Leave no holes in the graph

When a spec references an ancestor or a related lect that has no file, add a
cited stub rather than leaving a dangling reference. Grouping/family nodes are
allowed only when the literature recognises them (`clade: true`).

## A cartesian can hide behind a non-trivial value

The derivable-key audit looks for a key whose IPA is the **concatenation of its
parts**. That is necessary, not sufficient. `ru` hid 85 enumerated keys in
`graphemes` — ⟨ки⟩ → /kʲɪ/ is *not* к+и, because it bundles palatalisation AND
vowel reduction into one spelling, so it sailed through the test. The full ru
cartesian was 152 keys across both tables, and the audit reported it clean.

**Audit for the SHAPE, not just the value.** A block of keys that is every C×V
or C×C pair is a cartesian whatever its values say. Sweep
`positional_graphemes` too (see SKILLS.md).

## A shared alphabet is not a shared phonology

Generating a second language's rules from a first language's template gets it
wrong every time. Ukrainian is not Russian, and assuming otherwise produced five
separate bugs in one sitting:

- uk ⟨и⟩ is /ɪ/ and **hard** — treating it as a soft trigger (as ru's ⟨и⟩ is)
  palatalises before every ⟨и⟩ in the language.
- uk had a rule backing ⟨и⟩ to **/ɨ/, a phoneme Ukrainian does not have**.
- Russian's assimilative softening does not hold in uk: ⟨знання⟩ is [znɐˈnʲːɑ].
- uk ⟨л⟩ is velarised /ɫ/, so the geminate is /ɫː/ — ⟨Аллах⟩ is [ɐˈɫːax].
- uk ⟨я⟩ was /jɑ/ while ⟨а⟩ was /a/ — the same vowel spelled two ways, an
  inconsistency the enumerated ⟨ся⟩ key had been hiding.

Derive each language's inventory FROM ITS OWN SPEC, and check every generated
rule against a cited form in that language.

## Offset context

`preceded_by_2` / `followed_by_2` (neighbour classes) and
`preceded_by_phoneme_2` / `followed_by_phoneme_2` (phonemes) test the grapheme
**two** away. Some processes must look past the grapheme next door:

- ⟨любовь⟩ is [lʲʊˈbofʲ] — the ⟨в⟩ devoices because the ⟨ь⟩ after it is
  word-final, and the ⟨ь⟩ is itself silent. But ⟨возьми⟩ is [vɐzʲˈmʲi]: a
  NON-final ⟨ь⟩ does not devoice, which the enumerated keys got wrong.
- ⟨гости⟩ is [ˈɡosʲtʲɪ] (Avanesov 1984) — the ⟨с⟩ softens because the ⟨т⟩ after
  it stands before a soft vowel.

**A new rule field must be added to `json_loader` as well.** It builds
`AllophoneRule` from an explicit key list, so a field absent from that list is
dropped at LOAD time and the rule silently matches on its other conditions
alone. That hid this feature for an entire PR: the rules read correctly, the
spot-checks passed, and the PER gap had a plausible and completely wrong
explanation ready. `tests/test_allophone_rules.py::TestOffsetContext` pins it.

## Setup

```bash
pip install -e .
# validation extra (pydantic schema checks):
pip install -e .[validation]
```

Runtime deps are minimal: `numpy`, `langcodes` (see `requirements.txt`). `langcodes` is used for ISO 639-3 → BCP-47 normalisation, with a hand-maintained fallback alias table in `registry.py`.

## Test

```bash
pytest tests
# with coverage (as CI runs it):
pytest --cov=orthography2ipa --cov-report xml tests
```

`tests/pytest.ini` and `tests/conftest.py` configure the suite. There is a broad per-family test layout (`test_iberian.py`, `test_celtic.py`, `test_slavic.py`, `test_germanic.py`, `test_indo_iranian.py`, …) plus `test_all_languages.py` and `test_language_integrity.py` that sweep every data file.

## Lint/Typecheck

No linter or type checker is configured. Code uses `from __future__ import annotations` and typed dataclasses but there is no mypy/ruff/flake8 config.

## Layout

- `orthography2ipa/types.py` — frozen dataclasses: `LanguageSpec`, `Grapheme2IPA`, `AllophoneMap`, `Ancestor`, `PositionalGrapheme2IPA`, `SandhiRule`; enums `QualityTier`/`ScriptType`/`AncestorRole`.
- `orthography2ipa/data/*.json` — 356 language/dialect spec files (the actual payload). `data/SCHEMA.md` documents the format; dialects inherit via `graphemes_base`/`allophones_base`. No lexicons are bundled — a word list is a corpus, not a description of a language; callers register one (path / URL / `hf://` id).
- `orthography2ipa/json_loader.py` — loads JSON specs and lexicons, resolves multi-ancestor inheritance.
- `orthography2ipa/registry.py` — `get()`, `available_codes()`, `available_families()`; lazy cache + plugin discovery + ISO alias table.
- `orthography2ipa/phonetok.py` — `PhonetokTokenizer`, beam-search IPA expansion (`IPAPath`, `Token`, `TokenKind`).
- `orthography2ipa/distance.py` + `feats.py` + `script_distance.py` — phonological/inventory/grapheme/tone/script distance metrics and feature vectors.
- `orthography2ipa/transforms.py` + `sandhi.py` + `lm.py` — dialect transforms, sandhi rules, language-model scoring helpers.
- `orthography2ipa/g2p_plugin.py` — `G2PPlugin`/`WordContext` base types for richer language-specific engines built on this library; no in-tree implementation ships here. Downstream consumers include [arbtok](https://github.com/TigreGotico/arbtok) (Arabic) and [tugaphone](https://github.com/TigreGotico/tugaphone) (Portuguese).
- `orthography2ipa/cli.py` — `orthography2ipa` console entry point (`list`, `info`, `transcribe`, `distance`; all support `--json`).
- `examples/` — runnable usage demos; `docs/` — Markdown reference (architecture, data model, tokenizer, distance, adding a language, bibliography).

### Entry-point groups

- `[project.scripts]` → `orthography2ipa = orthography2ipa.cli:main` (CLI).

## Maintainability gate (part of every review)

Every PR that touches `orthography2ipa/*.py` answers to these before merge —
an adversarial review that skips them is incomplete. The engine stays small
because every addition pays this toll:

- **One concept, one place.** A new rule condition is ONE field, checked in
  ONE place in the match loop, documented in ONE docstring. If a behavior
  needs two cooperating helper predicates to express, collapse them into one
  named function with a single documented contract first — cooperating
  helpers whose combined semantics live in nobody's head are how the
  `-ers` regression happened (a vowel-silencing predicate silently
  generalized to consonant digraphs).
- **Additions mirror an existing pattern.** A positive condition mirrors its
  negative twin (`followed_by_grapheme` / `_not`); a new stress value
  extends the existing conditional; a new position slots into the existing
  precedence list. If the addition needs a new *shape* of code, that is an
  architecture change: name it in the PR body and justify it.
- **No language names in engine code.** Ever. Language facts live in
  `data/*.json`; the engine consumes declared fields. A closed set in
  engine code (like the transparent-suffix graphemes) is allowed only when
  it is a cross-linguistic class with a citation attached at the constant.
- **Flat beats clever.** The match loop is a flat sequence of independent
  condition checks; keep it that way. No nested dispatch, no
  metaprogramming, no condition that reads another condition's result.
- **Every new field lands in four places or it doesn't land**: `types.py`
  (dataclass + docstring), `schema.py` (validation), `json_loader.py`
  (parse), `data/SCHEMA.md` (user docs). Reviewers check all four.
- **Leave it cleaner.** If a PR's diff touches a function that already
  smells (three cooperating predicates, a boolean parameter that flips
  semantics, a >20-line conditional), the PR either cleans it or files the
  smell explicitly in its body. Silent accretion is a review finding.

### Lattice-ambiguity gate

First, the rule that governs every ending, list-valued or not: **every ending
is a cited linguistic claim about a suffix's realisation, never a PER-chasing
pattern.** A key earns its place from a published source — a paper or a
reference grammar — that states how the suffix is realised. Corpus frequency
supports a list's *ordering*; a frequency count alone is not a citation. This
is the same prohibition that keeps n-grams out of `graphemes`, and it is
enforced mechanically: every declared ending must be named in its spec's
`notes` with a citation traceable to that spec's `sources`.

A **list-valued `grammatical_endings` entry** injects a reading into the beam
that the spec cannot choose between (see
[`SCHEMA.md`](orthography2ipa/data/SCHEMA.md#ambiguous-endings)). It is the one
sanctioned way morphological ambiguity reaches the lattice, and it is exactly
the shape a lexicon or a paradigm table would take if it were smuggled in. So
an adversarial review of any PR that adds or widens one **must verify all
three**, and say in writing that it did:

1. **The lattice hole is proven.** The PR shows the **0-in-top-k**
   measurement on gold — the missing reading absent from `word_candidates` at a
   generous *k* and beam width, over a named sample — and the post-change
   reachability number. A reading that was already reachable at some *k* is a
   ranking problem; it is fixed with weights or downstream, and the entry is
   rejected.
2. **The ordering is cited.** Which reading holds rank 1 is a frequency claim.
   It carries a source or a measurement on gold, in the spec's `notes` beside
   the ending. "It felt more common" is a rejection.
3. **The oracle delta is segregated.** Oracle@k movement from an injected
   alternative is reported under its own heading, is not folded into any
   ranking-error headroom or per-phenomenon failure count, and appears in no
   beat-espeak or cross-system claim (see
   [`benchmark_methodology.md`](docs/benchmark_methodology.md#injected-alternatives-do-not-count-as-ranking-error)).
   A PR that reports it as a PER-headroom win is rejected on that alone.

Two mechanical gates back this up and must stay green:
`tests/test_grammatical_endings.py` asserts every list-valued ending is
documented with its evidence in its spec's `notes`, and caps how many a single
spec may declare. Raising that cap is an **amendment to this file**, argued
here first — the cap exists so paradigm enumeration hits a tripwire instead of
accreting one plausible entry at a time.

## Never carry the board backwards

The benchmark board — `benchmarks/results.json`, `benchmarks/results_ci_sample.json`
and `docs/scoreboard.md` — is regenerated whole, but a branch only rescores the
rows it worked on. Every other row is a copy of `dev` as it stood when the branch
was cut. That staleness on its own is harmless: the merge is a three-way merge,
so rows the branch never edited keep `dev`'s newer values and the copy never
lands.

What does land is a branch that edited the board and then took its own side for
rows it never measured — almost always by merging `dev` in and resolving the
conflicted board file by keeping its own file wholesale. Then `dev`'s newer
numbers really are written away, and nothing notices: both sides stay internally
consistent, so the regression gate is green on the branch and green on `dev`,
and the loss is visible only in the interaction. When you merge `dev` into a
branch, resolve the board files row by row, keeping `dev`'s value for every row
you did not rescore yourself, and regenerate `docs/scoreboard.md` afterwards.

`scripts/check_board_not_reverting.py` blocks the bad resolution in CI. It
performs the merge with `git merge-tree` and compares the resulting board
against `dev`'s. A row the merge would set to a value `dev` already carried and
moved past fails the check, unless the branch edits that language's spec —
re-measuring after a spec change can legitimately reproduce an old number, and a
spec edit is the one piece of evidence a stale copy cannot produce. Prose cannot
buy that row off. Board tags are resolved to a spec the way the harness resolves
them, so an edit to `de-DE.json` owns the `de` rows. Ownership follows the
inheritance chain: editing a proto or clade spec that a language's spec
descends from — through `parent` or a `*_base` table reference — owns that
language's rows too, so an edit to `ine.json` owns every Indo-European
dialect's row, not just the language it names directly.

A change that moves rows it does not own, such as a harness or engine rework,
declares them: a `Board-Rows: mr/wikipron, xh/kaikki` line, or
`Board-Rows: all - <reason>` when the whole board was rescored, in the pull
request body or a commit message.

The burn-down lists in the test suite fail the same way and are checked by the
same script. `_WIKIPEDIA_ONLY_SOURCES_BURNDOWN` in `tests/test_sources.py` is a
set of codes whose bare-Wikipedia `sources[]` predate the test that forbids
them; it may only shrink. Because it is an allowlist, restoring an entry makes
the suite *more* green, so a branch that resolves a rebase toward its own longer
copy resurrects the skips with nothing to show for it. The guard reads the set
off the merge result and fails if it grew. Resolve that file keeping `dev`'s
shorter set.

`_DOCS_PROSE_RATCHET` in `tests/test_docs_prose.py` works the same way for
documentation prose. Every page under `docs/` is linted by
`scripts/ste_lint.py` in its STE-flavoured mode, and a page must stay under two
violations per 100 words with no slop and no marketing adjectives. The listed
pages predate the gate and may fail; the list may only shrink. A page that is
not listed must pass, and a listed page that now passes has to be taken off the
list. Never add a page to make a red build green — rewrite the prose instead.
Generated pages are linted like any other, so a violation in
`docs/comparison.md` or `docs/spec_diagnostics.md` is fixed in the generator
script and the page is regenerated.

## Conventions (Org hard rules)

- Branches: `dev` for work, `master` for stable. NEVER `main`.
- Never edit `orthography2ipa/version.py` — gh-automations bumps semver from conventional-commit prefixes (`feat:`, `fix:`, `feat!:`).
- New repos private by default; do not make source public without asking.
- Commit identity: `JarbasAi <jarbasai@mailfence.com>`.
- Reference `TigreGotico`/`OpenVoiceOS` gh-automations reusable workflows at `@dev`.
- No Neon / `neon-*` references.
- No meta-commentary: describe current state only — no history, dates, or "design mistake" framing in docs/commits/PRs/comments.
- CI is provided by gh-automations reusable workflows.

## Gotchas

- This is **pure data + logic, no trained network weights** despite living in the ML cluster — the package ships no G2P plugin implementation; algorithmic diacritization/G2P pipelines (e.g. Arabic tashkeel) live in downstream consumers such as [arbtok](https://github.com/TigreGotico/arbtok).
- `dynamic = ["version", "dependencies"]`: version comes from `orthography2ipa/version.py` attr, deps from `requirements.txt`. The release workflows reference a `setup.py` that is not present in the tree — packaging is `pyproject`-only, so the `setup.py`-based release steps will fail.
- `QualityTier` ranges from `stub`/`skeleton` through `research`/`production`; not every one of the 356 specs is `production` quality. Check `spec.quality` before relying on a mapping.
- Graphemes ≠ allophones: `graphemes` maps a spelling to the phonemes it can represent; `allophones` maps a phoneme to its contextual surface forms. Keep them distinct.
- Many scratch report files (AUDIT.md, MAINTENANCE_REPORT.md, SUGGESTIONS.md, PLAN.md, QUICK_FACTS.md, FAQ.md) and 78 `.pyc` files are committed despite `.gitignore` — do not add more.
