# Data Model

## Overview

The entire package is built around a small, stable set of types defined in `types.py`. All types are immutable (frozen dataclasses or enums), enabling safe caching and sharing.

---

## `Grapheme2IPA`

```python
Grapheme2IPA = Dict[str, List[str]]
```

A mapping from orthographic strings to lists of IPA phoneme strings.

**Keys** are graphemes as they appear in the language's orthography. They may be:
- Single characters: `"a"`, `"b"`, `"ñ"`, `"ü"`
- Digraphs: `"ch"`, `"ll"`, `"sh"`, `"lh"`, `"nh"`, `"rr"`
- Trigraphs (rare): `"sch"` in German, `"tch"` in French
- Diphthong spellings: `"ai"`, `"ie"`, `"ua"`: for languages where diphthong graphemes need explicit handling

**Values** are lists of IPA strings representing the possible phonemic realisations of the grapheme, ordered from most to least common:

```python
# English
english = {
    "c":  ["k", "s"],      # /k/ is primary (cat, cup), /s/ secondary (cent, city)
    "th": ["θ", "ð"],      # /θ/ primary (think), /ð/ secondary (the, this)
}

# Castilian Spanish
castilian = {
    "c":  ["k", "θ"],      # /k/ before a,o,u; /θ/ before e,i
    "ll": ["ʎ", "ʝ"],      # /ʎ/ traditional lleísmo; /ʝ/ yeísmo variant
}

# Portuguese
portuguese = {
    "lh": ["ʎ"],                       # always /ʎ/: no ambiguity
    "x":  ["ʃ", "ks", "s", "z"],       # highly ambiguous in Portuguese
}
```

The ordering matters: the `PhonetokTokenizer`'s beam search treats the first value as the canonical (lowest-scoring) option.

---

## `AllophoneMap`

```python
AllophoneMap = Dict[str, List[str]]
```

A mapping from underlying phonemes to their contextual surface realisations (allophones).

**Keys** are IPA phoneme strings (the underlying phonological forms).

**Values** are lists of IPA strings representing the attested surface forms:

```python
# English /t/: multiple allophones depending on phonological context
english_t = ["t",   # canonical: word-initial (top, ten)
             "tʰ",  # aspirated: syllable-initial before stressed vowel
             "ɾ",   # flap: intervocalic in North American English (butter, water)
             "ʔ",   # glottal stop: before syllabic /n/ (button, cotton)
             "t̚"]  # unreleased: word-final (cat, but)

# Spanish /b/: lenition
spanish_b = ["b",   # after pause or nasal (burro, un barco)
             "β"]   # spirantised: all other positions (lobo, el barco)

# Spanish /n/: place assimilation
spanish_n = ["n",   # default alveolar
             "m",   # before bilabials (un padre → [um padre])
             "ɱ",   # labiodental before /f/ (enfermo)
             "n̪",  # dental before /t,d/
             "ŋ",   # velar before /k,g/ (banco)
             "ɲ"]   # palatal before /ʎ,ʝ/ (ancho)
```

The `AllophoneMap` does **not** encode the phonological rules that determine which allophone occurs: it only records *which allophones exist* for distance calculations and phonological inventory analysis. Allophone selection rules would need to be implemented separately by the consumer.

---

## `AncestorRole`

```python
class AncestorRole(Enum):
    PARENT          = "parent"
    PARENT_DIALECT  = "parent_dialect"
    PROTO_LANGUAGE  = "proto_language"
    ANCESTOR        = "ancestor"
    SUBSTRATE       = "substrate"
    SUPERSTRATE     = "superstrate"
    ADSTRATE        = "adstrate"
    LEXIFIER        = "lexifier"
    CREOLE_BASE     = "creole_base"
    RELATED         = "related"
```

Encodes the **type of historical relationship** between a language and its ancestor.

| Role | Definition | Example |
|---|---|---|
| `PARENT` | Primary genetic descent | Latin → Spanish |
| `PARENT_DIALECT` | Direct dialectal ancestor within the same language | A regional variety descending from a broader standard |
| `PROTO_LANGUAGE` | Reconstructed common ancestor at the top of a lineage | Proto-Indo-European, Proto-Germanic, Proto-Semitic |
| `ANCESTOR` | Earlier historical stage of the same lineage that is not the immediate parent | Old Spanish in the ancestry of modern Spanish |
| `SUBSTRATE` | Language of the pre-existing population | Basque substrate in Castilian |
| `SUPERSTRATE` | Language of a dominant group, later absorbed | Frankish superstrate in French |
| `ADSTRATE` | Peer contact influence from a neighbouring language | Arabic adstrate on Ibero-Romance |
| `LEXIFIER` | Vocabulary source in creole/pidgin formation | Portuguese lexifier of Papiamento |
| `CREOLE_BASE` | Grammar/structural source in creole formation | West African languages as creole base |
| `RELATED` | Sister language with shared features but no direct descent | Used for typological comparison rather than inheritance |

### Linguistic notes

**PARENT**: Every language (except reconstructed proto-languages) should have exactly one PARENT. This is the primary line of descent. The weight is typically 0.7-1.0.

**PARENT_DIALECT**: Direct dialectal ancestor within the same language, for example a regional variety descending from a broader standard.

**PROTO_LANGUAGE**: The reconstructed common ancestor at the top of a lineage, such as Proto-Indo-European, Proto-Germanic, or Proto-Semitic.

**ANCESTOR**: An earlier historical stage of the same lineage that is not the immediate parent, such as Old Spanish in the ancestry of modern Spanish.

**SUBSTRATE**: A substrate is the language of the original population that adopted the incoming language. Substrate effects show up as phonological features that deviate from the parent. The Basque substrate in Castilian is the classic explanation for the *f-* → *h-* sound change (Latin *filium* → Spanish *hijo*). Weight: 0.05-0.30.

**SUPERSTRATE**: A superstrate is a prestige language whose speakers eventually shift to the local language but leave phonological and lexical traces. Frankish (Proto-Germanic) contributed vocabulary and possibly phonological features to Old French. Weight: 0.10-0.40.

**ADSTRATE**: An adstrate is a neighbouring language at roughly equal social status that exerts ongoing influence. Arabic was an adstrate on medieval Ibero-Romance, contributing over 4,000 loanwords to Spanish and Portuguese. Weight: 0.05-0.20.

**RELATED**: A sister language that shares features with the descendant but is not on its line of descent. This role is used for typological comparison rather than inheritance.

---

## `Ancestor`

```python
@dataclass(frozen=True)
class Ancestor:
    code: str            # Language code of the ancestor
    role: AncestorRole   # Type of relationship
    weight: float = 0.5  # Contribution weight [0.0, 1.0]
    notes: str = ""      # Optional explanatory notes
```

### Weight guidelines

| Role | Typical weight range |
|---|---|
| PARENT | 0.70-1.00 |
| SUBSTRATE | 0.05-0.30 |
| SUPERSTRATE | 0.10-0.40 |
| ADSTRATE | 0.05-0.20 |
| LEXIFIER | 0.50-0.80 |
| CREOLE_BASE | 0.20-0.50 |

Weights do not need to sum to 1.0. They represent approximate phonological contribution, not relative proportion of heritage.

### Example

```python
from orthography2ipa.types import Ancestor, AncestorRole

Ancestor(
    code="la-x-hispania",
    role=AncestorRole.PARENT,
    weight=0.80,
    notes="Primary descent from Hispanic Vulgar Latin"
)
```

---

## `LanguageSpec`

`orthography2ipa/types.py:379`

```python
@dataclass(frozen=True)
class LanguageSpec:
    code: str                                          # BCP-47 or ISO 639 code
    name: str                                          # Human-readable name
    family: str                                        # DERIVED classification path
    family_path: Tuple[str, ...]                       # DERIVED clade chain, broadest first
    clade: bool                                        # classification-only node
    script: str                                        # Primary script
    graphemes: Grapheme2IPA                            # Grapheme → IPA
    allophones: AllophoneMap                           # Phoneme → allophones
    phonemes: Tuple[str, ...] = ()                     # The inventory, stated directly
    orthography_kind: OrthographyKind = OrthographyKind.NATIVE  # native | romanization | transliteration
    parent: Optional[str] = None                       # Primary parent code (shorthand)
    ancestors: Tuple[Ancestor, ...] = ()               # Full ancestry specification
    positional_graphemes: PositionalGrapheme2IPA = None # Context-sensitive IPA (normalised to {} in __post_init__)
    glottolog_code: str | None = None                  # Glottolog languoid code
    notes: str = ""                                    # Free-form notes
    quality: QualityTier = QualityTier.RESEARCH        # Data maturity tier
    script_type: ScriptType = ScriptType.ALPHABET      # Writing system typology
    inherent_vowel: Optional[str] = None               # For abugidas (e.g. "ə" for Hindi)
    vowel_graphemes: Tuple[str, ...] = ()              # Closed-inventory letters this spec declares vowels (e.g. Hmong RPA "w")
    iso639_3: Optional[str] = None                     # ISO 639-3 code
    wikidata_qid: Optional[str] = None                 # Wikidata item id: the linked-data hub
    phoible_id: Optional[str] = None                   # PHOIBLE inventory identifier
    wals_code: Optional[str] = None                    # WALS typological cross-reference
    sandhi_rules: Tuple[SandhiRule, ...] = ()          # Cross-word phonological rules
    allophone_rules: Tuple[AllophoneRule, ...] = ()    # Post-lexical phoneme → surface rewrites
    stress: Optional[StressRules] = None               # Stress placement rules
    tone_inventory: Optional[Dict[str, str]] = None    # IPA tone mark → label
    tone_marks_syllable_final: bool = False            # dock tone at the syllable end
    word_exceptions: Optional[Dict[str, str]] = None   # Whole-word overrides for an irregular set
    grammatical_endings: Optional[Dict[str, EndingValue]] = None  # Suffix morphology: ending → IPA (or an ordered candidate list) at the word end
    orthography_standard: Optional[OrthographyStandard] = None  # Official spelling norm
    location: Optional[Location] = None                # Representative point (lat/lon)
    timespan: Optional[TimeSpan] = None                # Attestation period
    sources: Tuple[LinguisticSource, ...] = ()         # Bibliographic references
    wikipedia: Tuple[str, ...] = ()                    # Wikipedia article URLs
    urls: Tuple[str, ...] = ()                         # Other reference URLs
```

### `phonemes`: the inventory, stated directly

`phonemes` is the language's phoneme inventory: the sounds it **has**, declared
independently of `graphemes`. A language's sounds are not a property of its writing
system. Most of the world's languages are unwritten or barely written and PHOIBLE
catalogues inventories for thousands of them; a logographic script encodes no sound
at all. Reading the inventory out of the spelling cannot work for either.

```python
import orthography2ipa

ine = orthography2ipa.get("ine")        # Proto-Indo-European: never written
len(ine.phonemes)                       # 30 reconstructed phonemes, declared

han = orthography2ipa.get("zh-Hani")    # Chinese in Han script
han.graphemes                           # {}: no grapheme→IPA rule exists to write
len(han.phonemes)                       # 60: the phonology is known regardless
```

When `phonemes` is empty the inventory is **derived** from `graphemes`: every
phoneme any grapheme can produce: so a spec that says nothing keeps exactly the
inventory it always had. That derivation is a *fallback*, not the definition: it
reads the sounds out of the spelling, which is backwards, and it is why a
reconstructed language once had to fake an identity orthography (`p` → [p]) merely
to have an inventory. `distance._extract_phonemes` applies the rule: declared
inventory wins, grapheme map is the fallback: and every phonological metric is
built on it.

**The integrity invariant:** a spec must declare `graphemes` **or** `phonemes`. It
may not be silent about both. "Every language has graphemes" is false: logographic
scripts and unwritten languages are the counterexamples: and no spec is allowed to
claim it has neither sounds nor spelling.

### `orthography_kind`: what kind of writing the graphemes are

`native` (default) | `romanization` | `transliteration`. Han characters, Pinyin and
Buckwalter ASCII make three different claims, and a consumer must be able to tell
them apart:

```python
import orthography2ipa

orthography2ipa.get("zh").orthography_kind                   # ROMANIZATION: Pinyin (ISO 7098)
orthography2ipa.get("zh-Hani").orthography_kind              # NATIVE: Han script
orthography2ipa.get("ar-Latn-buckwalter").orthography_kind   # TRANSLITERATION: Arabic in ASCII
```

Full treatment, with the reason a romanization is transcribable where the native
script is not: [orthography_kind.md](orthography_kind.md).

### `vowel_graphemes`: overriding a closed-inventory consonant letter

`orthography2ipa.vowels.grapheme_is_vowel` treats the Latin, Greek and
Arabic-harakat letter sets as **closed inventories**: a Latin letter absent
from them (⟨w⟩, ⟨y⟩, ⟨r⟩) is a consonant letter by construction, no matter
what IPA a spec maps it to. That is deliberate — Czech syllabic ⟨r⟩ and
English ⟨y⟩ must not become vowels just because their IPA output happens to
be a vowel elsewhere.

Some orthographies genuinely use one of those closed-inventory consonant
letters to spell a vowel: Hmong RPA ⟨w⟩ = /ɨ/, Welsh ⟨w⟩ = /ʊ, w/.
`vowel_graphemes` is the per-spec escape hatch — a tuple of whole grapheme
strings (matched in full, not by first character) that this spec declares
vowel letters, overriding the closed-inventory answer:

```python
import orthography2ipa

orthography2ipa.get("mww").vowel_graphemes   # ("w",)
```

Empty (the default) is the previous, closed-inventory-only behaviour exactly.
It is `OWN_ONLY` inherited — a dialect that shares a parent's grapheme table
does not thereby inherit its parent's script exceptions, exactly like
`script_type` and `optional_marks`.

### Derived classification: `family`, `family_path`, `clade`

`family` is not authored on a spec. Families are **clade nodes** in the ancestry
graph: specs flagged `clade=True`, which carry classification and nothing else: and the loader derives `family_path` by walking the `parent` chain upwards and
collecting the clade names it passes, broadest first. `family` is that path joined
with `" > "`:

```python
import orthography2ipa

orthography2ipa.get("pt-BR").family_path   # ('Indo-European', 'Italic', 'Romance', 'Ibero-Romance')
orthography2ipa.get("pt-BR").family        # 'Indo-European > Italic > Romance > Ibero-Romance'
orthography2ipa.get("pt-BR").clade         # False
```

A spec may still carry an explicit `family` string, which wins over the derived
one; that escape hatch exists for groupings that are not genetic clades (creoles,
constructed languages, isolates, unclassified languages). Clade nodes are excluded
from `available_codes()` unless `include_clades=True` is passed. See
[ancestry.md](ancestry.md#clade-nodes-and-the-derived-family).

### `Location`, `OrthographyStandard`, `TimeSpan`

| Type | Fields | Purpose |
|---|---|---|
| `Location` | `latitude`, `longitude`, `source`, `notes` | Representative point for the variety; feeds `geographic_distance`. A clade's point is a centroid of its descendants. |
| `OrthographyStandard` | `name`, `authority`, `year`, `url`, `notes` | The official published spelling norm, when one exists. A property of the language, not of every dialect of it: a dialect that spells by its standard language's norm omits the field and consumers walk the ancestry chain. |
| `TimeSpan` | `start_year`, `end_year` | Attestation period; decays ancestry weights across time and drives `temporal_distance`. |

```python
import orthography2ipa

orthography2ipa.get("gl").orthography_standard.authority
# 'Real Academia Galega / Instituto da Lingua Galega'
orthography2ipa.get("ca").location.latitude   # 41.453
```

### Accessor methods and properties

```python
import orthography2ipa
from orthography2ipa.types import AncestorRole, GraphemePosition

spec = orthography2ipa.get("es-ES")

# Primary parent code
spec.primary_parent          # "la-x-hispania"

# All ancestors, optionally filtered by role
spec.get_ancestors()                           # all ancestors
spec.get_ancestors(AncestorRole.SUBSTRATE)     # substrate ancestors only

# Convenience tuples of codes
spec.substrate_codes          # ('xaq',)
spec.superstrate_codes        # ('got',)
spec.contact_codes            # all non-parent ancestors: ('xaq', 'xaa', 'got')

# Positional grapheme resolution
spec.resolve_grapheme("b", GraphemePosition.INTERVOCALIC)  # ["β"]
spec.has_positional_data()           # bool
spec.positional_grapheme_keys()      # frozenset of graphemes with overrides
spec.positions_for_grapheme("b")     # tuple of GraphemePosition values
```

### The `parent` vs `ancestors` fields

For backward compatibility, `LanguageSpec` supports both `parent` (a simple string shorthand) and `ancestors` (the full typed tuple). If `ancestors` is empty but `parent` is set, `__post_init__` synthesises a `PARENT` ancestor with weight 1.0 automatically. Conversely, if `parent` is empty but `ancestors` contains a PARENT-role entry, `parent` is set from it.

For new languages and dialects, always populate the `ancestors` tuple for richest data. Simple dialects that differ minimally from a parent may use just `parent`.

### `QualityTier` enum

`orthography2ipa/types.py:174`

| Value | Description |
|---|---|
| `STUB` | Code + name + script + a parent to hang the clade chain on; no phoneme data |
| `SKELETON` | Graphemes + allophones from auto-generation (unvalidated) |
| `RESEARCH` | Validated against published phonology; positional rules present |
| `PRODUCTION` | Full coverage, regression-tested, cited sources |

### `ScriptType` enum

`orthography2ipa/types.py:194`

| Value | Description |
|---|---|
| `ALPHABET` | Latin, Cyrillic, Greek, Armenian, Georgian |
| `ABJAD` | Arabic, Hebrew: consonants primary, vowels optional |
| `ABUGIDA` | Devanagari, Bengali, Tamil, Thai: inherent vowel |
| `SYLLABARY` | Kana, Cherokee |
| `LOGOGRAPHIC` | Hanzi / CJK ideographs |
| `FEATURAL` | Hangul |
| `MIXED` | Japanese (logographic + syllabary) |
| `RECONSTRUCTION` | IPA-based phonological reconstruction for extinct languages |

### `LinguisticSource` dataclass

`orthography2ipa/types.py:258`

Fields: `id`, `author`, `year`, `title`, `publisher`, `url`, `wikipedia_url`, `pages`, `notes`.

### `SandhiRule` dataclass

`orthography2ipa/types.py:300`

Fields: `id`, `name`, `left_context`, `right_context`, `transform`,
`right_transform`, `obligatory`, `notes`.

A boundary has two sides and a rule may rewrite either or both:

- `transform` substitutes for the `left_context` match in the LEFT word: the
  shape a right-conditioned (regressive) process needs, e.g. European
  Portuguese coda-/s/ voicing. `None` leaves the left word alone.
- `right_transform` substitutes for the `right_context` match in the RIGHT word: the shape a left-conditioned (**progressive**) process needs. Catalan
  phrase-level spirantization is one: continuant-spreading rightwards across
  the boundary, so the *following* word's initial /b d ɡ/ lenites after a
  continuant (`de decidir` → [ðə ðəsiˈði]); Spanish and Galician have the same
  shape. `None` (the default) leaves the right word alone, so a rule that
  declares only `transform` behaves exactly as it always has.

Which side is rewritten is the rule's TARGET, not its direction of
conditioning; a rule must declare at least one of the two, or it matches a
boundary and changes nothing (the schema rejects it).

The two sides resolve independently: first matching rule wins *per side*: so
one boundary can both voice the left word's coda and lenite the right word's
onset (`els dos` → [əlz ðos]). Contexts are always matched against the original
words, so the side that fires first cannot mask the other's trigger.

### `StressRules.diphthongs`

The bundled syllabifier counts each maximal run of vowel LETTERS as one nucleus.
That is right for a language whose vowel runs are all diphthongs and wrong
wherever the orthography also writes hiatus. `diphthongs` lists the vowel
sequences that form a single nucleus; every other vowel letter in a run becomes
a nucleus of its own, so Catalan `ciutat` stays ci-u-tat = ciu-tat while `tenia`
splits te-ni-a. Empty (the default) merges the whole run, unchanged.

This matters far beyond the stress mark: in a language with stress-conditioned
vowel reduction, a wrong syllable count reduces the wrong vowel and the whole
word is wrong.

---

## Lexicon overlay (sidecar `word_exceptions` at scale)

`orthography2ipa/lexicon.py`

Deep-orthography languages (English, Danish, Irish) cannot reach production
accuracy from grapheme rules alone: too many words are irregular. The
`word_exceptions` field on `LanguageSpec` handles a *closed, tiny* set of
irregulars inline in the JSON, but it does not scale to thousands of entries.

A **lexicon** is an optional, convention-based sidecar file
a caller-registered TSV (`orthography2ipa.register_lexicon(code, path_or_url_or_hf_id)`; nothing is bundled): one `word<TAB>ipa` pair per line,
UTF-8, NFC-normalised, lowercase words, sorted, first-entry-wins. It needs **no
new spec field and no JSON change**: the file is discovered by its name (the
resolved language code, e.g. `en-GB.tsv`), so `FIELD_INHERITANCE` is untouched.

Contract:

- **Lazy.** Importing the package, and even loading a `LanguageSpec`, reads
  **no** lexicon. A language's TSV is read once, on the first transcription for
  that language (`get_lexicon(code)`), then cached for the process lifetime.
- **Same pathway as `word_exceptions`.** `G2P._override_for` folds a lexicon
  hit into the *exact* override path `word_exceptions` uses, so a lexicon entry
  still gets stress-mark insertion and cross-word sandhi applied and is
  reported with `confidence == 1.0` (a certain answer).
- **Precedence:** inline `word_exceptions` **>** lexicon **>** rules. An inline
  exception always wins; the sidecar always wins over the grapheme/positional
  beam.
- **Absent lexicon → no effect.** A language with no `{code}.tsv` gets an empty
  overlay, and the rules alone decide the transcription.

Every shipped TSV is validated (parseable `word<TAB>ipa`, NFC, lowercase word,
IPA-characters-only) by `lexicon.validate_lexicon_text`; the shipped-data test
runs that guard over every file. The one bundled pilot is `en-GB.tsv`
(CMUdict-derived: see [`bibliography.md`](bibliography.md) and
`scripts/build_en_lexicon.py`); the rules-only vs with-lexicon PER impact is
reported in [`lexicon_scoreboard.md`](lexicon_scoreboard.md). Full production
lexica belong downstream: see [`adding_a_language.md`](adding_a_language.md).

---

## Relationship between `graphemes` and `allophones`

These are **different layers** of phonological description:

```
orthography → graphemes → underlying phonemes → allophones → surface forms
    "butter"    b,u,tt,e,r     /b ʌ t ə r/     [b ʌ ɾ ə r]    "buh-der"
```

- `graphemes` maps **orthography → phonology** (abstract level)
- `allophones` maps **phonology → phonetics** (surface level)

A word like English *butter* goes through two mappings:
1. Grapheme step: `tt` → `/t/` (the underlying phoneme)
2. Allophone step: `/t/` → `[ɾ]` (the flap allophone in this context)

The package stores both levels; rule-based selection between allophones requires implementation by the consumer.

---

**Navigation:** [Docs home](index.md) · [Getting started](getting_started.md) · [Architecture](architecture.md) · [Languages](languages/index.md) · [Scoreboard](scoreboard.md)

*Related: [Architecture](architecture.md) · [Positional graphemes](positional_graphemes.md) · [Allophony](allophony.md) · [Registry](registry.md)*
