Metadata-Version: 2.4
Name: uniscriptipa
Version: 0.1.0
Summary: Selective IPA transliteration for multilingual text
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: phonemizer
Requires-Dist: pyicu
Requires-Dist: regex
Requires-Dist: uroman
Description-Content-Type: text/markdown

# UniScriptIPA

**UniScriptIPA** provides script-selective IPA transliteration for multilingual text.

It uses Phonemizer's eSpeak backend to generate IPA, but only replaces tokens that contain characters from a user-specified Unicode script. Digits, whitespace, punctuation, symbols, and text written in other writing systems are otherwise preserved in their original positions.

## Public API

UniScriptIPA provides three main functions:

| Function        | Purpose                                                                                    |
| --------------- | ------------------------------------------------------------------------------------------ |
| `tokenize()`    | Losslessly splits one string into tokens                                                   |
| `str_to_ipa()`  | Converts one string and returns its IPA text and token-level annotations                   |
| `list_to_ipa()` | Converts multiple strings together and returns their IPA texts and token-level annotations |

Two more descriptive aliases are also provided:

| Alias            | Equivalent function |
| ---------------- | ------------------- |
| `text_to_ipa()`  | `str_to_ipa()`      |
| `texts_to_ipa()` | `list_to_ipa()`     |

They can be imported directly from the package:

```python
from uniscriptipa import (
    tokenize,
    str_to_ipa,
    list_to_ipa,
    text_to_ipa,
    texts_to_ipa,
)
```

`text_to_ipa` and `str_to_ipa` are the same function. Likewise, `texts_to_ipa` and `list_to_ipa` are the same function.

## Installation

### Install UniScriptIPA

Install UniScriptIPA and its Python dependencies from PyPI:

```bash
pip install uniscriptipa
```

### Install eSpeak NG

UniScriptIPA uses Phonemizer's eSpeak backend. Therefore, eSpeak NG must also be installed on the system.

#### macOS with Homebrew

```bash
brew install espeak-ng
```

#### Debian or Ubuntu

```bash
sudo apt-get update
sudo apt-get install espeak-ng
```

#### Fedora

```bash
sudo dnf install espeak-ng
```

#### Windows

Install the appropriate eSpeak NG `.msi` package for the system architecture. Most modern Windows computers should use the x64 installer.

### Configure the eSpeak NG library path when necessary

Phonemizer can normally locate the eSpeak NG shared library automatically. No additional configuration is needed when automatic detection succeeds.

If UniScriptIPA raises an error such as:

```text
RuntimeError: espeak not installed on your system
```

set the `PHONEMIZER_ESPEAK_LIBRARY` environment variable to the absolute path of the eSpeak NG shared library.

#### macOS with Homebrew

A portable Homebrew-based configuration is:

```bash
export PHONEMIZER_ESPEAK_LIBRARY="$(brew --prefix espeak-ng)/lib/libespeak-ng.dylib"
```

On an Apple Silicon Mac, the path may also be:

```bash
export PHONEMIZER_ESPEAK_LIBRARY=/opt/homebrew/lib/libespeak-ng.dylib
```

On an Intel Mac, it may be located under:

```text
/usr/local/lib/libespeak-ng.dylib
```

#### Linux

First locate the shared library:

```bash
ldconfig -p | grep libespeak-ng
```

Then set the environment variable using the absolute path reported by the system:

```bash
export PHONEMIZER_ESPEAK_LIBRARY=/absolute/path/to/libespeak-ng.so
```

#### Windows PowerShell

```powershell
$env:PHONEMIZER_ESPEAK_LIBRARY = "C:\Program Files\eSpeak NG\libespeak-ng.dll"
```

The exact path may differ depending on where eSpeak NG was installed.

### Configure the path from Python

The environment variable may also be set inside a Python program. It must be set **before importing UniScriptIPA**:

```python
import os

os.environ["PHONEMIZER_ESPEAK_LIBRARY"] = (
    "/opt/homebrew/lib/libespeak-ng.dylib"
)

from uniscriptipa import text_to_ipa
```

Alternatively, the library may be configured through Phonemizer directly:

```python
from phonemizer.backend.espeak.wrapper import EspeakWrapper

EspeakWrapper.set_library(
    "/absolute/path/to/libespeak-ng"
)

from uniscriptipa import text_to_ipa
```

## Quick Start

### Convert one string

```python
from uniscriptipa import text_to_ipa

ipa_text, token_notes = text_to_ipa(
    "en-us",
    "Latin",
    input_str_text=(
        "Hello, world! This is a simple test "
        "and let's try the string iPhone17Pro."
    ),
)

print(ipa_text)
print(token_notes)
```

The converted text preserves the punctuations and the digits in `iPhone17Pro`:

```text
həlo͡ʊ, wɜːld! ðɪs ɪz e͡ɪ sɪmpə͡l tɛst ænd lɛts tɹa͡ɪ ðə stɹɪŋ a͡ɪfo͡ʊn17pɹo͡ʊ.
```

The token-level annotations indicate how each output token was produced:

```python
[
    ("həlo͡ʊ", "ipa"),
    (",", ""),
    (" ", ""),
    ("wɜːld", "ipa"),
    ("!", ""),
    # ...
    ("a͡ɪfo͡ʊn", "ipa"),
    ("17", ""),
    ("pɹo͡ʊ", "ipa"),
    (".", ""),
]
```

### Convert multiple strings

```python
from uniscriptipa import texts_to_ipa

ipa_texts, token_notes = texts_to_ipa(
    "es",
    "Latin",
    input_list_texts=[
        "Buenos días.",
        "Me gusta escuchar música.",
        "¿Dónde está el restaurante?",
        "Tenemos dos perros.",
    ],
)

print(ipa_texts)
print(token_notes)
```

The first returned value is a list containing one converted string for each input string:

```python
[
    "bwenos dias.",
    "me ɡusta eskut͡ʃaɾ musika.",
    "¿donde esta el resta͡ʊɾante?",
    "tenemos dos peros.",
]
```

The second returned value contains the corresponding token-level annotations for every string:
```python
[
	[('bwenos', 'ipa'), (' ', ''), ('dias', 'ipa'), ('.', '')], 
	[('me', 'ipa'), (' ', ''), ('ɡusta', 'ipa'), (' ', ''), ('eskut͡ʃaɾ', 'ipa'), (' ', ''), ('musika', 'ipa'), ('.', '')], 
	[('¿', ''), ('donde', 'ipa'), (' ', ''), ('esta', 'ipa'), (' ', ''), ('el', 'ipa'), (' ', ''), ('resta͡ʊɾante', 'ipa'), ('?', '')], 
	[('tenemos', 'ipa'), (' ', ''), ('dos', 'ipa'), (' ', ''), ('peros', 'ipa'), ('.', '')]
]
```

## `tokenize()`

```python
tokenize(text)
```

Losslessly divides one input string into tokens.

Tokenization is performed in two stages. First, the text is segmented with ICU's word `BreakIterator` using the root locale, following Unicode word-boundary rules. Each token produced by ICU is then further subdivided whenever the character class changes between:

1. Unicode letters (`L*`), combining marks (`M*`), or characters with the Unicode `Quotation_Mark` property; and
2. all other Unicode characters.

Consecutive characters belonging to the same class remain in the same token. This additional subdivision separates letters from adjacent digits and other non-letter content. For example, `iPhone17Pro` is divided into `"iPhone"`, `"17"`, and `"Pro"`. The second stage only subdivides the tokens produced by ICU and never merges separate ICU tokens.

### Parameter

| Parameter | Type  | Description          |
| --------- | ----- | -------------------- |
| `text`    | `str` | The text to tokenize |

### Return value

```python
list[str]
```

Returns the input as a list of tokens.

For example:

```python
from uniscriptipa import tokenize

tokens = tokenize('This is iPhone17Pro')
```

Result:

```python
['This', ' ', 'is', ' ', 'iPhone', '17', 'Pro']
```

Tokenization does not delete, replace, normalize, or reorder any character. The following condition is always enforced:

```python
"".join(tokenize(text)) == text
```

If the tokens cannot be joined to reproduce the original input exactly, `tokenize()` raises a `RuntimeError`.

## `str_to_ipa()` / `text_to_ipa()`

```python
str_to_ipa(
    lang_code,
    script,
    *,
    input_str_text,
    norm_arab_revers_punc=True,
    njobs=16,
    with_stress=False,
    tie=True,
)
```

Converts one input string using the selected eSpeak language and Unicode script.

### Parameters

| Parameter               | Type   | Default  | Description                                                                                                      |
| ----------------------- | ------ | -------- | ---------------------------------------------------------------------------------------------------------------- |
| `lang_code`             | `str`  | Required | The language code passed to Phonemizer as its `language` parameter, such as `"en-us"`, `"es"`, `"ru"`, or `"ug"` |
| `script`                | `str`  | Required | The target Unicode script, such as `"Latin"`, `"Cyrillic"`, `"Arabic"`, or `"Greek"`                             |
| `input_str_text`        | `str`  | Required | The single input string to convert                                                                               |
| `norm_arab_revers_punc` | `bool` | `True`   | Whether eligible Arabic or reversed punctuation characters should be normalized through Uroman                   |
| `njobs`                 | `int`  | `16`     | The number of parallel jobs passed to Phonemizer                                                                 |
| `with_stress`           | `bool` | `False`  | Whether Phonemizer should include stress marks in its output                                                     |
| `tie`                   | `bool` | `True`   | Whether Phonemizer should use tie characters for multi-character phones                                          |

`input_str_text`, `norm_arab_revers_punc`, `njobs`, `with_stress`, and `tie` are keyword-only and must therefore be written explicitly:

```python
str_to_ipa(
    "en-us",
    "Latin",
    input_str_text="Hello, world!",
)
```

### Return value

```python
tuple[
    str,
    list[tuple[str, str]],
]
```

The function returns two values:

```python
ipa_text, token_notes = str_to_ipa(...)
```

#### 1. `ipa_text`

A string reconstructed from all converted and preserved tokens in their original order.

#### 2. `token_notes`

A list of two-item tuples:

```python
(output_token, conversion_type)
```

The conversion type has one of three values:

| Value      | Meaning                                                           |
| ---------- | ----------------------------------------------------------------- |
| `"ipa"`    | The token was replaced with the IPA output produced by Phonemizer |
| `"uroman"` | The punctuation token was normalized by Uroman                    |
| `""`       | The token was preserved without IPA or Uroman conversion          |

For example:

```python
[
    ("həlo͡ʊ", "ipa"),
    (",", ""),
    (" ", ""),
    ("wɜːld", "ipa"),
    ("!", ""),
]
```

The first item in each tuple is the token appearing in the final output, not necessarily the original input token.

## `list_to_ipa()` / `texts_to_ipa()`

```python
list_to_ipa(
    lang_code,
    script,
    *,
    input_list_texts,
    norm_arab_revers_punc=True,
    njobs=16,
    with_stress=False,
    tie=True,
)
```

Converts multiple input strings using one shared Phonemizer call over the unique tokens collected from the complete input list.

This function is intended for batch processing and large text corpora.

### Parameters

| Parameter               | Type        | Default  | Description                                                                                    |
| ----------------------- | ----------- | -------- | ---------------------------------------------------------------------------------------------- |
| `lang_code`             | `str`       | Required | The language code passed to Phonemizer as its `language` parameter                             |
| `script`                | `str`       | Required | The target Unicode script                                                                      |
| `input_list_texts`      | `list[str]` | Required | A list of input strings to convert                                                             |
| `norm_arab_revers_punc` | `bool`      | `True`   | Whether eligible Arabic or reversed punctuation characters should be normalized through Uroman |
| `njobs`                 | `int`       | `16`     | The number of parallel jobs passed to Phonemizer                                               |
| `with_stress`           | `bool`      | `False`  | Whether Phonemizer should include stress marks                                                 |
| `tie`                   | `bool`      | `True`   | Whether Phonemizer should use tie characters for multi-character phones                        |

`input_list_texts`, `norm_arab_revers_punc`, `njobs`, `with_stress`, and `tie` are keyword-only:

```python
list_to_ipa(
    "es",
    "Latin",
    input_list_texts=[
        "Buenos días.",
        "Tenemos dos perros.",
    ],
)
```

### Return value

```python
tuple[
    list[str],
    list[list[tuple[str, str]]],
]
```

The function returns two values:

```python
ipa_texts, token_notes = list_to_ipa(...)
```

#### 1. `ipa_texts`

A list of converted strings.

The output list preserves the order of `input_list_texts`, and each input string produces exactly one output string.

```python
[
    "bwenos dias.",
    "tenemos dos peros.",
]
```

#### 2. `token_notes`

A nested list containing one token-annotation list for each converted string:

```python
[
    [
        ("bwenos", "ipa"),
        (" ", ""),
        ("dias", "ipa"),
        (".", ""),
    ],
    [
        ("tenemos", "ipa"),
        (" ", ""),
        ("dos", "ipa"),
        (" ", ""),
        ("peros", "ipa"),
        (".", ""),
    ],
]
```

The outer list corresponds to the input strings. Each inner list records how the output tokens of that string were produced.

## How Tokens Are Selected for IPA Conversion

A token (tokenized by tokenizer()) is replaced with Phonemizer's IPA result only when both of the following conditions are satisfied:

1. It contains at least one character that is a Unicode letter, or combining mark, or quotation-mark character.
2. It contains at least one character belonging to the Unicode script specified by `script`.

When a token satisfies these conditions, the entire token is replaced with the IPA result.

This script-based selection provides an approximate way to restrict IPA transliteration to words written in the expected form of the target language. It is intentionally not a strict language-identification system.

For example:

```python
text_to_ipa(
    "en-us",
    "Latin",
    input_str_text="iPhone17Pro",
)
```

The input is first divided into:

```python
["iPhone", "17", "Pro"]
```

The Latin-script tokens are converted, while the number is preserved:

```text
a͡ɪfo͡ʊn17pɹo͡ʊ
```

The `script` parameter selects which writing system is eligible for conversion. The `lang_code` parameter selects which eSpeak language interprets the eligible tokens.

This is script-based selection, not strict language identification. For example, Spanish and English words both use the Latin script and may therefore both be eligible when `script="Latin"`.

## Parallel  Processing

The `njobs` parameter is forwarded directly to Phonemizer:

```python
texts_to_ipa(
    "es",
    "Latin",
    input_list_texts=my_large_corpus,
    njobs=16,
)
```

It controls how many parallel jobs are used during phonemization. Parallel processing can substantially accelerate conversion when processing large corpora containing many unique tokens.

## Motivation

Recent studies suggest that converting multilingual orthographies into the International Phonetic Alphabet (IPA) as a shared phonemic representation can benefit natural language processing, including cross-lingual transfer, low-resource named entity recognition, multilingual tokenization, multilingual cognitive modeling, and phoneme-level language modeling (Imperial, 2022; Jung et al., 2024; Sohn et al., 2024; Goriely & Buttery, 2025; Miletić et al., 2026).

Grapheme-to-phoneme conversion is therefore useful not only for speech processing, but also as a preprocessing strategy for multilingual and low-resource NLP.

## Why UniScriptIPA?

When converting multilingual corpora into a shared IPA representation, we may want the conversion to behave like **selective machine transliteration**, rather than complete text-to-speech normalization.

In such a workflow, only text associated with the target language and grapheme should be converted to IPA. Other parts of the source text—such as punctuation, digits, whitespace, symbols, and obvious foreign words written in different writing systems—may carry useful information and should remain in their original positions.

Existing G2P systems make different trade-offs in this respect.

[Epitran] uses language-specific mapping tables together with preprocessing and postprocessing rules. When an input character cannot be matched by its mapping table, Epitran preserves that character in the output rather than discarding it. This produces useful transliteration-like behavior in which mapped orthographic content is converted while unmapped information is retained.

However, Epitran provides fewer built-in language–script configurations than eSpeak NG. The eSpeak NG backend used by Phonemizer supports a considerably broader range of languages, including languages such as Belarusian, Icelandic, and Latin.

Direct use of the Phonemizer eSpeak backend, however, is not always ideal for preservation-oriented corpus transliteration. eSpeak NG is primarily a text-to-speech system, so it attempts to assign pronunciations to the complete input. As a result:

* digits may be expanded into their spoken forms;
* foreign or mixed-language words may receive unexpected pronunciations across different target language settings;
* punctuation preservation may depend on the backend configuration and the configured punctuation inventory.

**UniScriptIPA** provides a selective, structure-preserving layer on top of Phonemizer's eSpeak backend. It uses eSpeak NG for its broad language coverage, while restricting phonemization to tokens in the target language's form containing characters from a user-specified target language's Unicode script.

For each token:

* if it contains at least one character belonging to the target language's Unicode script, the entire token is passed to the selected eSpeak language;
* otherwise, the token is copied to the output without phonemization.

This makes UniScriptIPA suitable for producing IPA-based multilingual NLP corpora while retaining non-target content and the overall structure of the original text.

## Lossless Tokenization

UniScriptIPA applies a two-stage tokenization procedure before selective phonemization.

### Stage 1: ICU word-boundary segmentation

The input is first segmented with ICU's word `BreakIterator` using the root locale. ICU word-boundary analysis is based on Unicode Standard Annex #29, *Unicode Text Segmentation*.

Under the default UAX #29 word-boundary rules, adjacent letters and numbers are not necessarily separated. For example, strings such as:

```text
iPhone17Pro
Year2027
```

may initially be treated as single word tokens.

Passing such a token directly to eSpeak NG could cause the complete token—including its digits—to be converted into a pronunciation.

### Stage 2: Unicode-category subdivision

Each token produced by ICU is therefore subdivided into two character classes.

**Class A**

* Unicode General Category `Letter` (`L*`)
* Unicode General Category `Mark` (`M*`)
* Unicode binary property `Quotation_Mark=Yes`

**Class B**

* every other Unicode character

A new boundary is inserted whenever the character class changes within an ICU token. Consecutive characters belonging to the same class remain together.

For example:

```text
iPhone17Pro
```

is segmented as:

```python
["iPhone", "17", "Pro"]
```

and:

```text
Year2027
```

is segmented as:

```python
["Year", "2027"]
```

The second stage only subdivides tokens produced by ICU; it never merges separate ICU tokens.

Tokenization is fully lossless: no character is deleted, replaced, normalized, or reordered. The following invariant is always enforced:

```python
"".join(tokenize(text)) == text
```

If reconstruction does not exactly reproduce the original input, `tokenize()` raises a `RuntimeError`.

## Selective IPA Conversion

After tokenization, UniScriptIPA collects all unique tokens from the input text.

For `str_to_ipa()`, the input text is provided through the `input_str_text` parameter. For `list_to_ipa()`, the list of multiple input texts is provided through the `input_list_texts` parameter, and unique tokens are collected across the entire input.

These unique tokens are passed to Phonemizer's `phonemize()` function using the eSpeak backend. The following UniScriptIPA parameters in str_to_ipa() and list_to_ipa() are forwarded to Phonemizer:

| UniScriptIPA parameter | Phonemizer parameter | Purpose                                                                |
| ---------------------- | -------------------- | ---------------------------------------------------------------------- |
| `lang_code`            | `language`           | Selects the eSpeak language                                            |
| `with_stress`          | `with_stress`        | Controls whether stress marks are included                             |
| `tie`                  | `tie`                | Controls whether multi-character phones are joined with tie characters |
| `njobs`                | `njobs`              | Controls the number of parallel phonemization jobs                     |

The `njobs` parameter allows Phonemizer to process tokens in parallel. Increasing this value can considerably accelerate the conversion of large multilingual corpora.

## Script-Based Token Selection

Although all unique tokens are passed through Phonemizer, UniScriptIPA only inserts the resulting IPA transliteration into the final output when a token satisfies both of the following conditions:

1. The token contains at least one character that is a Unicode letter, or combining mark, or character with the Unicode `Quotation_Mark` property.
2. The token contains at least one character belonging to the Unicode script specified by the user through the `script` parameter.

Examples of valid script names include:

```text
"Latin"
"Cyrillic"
"Arabic"
"Greek"
```

When both conditions are satisfied, the complete token is replaced with the IPA result returned by Phonemizer. 

This script-based selection provides an approximate way to restrict IPA transliteration to words written in the expected form of the target language. It is intentionally not a strict language-identification system.

For example, when using:

```python
lang_code="en-gb"
script="Latin"
```

a Spanish word such as:

```text
Cuándo
```

will also be replaced by the IPA result returned by Phonemizer because its characters belong to the Unicode Latin script.

Similarly, when using:

```python
lang_code="ru"
script="Cyrillic"
```

Kazakh words written in Cyrillic will also be replaced by the IPA result returned by Phonemizer.

The `script` parameter therefore controls the writing system that is eligible for transliteration, while `lang_code` controls the eSpeak language used to interpret eligible tokens.

## Optional Arabic and Reversed Punctuation Normalization

Tokens that do not satisfy the IPA-conversion conditions are normally preserved unchanged.

An optional normalization step is available through the `norm_arab_revers_punc` parameter, which is enabled by default:

```python
norm_arab_revers_punc=True
```

When this option is enabled, a non-phonemized token is passed to Uroman for punctuation normalization only when all of the following conditions are satisfied:

1. The token contains exactly one character.
2. The character belongs to a Unicode punctuation category (`P*`).
3. The Unicode character name contains either `ARABIC` or `REVERSED`.

Such characters are converted by Uroman into their corresponding ASCII-style punctuation forms.

When the option is disabled:

```python
norm_arab_revers_punc=False
```

Arabic and reversed punctuation characters are preserved exactly as they appear in the input.

## Lossless Reconstruction

After selective IPA replacement and optional punctuation normalization, all tokens are joined in their original order.

Every token that does not satisfy an enabled conversion rule is preserved without deletion, reordering, or modification. This retains the structure of the original text, including whitespace, digits, symbols, non-target scripts, and other unconverted content.

## Current Scope and Future Work

UniScriptIPA was initially designed primarily with languages from the Indo-European, Turkic, and Austronesian language families in mind. Languages of the Sinosphere and the Mainland Southeast Asia linguistic area may also be supported in principle, but they have not yet been systematically tested and were not the primary focus of the current version. In particular, mixed-script Japanese and Korean text—such as Japanese text combining Kanji and Kana, or Korean text combining Hangul and Hanja—can be difficult to handle effectively because it involves multiple Unicode scripts. We will continue exploring selective IPA transcription for languages of the Sinosphere and Mainland Southeast Asia, either through future UniScriptIPA updates or through separate libraries. Follow us to receive the latest feature updates and announcements about new libraries. ✨

## References

* Imperial, J. M. (2022). *NU HLT at CMCL 2022 Shared Task: Multilingual and Crosslingual Prediction of Human Reading Behavior in Universal Language Space*. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, 108–113. Association for Computational Linguistics.

* Jung, H., Oh, C., Kang, J., Sohn, J., Song, K., Kim, J., & Mortensen, D. R. (2024). *Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer*. In *Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024)*, 200–211. Association for Computational Linguistics.

* Sohn, J., Jung, H., Cheng, A., Kang, J., Du, Y., & Mortensen, D. R. (2024). *Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource Languages*. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 13595–13602. Association for Computational Linguistics.

* Goriely, Z., & Buttery, P. (2025). *IPA CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling*. In *Proceedings of the 29th Conference on Computational Natural Language Learning*, 502–521. Association for Computational Linguistics.

* Miletić, M., Kallini, J., & Shutova, E. (2026). *Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet*. In *Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, 40323–40349. Association for Computational Linguistics.

## License

UniScriptIPA is licensed under the GNU General Public License
version 3 or any later version.

See the [LICENSE](LICENSE) file for details.
