Metadata-Version: 2.4
Name: g2pflow
Version: 0.3.0
Summary: Composable multilingual grapheme-to-phoneme pipelines
Author: Team OpenVPI
License-Expression: MIT AND Apache-2.0 AND CC-BY-SA-3.0 AND CC-BY-SA-4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: licenses/cpp-kana-LICENSE.txt
License-File: licenses/cpp-pinyin-LICENSE.txt
License-File: src/g2pflow/converters/cpp_pinyin/dicts/cantonese/License.txt
License-File: src/g2pflow/converters/cpp_pinyin/dicts/mandarin/License.txt
Requires-Dist: pydantic<3,>=2.11
Provides-Extra: ja
Requires-Dist: fugashi<2,>=1.3; extra == "ja"
Requires-Dist: unidic<2,>=1.1; extra == "ja"
Requires-Dist: filelock; extra == "ja"
Provides-Extra: lstm
Requires-Dist: numpy; extra == "lstm"
Requires-Dist: onnxruntime; extra == "lstm"
Provides-Extra: all
Requires-Dist: fugashi<2,>=1.3; extra == "all"
Requires-Dist: unidic<2,>=1.1; extra == "all"
Requires-Dist: filelock; extra == "all"
Requires-Dist: numpy; extra == "all"
Requires-Dist: onnxruntime; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: numpy; extra == "dev"
Dynamic: license-file

# g2pflow

Composable multilingual grapheme-to-phoneme pipelines for Python 3.10+.

g2pflow preserves alternative readings, complete pronunciation paths and the
pronunciation groups within each path. Converters control word boundaries and
can be combined in priority order for mixed-language text.

## Install

```shell
pip install g2pflow
pip install "g2pflow[ja]"    # MeCab and UniDic
pip install "g2pflow[lstm]"  # ONNX inference for LSTM converters
pip install "g2pflow[all]"   # Both optional backends
```

These are the intended PyPI installation commands. Before publication, install
this checkout with `pip install -e .` (or `pip install -e ".[all,dev]"`).

The base package depends only on Pydantic. NumPy and ONNX Runtime are loaded for
LSTM OOV inference; filelock is loaded only when UniDic needs downloading.
The `lstm` and `ja` extras install these respective optional dependencies.

## Quick start

This example needs no downloaded dictionaries or models:

```python
from g2pflow import G2PPipelineConfig, build_pipeline_from_config

config = G2PPipelineConfig.model_validate({
    "preprocessors": [{"id": "lowercase"}],
    "converters": [{
        "id": "characters",
        "language": "en",
        "kwargs": {"mapping": {"h": ["HH"], "i": ["AY"]}},
    }],
})
pipeline = build_pipeline_from_config(config)
word = pipeline.convert("Hi", languages=["en"])[0]
assert word.text == "hi"
assert word.readings[0].paths[0][0].phonemes == ["HH", "AY"]
```

`G2PPipeline` also accepts preprocessor and converter instances directly.
Configuration is validated by independent Pydantic models; there is no YAML
loader, training configuration system or command-line interface.

## PFML markup

PFML (Pronunciation Flow Markup Language) adds language scopes, fixed word
boundaries and complete pronunciation candidate trees to ordinary text:

```python
from g2pflow import G2PPipeline, to_pfml

# Direct phonemes need no dictionaries, models or converters.
words = G2PPipeline().convert_pfml(
    '<scope language="ja"><word script="ka" phonemes="k a">か</word></scope>'
)
assert words[0].readings[0].paths[0][0].phonemes == ["k", "a"]
assert G2PPipeline().convert_pfml(to_pfml(words)) == words
```

`<scope language="ja">東京へ行く</scope>` routes a whole text range to Japanese
G2P; `<word language="zh">重庆</word>` fixes one output word. Automatic text
requires a pipeline configured for the corresponding language. Direct results
must provide all final phonemes; they preserve multiple readings, paths and
group boundaries without invoking G2P or filtering candidates. Missing outer
containers can be omitted, down to `<phoneme>ong</phoneme>`.

There is no required root tag. `convert()` still handles plain text, while
`convert_pfml()` handles XML fragments. Python outputs remain `G2PWord` objects;
`to_pfml()` serializes successful outputs losslessly, including all pronunciation
candidates and language markers. See the [PFML 1.0 specification and API](docs/pfml.md).

## Output

- `G2PWord(text, language, readings)` is a converter-defined semantic word.
- `G2PReading(paths)` groups the complete realizations of one reading.
- `G2PPath` is an ordered `list[G2PGroup]`.
- `G2PGroup(script, phonemes)` associates a pronunciation script with phones.

For example, a reading of Japanese `猫` may contain a path with groups
`("ne", ["n", "e"])` and `("ko", ["k", "o"])`. Different paths can have
different group boundaries. The framework filters silent groups and paths,
omits words left without pronunciation, and fills missing display labels.
Missing readings or path candidates are conversion failures. The package does
not select candidates against audio or encode phones into model vocabulary IDs.

## Components and resources

| Converter ID | Purpose | Caller-supplied resources |
| --- | --- | --- |
| `dictionary` | Word-to-phoneme dictionary | `dict_path` |
| `characters` | Character-to-phoneme mapping | `mapping` |
| `passthrough` | Return each token as its own phone | None |
| `chinese-pinyin` | Mandarin text through pinyin | Script-to-phone `dict_path` |
| `yue-jyutping` | Cantonese text through jyutping | Script-to-phone `dict_path` |
| `japanese-kana` | Kana through romaji | Script-to-phone `dict_path` |
| `japanese-mecab` | Japanese word segmentation and readings | Script-to-phone `dict_path`; optional `unidic_dir` |
| `lstm` | Dictionary lookup and ONNX OOV inference | `model_path`; optional `dict_path` |

Preprocessors: `filter-punctuation`, `strip-whitespace`, `lowercase`, and
`remove-accents`.

Mandarin and Cantonese character/phrase reading data are bundled. The caller
chooses the phoneme inventory through external pronunciation dictionaries.
English ONNX models are external. The Japanese backend manages UniDic's
first-use download and can also use a preinstalled directory. See
[resources and converter details](docs/usage.md).

## External plugins

```python
from g2pflow import register_plugin_paths

register_plugin_paths("./my_plugins", "../shared_plugins")
```

Each directory may contain both preprocessors and converters, registered with
`@preprocessor` and `@converter`. Registration scans immediately. Call
`discover_plugins()` to find newly added modules; existing modules load once
per process. Paths are supplied only through Python, not through configuration.
See [plugin discovery and multiprocessing](docs/plugins.md).

## Development

```shell
pip install -e ".[dev]"
pytest --cov=g2pflow --cov-report=term-missing
python -m build
python -m twine check dist/*
```

Default tests run offline with small fixtures and controlled backend doubles.
Real-resource tests and clean-install distribution tests are explicit opt-ins;
see [development and validation](docs/development.md).

## Licensing

Project code uses the [MIT license](LICENSE). Redistributed code and
data retain their respective licenses; the distribution's license expression
also includes Apache-2.0, CC-BY-SA-3.0 and CC-BY-SA-4.0.
See [third-party notices](docs/third-party.md).
