Metadata-Version: 2.4
Name: spokenform
Version: 0.4.1
Summary: Single-language written-to-spoken text normalization for speech applications
Author-email: Holger Nahrstaedt <nahrstaedt@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/buchwandler/spokenform
Project-URL: Repository, https://github.com/buchwandler/spokenform
Project-URL: Issues, https://github.com/buchwandler/spokenform/issues
Project-URL: Changelog, https://github.com/buchwandler/spokenform/blob/main/docs/changelog.md
Keywords: normalization,nlp,speech,text-to-speech,tts,verbalization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: abbr2words<0.3.0,>=0.2.13
Requires-Dist: cn2an<0.6,>=0.5.24
Requires-Dist: numeralform<0.2.0,>=0.1.4
Provides-Extra: lexhint
Requires-Dist: lexhint<0.3.0,>=0.1.2; extra == "lexhint"
Provides-Extra: spacy
Requires-Dist: spacy<4,>=3.7; extra == "spacy"
Provides-Extra: dev
Requires-Dist: build>=1; extra == "dev"
Requires-Dist: mypy>=1.11; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff<0.17,>=0.16; extra == "dev"
Requires-Dist: tomli>=2.0.1; python_version < "3.11" and extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Provides-Extra: all
Requires-Dist: spacy<4,>=3.7; extra == "all"
Requires-Dist: lexhint<0.3.0,>=0.1.2; extra == "all"
Dynamic: license-file

[![PyPI - Version](https://img.shields.io/pypi/v/spokenform)](https://pypi.org/project/spokenform/)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/spokenform)
![PyPI - Downloads](https://img.shields.io/pypi/dm/spokenform)
[![codecov](https://codecov.io/gh/buchwandler/spokenform/graph/badge.svg?token=FeOQeR94qo)](https://codecov.io/gh/buchwandler/spokenform)
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/buchwandler/spokenform/HEAD?urlpath=lab/tree/notebooks/spokenform_playground.ipynb)

# spokenform

`spokenform` converts plain written text in one selected language into reviewable
text intended for speech systems. It is a text-to-text frontend: written text in,
spoken form out.

The package provides:

- context-aware abbreviation and source-aligned numeric-unit expansion through `abbr2words`;
- a locale-aware structured-value stage for quantities, dates, times, currencies,
  temperatures, labels, and contextual ordinals;
- locale-policy-wrapped number, date, time, currency, decimal, and ordinal verbalization;
- optional provider-neutral spaCy annotations for POS-aware abbreviation rules;
- stage-level provenance through `PreparedText`;
- composed input-to-output offset maps with left/right boundary bias;
- caller-defined and automatically discovered protected ranges;
- conservative protection for URLs, email addresses, and arbitrary multi-dot
  semantic-version/ID sequences.

High-confidence URL, e-mail, semantic-version, and contextual Roman rendering
is available explicitly with `normalize_literals=True`; caller-protected spans
remain absolute. Structured fractions, identifiers, operator-shaped math,
music-context tokens, and controlled biological names use semantic precedence
and source-aligned mappings.

It intentionally does **not** detect languages, parse or render SSMD, segment mixed
languages, generate phonemes, or depend on `kokorog2p` or `piperg2p`.

## Installation

```bash
python -m pip install spokenform
```

For optional spaCy integration:

```bash
python -m pip install "spokenform[spacy]"
python -m spacy download de_core_news_sm
```

spaCy and its trained pipelines are separate packages. `spokenform` never downloads
a model automatically.

## Quickstart

````python
from spokenform import prepare

prepared = prepare(
    "Prof. Klein bringt am 14.05.2026 um 18:20 Uhr 2 kg mit.",
    language="de",
)

print(prepared.spoken_text)
print(prepared.render_changes())

## Language support

The exhaustive runtime language matrix is maintained in [`docs/language-coverage.md`](docs/language-coverage.md), with architecture notes in [`docs/languages.md`](docs/languages.md). Spokenform accepts 49 base families and 17 exact locale overlays. Abbreviation routing follows `abbr2words`; released number backends, reviewed punctuation, structured semantics, sequence policies, and KokoroG2P support are independent capabilities.

Swedish is available as `sv`, `sv-SE`, or `sv_SE`; `swe` and `swe-SE` are compatibility aliases.

```python
result = prepare(
    "Vi har t.ex. 2 kg kvar.",
    language="sv",
)
````

Swedish uses comma decimals, space/NBSP/NNBSP grouping, reviewed quantities and temperatures, and Swedish krona amounts. Dates, digital times, arbitrary initialisms, and unreviewed specialist sequence domains remain caller-managed or fail closed.

Russian (`ru`, `ru-RU`, or `ru_RU`) also accepts `rus` as a compatibility alias. It uses comma decimals, space/NBSP/NNBSP grouping, digitwise fractions, and reviewed canonical quantities with explicit numeral government.

```python
result = prepare(
    "Путь составляет 22 км.",
    language="ru",
    use_spacy=False,
 )
print(result.spoken_text)
```

This produces `Путь составляет двадцать два километра.` Dates, digital times, specialist sequences, and currency are caller-managed or fail closed; RUB remains caller-managed until `abbr2words` provides a reviewed identity.

Vietnamese (`vi`, `vi-VN`, or `vi_VN`) uses comma decimals, dot or space-family grouping, exact digitwise fractional speech, reviewed quantities and temperatures, VND/`₫`, and guarded `abbr2words` abbreviations. Dates, digital times, ordinals, arbitrary initialisms, and unreviewed specialist domains remain caller-managed or fail closed.

```python
result = prepare(
    "TP. Hà Nội có 2 kg hàng với giá 1000 VND.",
    language="vi",
)
```

> > > result.spoken_text
> > > 'thành phố Hà Nội có hai kilôgam hàng với giá một nghìn đồng Việt Nam.'
> > > The result contains:

- `source_text`: unchanged caller input;
- `clean_text`: plain text used by the normalization pipeline;
- `spoken_text`: readable normalized output;
- `language`: normalized processing-language code;
- ordered stages and mapped edits, including semantic rule and evidence metadata;
- composed `offset_map` and source-coordinate replacements;
- structured warnings;

`PreparedText.text` is an alias for `spoken_text`.

For new application code, use the strict language-explicit entry point:

```python
from spokenform import prepare_language

prepared = prepare_language("2 kg", language="de")
```

`prepare()` keeps its `language="en"` default for compatibility. Both APIs process
one language run only. Language detection, mixed-language segmentation, and markup
parsing remain caller responsibilities. `PreparationConfig.for_speech(language)`
is the generic TTS-neutral preset; `for_kokorog2p()` and `for_piperg2p()` are
downstream integration conveniences for their respective adapters.

## Thai runtime support

Thai (`th`, `th-TH`, or `th_TH`) uses point decimals, comma or space-family grouping, accepts Latin and Thai digits, and provides reviewed quantities, temperatures, and THB/`฿` amounts through `abbr2words`.

```python
prepare("ระยะ 5 กม.", language="th").spoken_text
# "ระยะ ห้า กิโลเมตร"
```

Thai date, era, and digital-time bodies remain caller-managed in this release. Ordinals, ranges, specialist sequences, and unsupported punctuation semantics fail closed rather than borrowing English vocabulary.

## kokorog2p adapter

Use `prepare_for_kokorog2p(text, language=...)` for one explicitly selected
language run. The adapter preserves caller-owned run whitespace and protected
overrides, emits exact source-coordinate replacements, and leaves tokenization,
G2P, phonemization, and model punctuation to kokorog2p. German, French, Spanish,
Italian, Portuguese, Czech, and English are parity-gated semantic migration
targets. English is active on the kokorog2p spokenform adapter for reviewed
structured semantics, contextual single-dot release labels, and safe
ordinary-number categories; phoneme-sensitive years, suffix ordinals, Roman
numerals, phone/ID and arbitrary multi-dot sequences, numeric suffixes, and G2P
decisions remain downstream in kokorog2p.

German quantity and currency symbols are recognized by `abbr2words.iter_unit_matches()`
or the reviewed symbol identities, while Spokenform owns the canonical German grammar
that realizes them. German now owns validated dates and times, conservative contextual
year speech via `render_year()`, reviewed Euro major/minor realization with explicit
Cent labels, exact excess-fraction preservation, and full lexical-boundary validation
against identifier adjacency. The reviewed `vgl.`, `i.d.R.`, `o.ä.`, and `u.U.` lexical
abbreviations are supplied by `abbr2words`, not duplicated in Spokenform. German
currencies without reviewed minor grammar use a safe exact decimal fallback or fail
closed. French likewise realizes canonical `abbr2words` quantity and currency
identities, including French dates, times, ordinals, decimal digits, plural
grammar, temperatures, and major/minor currency units. Spanish realizes
canonical quantities, temperatures, currencies, dates, and ordinary numbers;
Spanish `18:20`-style time expressions remain caller-managed. Italian realizes
reviewed dates, quantities, temperatures, currencies, and ordinary numbers;
Italian colon times remain caller-managed. Portuguese realizes reviewed dates,
quantities, temperatures, currencies, and ordinary numbers; Portuguese colon
times remain caller-managed. Czech realizes reviewed dates, ordinary numbers,
quantities, temperatures, currencies, and canonical extended units; Czech colon
times remain caller-managed. No locale copies raw symbol inventories or
downstream tokenizer/phoneme rules.
Swedish realizes comma-decimal numbers, reviewed quantities, temperatures, and Swedish krona amounts from canonical `abbr2words` identities. Swedish dates, digital times, arbitrary initialisms, and unreviewed specialist domains remain caller-managed or fail closed. No locale may borrow English fallback vocabulary for a supported language.

## piperg2p adapter

Use `prepare_for_piperg2p(text, language=...)` before passing prepared text to PiperG2P:

```python
from piperg2p import phonemize_prepared
from spokenform import prepare_for_piperg2p

prepared = prepare_for_piperg2p(
    "Pay $12.50 for 2 kg.",
    language="en",
)
result = phonemize_prepared(
    prepared.spoken_text,
    language="en-us",
    config="voice.onnx.json",
)
```

Spokenform owns written-to-spoken semantic normalization, source replacements, and offset mapping. PiperG2P owns voice configuration, tokenization, phonemization, phoneme IDs, lexicon overlays, raw Piper and eSpeak phoneme blocks, and backend compatibility. Spokenform does not load Piper models, interpret voice configs, generate phonemes, or require PiperG2P as a dependency.

The semantic language and selected Piper voice identifier are separate explicit choices. Caller-owned `[[...]]` blocks must be discovered by PiperG2P and passed to Spokenform as protected spans before preparation. Map source spans with `PreparedText.map_source_span()` before creating downstream overrides. Do not transfer source POS, tag, or lemma metadata across semantic replacements without reanalyzing the prepared text.

## Language boundary

Each call processes one language. Production callers should always pass
`language=...`; English remains the API default for compatibility and simple CLI
usage.

Language detection and mixed-language handling belong in the orchestration or G2P
layer. A foreign word may remain unchanged through normalization and be handled
afterward. Existing source spans can be transferred with `prepared.offset_map`.

Markup must also be parsed outside this package. Pass plain text to `prepare()` and
use `ProtectedSpan` for ranges generic normalization must not change.

## Configuration

```python
from spokenform import PreparationConfig, prepare

config = PreparationConfig(
    language="en",
    expand_abbreviations=True,
    expand_structured=True,
    expand_numbers=True,
    normalize_whitespace=True,
    context=True,
)

prepared = prepare("The board is 2 in. wide.", config=config)
```

When a `PreparationConfig` is supplied, it is authoritative for pipeline options.

### Residual symbols and acronym case

Residual punctuation and symbols are unchanged by default:

```python
PreparationConfig(language="en", symbol_mode="none")
```

Use `symbol_mode="remove"` to remove all residual Unicode punctuation and
symbols, or use an exact-codepoint allowlist with `symbol_mode="keep"`:

```python
PreparationConfig(language="en", symbol_mode="remove")
PreparationConfig(language="en", symbol_mode="keep", keep_symbols=":;,()-,.")
```

The filter runs after semantic recognition and does not modify protected spans.
For generic uppercase acronyms, `generic_acronym_case="lower"` renders `ABC`
as `a b c`; the default and `"upper"` render it as `A B C`. Lexical acronyms,
preserved terms, and known initialisms retain their existing policies.
The [`API policy reference`](docs/api.md#configuration-policy-modes) explains
the `generic_acronym_mode`, `registered_acronym_mode`, and `long_number_mode`
choices, including their false-positive tradeoffs.

## spaCy support

spaCy supplies POS annotations for abbreviation rules that opt into POS guards. The public normalization API remains provider-neutral.

`abbr2words` accepts POS annotations, but its bundled registries do not necessarily
require POS labels. Therefore installing spaCy alone may not change default
normalization output. The integration is usable for custom POS-guarded entries.

Load and inject a pipeline in the application:

```python
import spacy
from spokenform import prepare

nlp = spacy.load("en_core_web_sm")
prepared = prepare(
    "The board is 2 in. wide.",
    language="en",
    nlp=nlp,
)
```

Or ask `spokenform` to load an already installed model:

```python
prepared = prepare(
    "The board is 2 in. wide.",
    language="en",
    spacy_model="en_core_web_sm",
    strict=True,
)
```

Model names and paths are passed to `spacy.load()`. Loaded models are cached by
language/model key. `reset_spacy_cache()` clears that cache.

The adapter reads the token attributes `text`, `idx`, `pos_`, `tag_`, `lemma_`, and `lang_`. `lang_` is carried as provider metadata; it is not used as language detection. Annotation spans are validated against the exact input text and remapped when protected ranges are replaced by internal sentinels.
A trained pipeline with POS or morphological annotations is required for quality
improvement; `spacy.blank(...)` supplies tokenization but normally no useful POS
tags.

Explicit `annotations` take precedence over `nlp` and `spacy_model`.

## Protection

Use `ProtectedSpan(start, end)` or a `(start, end)` tuple to protect a source range:

```python
from spokenform import ProtectedSpan, prepare

text = "Keep Dr. literal, but verbalize 12."
start = text.index("Dr.")
prepared = prepare(
    text,
    language="en",
    protected_spans=[ProtectedSpan(start, start + 3)],
)
```

Invalid or overlapping ranges warn by default and raise `ProtectionError` with
`strict=True`. URLs, email addresses, and semantic versions are protected
automatically.

## Offset mapping

```python
from spokenform import prepare

source = "Prof. Klein has 2 kg."
prepared = prepare(source, language="de")

start = source.index("Prof.")
end = start + len("Prof.")
spoken_start, spoken_end = prepared.offset_map.map_source_span(start, end)

print(prepared.spoken_text[spoken_start:spoken_end])
```

Use `bias="left"` or `bias="right"` when mapping an individual boundary at an
expansion.

## CLI

```bash
spokenform --lang de "Prof. Klein hat 2 kg."
spokenform --lang de --changes "Prof. Klein hat 2 kg."
spokenform --lang de --json "Prof. Klein hat 2 kg."
spokenform --lang en --spacy-model en_core_web_sm --strict "The board is 2 in. wide."
echo "The value is 2." | spokenform --lang en
```

## Examples

Executable examples are in [`examples/`](examples/README.md):

```bash
python examples/basic.py
python examples/german.py
python examples/german.py --spacy-model de_core_news_sm
python examples/protected_text.py
python examples/offset_mapping.py
```

## Interactive notebook

Try `spokenform` in your browser without installing it locally:

[Launch the spokenform playground on Binder](https://mybinder.org/v2/gh/buchwandler/spokenform/HEAD?urlpath=lab/tree/notebooks/spokenform_playground.ipynb)

The Binder notebook runs the selected repository revision and includes interactive
controls for language, semantic stages, residual-symbol handling, and generic
acronym casing. Changes made in a Binder session are temporary.

## Documentation

Documentation sources use MyST Markdown. No reStructuredText source files are
required.

```bash
python -m pip install -e .
python -m pip install -r docs/requirements.txt
sphinx-build -W -b html docs docs/_build/html
```

## Reusable speech profiles

Use `SpeechProfile` for an isolated, reusable domain glossary. Profile entries can expand to their long form, spell the source as letters, or use a deterministic custom pronunciation:

```python
from spokenform import GlossaryEntry, SpeechProfile, prepare_language

profile = SpeechProfile(
    name="operations",
    language="en",
    glossary=(
        GlossaryEntry("AAR", "after-action review"),
        GlossaryEntry("AO", "area of operations", read_as="letters"),
        GlossaryEntry(
            "AAA",
            "anti-aircraft artillery",
            read_as="custom",
            spoken_form="Triple A",
        ),
    ),
)

result = prepare_language(
    "AAA enters the AO after the AAR.",
    language="en",
    profile=profile,
)
print(result.spoken_text)
```

The profile produces `Triple A enters the A O after the after-action review.`. Profiles are immutable and do not inherit process-global `add_abbreviation()` customizations. Calls without a profile continue to use the shared registry. Explicit profile entries override bundled meanings, while unrelated registered and generic acronym policies remain controlled by `PreparationConfig`. See [`docs/profiles.md`](docs/profiles.md) for validation, guards, aliases, and current v1 boundaries.

## Development

```bash
python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

python -m pytest
python -m ruff check .
python -m ruff format --check .
python -m mypy spokenform examples
python -m build
python -m twine check dist/*
```

On Windows, activate the environment with `.venv\Scripts\activate`.

## Current limits

- One processing language is supported per call.
- Language detection, mixed-language segmentation, and language marking are external.
- SSMD and other markup must be parsed before calling `spokenform`.
- Date, time, currency, and ordinal grammar is conservative and not exhaustive.
- Spanish parity ownership covers reviewed dates, quantities, temperatures,
  currencies, and ordinary numbers; time expressions remain caller-managed.
- Italian parity ownership covers reviewed dates, quantities, temperatures,
  currencies, and ordinary numbers; colon times remain caller-managed.
- Czech and English own reviewed structured and safe plain-number categories;
  Czech colon-time candidates remain caller-managed. English owns contextual
  single-dot release labels such as `bot 2.0` as `bot two point oh`, while
  ordinary decimals retain digit-wise zero wording. English years, suffix
  ordinals, Roman numerals, phone/ID and arbitrary multi-dot sequences, numeric
  suffixes, and G2P decisions remain downstream in kokorog2p.
- `abbr2words` lexical expansion exposes exact source-aligned replacement records, and Spokenform consumes those records directly. Deterministic stage-local diffs remain only for internal text-only stages such as Unicode, generic number, or whitespace normalization.
- Trained spaCy pipelines must be installed and version-compatible with the spaCy runtime.

## Dependency direction

```text
abbr2words ──────┐
cn2an ───────────┼──> spokenform ──────┐
numeralform ─────┘                    ├──> application / TTS orchestration
                                       │
piperg2p ────────────────────────────┘
```

`abbr2words` owns abbreviation, unit, and currency identities. Spokenform owns source recognition and semantic classification. Numeralform owns non-Chinese number realization through `spokenform.number_words`; `cn2an` remains the Chinese renderer for this migration.

spaCy is an optional quality dependency. `spokenform` remains independent of
language detection, markup parsing, and phoneme generation.

## Release versioning

`setuptools-scm` derives versions from Git tags and writes
`spokenform/_version.py` during builds. Create an annotated tag for the target
release, for example `vX.Y.Z`.
The source-tree fallback when SCM metadata has not been generated is the neutral
version `0+unknown`; release builds derive their version from the annotated tag.

Before publishing, ensure the released `abbr2words>=0.2.13,<0.3.0` prerequisite containing
the source-aligned replacement contract exists on the target package index and run the checklist in
[`docs/release-checklist.md`](docs/release-checklist.md).

## License

Apache License 2.0.

## Recognition modes and domains

The runtime interpretation policy is separate from rendering options:

```python
from spokenform import prepare_language

result = prepare_language(
    "The final was 3-2 and the sample contains H2O.",
    language="en",
    interpretation_mode="surface",
    disabled_domains={"chemistry"},
)
```

`interpretation_mode="contextual"` is the default and preserves the existing contextual behavior. `surface` is fail-closed: only recognizers with intrinsic evidence may claim a structured expression, so ambiguous context-dependent forms can remain unchanged. `disabled_domains` independently suppresses semantic families such as `chemistry`, `biology`, `sports`, or `finance`. Use `allowed_domains` for a fail-closed permitlist that remains stable when future domains are added. `sequence_fallback_mode="preserve"` is the default; `"spell"` provides conservative orthographic coverage for residual sequence-shaped spans without spelling ordinary prose. The legacy `context` option controls abbreviation context and is not the global interpretation mode.

## Optional Lexhint evidence

Lexhint can be supplied explicitly when lexical or positive semantic evidence is available:

```bash
python -m pip install "spokenform[lexhint]"
lexhint dataset download en --variant runtime
```

The optional extra supports Lexhint `0.1.2 <= x < 0.3.0`, including the Lexhint 0.2.x family. Lexhint artifacts are versioned independently of Spokenform: Lexhint 0.1.x uses schema 7, while Lexhint 0.2.x requires schema-8 artifacts.

For Lexhint 0.2.x, install a current schema-8 runtime dataset explicitly with `lexhint dataset download <language> --variant runtime`. Spokenform never downloads Lexhint data automatically; it only uses an installed artifact supplied by the caller.

The runtime provider boundary stays narrow and deterministic. Spokenform uses exact lexical evidence, segmentation, and positive semantic-domain corroboration; Lexhint fuzzy and dictionary-search APIs are not used for automatic recognition.

```python
from lexhint import Lexicon
from spokenform import prepare

lexicon = Lexicon("en", variant="runtime")
result = prepare(
    "Visit chatgpt.com.",
    language="en",
    normalize_literals=True,
    lexical_evidence=lexicon,
    use_spacy=False,
)
print(result.spoken_text)
# Visit chat g p t dot com.
```

The provider language must match Spokenform's base language, so `en_US` and `en` are compatible but `de` is rejected. Lexical-only providers can improve URL rendering; semantic evidence is optional and unavailable semantic capability is not negative evidence. Semantic evidence is used only in contextual mode. Surface mode ignores it, while lexical evidence for an already-recognized URL is still usable for rendering.

Lexhint remains below the interpretation layer. Spokenform owns structured candidate recognition, precedence, domain policy, URL syntax, and speech rendering. `abbr2words` remains the owner of ordinary prose abbreviation and initialism expansion. Lexhint is not a generic prose acronym detector.
