Metadata-Version: 2.5
Name: pantogloss
Version: 1.0.0rc1
Summary: TensorFlow/Keras many-to-English machine translation
Project-URL: Homepage, https://github.com/chrismattmann/pantogloss
Project-URL: Repository, https://github.com/chrismattmann/pantogloss
Project-URL: Model repository, https://huggingface.co/chrismattmann/pantogloss-500-en
Project-URL: Compact model repository, https://huggingface.co/chrismattmann/pantogloss-500-en-v7-compact
Project-URL: FP16 model repository, https://huggingface.co/chrismattmann/pantogloss-500-en-fp16
Project-URL: INT8 model repository, https://huggingface.co/chrismattmann/pantogloss-500-en-int8
Project-URL: V6 successor model repository, https://huggingface.co/chrismattmann/pantogloss-500-en-v6
Project-URL: V7 successor model repository, https://huggingface.co/chrismattmann/pantogloss-500-en-v7
Project-URL: Issues, https://github.com/chrismattmann/pantogloss/issues
Author: Chris A. Mattmann
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: keras,machine translation,multilingual,tensorflow
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: huggingface-hub<2,>=0.26
Requires-Dist: marisa-trie<2,>=1.2
Requires-Dist: numpy<3,>=1.26
Requires-Dist: sacremoses<0.3,>=0.2
Requires-Dist: tensorflow<2.19,>=2.18; sys_platform == 'darwin' and platform_machine == 'arm64'
Requires-Dist: tensorflow<2.22,>=2.18; sys_platform != 'darwin' or platform_machine != 'arm64'
Provides-Extra: conversion
Requires-Dist: ruamel-yaml>=0.17; extra == 'conversion'
Requires-Dist: torch<3,>=2.2; extra == 'conversion'
Provides-Extra: cuda
Requires-Dist: tensorflow[and-cuda]<2.22,>=2.18; (sys_platform == 'linux') and extra == 'cuda'
Provides-Extra: evaluation
Requires-Dist: sacrebleu<3,>=2.4; extra == 'evaluation'
Provides-Extra: metal
Requires-Dist: tensorflow-metal<1.3,>=1.2; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'metal'
Requires-Dist: tensorflow<2.19,>=2.18; (sys_platform == 'darwin' and platform_machine == 'arm64') and extra == 'metal'
Provides-Extra: server
Requires-Dist: fastapi<1,>=0.115; extra == 'server'
Requires-Dist: httpx<1,>=0.27; extra == 'server'
Requires-Dist: uvicorn<1,>=0.34; extra == 'server'
Provides-Extra: test
Requires-Dist: fastapi<1,>=0.115; extra == 'test'
Requires-Dist: httpx<1,>=0.27; extra == 'test'
Requires-Dist: nlcodec<0.6,>=0.5; (sys_platform != 'win32') and extra == 'test'
Requires-Dist: pypdf<7,>=6.16; extra == 'test'
Requires-Dist: pytest-cov<8,>=5; extra == 'test'
Requires-Dist: pytest<10,>=8; extra == 'test'
Requires-Dist: sacrebleu<3,>=2.4; extra == 'test'
Provides-Extra: tika
Requires-Dist: pypdf<7,>=6.16; extra == 'tika'
Requires-Dist: tika<4,>=3.3; extra == 'tika'
Description-Content-Type: text/markdown

# Pantogloss

<p align="center">
  <img src="assets/branding/pantogloss-mark-192.png" width="160" alt="Pantogloss cryptid moth mark">
</p>

[![Tests](https://github.com/chrismattmann/pantogloss/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/chrismattmann/pantogloss/actions/workflows/test.yml)
[![PyPI](https://img.shields.io/pypi/v/pantogloss.svg)](https://pypi.org/project/pantogloss/)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](LICENSE)

Pantogloss is a TensorFlow/Keras library that translates text from many
languages into English. It provides a Python API, command-line interface, and
optional persistent local server with a browser UI.

The default public model is
[`pantogloss-500-en-compact`](https://huggingface.co/chrismattmann/pantogloss-500-en-v7-compact),
a 539 MiB model validated on NVIDIA CUDA and Apple Metal. Model weights are
downloaded separately from Hugging Face and are never bundled in the Python
wheel. Public models normally require no Hugging Face login.

## What it can do

- Translate individual strings, batches, files, and bounded input streams.
- Run on CPU, NVIDIA CUDA, or Apple Silicon Metal.
- Keep a model resident behind a local HTTP API for fast repeated requests.
- Provide a simple Any language → English browser translator.
- Offer fast greedy and higher-quality beam-search decoding presets.
- Return plain strings or structured results with timing and runtime metadata.
- Attach caller-supplied language quality guidance and conservative warnings.
- Preserve input order and isolate individual failures in long-running streams.

Pantogloss translates text; it does not detect the source language or parse
document formats. See [Project scope](#project-scope).

## Installation

Pantogloss supports Python 3.10–3.12. Use the extra for your accelerator:

```bash
# CPU
python -m pip install pantogloss

# NVIDIA GPU on Linux
python -m pip install "pantogloss[cuda]"

# Apple Silicon Metal
python -m pip install "pantogloss[metal]"

# Persistent API and browser UI; combine with cuda or metal when needed
python -m pip install "pantogloss[server,metal]"
```

The first use downloads the selected model. Later loads use the local
Hugging Face cache.

## Python API

Load one translator and reuse it:

```python
from pantogloss import Translator

translator = Translator.from_pretrained(device="auto")
print(translator.translate("Comment allez-vous ?"))
```

Translate a batch or request structured results:

```python
translations = translator.translate([
    "Hola señor",
    "Wie geht es Ihnen?",
])

result = translator.translate_detailed("Bonjour le monde.")
print(result.text)
print(result.elapsed_seconds, result.execution_device)
```

For a large iterable, `translate_iter()` and `translate_iter_detailed()` process
bounded batches without retaining the complete input or output collection.

```python
with open("source.txt", encoding="utf-8") as source:
    for translation in translator.translate_iter(source, batch_size=16):
        print(translation)
```

Use the quality preset when latency is less important than beam-search quality:

```python
print(translator.translate("Comment allez-vous ?", preset="quality"))
```

## Command line

```bash
pantogloss translate "Comment allez-vous ?"
pantogloss translate --preset quality "Comment allez-vous ?"
pantogloss translate --input source.txt --output english.txt --batch-size 16
pantogloss models
pantogloss info --device gpu --json
pantogloss doctor
```

Translation output stays clean on stdout. Use `--verbose`, `--report`, or
`--tensorflow-logs` when diagnostics are wanted. Use `--offline` to require an
already cached model.

## Persistent server and browser UI

Each standalone CLI invocation reloads the model. For repeated requests, keep
one warmed translator in memory:

```bash
python -m pip install "pantogloss[server,metal]"  # or server,cuda
pantogloss serve --device gpu
```

Open `http://127.0.0.1:8765/` for the two-pane browser translator. The same
process exposes:

- `GET /health`
- `GET /info`
- `GET /metrics`
- `POST /translate`

```bash
curl http://127.0.0.1:8765/translate \
  -H 'Content-Type: application/json' \
  -d '{"text":"Comment allez-vous ?"}'
```

The server binds to loopback, warms the model before reporting readiness, and
serializes inference by default. Queuing is bounded, shutdown drains active
TensorFlow work, and optional request logs contain timing/count metadata rather
than source or translated text. On Apple Silicon, the server defaults to the
bounded-memory eager decoder because TensorFlow Metal retains memory during
compiled decoding; `--metal-compiled-decode` is an explicit faster but
memory-growing opt-in. See the
[Server and Web UI wiki page](https://github.com/chrismattmann/pantogloss/wiki/Server-and-Web-UI)
for deployment, authentication, limits, API schemas, benchmarks, and the
`pantogloss balance` front end for independently resident workers.
The balancer supports explicit graceful draining, connection-only safe retries,
passive circuit breaking, and Prometheus-compatible reliability metrics.

## Models

| Name | Role | Approximate artifact size |
| --- | --- | ---: |
| `pantogloss-500-en-compact` | Recommended default | 539 MiB |
| `pantogloss-500-en-compact-v6` | Previous V6-derived compact default | 538 MiB |
| `pantogloss-500-en-compact-v1` | Previous compact model | 538 MiB |
| `pantogloss-500-en-fp16` | Accelerator-oriented FP16 reference | 1.02 GiB |
| `pantogloss-500-en` | Historical FP32 numerical reference | 2.03 GiB |
| `pantogloss-500-en-int8` | Smaller experimental alternative | 539 MiB |
| [`pantogloss-500-en-v7`](https://huggingface.co/chrismattmann/pantogloss-500-en-v7) | Opt-in FP32 successor with conversational Spanish improvements | 2.03 GiB |
| [`pantogloss-500-en-v6`](https://huggingface.co/chrismattmann/pantogloss-500-en-v6) | Opt-in FP32 fine-tuned successor | 2.03 GiB |

Select a model explicitly when needed:

```python
translator = Translator.from_pretrained("pantogloss-500-en-fp16", device="gpu")
successor = Translator.from_pretrained("pantogloss-500-en-v7", device="gpu")
```

V7 builds on V6 with a bounded conversational Spanish-to-English fine-tune.
It improved chrF by 2.04–2.78 on two sealed Fisher/CALLHOME confirmation splits
while improving aggregate chrF by 0.217 on the 50-language development suite;
no evaluated language crossed the frozen −0.5 chrF retention floor. V7 remains
available as an opt-in full-size model and now supplies the recommended compact
default. Compact V7 retains the measured conversational Spanish gains while
remaining approximately 539 MiB. It slightly improves aggregate chrF over the
previous V6-derived compact default, though individual languages can vary. The
full-size FP32 V6 model remains available as an opt-in reference.
Caller-supplied language-quality guidance is tied to the evaluated model
revision: the default and original FP32 reference have separate measured
catalogs; other model variants return `unmeasured` rather than borrowing scores.
The INT8 alternative is not the default because it is substantially slower.
Detailed evidence and backend caveats live in the model cards, wiki, and
checked-in experiment reports.

## Stability and security

Pantogloss is preparing its 1.0 compatibility contract. See the
[compatibility and deprecation policy](docs/compatibility.md) for the proposed
stable surfaces and the [API contract inventory](docs/api-contract.md) for the
executable Python, CLI, server, and model-selection baseline. Security concerns
should be reported privately as described in [SECURITY.md](SECURITY.md).
The [research roadmap](docs/research-roadmap.md) records the paper thesis,
experimental matrix, required ablations, and publication-readiness gates.

## Documentation

- [Installation and Quickstart](https://github.com/chrismattmann/pantogloss/wiki/Installation-and-Quickstart)
- [Server and Web UI](https://github.com/chrismattmann/pantogloss/wiki/Server-and-Web-UI)
- [Decoding Presets](https://github.com/chrismattmann/pantogloss/wiki/Decoding-Presets)
- [Platform Validation](https://github.com/chrismattmann/pantogloss/wiki/Platform-Validation)
- [Language Quality Catalog](https://github.com/chrismattmann/pantogloss/wiki/Language-Quality-Catalog)
- [Quality Evaluation](https://github.com/chrismattmann/pantogloss/wiki/Quality-Evaluation)
- [Development and Testing](https://github.com/chrismattmann/pantogloss/wiki/Development-and-Testing)
- [Tika Document Translation](https://github.com/chrismattmann/pantogloss/wiki/Tika-Document-Translation)

The repository also retains reproducible model-conversion, parity, compression,
and evaluation artifacts under [`docs`](docs) and [`evaluation`](evaluation).

## Project scope

Pantogloss translates text and ordered text segments. It intentionally does not
identify languages, detect file types, or parse documents. `DocumentTranslator`
is a neutral text-segmentation helper. `pantogloss-tika` remains a compatibility
example and is not a direction for new core dependencies. See the
[architecture boundary](docs/architecture.md).

Translation quality varies by language, domain, and input. Pantogloss does not
provide calibrated confidence and should not be relied on without review for
medical, legal, safety-critical, or other high-stakes decisions.

## Provenance and license

The original model was described by Thamme Gowda, Zhao Zhang, Chris A. Mattmann,
and Jonathan May in *Many-to-English Machine Translation Tools, Data, and
Pretrained Models*, ACL-IJCNLP 2021 System Demonstrations,
[DOI 10.18653/v1/2021.acl-demo.37](https://doi.org/10.18653/v1/2021.acl-demo.37).

Pantogloss and its converted models are licensed under the
[Apache License, Version 2.0](LICENSE). See [NOTICE](NOTICE) for attribution.
