Metadata-Version: 2.5
Name: tinbox
Version: 0.2.0
Summary: A CLI translation tool using LLMs for document translation
Author-email: strickvl <your.email@example.com>
License-Expression: MIT
License-File: LICENSE
Keywords: cli,docx,llm,pdf,translation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.12
Requires-Dist: pillow>=10.0.0
Requires-Dist: pydantic<3.0.0,>=2.8.0
Requires-Dist: pypdf>=5.3.0
Requires-Dist: python-docx>=1.1.2
Requires-Dist: rich>=13.7.0
Requires-Dist: structlog>=24.1.0
Requires-Dist: tenacity>=8.2.3
Requires-Dist: typer>=0.9.0
Provides-Extra: all
Requires-Dist: anthropic>=0.40.0; extra == 'all'
Requires-Dist: openai>=1.92.0; extra == 'all'
Requires-Dist: pdf2image>=1.17.0; extra == 'all'
Requires-Dist: python-docx>=1.1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40.0; extra == 'anthropic'
Provides-Extra: docx
Requires-Dist: python-docx>=1.1.0; extra == 'docx'
Provides-Extra: openai
Requires-Dist: openai>=1.92.0; extra == 'openai'
Provides-Extra: pdf
Requires-Dist: pdf2image>=1.17.0; extra == 'pdf'
Description-Content-Type: text/markdown

# 🔄 Tinbox: Your Ultimate Translation Tool

**Tinbox** is a robust command-line tool designed to tackle the challenges of translating large documents using Large Language Models (LLMs). Unlike other tools, Tinbox excels in handling extensive document sizes and navigates around model limitations related to size and copyright issues, ensuring seamless and efficient translations.

## 🚀 Quick Start

```bash
# Install with all dependencies
pip install -e ".[all]"

# Set your API key
export OPENAI_API_KEY="your-api-key-here"

# Translate a text document
tinbox translate --to de --model openai:gpt-5-2025-08-07 ./examples/elara_story.txt

# Translate a PDF (requires poppler - see Installation section)
tinbox translate --to de --model openai:gpt-5-2025-08-07 --algorithm page ./examples/elara_story.pdf

# Use local models with Ollama
ollama serve  # In another terminal
tinbox translate --to de --model ollama:llama3.1:8b ./examples/elara_story.txt
```

## ✨ Key Features

- **🔍 No OCR Needed**: Directly translates PDFs using vision-capable multimodal models (GPT-4o, Claude Sonnet, Gemini Pro, etc.)
- **📄 Smart Document Handling**: Supports PDFs, Word documents, and text files
- **📏 Large Document Handling**: Efficiently processes large PDFs and other document types
- **📌 Overcomes Model Limitations**: Bypasses common model refusals due to size or copyright concerns.
- **🧠 Intelligent Algorithms**: Context-aware translation with smart text splitting
- **🌐 Local and Cloud Support**: Works with OpenAI, Anthropic, Google, or local Ollama models
- **📚 Glossary Support**: Maintains consistent terminology across documents
- **💾 Checkpoint & Resume**: Handle large documents with automatic progress saving
- **📊 Cost Tracking**: Estimate translation costs and time and monitor during translation

## 🎯 The Problems Tinbox Solves

1. **PDF Translation Challenges**

   - Most tools require OCR, leading to formatting loss and errors
   - Tinbox uses multimodal models to directly understand PDFs as images

2. **Large Document Limitations**

   - Traditional tools often fail with large documents
   - Models frequently refuse or timeout on big files
   - Tinbox smartly splits and processes documents while maintaining context

3. **Model Refusal Issues**

   - Many models refuse translation tasks due to:
     - Copyright concerns
     - Document size limitations
     - Rate limiting
   - Tinbox's algorithms work around these limitations intelligently

4. **Quality and Consistency**
   - Smart algorithms ensure consistent translations across document sections
   - Maintains context between pages and segments
   - Uses glossary to maintain consistent terminology

## ✨ Features

### 📄 Smart Document Handling

- **PDFs**: Processed directly as images - no OCR needed! ⚠️ **Requires a vision-capable model** (GPT-4o, Claude Sonnet, Gemini Pro Vision). Text-only models will fail on PDFs.
- **Word (docx)**: Preserves formatting while translating
- **Text files**: Efficient processing for large files

### 🧠 Intelligent Translation

- **Smart Algorithms**:
  - Page-by-Page (used for PDF documents)
  - Sliding Window for long text documents (deprecated)
  - Context-Aware algorithm with smart text splitting (default for text documents)

### 🤖 Flexible Model Support

- Use powerful cloud models (GPT-5, Claude 4 Sonnet, Google Gemini, etc.)
- Run translations locally with Ollama
- Mix and match models for different tasks

### 💾 Checkpoint & Resume

- Automatically save checkpoints and resume translations

### 🌐 Language Support

- Flexible source/target language specification using ISO 639-1 codes
- Common language aliases (e.g., "en", "zh", "es")

### 📊 Cost Tracking

- Track overall translation time and token usage/cost
- Compare algorithms or model providers side-by-side
- Full progress and cost tracking during translation

## 🚀 Getting Started

## 📦 Installation

### Basic Installation

**Requires Python 3.12 or higher.**

```bash
# Recommended: Install with all features (all providers + PDF + DOCX)
uv tool install --python 3.13 "tinbox[all]"

# Or install specific providers
uv tool install --python 3.13 "tinbox[openai]"      # OpenAI, Ollama, and Gemini support
uv tool install --python 3.13 "tinbox[anthropic]"   # Anthropic Claude support

# Base package only (no provider SDKs)
uv tool install --python 3.13 tinbox
```

For an existing PyPI installation with the desired extras and Python 3.12+, run `uv tool upgrade tinbox`.

When upgrading from 0.1.0, switching from a GitHub installation, or changing Python versions, reinstall from PyPI with explicit extras:

```bash
uv tool install --force --python 3.13 "tinbox[all]>=0.2.0"
tinbox --version
```

Use `tinbox[openai]` instead of `tinbox[all]` if you only need OpenAI, Ollama, Gemini, or oMLX. Version 0.2.0 requires Python 3.12+ and replaces LiteLLM with optional provider SDKs, so a base-only upgrade does not install those SDKs. For development from a source checkout, use `uv sync --locked --all-extras`.

> **Note:** Ollama and Gemini are accessed via OpenAI-compatible APIs, so they require the `openai` extra.
> For [OpenRouter](https://openrouter.ai), use `openai:model-name` with `OPENAI_BASE_URL=https://openrouter.ai/api/v1`.

### TranslateGemma with oMLX

TranslateGemma uses a translation-specific prompt rather than Tinbox's usual chat and JSON protocol. For an oMLX server hosting the model, select its plain-text protocol explicitly:

```bash
export OPENAI_BASE_URL="http://127.0.0.1:8000/v1"
export OPENAI_API_KEY="local"  # Or the key configured on your server

tinbox translate --from de --to en \
  --model openai:mlx-community--translategemma-27b-it-4bit \
  --translation-protocol translategemma --local \
  --format markdown ./translate-test.txt
```

Use the exact model ID listed by your server. This mode renders the text form of [TranslateGemma's template](https://huggingface.co/mlx-community/translategemma-27b-it-4bit/blob/main/chat_template.jinja) and sends it to `/v1/completions`, following [oMLX's guidance](https://github.com/jundot/omlx/issues/879#issuecomment-4293308928). It reads the plain translation response without requiring JSON.

The initial mode supports short `.txt` files, an explicit source language, and the `page` algorithm, which is selected automatically. Context-aware/sliding-window translation, glossary, reasoning options, and image/PDF inputs are rejected before translation. Each text file is sent as one request. [Google documents a 2K-token input context](https://huggingface.co/google/translategemma-27b-it#usage); Tinbox does not load the model tokenizer or enforce that limit, so split longer files before using this mode. Output is capped at 2,048 tokens; an incomplete response is reported as a failure.

`--local` declares that the endpoint runs a model without API charges. It applies zero API cost to both estimates and recorded usage, and also works with the default chat protocol. Use it only for a local model, not for a localhost proxy that forwards requests to paid providers. Without this flag, custom OpenAI-compatible endpoints retain Tinbox's approximate OpenAI provider rates. Ollama has zero API charges by default. Token and time estimates are approximate, and context-overhead notices do not indicate a detected translation-quality problem.

The protocol is covered by real-SDK tests over mocked HTTP and [live CLI checks](plans/omlx-live-validation.md) against oMLX 0.7.2 with `mlx-community/translategemma-4b-it-4bit`, including German ↔ English translation and text/JSON/Markdown exports. The 27B variant has not been tested live.

### System Requirements

**For PDF processing**, install `poppler-utils`:

```bash
# macOS
brew install poppler

# Linux (Ubuntu/Debian)
sudo apt-get install poppler-utils

# Windows (Chocolatey)
choco install poppler
```

### API Keys

Set up your API keys for cloud models:

```bash
# Choose your preferred model provider
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
export GOOGLE_API_KEY="your-google-api-key"
```

## 🎮 Basic Usage

```bash
# Simple document translation
tinbox translate --to es --model openai:gpt-5-2025-08-07 document.txt

# Specify output file
tinbox translate --to es --model openai:gpt-5-2025-08-07 --output document_es.txt document.txt

# Specify source language
tinbox translate --from zh --to en --model openai:gpt-5-2025-08-07 document.txt

# Large documents with checkpointing
tinbox translate --to de --checkpoint-dir ./checkpoints --model openai:gpt-5-2025-08-07 large_document.txt

# With glossary
tinbox translate --to es --glossary --model openai:gpt-5-2025-08-07 document.txt

# Setting a cost limit
tinbox translate --to es --max-cost 10.00 --model openai:gpt-5-2025-08-07 document.txt

# Use model reasoning
tinbox translate --to es --reasoning-effort low --max-cost 10.00 --model openai:gpt-5-2025-08-07 document.txt
```

## 💡 Best Practices

| Document Type         | Recommended Settings                    | Notes                                                        |
| --------------------- | --------------------------------------- | ------------------------------------------------------------ |
| **PDFs**              | `--algorithm page`                      | Page-by-page processing works best                           |
| **Large Text**        | `--context-size 2000`                   | Context-aware algorithm (default) handles size intelligently |
| **Technical Docs**    | `--glossary --save-glossary terms.json` | Maintains consistent terminology                             |
| **Long Translations** | `--checkpoint-dir ./checkpoints`        | Resume interrupted translations                              |
| **Cost Control**      | `--dry-run --max-cost 5.00`             | Preview costs before translating                             |
| **Reasoning**         | `--reasoning-effort minimal`            | Faster and cheaper, only use higher levels if needed         |

## 📖 Detailed Documentation

### Command-Line Reference

| Category         | Option                   | Description                                                       | Default         |
| ---------------- | ------------------------ | ----------------------------------------------------------------- | --------------- |
| **Core**         | `--from, -f`             | Source language (auto-detect if not specified)                    | Auto-detect     |
|                  | `--to, -t`               | Target language                                                   | English         |
|                  | `--model`                | Model to use (`openai:gpt-4o`, `anthropic:claude-3-sonnet`, etc.) | **Required**    |
|                  | `--output, -o`           | Output file (default: print to console)                           | Console         |
| **Algorithms**   | `--algorithm, -a`        | Translation algorithm (`context-aware`, `page`, `sliding-window`) | Auto-select     |
|                  | `--context-size`         | Target chunk size for context-aware algorithm                     | 2000 chars      |
|                  | `--split-token`          | Custom token to split text on                                     | None            |
| **PDF**          | `--pdf-dpi`              | DPI for PDF rasterization (higher = better quality, more tokens)  | 200             |
| **Checkpoints**  | `--checkpoint-dir`       | Directory to store checkpoints for resuming                       | None            |
|                  | `--checkpoint-frequency` | Save checkpoint every N pages/chunks                              | 1               |
| **Glossary**     | `--glossary`             | Enable glossary for consistent terminology                        | Disabled        |
|                  | `--glossary-file`        | Path to existing glossary file (JSON)                             | None            |
|                  | `--save-glossary`        | Path to save updated glossary                                     | None            |
| **Quality**      | `--reasoning-effort`     | Model reasoning level (`minimal`, `low`, `medium`, `high`)        | `minimal`       |
| **Output**       | `--format, -F`           | Output format (`text`, `json`, `markdown`)                        | `text`          |
| **Cost Control** | `--dry-run`              | Preview costs without translating                                 | Disabled        |
|                  | `--max-cost`             | Maximum cost limit (USD)                                          | No limit        |

### Supported Languages

Supports all language that is supported by the selected model.

### Advanced Examples

#### Algorithm Selection

```bash
# Context-aware (default) - best for most documents
tinbox translate --to es --context-size 1500 --model openai:gpt-5-2025-08-07 document.txt

# Page-by-page - best for PDFs
tinbox translate --to es --algorithm page --model openai:gpt-5-2025-08-07 document.pdf

# Sliding window (not recommended, use context-aware instead)
tinbox translate --to es --algorithm sliding-window --model openai:gpt-5-2025-08-07 large_file.txt

# Custom splitting for structured documents
tinbox translate --to fr --split-token "---" --model openai:gpt-5-2025-08-07 structured_document.txt
```

#### Model Providers

```bash
# OpenAI models
tinbox translate --to es --model openai:gpt-4o document.pdf

# Anthropic Claude
tinbox translate --to es --model anthropic:claude-3-sonnet document.pdf

# Local models with Ollama
tinbox translate --to fr --model ollama:mistral-small document.txt

# Quality vs cost control
tinbox translate --to es --reasoning-effort high --max-cost 10.00 document.pdf
```

#### Glossary & Consistency

When glossary is enabled, it will automatically detect important terms and add them to the glossary during translation.

```bash
# Use glossary during translation
tinbox translate --to es --glossary --model openai:gpt-4o medical_doc.pdf

# Build and save terminology in the end
tinbox translate --to es --glossary --save-glossary medical_terms.json --model openai:gpt-4o medical_doc.pdf

# Use existing initial glossary
tinbox translate --to fr --glossary-file company_terms.json --model openai:gpt-4o document.docx

# Use existing glossary and save extended glossary in the end
tinbox translate --to de --glossary-file base_terms.json --save-glossary extended_terms.json --model openai:gpt-4o doc.pdf
```

**Glossary File Format**

Glossary files use JSON format for easy editing and sharing:

```json
{
  "entries": {
    "CPU": "Processeur",
    "GPU": "Carte graphique",
    "API": "Interface de programmation"
  }
}
```

#### Output Formats

```bash
# JSON with metadata and costs
tinbox translate --to es --format json --model openai:gpt-4o document.pdf

# Markdown report
tinbox translate --to es --format markdown --model openai:gpt-4o document.pdf

# Save to file
tinbox translate --to es --output translated.txt --model openai:gpt-4o document.pdf
```

---

## 🔧 Troubleshooting

Run `tinbox doctor` to diagnose common setup issues:

```bash
tinbox doctor
```

This checks:
- ✅ Required Python dependencies (Pillow, python-docx)
- ✅ Optional PDF dependencies (Poppler, pdf2image)
- ✅ Model SDKs, API key configuration, and local model availability

Missing optional dependencies are shown with installation hints and do not cause `tinbox doctor` to fail. Poppler and pdf2image are required when translating PDFs, but are not needed for text files.

**Common Issues:**

| Problem | Solution |
|---------|----------|
| PDF translation fails | 1. Install poppler (`brew install poppler` on macOS) 2. Use a vision-capable model (GPT-4o, Claude Sonnet, Gemini Pro) |
| "No module named pdf2image" | Install PDF extras: `pip install tinbox[pdf]` |
| API key not found | Set environment variable: `export OPENAI_API_KEY="..."` |
| Translation times out | Try a smaller `--context-size` or use `--checkpoint-dir` for resumable translations |

---

## 🛠 Development

### Contributing Setup

```bash
# Install uv first: https://docs.astral.sh/uv/getting-started/installation/
# Clone and set up the development environment
git clone <repository-url>
cd tinbox
uv sync --locked --all-extras
```

This installs the project, all provider/document extras, and development tools into `.venv` using `uv.lock`. Development uses the Python version in `.python-version`. Install Poppler using the [system requirements](#system-requirements) above before running PDF tests.

Use `uv add` for runtime dependencies and `uv add --dev` for development tools, and include the resulting `uv.lock` changes with `pyproject.toml`.

Dependabot checks Python packages and GitHub Actions every Monday at 09:00 Europe/Amsterdam. Routine updates wait until releases are at least seven days old, with minor and patch updates grouped per ecosystem and major updates proposed separately. Security updates bypass the cooldown.

### Code Quality

```bash
uv run --locked --all-extras ruff check src tests
uv run --locked --all-extras ruff format --check src tests
uv run --locked --all-extras ty check
```

Use `ruff check --fix` and `ruff format` through the same `uv run` command to fix lint and formatting issues. CI runs these checks once, treats ty warnings as failures, and tests Python 3.12 and 3.13 with Poppler installed.

### Testing

```bash
# Run all tests
uv run --locked --all-extras pytest

# Run specific test categories
uv run --locked --all-extras pytest tests/test_core/test_cost.py -v          # Cost estimation
uv run --locked --all-extras pytest tests/test_core/test_processors/ -v     # Document processing
uv run --locked --all-extras pytest tests/test_core/test_translation/ -v    # Translation algorithms
uv run --locked --all-extras pytest tests/test_cli.py -v                    # CLI interface

# Coverage report
uv run --locked --all-extras pytest --cov-report=html
```

### Project Structure

```
tinbox/
├── src/tinbox/
│   ├── cli.py              # Command-line interface
│   ├── core/
│   │   ├── cost.py         # Cost tracking
│   │   ├── processor/      # Document processors (PDF, DOCX, text)
│   │   └── translation/    # Translation algorithms
│   └── utils/              # Utility functions
└── tests/                  # Comprehensive test suite
```
