Metadata-Version: 2.4
Name: aicorpusx
Version: 0.1.3
Summary: Resumable, multi-key translation for CSV and XLSX corpora.
Author-email: FENG YIFAN <yifan.f.academic@icloud.com>
License-Expression: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openpyxl>=3.1
Requires-Dist: rich>=13.0
Dynamic: license-file

# aicorpusx

`aicorpusx` is a concurrent translation library for Python programs and CSV/XLSX corpora. It supports OpenAI-compatible `/chat/completions` services, multiple API keys, terminology constraints, retries, resumable file translation, in-memory workflows, and memory-bounded streaming pipelines.

## Features

- Translate a single string, a sequence of strings, Python dictionaries, CSV files, or XLSX workbooks.
- Continue processing translated data directly in Python without writing an intermediate file.
- Read, translate, and write large datasets in bounded batches.
- Share work dynamically across multiple API keys or split it evenly.
- Retry rate limits, server failures, timeouts, and connection errors with exponential backoff.
- Resume `trans()` file jobs from their output file and JSON checkpoint.
- Apply multilingual glossaries in `prefer`, `strict`, or `off` mode.
- Use any provider that implements the OpenAI-compatible chat-completions protocol, or supply a custom provider object.

## Installation

```bash
python -m pip install aicorpusx
```

From a local checkout:

```bash
python -m pip install .
```

## Choose an interface

| Goal | Interface | Memory behavior | Checkpoint |
| --- | --- | --- | --- |
| Translate one value | `translate_text()` | In memory | No |
| Translate a list of values | `translate_texts()` | In memory | No |
| Translate Python dictionaries | `translate_rows()` | In memory | No |
| Translate an iterable in batches | `translate_rows_iter()` | Bounded by `batch_size` | No |
| Read/write all file rows | `read_rows()` / `write_rows()` | In memory | No |
| Stream file rows | `read_rows_iter()` / `write_rows_iter()` | Incremental | No |
| Translate a CSV/XLSX file directly | `trans()` | Loads the table | JSON checkpoint |

Use `trans()` for the simplest resumable file workflow. Use the iterator APIs for large files or when translation is one stage in a larger Python pipeline.

## API credentials

Do not publish API keys in source code or screenshots. Loading keys from environment variables is safer:

```python
import os

api_keys = [
    os.environ["TRANSLATION_API_KEY_1"],
    os.environ["TRANSLATION_API_KEY_2"],
]
```

Every example below assumes `api_keys` contains one or more valid credentials.

## Quick start: file to file

```python
from aicorpusx import trans

output_path = trans(
    "terms.xlsx",
    output="terms_translated.xlsx",  # optional
    source_column="source",
    targets={
        "ar": "Arabic",
        "en": "English",
    },
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

print(output_path)
```

The keys in `targets` identify target languages. The values are output column names. CSV input produces CSV output and XLSX input produces XLSX output. If `output` is omitted, the default name is `<input_stem>_translated.<extension>`.

### Resume and overwrite behavior

`trans()` defaults to `checkpoint=True` and `overwrite=False`.

- Non-empty target cells in an existing output file are skipped.
- Successful checkpoint results are restored into empty output cells.
- Re-running a fully completed job exits immediately because there are no remaining tasks.
- `overwrite=True` translates all eligible source cells again.
- `checkpoint=False` prevents creation of the checkpoint JSON file.

The checkpoint is stored next to the output file:

```text
terms_translated.xlsx.aicorpusx.checkpoint.json
```

It contains source-file identity information, completed translations, and failed-task details. It never stores API keys. The checkpoint may be deleted after a completed job, but doing so removes that resume record.

## Translate text in Python

### One string

```python
from aicorpusx import translate_text

english = translate_text(
    "Hello, world!",
    target_language="de",
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

print(english)
```

### Multiple strings and languages

```python
from aicorpusx import translate_texts

results = translate_texts(
    ["Hello", "Goodbye"],
    target_languages=["de", "ar"],
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

The result keeps the input order:

```python
[
    {
        "source": "Hello",
        "translations": {
            "de": "Hallo",
            "ar": "...",
        },
    },
    # ...
]
```

## Translate structured Python data

`translate_rows()` accepts any iterable of mappings. It copies the input rows, adds target columns, and returns new dictionaries; it does not mutate the original objects.

```python
from aicorpusx import translate_rows

rows = [
    {"id": 1, "text": "Hello"},
    {"id": 2, "text": "Goodbye"},
]

translated = translate_rows(
    rows,
    source_column="text",
    targets={"de": "German", "ar": "Arabic"},
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

for row in translated:
    print(row["id"], row["German"], row["Arabic"])
```

Existing non-empty target values are preserved by default. Pass `overwrite=True` to replace them.

## Read, process, translate, and write a file

Use this workflow when your program needs preprocessing or postprocessing:

```python
from aicorpusx import read_rows, translate_rows, write_rows

rows = read_rows("input.xlsx", sheet_name="Data")

for row in rows:
    row["source"] = str(row["source"]).strip().replace("\n", " ")

translated = translate_rows(
    rows,
    source_column="source",
    targets={"en": "English", "ar": "Arabic"},
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

for row in translated:
    row["English"] = row["English"].strip()

write_rows(translated, "output.xlsx", sheet_name="Translations")
```

The same helpers support `.csv` and `.xlsx`. The output extension selects the output format, so a program may read XLSX and write CSV or the reverse.

## Large-file streaming

`read_rows()` and `translate_rows()` keep all supplied rows in memory. For large datasets, compose the iterator APIs instead:

```python
from aicorpusx import read_rows_iter, translate_rows_iter, write_rows_iter

rows = read_rows_iter("large_input.csv")

cleaned_rows = (
    {**row, "source": str(row["source"]).strip()}
    for row in rows
)

translated_rows = translate_rows_iter(
    cleaned_rows,
    batch_size=500,
    source_column="source",
    targets={"en": "English", "ar": "Arabic"},
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)

final_rows = (
    {**row, "English": row["English"].strip()}
    for row in translated_rows
)

write_rows_iter(final_rows, "large_output.csv")
```

Only one input row stream and one translation batch are retained. A smaller `batch_size` lowers peak memory use; a larger batch reduces setup overhead and gives the scheduler more work to distribute.

Streaming output columns are taken from `columns=` when provided, otherwise from the first output row. Later rows must not introduce new columns.

```python
write_rows_iter(
    final_rows,
    "large_output.csv",
    columns=["id", "source", "English", "Arabic"],
)
```

### XLSX streaming limitations

Streaming XLSX input uses OpenPyXL read-only mode and streaming output uses write-only mode. This keeps memory bounded but transfers cell values only; it does not preserve the original workbook's styling, charts, merged cells, or macros. Use `trans()` when retaining the original XLSX workbook structure is more important than minimizing memory.

The iterator translation pipeline does not currently create a checkpoint. Process very large resumable jobs in application-defined chunks, or use `trans()` when built-in resume behavior is required.

## Languages

There is no fixed language whitelist. Target identifiers are passed to the selected model, so actual language coverage depends on that model or provider. ISO 639-1 codes are recommended because they are short and consistent; full names such as `"Japanese"` also work when understood by the model.

Common codes:

| Code | Language | Code | Language |
| --- | --- | --- | --- |
| `zh` | Chinese | `en` | English |
| `ja` | Japanese | `ko` | Korean |
| `ar` | Arabic | `de` | German |
| `fr` | French | `es` | Spanish |
| `ru` | Russian | `pt` | Portuguese |
| `it` | Italian | `tr` | Turkish |
| `vi` | Vietnamese | `th` | Thai |
| `id` | Indonesian | `ms` | Malay |
| `hi` | Hindi | `fa` | Persian |
| `nl` | Dutch | `pl` | Polish |

Use the same identifiers consistently in `targets`, glossary columns, and `glossary_target_columns`.

## Source language and content mode

The default source language is `"auto"`. Set it explicitly when useful:

```python
translated = translate_rows(
    rows,
    source_column="source",
    source_language="zh",
    targets={"en": "English"},
    mode="sentence",
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

Supported content modes are:

- `auto`: infer `term`, `sentence`, or `text` from each source value.
- `term`: short names, labels, and terminology.
- `sentence`: individual sentences or short passages.
- `text`: longer, possibly multiline content.

## Glossaries

### Dictionary glossary

For one target language, a flat mapping is sufficient:

```python
translated = translate_rows(
    rows,
    source_column="source",
    targets={"fr": "French"},
    glossary={
        "artificial intelligence": "intelligence artificielle",
        "machine learning": "apprentissage automatique",
    },
    glossary_mode="strict",
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

For multiple target languages, use language-first mappings:

```python
glossary = {
    "en": {"source term": "approved English term"},
    "de": {"source term": "approved German term"},
}
```

### Glossary file

A multilingual glossary file may have columns such as `source,en,de,ar`:

```python
trans(
    "corpus.xlsx",
    source_column="source",
    targets={"en": "English", "de": "German"},
    glossary="glossary.xlsx",
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

Map nonstandard column names explicitly:

```python
glossary_source_column="original term"
glossary_target_columns={"de": "approved German term"}
```

Glossary modes:

- `prefer`: include matched terminology in the prompt.
- `strict`: also reject and retry translations missing required target terms.
- `off`: ignore the glossary.

Terms are matched inside source text, with longer overlapping terms taking priority.

## Multiple API keys and resilience

- `strategy="dynamic"` is the default. Every live API worker pulls from a shared queue, so faster keys may process more tasks.
- `strategy="balanced"` splits work evenly at the start. Work assigned to a disabled key is handed to remaining workers.
- One source row translated into two languages creates two tasks. For example, six rows and two languages produce twelve tasks total, regardless of the number of API keys.
- HTTP 429, HTTP 5xx, timeouts, and connection failures use exponential backoff with jitter.
- HTTP 401 and 403 disable the affected credential.
- API keys are used only in request headers and are never written to progress output or checkpoints.

Main resilience options and defaults:

```python
trans(
    "corpus.csv",
    source_column="source",
    targets={"en": "English"},
    apis=api_keys,
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
    sleep=0.2,
    max_retries=5,
    backoff_base=1,
    max_backoff=60,
    api_failure_threshold=5,
    api_cooldown=30,
    max_api_cooldown=300,
)
```

## Endpoint handling

Set `base_url` to an API root such as:

```text
https://api.deepseek.com
https://api.openai.com/v1
```

The library appends `/chat/completions`. A URL already ending in `/chat/completions` is used unchanged. Extra request fields can be supplied through `provider_options`:

```python
provider_options={"temperature": 0.1}
```

For a service that does not implement the compatible endpoint, pass an object with a `translate(...)` method as `provider`.

## Public API summary

```python
from aicorpusx import (
    read_rows,
    read_rows_iter,
    trans,
    translate_rows,
    translate_rows_iter,
    translate_text,
    translate_texts,
    write_rows,
    write_rows_iter,
)
```

- `trans`: resumable CSV/XLSX translation to another file.
- `translate_text`: translate one value and return a string.
- `translate_texts`: translate multiple values and return structured results.
- `translate_rows`: translate mappings in memory and return copied mappings.
- `translate_rows_iter`: translate mappings in bounded batches and yield results.
- `read_rows`: load a CSV/XLSX file into a list of dictionaries.
- `read_rows_iter`: stream dictionaries from CSV/XLSX.
- `write_rows`: write dictionaries to CSV/XLSX.
- `write_rows_iter`: incrementally consume and write dictionaries.
