Metadata-Version: 2.4
Name: aicorpusx
Version: 0.1.2
Summary: Resumable, multi-key translation for CSV and XLSX corpora.
Author-email: FENG YIFAN <yifan.f.academic@icloud.com>
License-Expression: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openpyxl>=3.1
Requires-Dist: rich>=13.0
Dynamic: license-file

# aicorpusx

Resumable, multi-key concurrent translation for CSV / XLSX corpora. Any service that speaks OpenAI-compatible `/chat/completions` works: pass `apis`, `model`, and `base_url`.

```python
import aicorpusx

aicorpusx.trans(
    "terms.xlsx",
    source_column="source",
    targets={"ar": "Arabic", "en": "English"},
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

The default output is `terms_translated.xlsx`. CSV input produces CSV; XLSX input produces XLSX.

Set `base_url` to the API root (for example `https://api.deepseek.com` or `https://api.openai.com/v1`). The library appends `/chat/completions`. If you already pass a full endpoint, it is used as-is.

### Install

```bash
python -m pip install aicorpusx
```

From a local checkout:

```bash
python -m pip install .
```

### Examples

#### Languages

There is no fixed language whitelist. The keys in `targets` are passed to the translation model as target-language identifiers, so you can use ISO 639-1 codes (recommended) or full language names such as `"Japanese"`. Actual language coverage depends on the selected model or provider.

Common ISO 639-1 language codes:

| Code | Language | Code | Language |
| --- | --- | --- | --- |
| `zh` | Chinese | `en` | English |
| `ja` | Japanese | `ko` | Korean |
| `ar` | Arabic | `de` | German |
| `fr` | French | `es` | Spanish |
| `ru` | Russian | `pt` | Portuguese |
| `it` | Italian | `tr` | Turkish |
| `vi` | Vietnamese | `th` | Thai |
| `id` | Indonesian | `ms` | Malay |
| `hi` | Hindi | `fa` | Persian |
| `nl` | Dutch | `pl` | Polish |

The value paired with each language is the output column name:

```python
targets={
    "en": "English",
    "ja": "Japanese",
    "ar": "Arabic",
}
```

Use the same target identifiers in glossary column names or `glossary_target_columns` mappings.

#### Translation options

Worksheet, source language, and content mode:

```python
aicorpusx.trans(
    "corpus.xlsx",
    sheet_name="Data",
    source_column="source",
    source_language="zh",
    targets={"ar": "Arabic"},
    mode="sentence",  # auto / term / sentence / text
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

Pass a glossary dict:

```python
aicorpusx.trans(
    "corpus.csv",
    source_column="source",
    targets={"fr": "French"},
    glossary={
        "artificial intelligence": "intelligence artificielle",
        "machine learning": "apprentissage automatique",
    },
    glossary_mode="strict",  # strict / prefer / off
    apis=["key-1"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

A multilingual glossary file can use columns such as `source,ar,en,de`:

```python
aicorpusx.trans(
    "corpus.xlsx",
    source_column="source",
    targets={"ar": "Arabic", "en": "English", "de": "German"},
    glossary="glossary.xlsx",
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

Map irregular column names explicitly:

```python
aicorpusx.trans(
    "corpus.xlsx",
    source_column="source",
    targets={"ar": "Arabic"},
    glossary="glossary.xlsx",
    glossary_source_column="source term",
    glossary_target_columns={"ar": "approved Arabic term"},
    apis=["key-1"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
)
```

Terms are matched in-place, longest first. `strict` re-requests when a matched target term is missing from the result; `prefer` only constrains the prompt; `off` ignores the glossary.

### Scheduling and resume

- `strategy="dynamic"` (default): every live API pulls from a shared queue; faster keys take more work.
- `strategy="balanced"`: work is split evenly at start, and each API shows a fixed total. Unfinished tasks from a dead key are still handed to others.
- The default request gap is `sleep=0.2` seconds.
- 429, 5xx, timeouts, and connection errors back off exponentially with jitter. 401/403 disable that key. Ordinary 400 errors are not retried forever.
- `checkpoint=True` (default) stores successes and failed tasks. Re-runs with `overwrite=False` skip cells already filled in the output or checkpoint.
- Rich progress labels workers `API 1`, `API 2`, and so on. Logs and checkpoints never store API keys.

Main resilience knobs and defaults:

```python
aicorpusx.trans(
    "corpus.csv",
    source_column="source",
    targets={"en": "English"},
    apis=["key-1", "key-2"],
    model="deepseek-chat",
    base_url="https://api.deepseek.com",
    max_retries=5,
    backoff_base=1,
    max_backoff=60,
    api_failure_threshold=5,
    api_cooldown=30,
    max_api_cooldown=300,
)
```

### Non-compatible APIs

Requests use the OpenAI-compatible protocol by default. Extra fields such as `temperature` go in `provider_options`.

If the service is not `/chat/completions`, pass an object with `translate(...)` as `provider`.
