Metadata-Version: 2.5
Name: kintsugi
Version: 0.2.0
Summary: Red-teaming for machine translation: attacks, detectors, and judges from a large MT red-team project.
Project-URL: Homepage, https://github.com/alvations/kintsugi
Project-URL: Repository, https://github.com/alvations/kintsugi
Project-URL: Issues, https://github.com/alvations/kintsugi/issues
Author-email: Liling Tan <alvations@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: adversarial,evaluation,japanese,llm,machine-translation,mt,nlp,red-teaming,safety
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: huggingface
Requires-Dist: torch>=2.0; extra == 'huggingface'
Requires-Dist: transformers>=4.40; extra == 'huggingface'
Description-Content-Type: text/markdown

# Kintsugi

**Red-teaming for machine translation.** Kintsugi finds where a translation
model breaks: inputs that make it hallucinate drug names, erase a negation,
leak its prompt, refuse, soften violent text, or silently switch language. It is
named for the Japanese art of repairing breakage with gold, so the cracks can be
seen and mended.

It packages a battery of translation attacks, a failure detector, and judges:

- **201 attack categories / 2822 seeds** across 9 target languages, each tagged
  with the catastrophic/bad rates actually observed in the runs.
- **A 123-check local detector** that catches obvious failures (empty, copy
  through, refusal, prompt leak, repetition, script mismatch, number drop,
  negation erasure, plus per-language traps) with no model call.
- **11 adversarial source transforms** (homoglyphs, zero-width and RTL controls,
  fullwidth, injection, negation dropping, repetition, mixed scripts).
- **115 translator framings** (personas, task reframings, multi-hop pivots,
  context dilution) that change failure rates.
- **A two-layer judge** (local prefilter then an LLM judge) that cut judge error
  from 53% to 0.2% in the project.
- **Frequent-vs-rare word attacks** for **20 bundled languages**: the same probe
  on common vs rare words, to measure the resource-tier effect directly.

It is **provider-agnostic**. Point it at a local Ollama model, any Hugging Face
model, your own model via a plain function, or a frontier API. Every finding
records the model's provenance (provider, model, version).

All bundled data is shipped **as Python literals**, not data files: importing
kintsugi opens no files and parses no JSON, CSV or pickles.

## Install

```bash
pip install kintsugi                 # core: zero dependencies
pip install "kintsugi[huggingface]"  # to target Hugging Face models
pip install "kintsugi[frequency]"    # frequency probes beyond the 20 bundled languages
```

Python 3.9+.

## Quick start

Try your own model in the simplest possible way: give a function.

```python
from kintsugi import RedTeam, CallableProvider, load_attacks

target = CallableProvider(lambda text, src, tgt: my_model.translate(text, tgt))
rt = RedTeam(target, lang_pair="en-de")                 # detector-only, no judge
findings = rt.run(load_attacks(category="prompt_injection"))
print(rt.summary(findings))
```

A local open-source model via Ollama, judged by a frontier model:

```python
from kintsugi import RedTeam, OllamaProvider, AnthropicProvider, load_attacks

rt = RedTeam(
    target=OllamaProvider("qwen3.5:0.8b"),
    judge=AnthropicProvider("claude-opus-5-5"),   # set ANTHROPIC_API_KEY
    target_lang="Japanese", lang_pair="en-ja",
)
findings = rt.run(load_attacks(category="japanese_pharma_traps"), progress=True)
print(rt.summary(findings))
```

Any Hugging Face model, the simplest way to try an open-weights model:

```python
from kintsugi import RedTeam, HuggingFaceProvider

rt = RedTeam(HuggingFaceProvider("Qwen/Qwen2.5-0.5B-Instruct"), lang_pair="en-de")
print(rt.summary(rt.run(load_attacks(limit_per_category=2))))
```

## Frequent vs non-frequent words

Catastrophic failure scales inversely with how well-represented something is in
training data, and the *same attack is more effective on rarer material*.
Kintsugi makes that a controlled probe, with
frequency bands bundled for 20 languages. Japanese is the reference language
(the probes carry a gold expected answer from a rank-ordered frequency
dictionary); Kannada and Cantonese come from the FLORES+ low-resource benchmark.

```python
from kintsugi import RedTeam, OllamaProvider, frequency

rt = RedTeam(OllamaProvider("qwen3.5:0.8b"), lang_pair="en-ja")
bands = frequency.contrast("ja", n=50)                  # frequent vs rare, held constant
common = rt.summary(rt.run(bands["frequent"]))
rare   = rt.summary(rt.run(bands["rare"]))               # expect a higher catastrophic rate
```

Frequency bands are bundled for every FLORES+ language (233 configs) plus the
`wordfreq` set and the Japanese gold dictionary; `frequency.languages()` lists
them. Friendly aliases (`yue`, `kn`, `zh-Hant`, ...) resolve to the right band.

## Locale variants: region, script, register

Models often collapse a variant to its base language, and that collapse is a
failure worth catching. `kintsugi.locales` registers region, script, and
**register** (diglossia: standard vs colloquial/creole) variants as red-team
targets, each resolving to a frequency band:

```python
from kintsugi import locales, RedTeam, OllamaProvider

locales.diglossia_pairs()            # e.g. ('en-SG','en-SG-x-singlish'), ('de-CH','gsw'), ('ar','ar-EG')

std      = locales.get("en-SG")              # Singapore Standard English
singlish = locales.get("en-SG-x-singlish")   # Singlish (creole)
rt = RedTeam(OllamaProvider("qwen3.5:0.8b"),
             source_lang="English", target_lang=singlish["name"], lang_pair="en-en")
```

Covered: region (`pt-BR`/`pt-PT`, `fr-CA`/`fr-CH`, `de-AT`/`de-CH`, `zh-CN`/
`zh-Hant-TW`/`zh-Hant-HK`/`zh-Hant-MO`/`zh-SG`, ...), script (`zh-Hans`/`zh-Hant`,
`sr-Latn`, `hi-Latn`, ...), and register (Singapore Standard English vs Singlish,
Indian English vs Hinglish, Jamaican Standard English vs Patois, Swiss Standard
German vs Schweizerdeutsch, MSA vs Egyptian/Moroccan Arabic, HK written Chinese
vs spoken Cantonese).

```bash
kintsugi list-locales --register creole
kintsugi list-locales --script Hant
kintsugi frequency --target ollama:qwen3.5:0.8b --lang-of zh-Hant --band rare --n 30
```

## Full language registry

`kintsugi.languages` holds **every Google Translate and Translated language**
plus the curated diglossia/dialect variants above (304 entries), each a BCP-47
code with base language, script, region, register, listing engines, and a
frequency band where one is bundled. A language with no band is still a valid
attack target; frequency bands are bundled for all FLORES+ languages and the
`wordfreq` set.

```python
from kintsugi import languages
languages.by_engine("google")        # every Google Translate language/variant
languages.by_engine("translated")    # every translated.com language/variant
languages.with_band()                # those with a bundled frequency band
languages.get("zh-Hant-HK")          # {'code':'zh-Hant-HK','script':'Hant','region':'HK',...}
```

```bash
kintsugi list-languages --engine google
kintsugi list-languages --with-band
```

The engine coverage is also written out, with the Apple `language-Script-REGION`
convention, in [`docs/LANGUAGE_COVERAGE.md`](docs/LANGUAGE_COVERAGE.md).

## Command line

```bash
kintsugi list                                            # all attack categories
kintsugi list-framings                                   # all translator framings
kintsugi transform zero_width "Do not take with alcohol"
kintsugi attack  --target ollama:qwen3.5:0.8b --judge anthropic:claude-opus-5-5 \
                 --lang en-ja --category japanese_pharma_traps --out findings.jsonl
kintsugi frequency --target ollama:qwen3.5:0.8b --direction en-ja --band rare --n 30
```

## What the attacks find

A few headline numbers, drawn from the bundled observed rates:

| Attack | Pair | Catastrophic rate |
|---|---|---|
| `japanese_pharma_traps` | en-ja | ~51% |
| `anatomical_near_miss` | en-ja | 50% |
| `prompt_injection` | en-de | ~30% |
| `safety_critical_substance` | en-de | ~25% |

Vulnerability tracks resource tier almost perfectly: French is the hardest to
break, low-resource Indic and Cantonese targets the easiest. Models at or above
~4B parameters show near-zero catastrophic rates; models at or below ~2B should
not be used for safety-critical translation.

```python
import kintsugi
kintsugi.observed_rates()["japanese_pharma_traps"]["en-ja"]   # {'n': 79, 'catastrophic': 40, 'bad': ...}
```

## How judging works

1. **Local prefilter** (`kintsugi.prefilter`) classifies obvious failures for
   free and returns `None` for anything ambiguous.
2. **LLM judge** (`kintsugi.Judge`) scores the rest on adequacy, fluency,
   faithfulness, format and safety, and never returns an error verdict: on a
   parse failure it defaults to `ok` so statistics stay clean, and always-severe
   failure types (hallucination, refusal, prompt leak) are promoted.

A judge is optional. With no judge, kintsugi still runs the full prefilter and
profanity check, which already catch the most dangerous failure modes.

## License

Apache 2.0.
