Metadata-Version: 2.5
Name: kintsugi
Version: 0.1.0
Summary: Red-teaming for machine translation: attacks, detectors, and judges from a large MT red-team project.
Project-URL: Homepage, https://github.com/alvations/kintsugi
Project-URL: Repository, https://github.com/alvations/kintsugi
Project-URL: Issues, https://github.com/alvations/kintsugi/issues
Author-email: Liling Tan <alvations@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: adversarial,evaluation,japanese,llm,machine-translation,mt,nlp,red-teaming,safety
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: huggingface
Requires-Dist: torch>=2.0; extra == 'huggingface'
Requires-Dist: transformers>=4.40; extra == 'huggingface'
Description-Content-Type: text/markdown

# Kintsugi

**Red-teaming for machine translation.** Kintsugi finds where a translation
model breaks: inputs that make it hallucinate drug names, erase a negation,
leak its prompt, refuse, soften violent text, or silently switch language. It is
named for the Japanese art of repairing breakage with gold, so the cracks can be
seen and mended.

It packages a battery of translation attacks, a failure detector, and judges:

- **201 attack categories / 2822 seeds** across 9 target languages, each tagged
  with the catastrophic/bad rates actually observed in the runs.
- **A 123-check local detector** that catches obvious failures (empty, copy
  through, refusal, prompt leak, repetition, script mismatch, number drop,
  negation erasure, plus per-language traps) with no model call.
- **11 adversarial source transforms** (homoglyphs, zero-width and RTL controls,
  fullwidth, injection, negation dropping, repetition, mixed scripts).
- **115 translator framings** (personas, task reframings, multi-hop pivots,
  context dilution) that change failure rates.
- **A two-layer judge** (local prefilter then an LLM judge) that cut judge error
  from 53% to 0.2% in the project.
- **Frequent-vs-rare word attacks** for **20 bundled languages**: the same probe
  on common vs rare words, to measure the resource-tier effect directly.

It is **provider-agnostic**. Point it at a local Ollama model, any Hugging Face
model, your own model via a plain function, or a frontier API. Every finding
records the model's provenance (provider, model, version).

All bundled data is shipped **as Python literals**, not data files: importing
kintsugi opens no files and parses no JSON, CSV or pickles.

## Install

```bash
pip install kintsugi                 # core: zero dependencies
pip install "kintsugi[huggingface]"  # to target Hugging Face models
pip install "kintsugi[frequency]"    # frequency probes beyond the 20 bundled languages
```

Python 3.9+.

## Quick start

Try your own model in the simplest possible way: give a function.

```python
from kintsugi import RedTeam, CallableProvider, load_attacks

target = CallableProvider(lambda text, src, tgt: my_model.translate(text, tgt))
rt = RedTeam(target, lang_pair="en-de")                 # detector-only, no judge
findings = rt.run(load_attacks(category="prompt_injection"))
print(rt.summary(findings))
```

A local open-source model via Ollama, judged by a frontier model:

```python
from kintsugi import RedTeam, OllamaProvider, AnthropicProvider, load_attacks

rt = RedTeam(
    target=OllamaProvider("qwen3.5:0.8b"),
    judge=AnthropicProvider("claude-opus-5-5"),   # set ANTHROPIC_API_KEY
    target_lang="Japanese", lang_pair="en-ja",
)
findings = rt.run(load_attacks(category="japanese_pharma_traps"), progress=True)
print(rt.summary(findings))
```

Any Hugging Face model, the simplest way to try an open-weights model:

```python
from kintsugi import RedTeam, HuggingFaceProvider

rt = RedTeam(HuggingFaceProvider("Qwen/Qwen2.5-0.5B-Instruct"), lang_pair="en-de")
print(rt.summary(rt.run(load_attacks(limit_per_category=2))))
```

## Frequent vs non-frequent words

Catastrophic failure scales inversely with how well-represented something is in
training data, and the *same attack is more effective on rarer material*.
Kintsugi makes that a controlled probe, with
frequency bands bundled for 20 languages. Japanese is the reference language
(the probes carry a gold expected answer from a rank-ordered frequency
dictionary); Kannada and Cantonese come from the FLORES+ low-resource benchmark.

```python
from kintsugi import RedTeam, OllamaProvider, frequency

rt = RedTeam(OllamaProvider("qwen3.5:0.8b"), lang_pair="en-ja")
bands = frequency.contrast("ja", n=50)                  # frequent vs rare, held constant
common = rt.summary(rt.run(bands["frequent"]))
rare   = rt.summary(rt.run(bands["rare"]))               # expect a higher catastrophic rate
```

Bundled languages: `ar de en es fr hi id it ja kn ko nl pl pt ru sv tr vi yue zh zh-Hant`.

## Locale variants: region, script, register

Models often collapse a variant to its base language, and that collapse is a
failure worth catching. `kintsugi.locales` registers region, script, and
**register** (diglossia: standard vs colloquial/creole) variants as red-team
targets, each resolving to a frequency band:

```python
from kintsugi import locales, RedTeam, OllamaProvider

locales.diglossia_pairs()            # e.g. ('en-SG','en-SG-x-singlish'), ('de-CH','gsw'), ('ar','ar-EG')

std      = locales.get("en-SG")              # Singapore Standard English
singlish = locales.get("en-SG-x-singlish")   # Singlish (creole)
rt = RedTeam(OllamaProvider("qwen3.5:0.8b"),
             source_lang="English", target_lang=singlish["name"], lang_pair="en-en")
```

Covered: region (`pt-BR`/`pt-PT`, `fr-CA`/`fr-CH`, `de-AT`/`de-CH`, `zh-CN`/
`zh-TW`/`zh-HK`/`zh-MO`/`zh-SG`, ...), script (`zh-Hans`/`zh-Hant`, `sr-Cyrl`/
`sr-Latn`, `ms-Latn`/`ms-Arab`, `hi-Deva`/`hi-Latn`, ...), and register
(Singapore Standard English vs Singlish, Indian English vs Hinglish, Jamaican
Standard English vs Patois, Swiss Standard German vs Schweizerdeutsch, MSA vs
Egyptian/Moroccan Arabic, HK written Chinese vs spoken Cantonese).

```bash
kintsugi list-locales --register creole
kintsugi list-locales --script Hant
kintsugi frequency --target ollama:qwen3.5:0.8b --lang-of zh-Hant --band rare --n 30
```

The full Google Translate / Translated / Apple coverage catalogues are reference
material in [`docs/LANGUAGE_COVERAGE.md`](docs/LANGUAGE_COVERAGE.md), kept out of
the code on purpose (that list is never-ending).

## Command line

```bash
kintsugi list                                            # all attack categories
kintsugi list-framings                                   # all translator framings
kintsugi transform zero_width "Do not take with alcohol"
kintsugi attack  --target ollama:qwen3.5:0.8b --judge anthropic:claude-opus-5-5 \
                 --lang en-ja --category japanese_pharma_traps --out findings.jsonl
kintsugi frequency --target ollama:qwen3.5:0.8b --direction en-ja --band rare --n 30
```

## What the attacks find

A few headline numbers, drawn from the bundled observed rates:

| Attack | Pair | Catastrophic rate |
|---|---|---|
| `japanese_pharma_traps` | en-ja | ~51% |
| `anatomical_near_miss` | en-ja | 50% |
| `prompt_injection` | en-de | ~30% |
| `safety_critical_substance` | en-de | ~25% |

Vulnerability tracks resource tier almost perfectly: French is the hardest to
break, low-resource Indic and Cantonese targets the easiest. Models at or above
~4B parameters show near-zero catastrophic rates; models at or below ~2B should
not be used for safety-critical translation.

```python
import kintsugi
kintsugi.observed_rates()["japanese_pharma_traps"]["en-ja"]   # {'n': 79, 'catastrophic': 40, 'bad': ...}
```

## How judging works

1. **Local prefilter** (`kintsugi.prefilter`) classifies obvious failures for
   free and returns `None` for anything ambiguous.
2. **LLM judge** (`kintsugi.Judge`) scores the rest on adequacy, fluency,
   faithfulness, format and safety, and never returns an error verdict: on a
   parse failure it defaults to `ok` so statistics stay clean, and always-severe
   failure types (hallucination, refusal, prompt leak) are promoted.

A judge is optional. With no judge, kintsugi still runs the full prefilter and
profanity check, which already catch the most dangerous failure modes.

## License

Apache 2.0.
