Metadata-Version: 2.4
Name: colloquial
Version: 0.1.1
Summary: Rewrite machine text into colloquial, native-sounding language — humanize in the same language or translate-and-humanize, in any tongue.
Author: XScryer
License-Expression: MIT
Project-URL: Homepage, https://github.com/XScryer/colloquial
Project-URL: Repository, https://github.com/XScryer/colloquial
Project-URL: Issues, https://github.com/XScryer/colloquial/issues
Project-URL: Changelog, https://github.com/XScryer/colloquial/blob/main/CHANGELOG.md
Keywords: nlp,writing,humanize,translation,llm,slop
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: bandit>=1.7; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Dynamic: license-file

# Colloquial

[![PyPI](https://img.shields.io/pypi/v/colloquial)](https://pypi.org/project/colloquial/)
[![Python versions](https://img.shields.io/pypi/pyversions/colloquial)](https://pypi.org/project/colloquial/)
[![MIT license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
<!-- Add the CI badge once the repository is public. GitHub does not serve
     workflow badges for private repositories, and a broken image is worse than
     no image. -->

Rewrite machine text into colloquial, native-sounding language, in any tongue.

Colloquial takes text written by an LLM and produces text that reads like a
person wrote it. It works in the same language (`--same-lang`) or translates and
humanizes in one pass (`--to <lang>`). A deterministic detection core (no
network, no LLM) scores how machine a text sounds, a report ranks the worst
sections first, and an optional self-check loop re-evaluates the output.

The name says it. Colloquial. How people actually talk and write.

## Why this exists

Machine-written prose has a signature, and it is not just word choice. Research
on 61,608 stories (StoryScope, COLM 2026) shows that narrative features alone
separate human from machine text at 93.2% macro-F1, and that these structural
patterns survive paraphrasing. Surface detectors break when the wording
changes. The structure does not.

Colloquial is built on that finding. It reads how a text is built and rewrites
it the way a person would have built it. The result is not text with words
swapped. It is text with a different skeleton.

## What it does

| Command | What you get |
|---|---|
| `colloquial detect FILE` | A machine-likeness score from 0 to 100, ranked findings, and the worst sections first |
| `colloquial humanize FILE` | A rewrite that reads human, in the same language, offline and deterministic |
| `colloquial humanize FILE --mode llm` | The same rewrite driven by your own model, with a style profile built from the research |
| `colloquial humanize FILE --to ar` | Translate and humanize in one pass, without translationese |
| `colloquial humanize FILE --self-check` | Re-detect the output and iterate when it still reads machine-made |
| `colloquial mcp` | The engine as an MCP server, for Claude Code, Codex, Cursor and Hermes |

## How it works

Five layers, cheapest first. Each one feeds the next, and every layer is
grounded in published measurements.

1. **Surface rules.** Filler transitions, inflated significance, machine
   punctuation, AI self-reference. Hundreds of rules across eight languages.
   This layer is deliberately the weakest, because wordlists alone produce
   false positives on legitimate human text.
2. **Rhythm.** Burstiness (the coefficient of variation of sentence length),
   type-token ratio, trigram repetition. Machine text keeps an even tempo.
   People do not.
3. **Lexical diversity.** Six dimensions from arXiv 2508.00086, including
   dispersion (how close repeated words sit) and evenness (how uniformly
   vocabulary spreads). The counter-intuitive finding of that paper is that
   machine text is *more* lexically varied than human text and repeats words
   at *longer* distances. Dispersion alone carries a SHAP weight of 0.279,
   the strongest single discriminator measured.
4. **Discourse.** A ten-dimension narrative profile ported from StoryScope
   (revelation, events, perspective, plot, setting, style, situatedness,
   temporal structure, agents, social networks). It measures how the text is
   constructed, not what words it uses, and it produces a soft per-model
   fingerprint (Claude-like flat escalation, GPT-like gossip as plot
   mechanism, Gemini-like external description) on texts above 300 words.
5. **Persona.** AI-assisted writing makes the writer seem more confident, more
   positive and more privileged than they are (arXiv 2604.22503). This layer
   strips the tells and keeps the writer's actual voice.

The mechanical rewriter reverses what the detector finds, offline. The LLM
rewriter sends the text with a style profile built from the same research, so
either way the rules behind the rewrite are the same rules behind the score.

## The science behind it

Colloquial is not a prompt pack with a CLI glued on. Every layer cites its
source, and the sources are peer-reviewed.

| Source | What it contributed |
|---|---|
| **StoryScope** (arXiv 2604.03136, COLM 2026) | The ten discourse dimensions, the per-model fingerprints, and the finding that structural features survive paraphrasing (93.2% macro-F1 on 61,608 stories) |
| **Lexical diversity** (arXiv 2508.00086) | The six diversity dimensions and their AI/human directions; dispersion as the strongest discriminator |
| **AI Argues Differently** (arXiv 2604.22503, EMNLP 2025) | The persona-distortion catalog: over-confidence, forced positivity, compressed emotional range |
| **The Arabic AI Fingerprint** (arXiv 2505.23276, KFUPM) | Arabic stylometric tells: overused high-frequency words, lower vocabulary spread, domain-specific signatures |
| **Falsely Accused** (arXiv 2511.16690) | The calibration lesson: detectors misjudge slightly-polished Arabic (92% to 12% accuracy on polished text), so the Arabic profile fails open instead of accusing |
| **"Signs of AI writing"** (Wikipedia, WikiProject AI Cleanup) | The shared heritage behind the surface rule sets |

Two design commitments follow from the research. First, detection is
deterministic and offline, so a score is reproducible and a rewrite never
depends on a network call. Second, the tool fails open. It prefers a missed
detection to a false accusation, because the real cost of an AI detector is
sending a human's honest text back to them marked as machine-made.

## What makes it different

An honest comparison with the tools that came before.

- **Eight languages with real machinery.** English, Portuguese (Brazil),
  Arabic (neutral MSA), Chinese, Japanese, Korean, Hindi and Ukrainian. Each
  language ships its own surface rules, discourse lexicons, style profile and
  rewrite transforms. The Arabic profile includes a dialect-consistency lock.
  Egyptian, Levantine, Gulf, Maghrebi and Iraqi markers are rewritten to
  neutral MSA, so a text never comes back as a dialectal mix. The Ukrainian
  profile does the same for Russian calques, known as surzhyk.
- **Discourse, not just vocabulary.** Most humanizers work at the word level.
  Colloquial reads event escalation, dialogue balance, theme explicitness and
  temporal texture, because that is where the research says the signal lives.
- **Per-model fingerprints.** The report says which family the text resembles.
  This is a soft signal, never a verdict.
- **Two rewrite paths from one rulebook.** The offline path needs no key, no
  account and no network. The LLM path uses whatever provider the user
  configures, and both are judged by the same detector.
- **Zero runtime dependencies.** Standard library only, by design. The quality
  wall (ruff, mypy strict, bandit, pytest at 80% coverage minimum) runs on
  every push in CI.

## Install

```bash
pip install colloquial
```

Or run it without installing anything:

```bash
uvx colloquial detect README.md
```

## Quick start

```bash
colloquial detect README.md
colloquial detect docs/ --report --format json
colloquial humanize draft.md --lang pt-br --report
colloquial humanize draft.md --mode llm --self-check
colloquial humanize draft.md --to ar
colloquial langs
```

## Using it from coding agents (MCP)

Colloquial runs as an MCP server, so any MCP-capable agent (Claude Code,
Codex, Cursor, Cline, Hermes) can call it as native tools.

```bash
colloquial mcp
```

Tools: `detect_text`, `humanize_text`, `translate_humanize`, `list_languages`,
`get_report`. See [docs/plugins.md](docs/plugins.md) for per-agent setup, and
`skills/` for ready-made skill files.

## Choosing a provider (llm mode)

The mechanical rewriter is offline and needs nothing. The `--mode llm`
rewriter runs against any OpenAI-compatible API, and the provider is always
your choice. Your key, your model, your machine.

```bash
colloquial providers          # see the catalog
colloquial setup              # interactive: pick provider, key, model
colloquial models             # live model list from your provider
colloquial config             # show the resolved configuration
```

`colloquial setup` writes `~/.config/colloquial/config.toml` (mode 0600).
It queries the provider's live `/models` endpoint with your key, so the
model list is always current. Local providers (Ollama, vLLM, llama.cpp,
LM Studio) need no key at all.

The catalog covers OpenAI, Anthropic (Claude), Google Gemini, xAI (Grok),
Mistral, DeepSeek, Groq, Cerebras, MiniMax, Moonshot (Kimi), Z.ai (GLM),
Alibaba Qwen, Nous Research, OpenRouter, Together AI, Fireworks AI, and
the local servers above, plus a `custom` entry for any other
OpenAI-compatible endpoint.

Environment variables still work and take precedence over the config file:
`COLLOQUIAL_PROVIDER`, `COLLOQUIAL_API_KEY` (or `OPENAI_API_KEY`),
`COLLOQUIAL_BASE_URL`, `COLLOQUIAL_MODEL`.

## Supported languages

English (en) · Arabic (ar, neutral MSA) · Chinese (zh) · Japanese (ja) ·
Korean (ko) · Hindi (hi) · Ukrainian (uk) · Portuguese (pt-BR)

The Arabic, Chinese, Japanese, Korean, Hindi and Ukrainian profiles ship as
community-validated beta. Their rule sets were researched against public
sources but deserve review from native speakers. Pull requests that sharpen a
lexicon, add a tell, or fix a false positive are the most valuable
contribution this project can receive.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). The short version is that every change
runs through the quality wall, and language profiles need a minimum test
corpus.

## License

MIT
