Metadata-Version: 2.4
Name: colloquial
Version: 0.1.13
Summary: Rewrite machine text into colloquial, native-sounding language — humanize in the same language or translate-and-humanize, in any tongue.
Author: XScryer
License-Expression: MIT
Project-URL: Homepage, https://github.com/XScryer/colloquial
Project-URL: Repository, https://github.com/XScryer/colloquial
Project-URL: Issues, https://github.com/XScryer/colloquial/issues
Project-URL: Changelog, https://github.com/XScryer/colloquial/blob/main/CHANGELOG.md
Keywords: nlp,writing,humanize,translation,llm,slop
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: bandit>=1.7; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Dynamic: license-file

# Colloquial

[![PyPI](https://img.shields.io/pypi/v/colloquial)](https://pypi.org/project/colloquial/)
[![Python versions](https://img.shields.io/pypi/pyversions/colloquial)](https://pypi.org/project/colloquial/)
[![MIT license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
<!-- Add the CI badge once the repository is public. GitHub does not serve
     workflow badges for private repositories, and a broken image is worse than
     no image. -->

Rewrite machine text into colloquial, native-sounding language, in any tongue.

Colloquial takes text written by an LLM and produces text that reads like a
person wrote it. It works in the same language (`--same-lang`) or translates and
humanizes in one pass (`--to <lang>`). A deterministic detection core (no
network, no LLM) scores how machine a text sounds, a report ranks the worst
sections first, and an optional self-check loop re-evaluates the output.

The name says it. Colloquial. How people actually talk and write.

## Why this exists

Machine-written prose has a signature, and it is not just word choice. Research
on 61,608 stories (StoryScope, COLM 2026) shows that narrative features alone
separate human from machine text at 93.2% macro-F1, and that these structural
patterns survive paraphrasing. Surface detectors break when the wording
changes. The structure does not.

Colloquial is built on that finding. It reads how a text is built and rewrites
it the way a person would have built it. The result is not text with words
swapped. It is text with a different skeleton.

## What it does

| Command | What you get |
|---|---|
| `colloquial detect FILE` | A machine-likeness score from 0 to 100, ranked findings, and the worst sections first |
| `colloquial humanize FILE` | A rewrite that reads human, in the same language, on the provider you configured |
| `colloquial humanize FILE --to ar` | Translate and humanize in one pass, without translationese |
| `colloquial humanize FILE --self-check` | Re-detect the output and iterate when it still reads machine-made |
| `colloquial mcp` | The engine as an MCP server, for Claude Code, Codex, Cursor and Hermes |

## How it works

Five layers, cheapest first. Each one feeds the next, and every layer is
grounded in published measurements.

1. **Surface rules.** Filler transitions, inflated significance, machine
   punctuation, AI self-reference. Hundreds of rules across eight languages.
   This layer is deliberately the weakest, because wordlists alone produce
   false positives on legitimate human text.
2. **Rhythm.** Burstiness (the coefficient of variation of sentence length),
   type-token ratio, trigram repetition. Machine text keeps an even tempo.
   People do not.
3. **Lexical diversity.** Six dimensions from arXiv 2508.00086, including
   dispersion (how close repeated words sit) and evenness (how uniformly
   vocabulary spreads). The counter-intuitive finding of that paper is that
   machine text is *more* lexically varied than human text and repeats words
   at *longer* distances. Dispersion alone carries a SHAP weight of 0.279,
   the strongest single discriminator measured.
4. **Discourse.** A ten-dimension narrative profile ported from StoryScope
   (revelation, events, perspective, plot, setting, style, situatedness,
   temporal structure, agents, social networks). It measures how the text is
   constructed, not what words it uses. It reports no per-model attribution:
   measured on pt-BR, that signal labelled 4 of 4 non-Claude texts "claude".
5. **Persona.** AI-assisted writing makes the writer seem more confident, more
   positive and more privileged than they are (arXiv 2604.22503). This layer
   strips the tells and keeps the writer's actual voice.

The rewriter is the detector reversed, and it runs in three passes. A
deterministic pass strips the surface tells before the model sees the text. The
model then rewrites it with a style profile built from the same research that
feeds the score. A second deterministic pass guarantees that a tell the model
reintroduced does not survive to the output. Detection never needs a provider.
Rewriting always does, and it is your provider.

## The science behind it

Colloquial is not a prompt pack with a CLI glued on. Every layer cites its
source, and the sources are peer-reviewed.

| Source | What it contributed |
|---|---|
| **StoryScope** (arXiv 2604.03136, COLM 2026) | The ten discourse dimensions and the finding that structural features survive paraphrasing (93.2% macro-F1 on 61,608 stories) |
| **Lexical diversity** (arXiv 2508.00086) | The six diversity dimensions and their AI/human directions; dispersion as the strongest discriminator |
| **AI Argues Differently** (arXiv 2604.22503, EMNLP 2025) | The persona-distortion catalog: over-confidence, forced positivity, compressed emotional range |
| **The Arabic AI Fingerprint** (arXiv 2505.23276, KFUPM) | Arabic stylometric tells: overused high-frequency words, lower vocabulary spread, domain-specific signatures |
| **Falsely Accused** (arXiv 2511.16690) | The calibration lesson: detectors misjudge slightly-polished Arabic (92% to 12% accuracy on polished text), so the Arabic profile fails open instead of accusing |
| **"Signs of AI writing"** (Wikipedia, WikiProject AI Cleanup) | The shared heritage behind the surface rule sets |

Two design commitments follow from the research. First, detection is
deterministic and offline, so a score is reproducible and a rewrite never
depends on a network call. Second, the tool fails open. It prefers a missed
detection to a false accusation, because the real cost of an AI detector is
sending a human's honest text back to them marked as machine-made.

## What makes it different

An honest comparison with the tools that came before.

- **Eight languages with real machinery.** English, Portuguese (Brazil),
  Arabic (neutral MSA), Chinese, Japanese, Korean, Hindi and Ukrainian. Each
  language ships its own surface rules, discourse lexicons, style profile and
  rewrite transforms. The Arabic profile includes a dialect-consistency lock.
  Egyptian, Levantine, Gulf, Maghrebi and Iraqi markers are rewritten to
  neutral MSA, so a text never comes back as a dialectal mix. The Ukrainian
  profile does the same for Russian calques, known as surzhyk.
- **Discourse, not just vocabulary.** Most humanizers work at the word level.
  Colloquial reads event escalation, dialogue balance, theme explicitness and
  temporal texture, because that is where the research says the signal lives.
- **Punctuation is judged in context.** A dash that marks speech or sets off an
  aside is pt-BR punctuation, not a tell: `colloquial` reports and rewrites only
  the dashes dropped into running prose. A dash used to mark dialogue survives
  humanization untouched, because flattening it merges the speech into the
  narration and loses who was talking.
- **Two rewrite paths from one rulebook.** The offline path needs no key, no
  account and no network. The LLM path uses whatever provider the user
  configures, and both are judged by the same detector.
- **Zero runtime dependencies.** Standard library only, by design. The quality
  wall (ruff, mypy strict, bandit, pytest at 80% coverage minimum) runs on
  every push in CI.

## Install

```bash
pip install colloquial
```

Or run it without installing anything.

```bash
uvx colloquial detect README.md
```

If you prefer `uv tool install colloquial`, the executable lands in
`~/.local/bin` (on Windows, `C:\Users\<you>\.local\bin`), which `uv` may not
add to your PATH. Run `uv tool update-shell`, or add that directory to PATH
yourself.

## Quick start

```bash
colloquial detect README.md
colloquial detect docs/ --report --format json
colloquial humanize draft.md --lang pt-br --report
colloquial humanize draft.md --self-check
colloquial humanize draft.md --to ar
colloquial langs
```

## Using it from coding agents (MCP)

Colloquial runs as an MCP server, so any MCP-capable agent (Claude Code,
Codex, Cursor, Cline, Hermes) can call it as native tools.

```bash
colloquial mcp
```

Tools: `detect_text`, `humanize_text`, `translate_humanize`, `list_languages`,
`get_report`. See [docs/plugins.md](docs/plugins.md) for per-agent setup, and
`skills/` for ready-made skill files.

## Choosing a provider (rewriting)

Detection is offline and needs nothing. Rewriting runs against any
OpenAI-compatible API, and the provider is always your choice. Your key, your
model, your machine. Without a provider configured, `colloquial humanize` stops
with a clear message instead of pretending to work.

```bash
colloquial providers          # see the catalog
colloquial setup              # interactive: pick provider, key, model
colloquial models             # live model list from your provider
colloquial config             # show the resolved configuration
```

`colloquial setup` writes `~/.config/colloquial/config.toml` (mode 0600).
It queries the provider's live `/models` endpoint with your key, so the
model list is always current. Local providers (Ollama, vLLM, llama.cpp,
LM Studio) need no key at all.

The catalog covers OpenAI, Anthropic (Claude), Google Gemini, xAI (Grok),
Mistral, DeepSeek, Groq, Cerebras, MiniMax, Moonshot (Kimi), Z.ai (GLM),
Alibaba Qwen, Nous Research, OpenRouter, Together AI, Fireworks AI, and
the local servers above, plus a `custom` entry for any other
OpenAI-compatible endpoint.

Environment variables still work and take precedence over the config file:
`COLLOQUIAL_PROVIDER`, `COLLOQUIAL_API_KEY` (or `OPENAI_API_KEY`),
`COLLOQUIAL_BASE_URL`, `COLLOQUIAL_MODEL`.

## Is it safe to give it my API key?

That is the right question to ask, and it deserves a checkable answer rather
than a reassuring one. Here is what the code does, and how to confirm it in a
few minutes.

- **Detection needs no key and opens no socket.** `colloquial detect` and the
  reports are pure local computation, forever, with nothing configured.
- **There is exactly one outbound HTTP client in the package**, in
  `src/colloquial/rewrite/llm.py`. It calls two endpoints, `GET /models` and
  `POST /chat/completions`, on the base URL you configured, and the scheme is
  validated to be `http` or `https` before any request.
- **The request carries your text and nothing else.** The headers are
  `Content-Type` and `Authorization: Bearer <key>`. The body is the model, the
  style prompt and your text. No telemetry, no identifiers, no analytics, no
  call-home.
- **The key lives where you choose.** Preferred is `COLLOQUIAL_API_KEY` in the
  environment. Otherwise `~/.config/colloquial/config.toml`, written with mode
  0600 on POSIX. Windows has no POSIX modes, so there the file sits under your
  user profile and the environment variable is the safer choice. No command-line
  flag accepts a key, so it cannot leak into your shell history, and `colloquial
  config` prints it masked.
- **Zero dependencies.** Nothing else is installed, so no third-party code
  stands between your key and the provider. There is no `setup.py` either, which
  means installing the package runs no project code.
- **A guard test keeps it that way.** `tests/test_supply_chain_guard.py` fails
  the build if a module outside that single client imports a network stack,
  spawns a process, calls `eval`, or if any third-party import appears at all.
- **You can skip the cloud entirely.** A local model (Ollama, vLLM, llama.cpp,
  LM Studio on `http://localhost`) needs no key and no account.

To check the published artifact rather than the source, make PyPI prove where it
came from. Releases are published from this repository by GitHub Actions, so the
claim is signed and verifiable.

```bash
pip install pypi-attestations
WHEEL=$(python -c "import json,urllib.request as u; d=json.load(u.urlopen('https://pypi.org/pypi/colloquial/json')); print([f['url'] for f in d['urls'] if f['filename'].endswith('.whl')][0])")
pypi-attestations verify pypi --repository https://github.com/XScryer/colloquial "$WHEEL"
```

It prints `OK` when the wheel on PyPI was built and uploaded by this repository.
[SECURITY.md](SECURITY.md) documents the rest, including what falls inside the
scope of a report.

## Supported languages

English (en) · Arabic (ar, neutral MSA) · Chinese (zh) · Japanese (ja) ·
Korean (ko) · Hindi (hi) · Ukrainian (uk) · Portuguese (pt-BR)

The Arabic, Chinese, Japanese, Korean, Hindi and Ukrainian profiles ship as
community-validated beta. Their rule sets were researched against public
sources but deserve review from native speakers. Pull requests that sharpen a
lexicon, add a tell, or fix a false positive are the most valuable
contribution this project can receive.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). The short version is that every change
runs through the quality wall, and language profiles need a minimum test
corpus.

## License

MIT
