Metadata-Version: 2.5
Name: lexaudit
Version: 0.1.0
Summary: Locale vocabulary audit: inventeds, calques, foreign slips; page ranking for rewrite agents
Project-URL: Homepage, https://github.com/nareyko/lexaudit
Project-URL: Issues, https://github.com/nareyko/lexaudit/issues
Project-URL: Documentation, https://github.com/nareyko/lexaudit/tree/main/docs
Author: Vadim Nareyko
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Keywords: calque,false-friends,i18n,localization,markdown,nlp,proofreading,seo,spellcheck
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Natural Language :: English
Classifier: Natural Language :: German
Classifier: Natural Language :: Polish
Classifier: Natural Language :: Russian
Classifier: Natural Language :: Spanish
Classifier: Natural Language :: Swedish
Classifier: Natural Language :: Ukrainian
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.12
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.27
Requires-Dist: jsonschema>=4.23
Requires-Dist: pymorphy3-dicts-ru>=2.4
Requires-Dist: pymorphy3-dicts-uk>=2.4
Requires-Dist: pymorphy3>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: regex>=2024.11
Requires-Dist: spylls>=0.1.7
Provides-Extra: dev
Requires-Dist: hypothesis>=6.115; extra == 'dev'
Requires-Dist: libcst>=1.4; extra == 'dev'
Requires-Dist: mutmut<3,>=2.4; extra == 'dev'
Requires-Dist: mypy>=1.13; extra == 'dev'
Requires-Dist: pytest-cov>=6.0; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: radon>=6.0; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Requires-Dist: types-jsonschema>=4.23; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

# lexaudit

**Finds odd, wrong or copied-from-another-language words in your texts, and tells you which pages to fix first.**

## In plain words

You write or translate articles, courses or documentation. A normal spellchecker shouts about every brand name, abbreviation and long German word, and still misses the words that really hurt: a word copied from another language ("calque"), a real word used with the wrong meaning ("false friend"), or a word someone invented.

lexaudit reads your files, ignores the harmless things, and gives you a short ranked list: "look at this page first." It runs on an ordinary laptop in about a second. No graphics card, no internet, no account.

It only reports. It never changes your texts.

Unknown terms? See the [glossary](docs/glossary.md).

- **Languages:** English (`en`), German (`de`), Spanish (`es`), Polish (`pl`), Russian (`ru`), Swedish (`sv`), Ukrainian (`uk`)
- **Hardware:** CPU only
- **Author:** Vadim Nareyko ([github.com/nareyko](https://github.com/nareyko))
- **License:** [AGPL-3.0-or-later](LICENSE)

![How lexaudit works](docs/images/lexaudit-overview.jpg)

---

## Try it in 60 seconds

### 1. Install and get the dictionaries

```bash
uv sync
uv run lexaudit fetch-dicts     # downloads 7 spelling dictionaries, no admin rights
uv run lexaudit doctor          # shows what is ready and what is missing
```

`fetch-dicts` saves dictionaries in your user cache folder. `doctor` confirms that no GPU is needed and lists the installed languages. If the system `hunspell` program is installed, lexaudit uses it because it is a little faster. If not, a built-in Python checker is used, so nothing else has to be installed.

### 2. Check a file or a whole project

```bash
# One file or folder, no settings file needed
uv run lexaudit audit --path ./README.md --locale en

# Or create starter settings first, then check everything
uv run lexaudit init
uv run lexaudit audit --locale en
```

### 3. Open the report

The reports are in the output folder, one subfolder per language. Start with `queue.md`:

```markdown
| score | unique_suspects | path | top_words |
|---|---|---|---|
| 12.0 | 2 | docs/en/en-04-defect-known-calques.md | orientir surveyness |
| 6.0 | 1 | docs/en/en-05-defect-false-friend.md | denseness |
| 1.6 | 2 | docs/en/en-03-clean-unseen-jargon.md | vinylester haulout |
```

A higher score means "look here first". `top_words` shows the words behind the score.

### 4. Ask why a word was flagged

```bash
uv run lexaudit explain orientir --locale en
```

```text
word: orientir (normalized: orientir, locale: en)
status: calque (tier: confirmed)
reason: loan-calque-orientir | detectors: calque, hunspell
editorial hint: working rule / ballpark / figures - not orientir
```

### 5. Reproduce our test results

```bash
uv run lexaudit audit --config examples/synthetic-demo/lexaudit.config.yaml
```

Then open `tmp/synthetic-demo-out/<language>/queue.md`.

---

## How good is it? An honest answer

We built a 42-page test set in all 7 languages: 21 clean technical pages and 21 pages with planted problems. Some problems are known to the rules, some are brand new words the rules have never seen. Details: [docs/synthetic-benchmark.md](docs/synthetic-benchmark.md).

![Benchmark results](docs/images/lexaudit-synthetic-benchmark.jpg)

| Setup | Bad page in top 2 | Bad page in top 3 | MAP (0 to 1, higher is better) | Known problems found | Brand new problems found | Time |
|---|---:|---:|---:|---:|---:|---:|
| A. Plain spellchecker | 0.0% | 23.8% | 0.389 | 0 of 26 | 0 of 33 | 0.54 s |
| B. Plus your word lists | 7.1% | 23.8% | 0.421 | 0 of 26 | 0 of 33 | 0.62 s |
| C. Full lexaudit | 100.0% | 66.7% | 0.838 | 26 of 26 | 26 of 33 | 0.66 s |

What this means:

- Pages with known problems come first in every language. The plain spellchecker puts clean pages first.
- About 78% of all planted problems are found, including words the rules never saw. About 22% are missed.
- Weak spot: a clean page with a repeated new technical term can outrank a page with a single new invented word. Adding the term to your word list, or using the optional `label` step, fixes this.

---

## Documentation

Start with [`docs/README.md`](docs/README.md).

| Guide | What you learn |
|---|---|
| [Glossary](docs/glossary.md) | Every term in plain words |
| [Benchmark](docs/synthetic-benchmark.md) | The 42-page test, step by step |
| [Architecture](docs/architecture.md) | What goes in, what comes out |
| [Pipeline](docs/pipeline.md) | The gates every word passes |
| [CLI reference](docs/cli-reference.md) | All commands |
| [Agent workflow](docs/agent-workflow.md) | From report to fixed page |
| [Scores and metrics](docs/weights-and-metrics.md) | How pages are scored |

![Word classification ladder](docs/images/lexaudit-classify-pipeline.jpg)

---

## More commands

```bash
# Re-rank pages, filter by source (docs, academy, model_pack)
uv run lexaudit rank --locale ru --mode weighted --sources=docs

# Add every page that contains a known bad word to your to-do list
uv run lexaudit expand-needle --needle orientir --locale all \
  --write-queue /abs/path/to/rewrite-queue.md

# Optional: ask a language model for a second opinion (your own endpoint)
uv run lexaudit label --locale ru --top-pages 20 --top-words 50 --tiers likely,weak --profile cpu

# Compare two runs
uv run lexaudit diff --baseline .../report.json --current .../report.json --out /tmp/diff.json
```

---

## For contributors

Setup and checks:

```bash
uv sync --extra dev
uv run ruff check . && uv run ruff format --check .
uv run mypy --strict src/
uv run pytest
uv run python scripts/crap_report.py
uv run mutmut run && uv run python scripts/check_mutation_score.py --min-percent 80
uv build
```

Test layers: unit and integration tests (branch coverage 80% or more), the 7-language benchmark, property-based fuzzing (Hypothesis), golden regression with a read-only check of your files, JSON Schema contract, complexity gate (CRAP), mutation testing (kill rate 80% or more).

Pictures in `docs/images` are generated from code: run `python scripts/render_diagrams.py` (needs Inkscape and Pillow). Never edit the JPG files by hand.

## Origin and author

Created and maintained by Vadim Nareyko ([github.com/nareyko](https://github.com/nareyko)). It grew out of auditing multilingual courses and encyclopedias at [FairHelm](https://fairhelm.com).
