Metadata-Version: 2.4
Name: bluepencil
Version: 0.1.0
Summary: A quality gate for prose. Catches the constructions that mark machine-written text, with thresholds calibrated against human writing.
Project-URL: Homepage, https://github.com/wolfvswhale/bluepencil
Project-URL: Source, https://github.com/wolfvswhale/bluepencil
Project-URL: Demo, https://huggingface.co/spaces/wolfvswhale/bluepencil
Project-URL: Harness, https://github.com/wolfvswhale/prose-eval
Project-URL: Measurements, https://huggingface.co/datasets/wolfvswhale/prose-cadence-stats
Author: J. Alderman Lyell
License: MIT
License-File: LICENSE
Keywords: ai-detection,editorial,linter,prose,quality-gate,style,writing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Provides-Extra: calibrate
Requires-Dist: datasets>=2.14; extra == 'calibrate'
Requires-Dist: huggingface-hub>=0.20; extra == 'calibrate'
Requires-Dist: pandas>=2.0; extra == 'calibrate'
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# bluepencil

A quality gate for prose. It fails 46.9% of machine-written documents and 6.3% of human-written ones, and both of those numbers were measured rather than hoped for.

```bash
pip install bluepencil
bluepencil check draft.md
```

Live demo, nothing to install: [huggingface.co/spaces/wolfvswhale/bluepencil](https://huggingface.co/spaces/wolfvswhale/bluepencil)

Exit code 1 when something fires at error severity, so it drops into CI, a pre-commit hook, or an agent loop without wrapping.

## Why the numbers are the point

Most style linters ship a list of banned phrases and a threshold someone picked because it felt about right.

You install it, it fires on your good writing, you uninstall it.

Every threshold here was set by measuring the distribution of that statistic across 2,602 human-written documents spanning three registers: informal forum answers, formal scientific abstracts, and multi-paragraph encyclopedia articles. Each cadence gate is placed at the percentile that makes it fire on 5% of human writing. That rate is a property of the tool, not a hope, and `bluepencil gates` will print it back to you.

The companion repo [prose-eval](https://github.com/wolfvswhale/prose-eval) is the harness that produced those distributions.

## What it catches

```
$ bluepencil check draft.md
draft.md
        3:1  error            throat-clearing
             "In today's fast-paced world"
             -> Delete it. Start with the first real sentence.
       3:44  error            negative-parallelism
             "is not just a nice-to-have, it's"
             -> State the correct thing. Leave the wrong thing out.
       4:24  error            editorializing
             "It's important to note"
             -> Just write the note.
          -  warn             flat-cadence
             Sentence lengths vary less than human writing typically does.
             Measured 0.198, below the calibrated threshold of 0.237.
             -> Rewrite one sentence in every three-sentence run to a
                different length.

1 file checked, 0 clean, 3 errors, 1 warning
```

Two kinds of finding, and they need different responses.

**Pattern findings** point at a span with a line and column. The fix is unambiguous and takes seconds.

**Cadence findings** describe the document. There is no line to fix; the draft needs a pass. These are the ones that catch prose which has had every flagged word swapped out and still reads as machine-written, which is the failure mode a phrase blocklist cannot see.

## Measured performance

Fire rates across 1,500 human and 1,500 machine documents, held out from calibration. "Lift" is how much more often a gate fires on machine text than human text.

| gate | human | machine | lift |
|---|---|---|---|
| editorializing | 0.1% | 18.5% | 139x |
| assistant-residue | 0.1% | 7.9% | 119x |
| hollow-conclusion | 1.1% | 21.8% | 20x |
| flat-cadence | 3.4% | 23.1% | 6.8x |
| triplet-habit | 4.7% | 17.4% | 3.7x |
| transition-crutch | 5.0% | 6.9% | 1.4x |
| repeated-openers | 5.1% | 5.9% | 1.2x |
| **any finding** | **31.3%** | **70.1%** | 2.2x |
| **fails at error severity** | **6.3%** | **46.9%** | 7.4x |

## Where it doesn't work, and why

Nine of the eighteen gates fire *more* often on human writing than on machine writing here. All of them:

| gate | human | machine | lift |
|---|---|---|---|
| copula-avoidance | 5.8% | 0.8% | 0.1x |
| em-dash-habit | 5.5% | 0.0% | 0.0x |
| negative-parallelism | 2.4% | 0.7% | 0.3x |
| participle-tail | 1.5% | 0.1% | 0.0x |
| uniform-paragraphs | 1.5% | 0.0% | 0.0x |
| vague-attribution | 0.8% | 0.1% | 0.1x |
| inflated-significance | 0.7% | 0.2% | 0.3x |
| promotional | 0.3% | 0.2% | 0.8x |
| engagement-bait | 0.1% | 0.0% | 0.0x |

The obvious reading is that those rules are wrong.

The likelier reading is that the corpus is old.

The corpus used here, HC3, was collected around December 2022. Em dashes are the signature of *current* models: a pre-registered study of 69,632 medRxiv preprints ([arXiv:2606.29540](https://arxiv.org/abs/2606.29540)) puts em-dash prevalence in Discussion sections at 4.23% before ChatGPT, 8.0% in 2024, and 20.3% in 2025. HC3 predates almost all of that rise, so it cannot contain the behaviour the gate looks for. Newer paired corpora built on current models do exist, including MIRAGE, MAGA-Bench, and AITDNA, and recalibrating against one of them is the obvious next step. Until then these gates stay on and stay labelled unvalidated rather than disproven.

`uniform-paragraphs` is a different problem: the machine half of the corpus is single-paragraph, so the statistic is unmeasurable there and its 0.0x is an artifact of the data, not a result.

One gate shipped disabled. `no-short-sentences` could not be calibrated: more than 5% of human documents contain no short sentence at all, so no threshold below the target false-positive rate can ever fire. Calibration detects that and disables it rather than shipping a gate that silently never triggers.

## Running it where you already are

**Pre-commit.** Add to `.pre-commit-config.yaml`:

```yaml
repos:
  - repo: https://github.com/wolfvswhale/bluepencil
    rev: v0.1.0
    hooks:
      - id: bluepencil
```

**GitHub Actions.** `--format github` emits annotations that render inline on the pull request diff:

```yaml
- run: pip install bluepencil
- run: bluepencil check . --format github --fail-on never
```

Drop `--fail-on never` when you want it to block merges.

**Claude Code and other agent tools.** Copy `skill/SKILL.md` into your skills directory. It tells the agent when to run bluepencil, how to read the two kinds of finding, and, importantly, when to overrule one.

**Anything else.** `--format json` gives you the findings with line numbers, measurements, thresholds, and suggested fixes.

## Configuring it

`bluepencil.toml`, discovered from the file upward:

```toml
[bluepencil]
fail_on = "error"        # error | warn | never
min_words = 120          # documents shorter than this skip cadence gates

[bluepencil.gates]
em-dash-habit = "off"    # off | warn | error
promotional = "error"

[bluepencil.budgets]
transition-crutch = 6.0  # occurrences per 1000 words tolerated

[bluepencil.thresholds]
flat-cadence = 0.30      # override a calibrated value
```

Roughly one human document in twenty gets a cadence finding a good editor would overrule. That is the 5% target working as designed. Turn the gate off rather than mangling the sentence.

## Recalibrating for your own writing

The shipped thresholds describe general human prose. If your house style is genuinely different, point the calibration at your own corpus:

```bash
pip install "bluepencil[calibrate]"
python scripts/calibrate.py --target-fpr 0.05
python scripts/measure.py --limit 1500
```

The first writes `bluepencil/thresholds.json`. The second regenerates the tables above.

## What it will not tell you

Whether the text is true, whether the argument holds, or who wrote it. It measures the surface. A document can pass every gate and still be empty, and a good writer will sometimes trip three gates on purpose.

It is a gate, not a classifier. A document failing means it contains constructions the standard rejects, which is true of plenty of human writing, as the 6.3% says.

## Sources

Rules derived from Wikipedia's [Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing), maintained by WikiProject AI Cleanup, plus a house editing standard. Calibration corpora: [HC3](https://huggingface.co/datasets/Hello-SimpleAI/HC3), [AI-and-Human-Generated-Text](https://huggingface.co/datasets/Ateeqq/AI-and-Human-Generated-Text), and [wikitext-103](https://huggingface.co/datasets/Salesforce/wikitext).

MIT licensed. Built by J. Alderman Lyell ([@wolfvswhale](https://github.com/wolfvswhale)).
