Metadata-Version: 2.4
Name: k-uncanny
Version: 0.1.1
Summary: CLI tool that detects AI-generated text with sentence-level scoring and multiple detection strategies
Keywords: ai-detection,gpt,llm,text-analysis,cli
Author: Frank Chan
License-Expression: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: typer>=0.9
Requires-Dist: rich>=13.0
Requires-Dist: pytest>=7.0 ; extra == 'dev'
Requires-Dist: pytest-cov>=4.0 ; extra == 'dev'
Requires-Dist: torch>=2.0 ; extra == 'ml'
Requires-Dist: transformers>=4.30 ; extra == 'ml'
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/kpihx-labs/uncanny
Project-URL: Repository, https://github.com/kpihx-labs/uncanny
Project-URL: Upstream, https://github.com/dunkinfrunkin/uncanny
Project-URL: Issues, https://github.com/kpihx-labs/uncanny/issues
Provides-Extra: dev
Provides-Extra: ml
Description-Content-Type: text/markdown

# Uncanny

## KpihX-Labs distribution

This MIT-licensed adaptation retains Frank Chan's upstream implementation and history.
Upstream: https://github.com/dunkinfrunkin/uncanny.
The PyPI distribution is `k-uncanny`; the executable and Python import remain `uncanny`.
Install with `uv tool install k-uncanny`, or use `make uv-link` from this checkout.
Run `uncanny scan document.txt --format json` for structured output on stdout.
Use `make check`, `make uv-build`, `make uv-publish`, and `make push`.
`make puhsb` is an alias for `make push`, which pushes the current branch to both
`github` and `gitlab`, never to `upstream`.
This detector produces heuristic scores, not calibrated authorship probabilities.
No score proves who wrote a document or whether plagiarism occurred.

The remaining documentation describes the upstream tool. Install instructions naming
the upstream PyPI package do not install this KpihX-Labs distribution.

**Detect AI-generated text from your terminal.** Sentence-level scoring, multiple detection strategies, zero API calls. Runs locally, works offline.

```
$ uncanny scan proposal.md

  Uncanny Analysis — proposal.md
  ─────────────────────────────────────────────────

  Overall Score: 0.73 (likely AI-generated)

  ██████████████████████░░░░░░░░  73% AI probability

  Sentence Breakdown:
   0.91 │ The implementation of this feature requires careful
        │ consideration of several architectural patterns.
   0.42 │ I tried like three different approaches before
        │ landing on this one, honestly.
   0.88 │ By leveraging the existing infrastructure, we can
        │ ensure seamless integration with minimal overhead.

  Legend: ■ 0.8+ AI  ■ 0.5-0.8 mixed  ■ <0.5 human
```

## Why Uncanny?

Every other AI detector is a SaaS that sends your text to someone else's server. Uncanny runs **entirely on your machine**. No API keys, no subscriptions, no data leaving your network.

| Tool | Open Source | Offline | Sentence-level | Multi-method | CLI |
|------|:----------:|:-------:|:--------------:|:------------:|:---:|
| GPTZero | | | x | x | |
| Originality.ai | | | x | x | |
| Copyleaks | | | x | x | |
| Sapling | | | x | | |
| Zippy | x | x | | | x |
| **Uncanny** | **x** | **x** | **x** | **x** | **x** |

## Benchmarks

Tested against 36 labeled samples (18 AI-generated, 18 human-written) across diverse genres: academic papers, Slack messages, code reviews, personal essays, Reddit posts, cover letters, tweets, and more.

| Metric | Uncanny | Zippy | Random Baseline |
|--------|:-------:|:-----:|:---------------:|
| **Overall Accuracy** | **91.7%** | 80.6% | 50.0% |
| AI Detected (true positive) | 88.9% | 100% | 100% |
| Human Cleared (true negative) | **94.4%** | 61.1% | 0% |
| **False Accusation Rate** | **5.6%** | 38.9% | 100% |
| Speed (per sample) | 8ms | 1ms | 0ms |

**Uncanny beats Zippy by 11 points on accuracy.** Zippy catches everything but wrongly flags 39% of human text as AI — that means 2 out of 5 human writers get falsely accused. Uncanny's false positive rate is 7x lower.

> Run the benchmarks yourself: `uncanny bench --compare`

### How we test

The corpus includes adversarial cases:

- **Casual AI** — LLM output prompted to sound informal ("Hey! So basically...")
- **Formal humans** — Academic writing, technical docs, polished prose
- **Short text** — 2-3 sentence samples where signal is weakest
- **Edge cases** — LinkedIn posts, PR descriptions, commit messages, debug notes

Accuracy is enforced in CI. Every PR must pass `test_bench.py` which gates on:
- Overall accuracy >= 70%
- False positive rate < 20%
- Score separation > 0.10

## Install

```bash
pip install uncanny
```

Or with Homebrew:

```bash
brew tap dunkinfrunkin/tap
brew install uncanny
```

For ML-based analyzers (perplexity):

```bash
pip install "uncanny[ml]"
```

## Usage

```bash
# Scan a file
uncanny scan report.md

# Scan from stdin
cat essay.txt | uncanny scan

# Scan a string directly
uncanny scan --text "The implementation of this feature..."

# Output formats
uncanny scan report.md --format json      # structured JSON
uncanny scan report.md --format summary   # just the score

# Speed vs depth
uncanny scan report.md --fast             # compression only (~instant)
uncanny scan report.md --deep             # all analyzers (requires uncanny[ml])
uncanny scan report.md --analyzers compression,burstiness

# CI integration — exit code 1 if score exceeds threshold
uncanny scan report.md --threshold 0.7

# Benchmarks
uncanny bench                             # run accuracy benchmarks
uncanny bench --compare                   # head-to-head vs competitors
uncanny bench --compare --json            # machine-readable output
```

### JSON Output

```json
{
  "source": "proposal.md",
  "score": 0.73,
  "label": "likely_ai",
  "analyzers": {
    "compression": { "score": 0.81, "method": "lzma+zlib" },
    "burstiness": { "score": 0.71, "variance": 0.003 }
  },
  "sentences": [
    {
      "text": "The implementation of this feature requires...",
      "score": 0.91,
      "offset": { "start": 0, "end": 89 }
    }
  ]
}
```

## How It Works

Uncanny combines multiple detection strategies through an **ensemble scorer** with dynamic weighting:

### Compression Analysis (primary signal)
AI text is statistically predictable — it compresses better than human text. Uncanny cross-compresses your text against both an AI-generated prelude and a human-written prelude using LZMA and zlib. If your text shares compression patterns with the AI prelude, it scores higher. This is the fastest method (no model needed) and the most reliable single signal.

### Burstiness Analysis (tiebreaker)
Humans write with natural variation — short sentences mixed with long ones, simple ideas next to complex ones. AI text is uniformly smooth. Uncanny measures the variance in compression ratio and sentence length across a document. Low variance = AI-like uniformity. This signal gets dynamically weighted — it matters most when compression is ambiguous.

### Perplexity Analysis (optional, requires `uncanny[ml]`)
Measures how "surprised" GPT-2 is by each sentence. AI-generated text has low perplexity (the model predicted it easily). Human text has high perplexity (surprising word choices). Requires downloading a ~500MB model.

### Ensemble Scoring
Analyzers are combined with **confidence-aware dynamic weighting**. When compression is decisive (far from 0.5), it dominates. When compression is ambiguous, burstiness gets more influence. This avoids the classic problem of a weak signal overriding a strong one.

## Design Principles

1. **Offline-first** — No API calls. Everything runs locally. Models downloaded once.
2. **Fast by default** — Default mode uses compression + burstiness (~8ms/sample). No model download needed.
3. **Scores, not verdicts** — Uncanny reports probabilities, never binary "AI or human" labels. The tool informs, it doesn't judge.
4. **Sentence-level granularity** — Not just "this doc is AI" — shows exactly which sentences triggered detection.
5. **CI-friendly** — `--threshold` flag with exit codes. JSON output for parsing. Benchmarks in CI.

## Limitations

AI detection is an imperfect science. Be honest about it:

- **No detector is 100% accurate.** False positives and false negatives happen.
- **Short text is unreliable.** Confidence drops below ~250 words.
- **Edited AI text is harder to detect.** A human editing AI output blurs the signal.
- **Newer models are harder to detect.** As LLMs improve, statistical signals narrow.
- **Technical writing may false-positive.** Formal, structured text can look AI-like.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). PRs welcome — especially new analyzers, corpus samples, and accuracy improvements.

## License

MIT
