Metadata-Version: 2.4
Name: sieve-referee
Version: 1.0.0
Summary: Deterministic, zero-dependency plagiarism and text structure screening engine (Sieve-Core plugin)
Author: Kai IWASAKI
License: MIT
Project-URL: Homepage, https://github.com/neguseatama/sieve-referee
Project-URL: Repository, https://github.com/neguseatama/sieve-referee.git
Project-URL: Related Project (Sieve-Core), https://github.com/neguseatama/sieve-core
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Sieve Referee

English | [日本語](README.ja.md)

> **"Should we really use probabilistic AI models to evaluate text generated by other probabilistic AI models?"**
> A fully local, zero-dependency, and deterministic (100% reproducible) text structure and plagiarism screening engine. Developed as an extended application of [Sieve-Core](https://github.com/neguseatama/sieve-core).

---

## 💡 Concept & Philosophy

`Sieve-Core` is the core engine that applies the design philosophy of "Sieve-AI"—a "sieve of set theory" inspired by the Sieve of Eratosthenes—to the more general domain of text and data processing. `Sieve Referee`, designed by adding a "sieve of combinatorics" to this `Sieve-Core`, is a zero-dependency Python library that extracts reliable information from noise based on corroboration from multiple independent sources.

When evaluating text that may have been generated by probabilistic models (such as LLMs), relying on another probabilistic model to calculate a similarity score makes it difficult to mathematically trace *why* that score was produced.

`Sieve Referee` was built to address this problem: to ensure every judgment is mathematically traceable, 100% reproducible, and fully explainable.

- **Deterministic Approach**: Evaluation is based on a 16-state Look-Up Table (LUT, of which 12 states are reachable). Identical inputs always produce identical outputs no matter how many times it is run.
- **Explainable by Design**: Every result carries a `reason` string and a 4-bit hypothesis `mask`, so the basis of a judgment can always be inspected.
- **Careful Classification to Avoid False Accusations**: A shared template or boilerplate format is classified separately as `COINCIDENTAL_FORMAT_MATCH` (`1100` / `YELLOW`), rather than being treated the same as an actual content match.
- **Indirect ("Telephone Game") Chain Safety**: A pair with no direct similarity evidence (e.g. `A <-> C`, where `A <-> B` and `B <-> C` are both flagged) is never silently upgraded to a positive match solely based on cluster membership. Such pairs are reported separately on the dashboard as an "indirect chain (telephone game) warning," and the direct comparison result itself is left unmodified.

---

## 🎯 Primary Use Cases

Built with zero external dependencies and in-memory execution, `sieve_referee` is well suited as an embedded, privacy-focused screening engine.

### 1. Education & Learning Management Systems (EdTech / LMS)
Screen student submissions without uploading personal data or copyrighted text to a third-party cloud service. Common assignment templates are classified as `YELLOW`; unattributed copying or paraphrasing is flagged as `RED`.

### 2. Digital Publishing & CMS
Detect rewritten or recycled content submitted by freelance writers at draft-save time, without incurring LLM API costs for every check.

### 3. LLM Training Data Deduplication
Use as a deterministic pre-processing stage to remove near-duplicate or paraphrased records from a fine-tuning dataset, without GPU acceleration.

### 4. Legal, Contract & RFP Review
Separate boilerplate clauses (`COINCIDENTAL_FORMAT_MATCH`) from intentionally altered wording across contract revisions or vendor proposals.

### 5. CI/CD Documentation Auditing
Run in a GitHub Actions workflow or pre-commit hook to catch unauthorized copy-pasting across internal wikis or specification documents.

---

## ⚡ Key Technical Advantages

1. **Near-zero cold start** — no PyTorch, no Transformers, no model loading; well suited to AWS Lambda or other serverless environments.
2. **Zero per-call cost** — everything runs on local CPU.
3. **Zero maintenance burden** — zero third-party dependencies means no exposure to upstream CVEs or breaking API changes.

---

## 📦 Installation

```bash
pip install git+https://github.com/neguseatama/sieve-referee.git
```

or, from a local clone:

```bash
git clone https://github.com/neguseatama/sieve-referee.git
cd sieve-referee
pip install .
```

---

## 🐍 Usage (Python Library)

```python
from sieve_referee import batch_evaluate, EvaluationResult

# 1. Prepare input documents in memory
documents = {
    "doc_a.txt": "Quantum computing relies on superposition and entanglement to achieve fast factorization.",
    "doc_b.txt": "To achieve fast factorization, quantum computing makes use of entanglement and superposition."
}

# 2. Run the screening
results: list[EvaluationResult] = batch_evaluate(documents)

# 3. Read the results
for res in results:
    print(f"Pair:    {res.item_id} <-> {res.matched_peer_id}")
    print(f"Signal:  {res.evaluation_signal.value}")  # -> RED
    print(f"Mask:    {res.mask}")                      # -> 0010
    print(f"Pattern: {res.pattern_name}")               # -> UNFORMATTED_CONTENT_COPY
    print(f"Reason:  {res.reason}")
    print("-" * 40)
```

### `EvaluationResult` fields

| Property | Type | Description |
|---|---|---|
| `item_id` | `str` | Primary document identifier |
| `matched_peer_id` | `str` | Comparison peer document identifier |
| `evaluation_signal` | `SignalColor` | Result signal (`.value` returns `"GREEN"`, `"YELLOW"`, or `"RED"`) |
| `mask` | `str` | 4-bit hypothesis mask, e.g. `"1010"` |
| `pattern_name` | `str` | LUT pattern name, e.g. `"PARAPHRASE_DETECTED"` |
| `reason` | `str` | Human-readable explanation of the judgment |

---

## 💻 Usage (CLI Tool)

```bash
# Basic directory scan (writes reports/sieve_dashboard.html)
sieve-referee ./target_documents

# Single self-contained HTML report (no diff_htmls/ subfolder)
sieve-referee ./target_documents --standalone --output-dir ./reports

# JSON output for automated pipelines, with a non-zero exit code on RED
sieve-referee ./target_documents --json --fail-on-risk

# Multi-core evaluation
sieve-referee ./target_documents -j 4
```

| Option | Type | Description |
|---|---|---|
| `target_dir` | positional | Directory containing the `.txt` files to screen |
| `--output-dir` | string | Report output directory (default: `reports`) |
| `--standalone` | flag | Emit a single self-contained HTML report instead of a report + `diff_htmls/` folder |
| `--json` | flag | Print results as JSON to stdout |
| `--json-file` | string | Save results as JSON to the given path |
| `--fail-on-risk` | flag | Exit with status code `1` if any `RED` signal is found |
| `-j`, `--jobs` | int | Number of worker processes for evaluation (default: CPU count) |

---

## 📊 16-State Look-Up Table (LUT)

Four independent hypotheses (`H1`–`H4`) are combined into a 4-bit mask, which maps deterministically to a signal, pattern name, and reason.

| Mask | Signal | Pattern Name | Description |
|:---:|:---:|:---|:---|
| `1000` | 🟢 GREEN | `STANDARD_STRUCTURE_UNIQUE` | Independent document, no structural or content overlap detected. |
| `1100` | 🟡 YELLOW | `COINCIDENTAL_FORMAT_MATCH` | Shared format/template only; content itself is independent. |
| `1010` | 🔴 RED | `PARAPHRASE_DETECTED` | Content overlap despite a different structure (paraphrasing). |
| `1011` | 🔴 RED | `CLUSTER_PARAPHRASE_GROUP` | Paraphrased reuse confirmed within a 3+ document cluster. |
| `1111` | 🔴 RED | `CLUSTER_EXACT_COPY` | Exact/templated copy confirmed within a 3+ document cluster. |

(12 of the 16 possible mask states are reachable in practice; the remaining 4 states, where cluster membership is set without any direct similarity evidence, are structurally unreachable by design.)

> **Note on `H4` (cluster membership) and indirect chains**
> `H4` is only set when a pair has *both* direct similarity evidence (`H2` or `H3`) *and* cluster membership of size ≥ 3. A pair with no direct evidence is never upgraded through cluster membership alone — such "telephone game" chains are reported separately as an indirect-chain warning on the dashboard, and the direct pair's own result is left unmodified.

---

## ⚠️ Known Limitation: Threshold Calibration Is Japanese-Specific

All four thresholds (`H1 = 0.35`, `H2 = 0.80`, `H3 = 0.45`, cluster size `≥ 3`)
were calibrated using real-data testing on **Japanese** text (roughly 1,300+
pairs across unrelated documents, paraphrase pairs, and formal academic
prose).

The `H1` density metric in particular (`len(set(text)) / len(text)`) behaves
very differently across scripts: Japanese text draws from a large set of
kanji, so unique-character ratios tend to be high, while English and other
alphabetic-script text reuses a small ~26-letter alphabet and will
systematically score lower on this metric, independent of how generic or
templated the content actually is. As a result, non-Japanese text may be
under- or over-classified against the current defaults.

Multi-encoding file reading (UTF-8, Shift_JIS, CP932, EUC-JP) and NFKC
normalization work for any text regardless of language. What has **not**
been separately validated is the semantic quality of `H2`/`H3` splitting,
or the appropriateness of the default thresholds, for languages other than
Japanese. If you use this on non-Japanese corpora, re-run the false-positive
check described in [ARCHITECTURE.md](docs/ARCHITECTURE.md) against your own
data before trusting the default thresholds.

Additionally, the `reason` string returned by every `EvaluationResult` is
currently hard-coded in Japanese regardless of the input document's
language, as shown in the example output above. There is no English (or
other-language) localization of these messages yet.

---

## 📚 Documentation

- 🔌 [API & Integration Guide](docs/INTEGRATION.md) — using `sieve_referee` as a Python library or FastAPI microservice
- 🚀 [Deployment Guide](docs/DEPLOYMENT.md) — Docker and Terraform for Google Cloud Run
- 🔐 [CI/CD & Security Guide](docs/CI_CD_SETUP.md) — keyless GCP deployment via Workload Identity Federation
- 🏛️ [Architecture Specification](docs/ARCHITECTURE.md) — the mathematical model and safety invariants in detail

## 📄 License

MIT License. See [LICENSE](LICENSE).
