Metadata-Version: 2.4
Name: redup
Version: 0.4.22
Summary: Code duplication analyzer and refactoring planner for LLMs
Author-email: Tom Sapletta <tom@sapletta.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/semcod/redup
Project-URL: Repository, https://github.com/semcod/redup
Project-URL: Issues, https://github.com/semcod/redup/issues
Project-URL: Documentation, https://github.com/semcod/redup#readme
Keywords: duplication,refactoring,code-analysis,LLM,AST,static-analysis
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Requires-Dist: typer>=0.12.0
Requires-Dist: rich>=13.0
Requires-Dist: pydantic>=2.0
Requires-Dist: tomli>=2.0; python_version < "3.11"
Requires-Dist: python-dotenv>=1.0
Requires-Dist: goal>=2.1.0
Requires-Dist: costs>=0.1.20
Requires-Dist: pfix>=0.1.60
Provides-Extra: all
Requires-Dist: redup[fast]; extra == "all"
Requires-Dist: redup[fuzzy]; extra == "all"
Requires-Dist: redup[ast]; extra == "all"
Requires-Dist: redup[lsh]; extra == "all"
Requires-Dist: redup[semantic]; extra == "all"
Requires-Dist: redup[compare]; extra == "all"
Provides-Extra: compare
Requires-Dist: networkx>=3.0; extra == "compare"
Provides-Extra: llm
Requires-Dist: litellm>=1.0; extra == "llm"
Provides-Extra: fast
Requires-Dist: xxhash>=3.0; extra == "fast"
Requires-Dist: libcst>=1.0; extra == "fast"
Requires-Dist: pybloom-live>=4.0; extra == "fast"
Provides-Extra: fuzzy
Requires-Dist: rapidfuzz>=3.0; extra == "fuzzy"
Provides-Extra: ast
Requires-Dist: tree-sitter>=0.21; extra == "ast"
Requires-Dist: tree-sitter-javascript>=0.20; extra == "ast"
Requires-Dist: tree-sitter-typescript>=0.20; extra == "ast"
Requires-Dist: tree-sitter-go>=0.20; extra == "ast"
Requires-Dist: tree-sitter-rust>=0.20; extra == "ast"
Requires-Dist: tree-sitter-java>=0.20; extra == "ast"
Requires-Dist: tree-sitter-c>=0.20; extra == "ast"
Requires-Dist: tree-sitter-cpp>=0.20; extra == "ast"
Requires-Dist: tree-sitter-html>=0.20; extra == "ast"
Requires-Dist: tree-sitter-css>=0.20; extra == "ast"
Requires-Dist: tree-sitter-sql>=0.20; extra == "ast"
Requires-Dist: tree-sitter-json>=0.20; extra == "ast"
Requires-Dist: tree-sitter-yaml>=0.20; extra == "ast"
Requires-Dist: tree-sitter-toml>=0.20; extra == "ast"
Requires-Dist: tree-sitter-xml>=0.20; extra == "ast"
Requires-Dist: tree-sitter-markdown>=0.20; extra == "ast"
Requires-Dist: tree-sitter-bash>=0.20; extra == "ast"
Requires-Dist: tree-sitter-ruby>=0.20; extra == "ast"
Requires-Dist: tree-sitter-php>=0.20; extra == "ast"
Requires-Dist: tree-sitter-embedded-template>=0.20; extra == "ast"
Requires-Dist: tree-sitter-regex>=0.20; extra == "ast"
Requires-Dist: tree-sitter-scala>=0.20; extra == "ast"
Requires-Dist: tree-sitter-kotlin>=0.20; extra == "ast"
Requires-Dist: tree-sitter-swift>=0.20; extra == "ast"
Requires-Dist: tree-sitter-objc>=0.20; extra == "ast"
Requires-Dist: tree-sitter-c-sharp>=0.20; extra == "ast"
Requires-Dist: tree-sitter-lua>=0.20; extra == "ast"
Requires-Dist: tree-sitter-graphql>=0.20; extra == "ast"
Requires-Dist: tree-sitter-dockerfile>=0.20; extra == "ast"
Requires-Dist: tree-sitter-make>=0.20; extra == "ast"
Requires-Dist: tree-sitter-vim>=0.20; extra == "ast"
Requires-Dist: tree-sitter-nginx>=0.20; extra == "ast"
Requires-Dist: tree-sitter-svelte>=0.20; extra == "ast"
Requires-Dist: tree-sitter-vue>=0.20; extra == "ast"
Provides-Extra: lsh
Requires-Dist: datasketch>=1.6; extra == "lsh"
Provides-Extra: semantic
Requires-Dist: sentence-transformers>=2.0; extra == "semantic"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: goal>=2.1.0; extra == "dev"
Requires-Dist: costs>=0.1.20; extra == "dev"
Requires-Dist: pfix>=0.1.60; extra == "dev"
Dynamic: license-file

# reDUP

**Code duplication analyzer and refactoring planner for LLMs.**

[![PyPI](https://img.shields.io/pypi/v/redup)](https://pypi.org/project/redup/)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://python.org)
[![Version](https://img.shields.io/badge/version-0.4.22-green.svg)](https://pypi.org/project/redup/)


## AI Cost Tracking

![PyPI](https://img.shields.io/badge/pypi-costs-blue) ![Version](https://img.shields.io/badge/version-0.4.22-blue) ![Python](https://img.shields.io/badge/python-3.9+-blue) ![License](https://img.shields.io/badge/license-Apache--2.0-green)
![AI Cost](https://img.shields.io/badge/AI%20Cost-$7.50-orange) ![Human Time](https://img.shields.io/badge/Human%20Time-15.7h-blue) ![Model](https://img.shields.io/badge/Model-openrouter%2Fqwen%2Fqwen3--coder--next-lightgrey)

- 🤖 **LLM usage:** $7.5000 (63 commits)
- 👤 **Human dev:** ~$1568 (15.7h @ $100/h, 30min dedup)

Generated on 2026-04-16 using [openrouter/qwen/qwen3-coder-next](https://openrouter.ai/qwen/qwen3-coder-next)

---

reDUP scans codebases for duplicated functions, blocks, and structural patterns — then builds a prioritized refactoring map that LLMs can consume to eliminate redundancy systematically.

## Features

- **Exact duplicate detection** via SHA-256 block hashing
- **Structural clone detection** — same AST shape, different variable names
- **LSH near-duplicate detection** for large code blocks (>50 lines)
- **Multi-language support** — 35+ languages via tree-sitter (Python, JavaScript, TypeScript, Go, Rust, Java, C/C++, C#, Ruby, PHP, Bash, SQL, HTML, CSS, Lua, Scala, Kotlin, Swift, Objective-C, JSON, YAML, TOML, XML, Markdown, GraphQL, Dockerfile, Makefile, Nginx, Vim, Svelte, Vue, and more)
- **Parallel scanning** for large projects (2x+ performance improvement)
- **Fuzzy near-duplicate matching** via SequenceMatcher / rapidfuzz
- **Function-level analysis** using Python AST and tree-sitter extraction
- **Impact scoring** — prioritizes duplicates by `saved_lines × similarity`
- **Refactoring planner** — generates concrete extract/inline suggestions
- **Multiple output formats**: JSON, YAML, TOON, Markdown
- **Configuration system** — TOML files and environment variables
- **CLI commands**: `scan`, `compare`, `diff`, `check`, `config`, `info`
- **Cross-project comparison** — detect shared code between projects with merge/extract recommendations
- **CI integration** with configurable quality gates
- **Clean output** — no syntax warnings from external libraries

## New Features (v0.4.20)

### 🤖 MCP Server

Full MCP (Model Context Protocol) server for AI assistant integration:

```bash
# Start MCP server
redup-mcp

# Or HTTP mode
redup-mcp --transport http --port 8000
```

**Available Tools:**
- `analyze_project` — Full duplication analysis
- `find_duplicates` — Quick duplicate detection
- `check_project` — Quality gate check
- `compare_projects` — Cross-project comparison
- `suggest_refactoring` — AI-powered refactoring suggestions
- `project_info` — Project metadata

### 🌐 Universal Fuzzy Similarity Detection

Cross-language duplicate detection across all 35+ supported languages:

```bash
# Detect similar code across languages
redup scan . --fuzzy --fuzzy-threshold 0.65
```

**Cross-Language Matching:**
- JavaScript ↔ Python functions: ~65% similarity
- Docker ↔ YAML configs: ~40% similarity
- Auth patterns across languages: ~70% similarity

**Supported Patterns:**
- Functions, classes, API endpoints
- Database queries, web components
- Auth/validation, error handling, logging
- Configuration, infrastructure code

### 🌳 Modular Tree-Sitter Extractor

Refactored tree-sitter extraction with clean, modular architecture:

```
ts_extractor/
├── extractors/          # Modular per-language extractors
│   ├── c_family.py      # C, C++, C#, Objective-C
│   ├── go.py            # Go
│   ├── java.py          # Java, Scala, Kotlin
│   ├── markup.py        # HTML, XML, Svelte, Vue
│   ├── web.py           # JavaScript, TypeScript
│   └── ...
├── dispatcher.py        # Smart language routing
├── config.py            # Language registry
└── main.py              # Unified API
```

**Benefits:**
- Easier to add new languages
- Better testability
- Cleaner separation of concerns
- 35+ languages supported

---

## New Features (v0.5.0+)

### 🌐 Universal Fuzzy Similarity Detection

Cross-language fuzzy matching for detecting similar code patterns across **all 35+ supported languages**:

```bash
# Detect similar patterns across different languages
redup scan . --fuzzy --ext .py,.js,.ts

# Cross-project comparison with fuzzy matching
redup compare ./project-a ./project-b --fuzzy --threshold 0.65
```

**Features:**
- Detects similar functions, API endpoints, validation logic across languages (e.g., JS ↔ Python)
- Pattern recognition: authentication, error handling, database queries, web components
- Language-agnostic signature generation with identifier normalization
- Complexity scoring (0.0-1.0) for each detected pattern

**Example patterns detected:**
- Express.js route handler ↔ Flask endpoint (70% similarity)
- Docker Compose service ↔ Kubernetes deployment (40% similarity)
- Auth middleware patterns across frameworks

### 🧩 Modular ts_extractor Architecture

The tree-sitter multi-language extractor has been refactored from a 782-line god module into a clean package:

```
redup/core/ts_extractor/
├── extractors/
│   ├── web.py        # JavaScript/TypeScript
│   ├── c_family.py   # C/C++
│   ├── dotnet.py     # C#
│   ├── ruby.py       # Ruby
│   ├── php.py        # PHP
│   └── ...           # 10+ language-specific modules
```

**Benefits:**
- Better maintainability (avg 100 lines per module vs 782)
- Easier to add new language extractors
- Shared base utilities for common operations
- Full backward compatibility maintained

### 🎯 Enriched TOON Reporter

The TOON format now includes actionable sections for practical refactoring:

- **HOTSPOTS** — Top 7 files with most duplicated lines (where to focus effort)
- **QUICK_WINS** — Low-risk, high-savings suggestions (do first)
- **DEPENDENCY_RISK** — Duplicates spanning multiple packages (cross-module risk)
- **EFFORT_ESTIMATE** — Time estimates per task with difficulty (easy/medium/hard)

### 🤖 LLM-Powered Refactoring Plans

Generate AI-assisted refactoring TODO lists from cross-project comparisons:

```bash
redup compare ./project-a ./project-b --refactor-plan --env .env --output report.json
```

- Uses `litellm` for flexible LLM provider support
- Compact metadata-only prompts for efficiency
- Structured JSON output with prioritized tasks
- Token usage tracking

### 📊 Simplified Compare Reports

Cross-project comparison reports are now more compact and human-readable:

- Relative file paths instead of absolute
- Matches deduplicated by function pair
- Communities with compact member dicts
- Filtered trivial entries to reduce noise
- ~60% smaller JSON size

## Installation

```bash
pip install redup
```

With optional dependencies:

```bash
pip install redup[all]       # Everything
pip install redup[fuzzy]     # rapidfuzz for better similarity matching
pip install redup[ast]       # tree-sitter for multi-language AST
pip install redup[lsh]       # datasketch for LSH near-duplicate detection
pip install redup[compare]   # networkx for cross-project community detection
pip install redup[llm]       # litellm for LLM-powered refactoring plans
```

## Quick Start

### CLI

```bash
# Scan current directory, output TOON to stdout
redup scan .

# Scan with JSON output saved to file
redup scan ./src --format json --output ./reports/

# Parallel scanning for large projects
redup scan . --parallel --max-workers 4

# Multi-language scanning with 35+ supported languages
redup scan . --ext ".py,.js,.ts,.go,.rs,.java,.rb,.php,.html,.css,.sql,.lua,.scala,.kt,.swift,.m,.json,.yaml,.toml,.xml,.md,.graphql,.dockerfile,.svelte,.vue"

# CI gate with thresholds
redup check . --max-groups 10 --max-lines 100

# Compare two scans
redup diff before.json after.json

# Cross-project comparison (merge vs extract decision)
redup compare ./project-a ./project-b --threshold 0.75

# With LLM-powered refactoring plan (requires litellm + .env with API keys)
redup compare ./project-a ./project-b --refactor-plan --env .env --output comparison.json

# Specify custom LLM model
redup compare ./project-a ./project-b --refactor-plan --llm-model openrouter/anthropic/claude-3.5-sonnet

# Initialize configuration
redup config --init
```

```bash
# Scan with all formats
redup scan . --format all --output ./redup_output/

# Only function-level duplicates (faster)
redup scan . --functions-only

# Custom thresholds
redup scan . --min-lines 5 --min-sim 0.9

# Show installed optional dependencies
redup info
```

### Configuration

Create a `redup.toml` file:

```toml
[scan]
extensions = ".py,.js,.ts,.go,.rs,.java,.rb,.php,.html,.css,.sql,.lua,.scala,.kt,.swift,.m,.json,.yaml,.toml,.xml,.md,.graphql,.dockerfile,.svelte,.vue"
min_lines = 3
min_similarity = 0.85
include_tests = false

[lsh]
enabled = true
min_lines = 50
threshold = 0.8

[check]
max_groups = 10
max_lines = 100

[output]
format = "toon"
output = "redup_output"

[reporting]
include_snippets = true
generate_suggestions = true
```

Or use `[tool.redup]` in `pyproject.toml`. Environment variables with `REDUP_` prefix override file settings.

### Python API

```python
from pathlib import Path
from redup import ScanConfig, analyze
from redup.reporters.toon_reporter import to_toon
from redup.reporters.json_reporter import to_json

config = ScanConfig(
    root=Path("./my_project"),
    extensions=[".py", ".js", ".ts", ".go", ".rs", ".java", ".rb", ".php", ".html", ".css"],
    min_block_lines=3,
    min_similarity=0.85,
)

result = analyze(config=config, function_level_only=True)

print(f"Found {result.total_groups} duplicate groups")
print(f"Lines recoverable: {result.total_saved_lines}")

# For LLM consumption
print(to_toon(result))

# For tooling / CI
Path("duplication.json").write_text(to_json(result))
```

## Output Formats

### TOON (LLM-optimized)

```
# redup/duplication | 15 groups | 86f 10453L | 2026-04-16

SUMMARY:
  files_scanned: 86
  total_lines:   10453
  dup_groups:    15
  dup_fragments: 36
  saved_lines:   217
  scan_ms:       3620

HOTSPOTS[7] (files with most duplication):
  src/redup/core/ts_extractor.py  dup=74L  groups=4  frags=11  (0.7%)
  src/redup/core/scanner_utils.py  dup=70L  groups=3  frags=3  (0.7%)
  src/redup/core/scanner_loader.py  dup=52L  groups=1  frags=1  (0.5%)

DUPLICATES[15] (ranked by impact):
  [E0001] ! EXAC  _preload_files  L=52 N=2 saved=52 sim=1.00
      src/redup/core/scanner_loader.py:9-60  (_preload_files)
      src/redup/core/scanner_utils.py:53-104  (_preload_files)

REFACTOR[15] (ranked by priority):
  [1] ◐ extract_module     → src/redup/core/utils/_preload_files.py
      WHY: 2 occurrences of 52-line block across 2 files — saves 52 lines
      FILES: src/redup/core/scanner_loader.py, src/redup/core/scanner_utils.py

QUICK_WINS[8] (low risk, high savings — do first):
  [3] extract_function   saved=26L  → src/redup/core/utils/find_exact_duplicates_lazy.py
      FILES: lazy_grouper.py
  [4] extract_function   saved=21L  → src/redup/core/utils/_extract_functions_go.py
      FILES: ts_extractor.py

DEPENDENCY_RISK[3] (duplicates spanning multiple packages):
  validate_input  packages=2  files=2
      api/routes/users.py
      services/auth/validate.py

EFFORT_ESTIMATE (total ≈ 8.7h):
  hard   _preload_files                      saved=52L  ~156min
  hard   __init__                            saved=36L  ~108min
  medium find_exact_duplicates_lazy          saved=26L  ~52min
  easy   _is_test_file                       saved=12L  ~24min

METRICS-TARGET:
  dup_groups:  15 → 0
  saved_lines: 217 lines recoverable
```

### JSON (machine-readable)

```json
{
  "summary": {
    "total_groups": 3,
    "total_saved_lines": 84
  },
  "groups": [
    {
      "id": "E0001",
      "type": "exact",
      "normalized_name": "calculate_tax",
      "fragments": [
        {"file": "billing.py", "line_start": 1, "line_end": 8},
        {"file": "shipping.py", "line_start": 1, "line_end": 8}
      ],
      "saved_lines_potential": 16
    }
  ],
  "refactor_suggestions": [
    {
      "priority": 1,
      "action": "extract_function",
      "new_module": "utils/calculate_tax.py",
      "risk_level": "low"
    }
  ]
}
```

## Cross-Project Comparison

The `redup compare` command analyzes two separate projects to detect shared code and recommends a refactoring strategy:

- **Merge projects** — if >60% code overlap
- **Extract shared library** — if 5-60% overlap with well-defined clusters
- **Keep separate** — if <5% overlap

### CLI Usage

```bash
# Basic comparison
redup compare ./project-a ./project-b --threshold 0.75

# With semantic similarity (slower, more accurate)
redup compare ./project-a ./project-b --semantic --threshold 0.70

# Multi-language projects
redup compare ./backend ./frontend --ext ".py,.js,.ts" --threshold 0.80

# Skip community detection (faster, no networkx required)
redup compare ./a ./b --no-community

# Generate LLM-powered refactoring plan (requires redup[llm])
redup compare ./a ./b --refactor-plan --env .env --output plan.json
```

### Sample Output

```
Comparing project-a ↔ project-b (threshold=0.75)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃      Cross-Project Comparison                        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Metric                  │ Value                      │
├─────────────────────────┼────────────────────────────┤
│ Project A files         │ 42                         │
│ Project B files         │ 38                         │
│ Project A lines         │ 8500                       │
│ Project B lines         │ 7200                       │
│ Cross matches           │ 15                         │
│ Shared LOC (potential)  │ 1200                       │
└─────────────────────────┴────────────────────────────┘

Recommendation: extract_shared_lib
15% overlap (1200 shared lines, 5 clusters). Extract to shared library.
Confidence: 80%

Top Communities (shared code candidates):
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━┳━━━━━━━━━━┓
┃ ID ┃ Name                 ┃ Similarity ┃ LOC ┃ Members  ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━╇━━━━━━━━━━┩
│  0 │ validate_input       │ 0.89       │ 180 │ 5        │
│  1 │ parse_config         │ 0.82       │ 140 │ 4        │
│  2 │ format_response      │ 0.76       │ 100 │ 3        │
└────┴──────────────────────┴────────────┴─────┴──────────┘
```

### Report JSON Structure

```json
{
  "project_a": "./project-a",
  "project_b": "./project-b",
  "stats": {
    "a": {"files": 42, "lines": 8500},
    "b": {"files": 38, "lines": 7200}
  },
  "total_matches": 15,
  "shared_loc_potential": 1200,
  "recommendation": {
    "decision": "extract_shared_lib",
    "rationale": "15% overlap (1200 shared lines, 5 clusters). Extract to shared library.",
    "overlap_pct": 0.1523,
    "shared_loc": 1200,
    "confidence": 0.8
  },
  "communities": [
    {
      "name": "validate_input",
      "similarity": 0.89,
      "loc": 180,
      "members": [
        {"project": "A", "file": "api/validators.py", "function": "validate_input"},
        {"project": "B", "file": "utils/validation.py", "function": "validate_input"}
      ]
    }
  ],
  "matches": [...]
}
```

### Algorithm Overview

The comparison uses a **3-tier similarity detection**:

1. **Structural hash** — exact AST matches (fast, O(n+m))
2. **LSH (Locality Sensitive Hashing)** — near-duplicates via MinHash
3. **Semantic similarity** — CodeBERT embeddings (optional, slowest)

Matches are deduplicated by `(function_a, function_b, file_a, file_b)` with the highest similarity score retained.

### Community Detection

Requires `networkx` (`pip install redup[compare]`).

Uses **greedy modularity communities** on a similarity graph where:
- Nodes = functions from both projects
- Edges = similarity score (filtered by `--threshold`)
- Communities = clusters of mutually similar functions

Each community gets a generated name based on longest common prefix of its member functions (e.g., `validate_*` → `validate_input`).

## Architecture

```
src/redup/
├── __init__.py            # Public API
├── __main__.py            # python -m redup
├── mcp_server.py          # MCP server entry point (re-exports from mcp package)
├── mcp/                   # MCP server package
│   ├── __init__.py        # Public MCP API
│   ├── handlers.py        # Tool handlers
│   ├── schemas.py         # JSON-RPC schemas
│   ├── server.py          # JSON-RPC server core
│   └── utils.py           # Shared utilities
├── core/
│   ├── models.py          # Pydantic data models
│   ├── scanner.py         # File discovery + block extraction
│   ├── scanner/           # Scanner package
│   │   ├── __init__.py    # Public scanner API
│   │   ├── cache.py       # Memory cache
│   │   ├── filters.py     # File filtering
│   │   ├── loader.py      # File preloading
│   │   └── types.py       # Scanner types
│   ├── hasher.py          # SHA-256 / structural fingerprinting
│   ├── matcher.py         # Fuzzy similarity comparison
│   ├── planner.py         # Refactoring suggestion generator
│   ├── pipeline.py        # Legacy: re-exports from pipeline package
│   └── pipeline/          # Pipeline package (new)
│       ├── __init__.py    # analyze(), analyze_optimized(), analyze_parallel()
│       ├── phases.py      # scan_phase(), process_blocks()
│       ├── duplicate_finder.py  # Duplicate finding phases
│       └── groups.py      # Group creation, deduplication
│   └── ts_extractor/        # Tree-sitter extraction (35+ languages)
│       ├── __init__.py    # Public API
│       ├── main.py        # Core extraction API
│       ├── dispatcher.py  # Language routing
│       ├── config.py      # Language registry
│       └── extractors/    # Per-language extractors
├── reporters/
│   ├── json_reporter.py   # JSON output
│   ├── yaml_reporter.py   # YAML output
│   └── toon_reporter.py   # TOON output (LLM-optimized)
└── cli_app/
    └── main.py            # Typer CLI
```

## Analysis Pipeline

```
1. SCAN      Walk project, read files, extract function-level + sliding-window blocks
2. HASH      Generate exact (SHA-256) and structural (normalized AST) fingerprints
3. GROUP     Bucket by hash, keep only groups with 2+ blocks from different locations
4. MATCH     Verify candidates with fuzzy similarity (SequenceMatcher / rapidfuzz)
5. DEDUP     Remove overlapping groups (keep highest-impact)
6. PLAN      Generate prioritized refactoring suggestions with risk assessment
7. REPORT    Export to JSON / YAML / TOON
```

## Recent Improvements (v0.5.0)

### 🏗️ **Modular Architecture Refactoring**

Major internal restructuring for better maintainability and extensibility:

#### MCP Server Package
The MCP server has been split from a 675-line monolith into a clean package:
```
redup/mcp/
├── __init__.py      # Public API
├── handlers.py      # 8 tool handlers
├── schemas.py       # JSON-RPC schemas
├── server.py        # Server core
└── utils.py         # Utilities
```
- **82% code reduction** in main file
- **Backward compatible**: `mcp_server.py` re-exports all APIs
- **Better testability**: Isolated handlers can be tested independently

#### Pipeline Package
The analysis pipeline (714 lines) now lives in a modular package:
```
redup/core/pipeline/
├── __init__.py          # analyze(), analyze_optimized(), analyze_parallel()
├── phases.py            # scan_phase(), process_blocks()
├── duplicate_finder.py  # find_exact_groups(), find_structural_groups(), etc.
└── groups.py            # deduplicate_groups(), blocks_to_group(), etc.
```
- **66% reduction** in main orchestrator file
- **Phases can be used independently** for custom workflows
- **Cleaner separation** of concerns

#### Scanner Improvements
The scanner has been refactored with extracted helpers:
- `_init_strategy()` - Strategy initialization
- `_process_single_file()` - Per-file processing
- `_extract_blocks_for_file()` - Block extraction
- **Reduced CC** and **fan-out** in main `scan_project()` function

### 🎯 **Sprint 1 Refactoring Complete**
- **Reduced cyclomatic complexity** from CC̄=4.2 to CC̄=3.5
- **Eliminated all critical functions** (CC > 10): 2 → 0
- **Achieved HEALTHY status** with no structural issues
- **Dispatch pattern implementation** for AST node processing
- **Modular TOON reporter** split into 5 focused functions
- **CLI refactoring** with helper functions for better maintainability

### 🚀 **Technical Achievements**
- **`_process_ast_node`**: CC=14 → CC=6 (dispatch dict pattern)
- **`to_toon`**: CC=12 → CC=8 (5 helper functions)
- **CLI `scan()`**: fan-out=18 → ≤10 (4 helper functions)
- **Code quality**: 0 high-complexity functions
- **Test coverage**: 64/64 tests passing (100%)

### 📊 **Quality Metrics**
- **Health status**: ✅ HEALTHY (no critical issues)
- **Cyclomatic complexity**: CC̄=3.5 (target ≤ 3.0 achieved)
- **Maximum CC**: 9 (target ≤ 10 achieved)
- **Code maintainability**: Significantly improved
- **Duplication**: Minimal (2 groups, 6 lines - acceptable patterns)

### 🔧 **Code Architecture**
- **Dispatch tables** for extensible AST processing
- **Single responsibility** functions throughout codebase
- **Clean separation** of concerns in CLI pipeline
- **Type safety** improvements with proper annotations
- **Error handling** enhanced for edge cases

---

## Integration with wronai Toolchain

reDUP is part of the [wronai](https://github.com/wronai) developer toolchain:

- **[code2llm](https://github.com/wronai/code2llm)** — static analysis engine (health diagnostics, complexity)
- **reDUP** — deep duplication analysis and refactoring planning
- **[code2docs](https://github.com/wronai/code2docs)** — automatic documentation generation
- **[vallm](https://github.com/semcod/vallm)** — validation of LLM-generated code proposals

### 📈 **Typical workflow:**

1. `code2llm` analyzes the project → `.toon` diagnostics
2. `redup` finds duplicates → `duplication.toon.yaml`  
3. Feed both to an LLM for targeted refactoring
4. `vallm` validates the LLM's proposals before merging

### 🎯 **Why reDUP?**

- **LLM-ready**: TOON format optimized for LLM consumption
- **Actionable**: Generates concrete refactoring suggestions
- **Prioritized**: Ranks duplicates by impact and risk
- **Integrated**: Works seamlessly with wronai toolchain
- **Fast**: Scans 1000+ lines in < 1 second
- **Clean**: No syntax warnings, professional output

---

## Development

```bash
git clone https://github.com/semcod/redup.git
cd redup
pip install -e ".[dev]"
pytest
```

## License

Licensed under Apache-2.0.
## Author

Tom Sapletta
