Metadata-Version: 2.4
Name: gibsondedup
Version: 0.1.0
Summary: A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.
Author: Gibson Kwabena Aseda Mensah
Author-email: logosyninc@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Dynamic: author-email
Dynamic: requires-python

# gibsondedup

A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.

---

## The Problem

When aggregating search results from multiple providers (Google, Bing, DuckDuckGo), 40–60% of results are duplicates — same content, different URLs, different titles, different tracking parameters.

```
Source 1 (Google):
  url: https://www.python.org/docs?utm_source=google
  title: "Python Official Documentation"

Source 2 (Bing):
  url: http://python.org/docs/
  title: "Python Docs"

Source 3 (DuckDuckGo):
  url: https://python.org/docs?utm_medium=cpc
  title: "Learn Python - Official Docs"
```

These are the same resource. Without deduplication, users see noise instead of signal.

---

## The Solution

`gibsondedup` processes raw search results through a multi-stage pipeline:

```
Raw Results
    ↓
Parse & Validate         (malformed records skipped gracefully)
    ↓
URL Normalization        (remove tracking params, www, trailing slashes)
    ↓
Exact URL Grouping       (hash-based, O(n) complexity)
    ↓
Semantic Merge           (Jaccard similarity, configurable threshold)
    ↓
Canonicalization         (select best result per group)
    ↓
Clean Output             (deduplicated results + metadata)
```

**Result:** 100 noisy inputs → 60–70 clean, canonical results.

---

## Installation

```bash
pip install gibsondedup
```

---

## Quick Start

```python
from app.orchestration.pipeline import DeduplicationEngine

engine = DeduplicationEngine()

results = engine.process([
    {
        "title": "Python Official Documentation",
        "url": "https://www.python.org/docs?utm_source=google",
        "description": "Official Python docs",
        "source": "google"
    },
    {
        "title": "Python Docs",
        "url": "http://python.org/docs/",
        "description": "Python documentation",
        "source": "bing"
    },
    {
        "title": "Stack Overflow Python",
        "url": "https://stackoverflow.com/questions/tagged/python",
        "description": "Python questions",
        "source": "google"
    },
])

print(results)
```

**Output:**

```json
{
  "results": [
    {
      "title": "Python Official Documentation",
      "url": "https://www.python.org/docs?utm_source=google",
      "canonical_url": "python.org/docs",
      "description": "Official Python docs",
      "sources": ["google", "bing"],
      "duplicates_removed": 1
    },
    {
      "title": "Stack Overflow Python",
      "url": "https://stackoverflow.com/questions/tagged/python",
      "canonical_url": "stackoverflow.com/questions/tagged/python",
      "description": "Python questions",
      "sources": ["google"],
      "duplicates_removed": 0
    }
  ],
  "meta": {
    "total_input": 3,
    "total_parsed": 3,
    "total_output": 2,
    "duplicates_removed": 1,
    "processing_time_ms": 0.09,
    "similarity_threshold": 0.8
  }
}
```

---

## Configuration

### Custom Similarity Threshold

Control how aggressively similar-titled results are merged:

```python
# Conservative (default) — only merge highly similar titles
engine = DeduplicationEngine(similarity_threshold=0.8)

# Moderate — merge titles with moderate overlap
engine = DeduplicationEngine(similarity_threshold=0.5)

# Aggressive — merge titles with minimal overlap
engine = DeduplicationEngine(similarity_threshold=0.3)
```

### Input Contract

Each result in the input array accepts:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `title` | string | ✅ | Result title |
| `url` | string | ✅ | Result URL |
| `description` | string | ❌ | Result description |
| `source` | string | ❌ | Source provider (google, bing, etc.) |

Malformed records (missing title or URL) are **skipped gracefully** — the pipeline continues processing valid records.

---

## Architecture

### Pipeline Stages

**Stage 1: Parser**
Converts raw JSON payloads into structured internal contracts. Validates required fields. Skips malformed records without crashing.

**Stage 2: URL Normalizer**
Converts URLs to canonical form by:
- Removing protocol (`http://`, `https://`)
- Removing `www` prefix
- Removing tracking parameters (`utm_*`, `fbclid`, `gclid`)
- Removing trailing slashes
- Stripping default ports (80, 443)
- Preserving meaningful query parameters
- Lowercasing everything

**Stage 3: Exact Grouping**
Groups results by normalized canonical URL using a hash map. Time complexity: O(n). Avoids the O(n²) pairwise comparison trap.

**Stage 4: Semantic Merger**
Within groups sharing the same domain, compares representative titles using Jaccard similarity. Groups exceeding the similarity threshold are merged into one. Only compares within the same exact domain — `python.org` and `docs.python.org` are treated as separate domains.

**Stage 5: Canonicalizer**
For each group, selects the single best result using a combined title + description length heuristic. Collects source traceability from all merged results.

### Data Contracts

```python
# Input
RawSearchResult(title, url, description?, source?)

# Internal
NormalizedSearchResult(title, url, canonical_url, description, source)

# Output
CanonicalResult(title, url, canonical_url, description, sources[], duplicates_removed)
```

### System Invariants

1. Same normalized URL → same duplicate group (always)
2. Canonical results preserve source traceability (always)
3. Pipeline stages do not mutate upstream data (always)
4. Normalization happens once per result (always)

---

## Engineering Decisions

### Why hash-based grouping (O(n)) over pairwise comparison (O(n²))?

With 1000 results, O(n²) means 1,000,000 comparisons. O(n) means 1000. At scale this is the difference between milliseconds and seconds.

### Why exact domain matching for semantic merging?

`python.org/docs` and `docs.python.org` serve genuinely different content despite sharing a base domain. Exact domain matching prevents false merges while still catching real duplicates like `python.org/docs` and `python.org/reference`.

### Why Jaccard similarity over Levenshtein distance?

Jaccard operates on token sets (words), making it robust to word order changes and additions. Levenshtein operates on character sequences, making it sensitive to trivial differences like "Docs" vs "Documentation".

### Why keep normalization deterministic?

Same input must always produce same output. This ensures:
- Results are reproducible across runs
- Caching normalized URLs is safe
- Tests are reliable and meaningful

---

## Performance

| Input Size | Processing Time | Memory |
|------------|----------------|--------|
| 100 results | ~1ms | ~1MB |
| 1,000 results | ~10ms | ~5MB |
| 10,000 results | ~100ms | ~50MB |

*Benchmarked on Ubuntu 22.04, Python 3.10, Intel i5*

---

## Testing

```bash
# Install dev dependencies
pip install pytest pytest-cov

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=app --cov-report=term-missing
```

**Test coverage: 59 tests across 5 modules**

```
tests/test_normalizer.py    14 tests  (URL normalization)
tests/test_grouping.py       4 tests  (exact grouping)
tests/test_canonicalizer.py  7 tests  (best result selection)
tests/test_merger.py        11 tests  (semantic merging)
tests/test_similarity.py    13 tests  (Jaccard similarity)
tests/test_pipeline.py      10 tests  (end-to-end pipeline)
```

---

## Roadmap

### Phase 1 (Complete)
- URL normalization
- Exact duplicate grouping
- Canonicalization with source traceability

### Phase 2 (Complete)
- Jaccard similarity engine
- Domain-based semantic merging
- Configurable similarity threshold

### Phase 3 (Planned)
- Persistent caching (Redis)
- Database storage (PostgreSQL)
- REST API exposure
- Rails wrapper (gem)

### Phase 4 (Planned)
- Semantic embeddings (replace Jaccard with vector similarity)
- Distributed processing
- Production observability

---

## License

MIT License. See `LICENSE` file.

---

## Author

**Aseda Gibson**
Computer Engineering, University of Energy and Natural Resources (UENR), Ghana.
Backend systems, distributed architecture, infrastructure tooling.

GitHub: [github.com/asedagibson](https://github.com/asedagibson)

---

*Built with correctness-oriented engineering. Every architectural decision is documented. Every invariant is tested.*
