Metadata-Version: 2.4
Name: poi-dedupe
Version: 0.1.0
Summary: Deduplicate and conflate points of interest from OSM, Overture, Google exports, registries and scraped lists: spatial blocking, explainable similarity scoring, clustering and golden records with provenance.
Author-email: Yusuf Eminoğlu <yusufeminoglu@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/YusufEminoglu/poi_dedupe_sdk
Project-URL: Repository, https://github.com/YusufEminoglu/poi_dedupe_sdk
Project-URL: Issues, https://github.com/YusufEminoglu/poi_dedupe_sdk/issues
Project-URL: Changelog, https://github.com/YusufEminoglu/poi_dedupe_sdk/blob/main/CHANGELOG.md
Keywords: poi,points-of-interest,deduplication,conflation,entity-resolution,record-linkage,geospatial,gis,openstreetmap,overture-maps,fuzzy-matching,jaro-winkler,golden-record,data-quality
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Database
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: numpy
Requires-Dist: numpy>=1.20.0; extra == "numpy"
Provides-Extra: pandas
Requires-Dist: pandas>=1.3.0; extra == "pandas"
Provides-Extra: parquet
Requires-Dist: pyarrow>=8.0.0; extra == "parquet"
Provides-Extra: sklearn
Requires-Dist: scikit-learn>=1.0.0; extra == "sklearn"
Requires-Dist: numpy>=1.20.0; extra == "sklearn"
Provides-Extra: all
Requires-Dist: numpy>=1.20.0; extra == "all"
Requires-Dist: pandas>=1.3.0; extra == "all"
Requires-Dist: pyarrow>=8.0.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: numpy>=1.20.0; extra == "dev"
Requires-Dist: pandas>=1.3.0; extra == "dev"
Requires-Dist: pyarrow>=8.0.0; extra == "dev"
Dynamic: license-file

# poi-dedupe

[![CI](https://github.com/YusufEminoglu/poi_dedupe_sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/YusufEminoglu/poi_dedupe_sdk/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/poi-dedupe.svg)](https://pypi.org/project/poi-dedupe/)
[![Python](https://img.shields.io/pypi/pyversions/poi-dedupe.svg)](https://pypi.org/project/poi-dedupe/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**Deduplicate and conflate points of interest from several sources** (OpenStreetMap, Overture
Maps, Google exports, government registries, scraped lists) into one set of golden records,
with every field traceable to the record it came from.

```
sources ──► spatial blocking ──► similarity features ──► match score ──► clustering ──► golden records
            (grid / geohash,      (name, address, phone,   (weighted or     (union-find     (survivorship
             exact radius)         website, category,       logistic)        with conflict   rules +
                                   distance decay)                           checks)         provenance)
```

- **Zero required dependencies.** Matching, scoring, clustering, merging, CSV and GeoJSON are
  pure Python. NumPy is only needed to *train* a logistic model; pandas and pyarrow are
  optional extras for DataFrames and GeoParquet.
- **Complete blocking.** No pair within the radius is ever missed, at any latitude, across the
  antimeridian or over a pole; this is property-tested against brute force.
- **Explainable.** The default model is a weighted mean whose terms you can print per pair.
- **Multilingual normalisation**: Turkish dotted/dotless I, German ß and umlauts, accents, legal
  forms (`Ltd. Şti.`, `GmbH`, `A.Ş.`, `LLC`), stop-words in seven languages and cautious
  abbreviation expansion.

## Install

```bash
pip install poi-dedupe                 # core: pure Python
pip install "poi-dedupe[numpy]"        # + logistic-regression training
pip install "poi-dedupe[pandas]"       # + DataFrame in/out
pip install "poi-dedupe[parquet]"      # + GeoParquet in/out (pyarrow)
pip install "poi-dedupe[all]"          # all of the above
```

Python 3.9 to 3.13.

## Quick start (CLI)

```bash
poi-dedupe run osm.csv overture.csv registry.csv \
    --name name --lat lat --lon lon --id id \
    --category category --address address --phone phone --website website --updated updated \
    --country-code 90 --priority registry,osm,overture \
    -o merged.geojson --clusters-out clusters.csv
```

Each input file is one source (the source name defaults to the file stem; override it with
`--sources a,b,c`). Record ids become `"<source>:<id>"`. Inputs can be `.csv` (delimiter is
detected), `.geojson` (Point features) or `.parquet` (lat/lon columns or a WKB `geometry`
column); output can be `.geojson`, `.csv` or `.parquet` (GeoParquet 1.0).

| command | what it does |
|---|---|
| `poi-dedupe run ...` | block, score, cluster, merge; writes golden records (`--pairs-out`, `--clusters-out` optional) |
| `poi-dedupe pairs ... -o pairs.csv` | writes every candidate pair with all features and the score |
| `poi-dedupe evaluate pairs.csv --labels labels.csv` | precision / recall / F1, best-F1 threshold, optional `--curve` CSV |
| `poi-dedupe evaluate pairs.csv --truth truth.csv` | same, against complete ground truth (`id,entity_id`) |
| `poi-dedupe train pairs.csv --truth truth.csv -o model.json` | fits a logistic model, tunes its threshold; use with `run --model model.json` |
| `poi-dedupe synth -o bench/` | writes a synthetic three-source city plus `truth.csv` |
| `poi-dedupe benchmark` | runs the benchmark below |

Useful `run` options: `--radius` (metres, default 250), `--threshold`, `--blocking geohash`,
`--category-blocking`, `--cross-source-only`, `--clustering components`, `--max-diameter`,
`--one-per-source`, `--coordinates mean|medoid|source_priority`, `--rule phone=most_recent`,
`--taxonomy my_taxonomy.json`, `--lang de`.

## Quick start (Python)

```python
from poi_dedupe import POI, Deduper, Survivorship

records = [
    POI("osm:1", "Kahve Dünyası", 41.04380, 29.00940, source="osm", category="amenity=cafe",
        phone="+90 212 555 1234", updated="2024-01-01"),
    POI("overture:1", "KAHVE DUNYASI Besiktas", 41.04383, 29.00945, source="overture",
        category="coffee_shop", phone="0212 555 12 34", website="kahvedunyasi.com"),
    POI("registry:1", "Kahve Dunyasi Ltd. Şti.", 41.04378, 29.00938, source="registry",
        category="Coffee Shop", address="Barbaros Blv. No: 12"),
    POI("osm:2", "Garanti BBVA ATM", 41.04385, 29.00950, source="osm", category="amenity=atm"),
]

result = Deduper(
    radius_m=150,
    survivorship=Survivorship(source_priority=["registry", "osm", "overture"]),
).run(records)

len(result.golden)                     # 2: the cafe (3 records) and the ATM
g = result.golden[0]
g.fields["phone"], g.provenance["phone"]
# ('+90 212 555 1234', {'value': ..., 'record_id': 'osm:1', 'source': 'osm',
#                       'rule': 'majority', 'support': 2, 'candidates': 2})
```

Reading files instead:

```python
from poi_dedupe import ColumnMap, read_records, write_golden

cols = ColumnMap(name="name", lat="lat", lon="lon", id="id", category="category", phone="phone")
records = read_records("osm.csv", cols) + read_records("overture.geojson", cols)
write_golden(Deduper().run(records).golden, "merged.parquet")
```

DataFrames: `records_from_dataframe(df, cols, source="osm")`, `golden_to_dataframe(result.golden)`,
`pairs_to_dataframe(result.pairs)`.

## How it works

### 1. Candidate generation (blocking)

`grid` (default) or `geohash` buckets with their neighbour cells, then an exact haversine check
against `radius_m`. Bucket sizes come from great-circle bounds: two points `d` metres apart
differ by at most `d / R` in latitude and by at most `2 asin(sin(d / 2R) / cos φmax)` in
longitude, where `φmax` is the highest latitude present. Neighbouring cells are therefore
always enough, and the result equals brute force exactly (tested at the equator, 41° N,
60° N, 78° N, 89.995° N and on the antimeridian). Near the poles the bucketing falls back to
one longitude column, which is still complete but slower.

Options: `category_blocking=True` drops pairs whose categories are both known and
incompatible; `cross_source_only=True` never compares two records of the same source.

### 2. Similarity features

All in `[0, 1]`; `None` means "no evidence" (a field missing on either side).

| feature | how |
|---|---|
| `name_jw` | Jaro-Winkler on normalised names |
| `name_token_set` | token-set ratio (FuzzyWuzzy/RapidFuzz definition, InDel based): word order and extra words such as a district name do not hurt |
| `name_tfidf` | cosine of character-trigram TF-IDF vectors fitted on all names of the run: rare grams count more than common ones like "cafe" |
| `address` | street-token overlap (typo tolerant) with a house-number term: `0.75 * street + 0.25 * [numbers equal]` when both sides have a number |
| `phone` | E.164 normalisation; 1.0 equal, 0.9 same last nine digits (one side lacks the country code), else 0.0 |
| `website` | same registrable domain (`shop.example.co.uk` = `example.co.uk`); for shared hosts (Facebook, Instagram, ...) the page has to match too, otherwise no evidence |
| `category` | compatibility from the taxonomy: 1 same, 0.5 same group (cafe vs restaurant), overrides (bar~pub 0.9, bank~atm 0.5), 0 unrelated |
| `distance` | half-life decay `0.5 ** (d / scale)`, where the scale depends on category: 15 m for an ATM, 40 m for a cafe, 300 m for a mall, 2 km for an airport |

Name normalisation, step by step: case folding (`İ ı I i` all become `i`; `ß` becomes `ss`;
`lang="de"` turns `ü` into `ue`), accent folding (`ş→s`, `é→e`, `ø→o`, `ł→l`), dotted acronyms
joined (`A.Ş.→as`), apostrophes dropped (`McDonald's→mcdonalds`), `&→and`, other punctuation
removed, abbreviations expanded, trailing legal forms stripped, stop-words removed (never
down to an empty name).

Abbreviations that are ambiguous are expanded **only by position**: `St. Mary` → `saint mary`
(leading), `Main St` → `main street` (trailing), and a `st` in the middle stays `st`. The same
goes for `Dr.` (doctor/drive). Unambiguous ones (`ave`, `blvd`, `rd`, `cd` → `caddesi`, `sk` →
`sokak`, `hauptstr` → `hauptstrasse`, ...) are always expanded.

### 3. Scoring

**`WeightedScorer`** (default): a weighted mean over the features that are present, so a missing
phone neither helps nor hurts. Default weights: names 0.15 + 0.15 + 0.15, address 0.12, phone
0.12, website 0.06, category 0.10, distance 0.15; default threshold 0.70.
`scorer.explain(features)` returns each term's contribution.

**`LogisticScorer`**: trained from labelled pairs with `train_logistic` (NumPy Newton/IRLS with
an L2 penalty, or `backend="sklearn"`). Each optional feature gets a presence flag, so the
model can learn that a *different* phone number is evidence against a match. Inference is
pure Python; models are saved as small JSON files.

Threshold tuning: `precision_recall_curve(scores, labels)` and
`best_threshold(scores, labels, min_precision=0.98)` (or max F1). Pass `n_positive` when some
true pairs were never scored, so recall stays honest.

### 4. Clustering

- `union_find` (default): matches are applied strongest first, and two clusters merge only if
  the merged cluster stays within `max_diameter_m` (default 2 × radius), keeps at most one
  record per source (`one_per_source=True`) and violates no cannot-link pair. A weak bridging
  match cannot chain two different places together.
- `components`: connected components, and any component wider than `max_diameter_m` is split
  by removing its weakest edges.

Both return a partition of all records (singletons included). Invariants are tested on random
graphs.

### 5. Golden records

`Survivorship` sets one rule per field:

| rule | picks |
|---|---|
| `source_priority` | the value from the most trusted source that has one |
| `most_complete` | the longest value |
| `most_recent` | the value from the record with the latest `updated` timestamp |
| `majority` | the most frequent value after normalisation (names without legal suffixes, phones as E.164, websites as hosts) |

Defaults: name, category and phone use `majority`; address and website use `most_complete`;
every other column (`attrs`) uses `default_rule`. Ties go to source priority, then recency,
then record id, so results are deterministic. Coordinates use `mean` (spherical centroid),
`medoid` or `source_priority`. Every field records its provenance (`value`, `record_id`,
`source`, `rule`, `support`, `candidates`), and entity ids are a stable hash of the member ids.

### Taxonomy

`Taxonomy.default()` knows about 35 canonical categories with English, Turkish and German
aliases and understands OSM tags (`amenity=cafe`), Overture-style ids (`coffee_shop`) and
hierarchical labels (`eat_and_drink > cafe`, `food.cafe`). To extend it, use a JSON file
(`--taxonomy`):

```json
{
  "categories": {"vending": {"group": "retail", "distance_scale_m": 5, "aliases": ["vending machine"]}},
  "compatibility": [["vending", "convenience", 0.3]],
  "source_mappings": {"google": {"Espresso bar": "cafe"}}
}
```

## Benchmark

`poi-dedupe benchmark` generates two independent synthetic cities from different seeds. The
threshold is tuned and the logistic model trained on the first; all numbers below are measured
on the second, unseen one. Three sources (`osm`, `overture`, `registry`) contain each place with
probability 0.85 / 0.75 / 0.60, plus 2 % same-source duplicates. The noise includes coordinate
jitter scaled by category (5 % of records get three times the jitter), typos, case and accent
changes, legal suffixes, district suffixes, dropped words, `St.`/`Cd.` abbreviations, missing
fields, five phone formats, URL variants and a separate category vocabulary per source. Chain
brands (Starbucks, Migros, Shell, bank ATMs, ...) give hard negatives: same name and website,
different branch. All names and numbers are invented.

Metrics are pairwise against complete ground truth. "Pair" scores the matched pairs, "cluster"
scores the pairs implied by the final entities.

**Standard city**: 1000 entities in 5 × 5 km, 2221 records, radius 250 m. Blocking: 20 273
candidate pairs, pair completeness 0.977, reduction ratio 0.992.

| model | threshold | pair P | pair R | pair F1 | cluster P | cluster R | cluster F1 | entities (true 1000) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| weighted (default) | 0.700 | 0.990 | 0.966 | 0.978 | 0.973 | 0.975 | 0.974 | 1018 |
| weighted (tuned) | 0.692 | 0.988 | 0.966 | 0.977 | 0.960 | 0.975 | 0.967 | 1016 |
| logistic | 0.463 | 0.989 | 0.971 | 0.980 | 0.959 | 0.979 | 0.969 | 1012 |

**Dense city** (`--extent 1500 --radius 150`): the same 1000 entities in 1.5 × 1.5 km, so
far more look-alike neighbours. Blocking: 71 653 candidate pairs, pair completeness 0.947,
reduction ratio 0.971.

| model | threshold | pair P | pair R | pair F1 | cluster P | cluster R | cluster F1 | entities (true 1000) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| weighted (default) | 0.700 | 0.926 | 0.936 | 0.931 | 0.828 | 0.950 | 0.885 | 1006 |
| weighted (tuned) | 0.731 | 0.952 | 0.929 | 0.940 | 0.904 | 0.944 | 0.923 | 1035 |
| logistic | 0.575 | 0.982 | 0.923 | 0.952 | 0.944 | 0.940 | 0.942 | 1053 |

How to read these numbers:

- The **recall ceiling is set by blocking**. 2.3 % (standard) and 5.3 % (dense, smaller radius)
  of the true pairs lie beyond the radius because of the heavy-jitter records, and no model can
  recover them. Pick the radius from how far apart your sources really put the same place.
- In the dense city the **default threshold over-merges**: cluster precision (0.83) is well
  below pair precision (0.93) because transitive chains join look-alike neighbours. Tuning the
  threshold or training the logistic model on a few hundred labelled pairs from *your* data
  closes most of that gap.
- **Synthetic data is not your data.** The generator is a controlled stress test, not a claim
  about accuracy on real OSM / Overture / registry data, where coordinate offsets, naming
  conventions and missing fields follow other distributions. Label a sample of
  `poi-dedupe pairs` output and use `evaluate` / `train` before you trust a threshold.
- Runtime (pure Python, one core): about 2 s per configuration in the standard city and about
  6 s in the dense one (≈ 72 k pairs). Cost grows with the number of candidate pairs, that is
  with density × radius².

Reproduce with `poi-dedupe benchmark` and `poi-dedupe benchmark --extent 1500 --radius 150`
(add `--json` for the full report).

## API overview

| area | objects |
|---|---|
| data | `POI`, `CandidatePair`, `ScoredPair`, `GoldenRecord`, `pair_key` |
| pipeline | `Deduper`, `DedupeResult`, `dedupe` |
| blocking | `generate_candidates`, `grid_blocks`, `geohash_blocks`, `reduction_ratio` |
| normalisation | `normalize_text`, `normalize_name`, `parse_address`, `normalize_phone` |
| similarity | `jaro`, `jaro_winkler`, `ratio`, `token_sort_ratio`, `token_set_ratio`, `TfidfVectorizer`, `tfidf_cosine`, `address_similarity`, `phone_similarity`, `website_similarity`, `distance_decay` |
| features | `FeatureExtractor`, `FEATURE_NAMES` |
| taxonomy | `Taxonomy`, `Category` |
| scoring | `WeightedScorer`, `LogisticScorer`, `train_logistic`, `precision_recall_curve`, `best_threshold`, `save_model`, `load_model` |
| clustering | `cluster_union_find`, `cluster_components`, `UnionFind` |
| merge | `Survivorship`, `merge_cluster`, `merge_clusters` |
| evaluation | `Metrics`, `evaluate_labelled_pairs`, `evaluate_against_truth`, `evaluate_clusters`, `blocking_recall`, `truth_pairs` |
| synthetic | `make_synthetic`, `SyntheticDataset` |
| I/O | `ColumnMap`, `read_records`, `read_csv`, `read_geojson`, `read_parquet`, `write_golden`, `write_pairs`, `read_pairs`, `read_labels`, `read_truth`, `records_from_dataframe`, `golden_to_dataframe`, `pairs_to_dataframe` |

## Limitations

- Phone handling is deliberately simple (no numbering-plan validation); pass
  `default_country_code` when a source uses national formats.
- Registrable domains use a short built-in list of multi-part suffixes (`co.uk`, `com.tr`, ...),
  not the full Public Suffix List.
- Pure Python favours portability over speed: plan on roughly 10 k pair comparisons per
  second. Very dense inputs benefit from a smaller radius or `category_blocking=True`.
- Input geometries are points; polygons (malls, campuses) should be reduced to a representative
  point first.

## Development

```bash
pip install -e ".[dev]"
pytest --cov=poi_dedupe
ruff check . && ruff format --check src tests && mypy src
```

## Citation

See [CITATION.cff](CITATION.cff).

## License

MIT © Yusuf Eminoğlu
