Metadata-Version: 2.4
Name: pk_address_checker
Version: 1.0.1
Summary: Lenient Pakistani address checker with bundled registry data
License: Proprietary
Keywords: pakistan,address,validation
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: pdfplumber

# pk_address_checker — Documentation

Lenient, deterministic Pakistani address checker. The registry database
(CSVs) ships **inside** this package — install and check offline, no downloads,
no update run required. 


```bash
pip install pk_address_checker
```

Address checking only reads the data files, so any install can check
addresses. `update` is different: it rewrites the data files in place inside
the installed package, so the installed location must be writable by your
account. Install somewhere writable if you plan to update the data later:

```bash
# a virtual environment (recommended):
python -m venv pkvenv

# activate it — Windows:
pkvenv\Scripts\activate

# activate it — Linux/macOS:
source pkvenv/bin/activate

# then install:
pip install pk_address_checker

# or skip the venv and use your user site-packages:
pip install --user pk_address_checker
```

If your install is not writable and you cannot reinstall, you can point
`update` at a different data folder instead — see "Using a different data
folder" below.

To use in your python code:

```python
from pk_address_checker import check_address, is_valid_address

result = check_address("Shop 12, Tehsil Mankera, Rahim Yar Khan")
print(result["valid"])          # False
print(result["reason_codes"])   # ['tehsil_city_conflict']

is_valid_address("Street 5, Gulistan-e-Johar, Karachi")  # True
```

To use from Robot Framework:

```robot
*** Settings ***
Library    pk_address_checker

*** Test Cases ***
Check An Address
    ${result}=    Check Address    Shop 12, Main Bazar, Chak 45-JB, Tehsil Shorkot, District Jhang    province=Punjab
    Should Be True    ${result}[valid]

Reject A Contradiction
    ${result}=    Check Address    Shop 12, Tehsil Mankera, Rahim Yar Khan    province=Punjab
    Should Not Be True    ${result}[valid]
    Should Contain    ${result}[reason_codes]    tehsil_city_conflict

Quick Check
    ${ok}=    Is Valid Address    Street 5, Gulistan-e-Johar, Karachi
    Should Be True    ${ok}
```

Robot Framework turns each function into a keyword automatically
(`check_address` → `Check Address`, `is_valid_address` → `Is Valid Address`).
The dict result is read with `${result}[key]`.

- Stdlib only for checking addresses (`re`, `csv`, `difflib`); Python 3.8+.
  `pdfplumber` (installed automatically as a dependency) powers the data-update
  pipeline's PDF parsers.
- Ambiguity never causes rejection — only stated contradictions reject.
- Refresh the bundled data later with `python -m pk_address_checker update`
  — install somewhere writable first (see above) so the update can rewrite
  the data files in place.

## Using a different data folder

By default the data files live inside the installed package, in its
`pk_admin_data` folder. To use a different folder instead (for example, to
share one data folder between projects, or because the package is installed
somewhere read-only), set the `PK_ADMIN_DATA_DIR` environment variable
**before importing** `pk_address_checker` — it is read once, at import time.
When it is set, both address checking and `update` read and write that
folder.

The package never copies or creates its data files there on its own — a
fresh folder stays empty. Seed it yourself: copy the installed package's
`pk_admin_data` folder (all its CSV files) into your folder. Find that
folder with:

```bash
python -c "import pk_address_checker, os; print(os.path.join(os.path.dirname(pk_address_checker.__file__), 'pk_admin_data'))"
```

Point the variable only at a folder that already has the data files. Without
them the checker loses the registry it compares against: real contradictions
slip through undetected, and valid addresses can be wrongly rejected. Once
the data files are in place, `update` refreshes them in that folder and
leaves the installed package's own copy untouched.

- PowerShell: `$env:PK_ADMIN_DATA_DIR = "C:\pkdata"`
- Python, before the import: `os.environ["PK_ADMIN_DATA_DIR"] = r"C:\pkdata"`

## Updating the data

The package ships with a snapshot of the registry tables. To refresh them from
the official online sources — the PBS census tables, housing-authority scheme
lists, OpenStreetMap, Wikipedia and the Pakistan Post API — run:

```bash
python -m pk_address_checker update            # refresh in place
python -m pk_address_checker update --force    # also accept a much smaller result than before
```

This needs internet access. The update runs its phases one after another —
admin tables, societies, OpenStreetMap neighborhoods, Wikipedia landmarks,
postcodes and a Google phase — and prints what it found for each. The
city-by-city phases save their progress after every city, so an interrupted
update keeps everything it had already fetched. New rows are merged into
the existing tables inside the package's `pk_admin_data` folder (or the
`PK_ADMIN_DATA_DIR` folder, when set); old rows are never deleted, and a
source that fails (offline, or its page layout changed) keeps its previous
rows — an update can never leave you with less data than before.

- `update` rewrites the data files in place, so install somewhere writable
  first (see above). If your install is read-only: first copy the installed
  package's `pk_admin_data` files into a writable folder, then set
  `PK_ADMIN_DATA_DIR` to that folder (see "Using a different data folder"
  above) — the update writes there instead.
- `pdfplumber`, the PDF parser for the census and authority documents, is
  installed automatically with the package.

&nbsp;

# API Reference

Every function in the library, with its full documentation.

## Quick start

```python
from pk_address_checker import check_address, is_valid_address

# Plain True / False
is_valid_address("Shop 5, Skardu, Gilgit-Baltistan")   # True
is_valid_address("asdf qwerty")                         # False

# Full result, with the reasons an address was rejected
result = check_address("Street 12, House 12, Karachi")
result["reason_codes"]                                  # ['generic_address']
```

`check_address` returns a dict with the keys `valid`, `reasons`, `reason_codes` and `parsed`. See [`check_address`](#check_address) for every reason code.


## Contents

- [Public API](#public-api) (7) — The functions and the class you call directly.
- [Data tables](#data-tables) (5) — Loaders for the CSV-backed registries that the checks rely on.
- [Update pipelines](#update-pipelines) (7) — Functions that download source data and rebuild the CSV registries (run by the CLI `update` command).
- [Conflict checks](#conflict-checks) (10) — Checks that detect contradictions inside an address, such as a tehsil that is not in the stated district. Most return a reason string, or `None` when nothing conflicts.
- [Parsing helpers](#parsing-helpers) (15) — Internal helpers that read the address text: finding cities, anchors, markers and postcodes.
- [Matching and registry helpers](#matching-and-registry-helpers) (20) — Internal helpers for fuzzy name matching and for looking up the cached registries.
- [Download and parse helpers](#download-and-parse-helpers) (22) — Internal helpers that download and parse the source documents (PDFs, HTML pages, web APIs) behind the update pipelines.
- [Nested helpers](#nested-helpers) (8) — Methods of the two HTML parser classes.

---

## Public API

The functions and the class you call directly.

| Name | What it does |
|---|---|
| [`check_address`](#check_address) | Lenient plausibility check of one Pakistani address. |
| [`is_valid_address`](#is_valid_address) | True when the address passes, False when it is rejected. |
| [`parse_address`](#parse_address) | Extract the parts of an address deterministically (no model: every field is found in the text itself, so nothing can hallucinate). |
| [`Parsed`](#parsed) | The parts extracted from one address by `parse_address()`. |
| [`tehsil_district_conflict`](#tehsil_district_conflict) | True only when the tehsil is known, the stated district is itself a real district, AND none of the tehsil's known districts resembles it. |
| [`normalize_province`](#normalize_province) | Map a free-text province string to its canonical name. |
| [`norm`](#norm) | Normalise a tehsil/district name to a comparable token. |

### `check_address`

Lenient plausibility check of one Pakistani address.

```python
check_address(address: str, province: str | None = None) -> dict
```

**How it works**

1. Empty/whitespace address → `{"valid": False, "reasons": ["address is empty"], "reason_codes": ["empty_address"], "parsed": None}`.
2. `parse_address()` extracts the fields; every rejection below is added as a (code, reason) pair and the final "valid" is simply whether any pair fired:
   - **a.** `foreign_country`: parse says another country is stated.
   - **b.** `gibberish`: no readable place vocabulary.
   - **c.** `no_place`: no city/tehsil/district in the text, no known city word, no numbered place unit — judged on the TEXT so nothing can false-fire it.
   - **d.** `not_locatable` / `generic_address` / `city_conflict`: when the text fails to pin a specific point AND there is no tehsil/district AND no non-registry city (a small city/village is only `not_locatable` when BOTH tehsil and district are missing — a stated tehsil or district pins the area); else a two-city contradiction (`_city_mismatch`); else, in a STRICT big city, `generic_address` when there is neither a (named anchor + physical marker) nor a town anchor.
   - **e.** `province_conflict`: the stated province (from the form argument and/or the text) contradicts any extracted place field.
   - **f.** `tehsil_district_conflict`, `district_conflict`, `city_district_conflict`, `tehsil_city_conflict`, `tehsil_conflict`: the contradiction checks; when any fires, the vague `"not_locatable"` pair is dropped (a stated contradiction IS the verdict).
   - **g.** `sector_conflict`, `locality_conflict`, `society_conflict`, `postcode_conflict`.
3. Duplicate pairs are removed; the result dict carries "valid", the parallel "reasons" (human text) and `"reason_codes"` (stable codes) lists, and the parsed fields via `asdict(p)`.

**Parameters**

- `address` — Raw address text (any case).
- `province` — Optional province selected elsewhere in your form; free text, normalised internally, and checked together with any province in the text.

**Returns:** `dict` — `{"valid": bool, "reasons": [...], "reason_codes": [...], "parsed": {...}}`; reasons and `reason_codes` are empty when valid. Codes: `empty_address`, `foreign_country`, `gibberish`, `no_place`, `not_locatable`, `generic_address`, `city_conflict`, `province_conflict`, `tehsil_district_conflict`, `district_conflict`, `city_district_conflict`, `tehsil_city_conflict`, `tehsil_conflict`, `society_conflict`, `sector_conflict`, `locality_conflict`, `postcode_conflict`.

**Side effects:** None.

**Raises:** Never raises (every stage degrades to "no evidence" on bad input).

**Example**

```pycon
>>> check_address("Shop 12, Main Bazar, Chak 45-JB, Tehsil Shorkot, District Jhang", province="Punjab")["valid"]
True
>>> check_address("Postal Code: 54830, Karachi")["valid"]   # 54830 belongs to Lahore
False
>>> check_address("Street 12, House 12, Karachi")["reason_codes"]
['generic_address']
```

**Why it works this way:** Lenient by design: in a strict city a bare "Street 12, House 12" needs a named neighbourhood/town/sector/landmark (`generic_address`); smaller cities and villages stay lenient ("Chak 12, Tehsil X" is enough). Society/locality checks fire only for OFFICIAL places (authority-approved or well-known official geography) — those cannot exist in an unregistered city; unofficial-but-real places (e.g. Ghauri Town, Islamabad) pass.

**Related:** Called by `is_valid_address()`; calls `parse_address()` and every `_*_conflict` check.

---

### `is_valid_address`

True when the address passes, False when it is rejected.

```python
is_valid_address(address: str, province: str | None = None)
```

**How it works**

1. Run `check_address()`.
2. Return True only when "valid" is exactly True.

**Parameters**

- `address` — Raw address text.
- `province` — Optional province selected elsewhere in your form.

**Returns:** `bool` — True = valid, False = rejected. Always a plain bool — there is no third state.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> is_valid_address("Shop 5, Skardu, Gilgit-Baltistan")
True
>>> is_valid_address("asdf qwerty")
False
>>> is_valid_address("Hajira City, District Attock, District Quetta", "Khyber Pakhtunkhwa")
False
>>> is_valid_address("Street 13, Saddar, Rawalpindi", "Punjab")
True
>>> is_valid_address("Street 13, Saddar, Islamabad")   # no Saddar in the sector city
False
>>> is_valid_address("Street 232 GPO Islamabad")   # GPO anchors in the sector city
True
```

**Related:** Thin wrapper around `check_address()` for callers that want a plain bool.

---

### `parse_address`

Extract the parts of an address deterministically (no model: every field is found in the text itself, so nothing can hallucinate).

```python
parse_address(address: str) -> pk_address_checker.Parsed
```

**How it works**

1. Province: the LAST `PROVINCE_ALIASES` mention in the canonicalised text wins (the terminal province statement is the stated one — "Bank of Punjab, Lahore, Khyber Pakhtunkhwa" claims KPK), unless a road/place word follows the alias ("Islamabad Road, Khushab" is a road name).
2. District/tehsil: explicit markers in both orders ("Tehsil Shorkot" /"Shorkot Tehsil", "District X" / "X District"); district tries the LEADING order first (in "PO Hajira Dist Sudhnoti" the trailing order would capture "hajira" as the district). Captures are greedy multi-word, cut at suffix words, trimmed to a known name. The two fields parse INDEPENDENTLY — a district marker must not suppress the tehsil marker ("..., Tehsil X, District Y" states both). An unregistered marker value ("Tehsil Hassan Abdal" trimmed to "Hassan") is nulled so it cannot suppress the registry walk.
3. City: the earliest positioned `_city_hints` word (district/business mentions excluded). Fallback: a Title-Case phrase directly before a province/country word or postcode ("Village X, Choondiko, 66170, Sindh"), rejected when it is a known city, shorter than 4, or contains generic/business words.
4. Registry tehsil/district recognition (no markers): walk words; each field fills independently (a stated district must not suppress tehsil recognition — "PO Hajira Dist Sudhnoti" IS a tehsil claim). Multi-word tehsils match as whole phrases (longest span, cut at suffix words, locality names skipped, all-generic phrases skipped); exact registry names win, then near-miss spellings resolve at the 88 bar. Guards: business/road follower words ("Palandri Rd", "Multan Sohan Halwa", "store" ~ "astore"), a preceding locality-type word ("muhalla naseerabad"), a Title-Case proper noun before a tehsil word ("Badami Bagh" is a locality tail), and the cross-province guard (a registry word belonging to another province than the stated city is a locality name — unless comma-stated, which IS a place statement). A fuzzy district fills only when it would not contradict the filled tehsil (`tehsil_district_conflict` gate: fuzzy only ADDS acceptance).
5. No city/tehsil/district at all: lenient place fallback — with at least two Title-Case (or ALL-CAPS, grain-market style) non-generic names, the last one becomes the city ("Tatapani Rd Hajira Poonch" → Poonch).
6. Locality via `_locality_from_text`; a locality made only of registry tehsil/district tokens ("Hajira Poonch" is tehsil+district) is nulled.
7. Set `country_is_pakistan_or_unstated` (no `_foreign_country` hit), `is_gibberish` (`_is_gibberish`), and locatable (locality, `_locatable` markers, plus code, or known-postcode anchor).

**Parameters**

- `address` — Raw address text (any case).

**Returns:** `Parsed` — The extracted fields; None means "not found in the text".

**Side effects:** None.

**Raises:** Never raises (empty or unreadable input yields a `Parsed` of Nones).

**Example**

```pycon
>>> parse_address("Shop 12, Main Bazar, Chak 45-JB, Tehsil Shorkot, District Jhang, Punjab")
Parsed(country_is_pakistan_or_unstated=True, is_gibberish=False, locatable=True,
       business_name=None, locality='Tehsil Shorkot District', city='Jhang',
       tehsil='Shorkot', district='Jhang', province='Punjab')
```

**Why it works this way:** Lenient by design: ambiguity never causes rejection, and every guard above exists to stop a mention that is NOT a claim (business, road, locality tail) from being treated as one.

**Related:** Called by `check_address()` and the conflict checks (`_tehsil_city_conflict` consumers); calls `_canonicalize`, `_city_hints`, `_locality_from_text`, `_foreign_country`, `_is_gibberish`, `_locatable`, `_postcode_anchor`.

---

### `Parsed`

The parts extracted from one address by `parse_address()`.

```python
Parsed(
    country_is_pakistan_or_unstated: bool = True,
    is_gibberish: bool = False,
    locatable: bool = True,
    business_name: str | None = None,
    locality: str | None = None,
    city: str | None = None,
    tehsil: str | None = None,
    district: str | None = None,
    province: str | None = None,
) -> None
```

**Fields**

- `country_is_pakistan_or_unstated` — True when no foreign country/city is stated at the end of the text. None is never used here (default True).
- `is_gibberish` — True when the text has no readable words or only keyboard-mash tokens (default False).
- `locatable` — True when a specific point is pinned (locality, street/landmark marker, plus code or known postcode). None is never used here (default True).
- `business_name` — Absorbs shop/plaza/restaurant names; the parser always leaves it None.
- `locality` — The area/township name found in the text, or None.
- `city` — The city found in the text, or None.
- `tehsil` — The tehsil found in the text, or None.
- `district` — The district found in the text, or None.
- `province` — The province found in the text, or None.

None on locality/city/tehsil/district/province means "not found in the text" — the checks treat a missing field as no evidence, never as a contradiction.

---

### `tehsil_district_conflict`

True only when the tehsil is known, the stated district is itself a real district, AND none of the tehsil's known districts resembles it.

```python
tehsil_district_conflict(tehsil, district, min_score: float = 88) -> bool
```

**How it works**

1. Missing tehsil or district → False.
2. Resolve the tehsil through `_alias`/norm; if unknown, accept the BEST fuzzy match in the registry only when it scores >= `min_score`, else False.
3. Resolve the district; if it is not a real district even fuzzily, return False (a stray place name like "Shorkot" stated as a district must stay lenient).
4. Return False when any of the tehsil's registered districts matches the stated district exactly or fuzzily (>= `min_score`); True otherwise.

**Parameters**

- `tehsil` — Tehsil name (may be None or '').
- `district` — District name (may be None or '').
- `min_score` — Fuzzy acceptance bar (default 88).

**Returns:** `bool` — True = the pair contradicts the registry; False = consistent, unknown, or not enough evidence.

**Side effects:** None.

**Raises:** Never raises (an empty table yields no conflict).

**Example**

```pycon
>>> tehsil_district_conflict("Shorkot", "Sialkot")
True
>>> tehsil_district_conflict("Shorkot", "Jhang")
False
>>> tehsil_district_conflict(None, "Jhang")
False
```

**Related:** Called by `check_address()` and `parse_address()` (fuzzy district fill).

---

### `normalize_province`

Map a free-text province string to its canonical name.

```python
normalize_province(text: str | None) -> str | None
```

**How it works**

1. Empty/None → None.
2. Strip non-letter chars and lowercase; an exact `PROVINCE_ALIASES` key returns the canonical name ("kpk" → "Khyber Pakhtunkhwa").
3. Otherwise scan longer aliases as whole words inside the text ("Sindh Province" → "Sindh").
4. Unknown text → None.

**Parameters**

- `text` — Free-text province name, or None.

**Returns:** `Optional[str]` — The canonical province name, or None for empty/unknown text.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> normalize_province("kpk")
'Khyber Pakhtunkhwa'
>>> normalize_province("Sindh Province")
'Sindh'
>>> normalize_province("somewhere")
None
```

**Related:** Called by `province_conflict()`, `check_address()`, `parse_address()`.

---

### `norm`

Normalise a tehsil/district name to a comparable token.

```python
norm(s: str) -> str
```

**How it works**

1. Lowercase; every non-alphanumeric char becomes a space.
2. `_SUFFIX` strips trailing "tehsil"/"tahsil"/"taluka"/"taluk"/"district"/"city"/"town"/"cantt(cantonment)"/"sub-division" words.
3. All remaining whitespace is removed.

**Parameters**

- `s` — Any tehsil/district name string.

**Returns:** `str` — The squashed lowercase token ('Shor Kot Tehsil' and 'SHORKOT' both become 'shorkot').

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> norm("Dera Ghazi Khan")
'deraghazikhan'
>>> norm("Shor Kot Tehsil")
'shorkot'
```

**Related:** Called by nearly every registry lookup (`tehsil_district_conflict`, `_place_provinces`, `parse_address`, the conflict checks).

---

## Data tables

Loaders for the CSV-backed registries that the checks rely on.

| Name | What it does |
|---|---|
| [`load_admin_table`](#load_admin_table) | Load the full district/tehsil table with hand-maintained overrides applied. |
| [`load_society_table`](#load_society_table) | Load (name, location) society rows with hand-maintained overrides applied. |
| [`load_locality_table`](#load_locality_table) | Load (name, city) locality rows with hand-maintained overrides applied. |
| [`load_islamabad_sectors`](#load_islamabad_sectors) | Load the Islamabad sector grid with hand-maintained overrides applied. |
| [`load_postcodes`](#load_postcodes) | Load all known 5-digit Pakistani postcodes (existence only). |

### `load_admin_table`

Load the full district/tehsil table with hand-maintained overrides applied.

```python
load_admin_table() -> tuple
```

**How it works**

1. Read `admin_units.csv` as (province, district, tehsil) keys.
2. Apply `admin_overrides.csv`: `action=add` inserts the key, `action=remove` discards it.
3. Return a sorted tuple; the result is cached for the process lifetime (`lru_cache`) and only the update pipeline clears it.

**Returns:** `tuple` — Sorted (province, district, tehsil) triples.

**Side effects:** None (reads two CSVs; the `lru_cache` holds the result until cleared).

**Raises:** Never raises (missing CSVs read as empty).

**Example**

```pycon
>>> ("Azad Kashmir", "Poonch", "Hajira") in load_admin_table()
True
```

**Related:** Called by `tehsil_district_conflict()`, `_place_provinces()`, `parse_address()` and most conflict checks.

---

### `load_society_table`

Load (name, location) society rows with hand-maintained overrides applied.

```python
load_society_table() -> tuple
```

**How it works**

1. Read `societies.csv` rows lowercased.
2. Apply `society_overrides.csv`: rows hand-edited with unquoted commas spill into an extra field, which is joined back onto the location; `action=remove` filters out the name, `action=add` appends it.
3. Return a sorted tuple (cached for the process lifetime).

**Returns:** `tuple` — Sorted (name, location) pairs.

**Side effects:** None (reads two CSVs; the `lru_cache` holds the result until cleared).

**Raises:** Never raises (missing CSVs read as empty).

**Example**

```pycon
>>> ("chaudhari muhammad ali housing scheme", "faisalabad") in load_society_table()
True
```

**Related:** Called by `_locality_from_text()`, `_named_anchor()`, `_locality_conflict()`, `_society_conflict()`.

---

### `load_locality_table`

Load (name, city) locality rows with hand-maintained overrides applied.

```python
load_locality_table() -> tuple
```

**How it works**

1. Read `localities.csv` lowercased; skip rows whose name is itself a known city (bbox-overlap junk), a universal place name (`_UNIVERSAL_NAMES`: "Main Bazar", "Saddar" — they exist in every city), equal to the row's own city, or contains a comma (concatenated fragments, never a real place name).
2. Apply `locality_overrides.csv`: `action=remove` filters the name, `action=add` appends it.
3. Return a sorted tuple (cached for the process lifetime).

**Returns:** `tuple` — Sorted (name, city) pairs.

**Side effects:** None (reads two CSVs; the `lru_cache` holds the result until cleared).

**Raises:** Never raises (missing CSVs read as empty).

**Example**

```pycon
>>> ("hayatabad", "peshawar") in load_locality_table()
True
```

**Why it works this way:** Only official, well-known geography is registered; informal/unofficial area names are deliberately absent, so unknown names pass everywhere.

**Related:** Called by `_locality_from_text()`, `_locality_index()`, `_locality_conflict()`, `_locality_norms()` and `parse_address()`.

---

### `load_islamabad_sectors`

Load the Islamabad sector grid with hand-maintained overrides applied.

```python
load_islamabad_sectors() -> frozenset
```

**How it works**

1. Read `islamabad_sectors.csv` uppercased into a set.
2. Apply `islamabad_sector_overrides.csv`: `action=remove` discards the sector, `action=add` inserts it (values uppercased).
3. Return a frozenset. Not cached: the update pipeline may rewrite the CSV mid-run.

**Returns:** `frozenset` — Uppercase sector codes like "F-8".

**Side effects:** None.

**Raises:** Never raises (missing CSVs read as empty).

**Example**

```pycon
>>> "F-8" in load_islamabad_sectors()
True
```

**Related:** Called by `_marker()` and `_sector_conflict()`.

---

### `load_postcodes`

Load all known 5-digit Pakistani postcodes (existence only).

```python
load_postcodes() -> frozenset
```

**How it works**

1. Read the postcode column of `postcodes.csv` into a frozenset.

**Returns:** `frozenset` — Postcode strings ("54830", ...). Details live in the CSV.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises (a missing CSV reads as empty).

**Example**

```pycon
>>> "74000" in load_postcodes()
True
```

**Related:** Called by `_marker()`, `_named_anchor()`, `_postcode_anchor()`.

---

## Update pipelines

Functions that download source data and rebuild the CSV registries (run by the CLI `update` command).

| Name | What it does |
|---|---|
| [`update_admin_table`](#update_admin_table) | Download and rebuild the district/tehsil table (`admin_units.csv`) from the official PDFs. |
| [`update_society_table`](#update_society_table) | Rebuild `societies.csv` from the official housing-authority sources. |
| [`update_neighborhoods_osm`](#update_neighborhoods_osm) | OpenStreetMap phase of the neighborhood pipeline: union city neighborhoods + landmarks into `localities.csv` with per-city incremental commits. |
| [`update_neighborhoods_wiki`](#update_neighborhoods_wiki) | Wikipedia landmark phase of the neighborhood pipeline: union city landmark names into `localities.csv` with per-city incremental commits. |
| [`update_world_landmarks`](#update_world_landmarks) | Wikipedia landmark phase for world cities: union foreign landmarks into `localities.csv` with per-city incremental commits. |
| [`update_postcodes`](#update_postcodes) | Harvest Pakistan Post postcode records into `postcodes.csv` and merge their area names into `localities.csv`. |
| [`update_neighborhoods`](#update_neighborhoods) | Scrape Google Search results for each big city's neighborhoods and union them into `localities.csv`. |

### `update_admin_table`

Download and rebuild the district/tehsil table (`admin_units.csv`) from the official PDFs.

```python
update_admin_table(sources=None, force: bool = False) -> dict
```

**How it works**

1. Read the existing table and index it by (province, district, tehsil).
2. Per province in sources: fetch the first working PDF, extract its text, parse it. Failures are per-province: an empty parse, or a fresh parse less than half the previous row count (unless `force=True`), keeps the province's OLD rows and is recorded in failed — a broken link never wipes existing data.
3. Merge with UNION semantics: old rows are kept (people still use old addresses), and a freshly parsed tehsil row whose tehsil already belongs to a DIFFERENT district in the old table is dropped (the yearbook layouts attach boundary rows ambiguously, and a wrong pair is worse than a missing one).
4. Refuse an empty or tehsil-less result outright (RuntimeError).
5. Write via a .tmp file and atomic replace; clear the load caches.

**Parameters**

- `sources` — Optional `{province: [candidate URLs or paths]}` map; defaults to `PBS_SOURCES`.
- `force` — Accept a fresh parse even when it is much smaller than the previous one.

**Returns:** `dict` — `{"total", "added", "removed", "failed", "updated"}` — failed maps each failing province to its error text; added/removed are the sorted key triples.

**Side effects:** Downloads PDFs into the cache, writes `meta.json` (ETags), rewrites `admin_units.csv`, clears the `load_admin_table` and `_index` caches.

**Raises:** RuntimeError when nothing could be loaded, or when no province contributes any tehsil row.

**Example**

```pycon
>>> update_admin_table(force=False)["total"] >= 750
True
```

**Why it works this way:** Union-not-replace and drop-the-ambiguous-pair are deliberate: old addresses must keep validating, and a wrong tehsil/district pairing is worse than a missing one.

**Related:** Called by the CLI (`python -m pk_address_checker update`); feeds `load_admin_table()`.

---

### `update_society_table`

Rebuild `societies.csv` from the official housing-authority sources.

```python
update_society_table(sources=None, force: bool = False) -> dict
```

**How it works**

1. Load existing rows; legacy "Punjab" source labels become "Punjab (PHATA)".
2. Per source: fetch, parse; an empty parse or a fresh parse less than half the previous row count (unless `force=True`) records the failure and keeps that source's old rows — a broken page never wipes existing data.
3. Merge with UNION semantics: rows of successful sources replace their previous rows by (name, location) key, rows of failed sources and hand-added rows are kept, identical entries collapse into one.
4. Refuse an empty result outright (RuntimeError); write via a .tmp file and atomic replace; clear the load cache.

**Parameters**

- `sources` — Optional `{label: [(url, kind, default_city)]}` map; defaults to `SOCIETY_SOURCES`.
- `force` — Accept a fresh parse even when it is much smaller than the previous one.

**Returns:** `dict` — `{"total", "failed", "updated"}` — failed maps each failing source label to its error text.

**Side effects:** Downloads into the cache, writes `meta.json`, rewrites `societies.csv`, clears the `load_society_table` cache.

**Raises:** RuntimeError when nothing could be loaded.

**Example**

```pycon
>>> update_society_table(force=False)["total"] > 0
True
```

**Why it works this way:** Union-not-replace: people still use old scheme addresses, so old rows stay valid; explicit removals go through `society_overrides.csv` only.

**Related:** Called by the CLI; feeds `load_society_table()`.

---

### `update_neighborhoods_osm`

OpenStreetMap phase of the neighborhood pipeline: union city neighborhoods + landmarks into `localities.csv` with per-city incremental commits.

```python
update_neighborhoods_osm(cities=None, force: bool = False) -> dict
```

**How it works**

1. Rebuild clean (name, city, source) rows from `localities.csv`.
2. Per city: `_osm_neighborhoods()` (Nominatim bbox + Overpass queries); require 5..1000 rows else fail; a fresh parse less than half the previous osm rows fails unless `force=True`; failures print and keep the city's previous rows.
3. Rows get `source="osm"` and union in; after EACH city the CSV is rewritten (incremental commit — progress survives interruption and stays observable), then sleep 0.5s.
4. Clear the locality cache at the end.

**Parameters**

- `cities` — Optional list of city names; defaults to `_BIG_CITIES`.
- `force` — Accept a fresh parse even when it is much smaller than the previous one.

**Returns:** `dict` — `{"total", "failed", "updated"}` — failed maps each failing city to its error text.

**Side effects:** Nominatim + Overpass HTTP requests, prints per-city progress, rewrites `localities.csv` once per city, sleeps, clears the `load_locality_table` cache.

**Raises:** Never raises (per-city failures are recorded and skipped).

**Example**

```pycon
>>> update_neighborhoods_osm(cities=["Skardu"])["failed"]
{}
```

**Related:** Called by the CLI; sibling of `update_neighborhoods_wiki()` and `update_world_landmarks()`.

---

### `update_neighborhoods_wiki`

Wikipedia landmark phase of the neighborhood pipeline: union city landmark names into `localities.csv` with per-city incremental commits.

```python
update_neighborhoods_wiki(cities=None, force: bool = False) -> dict
```

**How it works**

1. Rebuild clean (name, city, source) rows from `localities.csv`.
2. Per city: `_wiki_landmarks()`; require 5..1000 rows else fail; a fresh parse less than half the previous wikipedia rows fails unless `force=True`; failures print and keep the city's previous rows (cities without landmark categories fail leniently).
3. Rows get `source="wikipedia"` and union in; after EACH city the CSV is rewritten (incremental commit), then sleep 0.5s.
4. Clear the locality cache at the end.

**Parameters**

- `cities` — Optional list of city names; defaults to `_BIG_CITIES`.
- `force` — Accept a fresh parse even when it is much smaller than the previous one.

**Returns:** `dict` — `{"total", "failed", "updated"}` — failed maps each failing city to its error text.

**Side effects:** MediaWiki API requests, prints per-city progress, rewrites `localities.csv` once per city, sleeps, clears the `load_locality_table` cache.

**Raises:** Never raises (per-city failures are recorded and skipped).

**Example**

```pycon
>>> update_neighborhoods_wiki(cities=["Lahore"])["failed"]
{}
```

**Related:** Called by the CLI; sibling of `update_neighborhoods_osm()`.

---

### `update_world_landmarks`

Wikipedia landmark phase for world cities: union foreign landmarks into `localities.csv` with per-city incremental commits.

```python
update_world_landmarks(cities=None, force: bool = False) -> dict
```

**How it works**

1. Rebuild clean (name, city, source) rows from `localities.csv`.
2. Per city: `_wiki_landmarks()`; require 3..1000 rows else fail; failures print and keep the city's previous rows.
3. Rows get `source="wikipedia-world"` and union in; after EACH city the CSV is rewritten (incremental commit), then sleep 0.5s.
4. Clear the locality cache at the end.

**Parameters**

- `cities` — Optional list of world city names; defaults to `_WORLD_CITIES`.
- `force` — Accepted for interface parity with the other `update_*` functions; not consulted in this one.

**Returns:** `dict` — `{"total", "failed", "updated"}` — failed maps each failing city to its error text.

**Side effects:** MediaWiki API requests, prints per-city progress, rewrites `localities.csv` once per city, sleeps, clears the `load_locality_table` cache.

**Raises:** Never raises (per-city failures are recorded and skipped).

**Example**

```pycon
>>> update_world_landmarks(cities=["Paris"])["failed"]
{}
```

**Why it works this way:** The landmarks land in the registry with the FOREIGN city, so any Pakistani city hint then contradicts them ("Statue of Liberty, Karachi" becomes impossible).

**Related:** Called by the CLI; feeds `_landmark_names()` and `_locality_conflict()`.

---

### `update_postcodes`

Harvest Pakistan Post postcode records into `postcodes.csv` and merge their area names into `localities.csv`.

```python
update_postcodes(force: bool = False) -> dict
```

**How it works**

1. Query every province plus every big/strict city (the province listing caps at ~200 rows, so city queries are needed for coverage); each query fails independently and keeps previous rows (a query returning fewer than 20 rows fails).
2. Union rows into `postcodes.csv` keyed by postcode (postcode, city, province, `area_name`) via a .tmp file and atomic replace.
3. Merge every `area_name` into `localities.csv` with `source="postcode"` (union, dedupe, hand-edited rows kept) — the area names feed the locality registry, and contradictions fire through the EXISTING `locality_conflict` with no new rejection logic.
4. Clear the postcode + locality caches.

**Parameters**

- `force` — Accepted for interface parity with the other `update_*` functions; not consulted in this one.

**Returns:** `dict` — `{"total", "failed", "updated"}` — failed maps each failing query label to its error text.

**Side effects:** HTTP requests to the postcode API, prints per-query progress, rewrites `postcodes.csv` and `localities.csv`, clears the `load_postcodes` and `load_locality_table` caches.

**Raises:** Never raises (per-query failures are recorded and skipped).

**Example**

```pycon
>>> update_postcodes()["total"] > 1000
True
```

**Related:** Called by the CLI; feeds `load_postcodes()` and `load_locality_table()`.

---

### `update_neighborhoods`

Scrape Google Search results for each big city's neighborhoods and union them into `localities.csv`.

> **Note:** this pipeline is currently dead — Google returns consent/bot pages — but stays wired for when it works again.

```python
update_neighborhoods(cities=None, force: bool = False) -> dict
```

**How it works**

1. Rebuild clean (name, city, source) rows from `localities.csv` (hand edits with stray commas are dropped: they would crash the write below).
2. Per city: request the SERP up to 3 times with growing sleeps (Google rate-limits); a page shorter than 20000 bytes is a consent/bot page and fails the attempt.
3. Parse name candidates (`_parse_google_neighborhoods`); require 5..1000 rows (the error message says 5-150 — a known stale string) else fail this city.
4. Guard: a fresh parse less than half the city's previous google-maps rows fails unless `force=True`. Failures record the error and keep the city's old rows.
5. Union rows with `source="google-maps"` (identical (name, city) entries collapse); sleep a random 2-6s between cities.
6. Write via a .tmp file and atomic replace; clear the locality cache.

**Parameters**

- `cities` — Optional list of city names; defaults to `_BIG_CITIES`.
- `force` — Accept a fresh parse even when it is much smaller than the previous one.

**Returns:** `dict` — `{"total", "failed", "updated"}` — failed maps each failing city to its error text.

**Side effects:** Makes Google Search requests, sleeps/retries, rewrites `localities.csv`, clears the `load_locality_table` cache.

**Raises:** Never raises (per-city failures are recorded and skipped).

**Example**

```pycon
>>> update_neighborhoods(cities=["Lahore"])["failed"]  # dead pipeline
{'Lahore': 'parsed 0 names, outside the plausible range (5-150) -- junk parse or Google blocked the request'}
```

**Related:** Called by the CLI after the other pipeline phases; feeds `load_locality_table()`.

---

## Conflict checks

Checks that detect contradictions inside an address, such as a tehsil that is not in the stated district. Most return a reason string, or `None` when nothing conflicts.

| Name | What it does |
|---|---|
| [`province_conflict`](#province_conflict) | True only when a known city clearly sits in a different province than stated. |
| [`_city_mismatch`](#_city_mismatch) | Two different known cities claimed in one address. |
| [`_district_conflict`](#_district_conflict) | Two different districts explicitly stated in one address. |
| [`_city_district_conflict`](#_city_district_conflict) | A stated city whose EVERY stated 'District X' contradicts it. |
| [`_tehsil_mismatch`](#_tehsil_mismatch) | Find two distinct known tehsils claimed in one address, or one claimed tehsil that contradicts the stated city / stated district. |
| [`_tehsil_city_conflict`](#_tehsil_city_conflict) | An explicitly MARKED tehsil whose registry districts exclude the stated city's districts. |
| [`_society_conflict`](#_society_conflict) | An OFFICIAL society appears with a city its registered location does not include. |
| [`_sector_conflict`](#_sector_conflict) | A known Islamabad sector in the text with a city that is not Islamabad. |
| [`_locality_conflict`](#_locality_conflict) | An official, well-known locality in the text with a city it does not belong to ('Hayatabad' is Peshawar, not Islamabad). |
| [`_postcode_conflict`](#_postcode_conflict) | An explicitly LABELED, KNOWN postcode whose registered city contradicts the text. |

### `province_conflict`

True only when a known city clearly sits in a different province than stated.

```python
province_conflict(place: str | None, province: str | None) -> bool
```

**How it works**

1. Normalise the province; missing place or province → False (nothing to compare, stay lenient).
2. Search the hand-curated `CITY_TO_PROVINCE` list as whole words in the place text; a hit returns whether the stated province is NOT among the allowed set.
3. Otherwise resolve the place via the admin table (`_place_provinces`): a name unique to one province conflicts when the stated province differs.
4. Unknown place → False.

**Parameters**

- `place` — An extracted place field (city/tehsil/district text), not raw address text.
- `province` — A province string (free text; normalised internally).

**Returns:** `bool` — True = known cross-province contradiction; False = consistent, unknown, or missing input.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> province_conflict("Karachi", "Punjab")
True
>>> province_conflict("Karachi", "Sindh")
False
>>> province_conflict("Depalpur", "Balochistan")
True
```

**Why it works this way:** Works on extracted fields, not raw text, so road names such as "Peshawar Road, Rawalpindi" or "Punjab Colony, Karachi" don't trigger it. The hand-curated `CITY_TO_PROVINCE` list is the authority; registry-only names resolve through the admin table.

**Related:** Called by `check_address()`.

---

### `_city_mismatch`

Two different known cities claimed in one address.

```python
_city_mismatch(address: str) -> str | None
```

**How it works**

1. Collect the positioned city hints, in text order; a "gilgit" hint directly followed by "baltistan" is the province phrase, not a city claim, and is skipped.
2. Fewer than two hints → None.
3. First vs last hint: identical → None. Different → conflict UNLESS the first hint is also a registered TEHSIL of the last hint's district ("Sahiwal" is a Sargodha tehsil, so "Sahiwal Road, Sargodha" names a tehsil, not a second city). The last mention is the stated city; an earlier different one contradicts it.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — A reason string ("'attock' is not in 'quetta'"), or None when there is no two-city contradiction.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _city_mismatch("Attock, Quetta")
"'attock' is not in 'quetta'"
>>> _city_mismatch("Shop 5, Sahiwal, Sargodha")
None
```

**Related:** Called by `check_address()`.

---

### `_district_conflict`

Two different districts explicitly stated in one address.

```python
_district_conflict(address: str) -> str | None
```

**How it works**

1. Canonicalise the text and collect stated districts (`_stated_districts`).
2. Drop values that are not real districts — and not real cities either ("District Karachi" counts too).
3. Keep only the first occurrence of each distinct district norm, in order.
4. Two or more distinct districts → a conflict, UNLESS the two are fuzzy-similar (ratio >= 88, i.e. repeats or near-miss spellings of the SAME district, which stay silent).

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — A reason string naming the two districts, or None when there is no two-district contradiction.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _district_conflict("District Lodhran ... District Lahore")
"'Lodhran' and 'Lahore' are two different districts"
>>> _district_conflict("District Lahore ... District Lahore")
None
```

**Related:** Called by `check_address()`.

---

### `_city_district_conflict`

A stated city whose EVERY stated 'District X' contradicts it.

```python
_city_district_conflict(address: str, district: str | None) -> str | None
```

**How it works**

1. Missing district, no stated districts, or no positioned city hints → None.
2. For each positioned hint with a registry entry (`_city_districts`): if NO stated district matches any of its districts (exact or fuzzy 88), return the conflict naming the hint.
3. One matching district statement ('District Lahore' for a Lahore address) or an unknown city silences it.

**Parameters**

- `address` — Raw address text.
- `district` — The parsed district field (used only for the message).

**Returns:** `Optional[str]` — A reason string, or None when there is no city-vs-districts contradiction.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _city_district_conflict("BHAKKAR, District Ludhran", "Ludhran")
"'bhakkar' is not in district 'Ludhran'"
```

**Why it works this way:** Only explicit district markers count as evidence — a registry-filled district is not a statement.

**Related:** Called by `check_address()`.

---

### `_tehsil_mismatch`

Find two distinct known tehsils claimed in one address, or one claimed tehsil that contradicts the stated city / stated district.

```python
_tehsil_mismatch(address: str) -> str | None
```

**How it works**

1. Canonicalise the text and walk its words; commas delimit fields (a follower word in the NEXT field cannot qualify the claim).
2. Match known tehsil names as whole phrases, longest span first ("Kot Addu", "Dera Ghazi Khan"); alias-resolve each candidate.
3. Drop a match when: a registered locality name covers the same or a longer span (locality-first), every token is generic/descriptor ("Model Town" names no specific tehsil), the name is itself a district ("Baloch Tehsil, Sudhnoti": Sudhnoti states the district), a road/business/place-type follower word appears within the next 3 same-field words ("Shah Faisal - Korangi Link", "Multan Sohan Halwa"), or the previous word is a district word ("District Jhang" states a district, not a tehsil). The follower checks are typo-tolerant (`_looks_like`).
4. Two or more DISTINCT claims → a conflict naming the first and last.
5. One claim: it still contradicts the stated city when its province (unique registry names only) differs from the last positioned city hint's province ("Jand, Quetta"); same-province pairs ("Jand, Attock") stay valid. It also contradicts an explicitly stated "District X" that excludes all of its registered districts ("Silanwali, District Sialkot").

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — A reason string, or None when there is no tehsil contradiction.

**Side effects:** None.

**Raises:** Never raises (an empty table yields no claims).

**Example**

```pycon
>>> _tehsil_mismatch("Palandri Hajira Poonch")
"'Palandri' is not in 'Hajira' tehsil"
>>> _tehsil_mismatch("Palandri Rd, Hajira, Poonch")
None
```

**Why it works this way:** Multi-word tehsils are matched as whole phrases: a single-word scan never recognises "Kot Addu" (one token, "kotaddu"), so real contradictions involving one would silently pass.

**Related:** Called by `check_address()`; calls `_alias()`, `_locality_norms()`, `_place_provinces()`, `_stated_districts()`, `_city_hints()`.

---

### `_tehsil_city_conflict`

An explicitly MARKED tehsil whose registry districts exclude the stated city's districts.

```python
_tehsil_city_conflict(address: str, p) -> str | None
```

**How it works**

1. Look for a "Tehsil X" marker (leading or trailing order); no marker → None (ambient names stay ambiguous).
2. Missing parsed tehsil, unknown tehsil, no positioned city hints → None.
3. Exclude city hints INSIDE the marker's own span ("TEHSIL FATEH JHANG" — "jhang" is a typo tail, not a city); the last remaining hint is the stated city.
4. Conflict when the city has registry districts and none of them overlaps the tehsil's districts, OR the tehsil's province (unique names only) differs from the city's province. Any district overlap silences it (same-district pairs like "Tehsil Taxila, Rawalpindi" stay valid).

**Parameters**

- `address` — Raw address text.
- `p` — The `Parsed` object from `parse_address()`.

**Returns:** `Optional[str]` — A reason string, or None when there is no marked-tehsil contradiction.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _tehsil_city_conflict("Tehsil Mankera, Rahim Yar Khan", parse_address("Tehsil Mankera, Rahim Yar Khan"))
"'Mankera' is not in 'rahim yar khan'"
```

**Related:** Called by `check_address()`.

---

### `_society_conflict`

An OFFICIAL society appears with a city its registered location does not include.

```python
_society_conflict(address: str) -> str | None
```

**How it works**

1. No city hints → None.
2. Match registered society names in the text (`_name_matches`); no matches → None.
3. Scope gate: the check only polices the STRICT big-city set (the scheme lists are curated per-city for the big metros, not for every small town) — EXCEPT mega societies (Bahria Town, DHA), which are authority-approved and impossible outside their registered cities even in small towns.
4. Conflict when NO matched society has ANY registered location consistent with a city hint (exact membership or fuzzy 88 against each location word). Any consistent hint silences it.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — A reason string naming the first matched society, or None.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _society_conflict("Bahria Town Gujarat DHA 73")
"'bahria town' is not located in the city stated"
```

**Why it works this way:** Official societies cannot exist in unregistered cities, so this is impossible; UNOFFICIAL societies (not in the registry, e.g. Ghauri Town) pass everywhere.

**Related:** Called by `check_address()`.

---

### `_sector_conflict`

A known Islamabad sector in the text with a city that is not Islamabad.

```python
_sector_conflict(address: str) -> str | None
```

**How it works**

1. `_SECTOR_RE` finds sector codes in either order ("F-8 Markaz" or "Markaz F-8"); the markaz/sector suffix keeps bare letter-number shop codes ("Shop F-8, Karachi") from matching.
2. Intersect with the registered sector grid; unknown sectors → None.
3. City hints: none, or only islamabad/rawalpindi → None (Rawalpindi borders Islamabad and people write sectors there). Any other hint conflicts.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — A reason string naming the first known sector, or None.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _sector_conflict("Shop 3, F-8 Markaz, Karachi")
"'F-8' is an Islamabad sector, not in the city stated"
>>> _sector_conflict("Shop 3, F-8 Markaz, Islamabad")
None
```

**Related:** Called by `check_address()`.

---

### `_locality_conflict`

An official, well-known locality in the text with a city it does not belong to ('Hayatabad' is Peshawar, not Islamabad).

```python
_locality_conflict(address: str) -> str | None
```

**How it works**

1. No city hints, or no STRICT-city hint → None (the data is scraped — Google/OSM/Wikipedia — and reliably curated only for the true metros).
2. Collect candidate locality names via the token index; keep those matching the text (`_name_matches`).
3. Drop matches that cannot drive a conflict: names made only of generic place-type words ("bazaar"), names containing a marker/landmark-type word ("Civil Hospital Chowk" exists everywhere) — except the explicit `_LANDMARK_EXCEPTIONS` (Kashmir Highway, Qissa Khwani Bazar, Tariq Road) — and numbered units ("block 7" exists in every city).
4. A registered SOCIETY that is consistent with the city silences the conflict ("DHA Phase 5, Lahore" vs an Islamabad-only "dha phase 5" row).
5. Conflict only when EVERY matched locality contradicts every city hint (fuzzy 88 against each registered city); any consistent pair passes.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — A reason string naming the first matched locality and its registered cities, or None.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _locality_conflict("Shop 3, Hayatabad, Islamabad")
"'hayatabad' is in peshawar, not the city stated"
```

**Why it works this way:** Unknown names (informal/unofficial areas) pass everywhere, and the strict-city gate keeps a small town's generic locality/market name from being judged by some unrelated big city's scrape (e.g. "Bara Bazaar" registered under Peshawar) — the checker's contract is to reject only clear problems.

**Related:** Called by `check_address()`.

---

### `_postcode_conflict`

An explicitly LABELED, KNOWN postcode whose registered city contradicts the text.

```python
_postcode_conflict(address: str) -> str | None
```

**How it works**

1. Extract the labeled postcode; no label → None.
2. Look up its registered city; an UNKNOWN code returns None (positive-only: a stale postcode table must never reject a real address).
3. Compare the city hints in the text against the registered city (fuzzy bar 88); hints that resemble the registered city silence the check, any other hint conflicts.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — A reason string naming the postcode and its registered city, or None when there is no labeled known postcode or the hints agree.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _postcode_conflict("Postal Code: 54830, Karachi")   # 54830 belongs to Lahore
'postcode 54830 belongs to lahore, not the city stated'
>>> _postcode_conflict("Postal Code: 54830, Lahore")
None
>>> _postcode_conflict("Postal Code: 99999, Karachi")   # unknown code
None
```

**Related:** Called by `check_address()`.

---

## Parsing helpers

Internal helpers that read the address text: finding cities, anchors, markers and postcodes.

| Name | What it does |
|---|---|
| [`_city_hints`](#_city_hints) | Find known cities mentioned in the text, exact or near-miss. |
| [`_hint_positions`](#_hint_positions) | Find the text position of each city hint. |
| [`_stated_districts`](#_stated_districts) | Extract all explicitly stated districts as (display, norm) pairs. |
| [`_locality_from_text`](#_locality_from_text) | Determine the locality deterministically: the longest registry locality/society name present in the text, else a Title-Case area-name phrase. |
| [`_marker`](#_marker) | Test whether the text carries a physical marker (road/street/near/opp/shop/house number, plus code, postcode or sector code). |
| [`_named_anchor`](#_named_anchor) | Test whether the text names a real place anchor: a locality that traces back to the written text, a tehsil/district, a plus code, a labeled known postcode, a near/opposite landmark reference, or a registry-known locality or society name. |
| [`_town_anchor`](#_town_anchor) | A known tehsil that is not the STRICT city in the text names a town. |
| [`_foreign_country`](#_foreign_country) | Find the first clearly-stated foreign country or foreign CITY at the END of the address. |
| [`_locatable`](#_locatable) | True when the text pins a specific point. |
| [`_is_gibberish`](#_is_gibberish) | No readable words at all, or every word is a keyboard mash. |
| [`_postcode_anchor`](#_postcode_anchor) | A clearly mentioned, KNOWN postcode pins a specific postal area. |
| [`_clear_postcode`](#_clear_postcode) | Extract a clearly mentioned postcode: labeled, or a bare 5-digit code in the standard trailing position AFTER a known city name ('Karachi 74000'). |
| [`_labeled_postcode`](#_labeled_postcode) | Extract an explicitly LABELED postcode ('Postal Code: 54830'). |
| [`_canonicalize`](#_canonicalize) | Map near-miss words to their canonical keyword so marker/district detection tolerates misspells. |
| [`_canon_word`](#_canon_word) | Map a near-miss word to its canonical keyword. |

### `_city_hints`

Find known cities mentioned in the text, exact or near-miss.

```python
_city_hints(text: str) -> set
```

**How it works**

1. Canonicalise the text (marker/district keywords typo-corrected) and index the words; commas split the text into fields (a word in the NEXT field cannot qualify the city: "BHAKKAR, District Layyah" still states Bhakkar).
2. Walk the words; multi-word cities are matched as whole phrases, longest first, so "Rahim Yar Khan" beats a stray "khan". Each candidate is dropped (`_excluded`) when the next same-field word is a district word ("Wah Cantt, Rawalpindi District" is NOT a Rawalpindi-city claim), a business word within the next 3 same-field words ("Hyderabad Sweets", "London Bakery"), or a road/place-type follower ("Peshawar Road").
3. A single-word city that heads a known tehsil/district phrase ("Nowshera Virkan") is part of that phrase, not a city claim — unless it is comma-stated, which IS a city statement.
4. Unmatched words are checked against `CITY_ALIASES` (deterministic variants), then a fuzzy bar of 88 for near-miss city words (only when >= 4 letters and not excluded).
5. All found hints are re-mapped through `CITY_ALIASES` so consumers see canonical names.

**Parameters**

- `text` — Address text (any case; canonicalised internally).

**Returns:** `set` — Canonical lowercase city names found in the text; empty when none.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _city_hints("Hyderabad Sweets, Karachi")
{'karachi'}
>>> _city_hints("Shop 5, Rawalpindi District")
set()   # a district mention is not a city claim
```

**Why it works this way:** The exclusion words are matched loosely (`_looks_like`) because a misspelled follower ("Sweeets", "Distt") must not turn a business or landmark name into a false city claim — a missed follower here cascades into false city/locality/society conflicts downstream.

**Related:** Called by `parse_address()` and every city-aware conflict check.

---

### `_hint_positions`

Find the text position of each city hint.

```python
_hint_positions(text: str, hints) -> dict
```

**How it works**

1. Tokenise the lowercased text with spans.
2. For each hint, search the exact hint word, then any `CITY_ALIASES` variant word ("sarghodha" for "sargodha").
3. Hints with no position are omitted — fuzzy hints get no position, because fuzzy matching must never drive a rejection, only resolve real variants.

**Parameters**

- `text` — Lowercased address text.
- `hints` — Iterable of city hints (from `_city_hints`).

**Returns:** `dict` — `{hint: character position}`; hints found only fuzzily are absent.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _hint_positions("shop 5, sargodha", {"sargodha"})
{'sargodha': 8}
```

**Related:** Called by `_city_mismatch()`, `_city_district_conflict()`, `_tehsil_city_conflict()`.

---

### `_stated_districts`

Extract all explicitly stated districts as (display, norm) pairs.

```python
_stated_districts(text: str) -> list
```

**How it works**

1. Run both district-marker regexes ("District X" leading, "X District" trailing) over the canonicalised text.
2. Skip values naming province/country/state; cut the capture at the first suffix word (tehsil/district/city/...).
3. Trim trailing words until the remainder (alias-resolved) is a KNOWN district norm — the capture is greedy ("District Rahim Yar Khan"), and words that are not districts ("District Court") drop away entirely.
4. Unknown values never reach the output.

**Parameters**

- `text` — Canonicalised address text.

**Returns:** `list of tuple` — `[(display name, district norm), ...]` in text order.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _stated_districts("District Rahim Yar Khan")
[('Rahim Yar Khan', 'rahimyarkhan')]
>>> _stated_districts("District Court Road, Lahore")   # "Court" is not a district
[]
```

**Related:** Called by `_district_conflict()`, `_city_district_conflict()`, `_tehsil_mismatch()`.

---

### `_locality_from_text`

Determine the locality deterministically: the longest registry locality/society name present in the text, else a Title-Case area-name phrase.

```python
_locality_from_text(address: str, city) -> str | None
```

**How it works**

1. Collect candidate registry locality names via the token index; the LONGEST name that matches the text and contains no road word wins (road words are roads, not areas).
2. No registry locality: try registered society names (same longest-wins rule); mega societies (DHA, Bahria Town) in a BIG city need a numbered phase/sector/block, like in `_named_anchor`.
3. Still nothing: walk Title-Case words; the first 2-3 word phrase (span 3 then 2) that is not a known city, does not end in the stated city, contains no road word, and LEADS with a proper noun (not all-generic "main bazar", not descriptor-led "Main Qambrani") becomes the locality. "Kabul Town" qualifies; "House 5 Kabul" does not. A single universal place name that is a proper noun ("Saddar", "gpo") also qualifies — it names the city's own central area and anchors (acceptance only; it can never drive a rejection). One exception: "Saddar" does NOT anchor in Islamabad (a planned sector city with no Saddar area).

**Parameters**

- `address` — Raw address text.
- `city` — The parsed city (used to exclude phrases that are the city or end in it, and for the mega-society gate).

**Returns:** `Optional[str]` — The Title-Cased locality name, or None.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _locality_from_text("Shop 3, Tibbi Saikhwaan, Jhang", "Jhang")
'Tibbi Saikhwaan'
>>> _locality_from_text("Main Bazar, Rawalakot", None)
None
>>> _locality_from_text("Street 232 GPO Islamabad", "Islamabad")
'Gpo'
```

**Related:** Called by `parse_address()`.

---

### `_marker`

Test whether the text carries a physical marker (road/street/near/opp/shop/house number, plus code, postcode or sector code).

```python
_marker(address: str) -> bool
```

**How it works** *(any hit → True)*

1. A plus code (`_PLUS_CODE`) in the text.
2. A clearly mentioned postcode (`_clear_postcode`) that is in the known-postcodes registry.
3. `_MARKER_RE` against the canonicalised text: road/street/lane/link/highway/avenue words, chowk/market/bazar, near/opp/opposite/behind, masjid/school/hospital/station/bank/hotel/restaurant/plaza, shop/house/plot/flat/office/stand numbers, markaz/chak/mouza/mandi.
4. `_SECTOR_RE`: a sector code in either order ("F-8 Markaz", "Markaz F-8").
5. A registered LANDMARK name (`_landmark_names`) matched in the text — "Mazar-e-Quaid, Karachi" needs no road; area rows like "ghauri town phase 1" are NOT landmarks.
6. A bare Islamabad sector code that exists in the registry ("F-7 Islamabad").
7. A named business (`_NAMED_BUSINESS`: "Karachi Biryani") — it pins its own location.

**Parameters**

- `address` — Raw address text.

**Returns:** `bool` — True when any marker is present; False otherwise.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _marker("Main Road, DHA 3 Lahore")
True
>>> _marker("DHA 3 Lahore")
False
```

**Related:** Called by `check_address()` (the `generic_address` rule).

---

### `_named_anchor`

Test whether the text names a real place anchor: a locality that traces back to the written text, a tehsil/district, a plus code, a labeled known postcode, a near/opposite landmark reference, or a registry-known locality or society name.

```python
_named_anchor(address: str, parsed) -> bool
```

**How it works** *(any hit → True)*

1. `_field_anchor(parsed.locality)`, then (`parsed.tehsil` / `parsed.district`). A field value anchors when it is non-empty, not a bare city word, not a marker-led phrase, not all-generic ("street 5"), not a road name, not a district (broad area), a tehsil that differs from the stated city ("Nowshera Virkan" inside Gujranwala), or any other value that matches the written text (`_name_matches`).
2. A plus code in the address.
3. A clearly mentioned, KNOWN postcode (bare numbers never do).
4. A near/opposite/behind/in-front-of reference.
5. A registered locality name matched in the text (candidates come from the token index; road-word names excluded).
6. A registered society name matched in the text — except MEGA societies (DHA, Bahria Town) in a BIG city, where a bare mention is too vague and only a numbered phase/sector/block anchors ("DHA 3 Lahore").
7. A NAMED road ("Peshawar Road", "4-Lyton Road") whose proper noun leads (descriptors like "Main Qambrani" don't count).
8. A numbered place unit ("Chak No 42", "Mouza 116") or a named business ("Karachi Biryani").

**Parameters**

- `address` — Raw address text.
- `parsed` — The `Parsed` object from `parse_address()`.

**Returns:** `bool` — True when any named anchor is present; False otherwise.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> p = parse_address("Nowshera Virkan, Gujranwala")
>>> _named_anchor("Nowshera Virkan, Gujranwala", p)
True
```

**Related:** Called by `check_address()` (the `generic_address` rule); calls `_name_matches()`, `_locality_index()`, `load_society_table()`.

---

### `_town_anchor`

A known tehsil that is not the STRICT city in the text names a town.

```python
_town_anchor(address: str, p) -> bool
```

**How it works**

1. No parsed tehsil → False.
2. A universal name ("Saddar") with the stated city Islamabad → False (the planned sector city has no Saddar area).
3. True when the tehsil is a registered tehsil name AND is not one of the strict-city hints in the text.

**Parameters**

- `address` — Raw address text.
- `p` — The `Parsed` object from `parse_address()`.

**Returns:** `bool` — True when the parsed tehsil pins a specific town.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _town_anchor("Nowshera Virkan, Gujranwala", parse_address("Nowshera Virkan, Gujranwala"))
True
```

**Why it works this way:** A known tehsil inside a stated city names a town, which is specific enough on its own — no separate road/marker is needed.

**Related:** Called by `check_address()` (the `generic_address` rule).

---

### `_foreign_country`

Find the first clearly-stated foreign country or foreign CITY at the END of the address.

```python
_foreign_country(address: str) -> str | None
```

**How it works**

1. Scan the `_COUNTRIES` list for a name at the very end of the text (optionally followed by digits); return the first hit.
2. Then scan the `_FOREIGN_CITIES` list the same way, tolerating a city suffix word ("Kabul City", "Kabul Shehr").
3. Mid-text mentions are business/road names, not claims, and return None.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — The foreign country/city name, or None when none is stated at the end.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _foreign_country("Shop 3, Lahore, Dubai")
'dubai'
>>> _foreign_country("Kabul Hotel, Shorkot")
None
```

**Why it works this way:** Checked only at the end (possibly followed by a code) so a business name like "India Plaza, Lahore" is not flagged.

**Related:** Called by `parse_address()`.

---

### `_locatable`

True when the text pins a specific point.

```python
_locatable(address: str) -> bool
```

**How it works**

1. A plus code (`_PLUS_CODE`) in the text, or
2. `_ANCHOR_RE` against the canonicalised text: street/road/lane/link/highway/avenue, chowk/market/bazar/colony/sector/phase/block/society/mohalla/town/goth/basti/chak/mouza/village/dhera/dera, near/opposite/behind, masjid/school/hospital/station/bank/hotel/restaurant/plaza, or a shop/house/plot/flat/office/stand number.

**Parameters**

- `address` — Raw address text.

**Returns:** `bool` — True when a street, landmark, village marker or number is present.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _locatable("Street 12, House 12, Karachi")
True
>>> _locatable("Karachi")
False
```

**Related:** Called by `parse_address()`.

---

### `_is_gibberish`

No readable words at all, or every word is a keyboard mash.

```python
_is_gibberish(address: str) -> bool
```

**How it works**

1. Collect lowercase words of length >= 2.
2. True when no word reaches 3 letters, or when all words are in the `_MASH` set (asdf/qwerty rows, aaa/bbb repeats, test/dummy/sample/lorem/ipsum...).

**Parameters**

- `address` — Raw address text.

**Returns:** `bool` — True when the text carries no readable place vocabulary.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _is_gibberish("asdf qwerty zxcv")
True
>>> _is_gibberish("Shop 5, Karachi")
False
```

**Related:** Called by `parse_address()`.

---

### `_postcode_anchor`

A clearly mentioned, KNOWN postcode pins a specific postal area.

```python
_postcode_anchor(address: str) -> bool
```

**How it works**

1. `_clear_postcode` extracts the labeled or trailing postcode.
2. True only when it is in the known-postcodes registry; unknown codes do not anchor.

**Parameters**

- `address` — Raw address text.

**Returns:** `bool` — True when a known postcode is clearly mentioned.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _postcode_anchor("Karachi 74000")
True
>>> _postcode_anchor("Karachi 99999")
False
```

**Related:** Called by `parse_address()`.

---

### `_clear_postcode`

Extract a clearly mentioned postcode: labeled, or a bare 5-digit code in the standard trailing position AFTER a known city name ('Karachi 74000').

```python
_clear_postcode(address: str) -> str | None
```

**How it works**

1. A labeled postcode (`_labeled_postcode`) always wins.
2. Otherwise take the LAST 5-digit group in the text; it counts only when it ends the address and a known city word appears before it.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — The 5-digit code, or None when no clear postcode is present.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _clear_postcode("Postal Code: 54830")
'54830'
>>> _clear_postcode("Karachi 74000")
'74000'
>>> _clear_postcode("Plot 54830, Clifton, Karachi")
None
```

**Why it works this way:** Used for ANCHORING only — the bare trailing form never triggers a rejection, because no model can tell a plot number from a postcode.

**Related:** Called by `_marker()`, `_named_anchor()`, `_postcode_anchor()`.

---

### `_labeled_postcode`

Extract an explicitly LABELED postcode ('Postal Code: 54830').

```python
_labeled_postcode(address: str) -> str | None
```

**How it works**

1. Regex for "post code" / "postal code" / "postcode" / "zip(code)" with an optional colon/hash followed by exactly 5 digits.
2. Return the digits or None.

**Parameters**

- `address` — Raw address text.

**Returns:** `Optional[str]` — The 5-digit code when an explicit label precedes it, else None (bare digits in the text are never "labeled").

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _labeled_postcode("Postal Code: 54830, Lahore")
'54830'
>>> _labeled_postcode("Karachi 74000")
None
```

**Why it works this way:** This is the ONLY postcode form allowed to trigger a rejection.

**Related:** Called by `_clear_postcode()` and `_postcode_conflict()`.

---

### `_canonicalize`

Map near-miss words to their canonical keyword so marker/district detection tolerates misspells.

```python
_canonicalize(text: str) -> str
```

**How it works**

1. Lowercase the text, then replace every [a-z]+ word through `_canon_word`.
2. Punctuation and digits are preserved (commas delimit fields for the district/tehsil regexes).

**Parameters**

- `text` — Address text.

**Returns:** `str` — The text with keyword-like words canonicalised.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _canonicalize("Shop 5, Distict Lahore")
'shop 5, district lahore'
```

**Related:** Called by `parse_address()`, `_city_hints()`, `_district_conflict()`, `_tehsil_mismatch()`, `_tehsil_city_conflict()`, `_locatable()`.

---

### `_canon_word`

Map a near-miss word to its canonical keyword.

```python
_canon_word(w: str) -> str
```

**How it works**

1. Exact `_WORD_ALIASES` entries win ("dist" → "district", "distt" → "district").
2. Words shorter than 3 chars, or the connectors "e"/"i", stay unchanged.
3. Otherwise the best SequenceMatcher match among `_KEYWORDS` wins when it reaches the bar: 75 for short words (<= 5 chars: "rode" → road, "neer" → near), 85 for longer ones ("distict" → district at 87.5). Words far from every keyword stay unchanged.

**Parameters**

- `w` — A lowercase word from the address.

**Returns:** `str` — The canonical keyword, or the input word unchanged.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _canon_word("distict")
'district'
>>> _canon_word("rode")
'road'
>>> _canon_word("karachi")
'karachi'
```

**Related:** Called by `_canonicalize()`.

---

## Matching and registry helpers

Internal helpers for fuzzy name matching and for looking up the cached registries.

| Name | What it does |
|---|---|
| [`_name_matches`](#_name_matches) | Test whether a registry name appears in the text. |
| [`_name_pattern`](#_name_pattern) | Build the flexible phrase matcher for a registry name. |
| [`_looks_like`](#_looks_like) | Typo-tolerant membership test: exact hit, or a close fuzzy match against a vocabulary. |
| [`_ratio`](#_ratio) | Similarity score 0-100 (stdlib SequenceMatcher). |
| [`_alias`](#_alias) | Resolve a normalised tehsil/district spelling variant to its canonical registry norm. |
| [`_index`](#_index) | Build the normalised tehsil → set of normalised districts index. |
| [`_place_provinces`](#_place_provinces) | Build the normalised tehsil/district name → province map for names unique to one province. |
| [`_universal_norms`](#_universal_norms) | Normalised forms of the universal place names (`_UNIVERSAL_NAMES`: "Saddar", "Main Bazar"...). |
| [`_locality_norms`](#_locality_norms) | Normalise every registered locality name. |
| [`_td_norms`](#_td_norms) | Normalise all known tehsil + district names. |
| [`_locality_index`](#_locality_index) | Build the token → registry names index for fast locality lookups. |
| [`_landmark_names`](#_landmark_names) | Collect registry names that are themselves landmarks. |
| [`_place_name_tokens`](#_place_name_tokens) | Collect every individual word inside a known city, tehsil or district name. |
| [`_claim_followers`](#_claim_followers) | Collect the road/business/place-type words that turn a place name into a phrase, not a claim. |
| [`_city_districts`](#_city_districts) | Build the city name norm → its districts map. |
| [`_postcode_city`](#_postcode_city) | Build the postcode → registered city map from the official data. |
| [`_clean`](#_clean) | Strip a string field; non-strings and empty strings become None. |
| [`_split`](#_split) | Split a table text line into its leading label and whether numeric columns follow. |
| [`_strip_div`](#_strip_div) | Strip a leading AJK division name from a label when more words follow. |
| [`_ok_osm_name`](#_ok_osm_name) | Reject OSM names containing vowel-less abbreviation chunks ('bahria twon skt'). |

### `_name_matches`

Test whether a registry name appears in the text.

```python
_name_matches(name: str, text: str) -> bool
```

**How it works**

1. Try the flexible phrase pattern (`_name_pattern`) first — spaces/dashes and "e"/"i" connectors are interchangeable.
2. Fall back to fuzzy window matching: the name's tokens (connectors dropped) are compared as a whole joined string against every same-length window of the text's words at the 88 bar, so near-miss spellings still match (the bar is strict: "mazaar" vs "mazar" scores 85.7 and does NOT count — the same bar that keeps "no karachi" from matching "north karachi"). Single-token names skip fuzz (too risky).
3. Additionally compare every consecutive token pair (excluding pairs containing a generic landmark-type word like "museum", which would fragment-match unrelated rows) against every word pair window — extra words around the name ("Qila Bala Hisar" vs "Bala Hissar Fortress") must not break the match.

**Parameters**

- `name` — Lowercase registry name.
- `text` — Lowercased address text.

**Returns:** `bool` — True when the name is present (pattern or fuzzy); False otherwise.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _name_matches("mazar-e-quaid", "main mazar e quaid road, karachi")
True
>>> _name_matches("north karachi", "street 12, house 12, karachi")
False
```

**Related:** Called by `_locality_from_text()`, `_marker()`, `_named_anchor()`, `_locality_conflict()`, `_society_conflict()`.

---

### `_name_pattern`

Build the flexible phrase matcher for a registry name.

```python
_name_pattern(name: str) -> re.Pattern | None
```

**How it works**

1. Split the name on spaces/dashes/commas, dropping connector words "e"/"i".
2. One token → a plain word-boundary regex. Several tokens → the first is escaped, each following token is prefixed with optional spaces/dashes and an optional "e" connector, so "gulistan-e-johar" matches "gulistan johar" and "gulshan-e-maymar" matches "gulshan maymar".
3. Patterns are memoised in the module-level `_NAME_PATTERNS` dict.

**Parameters**

- `name` — A lowercase registry name.

**Returns:** `Optional[re.Pattern]` — The compiled pattern, or None when the name had no usable tokens.

**Side effects:** Caches compiled patterns in the module-level `_NAME_PATTERNS` dict.

**Raises:** Never raises.

**Example**

```pycon
>>> bool(_name_pattern("gulistan-e-johar").search("main gulistan johar road"))
True
```

**Related:** Called by `_name_matches()`.

---

### `_looks_like`

Typo-tolerant membership test: exact hit, or a close fuzzy match against a vocabulary.

```python
_looks_like(word: str, vocab, min_len: int = 3, threshold: float = 88) -> bool
```

**How it works**

1. Empty word → False.
2. Exact membership → True regardless of length (so short followers "rd"/"st" count).
3. Words shorter than `min_len` → False.
4. Words that are themselves place-name vocabulary (`_place_name_tokens`) → False (guarded by name, see below).
5. Otherwise compare only vocab words whose length differs by at most 2, and return True when the best SequenceMatcher score reaches threshold.

**Parameters**

- `word` — A single lowercase word from the address.
- `vocab` — A set/frozenset of canonical vocabulary words.
- `min_len` — Minimum word length for the fuzzy branch (default 3).
- `threshold` — Fuzzy acceptance bar (default 88).

**Returns:** `bool` — True when the word is (or closely resembles) a member of vocab.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _looks_like("sweeets", _BUSINESS_WORDS)
True
>>> _looks_like("main", _BUSINESS_WORDS)
False
>>> _looks_like("rd", _ROAD_WORDS)
True
```

**Why it works this way:** `_canon_word`'s looser short-word bar (75) is calibrated for `_KEYWORDS`, a small set of specific, mutually-dissimilar admin/marker words. The business vocabulary is 20x larger with short generic entries ("mini", "co", "point"), where the loose bar wrongly swallowed ordinary words like "main" against "mini" — so this keeps a stricter threshold plus a length-difference guard while the place-name-token guard preserves genuine typos ("Sweeets" → "Sweets").

**Related:** Called by `_city_hints()`, `_tehsil_mismatch()`, `parse_address()`.

---

### `_ratio`

Similarity score 0-100 (stdlib SequenceMatcher).

```python
_ratio(a: str, b: str) -> float
```

**How it works**

1. Compute `SequenceMatcher(None, a, b).ratio() * 100`.

**Parameters**

- `a`, `b` — The two strings to compare.

**Returns:** `float` — 0 (no similarity) to 100 (identical).

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _ratio("karachi", "karachi")
100.0
>>> round(_ratio("nokarachi", "northkarachi"), 2)
85.71
```

**Why it works this way:** stdlib-only and deterministic on purpose: the checker's decisions must be the same in every environment.

**Related:** Called by `_looks_like`, `_canon_word`, `tehsil_district_conflict`, `_name_matches` and the conflict checks.

---

### `_alias`

Resolve a normalised tehsil/district spelling variant to its canonical registry norm.

```python
_alias(n: str) -> str
```

**How it works**

1. Look up the normalised name in the curated `_TD_ALIASES` map ("silianwali" → "sillanwali"); unknown names pass through unchanged.

**Parameters**

- `n` — A normalised (`norm()`) tehsil/district name.

**Returns:** `str` — The canonical registry norm, or the input itself.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _alias("sudhanoti")
'sudhnoti'
>>> _alias("hajira")
'hajira'
```

**Why it works this way:** Variants resolve through this map BEFORE any fuzzy fallback, so alternations like "Silanwali" (one L) are deterministic; the fuzzy 88-bar remains only as a safety net for unknown variants.

**Related:** Called by `tehsil_district_conflict()`, `_stated_districts()`, `parse_address()`, `_tehsil_mismatch()`, `_tehsil_city_conflict()`.

---

### `_index`

Build the normalised tehsil → set of normalised districts index.

```python
_index() -> dict
```

**How it works**

1. Walk `load_admin_table()`; for each tehsil row, add the district norm to the tehsil norm's set.

**Returns:** `dict` — `{tehsil_norm: {district_norm, ...}}`.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises (an empty table yields `{}`).

**Example**

```pycon
>>> "jhang" in _index().get("shorkot", set())
True
```

**Related:** Called by `tehsil_district_conflict()`, `_tehsil_mismatch()`, `_tehsil_city_conflict()`.

---

### `_place_provinces`

Build the normalised tehsil/district name → province map for names unique to one province.

```python
_place_provinces() -> dict
```

**How it works**

1. Walk `load_admin_table()`; collect every province each district/tehsil norm appears in.
2. Keep only names with exactly ONE province; ambiguous names ("khanpur", "saddar", "landhi") are left out and can never drive a conflict.

**Returns:** `dict` — `{name_norm: province}` for unambiguous names only.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises (an empty table yields `{}`).

**Example**

```pycon
>>> _place_provinces().get("depalpur")
'Punjab'
>>> _place_provinces().get("saddar")   # shared by several provinces
None
```

**Related:** Called by `province_conflict()`, `_tehsil_mismatch()`, `_tehsil_city_conflict()`.

---

### `_universal_norms`

Normalised forms of the universal place names (`_UNIVERSAL_NAMES`: "Saddar", "Main Bazar"...). A universal name exists in virtually every city, so it can NEVER be claimed as a specific tehsil and can never drive a conflict — ambiguity never causes rejection.

```python
_universal_norms() -> frozenset
```

---

### `_locality_norms`

Normalise every registered locality name.

```python
_locality_norms() -> frozenset
```

**How it works**

1. `norm()` each name in `load_locality_table()` into a frozenset.

**Returns:** `frozenset` — Normalised locality names.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises.

**Example**

```pycon
>>> norm("gulistan e johar") in _locality_norms()
True
```

**Why it works this way:** A name that IS a locality is treated as a locality (user rule: locality-first), never claimed as a tehsil.

**Related:** Called by `_tehsil_mismatch()` and `parse_address()`.

---

### `_td_norms`

Normalise all known tehsil + district names.

```python
_td_norms() -> frozenset
```

**How it works**

1. `norm()` every tehsil and district name from `load_admin_table()` into one frozenset.

**Returns:** `frozenset` — Normalised tehsil and district names.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises.

**Example**

```pycon
>>> norm("kot addu") in _td_norms()
True
```

**Related:** Called by `_city_hints()` (the city-phrase guard).

---

### `_locality_index`

Build the token → registry names index for fast locality lookups.

```python
_locality_index() -> dict
```

**How it works**

1. For each locality name, index every token of length >= 4 (connectors "e"/"i" excluded) into a set keyed by that token.

**Returns:** `dict` — `{token: {locality names containing it}}`.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises.

**Example**

```pycon
>>> "hayatabad" in _locality_index()
True
```

**Why it works this way:** Matching checks only candidates instead of scanning every one of the thousands of registry rows (keeps the check fast).

**Related:** Called by `_locality_from_text()`, `_named_anchor()`, `_locality_conflict()`.

---

### `_landmark_names`

Collect registry names that are themselves landmarks.

```python
_landmark_names() -> tuple
```

**How it works**

1. Read `localities.csv` directly (bypassing overrides); keep rows whose source is "wikipedia"/"wikipedia-world", plus any row whose name contains a landmark word (`_LANDMARK_WORDS`: statue, darbar, shrine, masjid, fort, ...).
2. Return the lowercased names as a tuple.

**Returns:** `tuple` — Lowercase landmark names (Wikipedia-sourced or landmark-word names). Plain area/town names are NOT included.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises (a missing CSV yields ()).

**Example**

```pycon
>>> "mazar-e-quaid" in _landmark_names()
True
>>> "ghauri town phase 1" in _landmark_names()
False
```

**Related:** Called by `_marker()`.

---

### `_place_name_tokens`

Collect every individual word inside a known city, tehsil or district name.

```python
_place_name_tokens() -> frozenset
```

**How it works**

1. Split every `CITY_TO_PROVINCE` city name into words.
2. Split every district and tehsil name from `load_admin_table()` into words.
3. Return the frozen set.

**Returns:** `frozenset` — Place-name vocabulary words ('khan' from 'Dera Ghazi Khan', 'addu' from 'Kot Addu').

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises.

**Example**

```pycon
>>> "addu" in _place_name_tokens()
True
```

**Why it works this way:** A word that is itself genuine Pakistani place-name vocabulary must never be swallowed as a fuzzy near-miss of an unrelated business/follower word: 'khan' resembles 'khana' (a listed business word meaning eatery) closely enough that no single similarity threshold can separate that collision from a genuine typo like 'resturent' → 'restaurant', so it is guarded by name here instead.

**Related:** Called by `_looks_like()`.

---

### `_claim_followers`

Collect the road/business/place-type words that turn a place name into a phrase, not a claim.

```python
_claim_followers() -> frozenset
```

**How it works**

1. Union `_ROAD_WORDS`, `_BUSINESS_WORDS` and `_PLACE_FOLLOWERS` into one frozenset.

**Returns:** `frozenset` — The follower vocabulary ("Palandri Rd", "Palandri Hotel", "Palandri Sweets" are not tehsil claims).

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises.

**Example**

```pycon
>>> "rd" in _claim_followers()
True
```

**Related:** Called by `parse_address()` (the registry walk) and `_tehsil_mismatch()`.

---

### `_city_districts`

Build the city name norm → its districts map.

```python
_city_districts() -> dict
```

**How it works**

1. Walk `load_admin_table()`; every district norm maps to itself, and every tehsil norm maps to its district `norm(s)`.
2. Cities split into several districts (Karachi) end up with a multi-value set, which the callers treat as "no entry" for their purposes.

**Returns:** `dict` — `{name_norm: {district_norm, ...}}`.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises.

**Example**

```pycon
>>> _city_districts()["shorkot"]
{'jhang'}
```

**Related:** Called by `_city_district_conflict()` and `_tehsil_city_conflict()`.

---

### `_postcode_city`

Build the postcode → registered city map from the official data.

```python
_postcode_city() -> dict
```

**How it works**

1. Read `postcodes.csv`; keep rows that have a city; key by postcode.

**Returns:** `dict` — `{postcode: city}`, lowercase cities.

**Side effects:** None (the `lru_cache` holds the result until cleared).

**Raises:** Never raises (a missing CSV reads as empty).

**Example**

```pycon
>>> _postcode_city().get("54830")
'lahore'
```

**Related:** Called by `_postcode_conflict()`.

---

### `_clean`

Strip a string field; non-strings and empty strings become None.

```python
_clean(value) -> str | None
```

**How it works**

1. Strings are stripped; an all-whitespace string becomes None.
2. Non-strings become None.

**Parameters**

- `value` — Any extracted field value.

**Returns:** `Optional[str]` — The stripped string, or None.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _clean("  Lahore  ")
'Lahore'
>>> _clean("   ")
None
```

**Related:** Currently unused.

---

### `_split`

Split a table text line into its leading label and whether numeric columns follow.

```python
_split(line: str)
```

**How it works**

1. Split on whitespace and scan tokens left to right.
2. The first token matching `_NUM` (a number, optionally with commas/decimals, or a bare dash) ends the label; everything before it is the label text.
3. A line with no numeric token returns the whole line with `found=False`.

**Parameters**

- `line` — One raw text line from a census table.

**Returns:** `tuple` — (label, `has_num`) — label is the text before the first numeric column ('' when the line starts with a number), `has_num` is True when a numeric token was found.

**Side effects:** None.

**Raises:** Never raises (pure function).

**Example**

```pycon
>>> _split("ATTOCK DISTRICT 123 456")
('ATTOCK DISTRICT', True)
>>> _split("some header words")
('some header words', False)
```

**Related:** Called by `parse_pbs_lines()`, `_parse_ajk_yearbook()`, `_parse_gb_glance()`.

---

### `_strip_div`

Strip a leading AJK division name from a label when more words follow.

```python
_strip_div(label: str) -> str
```

**How it works**

1. Split into words; if there is more than one word and the first is a division name (muzaffarabad / poonch / mirpur), drop it.
2. A single-word label is returned unchanged (a bare "Mirpur" IS the name).

**Parameters**

- `label` — A label from the AJK year book.

**Returns:** `str` — The label without its leading division word ('Mirpur Mirpur' → 'Mirpur', 'Muzaffarabad Sharda' → 'Sharda'), or the input unchanged.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _strip_div("Muzaffarabad Sharda")
'Sharda'
>>> _strip_div("Mirpur")
'Mirpur'
```

**Related:** Called by `_parse_ajk_yearbook()`.

---

### `_ok_osm_name`

Reject OSM names containing vowel-less abbreviation chunks ('bahria twon skt').

```python
_ok_osm_name(name: str) -> bool
```

**How it works**

1. Split the name into tokens; any token that is alphabetic, at most 4 chars and contains no vowel (a e i o u) fails the name.
2. All other names pass.

**Parameters**

- `name` — A raw OpenStreetMap place name.

**Returns:** `bool` — True when the name has no vowel-less <=4-letter alphabetic token.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _ok_osm_name("Bahria Town")
True
>>> _ok_osm_name("Bahria Town Skt")
False
```

**Why it works this way:** Abbreviation chunks are junk that must not enter the registry.

**Related:** Called by `_osm_neighborhoods()`.

---

## Download and parse helpers

Internal helpers that download and parse the source documents (PDFs, HTML pages, web APIs) behind the update pipelines.

| Name | What it does |
|---|---|
| [`fetch`](#fetch) | Download a URL into the local cache and return its path; local paths pass through. |
| [`fetch_first`](#fetch_first) | Try each candidate source in order and return the first that downloads. |
| [`pdf_lines`](#pdf_lines) | Yield the text lines of a PDF file, page by page. |
| [`parse_pbs_lines`](#parse_pbs_lines) | Walk PBS Census Table 1 text and emit (province, district, tehsil) rows. |
| [`_parse_ajk_yearbook`](#_parse_ajk_yearbook) | Parse the AJK Statistical Year Book tables 7.10 (districts) and 7.11 (tehsils). |
| [`_parse_gb_glance`](#_parse_gb_glance) | Parse the GB "at a Glance" district table (district-level rows only). |
| [`_parse_admin_lines`](#_parse_admin_lines) | Route each province's source document to its parser. |
| [`_parse_society_rows`](#_parse_society_rows) | Extract (name, location) society rows from an HTML table page (PHATA style). |
| [`_parse_cda_rows`](#_parse_cda_rows) | Extract scheme names from CDA card links (`/housing-schemes/<slug>`). |
| [`_parse_pdf_rows`](#_parse_pdf_rows) | Read (name, location) rows from a generic authority PDF table (LDA/RDA style). |
| [`_parse_society_source`](#_parse_society_source) | Parse one approved-scheme source file into (name, location) rows by its kind. |
| [`_fetch_postcode_rows`](#_fetch_postcode_rows) | Fetch ALL result rows for one postcode API query, paging past the 200-row cap. |
| [`_wiki_landmarks`](#_wiki_landmarks) | Collect a city's landmark names from Wikipedia's category system. |
| [`_osm_neighborhoods`](#_osm_neighborhoods) | Collect a city's neighborhood + landmark names from OpenStreetMap. |
| [`_overpass`](#_overpass) | Run an Overpass query against the public mirrors with retries. |
| [`_parse_google_neighborhoods`](#_parse_google_neighborhoods) | Pull neighborhood/town name candidates out of the Google results text. |
| [`_google_query`](#_google_query) | Build the Google Search URL for a city's neighbourhood query (dead pipeline). |
| [`_meta`](#_meta) | Read the pipeline metadata file. |
| [`_read_csv`](#_read_csv) | Read a CSV file as a list of dicts; a missing file reads as []. |
| [`_main`](#_main) | CLI entry point: `update` (rebuild data). |
| [`_TableParser`](#_tableparser) | Collect the first two text cells of every `<tr>` as (name, location) pairs. |
| [`_SearchText`](#_searchtext) | Collect the plain text of a page with `<script>`/`<style>` blocks dropped. |

### `fetch`

Download a URL into the local cache and return its path; local paths pass through.

```python
fetch(source: str) -> pathlib.Path
```

**How it works**

1. Non-http(s) sources are returned as-is as a Path (never modified).
2. The cache directory is created; the destination name is the first 12 hex chars of the URL's sha1 plus the URL's file suffix (defaults to ".pdf").
3. A previously stored ETag (from `meta.json`) is sent as If-None-Match when the cached file exists.
4. On success the body is written and the new ETag + today's date are recorded in `meta.json`.
5. An HTTP 304 (unchanged) is swallowed: the cached copy is kept and returned.

**Parameters**

- `source` — URL to download, or a local path.

**Returns:** `Path` — The cached file path (the local path itself when source is not a URL).

**Side effects:** Writes the downloaded body into the cache directory; updates `meta.json`.

**Raises:** urllib.error.HTTPError for non-304 status codes; urllib.error.URLError or OSError for network/IO failures.

**Example**

```pycon
>>> fetch("https://www.pbs.gov.pk/wp-content/uploads/census_tables/tables/table_1_punjab_districts.pdf")
WindowsPath('.../pk_admin_data/cache/a1b2c3d4e5f6.pdf')
>>> fetch("C:/data/table.pdf")
WindowsPath('C:/data/table.pdf')
```

**Related:** Called by `fetch_first()`; calls `_meta()`.

---

### `fetch_first`

Try each candidate source in order and return the first that downloads.

```python
fetch_first(candidates) -> pathlib.Path
```

**How it works**

1. For each candidate, call `fetch()`; catch OSError and URLError (HTTPError is a URLError subclass) and remember the error text.
2. Return the first successful download's path.
3. If every candidate failed, raise one RuntimeError listing all of them.

**Parameters**

- `candidates` — Iterable of URLs or local paths, tried left to right.

**Returns:** `Path` — The first successfully fetched file.

**Side effects:** None directly (`fetch()` writes the cache).

**Raises:** RuntimeError when every candidate fails (the message lists each source and its error).

**Example**

```pycon
>>> fetch_first(["https://mirror1.example/x.pdf", "https://mirror2.example/x.pdf"])
WindowsPath('.../pk_admin_data/cache/ab12cd34ef56.pdf')
```

**Related:** Called by `update_admin_table()`.

---

### `pdf_lines`

Yield the text lines of a PDF file, page by page.

```python
pdf_lines(path)
```

**How it works**

1. Import pdfplumber on first call (installed automatically, but needed only by the update pipelines, so the import is deferred).
2. For each page, `extract_text()` returns the page text ('' for image-only pages), which is split into lines and yielded one at a time.

**Parameters**

- `path` — Path to a PDF file (a cached download or a local file).

**Returns:** `generator of str` — One line of extracted text per yield, in page order.

**Side effects:** None (reads the given file only).

**Raises:** pdfplumber's own errors for unreadable files (propagated; the caller decides how to handle them).

**Example**

```pycon
>>> list(pdf_lines(Path("table_1_punjab_districts.pdf")))[:2]
['TABLE 1', 'ATTOCK DISTRICT']
```

**Related:** Called by `update_admin_table()` via `_parse_admin_lines()`.

---

### `parse_pbs_lines`

Walk PBS Census Table 1 text and emit (province, district, tehsil) rows.

```python
parse_pbs_lines(lines, province: str) -> list
```

**How it works**

1. Split each line into a label and its numeric columns (`_split`). Lines without numbers are held back as a pending continuation IF they are letters-only (`_LETTERS_ONLY`) and not all header words (`_HEADER_WORDS`) — long names wrap onto two lines with the number columns on the second.
2. A numbers-only line (empty label) clears the pending fragment.
3. A pending fragment is prepended to the next line's label.
4. `DISTRICT_RE` matches labels ending in "district"/"dist." and starts a district row (tehsil=''); `TEHSIL_RE` matches labels ending in "tehsil"/"tahsil"/"taluka"/"taluk"/"sub-division"/"town" and attaches the label to the current district. Lines before any district is known are skipped.

**Parameters**

- `lines` — Iterable of table text lines.
- `province` — Province name stamped on every row.

**Returns:** `list of dict` — `[{'province', 'district', 'tehsil'}, ...]` in reading order; district rows carry tehsil == ''.

**Side effects:** None.

**Raises:** Never raises (unrecognised lines are skipped).

**Example**

```pycon
>>> parse_pbs_lines(["ATTOCK DISTRICT 1 2", "HASSAN ABDAL TEHSIL 3 4"], "Punjab")
[{'province': 'Punjab', 'district': 'Attock', 'tehsil': ''},
 {'province': 'Punjab', 'district': 'Attock', 'tehsil': 'Hassan Abdal'}]
```

**Related:** Called by `_parse_admin_lines()` for Punjab, Sindh, Khyber Pakhtunkhwa, Balochistan and Islamabad Capital Territory.

---

### `_parse_ajk_yearbook`

Parse the AJK Statistical Year Book tables 7.10 (districts) and 7.11 (tehsils).

```python
_parse_ajk_yearbook(lines, province: str) -> list
```

**How it works**

1. Join the lines; regex the block between "Table 7.10" and "Table 7.11": every label+numeric line becomes a district name (deduped, division prefixes stripped, `_SKIP_LABELS` filtered). No 7.10 block → RuntimeError.
2. Regex the 7.11 block after the "Sub Division (Tehsil)" header up to "Source:".
3. Walk the 7.11 lines in reading order:
   - **a.** A label line WITHOUT numbers that repeats a district name re-attaches the previous tehsil row to that district (the yearbook prints some district labels AFTER their first tehsil row — the Poonch/Sudhnoti late-row quirks).
   - **b.** A numbered line naming a district starts a new district block (also handles district+tehsil merged on one line, "Bagh Dhirkot").
   - **c.** Any other numbered label becomes a tehsil row.
4. Attach each tehsil to the NEAREST district row above or below it (a re-attach marker wins); emit one row per district name (tehsil='') plus one per tehsil.

**Parameters**

- `lines` — Extracted text lines of the yearbook PDF.
- `province` — Always "Azad Kashmir" here.

**Returns:** `list of dict` — `[{'province','district','tehsil'}, ...]`.

**Side effects:** None.

**Raises:** RuntimeError when the 7.10 district block or the 7.11 tehsil table cannot be found (layout changed).

**Example**

```pycon
>>> rows = _parse_ajk_yearbook(lines, "Azad Kashmir")
>>> ("Azad Kashmir", "Poonch", "Hajira") in {(r["province"], r["district"], r["tehsil"]) for r in rows}
True
```

**Related:** Called by `_parse_admin_lines()` for "Azad Kashmir".

---

### `_parse_gb_glance`

Parse the GB "at a Glance" district table (district-level rows only).

```python
_parse_gb_glance(lines, province: str) -> list
```

**How it works**

1. Skip lines until the table header ("District Wise Population..." /"Population and Area of GB") appears.
2. Stop at a "Source:" line or a "GB " prefix (the footer).
3. Each remaining label+numeric line becomes a district row; labels in `_SKIP_LABELS` or shorter than 4 chars are dropped. The official document has tehsil COUNTS, not names, so no tehsil rows are produced.
4. The row count must be 5..15 (GB has 10 districts) or RuntimeError — a wrong count means the layout drifted.

**Parameters**

- `lines` — Extracted text lines of the GB P&D document.
- `province` — Always "Gilgit-Baltistan" here.

**Returns:** `list of dict` — `[{'province','district','tehsil':''}, ...]`.

**Side effects:** None.

**Raises:** RuntimeError when fewer than 5 or more than 15 district rows parse.

**Example**

```pycon
>>> rows = _parse_gb_glance(lines, "Gilgit-Baltistan")
>>> rows[0]
{'province': 'Gilgit-Baltistan', 'district': 'Gilgit', 'tehsil': ''}
```

**Related:** Called by `_parse_admin_lines()` for "Gilgit-Baltistan".

---

### `_parse_admin_lines`

Route each province's source document to its parser.

```python
_parse_admin_lines(lines, province: str) -> list
```

**How it works**

1. "Azad Kashmir" → `_parse_ajk_yearbook` and "Gilgit-Baltistan" → `_parse_gb_glance` (PBS publishes no tehsil tables for the two territories).
2. Every other province → `parse_pbs_lines`.

**Parameters**

- `lines` — Extracted text lines of the province's source PDF.
- `province` — Province name (decides the parser).

**Returns:** `list of dict` — The chosen parser's output, `[{'province','district','tehsil'}, ...]`.

**Side effects:** None.

**Raises:** The chosen parser's errors (RuntimeError on layout drift).

**Example**

```pycon
>>> _parse_admin_lines(["ATTOCK DISTRICT 1 2"], "Punjab")[0]
{'province': 'Punjab', 'district': 'Attock', 'tehsil': ''}
```

**Related:** Called by `update_admin_table()`.

---

### `_parse_society_rows`

Extract (name, location) society rows from an HTML table page (PHATA style).

```python
_parse_society_rows(html_text: str) -> list
```

**How it works**

1. Feed the HTML to `_TableParser`.
2. Keep rows with at least 5 cells: cell index 1 is the name (the generic suffix regex `_GENERIC_SUFFIX` strips a trailing "housing scheme/society/homes/town/city") and cell index 2 is the location.
3. Require: name length >= 4 AND a space inside it AND not "name of", plus at least two letters in the location. The multi-word rule stops a single word like "akbar" from false-matching road names later.

**Parameters**

- `html_text` — The full HTML of an authority page.

**Returns:** `list of dict` — `[{"name": ..., "location": ...}]`, lowercase.

**Side effects:** None.

**Raises:** Never raises (malformed HTML yields fewer rows).

**Example**

```pycon
>>> _parse_society_rows(open("phata.html", encoding="utf-8").read())[0]
{'name': 'al noor orchard housing scheme', 'location': 'lahore'}
```

**Related:** Called by `_parse_society_source()` for `kind="table"`.

---

### `_parse_cda_rows`

Extract scheme names from CDA card links (`/housing-schemes/<slug>`).

```python
_parse_cda_rows(html_text: str, city: str) -> list
```

**How it works**

1. Find every href whose path is `housing-schemes/<slug>`.
2. Turn the slug's dashes into spaces; skip duplicates, names shorter than 8 chars, and names without 3 consecutive letters.
3. Every row gets the authority's city as location.

**Parameters**

- `html_text` — Full HTML of the CDA page.
- `city` — The authority city stamped on each row ("islamabad").

**Returns:** `list of dict` — `[{"name": ..., "location": city}]`.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _parse_cda_rows(html, "islamabad")[0]
{'name': 'capital enclave', 'location': 'islamabad'}
```

**Why it works this way:** The slug is used because the visible anchor text is CSS-truncated and unreliable.

**Related:** Called by `_parse_society_source()` for `kind="cards"`.

---

### `_parse_pdf_rows`

Read (name, location) rows from a generic authority PDF table (LDA/RDA style).

```python
_parse_pdf_rows(path: pathlib.Path, city: str) -> list
```

**How it works**

1. Open with pdfplumber; for every page and every extracted table, keep rows with at least two non-empty cells.
2. Cell index 1 is the name: whitespace-collapsed, must be at least 8 chars and contain 3 consecutive letters.
3. Every row gets the authority's city as location (the whole list belongs to one authority).

**Parameters**

- `path` — Path to the PDF.
- `city` — The authority city stamped on each row.

**Returns:** `list of dict` — `[{"name": ..., "location": city}]`.

**Side effects:** None.

**Raises:** pdfplumber's own errors for unreadable PDFs (propagated to the caller).

**Example**

```pycon
>>> _parse_pdf_rows(Path("lda.pdf"), "lahore")[0]
{'name': 'eden valley housing scheme', 'location': 'lahore'}
```

**Related:** Called by `_parse_society_source()` for `kind="pdf"`.

---

### `_parse_society_source`

Parse one approved-scheme source file into (name, location) rows by its kind.

```python
_parse_society_source(path: pathlib.Path, kind: str, city: str | None) -> list
```

**How it works**

1. kind "table" → `_parse_society_rows` on the HTML file text.
2. kind "cards" → `_parse_cda_rows` with the default city.
3. kind "pdf" → `_parse_pdf_rows`.
4. Any other kind raises ValueError.

**Parameters**

- `path` — Downloaded source file.
- `kind` — "table", "cards" or "pdf".
- `city` — Default authority city for kinds that do not carry one; may be None.

**Returns:** `list of dict` — `[{"name", "location"}, ...]`.

**Side effects:** None.

**Raises:** ValueError for an unknown kind.

**Example**

```pycon
>>> _parse_society_source(Path("cda.html"), "cards", "islamabad")[0]
{'name': 'capital enclave', 'location': 'islamabad'}
```

**Related:** Called by `update_society_table()`.

---

### `_fetch_postcode_rows`

Fetch ALL result rows for one postcode API query, paging past the 200-row cap.

```python
_fetch_postcode_rows(query: str) -> list
```

**How it works**

1. Request `/postal-codes?<query>&limit=200&page=N`, retrying up to 4 times with 4s sleeps (the API times out under load).
2. Append the batch and stop when a page returns fewer than 200 rows (the last page).

**Parameters**

- `query` — URL-encoded query string, e.g. "`province=Punjab`" or "`city=Karachi`".

**Returns:** `list of dict` — All result rows for the query (each with `postal_code`, city, province, `area_name` fields from the API).

**Side effects:** HTTP GET requests to the postcode API; sleeps 4s per failed attempt.

**Raises:** The last attempt's exception (URLError/HTTPError/JSONDecodeError) after 4 failures.

**Example**

```pycon
>>> len(_fetch_postcode_rows("city=Karachi")) >= 20
True
```

**Related:** Called by `update_postcodes()`.

---

### `_wiki_landmarks`

Collect a city's landmark names from Wikipedia's category system.

```python
_wiki_landmarks(city: str) -> list
```

**How it works**

1. For six categories ("Landmarks in X", "Tourist attractions in X", "Shrines in X", "Forts in X", "Monuments and memorials in X", "Statues in X") query the MediaWiki API (categorymembers + redirects, 500 each).
2. A failed request just skips the category (a missing category is normal).
3. Each page title plus its redirect titles is cleaned: namespaced titles (containing ":"), "(City)" parentheticals, a trailing ", City", and "List of ..." / "... in City" titles are dropped (list pages are not places).
4. Titles 4..40 chars with a letter are kept, title-cased.

**Parameters**

- `city` — City name (substituted into the category titles).

**Returns:** `list of dict` — `[{"name", "city"}, ...]` lowercase names, sorted.

**Side effects:** Up to 6 MediaWiki API requests per city.

**Raises:** Never raises (failed categories are skipped).

**Example**

```pycon
>>> [r["name"] for r in _wiki_landmarks("Lahore")][:2]
['badshahi mosque', 'lahore fort']
```

**Related:** Called by `update_neighborhoods_wiki()` and `update_world_landmarks()`.

---

### `_osm_neighborhoods`

Collect a city's neighborhood + landmark names from OpenStreetMap.

```python
_osm_neighborhoods(city: str) -> list
```

**How it works**

1. Nominatim search resolves the city to a bounding box (country restricted to Pakistan); no result → RuntimeError.
2. An Overpass query returns every node/way/relation tagged `place=neighbourhood|suburb|quarter` inside the bbox.
3. Names are kept when they have a letter, are 4..40 chars, and pass `_ok_osm_name`.
4. A SECOND, nodes-only Overpass query collects landmarks (`historic=*`, `tourism=attraction|museum|...`, `leisure=park|garden`, shrines/darbars); landmark names must be multi-word so bare category words like "Park" don't enter the registry. This query is best-effort: any failure is swallowed (landmarks are a bonus, they never fail the city).

**Parameters**

- `city` — City name (also used in the Nominatim query "`<city>`, Pakistan").

**Returns:** `list of dict` — `[{"name", "city"}, ...]` lowercase names, sorted.

**Side effects:** HTTP requests to Nominatim and the Overpass mirrors.

**Raises:** RuntimeError when Nominatim finds no city; `_overpass` errors propagate for the neighborhood query (landmark failures are caught).

**Example**

```pycon
>>> [r["name"] for r in _osm_neighborhoods("Karachi")][:2]
['azizabad', 'bahadurabad']
```

**Why it works this way:** OpenStreetMap is free, keyless and bot-tolerant — unlike Google's front pages — so the neighborhood phase uses it as the primary source.

**Related:** Called by `update_neighborhoods_osm()`; calls `_overpass()` and `_ok_osm_name()`.

---

### `_overpass`

Run an Overpass query against the public mirrors with retries.

```python
_overpass(query: str) -> dict
```

**How it works**

1. Try overpass-api.de first, then overpass.kumi.systems (the fallback mirror).
2. Each endpoint gets 3 attempts with 8s sleeps between; a successful response is returned immediately.
3. When both endpoints are exhausted, re-raise the last captured error (or a generic RuntimeError if none was recorded).

**Parameters**

- `query` — Overpass QL query text (POSTed as form data).

**Returns:** `dict` — `Parsed` JSON response (`{"elements": [...]}` for successful queries).

**Side effects:** HTTP POSTs to the Overpass mirrors; sleeps 8s per failed attempt.

**Raises:** The last captured exception (URLError, HTTPError, JSONDecodeError...) when every attempt fails.

**Example**

```pycon
>>> _overpass('[out:json];node(1);out;')["elements"]
[]
```

**Related:** Called by `_osm_neighborhoods()`.

---

### `_parse_google_neighborhoods`

Pull neighborhood/town name candidates out of the Google results text.

```python
_parse_google_neighborhoods(html_text: str, city: str) -> list
```

**How it works**

1. Feed the HTML to `_SearchText` and join the chunks.
2. Anchor on Google's echo of the query: the first occurrence of "neighbourhoods in `<city>`" / "neighborhoods in `<city>`" / "areas of `<city>`" / "areas in `<city>`"; only the next 4000 chars are searched (no anchor → the whole text).
3. `_NEIGHBOR_RE` harvests title-cased proper-noun phrases (a capital-led word plus up to 3 more) from that segment — area names are proper nouns; lowercase boilerplate falls away.
4. Filter: length 5..35, not seen, not the city itself, not a `_GOOGLE_STOP` word (chrome/boilerplate vocabulary), not containing google/cookies/copyright/wikipedia/http, and either 2+ capitalised words or length >= 8.

**Parameters**

- `html_text` — The full Google results page HTML.
- `city` — The city being queried (echo anchor + self-exclusion).

**Returns:** `list of dict` — `[{"name", "city"}, ...]` lowercase.

**Side effects:** None.

**Raises:** Never raises (malformed HTML yields fewer rows).

**Example**

```pycon
>>> _parse_google_neighborhoods(html, "Lahore")[:2]
[{'name': 'gulberg', 'city': 'lahore'}, ...]
```

**Why it works this way:** Tolerant on purpose: Google's markup changes constantly, and the per-city plausibility guard in `update_neighborhoods` rejects junk parses.

**Related:** Called by `update_neighborhoods()`.

---

### `_google_query`

Build the Google Search URL for a city's neighbourhood query (dead pipeline).

```python
_google_query(city: str) -> str
```

**How it works**

1. URL-encode the city into a "neighbourhoods in `<city>` pakistan" Google Search query.

**Parameters**

- `city` — City name.

**Returns:** `str` — The full Google Search URL.

**Side effects:** None.

**Raises:** Never raises.

**Example**

```pycon
>>> _google_query("Lahore")
'https://www.google.com/search?q=neighbourhoods+in+Lahore+pakistan&hl=en&num=20'
```

**Related:** Called by `update_neighborhoods()`.

---

### `_meta`

Read the pipeline metadata file.

```python
_meta() -> dict
```

**How it works**

1. If `meta.json` exists next to the data tables, parse and return it.
2. Otherwise return `{}` (no file, no error — a fresh pipeline starts empty).

**Returns:** `dict` — The parsed metadata (download ETags, last-run dates, row counts), or `{}` when `meta.json` is missing.

**Side effects:** None.

**Raises:** json.JSONDecodeError when `meta.json` exists but is corrupt.

**Example**

```pycon
>>> _meta()   # after an update run
{'https://...table_1_punjab_districts.pdf': {'etag': '"abc"', 'fetched': '2026-10-03'}}
```

**Related:** Called by `fetch()` to read and record ETags.

---

### `_read_csv`

Read a CSV file as a list of dicts; a missing file reads as [].

```python
_read_csv(path: pathlib.Path) -> list
```

**How it works**

1. If the file does not exist, return [] immediately — absent registry data must never crash the checks.
2. Otherwise read with csv.DictReader (utf-8; the header row becomes the keys).

**Parameters**

- `path` — CSV file path.

**Returns:** `list of dict` — One dict per data row (keys = header names, values = strings), or [] when the file is missing.

**Side effects:** None.

**Raises:** Never raises for a missing file; UnicodeDecodeError on non-utf-8 content.

**Example**

```pycon
>>> _read_csv(TABLE_CSV)[0]
{'province': 'Azad Kashmir', 'district': 'Bagh', 'tehsil': ''}
```

**Why it works this way:** Lenient on missing files on purpose: the checks must never fail because a registry file is absent.

**Related:** Called by every `load_*` and `update_*` function.

---

### `_main`

CLI entry point: `update` (rebuild data).

```python
_main(argv=None)
```

**How it works**

1. argparse defines one subcommand, update [--force].
2. update runs, in order: `update_admin_table`, `update_society_table`, `update_neighborhoods_osm`, `update_neighborhoods_wiki`, `update_postcodes`, `update_neighborhoods` (Google, currently dead); each prints totals and per-province/city failures.

**Parameters**

- `argv` — Optional argument list; defaults to sys.argv.

**Returns:** None.

**Side effects:** The update subcommand downloads from the internet and rewrites the data CSVs.

**Raises:** argparse exits on bad arguments.

**Related:** Run as `python -m pk_address_checker ...` under `__main__`.

---

### `_TableParser`

Collect the first two text cells of every `<tr>` as (name, location) pairs.

```python
_TableParser()
```

**How it works**

1. `<tr>` starts a new row list; `<td>` inside a row starts a new cell buffer.
2. `handle_data` appends text into the open cell buffer (separate runs join).
3. `</td>` closes the cell (whitespace-collapsed); `</tr>` closes the row and appends it if non-empty.

**Side effects:** Mutates `self.rows` (the collected rows).

**Raises:** Never raises (malformed HTML simply yields fewer rows).

**Related:** Used by `_parse_society_rows()`; the content filters live there.

---

### `_SearchText`

Collect the plain text of a page with `<script>`/`<style>` blocks dropped.

```python
_SearchText()
```

**How it works**

1. `handle_starttag` increments a skip counter on `<script>`/`<style>`; `handle_endtag` decrements it (they can nest).
2. `handle_data` appends text chunks only while the counter is zero — script and style content is mostly JavaScript/CSS noise, not readable content.

**Side effects:** Mutates `self.chunks` (the collected text pieces).

**Raises:** Never raises (malformed HTML simply yields fewer chunks).

**Related:** Used by `_parse_google_neighborhoods()`.

---

## Nested helpers

Methods of the two HTML parser classes.

| Name | What it does |
|---|---|
| [`_TableParser.__init__`](#_tableparser__init__) | Initialise the rows list and the current row/cell state. |
| [`_TableParser.handle_starttag`](#_tableparserhandle_starttag) | Start a row list on `<tr>` and a cell buffer on `<td>` inside a row. |
| [`_TableParser.handle_data`](#_tableparserhandle_data) | Append text into the open cell buffer (ignored outside cells). |
| [`_TableParser.handle_endtag`](#_tableparserhandle_endtag) | Close a cell (join and collapse whitespace) on `</td>`; close a row on `</tr>`. |
| [`_SearchText.__init__`](#_searchtext__init__) | Initialise the text chunk list and the skip counter. |
| [`_SearchText.handle_starttag`](#_searchtexthandle_starttag) | Increment the skip counter on `<script>`/`<style>`. |
| [`_SearchText.handle_endtag`](#_searchtexthandle_endtag) | Decrement the skip counter on `</script>`/`</style>` (guarded against 0). |
| [`_SearchText.handle_data`](#_searchtexthandle_data) | Append text into chunks while no script/style block is open. |

### Methods of `_TableParser`

#### `_TableParser.__init__`

Initialise the rows list and the current row/cell state.

```python
__init__(self)
```

---

#### `_TableParser.handle_starttag`

Start a row list on `<tr>` and a cell buffer on `<td>` inside a row.

```python
handle_starttag(self, tag, attrs)
```

---

#### `_TableParser.handle_data`

Append text into the open cell buffer (ignored outside cells).

```python
handle_data(self, data)
```

---

#### `_TableParser.handle_endtag`

Close a cell (join and collapse whitespace) on `</td>`; close a row on `</tr>`.

```python
handle_endtag(self, tag)
```

### Methods of `_SearchText`

#### `_SearchText.__init__`

Initialise the text chunk list and the skip counter.

```python
__init__(self)
```

---

#### `_SearchText.handle_starttag`

Increment the skip counter on `<script>`/`<style>`.

```python
handle_starttag(self, tag, attrs)
```

---

#### `_SearchText.handle_endtag`

Decrement the skip counter on `</script>`/`</style>` (guarded against 0).

```python
handle_endtag(self, tag)
```

---

#### `_SearchText.handle_data`

Append text into chunks while no script/style block is open.

```python
handle_data(self, data)
```
