Metadata-Version: 2.4
Name: geobatch
Version: 0.1.1
Summary: Polite, resumable batch geocoding and reverse geocoding of spreadsheets over Nominatim, Photon, Pelias and API-key providers, with caching, rate limiting, fallback and quality flags.
Author-email: Yusuf Eminoğlu <yusufeminoglu@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/YusufEminoglu/geobatch_sdk
Project-URL: Repository, https://github.com/YusufEminoglu/geobatch_sdk
Project-URL: Issues, https://github.com/YusufEminoglu/geobatch_sdk/issues
Project-URL: Changelog, https://github.com/YusufEminoglu/geobatch_sdk/blob/main/CHANGELOG.md
Keywords: geocoding,reverse-geocoding,batch-geocoding,nominatim,photon,pelias,openstreetmap,address,csv,geojson,geopackage,gis,rate-limiting,cache
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Environment :: Console
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Utilities
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: pandas
Requires-Dist: pandas>=1.3.0; extra == "pandas"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Dynamic: license-file

# geobatch

**Polite, resumable batch geocoding for spreadsheets.**

`geobatch` turns a CSV of thousands of addresses into coordinates (or a CSV of
coordinates into addresses) reliably and without upsetting the services you
depend on. It runs on the Python standard library alone and puts several
providers behind one interface:

- **Nominatim** (public or self-hosted). The OpenStreetMap Foundation usage policy is enforced on the public instance.
- **Photon** (komoot public demo or self-hosted).
- **Pelias** (self-hosted or hosted).
- **Custom HTTP template** for any JSON geocoding API.
- Thin adapters for API-key services: **OpenCage**, **Geoapify**, **Mapbox** and **Google**. Keys are read only from environment variables and are never logged.

Every lookup goes through a persistent SQLite cache, per-provider token-bucket
rate limiting, retries that honour `Retry-After`, a fallback chain and
quality checks. A batch that stops halfway can be resumed: run the same
command again.

```bash
pip install geobatch            # stdlib only
pip install "geobatch[pandas]"  # + DataFrame helpers
```

Requires Python 3.9 or newer.

---

## Quick start (command line)

```bash
geobatch geocode addresses.csv --address-col address \
    --provider nominatim \
    --user-agent "MyProject/1.0 (https://my-project.org; gis-team@my-project.org)" \
    -o geocoded.csv
```

Illustrative output (abridged):

```
geobatch: Nominatim usage policy applies: https://operations.osmfoundation.org/policies/nominatim/ (1 req/s, results cached, at most 5000 new lookups per run)
geocoding [##############----------------] 1412/3000  47.1% cached=388 failed=3 elapsed=17:04 eta=19:10
...
geobatch forward summary
  rows            3000
  ok              2941
  not found       52
  errors          3
  skipped (empty) 4
  unique lookups  2710
  cache hits      388
  HTTP requests   2325 (retries 3)
  by provider     nominatim=2941
  by precision    locality=61, rooftop=2402, street=478
  quality flags   country_mismatch=4, duplicate_coordinates=18, region_centroid=61
```

The output keeps every input column and adds `geobatch_status`, `geobatch_lat`,
`geobatch_lon`, `geobatch_display_name`, `geobatch_precision`,
`geobatch_confidence`, `geobatch_importance`, `geobatch_country_code`,
`geobatch_provider`, `geobatch_flags`, `geobatch_from_cache` and
`geobatch_error`.

More examples:

```bash
# Structured columns, Germany only, fallback to Photon, GeoJSON output
geobatch geocode customers.csv --street-col street --postcode-col zip --city-col city \
    --country-codes de --provider nominatim,photon \
    --user-agent "MyProject/1.0 (https://my-project.org)" -o customers.geojson

# Self-hosted Nominatim: no public-policy limits, so raise the rate and parallelism
geobatch geocode big.csv --address-col addr --url nominatim=http://localhost:8080 \
    --rate 50 --concurrency 8 -o big.gpkg

# Reverse geocoding
geobatch reverse points.csv --lat-col lat --lon-col lon \
    --user-agent "MyProject/1.0 (https://my-project.org)" -o points_addresses.csv

# A commercial provider (key from the environment, never from the command line)
export OPENCAGE_API_KEY=...
geobatch geocode addresses.csv --address-col address --provider opencage -o out.csv

# Cache maintenance
geobatch cache stats
geobatch cache clear --provider photon
geobatch cache clear --expired

# Provider terms and key variables, quality flag meanings
geobatch providers
geobatch flags
```

Input files: UTF-8 CSV, including the BOM that Excel writes. The delimiter
(`,` `;` tab `|`) is detected automatically, so semicolon "CSV" exports from
European Excel work as they are. `--excel` writes the output with a BOM so
Excel shows accented characters correctly. Output format follows the file
extension: `.csv`, `.tsv`, `.geojson` or `.gpkg`. GeoPackage is written with
the standard-library `sqlite3` as an OGC GeoPackage 1.3 point layer in EPSG:4326,
so GDAL is not needed. QGIS and GDAL read it directly.

Exit codes: `0` done, `2` usage or configuration error (including policy
refusals), `3` aborted after too many consecutive failures, `130` interrupted
(a partial output is still written).

---

## Responsible use

Free geocoders are run by volunteers and small teams on donated hardware. A
careless batch job can get your IP blocked, and it also costs everyone else
capacity. geobatch builds the rules in rather than leaving them to the
documentation.

| Provider | Terms / policy | What geobatch enforces |
|---|---|---|
| Nominatim (public `nominatim.openstreetmap.org`) | [OSMF Nominatim usage policy](https://operations.osmfoundation.org/policies/nominatim/), [OSM copyright & ODbL](https://www.openstreetmap.org/copyright) | Refuses to run without an identifying `user_agent=` / `--user-agent`. Stock library agents, geobatch's own default agent and placeholders such as `example.org` are rejected, and geobatch never fills in an e-mail address for you. Max **1 request/second**, **single thread**, cache required, at most **5000 uncached lookups per run** ("smaller one-time bulk tasks" only). The policy link is printed on every run, and the policy cannot be switched off for this host. |
| Photon (public `photon.komoot.io`) | [photon.komoot.io](https://photon.komoot.io/): fair use, heavy use is throttled; [source](https://github.com/komoot/photon) | 1 request/second, single thread, cache required (a conservative choice by geobatch). |
| Pelias | [documentation](https://github.com/pelias/documentation); hosted services such as [geocode.earth](https://geocode.earth) have their own terms | Optional `PELIAS_API_KEY`, redacted from logs. |
| OpenCage | [terms](https://opencagedata.com/terms) | Key from `OPENCAGE_API_KEY`; default 1 req/s (free-trial limit). |
| Geoapify | [terms](https://www.geoapify.com/terms-and-conditions/) | Key from `GEOAPIFY_API_KEY`; default 5 req/s. Attribution is required by Geoapify. |
| Mapbox | [terms of service](https://www.mapbox.com/legal/tos) | Token from `MAPBOX_ACCESS_TOKEN`. Temporary geocoding results are **not cached**; pass `permanent=True` / `--mapbox-permanent` only if your account includes permanent geocoding. |
| Google Geocoding | [Maps Platform terms](https://cloud.google.com/maps-platform/terms), [geocoding policies](https://developers.google.com/maps/documentation/geocoding/policies) | Key from `GOOGLE_MAPS_API_KEY`. Cache entries **expire after 30 days**. Google requires results to be used with a Google map. |

More rules that apply to every provider:

- **Keys** are read from environment variables only and never accepted as
  arguments. They are not part of cache keys, and they are replaced by `***` in
  every error message.
- **Retries** use exponential backoff with jitter, and a server's
  `Retry-After` always wins. A 429 pauses the provider's token bucket for all
  threads, not just the thread that received it. If a server asks you to wait
  more than `max_retry_after` (300 s by default; usually a used-up daily
  quota), geobatch gives up on that lookup instead of hammering or blocking
  for hours.
- **Circuit breaker:** after 25 consecutive failed lookups the batch stops,
  because the service is down or blocking you.
- **De-duplication:** identical addresses (after normalisation) are looked up
  once.
- **Attribution:** OpenStreetMap-based results (Nominatim, Photon, most Pelias
  installations) are © OpenStreetMap contributors under the ODbL. Credit them
  wherever you publish the data.

For large or recurring jobs, run your own Nominatim, Photon or Pelias, or use a
commercial provider. Self-hosted instances have no built-in limits in
geobatch. Set `--rate` and `--concurrency` to what your server can handle.

---

## Python API

```python
from geobatch import Geocoder, Nominatim, Photon, Query, SearchOptions, QualityConfig

ua = "MyProject/1.0 (https://my-project.org)"
geocoder = Geocoder(
    [Nominatim(user_agent=ua), Photon(user_agent=ua)],   # fallback chain
    cache="geocache.sqlite",                             # default: per-user cache dir
    options=SearchOptions(country_codes=("de", "at"), language="en"),
    quality=QualityConfig(min_confidence=0.5, expected_countries=("de", "at")),
)

result = geocoder.geocode_many(
    [
        "Pariser Platz 1, 10117 Berlin",
        Query(street="Stephansplatz 3", postcode="1010", city="Wien", country="Austria"),
        {"city": "Hamburg"},
    ],
    progress=True,
)
for rec in result:
    if rec.ok:
        r = rec.result
        print(r.lat, r.lon, r.display_name, r.precision, r.confidence, rec.flags)
    else:
        print(rec.status, rec.error)
print(result.summary.format())

# one-offs
rec = geocoder.geocode("Brandenburger Tor, Berlin")
rec = geocoder.reverse(52.5163, 13.3777)
```

`GeocodeResult` fields: `provider`, `lat`, `lon`, `display_name`,
`components` (normalised keys such as `house_number`, `road`, `postcode`,
`city`, `state`, `country`, `country_code`), `confidence` (0..1), `importance`,
`bbox` `(min_lon, min_lat, max_lon, max_lat)`, `precision` (`rooftop`,
`interpolated`, `street`, `postcode`, `locality`, `region`, `country`,
`unknown`), `result_type` and `raw` (the provider's original JSON). If a
provider gives no confidence score (Nominatim, Photon), `confidence` is
derived from `precision`.

Reading and writing files:

```python
from geobatch import read_table, build_queries, write_output

columns, rows = read_table("addresses.csv")
queries = build_queries(columns, rows, street_col="street", city_col="city", postcode_col="zip")
result = geocoder.geocode_many(queries, rows=rows)
write_output(result.records, "addresses.gpkg")
```

With pandas (`pip install "geobatch[pandas]"`):

```python
from geobatch.pandas_ext import geocode_dataframe, reverse_dataframe

df = geocode_dataframe(df, geocoder, address_col="address", expected_country_col="iso2")
df.attrs["geobatch_summary"]
```

### Custom HTTP template provider

Any JSON API can be described by URL templates and dotted JSON paths:

```python
from geobatch import TemplateProvider

inhouse = TemplateProvider(
    "https://geo.internal/find?address={query}&token={env:GEO_TOKEN}",
    reverse_url="https://geo.internal/reverse?lat={lat}&lon={lon}&token={env:GEO_TOKEN}",
    results_path="data.matches", lat_field="y", lon_field="x",
    display_field="label", confidence_field="score", confidence_scale=0.01,
    name="inhouse", rate=10, concurrency=4,
)
```

The available placeholders are `{query}`, `{street}`, `{city}`, `{county}`,
`{state}`, `{postcode}`, `{country}`, `{country_codes}`, `{language}`,
`{limit}`, `{lat}`, `{lon}` and `{env:NAME}`. A parameter that uses `{env:...}`
is redacted from logs and kept out of the cache key. GeoJSON
FeatureCollections are recognised automatically. On the command line, use
`--provider template --template-url ...`.

---

## How it works

**Normalisation.** Before lookup and before computing cache keys, addresses
are cleaned: Unicode NFKC (full-width digits, non-breaking spaces), control
characters, collapsed whitespace, surrounding quotes, doubled or trailing
separators, and spreadsheet null tokens (`NaN`, `N/A`, `-`) become empty.
Wording is never rewritten. Cache keys are case-insensitive.

**Cache and resume.** The SQLite cache (WAL mode, thread-safe) is keyed by
provider and endpoint, query kind, normalised query and search options. It
stores found results and "not found" answers, and each one is committed as
soon as it arrives. Network errors are never cached. If a batch is
interrupted (Ctrl+C, crash, laptop lid), run the same command again: finished
lookups come from the cache and only the rest goes to the network. In Python,
`BatchInterrupted.result` holds the partial result. The default location is
the per-user cache directory (`%LOCALAPPDATA%\geobatch`, `~/Library/Caches/geobatch`
or `~/.cache/geobatch`). Override it with `--cache` or `GEOBATCH_CACHE`.

**Rate limiting and concurrency.** Each provider has a token bucket
(`rate` requests per second, no initial burst) and a concurrency limit, shared
by all worker threads.

**Fallback.** Providers are tried in order. The next one is asked when a
provider fails, and also when it finds nothing (turn that off with
`fallback_on_not_found=False` / `--no-fallback-on-not-found`).

**Biasing.** `country_codes`, `bbox` and `bounded` are translated into each
provider's own parameters, for example Nominatim
`countrycodes`/`viewbox`/`bounded`, Pelias `boundary.*`/`focus.point`,
Geoapify `filter`/`bias` and Google `components`/`bounds`. Structured queries
use native structured endpoints where they exist (Nominatim, Pelias, Geoapify,
Mapbox) and are joined into one line elsewhere.

### Quality flags

| Flag | Meaning |
|---|---|
| `low_confidence` | confidence below `min_confidence` (default 0.5) |
| `country_mismatch` | result is outside the expected countries (`--country-codes` or a per-row `--expected-country-col`) |
| `outside_bbox` | result is outside the expected bounding box (`--bbox --bounded`) |
| `duplicate_coordinates` | the same point was returned for *different* addresses, a typical sign of a fallback to a street or town centroid |
| `region_centroid` | a postcode, city or region centroid was returned for a query that had street-level detail, or a region or country centroid was returned for any query |
| `fallback_provider` | answered by a provider other than the first in the chain |

---

## Development

```bash
pip install -e ".[dev,pandas]"
pytest --cov=geobatch
ruff check . && ruff format --check src tests && mypy src
```

The test suite never touches the real network. It runs a local stdlib HTTP
server (`tests/mockserver.py`) that imitates every provider's JSON, including
429/503 with `Retry-After`, malformed JSON, timeouts, rate-limit timing,
cache hits, resume and fallback.

## License

MIT © 2026 Yusuf Eminoğlu. See [LICENSE](LICENSE). To cite geobatch, see
[CITATION.cff](CITATION.cff).
