Metadata-Version: 2.4
Name: tilefetch
Version: 0.1.1
Summary: Download XYZ/TMS/WMTS map tiles for an area into MBTiles, PMTiles, z/x/y folders or one georeferenced GeoTIFF/PNG, politely and within provider terms.
Author-email: Yusuf Eminoğlu <yusufeminoglu@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/YusufEminoglu/tilefetch_sdk
Project-URL: Repository, https://github.com/YusufEminoglu/tilefetch_sdk
Project-URL: Issues, https://github.com/YusufEminoglu/tilefetch_sdk/issues
Project-URL: Changelog, https://github.com/YusufEminoglu/tilefetch_sdk/blob/main/CHANGELOG.md
Keywords: gis,tiles,xyz,tms,wmts,mbtiles,pmtiles,geotiff,web-mercator,basemap,imagery,offline-maps,remote-sensing,machine-learning,mosaic
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Environment :: Console
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Topic :: Scientific/Engineering :: Image Processing
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Pillow>=9.1.0
Provides-Extra: rasterio
Requires-Dist: rasterio>=1.3.0; extra == "rasterio"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: pmtiles>=3.4.0; extra == "dev"
Dynamic: license-file

<div align="center">

# tilefetch

[![PyPI version](https://img.shields.io/pypi/v/tilefetch.svg?color=10b981)](https://pypi.org/project/tilefetch/)
[![Python version support](https://img.shields.io/pypi/pyversions/tilefetch.svg?color=3b82f6)](https://pypi.org/project/tilefetch/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/YusufEminoglu/tilefetch_sdk/blob/main/LICENSE)
[![Code style: Ruff](https://img.shields.io/badge/code%20style-ruff-D7FF64.svg)](https://docs.astral.sh/ruff/)

**Download XYZ / TMS / WMTS map tiles for an area and get one georeferenced GeoTIFF/PNG,
an MBTiles file, a PMTiles v3 archive or a `z/x/y` folder - politely, resumably, and within
each provider's terms.**

[📦 PyPI](https://pypi.org/project/tilefetch/) • [💻 GitHub](https://github.com/YusufEminoglu/tilefetch_sdk) • [🐛 Issues](https://github.com/YusufEminoglu/tilefetch_sdk/issues)

</div>

`tilefetch` is for GIS and machine-learning work that needs a basemap or imagery for an
area: as **one image with correct georeferencing** (to drop into QGIS, rasterio or a
training pipeline), or as an **offline tile package**. It has a single required dependency
(Pillow) and writes GeoTIFF, MBTiles and PMTiles itself.

## ⚠️ Responsible use - read this first

Tile servers are run by people and organisations who pay for the bandwidth. Most free
community tile servers **forbid bulk or offline downloading**; for example the
[OSMF Tile Usage Policy](https://operations.osmfoundation.org/policies/tiles/) forbids it
for `tile.openstreetmap.org`. `tilefetch` is built to keep you inside the rules:

- **Known no-bulk hosts are refused.** URLs on `tile.openstreetmap.org`, `opentopomap.org`,
  Google, Bing (`virtualearth.net`) and Esri's `arcgisonline.com` raise `PolicyError`
  (see `tilefetch.RESTRICTED_HOSTS`). There is no override flag: run your own tile server, or
  use a service whose terms allow offline use.
- **Built-in providers have hard limits.** Each entry carries its terms URL, its required
  attribution, a maximum number of tiles per job, a request rate and a thread limit. You can
  lower these, never raise them.
- **Custom URLs get conservative defaults**: 10,000 tiles per job and 10 requests/s. Raise the
  cap (`max_tiles=` / `--max-tiles`) only for servers you operate or whose terms allow it.
- **Polite by default**: 2 threads, retries with exponential back-off that honour
  `Retry-After`, an on-disk cache with `ETag` revalidation, and resume, so nothing is
  downloaded twice.
- **Honest identification**: requests carry `User-Agent: tilefetch/<version> (+repo URL)`.
  Browser User-Agent strings are refused; set your own application name and contact instead
  (`user_agent="myproject/1.0 (me@example.org)"`).
- **Attribution travels with the data**: it is printed when a job starts and stored in
  MBTiles/PMTiles metadata, GeoTIFF `Copyright`/`ImageDescription` tags and PNG text chunks.
  Display it wherever you show the data.

You remain responsible for complying with the terms of every service you use.

## 📦 Installation

```bash
pip install tilefetch                 # Pillow only
pip install "tilefetch[rasterio]"     # optional: write GeoTIFFs through GDAL/rasterio
pip install "tilefetch[dev]"          # pytest, ruff, mypy, pmtiles (reference reader for tests)
```

Python 3.9+ on Windows, macOS and Linux.

## ⚡ Quickstart

```python
import tilefetch

bbox = (-105.30, 39.95, -105.20, 40.05)   # west, south, east, north (degrees)

# 1. How big is the job?
est = tilefetch.estimate("usgs-imagery", bbox, "12-15")
print(est.total_tiles, est.estimated_bytes, est.within_limit)

# 2. An offline package (MBTiles, PMTiles or a z/x/y folder, chosen by extension)
tilefetch.fetch("usgs-topo", bbox, "12-15", out="boulder.mbtiles")
tilefetch.fetch("usgs-topo", bbox, "12-15", out="boulder.pmtiles")

# 3. One georeferenced image cropped to exactly the bbox
res = tilefetch.mosaic("usgs-imagery", bbox, 15, out="boulder.tif")
print(res.image.size, res.bounds, res.crs)          # EPSG:3857 bounds of the GeoTIFF

# ... or in EPSG:4326, as PNG + world file (.pgw) + .prj
tilefetch.mosaic("usgs-imagery", bbox, 15, out="boulder_4326.png", crs="EPSG:4326")

# 4. Your own tile server (any XYZ template)
tilefetch.fetch(
    "https://tiles.example.org/{z}/{x}/{y}.png",
    bbox, 14, out="area.pmtiles",
    attribution="© My Org", user_agent="myproject/1.0 (me@example.org)",
)
```

The same from the shell:

```bash
tilefetch providers
tilefetch estimate -p usgs-imagery --bbox=-105.30,39.95,-105.20,40.05 -z 12-15
tilefetch get -p usgs-topo --bbox=-105.30,39.95,-105.20,40.05 -z 12-15 -o boulder.pmtiles
tilefetch get --url "https://tiles.example.org/{z}/{x}/{y}.png" \
    --bbox=-105.3,39.95,-105.2,40.05 --zoom 12-15 -o out.pmtiles --attribution "© My Org"
tilefetch mosaic -p usgs-imagery --bbox=-105.30,39.95,-105.20,40.05 -z 15 -o out.tif
tilefetch mosaic --input boulder.mbtiles --bbox=-105.25,39.98,-105.22,40.0 -z 15 -o crop.png --crs EPSG:4326
tilefetch info boulder.pmtiles
```

Use `--bbox=...` (with `=`) when the first coordinate is negative, so it is not read as an option.

## 🗺️ Built-in providers

Only services whose published terms allow scripted/offline use are included.

| Name | Data | Zoom | Limits per job | Terms |
|---|---|---|---|---|
| `usgs-topo` | USGS Topo base map (US & territories) | 0-16 | 50,000 tiles, 8 req/s, 4 threads | [USGS: public domain, no restrictions](https://www.usgs.gov/faqs/what-are-terms-uselicensing-map-services-and-data-national-map) |
| `usgs-imagery` | USGS orthoimagery (US & territories) | 0-16 | same | same |
| `usgs-shaded-relief` | USGS shaded relief (US & territories) | 0-16 | same | same |
| `nasa-bluemarble` | NASA Blue Marble Next Generation (global, 500 m) | 0-8 | 100,000 tiles, 16 req/s, 8 threads | [NASA GIBS bulk-download guidance](https://nasa-gibs.github.io/gibs-api-docs/map-library-usage/#bulk-downloading) |
| `nasa-bluemarble-relief` | Blue Marble with shaded relief and bathymetry | 0-8 | same | same |
| `nasa-modis-terra` | MODIS Terra daily true colour (`params={"time": "YYYY-MM-DD"}`) | 0-9 | same | same |

- USGS asks for the credit *"Map services and data available from U.S. Geological Survey,
  National Geospatial Program."*
- NASA GIBS defines a bulk download as more than 1,000,000 tiles in 24 hours and asks for
  advance coordination above that; the per-job cap keeps one job far below it. Credit:
  *"Imagery provided by services from NASA's Global Imagery Browse Services (GIBS), part of
  NASA's Earth Science Data and Information System (ESDIS)."*

Register your own entries with `tilefetch.register_provider(TileSource(...))`.

## 🧩 Sources

```python
from tilefetch import TileSource, wmts_source

# XYZ / TMS templates
TileSource.from_url("https://t.example.org/{z}/{x}/{y}.png")              # XYZ
TileSource.from_url("https://t.example.org/{z}/{x}/{y}.png", scheme="tms")  # TMS rows
TileSource.from_url("https://t.example.org/{z}/{x}/{-y}.png")             # TMS row placeholder
TileSource.from_url("https://{s}.t.example.org/{z}/{x}/{y}.png", subdomains=("a", "b"))
TileSource.from_url("https://t.example.org/q/{q}.jpeg")                   # Bing-style quadkey
TileSource.from_url("https://wms.example.org/?SERVICE=WMS&REQUEST=GetMap&LAYERS=x"
                    "&SRS=EPSG:3857&BBOX={bbox}&WIDTH=256&HEIGHT=256&FORMAT=image/png")
TileSource.from_url("https://t.example.org/{z}/{x}/{y}.png?key={key}", params={"key": "..."})

# WMTS: REST (ResourceURL) or KVP GetTile, from a GetCapabilities URL, file or XML string
src = wmts_source(
    "https://gibs.earthdata.nasa.gov/wmts/epsg3857/best/1.0.0/WMTSCapabilities.xml",
    "MODIS_Terra_CorrectedReflectance_TrueColor",
    dimensions={"Time": "2024-06-01"},
    attribution="NASA GIBS",
)
```

`wmts_source` parses the `TileMatrixSet`s, keeps only Web Mercator ones and maps each
`TileMatrix` to an XYZ zoom from its scale denominator (OGC 0.28 mm pixel), so matrix
identifiers such as `EPSG:3857:12`, 512-px tiles and regional matrix sets whose top-left
corner is offset from the world origin all work.

## 📤 Outputs

| Output | How | Notes |
|---|---|---|
| **MBTiles** | `out="x.mbtiles"` | MBTiles 1.3: `metadata` + `tiles` tables, rows in TMS order, `name`, `format`, `bounds`, `center`, `minzoom`, `maxzoom`, `attribution`, `description`, `type`, `version` |
| **PMTiles v3** | `out="x.pmtiles"` | 127-byte header, Hilbert tile IDs, varint directories, gzip internal compression, root directory in the first 16 KiB with leaf directories, de-duplicated tiles and run-lengths, clustered. Validated against the reference `pmtiles` reader |
| **Folder** | any other path | `z/x/y.png` (extension from the tile bytes) + `metadata.json` |
| **GeoTIFF** | `.tif` / `.tiff` | Pillow-only writer with ModelPixelScale, ModelTiepoint and GeoKeys for EPSG:3857 or EPSG:4326, deflate-compressed; or `engine="rasterio"` |
| **PNG / JPEG / WebP** | `.png` / `.jpg` / `.webp` | plus world file (`.pgw`/`.jgw`/`.wld`) and `.prj` (ESRI WKT) |

Mosaics (`tilefetch.mosaic` or an image `out=` in `fetch`) are built at one zoom level:

- `crop=True` (default) resamples to **exactly** the requested bbox (bilinear by default,
  `resampling="nearest"`/`"bicubic"` available);
- `crop="snap"` crops to the enclosing source pixels with no resampling;
- `crop=False` keeps whole tiles;
- `crs="EPSG:4326"` resamples to a lon/lat grid with square-degree pixels; the inverse mapping
  is exact per row (Mercator x is linear in longitude, each row's latitude is projected).

Missing tiles (HTTP 404/204) stay transparent. A mosaic can also be built offline from an
existing `.mbtiles`, `.pmtiles` or folder: `tilefetch.mosaic("area.mbtiles", bbox, 14, out=...)`.

## 🔁 Downloading behaviour

- `threads=2`, `rate=` requests/s (shared across threads), `retries=4`, `backoff=0.5` s
  (exponential with jitter), `timeout=30` s.
- Retries on timeouts, connection errors and HTTP 408/425/429/500/502/503/504, honouring
  `Retry-After`. 404/204/410 count as *missing*; 401/403 and other client errors fail at once.
  An HTML/XML page served with status 200 is treated as a failure, not stored as a tile.
- `cache_dir=` keeps an HTTP cache (`ETag`, `Last-Modified`); `revalidate=True` sends
  `If-None-Match` / `If-Modified-Since` and reuses the cached tile on `304 Not Modified`.
- **Resume**: re-running a job skips tiles already in the output. PMTiles jobs stage tiles in
  `<out>.part` (SQLite) and write the archive at the end, so an interrupted job continues
  where it stopped. `overwrite=True` starts again.

## 🧮 Tile math

```python
from tilefetch import (tiles_for_bbox, tiles_for_geometry, lonlat_to_tile, tile_bounds_mercator,
                       tile_to_quadkey, quadkey_to_tile, flip_y, zxy_to_tileid)

tiles_for_bbox((28.97, 41.00, 28.99, 41.02), "14-15")   # antimeridian boxes (west > east) work
tiles_for_geometry("area.geojson", 14)                   # exact tile/polygon test, holes included
lonlat_to_tile(13.3777, 52.5163, 12)                     # Tile(z=12, x=2200, y=1343)
tile_to_quadkey((3, 3, 5))                               # '213'
flip_y(3, 5)                                             # 2  (XYZ <-> TMS)
zxy_to_tileid(1, 1, 0)                                   # 4  (PMTiles Hilbert ID)
```

## 🧪 Development

```bash
git clone https://github.com/YusufEminoglu/tilefetch_sdk.git
cd tilefetch_sdk
pip install -e ".[dev,rasterio]"
python -m pytest --cov=tilefetch
python -m ruff check . && python -m ruff format --check src tests && python -m mypy src
```

The tests never touch the internet: they start an HTTP tile server on `127.0.0.1` that
serves generated tiles (and simulates 429/503/timeouts, ETags, WMTS REST/KVP and
capabilities). Mosaic tests check georeferencing pixel by pixel; GeoTIFFs are re-read with
rasterio when it is installed, and PMTiles archives with the reference `pmtiles` reader.

## 📄 License

MIT © Yusuf Eminoğlu. Map data and imagery downloaded with this tool remain under their
providers' licences and terms.
