Metadata-Version: 2.5
Name: gla-data
Version: 0.3.0
Summary: Shared tools for loading, transforming, and visualising the GLA datasets used in City Data analysis.
Author-email: Conor Dempsey <conor.dempsey@london.gov.uk>
License: MIT
Requires-Python: >=3.12
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: boto3>=1.35
Requires-Dist: botocore>=1.35
Requires-Dist: filelock>=3.12
Requires-Dist: geopandas>=1.0
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.2
Requires-Dist: pandera>=0.32.1
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: pydantic-settings
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv
Requires-Dist: requests>=2.32
Requires-Dist: rich>=13.0
Requires-Dist: shapely>=2.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typer>=0.26.8
Description-Content-Type: text/markdown

# gla-data

Shared tools for loading, transforming, and visualising the GLA datasets used in City Data analysis.

## Installation

### CLI

To install `gla-data` globally so the `gla-data` CLI is available anywhere in your terminal:

```bash
uv tool install gla-data
```

This requires [uv](https://docs.astral.sh/uv/) to be installed.

### pip

```bash
pip install gla-data
```

### pixi

```bash
pixi add gla-data
```

Pixi pulls geospatial dependencies (geopandas, pyproj, etc.) from conda-forge, which provides reliable pre-built binaries across all platforms and Python versions.

## Usage

### Python API

```python
import gla_data

# Load 2021 LSOA boundaries for London
lsoa_gdf = gla_data.load_boundaries(geography='lsoa', year=2021)

# Load census dwelling counts
dwellings_2021 = gla_data.load_census_dwellings(year=2021)
dwellings_2011 = gla_data.load_census_dwellings(year=2011)

# Crosswalk 2011 dwelling counts onto 2021 LSOA boundaries
dwellings_2011_xw = gla_data.crosswalk(
    dwellings_2011,
    from_year=2011,
    to_year=2021,
    value_cols=['dwellings_total'],
)

# Load an OA → LSOA geography lookup
lookup = gla_data.load_geography_lookup(year=2021, smallest_geography='oa')

# Aggregate OA-level data up to LSOA level
lsoa_totals = gla_data.aggregate(
    df,
    from_geography='oa',
    to_geography='lsoa',
    value_cols=['count'],
    year=2021,
)

# OS MasterMap buildings, with heights, in one 1km grid square plus a 100m
# margin (GLA only; needs the database settings below)
buildings = gla_data.load_mastermap_topography(
    bbox=(530000, 180000, 531000, 181000),
    buffer_m=100,
    descriptive_group='Building',
)

# GLA tree canopy in the same square plus a 100m margin. The first call
# downloads the data (about 270MB) and builds a local file from it, which
# takes several minutes; later calls read that file.
canopy = gla_data.load_tree_canopy(
    bbox=(530000, 180000, 531000, 181000),
    buffer_m=100,
)
```

### CLI

The package installs a `gla-data` command. Run `catalogue` to list all available datasets:

```bash
gla-data catalogue
```

This prints a table of every dataset, the function used to load it, a description, and its access tier (`public`, `registered`, `licensed` or `internal`; see Configuration for the last three).

## Configuration

Public datasets need no setup. Datasets held inside the GLA (the GIS team's Postgres database and S3 buckets) need settings that are not included in this package: connection details, table names and bucket names. GLA staff can get the values from the City Intelligence Data Science team.

### Where to put settings

Every setting is named `GLA_DATA_<NAME>` and can be set in any of these places. Higher in the list wins:

1. An environment variable, e.g. `GLA_DATA_BLUESKY_IMAGERY_BUCKET`.
2. A `.env` file in the folder you run your code from (a project's own settings).
3. Your user-level settings file, `~/.config/gla-data/.env` (on Windows, `C:\Users\<you>\.config\gla-data\.env`). Use this for settings shared by all your projects, such as database connections and bucket names. Set `GLA_DATA_USER_ENV_FILE` to use a different path.

A settings file holds one setting per line:

```
GLA_DATA_BLUESKY_IMAGERY_BUCKET=<bucket name>
```

A misspelt `GLA_DATA_` setting in a settings file raises an error rather than being ignored. Other keys in a project's `.env` are left alone.

### Settings for GLA-internal datasets

| Setting | Needed for |
|---|---|
| `GLA_DATA_GIS_DB_HOST`, `_PORT`, `_NAME`, `_USER`, `_PASSWORD` | Anything from the GIS team's Postgres database (`source="postgres"`, `load_gis_layer`, `load_front_gardens`, `load_mastermap_topography`, `gla-data db ...`). Use a read-only login. |
| `GLA_DATA_GIS_DB_SCHEMA` | Optional: the AddressBase schema, if not the default search path. |
| `GLA_DATA_ADDRESSBASE_BLPU_TABLE`, `_CLASSIFICATION_TABLE` | `load_addressbase(source="postgres")`, if the tables are not named `blpu` / `classification`. |
| `GLA_DATA_GISAPDATA_<LAYER>_TABLE` | Each `gisapdata` layer used by `load_boundaries(source="postgres")`, `load_gis_layer` and `load_front_gardens`. |
| `GLA_DATA_GISAPDATA_MASTERMAP_TOPO_2024_TABLE` | `load_mastermap_topography` (2024 snapshot). |
| `GLA_DATA_BLUESKY_IMAGERY_BUCKET` | Bluesky aerial imagery (`build_imagery_tile_index`, `download_imagery_tiles`). |
| `GLA_DATA_BLUESKY_IMAGERY_DIR` | Optional: folder for the imagery tile index and downloads (default `data/bluesky_imagery`). Tiles are 256MB each. |
| `GLA_DATA_AWS_PROFILE`, `GLA_DATA_AWS_REGION` | Optional: pin the AWS profile or region used for S3. |
| `GLA_DATA_TREE_CANOPY_DIR` | Optional: folder for the tree canopy download and the GeoPackage built from it (default `data/tree_canopy`). Point every project at the same folder so the data is downloaded once. |

AWS credentials are never set here. gla-data uses the standard AWS sources: an AWS profile (`~/.aws`), `AWS_*` environment variables, or the machine's own role on GLA cloud machines. Keep database passwords in your user-level file, never in a project's repository.

## Development

```bash
git clone <repo-url>
cd gla-data
pixi install

pixi run test       # run tests
pixi run lint       # check code style
pixi run format     # auto-format
pixi run typecheck  # type checking
```
