Architecture

How the pieces fit together, for anyone modifying the system.

Overview

  Web portal (Cloudflare Pages)        Python package (PyPI)
  weather-data-gui.pages.dev           TeamOverbyeWeather
              |                                  |
              +----------------+-----------------+
                               |
                     FastAPI backend (Railway)
                weather-data-gui.up.railway.app
                               |
                     Google Drive (service account)

Both front ends are thin. All catalog logic, cropping, and Drive access live in the backend; the browser and the Python package only make HTTP requests.

Repository layout

frontend/                  static site — index.html, main.js, styles.css
backend/
  main.py                  FastAPI app, endpoints, concurrency gates
  catalog.py               Drive scanning, filename matching, 30-min cache
  download.py              file lookup, ZIP bundling, fetch_and_crop()
  pww_io.py                PWW read / crop / write
  regions.py               state bounding boxes and ISO zones
  status.py                pipeline health
package/TeamOverbyeWeather/
  client.py                WeatherClient.download() and discovery
  registry.py              (source, type) -> API key mapping
  localcrop.py             client-side crop fallback
  pww_io.py                copy of the backend module
  errors.py                exception types
  sources/                 back-compatible per-source wrappers
docs/                      this documentation

Three places that must agree

Adding or renaming a data source means touching all three:

Location

What it defines

backend/download.py — _SOURCE_LOOKUP

API key → catalog key and list key

package/.../registry.py — _API_KEYS, _TYPES

user-facing source/type → API key

frontend/main.js — TYPE_DEFS, getApiSourceKey()

the portal’s picker

They are kept deliberately parallel. If a source works in the portal but not the package, this mismatch is the first thing to check.

package/TeamOverbyeWeather/pww_io.py is a copy of backend/pww_io.py. Change one, copy to the other — a divergence here produces crops that differ between server and client.

The crop pipeline

fetch_and_crop() in backend/download.py is the core:

  1. Resolve (source, date_key) to a Drive file id

  2. Stream the file to /tmp — never fully into memory

  3. Detect the shape by content, not by source name

  4. Crop each part to the bounding box, then stitch along time

  5. Apply the time crop if requested

  6. Write PWW bytes and return them

Step 3 matters. zipfile.is_zipfile() decides whether to unzip, so a source that changes format does not need a code change. Four on-disk shapes are handled: a bare .pww, a zip with one .pww, a zip with four quarter .pww files, and a zip of daily zips.

Cropping each member before concatenating keeps peak memory at one cropped member instead of a whole month.

package/TeamOverbyeWeather/localcrop.py mirrors this for the client-side fallback, reading from disk rather than Drive.

Concurrency and load shedding

Heavy endpoints sit behind _Gate(concurrency, max_wait) in main.py. Each gate runs a few requests concurrently, queues the rest, and returns 503 with Retry-After once too many are waiting — shedding load rather than accepting work that would exhaust memory.

Gate

Covers

Environment variable

Default

_single_gate

single-file streams

SINGLE_CONCURRENCY

6

_region_gate

region crops

REGION_CONCURRENCY

2

_download_gate

multi-file ZIP builds

BUNDLE_CONCURRENCY

2

all

queue depth before 503

MAX_DOWNLOAD_QUEUE

50

All are tunable on Railway without a code change. Region crops are the heaviest path — lower REGION_CONCURRENCY first if memory is tight.

For streamed responses the slot is taken before the response is returned and released in the generator’s finally, so the limit holds for the whole transfer rather than just the headers.

Why cropping is server-side

Client-side cropping always requires transferring the whole file first. Measured:

Request

Transferred

NOAA cycle, uncropped

120 MB

Cropped to Texas

7.3 MB

Texas plus a 6-hour window

240 KB

HRRR 15-minute day, uncropped

534 MB

The server is the only place that can avoid moving the large number. The gates above exist so it can do that safely; the 413 fallback exists for the one case where it genuinely cannot.

Deployment

Frontend — Cloudflare Pages, auto-deploys from main. Output directory frontend, no build step.

Backend — Railway, auto-deploys from main, roughly 90 seconds. The service root directory must be backend, since the Dockerfile lives there. Required environment variables are GDRIVE_CREDENTIALS_JSON_CONTENT (the whole service-account JSON) and CORS_ORIGINS.

After deploying, verify:

curl <url>/api/health
curl <url>/api/debug/folders
curl <url>/api/catalog/refresh

Do not set GDRIVE_*_FOLDER_ID variables — several are stale, so folder ids are hard-coded in catalog.py.

Package — built from package/ and published to PyPI.

Documentation — this site, built by Read the Docs from docs/.