Metadata-Version: 2.5
Name: gtfs-parquet
Version: 0.7.1
Summary: Parse GTFS feeds to/from Parquet via Polars — fast, compact, typed.
Project-URL: Homepage, https://github.com/GaspardMerten/gtfs-parquet
Project-URL: Repository, https://github.com/GaspardMerten/gtfs-parquet
Project-URL: Issues, https://github.com/GaspardMerten/gtfs-parquet/issues
Author-email: Gaspard Merten <gaspard.mp.work@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: gtfs,parquet,polars,transit,transport
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: GIS
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: polars>=1.0
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# gtfs-parquet

Parse [GTFS](https://gtfs.org/) feeds to/from [Parquet](https://parquet.apache.org/) via [Polars](https://pola.rs/) — fast, compact, typed.

## Features

- Parse GTFS from **zip**, **directory**, or **URL**
- Read **NeTEx** timetables (European profile) as the same GTFS tables
- Write to **Parquet** (zstd-compressed) or back to **GTFS**
- Strongly typed schemas with optimised dtypes (Float32 coords, Int16 sequences)
- Built on Polars — zero-copy reads, lazy evaluation ready
- Operations: calendar expansion, network analysis, route/stop/trip stats, timetable graphs, CSA connections

## Installation

```bash
pip install gtfs-parquet
```

## Quick start

```python
from gtfs_parquet import parse_gtfs, write_parquet, read_parquet, write_gtfs

# Parse a GTFS zip (local path or URL)
feed = parse_gtfs("gtfs.zip")

# Write to Parquet — directory, .zip, or .tar (auto-detected by extension)
write_parquet(feed, "output/")           # directory of .parquet files
write_parquet(feed, "output.zip")        # zip archive (no extra compression)
write_parquet(feed, "output.tar")        # tar archive (single file, no extra compression)

# Read back (same formats)
feed = read_parquet("output/")
feed = read_parquet("output.zip")
feed = read_parquet("output.tar")

# Convert back to GTFS zip
write_gtfs(feed, "roundtrip.zip")
```

Additional top-level helpers: `parse_gtfs_dir`, `parse_gtfs_zip`, `write_gtfs_dir`, and
`to_parquet_bytes(feed)`, which returns `{table_name: parquet_bytes}` without writing to disk
(handy for uploading to object storage).

## Large feeds

`parse_gtfs_zip` loads the whole feed in memory. For a national feed that takes several GB of RAM. When you only
need the Parquet files, `convert_gtfs_zip` streams the conversion instead:

```python
from gtfs_parquet import convert_gtfs_zip

paths = convert_gtfs_zip("gtfs.zip", "output/")   # {"stops": Path("output/stops.parquet"), ...}
```

Tables are converted one at a time, and large files are read in 16 MB blocks,
stored as temporary Parquet files in `out_dir` until the table is written.
Rows keep the order of the source file (`sort=True` sorts them). A file with an
unescaped quote (`12" Street` outside quotes) raises a `ValueError` instead of
being read whole.

| German national feed (298 MB zip, 40M stop times) | Peak RAM | Parquet |
|---------------------------------------------------|----------|---------|
| `parse_gtfs_zip` + `write_parquet`                  | 10.7 GB  |  320 MB |
| `convert_gtfs_zip`, all 20 threads                  |  1.6 GB  |  120 MB |
| `convert_gtfs_zip`, `POLARS_MAX_THREADS=4`           |  0.5 GB  |  120 MB |

Memory grows with the number of Polars threads, which defaults to the number of
CPU cores: set `POLARS_MAX_THREADS` before importing Polars to limit it.

## NeTEx

Some operators publish their timetable only in [NeTEx](https://netex-cen.eu/), the European XML format.
`parse_netex` reads it as GTFS tables, typed like a parsed GTFS feed; `convert_netex` writes them as Parquet,
the same files `convert_gtfs_zip` writes:

```python
from gtfs_parquet import convert_netex, parse_netex

feed = parse_netex("IT-IT-TRENITALIA_L1.xml.gz")   # .xml, .xml.gz, or a .zip of XML files
paths = convert_netex("IT-IT-TRENITALIA_L1.xml.gz", "output/")

# Several files read as one timetable, e.g. Entur's timetable and its stop places (NSR)
feed = parse_netex(["rb_vyg-aggregated-netex.zip", "Current_latest.zip"])
```

| NeTEx | GTFS |
|-------|------|
| `Operator` / `Authority` | agency |
| `StopPlace`, its `Quay`s (nested or apart) | stops: stations and their stops |
| `Line` (`TransportMode`) | routes (basic `route_type`); journeys of another mode (rail replacement buses) get a route `<line>:<mode>` |
| `ServiceJourney` (public code, `TrainNumber`, private code or `Name`) | trips (`trip_short_name`), cancelled ones left out |
| `TimetabledPassingTime` | stop_times, day offsets as times past 24:00; `RequestStop` as pickup/drop off type 3 |
| `DayType`, `DayTypeAssignment`, `OperatingPeriod` / `UicOperatingPeriod`, `DatedServiceJourney` | calendar_dates, one service per set of dates |
| `ServiceLink` geometry along the journey pattern | shapes |

Tested on the Italian (Trenitalia), Nordic (Entur: Vy, Flytoget) and French (SNCF, BreizhGo) profiles:
the Entur and SNCF timetables give the same trips, dates and stop times as the operators' own GTFS.
The time zone is the file's, else the most common one of its stop places (SNCF's says `CET`: pass
`timezone="Europe/Paris"`), else UTC. Nordic exports leave stop places to a separate file: without it, the
stops are the scheduled stop points, without coordinates.

The file is read in one streaming pass. Trenitalia's timetable (302 MB of XML, 16,675 journeys) converts in
about 10 seconds with 0.6 GB of RAM; SNCF's (625 MB, 58,813 journeys) in 15 seconds with 0.7 GB.

## Compression

Parquet output is **significantly smaller** than the original GTFS zip thanks to
zstd compression and optimised column types:

| Feed               | GTFS zip | Parquet | Sorted (`sort=True`) |
|--------------------|----------|---------|----------------------|
| MBTA               |  24.8 MB |  8.3 MB |   8.3 MB             |
| FlixBus (EU)       |  30.6 MB | 13.5 MB |  16.3 MB             |
| Germany (national) | 298.2 MB |  119 MB |   320 MB             |

Since 0.6.0, rows keep the order of the source files by default: feeds usually
list related rows together, and sorting by id often breaks that up.

## Feed object

`Feed` is a plain dataclass with one optional `polars.DataFrame` attribute per GTFS
file (e.g. `feed.stops`, `feed.routes`, `feed.stop_times`). Only files present
in the source feed are populated.

```python
feed.tables()    # dict of all non-None tables
feed.validate()  # check required files and columns
```

## Operations

All operations are standalone functions that take a `Feed` as their first argument.
Import from the `ops` submodules:

```python
from gtfs_parquet.ops.calendar import (
    get_dates, get_first_week, get_week, get_active_services,
    subset_dates, compute_trip_activity, compute_busiest_date,
)
from gtfs_parquet.ops.trips import get_trips, compute_trip_stats
from gtfs_parquet.ops.routes import get_routes, build_route_timetable, compute_route_stats
from gtfs_parquet.ops.stops import (
    get_stops, get_stop_times, get_start_and_end_times,
    build_stop_timetable, compute_stop_activity, compute_stop_stats,
)
from gtfs_parquet.ops.network import describe, compute_network_stats
from gtfs_parquet.ops.restrict import restrict_to_routes, restrict_to_dates, restrict_to_trips
from gtfs_parquet.ops.clean import clean, clean_ids, drop_zombies
from gtfs_parquet.ops.graph import (
    build_timetable_graph, get_service_day_counts, build_stop_lookup,
    compute_segment_frequencies, compute_connections, served_stations,
)

dates = get_dates(feed)
week = get_first_week(feed)
services = get_active_services(feed, dates[0])

trip_stats = compute_trip_stats(feed)
route_stats = compute_route_stats(feed, [dates[0]], trip_stats)
stop_stats = compute_stop_stats(feed, [dates[0]])

# Timetable graph for routing
graph = build_timetable_graph(feed, services, hour_filter=(6, 22))

# CSA-compatible connections
connections = compute_connections(feed, services)

# Segment frequencies weighted by service days
day_counts = get_service_day_counts(feed, dates)
freqs = compute_segment_frequencies(feed, services, service_day_counts=day_counts)
```

The API is inspired by [gtfs-kit](https://github.com/mrcagney/gtfs_kit),
re-implemented on Polars for significantly better performance.

### Performance vs gtfs-kit

Benchmarked on the STIB (Brussels) feed (~5.5 MB, ~9 000 trips):

| Operation             | gtfs-kit (pandas) | gtfs-parquet (Polars) | Speedup |
|-----------------------|------------------:|----------------------:|--------:|
| Load feed             |           2.97 s  |              0.40 s   |     7x  |
| `compute_trip_stats`  |          57.46 s  |              0.05 s   |  1149x  |
| `compute_stop_stats`  |           9.67 s  |              0.19 s   |    51x  |
| `compute_route_stats` |           2.12 s  |              0.08 s   |    27x  |
| `compute_busiest_date`|           0.07 s  |              0.06 s   |     1x  |

Peak process memory: **1020 MB** (gtfs-kit) vs **744 MB** (gtfs-parquet).

## License

[MIT](LICENSE)
