Metadata-Version: 2.4
Name: wikipediaGATN
Version: 0.1.4
Summary: Derive the global air transportation networks (pax and cargo) from Wikipedia
Project-URL: Homepage, https://github.com/julien-arino/wikipediaGATN
Project-URL: Repository, https://github.com/julien-arino/wikipediaGATN
Project-URL: Bug Tracker, https://github.com/julien-arino/wikipediaGATN/issues
Author-email: Julien Arino <julien.arino@umanitoba.ca>, Adriana-Stefania Ciupeanu <ciupeana@myumanitoba.ca>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Keywords: IATA,Wikipedia scraping,air transportation network,epidemiology,graph,network science
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: click>=8.0
Requires-Dist: geopy>=2.3
Requires-Dist: mwparserfromhell>=0.6
Requires-Dist: networkx>=3.0
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=1.5
Requires-Dist: plotly>=5.14
Requires-Dist: pycountry-convert>=0.7
Requires-Dist: pycountry>=22.3
Requires-Dist: pydot>=1.4
Requires-Dist: python-dateutil>=2.8
Requires-Dist: requests>=2.28
Requires-Dist: reverse-geocoder>=1.5.1
Requires-Dist: scipy>=1.10
Requires-Dist: spacy>=3.5
Provides-Extra: dev
Requires-Dist: black>=23.0; extra == 'dev'
Requires-Dist: mypy>=1.5; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: responses>=0.23; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Provides-Extra: docs
Requires-Dist: myst-parser>=2.0; extra == 'docs'
Requires-Dist: sphinx-rtd-theme>=1.3; extra == 'docs'
Requires-Dist: sphinx>=7.0; extra == 'docs'
Provides-Extra: test
Requires-Dist: pytest-cov>=4.1; extra == 'test'
Requires-Dist: pytest>=7.4; extra == 'test'
Requires-Dist: responses>=0.23; extra == 'test'
Description-Content-Type: text/markdown

# wikipediaGATN

[![Tests](https://github.com/julien-arino/wikipediaGATN/actions/workflows/tests.yml/badge.svg)](https://github.com/julien-arino/wikipediaGATN/actions/workflows/tests.yml)
[![Docs](https://github.com/julien-arino/wikipediaGATN/actions/workflows/docs.yml/badge.svg)](https://github.com/julien-arino/wikipediaGATN/actions/workflows/docs.yml)
[![PyPI version](https://img.shields.io/pypi/v/wikipediaGATN.svg)](https://pypi.org/project/wikipediaGATN/)
[![Checked with mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)

## Overview

`wikipediaGATN` scrapes Wikipedia airport pages to assemble the **Global Air Transportation Networks (GATN)**: two directed graphs in which each node is an airport (identified by its IATA code) and each directed edge represents a scheduled route between two airports for passengers (pax) or cargo. 

## Data

The collected data is available in the [`data/public/`](https://github.com/julien-arino/wikipediaGATN/tree/main/data/public) directory. See ``data/public/README.md`` for a detailed description of the files.

## Using Pre-built Data & Network Analysis

The global network data in `data/public/` is automatically updated weekly via GitHub Actions. **You do not need to run your own crawl to use this data.** You can clone this repository (or download the `data/` folder) and immediately load the provided sparse matrices to perform graph analysis.

For more details on available files and advanced analysis examples, see the [Quickstart Tutorials](https://julien-arino.github.io/wikipediaGATN/quickstart.html).

Here is an end-to-end example showing how to load the passenger network from the pre-built `.npz` file and analyze it with `networkx`:

```python
import numpy as np
import scipy.sparse
import networkx as nx

# 1. Load the node labels (IATA codes)
with open('data/public/nodes_pax.txt', 'r') as f:
    iata_codes = [line.strip() for line in f]

# 2. Load the sparse adjacency matrix
matrix = scipy.sparse.load_npz('data/public/adjacency_matrix_pax.npz')

# 3. Create a NetworkX directed graph
G = nx.from_scipy_sparse_array(matrix, create_using=nx.DiGraph)

# 4. Map node indices to their actual IATA codes
mapping = {i: code for i, code in enumerate(iata_codes)}
nx.relabel_nodes(G, mapping, copy=False)

print(f"Network has {G.number_of_nodes()} airports and {G.number_of_edges()} routes.")

# 5. Basic analysis: find the airport with the most incoming flights
in_degrees = dict(G.in_degree())
most_connected = max(in_degrees, key=in_degrees.get)
print(f"Most connected destination: {most_connected} ({in_degrees[most_connected]} incoming routes)")
```

## Package pipeline

The package handles the full pipeline:

1. **Crawling** — breadth-first traversal from a seed airport, following destination links to neighbouring airport pages.
2. **Parsing** — extraction of IATA/ICAO codes, geographic coordinates, and route tables from Wikipedia infoboxes and HTML tables, supplemented by the authoritative [OurAirports](https://ourairports.com/) database for metadata.
3. **IATA recovery** — resolution of destination URLs that lack an obvious code, prioritizing offline lookups in the [OurAirports](https://ourairports.com/) database before falling back to Wikipedia scraping.
4. **Export** — sparse adjacency matrices (`.npz`), node lists, airport metadata CSVs ready for network analysis, and interactive Plotly visualisations (`.html`).
5. **Updates** — on demand maintenance of the network through incremental scraping and synchronization with upstream [OurAirports](https://ourairports.com/) metadata changes, keeping the graphs up to date. Updates are easy to perform as GitHub actions (see .github/workflows/refresh_data.yml).

The resulting networks can be used for empirical studies of air-travel connectivity, epidemic-spread modelling and transportation network analysis, among other things. 
They also provide great examples in courses about graphs/networks, data science and computational social science.

## Installation

### From PyPI (Recommended)

To install the latest stable version directly from PyPI:

```bash
pip install wikipediaGATN
```

### From Source

For development or to access the latest changes, clone the repository and install it:

```bash
git clone https://github.com/julien-arino/wikipediaGATN.git
cd wikipediaGATN
pip install .
```

If you wish to contribute, install in editable mode with development dependencies:

```bash
pip install -e ".[dev]"
```

If running from the source repository before deploying the package, set:

```bash
export PYTHONPATH=src
```

and then call the code using, e.g.,

```
python -m scripts.grab_info_from_IATA
```

Note the nonstandard call: `-m`, `.` instead of `/` to indicate a subdirectory and no `.py` extension.


## Required post-install step — spaCy language model

The NLP fallback for airline/destination extraction requires the
`en_core_web_sm` model, which cannot be declared as a standard PyPI
dependency:

```bash
python -m spacy download en_core_web_sm
```

### Dependencies

| Package | Purpose |
|---|---|
| `requests`, `beautifulsoup4` | Wikipedia HTTP requests and HTML parsing |
| `mwparserfromhell` | Wikitext infobox parsing |
| `spacy` | NLP fallback for unstructured route tables |
| `geopy`, `pycountry` | Coordinate and ISO 3166-2 parsing |
| `numpy`, `scipy` | Sparse adjacency matrix construction |
| `pandas` | CSV I/O and data manipulation |
| `networkx` | Graph construction and layout |
| `plotly` | Interactive HTML visualisation |

---

## Example use

Some scripts are provided as examples in the `scripts/` directory, which comprise instructions to carry out all steps of the pipeline..

For function example, the following builds a network for all airports reachable within two hops of Winnipeg (YWG) and exports it as a sparse adjacency matrix:

```python
from wikipediaGATN.wikipedia_network_level import iterate_search_until_distance_N
from wikipediaGATN import (
    export_all_airport_data,
    create_outbound_connections_list,
    run_two_pass_iata_extraction,
    create_outbound_adjacency_matrix,
)

# 1. Crawl Wikipedia — save one JSON file per airport to data/tmp_results/
iterate_search_until_distance_N("YWG", dist=2, delay=0.5, verbose=True)

# 2. Process and export crawled airport data (creates public/airport_data/ and airports_information.csv)
export_all_airport_data(use_new_data=True, verbose=True)

# 3. Build connections CSV (maps destination URLs to IATA codes)
connections_csv, cargo_csv, unmapped_csv = create_outbound_connections_list(
    verbose=True, export_unmapped=True
)

# 4. Recover IATA codes for any destinations that could not be mapped automatically
#    (scrapes Wikipedia; allow ~15 minutes for a large unmapped set)
run_two_pass_iata_extraction(batch_size=50, delay=0.5, verbose=True)

# 5. Re-export airport data to apply the recovered manual mappings
export_all_airport_data(use_new_data=True, verbose=True)

# 6. Re-run connections with the enriched mapping
create_outbound_connections_list(verbose=True)

# 7. Export sparse adjacency matrices to data/public/
matrix_npz, nodes_txt = create_outbound_adjacency_matrix(symmetric=False, verbose=True)
matrix_sym_npz, nodes_sym_txt = create_outbound_adjacency_matrix(symmetric=True, verbose=True)
```

For a full global crawl (several hours) replace step 1 with:

```python
from wikipediaGATN.wikipedia_network_level import iterate_search_until_empty
iterate_search_until_empty("YWG", delay=0.5, verbose=True)
```

To resume after an interruption:

```python
from wikipediaGATN.wikipedia_network_level import continue_existing_search_until_empty
continue_existing_search_until_empty(delay=0.5, verbose=True)

