Metadata-Version: 2.4
Name: statcan-dl
Version: 0.1.0
Summary: Snapshot Statistics Canada tables into analysis-ready Parquet
License-Expression: MIT
Project-URL: Homepage, https://github.com/ink-waffle/statcan-dl
Keywords: statcan,statistics-canada,health,open-data,parquet,dataset
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31
Requires-Dist: pyarrow>=14.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Dynamic: license-file

# statcan-dl

Takes a local snapshot of [Statistics Canada](https://www150.statcan.gc.ca/)'s health tables — all 400-odd of them — and leaves you with Parquet files you can open in pandas.

This is not a query API. It mirrors what StatCan publishes to your disk in one shot: no API key, no auth, no rate limits to negotiate. Run it once to get the tables, re-run it later to pick up revisions. Every step skips work already on disk, so an interrupted run resumes where it stopped.

Expect roughly **10 GB** and 20–60 minutes on a decent connection for the full health subject.

## Install

```bash
pip install git+https://github.com/ink-waffle/statcan-dl
```

## Get the data

```bash
statcan-dl
```

That writes to `output/statcan/` relative to your current directory. The part you'll actually use is `output/statcan/tables/`:

```python
import pandas as pd

df = pd.read_parquet("output/statcan/tables/13100489 — Perceived health.parquet")
df.head()
```

Every table arrives in StatCan's long format — one row per (period, geography, characteristic) combination:

| Column | Meaning |
|---|---|
| `REF_DATE` | reference period |
| `GEO` | geography (Canada, a province, a health region…) |
| `VALUE` | the number, as a float |
| `UOM`, `SCALAR_FACTOR` | unit and multiplier — **read these before comparing values** |
| `STATUS` | suppression / quality flag; non-empty means `VALUE` needs care |
| remaining columns | that table's own dimensions — `Sex`, `Perceived health`, `Characteristics` for the table above |

Don't need all of it? Narrow it:

```bash
statcan-dl --pids 13100489,13100098       # just these tables
statcan-dl --limit 20                     # first 20, useful for a trial run
statcan-dl --subject 1310                 # a narrower branch of the subject tree
statcan-dl -o ~/data/statcan -w 16        # elsewhere, more parallelism
```

Subject codes are a hierarchy and match by prefix, so `13` is all of Health and `1310` is a branch within it. List what you'd get without downloading:

```bash
python -m statcan_dl.download --list
```

## Dimension metadata

The CSVs give you dimension *values* but not the structure behind them. `output/statcan/table_metadata.json` holds that, keyed by product ID — which dimensions a table has, every member of each, and how members nest via `parent`:

```python
import json

meta = json.load(open("output/statcan/table_metadata.json"))
for dim in meta["13100489"]["dimensions"]:
    print(dim["position"], dim["name"], len(dim["members"]))
```

That `parent` field is what lets you tell a total apart from its components — necessary if you're aggregating, since totals sit in the same column as the things they total.

## What it produces

```
output/statcan/
├── csv/<pid>/              raw CSV archives, as extracted
├── tables/                 one Parquet per table  ← use these
└── table_metadata.json     dimensions and members, per table
```

`csv/` is an intermediate; it exists so re-runs don't re-download and so you can go back to the source if a conversion looks wrong. Delete it once you have `tables/` and you save most of the disk.

Converting drops seven bookkeeping columns that carry no information once a table stands alone: `DGUID`, `UOM_ID`, `SCALAR_ID`, `VECTOR`, `COORDINATE`, `TERMINATED`, `SYMBOL`.

## Pipeline steps

| Step | Does |
|---|---|
| `download` | lists cubes via the WDS API, fetches each table's CSV archive |
| `metadata` | fetches dimensions and members via `getCubeMetadata` |
| `combine` | CSV → Parquet, dropping bookkeeping columns and typing `VALUE` |

Run one on its own with `--only`, or exclude one with `--skip-<step>`:

```bash
statcan-dl --only combine
statcan-dl --skip-metadata
```

Individual steps are also importable and directly runnable:

```bash
python -m statcan_dl.download --limit 5
python -m statcan_dl.combine
```

## Caveats

- `VALUE` becomes text instead of a float when a table mixes numbers with suppression markers that won't parse. Check the dtype before doing arithmetic.
- Archived tables are excluded. They're still on StatCan's site, but they're superseded and often overlap live tables.
- Titles are truncated to 120 characters in filenames. The full title is in the Parquet metadata under `statcan_title`, and in `table_metadata.json`.
- English only — the loader requests `-eng.zip` archives.

## Tests

```bash
pip install -e ".[dev]"
pytest                  # runs the real pipeline over 3 small tables
pytest -m "not network" # offline only
```

## Licence

MIT. The data itself is published by Statistics Canada under the [Open Licence](https://www.statcan.gc.ca/en/reference/licence).
