{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b43ee99c",
   "metadata": {},
   "source": [
    "# StatCan parser walkthrough\n",
    "\n",
    "This notebook is the practical guide for parsing Statistics Canada supply-use and symmetric input-output tables in MARIO.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfc07c40",
   "metadata": {},
   "source": [
    "## What this notebook covers\n",
    "\n",
    "- where the StatCan tables come from;\n",
    "- when direct online parsing is enough and when a local cache is useful;\n",
    "- how `SUT` and `IOT` workflows differ;\n",
    "- how `year=`, `level=`, `geo=`, and `valuation=` are used;\n",
    "- what to expect from `download=True`;\n",
    "- which caveats matter for this API-driven parser.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a8602e",
   "metadata": {},
   "source": [
    "## Relevant source pages\n",
    "\n",
    "- Official StatCan SUT catalogue: [Supply and use tables](https://www150.statcan.gc.ca/n1/en/catalogue/36100478)\n",
    "- Official StatCan IOT catalogue: [Symmetric input-output tables](https://www150.statcan.gc.ca/n1/en/catalogue/36100001)\n",
    "- StatCan WDS documentation: [WDS user guide](https://www.statcan.gc.ca/en/developers/wds/user-guide)\n",
    "\n",
    "MARIO can query the official WDS API directly, so there is no need to download the raw `.csv` bundles manually unless you want a local cache.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a98f11b6",
   "metadata": {},
   "source": [
    "## Main entry point\n",
    "\n",
    "For normal user workflows, the public entry point is:\n",
    "\n",
    "- `mario.parse_statcan(...)`\n",
    "\n",
    "The same function supports:\n",
    "\n",
    "- `SUT` tables;\n",
    "- `IOT` tables;\n",
    "- `summary`, `detail`, and `link1997` levels where available;\n",
    "- direct online parsing or local cached raw files.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "473a4e6d",
   "metadata": {},
   "source": [
    "## Key arguments\n",
    "\n",
    "The key public arguments are:\n",
    "\n",
    "- `year`: reference year to parse;\n",
    "- `table`: choose `\"SUT\"` or `\"IOT\"`;\n",
    "- `level`: choose `summary`, `detail`, or `link1997` when supported;\n",
    "- `geo`: geography label such as `Canada` or one province/territory;\n",
    "- `valuation`: only for `IOT`, usually `basic` or `purchaser`;\n",
    "- `path`: optional directory for locally cached raw files;\n",
    "- `download`: when `True`, MARIO downloads the raw WDS bundle into `path` and then parses it locally.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69ca5b6",
   "metadata": {},
   "source": [
    "## `SUT` versus `IOT`\n",
    "\n",
    "Use `table=\"SUT\"` when you want the native split structure with `S`, `U`, `Yc`, `Ya`, `Va`, and `Vc`.\n",
    "\n",
    "Use `table=\"IOT\"` when you want the symmetric input-output representation with `Z`, `Y`, and `V`.\n",
    "\n",
    "For `SUT`, `level=\"link1997\"` is also available. For `IOT`, `valuation=` matters because the tables are exposed at both basic and purchaser prices.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e691f8e5",
   "metadata": {},
   "source": [
    "## Direct online parsing versus local cache\n",
    "\n",
    "Use direct online parsing when you just want the database and do not need the raw files afterwards.\n",
    "\n",
    "Use `download=True` together with `path=...` when you want MARIO to keep the raw WDS `.csv` bundle locally. This is the most useful option when you parse the same StatCan table repeatedly or want a reproducible local cache.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2ede926",
   "metadata": {},
   "source": [
    "## Cache layout and supported values\n",
    "\n",
    "`path` is optional and acts as a local WDS cache:\n",
    "\n",
    "```text\n",
    "StatCan/\n",
    "└── cache/\n",
    "    ├── 36-10-0438-01_*.csv\n",
    "    └── 36-10-0001-01_*.csv\n",
    "```\n",
    "\n",
    "Supported levels:\n",
    "\n",
    "- `SUT`: `summary`, `detail`, `link1997`;\n",
    "- `IOT`: `summary`, `detail`.\n",
    "\n",
    "For `IOT`, `valuation` can be `basic` or `purchaser`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c04cfdfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mario"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6430ecbf",
   "metadata": {},
   "source": [
    "## Parse a Canada `SUT` directly from WDS\n",
    "\n",
    "This is the simplest StatCan workflow.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b5ff1bf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: requesting StatCan WDS metadata for table 36100438.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: downloading StatCan table 36100438 from https://www150.statcan.gc.ca/n1/tbl/csv/36100438-eng.zip.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: StatCan SUT payload ready with shapes S=(32, 63), U=(63, 32), Yc=(63, 18), Va=(9, 32), Vc=(9, 63).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = StatCan SUT summary Canada 2022\n",
       "table = SUT\n",
       "tech_assumption = industry-based\n",
       "scenarios = ['baseline']\n",
       "Activity = 32\n",
       "Commodity = 63\n",
       "Factor of production = 9\n",
       "Satellite account = 1\n",
       "Consumption category = 18\n",
       "Region = 1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_statcan(\n",
    "    year=2022,\n",
    "    table=\"SUT\",\n",
    "    level=\"summary\",\n",
    "    geo=\"Canada\",\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d3306f",
   "metadata": {},
   "source": [
    "## Parse a provincial `SUT`\n",
    "\n",
    "The `SUT` tables expose provinces and territories as separate geographies. The parser expects the published StatCan geography label.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0b3e1e42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: requesting StatCan WDS metadata for table 36100478.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: downloading StatCan table 36100478 from https://www150.statcan.gc.ca/n1/tbl/csv/36100478-eng.zip.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: StatCan SUT payload ready with shapes S=(216, 473), U=(473, 216), Yc=(473, 289), Va=(9, 216), Vc=(9, 473).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = StatCan SUT detail Ontario 2022\n",
       "table = SUT\n",
       "tech_assumption = industry-based\n",
       "scenarios = ['baseline']\n",
       "Activity = 216\n",
       "Commodity = 473\n",
       "Factor of production = 9\n",
       "Satellite account = 1\n",
       "Consumption category = 289\n",
       "Region = 1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_statcan(\n",
    "    year=2022,\n",
    "    table=\"SUT\",\n",
    "    level=\"detail\",\n",
    "    geo=\"Ontario\",\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8afe5538",
   "metadata": {},
   "source": [
    "## Parse an `IOT` at basic prices\n",
    "\n",
    "For `IOT`, pass `valuation=\"basic\"` or `valuation=\"purchaser\"`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ce967a52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: requesting StatCan WDS metadata for table 36100001.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: downloading StatCan table 36100001 from https://www150.statcan.gc.ca/n1/tbl/csv/36100001-eng.zip.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: StatCan IOT payload ready with shapes Z=(252, 252), Y=(252, 306), V=(8, 252).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = StatCan IOT detail Canada 2022 Basic price\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 8\n",
       "Satellite account = 1\n",
       "Consumption category = 306\n",
       "Region = 1\n",
       "Sector = 252"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_statcan(\n",
    "    year=2022,\n",
    "    table=\"IOT\",\n",
    "    level=\"detail\",\n",
    "    valuation=\"basic\",\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49e46772",
   "metadata": {},
   "source": [
    "## Parse an `IOT` at purchaser prices\n",
    "\n",
    "The public API stays the same; only the `valuation` selector changes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "92a7aba9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: requesting StatCan WDS metadata for table 36100084.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: downloading StatCan table 36100084 from https://www150.statcan.gc.ca/n1/tbl/csv/36100084-eng.zip.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: StatCan IOT payload ready with shapes Z=(32, 32), Y=(32, 28), V=(7, 32).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = StatCan IOT summary Canada 2022 Purchaser price\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 7\n",
       "Satellite account = 1\n",
       "Consumption category = 28\n",
       "Region = 1\n",
       "Sector = 32"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_statcan(\n",
    "    year=2022,\n",
    "    table=\"IOT\",\n",
    "    level=\"summary\",\n",
    "    valuation=\"purchaser\",\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edc9014",
   "metadata": {},
   "source": [
    "## Keep the raw WDS files locally\n",
    "\n",
    "When `download=True`, MARIO stores the downloaded raw `.csv` bundle inside `path` and then parses the local file. This is useful when you want to avoid re-downloading the same table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "643be669",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading local StatCan CSV statcan_36100478_sut_detail.csv.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: StatCan SUT payload ready with shapes S=(217, 472), U=(472, 217), Yc=(472, 289), Va=(9, 217), Vc=(9, 472).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = StatCan SUT detail Quebec 2022\n",
       "table = SUT\n",
       "tech_assumption = industry-based\n",
       "scenarios = ['baseline']\n",
       "Activity = 217\n",
       "Commodity = 472\n",
       "Factor of production = 9\n",
       "Satellite account = 1\n",
       "Consumption category = 289\n",
       "Region = 1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_statcan(\n",
    "    year=2022,\n",
    "    table=\"SUT\",\n",
    "    level=\"detail\",\n",
    "    geo=\"Quebec\",\n",
    "    path=\"/path/to/StatCan\",\n",
    "    download=True,\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd1aebe",
   "metadata": {},
   "source": [
    "## Caveats\n",
    "\n",
    "- the parser is API-driven, so network availability matters more than for local-file parsers;\n",
    "- `download=True` is optional, not required;\n",
    "- this walkthrough focuses on the economic StatCan tables only;\n",
    "- environmental extensions are intentionally left out here.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mario",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
