{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ceads-title",
   "metadata": {},
   "source": [
    "# CEADS parser walkthrough\n",
    "\n",
    "This notebook is the practical guide for parsing the currently supported CEADS China provincial MRIO workbooks in MARIO."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-coverage",
   "metadata": {},
   "source": [
    "## What this notebook covers\n",
    "\n",
    "- where the supported CEADS workbooks come from;\n",
    "- which workbook family MARIO currently parses;\n",
    "- what `format=` means;\n",
    "- how to use `path=` and `year=`;\n",
    "- how MARIO maps imports, exports, and the optional `CO2` row."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-sources",
   "metadata": {},
   "source": [
    "## Source pages\n",
    "\n",
    "- CEADS portal: [Input-Output Tables](https://www.ceads.net/data/input_output_tables/)\n",
    "- Scientific Data descriptor: [China's Provincial Multi-Regional Input-Output Database for 2018 and 2020](https://www.nature.com/articles/s41597-025-06543-y)\n",
    "- Dataset DOI: [10.6084/m9.figshare.29927291](https://doi.org/10.6084/m9.figshare.29927291)\n",
    "\n",
    "MARIO does not download the CEADS workbook automatically. Download the workbook locally before running this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-entrypoint",
   "metadata": {},
   "source": [
    "## Main entry point\n",
    "\n",
    "For normal user workflows, the public entry point is `mario.parse_ceads(...)`.\n",
    "\n",
    "At the moment, that entry point supports only `table=\"IOT\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-supported-family",
   "metadata": {},
   "source": [
    "## What MARIO currently parses\n",
    "\n",
    "The current backend is intentionally narrow:\n",
    "\n",
    "- it parses local Excel workbooks only;\n",
    "- it currently supports the verified CEADS provincial MRIO workbook family for `2018` and `2020`;\n",
    "- it reads the English table sheet, not the Chinese one;\n",
    "- it uses the `Sector` and `Province` sheets as metadata support."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-format",
   "metadata": {},
   "source": [
    "## What `format` means\n",
    "\n",
    "`format=` is the parser-side layout selector. It does not mean \"which CEADS dataset\" you are using; it means \"which workbook structure\" MARIO should expect.\n",
    "\n",
    "At the moment, MARIO supports:\n",
    "\n",
    "- `format=\"auto\"`: inspect the workbook and resolve the known layout automatically;\n",
    "- `format=\"ceads_provincial_workbook\"`: force the verified 2018/2020 workbook structure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-paths",
   "metadata": {},
   "source": [
    "## Configure local paths\n",
    "\n",
    "Set `MARIO_DOC_DATA` to the root folder where you keep real parser datasets, or set `MARIO_CEADS_PATH` directly to the CEADS folder.\n",
    "\n",
    "Expected local structure:\n",
    "\n",
    "```text\n",
    "CEADS/\n",
    "├── MRIO 2018.xlsx\n",
    "└── MRIO 2020.xlsx\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ceads-imports",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "import mario"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-parse-file",
   "metadata": {},
   "source": [
    "## Parse one explicit workbook\n",
    "\n",
    "Use this form when you want to parse a specific downloaded workbook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ceads-parse-2018",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: inspecting CEADS workbook MRIO 2018.xlsx.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: detected CEADS workbook format ceads_provincial_workbook for year 2018 using sheet Table_2018_English Version.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading CEADS workbook MRIO 2018.xlsx sheet Table_2018_English Version.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: CEADS payload ready with shapes Z=(1302, 1302), Y=(1302, 186), V=(5, 1302), E=(1, 1302), EY=(1, 186).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = CEADS China Provincial MRIO 2018\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 5\n",
       "Satellite account = 1\n",
       "Consumption category = 6\n",
       "Region = 31\n",
       "Sector = 42"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_2018 = mario.parse_ceads(\n",
    "    path=\"/path/to/MRIO 2018.xlsx\",\n",
    "    format=\"auto\",\n",
    "    calc_all=False,\n",
    ")\n",
    "\n",
    "db_2018"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-parse-directory",
   "metadata": {},
   "source": [
    "## Parse from a directory\n",
    "\n",
    "Use this form when one folder contains multiple CEADS workbooks. In that case, `year=` selects which workbook to parse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ceads-parse-directory-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: inspecting CEADS workbook MRIO 2020.xlsx.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: detected CEADS workbook format ceads_provincial_workbook for year 2020 using sheet Table_2020_English Version.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading CEADS workbook MRIO 2020.xlsx sheet Table_2020_English Version.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: CEADS payload ready with shapes Z=(1302, 1302), Y=(1302, 186), V=(5, 1302), E=(1, 1302), EY=(1, 186).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = CEADS China Provincial MRIO 2020\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 5\n",
       "Satellite account = 1\n",
       "Consumption category = 6\n",
       "Region = 31\n",
       "Sector = 42"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_2020 = mario.parse_ceads(\n",
    "    path=\"/path/to/MRIO 2020.xlsx\",\n",
    "    year=2020,\n",
    "    calc_all=False,\n",
    ")\n",
    "\n",
    "db_2020"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-axis-semantics",
   "metadata": {},
   "source": [
    "## How the parsed table is mapped\n",
    "\n",
    "For the verified CEADS workbook family:\n",
    "\n",
    "- `Z` comes from the main intermediate-transactions block;\n",
    "- `Y` contains the domestic final-demand block plus one `Exports` column per province;\n",
    "- `V` contains `Imports` plus the four value-added rows exposed by the workbook;\n",
    "- `E` contains the optional `CO2 (Mt)` row when it exists;\n",
    "- `EY` is initialized at zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ceads-blocks",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['Z', 'Y', 'V', 'E', 'EY'])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_2020[\"baseline\"].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-inspection",
   "metadata": {},
   "source": [
    "## inspect the parsed database\n",
    "\n",
    "Once parsed, the result is a standard MARIO database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ceads-regions",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Beijing',\n",
       " 'Tianjin',\n",
       " 'Hebei',\n",
       " 'Shanxi',\n",
       " 'Inner Mongolia',\n",
       " 'Liaoning',\n",
       " 'Jilin',\n",
       " 'Heilongjiang',\n",
       " 'Shanghai',\n",
       " 'Jiangsu',\n",
       " 'Zhejiang',\n",
       " 'Anhui',\n",
       " 'Fujian',\n",
       " 'Jiangxi',\n",
       " 'Shandong',\n",
       " 'He0',\n",
       " 'Hubei',\n",
       " 'Hu0',\n",
       " 'Guangdong',\n",
       " 'Guangxi',\n",
       " 'Hai0',\n",
       " 'Chongqing',\n",
       " 'Sichuan',\n",
       " 'Guizhou',\n",
       " 'Yun0',\n",
       " 'Xizang',\n",
       " 'Shaanxi',\n",
       " 'Gansu',\n",
       " 'Qinghai',\n",
       " 'Ningxia',\n",
       " 'Xinjiang']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_2020.get_index(\"Region\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ceads-sectors",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Agriculture, Forestry, Animal Husbandry and Fishery',\n",
       " 'Mining and washing of coal',\n",
       " 'Extraction of petroleum and natural gas',\n",
       " 'Mining and processing of metal ores',\n",
       " 'Mining and processing of nonmetal and other ores',\n",
       " 'Food and tobacco processing',\n",
       " 'Textile industry',\n",
       " 'Manufacture of leather, fur, feather and related products',\n",
       " 'Processing of timber and furniture',\n",
       " 'Manufacture of paper, printing and articles for culture, education and sport activity',\n",
       " 'Processing of petroleum, coking, processing of nuclear fuel',\n",
       " 'Manufacture of chemical products',\n",
       " 'Manuf. of non -metallic mineral products',\n",
       " 'Smelting and processing of metals',\n",
       " 'Manufacture of metal products',\n",
       " 'Manufacture of general purpose machinery',\n",
       " 'Manufacture of special purpose machinery',\n",
       " 'Manufacture of transport equipment',\n",
       " 'Manufacture of electrical machinery and equipment',\n",
       " 'Manufacture of communication equipment, computers and other electronic equipment',\n",
       " 'Manufacture of measuring instruments',\n",
       " 'Other manufacturing',\n",
       " 'Comprehensive use of waste resources',\n",
       " 'Repair of metal products, machinery and equipment',\n",
       " 'Production and distribution of electric power and heat power',\n",
       " 'Production and distribution of gas',\n",
       " 'Production and distribution of tap water',\n",
       " 'Construction',\n",
       " 'Wholesale and retail trades',\n",
       " 'Transport, storage, and postal services',\n",
       " 'Accommodation and catering',\n",
       " 'Information transfer, software and information technology services',\n",
       " 'Fi0ce',\n",
       " 'Real estate',\n",
       " 'Leasing and commercial services',\n",
       " 'Scientific research and polytechnic services',\n",
       " 'Administration of water, environment, and public facilities',\n",
       " 'Resident, repair and other services',\n",
       " 'Education',\n",
       " 'Health care and social work',\n",
       " 'Culture, sports, and entertainment',\n",
       " 'Public administration, social insurance, and social organizations']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_2020.get_index(\"Sector\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ceads-satellites",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['CO2']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_2020.get_index(\"Satellite account\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceads-caveats",
   "metadata": {},
   "source": [
    "## Caveats\n",
    "\n",
    "- The CEADS portal lists older provincial MRIO releases too, such as `2012`, `2015`, and `2017`. MARIO does not assume those older workbook families are compatible until they are checked explicitly.\n",
    "- The parser intentionally relies on the English worksheet. If only a Chinese worksheet is available, that workbook is not currently supported.\n",
    "- The workbook metadata sheets can contain minor naming inconsistencies. For the verified 2018/2020 family, MARIO treats the main transaction axis as canonical when such conflicts appear."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mario",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
