{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b0f1a54d",
   "metadata": {},
   "source": [
    "# EMERGING parser walkthrough\n",
    "\n",
    "This notebook is the practical guide for parsing the EMERGING MATLAB bundles supported by MARIO.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74bb736f",
   "metadata": {},
   "source": [
    "## What this notebook covers\n",
    "\n",
    "- which official Zenodo version records are relevant for MARIO;\n",
    "- how local file naming works for `v1.0`, `v2.x`, and `variant=\"E\"` bundles;\n",
    "- how `download_emerging(...)` maps to the supported version records;\n",
    "- how `year=`, `regions=`, `load_co2=`, `co2_path=`, and `variant=` are used;\n",
    "- which caveats matter for very large EMERGING and EMERGING-E databases.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca35af5e",
   "metadata": {},
   "source": [
    "## Supported official version records\n",
    "\n",
    "The relevant official records are:\n",
    "\n",
    "- `v2.2`: [Zenodo 19461860](https://doi.org/10.5281/zenodo.19461860)\n",
    "- `v2.0`: [Zenodo 17557778](https://doi.org/10.5281/zenodo.17557778)\n",
    "- `v1.0`: [Zenodo 10956623](https://doi.org/10.5281/zenodo.10956623)\n",
    "- `E`: [Zenodo 18303090](https://doi.org/10.5281/zenodo.18303090)\n",
    "\n",
    "Any other version reference should be treated as deprecated in the MARIO documentation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f49fcea4",
   "metadata": {},
   "source": [
    "## Main entry point\n",
    "\n",
    "For normal user workflows, the public entry point is:\n",
    "\n",
    "- `mario.parse_emerging(...)`\n",
    "\n",
    "The parser currently supports only the multiregional `IOT` workflow.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9baf06b0",
   "metadata": {},
   "source": [
    "## Key arguments\n",
    "\n",
    "The key public arguments are:\n",
    "\n",
    "- `path`: one EMERGING `.mat` file or a directory containing multiple yearly bundles;\n",
    "- `table`: currently only `\"IOT\"` is supported;\n",
    "- `year`: use it when one directory contains more than one EMERGING year;\n",
    "- `regions`: optional ISO3 subset to keep only one manageable part of the database;\n",
    "- `load_co2`: enable or disable automatic companion CO2 import;\n",
    "- `co2_path`: explicit path to one companion CO2 file when auto-detection is not enough;\n",
    "- `labels_path`: optional Excel workbook with explicit sector labels; for `variant=\"E\"` it overrides workbook auto-detection next to the bundle;\n",
    "- `variant`: keep `\"standard\"` for the original EMERGING releases and use `\"E\"` for the EMERGING-E power-disaggregated bundle.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "033d39ad",
   "metadata": {},
   "source": [
    "## Local file naming conventions\n",
    "\n",
    "In practice MARIO accepts these local naming conventions:\n",
    "\n",
    "- `global_mrio_<year>.mat` for `v1.0`;\n",
    "- `EMERGING_V2_<year>_m.mat` for `v2.x`;\n",
    "- `EMERGING_V2_<year>.mat` for older local `v2.x` copies when the internal MATLAB structure is still compatible;\n",
    "- `EMERGING_E_<year>.mat` for the EMERGING-E power-disaggregated variant.\n",
    "\n",
    "For local `v2.x` files, MARIO does not try to infer the exact sub-version `2.0` versus `2.1` versus `2.2` from the filename alone, because the public naming convention is shared across those releases.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cc5c1a2",
   "metadata": {},
   "source": [
    "## Expected path structure\n",
    "\n",
    "`path` can point either to one yearly `.mat` file or to a directory containing the yearly EMERGING bundle and, optionally, the companion CO2 file.\n",
    "\n",
    "Typical directory layout:\n",
    "\n",
    "```text\n",
    "EMERGING/2.2/\n",
    "├── EMERGING_V2_2023_m.mat\n",
    "└── EMERGING_CO2_2023.mat\n",
    "```\n",
    "\n",
    "For the older `v1.0` naming convention, the economic bundle is typically named like `global_mrio_<year>.mat`.\n",
    "For EMERGING-E, the local bundle is typically named like `EMERGING_E_<year>.mat` and may ship a sibling `Figure data.xlsx` workbook that MARIO uses to recover ISO3 region codes.\n",
    "\n",
    "When `path` is a directory, use `year=` to select the economic bundle. With `load_co2=True`, MARIO first looks for the matching CO2 file in the same directory, for example `EMERGING_CO2_2023.mat` for `year=2023`. Use `co2_path=` when the companion file is stored elsewhere or has a non-standard name."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a26ce2",
   "metadata": {},
   "source": [
    "## Download workflow\n",
    "\n",
    "Use `mario.download_emerging(...)` when you want MARIO to fetch one of the supported official version records, including the newer `version=\"E\"` EMERGING-E record.\n",
    "\n",
    "`latest` currently resolves to `v2.2`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7de03057",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mario"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9090717",
   "metadata": {},
   "source": [
    "## Download one official version explicitly\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dca14d51",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source': 'https://doi.org/10.5281/zenodo.10956622',\n",
       " 'version': '2.2',\n",
       " 'version_record': 'https://doi.org/10.5281/zenodo.19461860',\n",
       " 'download_dir': '/path/to/emerging_directory',\n",
       " 'years': [2023],\n",
       " 'files': ['/path/to/emerging_directory/EMERGING_CO2_2023.mat',\n",
       "  '/path/to/EMERGING_V2_2023_m.mat']}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mario.download_emerging(\n",
    "    path=\"/path/to/2.2\",\n",
    "    version=\"2.2\",\n",
    "    years=[2023],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f90884b3",
   "metadata": {},
   "source": [
    "## Download the EMERGING-E record\n",
    "\n",
    "Use `version=\"E\"` when you want MARIO to fetch the public EMERGING-E Zenodo record. With `years=[2018]`, MARIO keeps the main `.mat` file and the companion `Figure data.xlsx` workbook used to recover ISO3 region codes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c11aaf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "mario.download_emerging(\n",
    "    path=\"/path/to/emerging-e\",\n",
    "    version=\"E\",\n",
    "    years=[2018],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2e72cf9",
   "metadata": {},
   "source": [
    "## Parse one local EMERGING bundle\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5820caa9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING bundle EMERGING_V2_2023_m.mat.\n",
      "INFO Parser: reading EMERGING CO2 file EMERGING_CO2_2023.mat.\n",
      "INFO Parser: EMERGING parsed with 245 regions, 133 sectors, 735 final-demand columns and 7 satellite rows.\n",
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = EMERGING 2023\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 1\n",
       "Satellite account = 7\n",
       "Consumption category = 3\n",
       "Region = 245\n",
       "Sector = 133"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_emerging(\n",
    "    path=\"/path/to/EMERGING_V2_2023_m.mat\",\n",
    "    table=\"IOT\",\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0375cf5a",
   "metadata": {},
   "source": [
    "## Parse one local EMERGING-E bundle\n",
    "\n",
    "`EMERGING-E` is currently a local-file workflow. Use `variant=\"E\"` and point `path` to `EMERGING_E_<year>.mat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86663529",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING-E bundle EMERGING_E_2018.mat.\n",
      "INFO Parser: EMERGING-E parsed with 245 regions, 146 sectors, 735 final-demand columns and 1 satellite rows.\n",
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = EMERGING-E 2018\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 1\n",
       "Satellite account = 1\n",
       "Consumption category = 3\n",
       "Region = 245\n",
       "Sector = 146"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_e = mario.parse_emerging(\n",
    "    # path=\"/path/to/EMERGING_E_2018.mat\",\n",
    "    path='/Users/lorenzorinaldi/Library/CloudStorage/OneDrive-SharedLibraries-PolitecnicodiMilano/DENG-SESAM - Documenti/c-Research/a-Datasets/_Input Output Databases/EMERGING/EMERGING-E/EMERGING_E_2018.mat',\n",
    "    table=\"IOT\",\n",
    "    variant=\"E\"\n",
    ")\n",
    "\n",
    "db_e"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38a7e3c4",
   "metadata": {},
   "source": [
    "## Parse from a directory and select one year\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8f095295",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING bundle EMERGING_V2_2023_m.mat.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING CO2 file EMERGING_CO2_2023.mat.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: EMERGING parsed with 245 regions, 133 sectors, 735 final-demand columns and 7 satellite rows.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = EMERGING 2023\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 1\n",
       "Satellite account = 7\n",
       "Consumption category = 3\n",
       "Region = 245\n",
       "Sector = 133"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_emerging(\n",
    "    path=\"/path/to/2.2\",\n",
    "    table=\"IOT\",\n",
    "    year=2023,\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c3810b5",
   "metadata": {},
   "source": [
    "## Restrict the region set\n",
    "\n",
    "The full EMERGING matrix is very large, so `regions=` is often the right first step.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "40930d10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING bundle EMERGING_V2_2023_m.mat.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING CO2 file EMERGING_CO2_2023.mat.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: EMERGING parsed with 3 regions, 133 sectors, 9 final-demand columns and 7 satellite rows.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = EMERGING 2023\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 1\n",
       "Satellite account = 7\n",
       "Consumption category = 3\n",
       "Region = 3\n",
       "Sector = 133"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_emerging(\n",
    "    path=\"/path/to/2.2\",\n",
    "    table=\"IOT\",\n",
    "    year=2023,\n",
    "    regions=[\"DEU\", \"FRA\", \"ITA\"],\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "822b1e7c",
   "metadata": {},
   "source": [
    "## Control CO2 loading explicitly\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d6b415f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING bundle EMERGING_V2_2023_m.mat.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading EMERGING CO2 file EMERGING_CO2_2023.mat.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: EMERGING parsed with 245 regions, 133 sectors, 735 final-demand columns and 7 satellite rows.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = EMERGING 2023\n",
       "table = IOT\n",
       "scenarios = ['baseline']\n",
       "Factor of production = 1\n",
       "Satellite account = 7\n",
       "Consumption category = 3\n",
       "Region = 245\n",
       "Sector = 133"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = mario.parse_emerging(\n",
    "    path=\"/path/to/2.2\",\n",
    "    table=\"IOT\",\n",
    "    year=2023,\n",
    "    co2_path=\"/path/to/EMERGING_CO2_2023.mat\",\n",
    ")\n",
    "\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec663be5",
   "metadata": {},
   "source": [
    "## Caveats\n",
    "\n",
    "- EMERGING parsing currently supports only `IOT` tables;\n",
    "- local `v2.x` file names do not identify the exact official sub-version;\n",
    "- `regions=` is often necessary to keep the database manageable;\n",
    "- `load_co2=False` is useful when you want to parse the core IOT first and deal with extensions separately.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2896bab0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'IOT'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.table_type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "88a250e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ABW', 'AFG', 'AGO', 'AIA', 'ALB', 'AND', 'ANT', 'BES', 'CUW', 'ARE']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.get_index(\"Region\")[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3cfa1b26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Coal',\n",
       " 'Natural gas',\n",
       " 'Oil products',\n",
       " 'Crude, NGL, Ref Feeds.',\n",
       " 'Other',\n",
       " 'Oil shale & oil sands',\n",
       " 'Peat & Peat products']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.get_index(\"Satellite account\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5a7cfdf1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'EMERGING MATLAB bundle via Zenodo; (concept DOI https://doi.org/10.5281/zenodo.10956622); Huo, J., Chen, P., Hubacek, K., Zheng, H., Meng, J., & Guan, D. (2022). Full-scale, near real-time multi-regional input-output table for the global emerging economies (EMERGING). Journal of Industrial Ecology, 26, 1218–1232. https://doi.org/10.1111/jiec.13264'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.meta.source"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mario",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
