{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "title",
   "metadata": {},
   "source": [
    "# FIGARO parser walkthrough\n",
    "\n",
    "This notebook is the practical guide for parsing FIGARO from the Eurostat API in MARIO."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "covers",
   "metadata": {},
   "source": [
    "## What this notebook covers\n",
    "\n",
    "- which Eurostat dataflows MARIO uses;\n",
    "- how `SUT` and `IOT` parsing differ;\n",
    "- how `year=`, `countries=`, and `iot_mode=` control the request;\n",
    "- why no local FIGARO path is needed anymore."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sources",
   "metadata": {},
   "source": [
    "## Source links\n",
    "\n",
    "- Eurostat Statistics API: https://ec.europa.eu/eurostat/web/user-guides/data-browser/api-data-access/api-introduction\n",
    "- FIGARO data browser folder: https://ec.europa.eu/eurostat/databrowser/explore/all/naio?lang=en&subtheme=naio.naio_10.naio_10_fcp&display=list&sort=category\n",
    "\n",
    "MARIO downloads FIGARO through the Eurostat API. The old local flat-file `path` workflow is deprecated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dataflows",
   "metadata": {},
   "source": [
    "## Dataflow groups\n",
    "\n",
    "The parser selects the dataflow suffix from `year`:\n",
    "\n",
    "- `2010`-`2013`: `s1`, `u1`, `ip1`, `ii1`\n",
    "- `2014`-`2017`: `s2`, `u2`, `ip2`, `ii2`\n",
    "- `2018`-`2021`: `s3`, `u3`, `ip3`, `ii3`\n",
    "- `2022` onwards: `s4`, `u4`, `ip4`, `ii4`\n",
    "\n",
    "For example, a 2022 SUT uses `naio_10_fcp_s4` and `naio_10_fcp_u4`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "key-arguments",
   "metadata": {},
   "source": [
    "## Key arguments\n",
    "\n",
    "- `table`: choose `\"SUT\"` or `\"IOT\"`.\n",
    "- `year`: required.\n",
    "- `countries`: optional list of FIGARO country codes. Use it while developing to avoid downloading the full table.\n",
    "- `iot_mode`: only for `table=\"IOT\"`; use `\"auto\"`, `\"product\"`, or `\"industry\"`.\n",
    "- `unit`: currently `\"MIO_EUR\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ea216b6",
   "metadata": {},
   "source": [
    "## API notes and caveats\n",
    "\n",
    "FIGARO is fully API-based. The old `path` argument is deprecated and ignored.\n",
    "\n",
    "MARIO chunks requests by `c_orig` to avoid Eurostat extraction-size limits and then rebuilds the MARIO matrices locally. For `SUT`, `DOM` rows in the use table are interpreted as value-added rows. For `IOT`, `DOM` rows are interpreted in the same way for the `V` matrix.\n",
    "\n",
    "Full FIGARO payloads are large, so `countries=` is the practical way to keep exploratory runs small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "import-mario",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mario"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sut",
   "metadata": {},
   "source": [
    "## Parse a FIGARO SUT\n",
    "\n",
    "The example below limits the request full FIGARO table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "sut-code",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading cached FIGARO API payload from /Users/lorenzorinaldi/Library/CloudStorage/OneDrive-SharedLibraries-PolitecnicodiMilano/DENG-SESAM - Documenti/c-Research/a-Datasets/_Input Output Databases/FIGARO/SUT/naio_10_fcp_s4_2022_MIO_EUR_8ae0a3260d8f.csv.\n",
      "INFO Parser: reading cached FIGARO API payload from /Users/lorenzorinaldi/Library/CloudStorage/OneDrive-SharedLibraries-PolitecnicodiMilano/DENG-SESAM - Documenti/c-Research/a-Datasets/_Input Output Databases/FIGARO/SUT/naio_10_fcp_u4_2022_MIO_EUR_8be2fc37e8a5.csv.\n",
      "INFO Parser: FIGARO SUT parsed from Eurostat API with 50 regions, 64 activities, 64 commodities, 6 factor rows.\n",
      "INFO Metadata: initialized.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name = FIGARO SUT 2022\n",
       "table = SUT\n",
       "tech_assumption = industry-based\n",
       "scenarios = ['baseline']\n",
       "Activity = 64\n",
       "Commodity = 64\n",
       "Factor of production = 6\n",
       "Satellite account = 1\n",
       "Consumption category = 5\n",
       "Region = 50"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_sut = mario.parse_figaro(\n",
    "    table=\"SUT\",\n",
    "    year=2022,\n",
    "    # path=\"/path/to/figaro/sut_directory\",\n",
    "    path='/Users/lorenzorinaldi/Library/CloudStorage/OneDrive-SharedLibraries-PolitecnicodiMilano/DENG-SESAM - Documenti/c-Research/a-Datasets/_Input Output Databases/FIGARO/SUT'\n",
    ")\n",
    "\n",
    "db_sut"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8387427e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name = FIGARO SUT 2022\n",
       "table = SUT\n",
       "tech_assumption = industry-based\n",
       "scenarios = ['baseline']\n",
       "Activity = 64\n",
       "Commodity = 64\n",
       "Factor of production = 6\n",
       "Satellite account = 1\n",
       "Consumption category = 5\n",
       "Region = 50"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_sut"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "iot",
   "metadata": {},
   "source": [
    "## Parse a FIGARO IOT\n",
    "\n",
    "For `IOT` parsing, `iot_mode=\"auto\"` defaults to the product-by-product dataflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "iot-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "from importlib.resources import path\n",
    "\n",
    "\n",
    "db_iot = mario.parse_figaro(\n",
    "    table=\"IOT\",\n",
    "    year=2022,\n",
    "    iot_mode=\"product\",\n",
    "    path=\"/path/to/figaro/iot_directory\",\n",
    ")\n",
    "\n",
    "db_iot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "inspect",
   "metadata": {},
   "source": [
    "## First inspection\n",
    "\n",
    "Once parsed, the result is a standard MARIO database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "inspect-code",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Belgium', 'Bulgaria', 'Czechia', 'Denmark', 'Germany']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_sut.get_index(\"Region\")[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2730933e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Crop and animal production, hunting and related service activities',\n",
       " 'Forestry and logging',\n",
       " 'Fishing and aquaculture',\n",
       " 'Mining and quarrying',\n",
       " 'Manufacture of food products; beverages and tobacco products']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_sut.get_index(\"Activity\")[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8b91f880",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Products of agriculture, hunting and related services',\n",
       " 'Products of forestry, logging and related services',\n",
       " 'Fish and other fishing products; aquaculture products; support services to fishing',\n",
       " 'Mining and quarrying',\n",
       " 'Food, beverages and tobacco products']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_sut.get_index(\"Commodity\")[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "warnings",
   "metadata": {},
   "source": [
    "## Practical warnings\n",
    "\n",
    "- Full FIGARO tables are large to be downloaded; use `countries=` while developing workflows.\n",
    "- MARIO chunks Eurostat API requests by `c_orig` internally to avoid extraction-size limits.\n",
    "- The old local `path` argument is deprecated and ignored."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mario",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
