{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "from_txt-01",
   "metadata": {},
   "source": [
    "# TXT, CSV, and Parquet custom parser walkthrough\n",
    "\n",
    "This walkthrough covers the directory-based parsers for TXT, CSV, and Parquet payloads.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "from_txt-02",
   "metadata": {},
   "source": [
    "## Main Entry Points\n",
    "\n",
    "- `mario.parse_from_txt(...)`\n",
    "- `mario.parse_from_parquet(...)`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "850b213a",
   "metadata": {},
   "source": [
    "## Key arguments and directory layout\n",
    "\n",
    "Use `mario.parse_from_txt(...)` for TXT or CSV payloads and `mario.parse_from_parquet(...)` for Parquet payloads.\n",
    "\n",
    "Key arguments:\n",
    "\n",
    "- `path`: directory containing the files to parse;\n",
    "- `table`: choose `\"IOT\"` or `\"SUT\"`;\n",
    "- `mode`: choose `\"flows\"` or `\"coefficients\"`;\n",
    "- `flat`: set `True` for long-format payloads;\n",
    "- `sep` and `_format`: TXT or CSV only;\n",
    "- `matrix_layouts`: optional semantic declaration for non-standard matrix layouts;\n",
    "- `tech_assumption`: optional `SUT` selector for `IT` or `PT`.\n",
    "\n",
    "Matrix-per-file payloads look like:\n",
    "\n",
    "```text\n",
    "custom_txt_database/\n",
    "├── Z.csv\n",
    "├── Y.csv\n",
    "├── V.csv\n",
    "├── E.csv\n",
    "├── EY.csv\n",
    "└── units.csv\n",
    "```\n",
    "\n",
    "Flat payloads can use one combined `data` file plus `units`, or one flat file per matrix plus `units`. The same directory logic applies to Parquet."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf91a844",
   "metadata": {},
   "source": [
    "## Packaged example directories\n",
    "\n",
    "The parser examples below use exported database folders bundled with the documentation:\n",
    "\n",
    "- [CSV export archive](../../../_static/data/supporting_files/iot_export_csv.zip)\n",
    "- [Parquet export archive](../../../_static/data/supporting_files/iot_export_parquet.zip)\n",
    "\n",
    "Extract each archive locally and point `path` to the inner `flows` or `coefficients` directory shown in the code examples below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "from_txt-03",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mario"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "from_txt-04",
   "metadata": {},
   "source": [
    "## Matrix-per-file TXT or CSV\n",
    "\n",
    "Use `flat=False` for the historical matrix-per-file layout.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "from_txt-05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: txt reading IOT flows from /path/to/MARIO/mario/test/supporting_files/iot_export_csv/flows in matrix mode (csv).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: Reading flows from txt files.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: Reading files finished.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: Investigating possible identifiable errors.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: parsing database finished.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: state payload ready with 6 canonical blocks.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: txt state ready for IOT.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    }
   ],
   "source": [
    "db = mario.parse_from_txt(\n",
    "    path=\"/path/to/iot_export_csv/flows\",\n",
    "    table=\"IOT\",\n",
    "    mode=\"flows\",\n",
    "    _format=\"csv\",\n",
    "    flat=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "from_txt-06",
   "metadata": {},
   "source": [
    "## Flat TXT or CSV\n",
    "\n",
    "Use `flat=True` for long-format payloads. MARIO accepts either one combined `data` file or separate flat files per matrix, as long as `units` is present.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "from_txt-07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: txt reading IOT coefficients from /path/to/MARIO/mario/test/supporting_files/iot_export_csv/coefficients in flat mode (csv).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: reading coefficients from flat txt files.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: state payload ready with 6 canonical blocks.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: txt state ready for IOT.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    }
   ],
   "source": [
    "db = mario.parse_from_txt(\n",
    "    path=\"/path/to/iot_export_csv\",\n",
    "    table=\"IOT\",\n",
    "    mode=\"coefficients\",\n",
    "    _format=\"csv\",\n",
    "    flat=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "from_txt-08",
   "metadata": {},
   "source": [
    "## Flat Parquet\n",
    "\n",
    "The same logic applies to Parquet exports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "from_txt-09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: parquet reading IOT flows from /path/to/MARIO/mario/test/supporting_files/iot_export_parquet/flows in flat mode.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: state payload ready with 6 canonical blocks.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Parser: parquet state ready for IOT.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Metadata: initialized.\n"
     ]
    }
   ],
   "source": [
    "db = mario.parse_from_parquet(\n",
    "    path=\"/path/to/iot_export_parquet\",\n",
    "    table=\"IOT\",\n",
    "    mode=\"flows\",\n",
    "    flat=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c9fa99a",
   "metadata": {},
   "source": [
    "## Reference notes and caveats\n",
    "\n",
    "These parsers are directory-based. `path` must point to one directory, not to one individual file.\n",
    "\n",
    "Use `flat=True` for long-format payloads. For TXT or CSV parsing, `_format` and `sep` matter; for Parquet parsing they do not.\n",
    "\n",
    "The same semantic rules used for custom Excel parsing also apply here: if one IOT layout carries extra semantic levels, declare them through `matrix_layouts=` instead of relying on filename conventions alone.\n",
    "\n",
    "These formats are usually preferable to Excel when the data already comes from a MARIO export or from an automated preprocessing workflow."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mario",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
