Metadata-Version: 2.4
Name: adls-pandas-utils
Version: 0.0.15
Summary: Package that simplifies working with parquet files stored in Azure Data Lake Storage Gen2 in pandas.
Author-email: Niels Teunissen <n.teunissen@datalier.nl>, Luuk Tijssen <l.tijssen@datalier.nl>
License: Proprietary
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: pandas==3.0.5
Requires-Dist: azure-storage-blob==12.30.0
Requires-Dist: pyarrow==25.0.1
Requires-Dist: azure-storage-file-datalake==12.25.0
Requires-Dist: fastparquet==2026.5.0

# adls-pandas-utils

Package that simplifies working with parquet files stored in Azure Data Lake Storage Gen2 (ADLS Gen2) from pandas. Developed by Datalier for use in data pipelines (bronze/silver processing and Power BI source data).

**Authors:** Niels Teunissen, Luuk Tijssen | Datalier

## Installation

Install the package from the Azure DevOps Artifacts feed:

```bash
pip install adls-pandas-utils --index-url https://pkgs.dev.azure.com/<organization>/_packaging/<feed-name>/pypi/simple/
```

Requires Python **3.12+**. Main dependencies: `pandas`, `pyarrow`, `azure-storage-blob`, `azure-storage-file-datalake`.

## Modules

| Module | Purpose |
| --- | --- |
| `blob_helpers` | Reading and writing parquet files in ADLS Gen2 as pandas DataFrames (documented in this README) |
| `full_incr_load` | Full/incremental load patterns (load silver curated data, combine new + existing, move processed files) |
| `utils` | `BearerTokenCredential` (wraps an access token as a `TokenCredential`) and `strtobool` |

---

## `blob_helpers` documentation

### Public functions

#### `combine_files_in_directory_as_dataframe(path_to_files, container_name, access_token, sa_name) -> pd.DataFrame | None`

Reads **all parquet files** in a directory within an ADLS Gen2 container and combines them into a single DataFrame (`pd.concat`, index is reset).

| Parameter | Type | Description |
| --- | --- | --- |
| `path_to_files` | `str` | Path within the container where the parquet files are stored |
| `container_name` | `str` | Name of the container (file system) in ADLS Gen2 |
| `access_token` | `str` | Service principal access token for the storage account |
| `sa_name` | `str` | Name of the storage account |

**Returns:** a single combined `pd.DataFrame`, or `None` when there are no files or reading fails (the error is printed, not raised).

```python
from adls_pandas_utils.blob_helpers import combine_files_in_directory_as_dataframe

df = combine_files_in_directory_as_dataframe(
    path_to_files="silver/new/",
    container_name="datalake",
    access_token=token,
    sa_name="mystorageaccount",
)
```

### Internal functions (`_` prefix)

These functions are used internally, but can also be called directly from pipelines.

#### `_dataframe_to_adls_gen2_parquet_file(df, access_token, destination_container, blob_directory, blob_name, sa_name) -> dict`

Writes a DataFrame as a parquet file to ADLS Gen2. An existing blob with the same name is deleted first (overwrite behavior).

Before writing, the DataFrame is preprocessed so the parquet file stays **Power BI-compatible** under the pyarrow engine (see the pipeline below).

| Parameter | Type | Description |
| --- | --- | --- |
| `df` | `pd.DataFrame` | The DataFrame to upload |
| `access_token` | `str` | Service principal access token |
| `destination_container` | `str` | Destination container in ADLS Gen2 |
| `blob_directory` | `str` | Directory path within the container (including trailing `/`) |
| `blob_name` | `str` | File name of the blob |
| `sa_name` | `str` | Name of the storage account |

**Returns:** metadata dict from `upload_blob` (e.g. `etag`, `last_modified`).

**Note:** during upload the file is temporarily written to the local working directory under `blob_name` and removed afterwards. Timestamps are written as microseconds (`coerce_timestamps="us"`, truncation allowed).

#### `_read_parquet_file_as_dataframe(file_path, container_system) -> pd.DataFrame`

Downloads a single parquet file from ADLS Gen2 and reads it into a DataFrame (pyarrow engine, via an in-memory buffer).

| Parameter | Type | Description |
| --- | --- | --- |
| `file_path` | `str` | Path to the parquet file within the container |
| `container_system` | `FileSystemClient` | ADLS Gen2 file system client |

#### `_load_file_as_dataframe(file_path, container_system) -> pd.DataFrame`

Wrapper around `_read_parquet_file_as_dataframe` that accepts either a `str` path or a path item from `FileSystemClient.get_paths()` (`ItemPaged`).

### Preprocessing pipeline (Power BI compatibility)

When writing via `_dataframe_to_adls_gen2_parquet_file`, columns are coerced in this order so pyarrow can serialize them and Power BI's parquet connector interprets them correctly:

1. **`_coerce_bool_object_cols`** — object columns containing only `True`/`False`/`None` → nullable `boolean` (parquet BOOLEAN). Runs before the numeric coercion, otherwise booleans would silently become 1/0.
2. **`_coerce_date_object_cols`** — date-like object/string columns → `datetime64`. Recognizes `datetime.date`/`datetime.datetime` instances as well as date strings (`YYYY-MM-DD`, `YYYY/MM/DD`, `DD-MM-YYYY`, `DD/MM/YYYY`, ISO timestamps). Ambiguous formats are interpreted **dayfirst** (Dutch/EU); blank strings and unparseable values become `NaT`.
3. **`_coerce_numeric_object_cols`** — object columns with `Decimal`/mixed-numeric values or numeric strings → numeric (`pd.to_numeric`). Prevents parquet DECIMAL columns from landing in Power BI as text.
4. **`_datetime_to_us_naive`** — tz-aware datetimes → UTC without timezone, downcast to microseconds (`datetime64[us]`). Power BI's parquet connector fails on tz-aware TIMESTAMP and on TIMESTAMP_NANOS.
5. **`_object_to_str`** — remaining object columns (including all-`None`) → `StringDtype` (parquet STRING), because pyarrow cannot infer a type for mixed or empty object columns.

---

## Build & publish

See [how-to-build-publish.md](how-to-build-publish.md). In short:

```bash
py -m build
py -m twine upload --repository pypi dist/*
```

Always bump the version in `pyproject.toml` first.

## Tests

```bash
pytest --cov=src/adls_pandas_utils --cov-report=term-missing
```
