Metadata-Version: 2.5
Name: f9columnar
Version: 0.8.0
Summary: Columnar analysis utils for high-energy physics data analysis.
Project-URL: Repository, https://gitlab.cern.ch/ijs-f9-ljubljana/f9columnar
Author-email: Jan Gavranovic <jan.gavranovic@cern.ch>
License: MIT
License-File: LICENSE
Keywords: analysis,columnar,data-analysis,hep,physics
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: <3.15,>=3.11
Requires-Dist: act-client
Requires-Dist: aiohttp<4,>=3.13.3
Requires-Dist: awkward<3,>=2.13.0
Requires-Dist: corner<3,>=2.2.2
Requires-Dist: dill<1,>=0.4.1
Requires-Dist: flask<4,>=3.1.2
Requires-Dist: h5py<4,>=3.11.0
Requires-Dist: hdf5plugin<7,>=6.0.0
Requires-Dist: hist[plot]<3,>=2.7.3
Requires-Dist: imbalanced-learn<1,>=0.14.1
Requires-Dist: lightning<3,>=2.5.3
Requires-Dist: matplotlib<4,>=3.9.2
Requires-Dist: mplhep<1,>=0.3.50
Requires-Dist: networkx<4,>=3.3
Requires-Dist: numba<1,>=0.60.0
Requires-Dist: numpy<3,>=2.0.2
Requires-Dist: pandas<3,>=2.2.2
Requires-Dist: plothist<2,>=1.2.6
Requires-Dist: psutil<7,>=6.1.0
Requires-Dist: pyarrow<24,>=17.0.0
Requires-Dist: pydot<4,>=3.0.1
Requires-Dist: requests<3,>=2.28.1
Requires-Dist: rich<14,>=13.7.0
Requires-Dist: scikit-learn<1.9,>=1.8.0
Requires-Dist: scipy<2,>=1.14.1
Requires-Dist: seaborn<1,>=0.13.2
Requires-Dist: sympy<2,>=1.13.1
Requires-Dist: tables<4,>=3.10.1
Requires-Dist: tqdm<5,>=4.65.0
Requires-Dist: uncertainties<4,>=3.2.2
Requires-Dist: uproot<6,>=5.7.6
Requires-Dist: vector<2,>=1.4.1
Provides-Extra: torch-cpu
Requires-Dist: torch<2.14,>=2.9.0; extra == 'torch-cpu'
Provides-Extra: torch-cuda
Requires-Dist: torch<2.14,>=2.9.0; extra == 'torch-cuda'
Description-Content-Type: text/markdown

# F9 Columnar

<div align="center">

[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![python](https://img.shields.io/badge/-Python_3.11+-3776AB?logo=python&logoColor=white)](https://www.python.org/)
[![pytorch](https://img.shields.io/badge/-PyTorch_2.9+-EE4C2C?logo=pytorch&logoColor=white)](https://pytorch.org/)
[![License](https://img.shields.io/badge/license-MIT-green)](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/LICENSE)
[![pypi](https://img.shields.io/pypi/v/f9columnar)](https://pypi.org/project/f9columnar/)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19312298.svg)](https://doi.org/10.5281/zenodo.19312298)

</div>

A PyTorch-based library for processing event data in high energy physics for ML applications.

### Project description

F9Columnar is a Python library designed to streamline the processing of physics datasets (ROOT, HDF5, parquet) for machine learning. It is built on top of PyTorch, Uproot and Awkward Arrays. It provides a modular framework for efficient data loading, transformation, and analysis in high-energy physics research.

The library bridges the gap between traditional high-energy physics data formats and modern machine learning workflows. It enables physicists and data scientists to:

- Process large datasets in many formats using PyTorch DataLoaders
- Apply event selections, calculate variables, and create histograms
- Convert data to piles, HDF5 files with events prepared for fast random-access reads during ML training

The library is designed to be modular and extensible, allowing users to easily add new processors and functionality as needed. A common workflow is illustrated in the diagram below:

<div align="center">

![F9Columnar](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/raw/main/docs/f9columnar_flow.svg)

</div>

Check out the [PyHEP 2025 talk](https://indico.cern.ch/event/1566263/timetable/?view=standard#14-from-root-to-pytorch-seamle) for a good overview of what the library can do.

## Setup

The library comes without any PyTorch dependencies by default to allow flexible installations. You can install it with or without PyTorch GPU support depending on your needs.

### Install with PyTorch for GPU

```shell
pip install f9columnar
pip install torch
```

### Install with PyTorch for CPU

```shell
pip install f9columnar
pip install torch --index-url https://download.pytorch.org/whl/cpu
```

### Install for development

```shell
git clone https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar.git
cd F9Columnar
uv sync
```

Run the tests with `uv run pytest`.

## Getting started

The [examples](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/tree/main/examples) folder walks through the library step by step. The examples are self-contained, you do not need real data. Example 1 generates dummy ROOT files that the other examples use. The last example is the exception, it shows the grid workflow and needs a grid environment to run.

1. [Making dummy ROOT inputs](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example1.py)
2. [Basic data loading and histogramming with awkward arrays](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example2.py)
3. [Data loading and processing with a processor graph](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example3.py)
4. [Converting ROOT files into HDF5 piles used for ML training](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example4.py)
5. [One ML training loader from two sources](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example5.py)
6. A Lightning training loop over [partitioned ROOT files](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example6a.py) and over [HDF5 piles](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example6b.py)
7. [Inference on the trained model with the inference server](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example7.py)
8. [Feature scaling with fitted scalers as torch modules](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example8.py)
9. [Rucio ntuples and batch processing on the Vega HPC with aCT](https://gitlab.cern.ch/ijs-f9-ljubljana/F9Columnar/-/blob/main/examples/example9.py)

## Projects using F9Columnar

- [SeeSawML](https://gitlab.cern.ch/ijs-f9-ljubljana/SeeSawML): multi-dataset, multi-modal and multi-task ML training framework.

## Citation

If you found F9Columnar useful, please consider citing its Zenodo record [10.5281/zenodo.19312298](https://zenodo.org/records/19312298).

You can find an example bibtex entry below.
```
@software{F9Columnar,
  author       = {Gavranovic, Jan},
  title        = {F9Columnar: A PyTorch-Based Library for High Energy Physics Data Processing and Analysis},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.19312298},
  url          = {https://doi.org/10.5281/zenodo.19312298},
}
```
