Metadata-Version: 2.4
Name: xllim
Version: 2.0.11
Summary: xLLiM is a C++ with Python bindings implementation of Gaussian Locally-Linear Mapping.
Author-Email: Luc Meyer <luc.meyer@inria.fr>
Maintainer-Email: xLLiM Team <xllim-contact@inria.fr>
License-Expression: CECILL-C
License-File: LICENCE.txt
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Project-URL: Repository, https://github.com/xllim-tools/xllim
Project-URL: Documentation, https://xllim-tools.github.io/xllim/
Project-URL: Issues, https://github.com/xllim-tools/xllim/issues
Requires-Python: >=3.10
Requires-Dist: numpy<3,>=2.0
Requires-Dist: h5py<4,>=3.8
Description-Content-Type: text/markdown


# xLLiM

`xLLiM` is a C++ with Python bindings implementation of Gaussian Locally-Linear Mapping.
It can be used to perform efficient inversion of high-dimensional models. 
`xLLiM` integrates features such as:
* Forward model functionals. These can be implemented as C++ or as pure Python functions. The forward model can be used to generate data following a distribution and refine GLLiM results by sampling the PDF using various strategies.
* Multi-initialization options
* Post-GLLiM refinement methods such as Importance Sampling (IS) and Iterative Mixture Importance Sampling (IMIS)
* Post-processing analysis, including confidence quantification on predictions, detection of multiple solutions, and permutation of predictions in case of signal regularity.
Currently, `xLLiM` supports only Gaussian probability distributions, but other distributions may be added to it in the future.

`xLLiM` is distributed as a compiled shared library and can be installed via conda, pip, or as a Docker container. It is integrated in the [Planet-GLLiM](https://gitlab.inria.fr/xllim/planet-gllim) astrophysics application, that is distributed as a Docker image for local use and as a data processing service
on [Allgo-18](https://allgo18.inria.fr/).

> **Repository migration:** `xLLiM` has moved to GitHub. The current repository is [https://github.com/xllim-tools/xllim](https://github.com/xllim-tools/xllim). The former GitLab repository ([https://gitlab.inria.fr/xllim/xllim](https://gitlab.inria.fr/xllim/xllim)) is no longer maintained.

The `xLLiM` method was previously implemented in `R` and is available on [CRAN](https://cran.r-project.org/web/packages/xLLiM/index.html).
Other implementations of GLLiM also exist:
- A pure Python implementation: [pyGLLiM](https://github.com/Chutlhu/pyGLLiM)
- A Julia implementation with IS and IMIS: [Fast Bayesian Inversion](https://gitlab.inria.fr/bkugler/fastbayesianinversion)

This `xLLiM` implementation is derived from [Kernelo](https://gitlab.inria.fr/kernelo-mistis/kernelo-gllim-is).
`Kernelo` is obsolete and is no longer maintained.

## Table of Contents

- [Documentation and API reference](#documentation-and-api-reference)
- [Runtime Dependencies](#runtime-dependencies)
  - [Mandatory](#mandatory)
  - [Optional](#optional)
- [Platform support](#platform-support)
- [xLLiM installation options](#xllim-installation-options)
  - [1. Conda-forge (recommended)](#1-conda-forge-recommended)
    - [Installing a conda distribution](#installing-a-conda-distribution)
    - [Installing xLLiM](#installing-xllim)
  - [2. PyPI](#2-pypi)
  - [3. Docker / Apptainer (Singularity)](#3-docker--apptainer-singularity)
    - [Docker](#docker)
    - [Apptainer / Singularity (HPC)](#apptainer--singularity-hpc)
  - [4. Manual installation (optional)](#4-manual-installation-optional)
- [Using xLLiM](#using-xllim)
  - [Python API vs Command-Line Interface](#python-api-vs-command-line-interface)
  - [Quick start with the Python API](#quick-start-with-the-python-api)
  - [Command-Line Interface (xllim_cli)](#command-line-interface-xllim_cli)
- [Further documentation](#further-documentation)
- [Licence](#licence)

# Documentation and API reference

The API reference of the xLLiM module is available at [xLLiM API reference](https://xllim-tools.github.io/xllim/). 
For more information you can find a complete scientific documentation in the [Planet-GLLiM documentation](https://xllim.gitlabpages.inria.fr/planet-gllim/).

# Runtime Dependencies

## Mandatory:
These packages are required for the core functionality of the library.
| Name                 | version           |
|----------------------|-------------------|
| Python               | >=3.11, < 3.13    |
| Numpy                | >=2.0, < 3        |
| h5py                 | >=3.8, < 4        |

For other python versions, see [Other python versions](#other-python-versions).

## Optional:

### ENVI / GDAL Support
xllim requires GDAL specifically for ENVI hyperspectral file support. Because GDAL depends on native system libraries, installation can be complex.

#### Recommended (Most reliable)
Using Conda is the simplest way to handle linked C libraries:
```bash
conda install gdal
```

#### Alternative: pip + system package
If installing via apt or brew, the Python GDAL version must match the system GDAL version.
If the versions do not match, the installation will fail.

Example on Ubuntu:

1. Install system libraries
```bash
sudo apt install libgdal-dev gdal-bin
```
2. Check the system version
```bash
gdal-config --version
```
3. Install the matching python wrapper (e.g., if version is 3.8.4)
```bash
pip install "gdal==3.8.4"
```

# Platform support

The table below summarises which platforms and architectures have pre-built packages available for each distribution method.

| Platform | pip | conda |
|---|---|---|
| Linux x86_64 | ✅ | ✅ |
| Linux aarch64 (ARM) | — | ✅ |
| macOS x86_64 | — | ✅ |
| macOS arm64 (Apple Silicon) | — | ✅ |
| Windows x86_64 | — | ✅ |

— = no pre-built package; source build may be possible (see [Section 4](#4-manual-installation-optional)).

**Docker and Apptainer**: the container image is built for Linux x86_64, but containers are designed to run on any host OS (Linux, macOS, or Windows) without requiring any native installation. If you are on macOS or Windows and do not want to manage a conda environment, this is the simplest path. See [Section 3](#3-docker--apptainer-singularity) for details.

# xLLiM installation options

`xLLiM` can be installed and used in several ways, depending on your environment and preference.

## 1. Conda-forge (recommended)
The easiest way to install `xLLiM` is via a conda-compatible package manager. `xLLiM` is published on [conda-forge](https://conda-forge.org/), a community-maintained channel that provides up-to-date, cross-platform packages. The corresponding [xllim-feedstock](https://github.com/conda-forge/xllim-feedstock) contains the conda packaging recipe and tracks each release. That is where the conda recipe, managing xllim conda builds, should be updated if needed (refer to xllim-feedstock README for instructions).

### Installing a conda distribution

If you don't already have conda installed, several distributions are available. They all provide the `conda` command (or a drop-in equivalent) and can install packages from conda-forge:

| Distribution | Description | Docs |
|---|---|---|
| **Micromamba** | Standalone, fast C++ reimplementation of conda. No base environment, no Python required. Defaults to conda-forge. | [mamba.readthedocs.io](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html) |
| **Miniforge** | Community-maintained minimal installer. Defaults to conda-forge. | [github.com/conda-forge/miniforge](https://github.com/conda-forge/miniforge) |
| **Miniconda** | Minimal installer from Anaconda. Defaults to the `defaults` channel (see warning below). | [docs.conda.io](https://docs.conda.io/en/latest/miniconda.html) |
| **Anaconda** | Full-featured distribution with many pre-installed packages and a GUI. Defaults to `defaults` channel (see warning below). Heavier download. | [docs.anaconda.com](https://docs.anaconda.com/anaconda/install/) |

> **Recommended: Micromamba or Miniforge.** Both default to conda-forge exclusively, so environments are clean and consistent. They are also significantly faster than the classic `conda` solver.

#### Example: installing Micromamba on Linux

```bash
"${SHELL}" <(curl -L micro.mamba.pm/install.sh)
```

This installs the `micromamba` binary and adds a shell initialisation block to your shell config. Restart your shell (or `source ~/.bashrc`), then verify:

```bash
micromamba --version
```

> On macOS the same command works. On Windows, see the [Micromamba installation docs](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).

### Installing xLLiM

Once you have a conda distribution available, create and activate an environment (`<my-env>` is a placeholder):

```bash
# With conda or mamba
conda create -n <my-env>
conda activate <my-env>

# With micromamba
micromamba create -n <my-env>
micromamba activate <my-env>
```

Then install `xLLiM` (replace `conda` with `micromamba` if applicable). This will also install the required dependencies.
```bash
conda install -c conda-forge xllim
```

> ⚠️ **Anaconda/Miniconda users: channel conflict warning.** Anaconda and Miniconda default to Anaconda's proprietary `defaults` channel, which is **not** the same as conda-forge. Mixing channels (e.g., adding `-c conda-forge` to a `defaults`-based environment) can lead to subtle dependency conflicts and broken environments, because the two channels build packages independently and their ABI guarantees are not always compatible. If you use Anaconda or Miniconda, we strongly recommend creating a **conda-forge-only** environment before installing `xLLiM`:
> ```bash
> conda create -n <my-env> -c conda-forge --override-channels
> conda activate <my-env>
> conda config --env --add channels conda-forge
> conda config --env --set channel_priority strict
> ```
> See the [conda-forge documentation on channel conflicts](https://mamba.readthedocs.io/en/latest/user_guide/troubleshooting.html#defaults-channels) for more details.

## 2. PyPI

#### Available wheels

We've built and tested wheels for **Linux x86_64** (manylinux_2_28) - **Python 3.11** and **3.12**. If you're in one of these configurations, you can install xllim easily with:

```bash
pip install xllim
```

#### Mac/Windows

If you're on Mac or Windows, and are fine with Python 3.11/12, we recommend you either:
- install with conda ([Section 1.](#1-conda-forge-recommended))
- use our pre-built docker image ([Section 3.](#3-docker))

#### Other Python versions

If you want to use a different Python version (3.10, 3.13, 3.14...):
They've not been tested as some of `xLLiM` dependencies exclude these versions. It might be possible though. Then you may try:
- Building from the sdist (```pip install xllim```): when no pre-built wheel matches your Python version or platform, pip automatically falls back to downloading the source distribution and compiling the C++ extension on your machine. The Python-side build tools (scikit-build-core, cmake, ninja, numpy, pybind11) are installed automatically. However, you must first install the native system libraries manually (compiler, Armadillo, a BLAS/LAPACK implementation, Boost, carma - see [Section 4.](#4-manual-installation-optional) for details).
.
- Compiling, building and installing from scratch (advanced users) ([Section 4.](#4-manual-installation-optional))


## 3. Docker / Apptainer (Singularity)

Pre-built container images are published for every release. Two formats are available: a standard **Docker** image for workstations and cloud environments, and an **Apptainer / Singularity** SIF image for HPC clusters where Docker is unavailable.

### Docker

A minimal Docker image based on `python:3.11-slim` is available, with `xLLiM` pre-installed.

**Prerequisites:** install [Docker Engine](https://docs.docker.com/engine/install/) (Linux/macOS) or [Docker Desktop](https://docs.docker.com/desktop/windows/install/) (Windows).

Pull the image:

```bash
docker pull ghcr.io/xllim-tools/xllim/xllim:latest
```
> Note: specific versions are also available. See [xLLiM GHCR](https://github.com/xllim-tools/xllim/pkgs/container/xllim%2Fxllim/versions).

Run your project by mounting your source directory:

```bash
docker run -it \
  -v $(pwd):/workspace \
  -w /workspace \
  ghcr.io/xllim-tools/xllim/xllim:latest \
  python main.py
```

You can also open an interactive shell:

```bash
docker run -it -v $(pwd):/workspace -w /workspace ghcr.io/xllim-tools/xllim/xllim:latest bash
```

### Apptainer / Singularity (HPC)

[Apptainer](https://apptainer.org/) (formerly Singularity) is a container runtime designed for HPC environments where Docker is typically not available or not allowed. It runs rootless and is natively supported on most HPC schedulers (SLURM, PBS, etc.).

For every release, a **SIF** (Singularity Image Format) image is automatically built from the Docker image and published to the same GHCR registry with a `-sif` tag suffix.

**Prerequisite:** [Apptainer](https://apptainer.org/docs/user/latest/quick_start.html) must be installed on your system (typically already available on HPC clusters - check with your sysadmin).

Pull the latest SIF image:

```bash
apptainer pull xllim.sif oras://ghcr.io/xllim-tools/xllim/xllim:latest-sif
```

Or pull a specific version (replace `<version>` with the desired release tag):

```bash
apptainer pull xllim.sif oras://ghcr.io/xllim-tools/xllim/xllim:<version>-sif
```
> Available versions: [xLLiM GHCR](https://github.com/xllim-tools/xllim/pkgs/container/xllim%2Fxllim/versions).

Run your project by binding your working directory:

```bash
apptainer run -B $(pwd):/workspace --pwd /workspace xllim.sif python main.py
```

Or open an interactive shell:

```bash
apptainer shell -B $(pwd):/workspace xllim.sif
```

## 4. Manual installation (optional)
If you need to compile `xLLiM` from source (e.g., for an unsupported platform or Python version, or for development), please refer to the requirements below.

### Build Dependencies

These are required only if you are compiling the library from source.
| Name                 | Version           | Notes                                                              |
|----------------------|-------------------|--------------------------------------------------------------------|
| C++ compiler         | C++17 support     | System-installed (e.g., g++ >= 9, clang++ >= 5)                    |
| CMake                | >= 3.21           | Auto-installed by pip when using `pip install .`                   |
| Ninja                | any               | Auto-installed by pip when using `pip install .`                   |
| scikit-build-core    | >= 0.7.0, < 1     | Auto-installed by pip when using `pip install .`                   |
| Python               | >= 3.11, < 3.13   | System-installed; see [Other Python versions](#other-python-versions) for other versions |
| Numpy                | >= 2              | Auto-installed by pip when using `pip install .`                   |
| Pybind11             | >= 2.12           | Auto-installed by pip when using `pip install .`                   |
| Armadillo            | >= 12.6, < 13     | System-installed                                                   |
| Boost                | >= 1.78, < 2      | System-installed; components: `system`, `thread`, `random`         |
| BLAS / LAPACK        | any               | System-installed; any conforming implementation is accepted (OpenBLAS, MKL, Apple Accelerate…) |
| Carma                | >= 0.8.0          | System-installed (see installation notes below)                    |

### Build from source

Since `xLLiM` contains C++ extensions with Python bindings, building from source requires native system libraries that pip cannot install automatically. The instructions below use a Debian/Ubuntu-based system as example - adapt package names for your distribution.

1. Clone the project
```bash
git clone https://github.com/xllim-tools/xllim.git
cd xllim
```

2. Install system dependencies
```bash
sudo apt update

# Compilation tools
sudo apt-get install -y --no-install-recommends g++ cmake ninja-build

# BLAS/LAPACK - OpenBLAS is used here as an example; any conforming implementation works (MKL, etc.)
sudo apt-get install -y --no-install-recommends libopenblas-dev liblapack-dev

# Armadillo
sudo apt-get install -y --no-install-recommends libarmadillo-dev

# Python development headers
sudo apt-get install -y --no-install-recommends python3-dev

# Boost (components: system, thread, random)
sudo apt-get install -y --no-install-recommends libboost-dev libboost-system-dev libboost-thread-dev libboost-random-dev
```

> ⚠️ **Boost version:** `xLLiM` requires Boost >= 1.78.0 (enforced by CMake). Ubuntu 24.04+ ships a compatible version out of the box. On older distributions (e.g., Ubuntu 22.04 ships 1.74), you will need to build Boost >= 1.78.0 from source:
> ```bash
> curl -L https://archives.boost.io/release/1.78.0/source/boost_1_78_0.tar.gz -o boost.tar.gz
> tar -xzf boost.tar.gz
> cd boost_1_78_0
> ./bootstrap.sh --prefix=/usr/local
> ./b2 --with-system --with-thread --with-random link=static variant=release -j$(nproc) install
> cd .. && rm -rf boost_1_78_0 boost.tar.gz
> ```
> After building Boost to `/usr/local`, you must tell CMake where to find it by setting the `CMAKE_ARGS` environment variable before calling `pip install .`:
> ```bash
> export CMAKE_ARGS="-DBoost_ROOT=/usr/local -DBoost_NO_SYSTEM_PATHS=ON -DBoost_USE_STATIC_LIBS=ON"
> ```

**carma** - not available in standard apt repositories; build from source:
```bash
curl -L https://github.com/RUrlus/carma/archive/refs/tags/v0.8.0.tar.gz -o carma.tar.gz
tar -xzf carma.tar.gz
cd carma-0.8.0 && mkdir build && cd build
cmake -DCARMA_INSTALL_LIB=ON ..
cmake --build . --config Release --target install
cd ../..
rm -rf carma-0.8.0 carma.tar.gz
```

> **Tip:** if you are using conda, all of the above can be installed in one step:
> ```bash
> conda install armadillo boost libblas liblapack carma
> ```
> This pulls OpenBLAS by default; substitute a different `blas` build string (e.g., `libblas * *mkl`) to use an alternative BLAS implementation.

3. Build and install
```bash
pip install .
```
This uses [scikit-build-core](https://scikit-build-core.readthedocs.io/) to drive the CMake build automatically. The Python-side build tools (`scikit-build-core`, `cmake`, `ninja`, `numpy`, `pybind11`) are fetched by pip as needed.

4. Verify the installation
```
python3
>>> import xllim
```

#### Manual CMake build (for development)

For C++ development or debugging, you can drive CMake directly:
```
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DXLLIM_BUILD_TESTS=ON
cmake --build build
ctest --test-dir build
```
The build type can be either *Debug* (`-O0 -g`) or *Release* (`-O2 -DNDEBUG`). To install the compiled module into your Python environment, use `pip install .` as described above.

# Using xLLiM

`xLLiM` can be used in two ways: as a **Python library** (`import xllim`) for full programmatic control, or via the **command-line interface** (`xllim_cli`) for a guided, file-based workflow. Both interfaces use the same underlying C++ engine.

## Python API vs Command-Line Interface

`xLLiM` exposes two interfaces that address different needs, and they can be used together.

The **Python API** (`import xllim`) gives you direct access to all objects and intermediate results. You pass parameters as function arguments, work with NumPy arrays in memory, and have full control over the workflow. It is the natural choice for Jupyter notebooks, custom scripts, or integration into a larger pipeline.

The **CLI** (`xllim_cli`) provides a guided, file-based workflow: all configuration, training data, and trained model parameters are stored in a single HDF5 file. You configure it once via interactive prompts, then run `generate`, `train`, and `predict` as separate commands. The resulting `.h5` file is a self-contained, shareable experiment record, useful for reproducibility or when you want to run the full pipeline without writing code.

## Quick start with the Python API

The [`examples/`](examples/) directory contains two Jupyter notebooks that walk you through the full workflow:

- **[`example.ipynb`](examples/example.ipynb)**: Complete walkthrough using the basic `TestModel`: data generation, GLLiM training, prediction, importance sampling, and post-processing analysis with plots.
- **[`example_hapke.ipynb`](examples/example_hapke.ipynb)**: Same workflow applied to the `HapkeModel` photometric model using real BRDF geometry data.

For details on array shapes, dimension conventions (`K`, `D`, `L`, `N`, `L_t`, `L_w`), and parameter matrices (`Pi`, `A`, `B`, `C`, `Gamma`, `Sigma`), see the [Useful information](https://xllim-tools.github.io/xllim/getting_started/useful_info.html) page in the documentation.

## Command-Line Interface (`xllim_cli`)

The CLI provides a guided, no-code workflow where all configuration, training data, and trained model parameters are stored in a single **HDF5 file**.

After installing `xllim`, the `xllim_cli` command is available directly in your terminal:

```bash
xllim_cli --help
```

> **Windows with conda:** on Windows, the `xllim_cli` console script may not work when xLLiM was installed via conda (the launcher embeds a build-time Python path that doesn't exist at runtime). Use `python -m xllim.cli` instead:
> ```bash
> python -m xllim.cli --help
> ```

### Available commands

| Command | Description |
|---------|-------------|
| `edit <model.h5>` | Interactively configure all sections (functional model, generator, GLLiM, prediction, importance sampling, output) |
| `print <model.h5>` | Print a summary of the HDF5 file contents (`-v` for detailed view) |
| `generate <model.h5>` | Generate synthetic training data using the configured functional model and generator |
| `train <model.h5>` | Train the GLLiM model using training data and configuration from the file |
| `predict <model.h5> <observations> <output>` | Run the full prediction workflow (load model, predict, optional IS, write results) |
| `import <type> <source> <model.h5>` | Import `geometries` (JSON/NPZ) or `train_data` (NPZ) into the HDF5 file |
| `export <type> <model.h5> <output>` | Export `geometries`, `train_data`, or `trained_gllim` to NPZ |
| `copy <source.h5> <dest.h5>` | Interactively copy sections between two HDF5 files |

### Typical CLI workflow

```bash
# 1. Create and configure a new model file (interactive prompts)
xllim_cli edit my_model.h5

# 2. (Optional) Import geometries from an existing JSON file
xllim_cli import geometries geometries.json my_model.h5

# 3. Generate training data
xllim_cli generate my_model.h5

# 4. Train the GLLiM model
xllim_cli train my_model.h5

# 5. Inspect the file
xllim_cli print my_model.h5 -v

# 6. Run predictions on observations
xllim_cli predict my_model.h5 observations.json results/ -f npz

# 7. Export trained parameters for use elsewhere
xllim_cli export trained_gllim my_model.h5 trained_params.npz
```

The `edit` command walks through six configuration sections in order:
1. **Functional model**: choose from `Hapke`, `Shkuratov`, `External` (custom Python class), or `Test model`, each with its own sub-options.
2. **Data generator**: dataset size `N`, generator type (`sobol`, `random`, `latin hypercube`), noise covariance, seed.
3. **GLLiM model**: number of components `K`, covariance types (`full`/`diag`/`iso`), initialisation parameters, and training variant (`GLLiM` or `JGMM`).
4. **Importance sampling**: IMIS parameters (`N_zero`, `B`, `J`, covariance, seed).
5. **Prediction module**: merging options (`K_merged`, threshold), IS flags.
6. **Output**: which results to write (fullGMM, mergedGMM, IS variants).

The `predict` command accepts observations in **JSON**, **NPZ**, or **ENVI** format (ENVI requires GDAL, see [Optional dependencies](#envi--gdal-support)). Results are written as NPZ files by default.

# Further documentation

- [API reference](https://xllim-tools.github.io/xllim/): Full module documentation (classes, methods, parameters).
- [Useful information](https://xllim-tools.github.io/xllim/getting_started/useful_info.html):  Dimension conventions, array shapes, and workflow summaries.
- [Planet-GLLiM documentation](https://xllim.gitlabpages.inria.fr/planet-gllim/): Scientific background and application-level documentation.

# Licence
This software is licensed under the GNU GPL-compatible [CeCILL-C License](LICENCE.txt).
While the software is free, we would appreciate it if you send us an email at `xllim-contact@inria.fr` to let us know how you use it.
Also, please contact us if the licence does not meet your needs.
