Metadata-Version: 2.4
Name: mlindex
Version: 0.1.3
Summary: A data driven approach to powder diffraction indexing
Author-email: David Mittan-Moreau <dwmoreau@lbl.gov>
License-Expression: MIT
Project-URL: Homepage, https://github.com/dwmoreau/MLI
Project-URL: Issues, https://github.com/dwmoreau/MLI/issues
Keywords: powder diffraction,diffraction,crystallography,indexing,machine learning
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: cctbx-base
Requires-Dist: huggingface_hub>=0.25
Requires-Dist: matplotlib
Requires-Dist: numpy
Requires-Dist: numba
Requires-Dist: onnxruntime
Requires-Dist: pandas
Requires-Dist: scikit-learn
Requires-Dist: scipy
Provides-Extra: mpi
Requires-Dist: mpi4py; extra == "mpi"
Provides-Extra: dataset-generation
Requires-Dist: gemmi; extra == "dataset-generation"
Requires-Dist: pyarrow; extra == "dataset-generation"
Requires-Dist: openpyxl; extra == "dataset-generation"
Requires-Dist: tqdm; extra == "dataset-generation"
Provides-Extra: training
Requires-Dist: pyarrow; extra == "training"
Requires-Dist: openpyxl; extra == "training"
Requires-Dist: skl2onnx; extra == "training"
Requires-Dist: keras; extra == "training"
Requires-Dist: torch; extra == "training"
Requires-Dist: torchvision; extra == "training"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# MLINDEX - A data driven approach to powder diffraction indexing

A powder diffraction indexing program that uses machine learning models to initialize the SVD-Index algorithm. It takes an input peak list and returns a list of unit cells ranked by Figure of Merit.

> **Note:** This application is in beta stage. Usage and feedback would be greatly appreciated to improve user experience.

## Installation

### Standard installation (pip)

```bash
pip install mlindex
mlindex.download_models
```

`mlindex.download_models` fetches the ML model files (~545 MB) from the [Hugging Face Hub](https://huggingface.co/dwmoreau/mlindex-models) and installs them to `~/.local/share/mlindex/models/`. No git or git-lfs is required. Each mlindex release pins a specific model revision, so you always get the models that version was tested against.

Re-running the command is cheap: files that are already present and up to date are not downloaded again, so an interrupted download can be resumed by simply running it again.

### Installing models somewhere else

The model directory can be customized with `--models-dir` or the `MLINDEX_MODELS_DIR` environment variable. Both must name **the same directory**, and it must be the directory that *directly contains* the model subdirectories `cubic_1/`, `hexagonal_1/`, ...:

```bash
mlindex.download_models --models-dir /path/to/models
export MLINDEX_MODELS_DIR=/path/to/models
```

On Windows:

```
mlindex.download_models --models-dir D:\mlindex-models
set MLINDEX_MODELS_DIR=D:\mlindex-models
```

### Troubleshooting

- **`MLINDEX_MODELS_DIR=... does not look like a models directory`** — the variable is pointing one level too high. It must name the directory holding `cubic_1/`, not its parent. The error message suggests the right path when it can find it.
- **Network blocks huggingface.co** — set `HF_ENDPOINT` to a mirror, or fall back to the legacy git-lfs download with `mlindex.download_models --source github` (this one does require git and git-lfs).
- **Re-download everything from scratch** — `mlindex.download_models --redownload`.

### Developer installation (git clone)

Required for model training, dataset generation, or contributing to the codebase. The machine learning models are version controlled through [git-lfs](https://www.git-lfs.com).

1. **Clone the repository:**
   ```bash
   git clone git@github.com:dwmoreau/MLI.git
   ```

2. **Retrieve the model files:**
   ```bash
   git lfs pull
   ```

3. **Install the project:**
   ```bash
   cd /path/to/the/cloned/repo
   pip install .
   ```

## Usage

### Peak List Generation

Peak list files generated by GSAS-II can be used directly. GSAS-II provides tutorials for creating peak lists:
- [GSAS-II Fit Peaks Tutorial](https://advancedphotonsource.github.io/GSAS-II-tutorials/FitPeaks/Fit%20Peaks.htm)

Alternatively, provide the d-spacings of the observed diffraction peaks in units of q², where q² = (2 sin θ / λ)² = 1/d² (Å⁻²). Save this list to a numpy array.

> **Note:** Only the first 20 peaks in the list are used internally.

### Code Execution

#### Using a numpy array
```bash
mlindex.run --peak-file /path/to/your/file/peaks.npy
```

#### Using a GSAS-II pkslst file
When using a GSAS-II pkslst file, you must supply the wavelength:
```bash
mlindex.run --peak-file /path/to/your/file/peaks.pkslst --wavelength 0.413128
```

#### Parallel execution (recommended)
Use `--nproc N` to run with N parallel worker processes. This is the recommended way to speed up indexing:
```bash
mlindex.run --peak-file /path/to/your/file/peaks.npy --nproc 4
```

#### Zero-point error correction
If your instrument has a systematic 2θ offset, use `--zero-error` to correct for it during indexing. This option requires a wavelength to be specified:
```bash
mlindex.run --peak-file /path/to/your/file/peaks.pkslst --wavelength 0.413128 --zero-error
```

#### MPI mode (HPC clusters)
MPI mode is available for use on HPC clusters with MPI infrastructure. It requires exactly 6 MPI ranks and the `--mpi` flag:
```bash
mpiexec -n 6 mlindex.run --peak-file /path/to/your/file/peaks.pkslst --wavelength 0.413128 --mpi
```

### Analytical Indexer (lightweight alternative)

`mlindex.run_analytical` uses a geometry-based guess-and-check approach instead of ML models. It covers the 11 higher-symmetry Bravais lattices (cF, cI, cP, hP, hR, tI, tP, oC, oF, oI, oP) and requires no model files.

#### Basic usage
```bash
mlindex.run_analytical --peak-file /path/to/your/file/peaks.npy
```

#### Using a GSAS-II pkslst file
```bash
mlindex.run_analytical --peak-file /path/to/your/file/peaks.pkslst --wavelength 0.413128
```

#### Parallel execution
```bash
mlindex.run_analytical --peak-file /path/to/your/file/peaks.npy --nproc 4
```

#### Zero-point error correction
```bash
mlindex.run_analytical --peak-file /path/to/your/file/peaks.pkslst --wavelength 0.413128 --zero-error
```

#### MPI mode
```bash
mpiexec -n 6 mlindex.run_analytical --peak-file /path/to/your/file/peaks.pkslst --wavelength 0.413128 --mpi
```

Results are written to `analytic_results.json`.

---

### Results Interpretation

The program outputs the top 20 unit cell candidates ranked by M20 score and writes them to `indexing_results.json`:

![Indexing Results](assets/indexing_results.jpg)

#### Column Descriptions

| Column | Description |
|--------|-------------|
| **M20** | de Wolff Figure of Merit (Wolff 1968) |
| **Minfo** | Figure of Merit from Taupin (1988) |
| **n_indexed** | Number of indexed peaks, using a probability from Taupin (1988) and a 95% threshold |
| **bravais_lattice** | Assumed Bravais lattice for the unit cell optimization |
| **spacegroup** | Spacegroup whose systematic absences best align with the observed peak list |
| **volume** | Unit cell volume (Å³) |
| **a, b, c** | Unit cell edge lengths (Å) |
| **alpha, beta, gamma** | Unit cell angles (°) |

## Acknowledgements

The US Department of Energy Integrated Computational and Data Infrastructure for Scientific Discovery supported this work via grant DE-SC0022215 to Aaron S. Brewster (LBL), Tess Smidt (MIT), and Nate Hohmann (UCONN).

## Citations

- Taupin, D. (1988). *J. Appl. Cryst.* **21**, 485-489.
- Wolff, P. M. D. (1968). *J. Appl. Cryst.* **1**, 108.
