Metadata-Version: 2.4
Name: matensemble
Version: 0.5.9
Summary: An adaptive and highly asynchronous ensemble simulation workflow manager MatEnsemble (https://github.com/Q-CAD/MatEnsemble) built jointly on top of the hierarchical graph based scheduler FLUX and concurrent-futures infrastructure of python
Author: Soumendu Bagchi, Kaleb Duchesneau
Author-email: Soumendu Bagchi <soumendubagchi@gmail.com>, Kaleb Duchesneau <kalebduchesneau@gmail.com>
License-File: LICENSE
Requires-Dist: ase>=3.22.0
Requires-Dist: cloudpickle>=3.1.2
Requires-Dist: matplotlib>=3.4.0
Requires-Dist: networkx>=3.6.1
Requires-Dist: numpy>=1.21.0
Requires-Dist: ovito>=3.7.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: pymatgen>=2022.0.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: seaborn>=0.11.0
Requires-Dist: starlette>=1.0.0
Requires-Dist: uvicorn>=0.42.0
Requires-Dist: flux-python==0.66.0 ; extra == 'flux'
Requires-Python: >=3.12
Provides-Extra: flux
Description-Content-Type: text/markdown

[![PyPI version](https://badge.fury.io/py/matensemble.svg)](https://pypi.org/project/matensemble/)
[![Documentation](https://readthedocs.org/projects/matensemble/badge/?version=latest)](https://matensemble.readthedocs.io/en/latest/)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)

<p align="center">
  <img src="media/Logo-Matensemble.png" alt="MatEnsemble" width="720" />
</p>

# MatEnsemble

MatEnsemble is a framework to build, orchestrate, and asynchronously manage extremely scalable
adaptive-learning workflows, especially targeted for compute-intensive AI-driven high-throughput and
ensemble-driven materials modeling simulations (e.g., atomistic modeling, Phase-Field, etc.) as
efficiently as possible.

While it can in general run on your personal Mac/Linux workstation and
orchestrate arbitrary python callables, shell commands with explicit resource and dependency-aware
execution graphs from a single python workflow driver process,  MatEnsemble shines with
***user-defined autonomous strategic*** execution of large batches of adaptively and
hierarchically-scheduled tasks on HPC systems, specifically on Peta and Exascale computing
facilities, e.g., Perlmutter, Frontier, Aurora etc.

Here is an example which defines a simple MPI chore, adds ten independent instances of it to a pipeline, and submits the workflow.

```python
pipe = Pipeline()

# register a function the MatEnsemble
@pipe.chore(num_tasks=10, cores_per_task=1, gpus_per_task=0, mpi=True)
def mpi_hello_world():
    size = MPI.COMM_WORLD.Get_size()
    rank = MPI.COMM_WORLD.Get_rank()
    name = MPI.Get_processor_name()

    print(f"Hello World! I am process {rank} of {size} on {name}.")

# adding 10 mpi_hello_world chores to the pipeline
for _ in range(10):
    mpi_hello_world()

pipe.submit(log_delay=1)
```

Rather than launching the MPI jobs directly, you describe the work declaratively with the @pipe.chore decorator and enqueue each invocation. MatEnsemble then schedules the chores through Flux, allocating the requested resources (num_tasks, CPU cores, GPUs, and MPI support) for each job.

Hs, GPUs, and MPI support) for each job.

## Installation

To install MatEnsemble you can follow our [documentation](https://matensemble.readthedocs.io/en/latest/) which will walk you through manual
installation for different systems, and also give many helpful tips for working with containers

You can also use our installation script to install MatEnsemble along side the MCP server with:

```bash
curl -fsSL https://raw.githubusercontent.com/Q-CAD/MatEnsemble/refs/heads/main/install.sh | bash
```

## Minimal Code Example

MatEnsemble workflows are ordinary Python scripts (and/or shell commands) which can be use to:

1. define resource-aware chores
2. Pass chore outputs into later chores to create a DAG
3. Add a strategy when the workflow should decide what to launch next while the campaign is already running.

```python
from matensemble.pipeline import Pipeline
from matensemble.model import Resources
from matensemble.chore import ChoreSpec

pipe = Pipeline()

md_resources = dict(num_tasks=128, cores_per_task=1, gpus_per_task=4, mpi=True)
analysis_resources = dict(num_tasks=1, cores_per_task=8)


@pipe.chore(name="simulate", **md_resources)
def simulate(candidate):
    # Run LAMMPS, DFT, phase-field, or another science application here.
    return {"trajectory": "traj.dump", "candidate": candidate}


@pipe.chore(name="score", **analysis_resources)
def score(simulation):
    # Analyze the completed simulation and propose the next high-value sample.
    return {
        "uncertainty": 0.18,
        "next_candidate": {"temperature": 1750, "composition": "SiO2"},
    }


@pipe.strategy(bolo_list=["score"], **analysis_resources)
def adapt(report):
    if report["uncertainty"] < 0.05:
        return None

    return ChoreSpec(
        args=(report["next_candidate"],),
        kwargs={},
        resources=Resources(**md_resources),
        qualname="simulate",
    )


seed = {"temperature": 1600, "composition": "SiO2"}
trajectory = simulate(seed)
score(trajectory)  # OutputReference creates the simulate -> score DAG edge.

future = pipe.submit(log_delay=10)
results = future.result()
```

## Publications

1. Bagchi, Soumendu, et al. "Towards “on-demand” van der Waals epitaxy with adaptive ensemble sampling atomistic workflows." Digital Discovery (2026) https://doi.org/10.1039/d6dd00049e.
2. Morelock, Ryan, et al. "pyRMG: A framework for high-throughput, large-cell DFT calculations on supercomputers." The Journal of Chemical Physics 164.5 (2026).
