Metadata-Version: 2.4
Name: nomad-ml-workflows
Version: 0.0.15
Summary: A NOMAD plugin for managing ML workflows.
Author-email: Sarthak Kapoor <sarthak.kapoor@physik.hu-berlin.de>
Maintainer-email: Sarthak Kapoor <sarthak.kapoor@physik.hu-berlin.de>
Project-URL: Repository, https://github.com/FAIRmat-NFDI/nomad-ml-workflows
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: nomad-lab>=1.4.3
Requires-Dist: huggingface-hub
Requires-Dist: pydantic
Requires-Dist: temporalio
Provides-Extra: dev
Requires-Dist: nomad-lab[infrastructure]>=1.4.3; extra == "dev"
Requires-Dist: nomad_ml_workflows[cpu-action]; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: structlog; extra == "dev"
Requires-Dist: mkdocs; extra == "dev"
Requires-Dist: mkdocs-material>=8.1.1; extra == "dev"
Requires-Dist: pymdown-extensions; extra == "dev"
Requires-Dist: mkdocs-click; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Provides-Extra: cpu-action
Requires-Dist: pyarrow; extra == "cpu-action"
Requires-Dist: boto3; extra == "cpu-action"
Requires-Dist: nomad-forces-export; extra == "cpu-action"
Dynamic: license-file

# nomad-ml-workflows

A NOMAD plugin for supporting machine-learning workflows. It currently provides the
Export Entries action, which turns a permission-scoped NOMAD search into a reusable
JSON, Parquet, or CSV dataset.

## 📦 Installation

Install the package with pip:

```sh
pip install nomad-ml-workflows
```

To add the plugin to your NOMAD deployment, see
[Adding this plugin to NOMAD](#-adding-this-plugin-to-nomad).

## ✨ Key functionality

The Export Entries action lets users:

- select entries with a NOMAD search query and ownership scope;
- export complete archives or selected archive paths, optionally resolving references;
- generate JSON, Parquet, or CSV datasets; and
- save the generated dataset to a staging project/upload as a ZIP archive or directory.

JSON preserves the selected archive structure. Parquet provides typed tabular data
and preserves nested quantity values as a sharded dataset, while CSV represents nested
values as JSON text.

### Action input

The action form groups its fields into **Search options** and **Export options**. Its
underlying input shape is:

```json
{
  "user_id": "<injected by NOMAD>",
  "upload_id": "<destination-project-id>",
  "search_settings": {
    "owner": "visible",
    "max_entries": 1000,
    "query": "{\n  \"entry_type\": \"ELNSample\"\n}",
    "required": [
      {
        "type": "include",
        "path": "results.method",
        "resolve_references": false
      }
    ]
  },
  "export_settings": {
    "file_format": "parquet",
    "create_zip_archive": true
  }
}
```

`user_id` is required by the workflow but is not shown in the action form. The other
fields have these semantics:

| Field | Values and behavior |
| --- | --- |
| `upload_id` | ID of the staging project/upload that receives the exported artifacts. |
| `search_settings.owner` | `visible`, `public`, `user`, `shared`, or `staging`; defaults to `visible`. |
| `search_settings.max_entries` | Positive requested limit, bounded by `max_entries_export_limit`; defaults to the smaller of 1,000 and the deployment limit. |
| `search_settings.query` | NOMAD search query supplied as JSON text. The query can be copied from **View API Call** in a NOMAD search app. |
| `search_settings.required` | Archive paths to include. An empty list exports the complete archive. Set `resolve_references` to include content reached through references below a path. |
| `export_settings.file_format` | `parquet`, `csv`, or `json`; defaults to `parquet`. |
| `export_settings.create_zip_archive` | Defaults to `true`. Set it to `false` to publish a project subdirectory instead of a ZIP file. |

The current public model supports include directives only. Exclusion is not yet
available through the action form.

### Generated output

The generated ZIP archive or directory contains:

| File | Contents |
| --- | --- |
| `data.json` or `data.csv` | The exported archive data in the selected single-file format. This file is omitted when no entries match. |
| `data.parquet/part-NNNNN.parquet` | The exported Parquet dataset. Each deterministic part uses the same schema and the directory is omitted when no entries match. |
| `selected_entries.json` | The ordered `entry_id` and `upload_id` pairs selected for export. |
| `metadata.json` | The export metadata under `data`, together with its JSON schema and an explanatory note. |

The metadata records the original input, search timing, export-limit state, and the
`num_entries_available`, `num_entries_selected`, and `num_entries_exported` counts. It
also records `nomad_deployment_api_host`, `nomad_version`, and
`nomad_ml_workflows_version`. A zero-match export still contains `metadata.json` and
an empty `selected_entries.json`.

PyArrow reads the complete Parquet dataset directly from its directory:

```python
import pyarrow.parquet as pq

table = pq.read_table('data.parquet')
```

## ⚙️ Configuration

Configure the action entry point in the `nomad.yaml` of the NOMAD Oasis:

```yaml
plugins:
  entry_points:
    options:
      nomad_ml_workflows.actions:export_entries:
        max_entries_export_limit: 100000
        # Deployment cap for one Export Entries action.

        read_archives_timeout: 7200
        # Start-to-close timeout, in seconds, for reading archives and
        # writing the selected output format.

        write_tabular_timeout: 7200
        # Start-to-close timeout, in seconds, for writing the output 
        # tabular artifact.

        max_write_buffer_size_bytes: 67108864  # 64 MB
        # Maximum number of encoded NDJSON input bytes represented by parsed
        # rows buffered before writing to the output tabular file. Protects
        # against rows with large encoded representations. One oversized row
        # may exceed this target.

        max_write_buffer_size_rows: 1024
        # Maximum number of parsed rows buffered before writing to the output
        # tabular file. Protects against many small or sparse rows whose Python
        # and Arrow representations are much larger than their NDJSON bytes.
```

For deployments where PyArrow uses the `mimalloc` memory-pool backend, the CPU
action worker can be started with immediate memory purging enabled:

```sh
MIMALLOC_PURGE_DELAY=0
```

The tabular exporter requests that unused Arrow memory be released after every
batch. Setting the environment variable `MIMALLOC_PURGE_DELAY=0` asks mimalloc
to return unused pages to the operating system immediately. In local measurements,
this produced slightly lower post-export RSS.

This setting is optional and must be present in the CPU worker environment before
the worker starts. Immediate purging may trade some allocation performance for
lower retained RSS.

## 🚀 Adding this plugin to NOMAD

### NOMAD Oasis

Follow the
[NOMAD Oasis configuration documentation](https://docs.nomad-lab.eu/howto/oasis/configure.html#plugins)
to add and enable the plugin in an Oasis.

### Local NOMAD development installation

Use the dedicated
[`nomad-distro-dev`](https://github.com/FAIRmat-NFDI/nomad-distro-dev) repository for
an integrated local NOMAD development environment.

## 🛠️ Development

Clone the repository and create a virtual environment with Python 3.10, 3.11, or
3.12:

```sh
git clone https://github.com/FAIRmat-NFDI/nomad-ml-workflows.git
cd nomad-ml-workflows
python3.12 -m venv .pyenv
. .pyenv/bin/activate
python -m pip install --upgrade pip
python -m pip install uv
uv pip install -e '.[dev]'
```

Run the focused Export Entries tests:

```sh
pytest tests/actions/export_entries/test_models.py \
  tests/actions/export_entries/test_utils.py
```

Run linting and formatting checks with Ruff:

```sh
ruff check .
ruff format . --check
```

For interactive test debugging, pass `--pdb` to pytest. To serve the documentation
locally, use the development dependencies and run:

```sh
mkdocs serve
```

## 👥 Main contributors

| Name | Email |
| --- | --- |
| Sarthak Kapoor | [sarthak.kapoor@physik.hu-berlin.de](mailto:sarthak.kapoor@physik.hu-berlin.de) |

## 📄 License

This project is licensed under the MIT License. See [LICENSE](LICENSE).
