Metadata-Version: 2.5
Name: nomad-fsspec
Version: 0.4.1
Summary: An fsspec filesystem for the raw data of NOMAD.
Project-URL: Homepage, https://github.com/arash77/nomad-fsspec
Project-URL: Issues, https://github.com/arash77/nomad-fsspec/issues
Author: nomad-fsspec contributors
License-Expression: MIT
License-File: LICENSE
Keywords: filesystem,fsspec,materials science,nomad
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: >=3.10
Requires-Dist: fsspec>=2024.12.0
Requires-Dist: requests>=2.31
Requires-Dist: urllib3>=1.26
Description-Content-Type: text/markdown

# nomad-fsspec

An [fsspec](https://filesystem-spec.readthedocs.io) filesystem for the raw data of
[NOMAD](https://nomad-lab.eu), the materials science repository. It is read-only.

```python
import fsspec

fs = fsspec.filesystem("nomad")  # https://nomad-lab.eu/prod/v1 by default
fs.ls("/")  # two folders: /datasets and /uploads
fs.ls("/datasets")  # public datasets
fs.ls("/datasets/wWgAnNZNQxOHLf97H62dgw", detail=False)  # the uploads of one dataset, by id
fs.upload_names(["Xa2LB__ESiKu-odRBPJhjA"])  # {"Xa2LB__ESiKu-odRBPJhjA": "nomad-demo-data.zip"}
fs.get_file(
    "/datasets/wWgAnNZNQxOHLf97H62dgw/Xa2LB__ESiKu-odRBPJhjA/nomad-demo-data/BrK_svSe/TBCC006.ABC/vasprun.xml.relax2",
    "vasprun.xml",
)
```

The top level has two folders, so a path means the same thing with or without a token:

- `/datasets/<dataset id>/<upload id>/<path inside the upload>` for public datasets. Datasets
  have their names in `display_name`. Uploads inside a dataset are listed by id, since naming
  all uploads of a large dataset at once makes NOMAD time out; `upload_names` looks names up
  8 at a time and keeps them, and `search_uploads` finds the ids of uploads by name.
- `/uploads/<upload id>/<path inside the upload>` for your uploads and those shared with you,
  published or not, as NOMAD's uploads page lists them. Listing `/uploads` needs a token.
  Published uploads can be read through it without one.

```python
fs = fsspec.filesystem("nomad", token="...")
fs.ls("/uploads")  # your uploads, with their names in display_name
```

A published upload with more than 300 entries is listed from the central directory of its ZIP,
which takes one or two byte ranges of `/uploads/<upload id>/raw` without the token, even for
uploads with hundreds of thousands of files. NOMAD's own folder listing adds up every size below a
folder and can take a minute on such uploads. The listing is kept for the life of the process and
shared, since published uploads do not change: about 19 bytes per file, so 7 MB for the largest
public upload, and up to 64 MB in all, the oldest dropped first. Smaller, unpublished and embargoed
uploads are listed folder by folder. NOMAD allows one new connection to the ZIP route per client
every 5 seconds; a busy answer is retried.

A [personal access token](https://docs.nomad-lab.eu/howto/manage/program/auth.html) with the
`uploads:read` scope is enough. To use a NOMAD Oasis, pass `base_url`:

```python
fs = fsspec.filesystem("nomad", base_url="https://my-oasis.example.org/nomad-oasis")
```

NOMAD's rate limits (HTTP 429 and 503) are retried. Redirects are not followed: a base URL that
redirects is reported with the address it redirects to.

## Development

```bash
uv sync
uv run pytest -m "not live"  # unit tests
uv run pytest -m live        # a few calls to nomad-lab.eu
uv run ruff check . && uv run ruff format --check . && uv run mypy src
```

## License

MIT
