Metadata-Version: 2.4
Name: ccpem-pipeliner
Version: 1.6.0
Summary: CCP-EM Pipeliner
License-Expression: MPL-2.0
Project-URL: Repository, https://gitlab.com/ccpem/ccpem-pipeliner
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: ccpem-utils==0.0.1.dev34
Requires-Dist: gemmi==0.7.4
Requires-Dist: importlib-metadata>=3.6.0
Requires-Dist: matplotlib
Requires-Dist: mrcfile
Requires-Dist: numpy>=2.0.2
Requires-Dist: pandas
Requires-Dist: Pillow>=11.3.0
Requires-Dist: plotly
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: scipy>=1.13.1
Requires-Dist: imodmodel
Requires-Dist: pdbe-mmcif-validator==0.1.92
Provides-Extra: dev
Requires-Dist: black==25.11.0; extra == "dev"
Requires-Dist: flake8==7.3.0; extra == "dev"
Requires-Dist: mypy==1.19.1; extra == "dev"
Requires-Dist: flake8-pyproject; extra == "dev"
Requires-Dist: pandas-stubs; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: sniffio; extra == "dev"
Requires-Dist: types-pillow; extra == "dev"
Requires-Dist: types-setuptools; extra == "dev"
Requires-Dist: types-pyyaml; extra == "dev"
Provides-Extra: docs
Requires-Dist: myst-parser; extra == "docs"
Requires-Dist: sphinx; extra == "docs"
Requires-Dist: sphinx-argparse; extra == "docs"
Requires-Dist: sphinx-rtd-theme; extra == "docs"
Dynamic: license-file

# CCPEM-pipeliner

[![Documentation](https://readthedocs.org/projects/ccpem-pipeliner/badge/ "Documentation on ReadTheDocs")](https://ccpem-pipeliner.readthedocs.io/)

![ccpem-pipeliner header image](docs/source/header.png)

The CCP-EM-pipeliner is a Python library which provides the ‘business logic’ layer for 
the CCP-EM software suite (v2). It can be used separately and is open source 
(MPL 2.0). 

Pipeliner provides command line functionality to:

- Run image processing tasks (jobs)
- Continue/overwrite existing jobs
- Create scripts for automated workflows
- Cleanup to remove tmp/unneeded files
- Create archives to preserve workflows
- Generate metadata reports
- Generate literature reference lists
- Generate results for viewing in the CCP-EM Doppio GUI 

CCP-EM pipeliner allows the design of custom, automated cryoEM workflows for SPA, STA 
and model building and is used as part of eBIC/DLS’s automated processing pipeliner 
and the CZII CryoET Object Identification Kaggle challenge. 

It produces Doppio (or RELION) projects which can be viewed and continued in the UI.
Project created in pipeliner are fully compatible with projects 
created using the RELION GUI.


## Installation

Install from PyPI with ``pip install ccpem-pipeliner``

To install the package from source navigate into the `ccpem-pipeliner` directory and
install the pipeliner with the command:

``pip install -e .``

## Check the installation

Once the pipeliner is installed use the command `pipeliner.check_setup` to check that
the setup is complete and the pipeliner can find the Relion programs it needs to run.

## Documentation

Documentation is available online at:
[ccpem-pipeliner.readthedocs.io/en/latest/](https://ccpem-pipeliner.readthedocs.io/en/latest/)

To build the documentation yourself, install the documentation build requirements
as follows:

`pip install -e .[docs]`

Then navigate to the `ccpem-pipeliner/docs` directory and run the command:

`make html`

Then open the file `ccpem-pipeliner/docs/_build/html/index.html` in a web browser to access the
documentation.

## Adding plugins

To add additional job plugins the plugin must first be written as a `PipelinerJob` object.
See the [pipeliner documentation](https://ccpem-pipeliner.readthedocs.io/en/latest/source/pipeliner_jobs.html)
for a description of the format.

The plugin must then be placed in the `Pipeliner/jobs/` folder or one of its subfolders, or create a new subfolder.

Once the plugin file is in place add it to the list of entry points in `pyproject.toml`.

If `pyproject.toml` has been modified update the pipeliner by running `pip install -e .` for the changes to
take effect.

## For Developers

It's a good idea to work in a virtual environment for this project. Set one up as
follows:

```
python3 -m venv venv/
source venv/bin/activate
pip install -e .[dev]
```

This project uses [pre-commit](https://pre-commit.com/) to run
[Black](https://black.readthedocs.io/) for code formatting,
[flake8](https://flake8.pycqa.org/) for linting and some other simple checks,
and [mypy](https://mypy.readthedocs.io/en/stable/) for type checks.

You might want to install Black separately yourself too, so you can run it from your
IDE.

According to the flake8 documentation, flake8 should not stop a git commit from going
ahead unless `flake8.strict` is set in the git config. That doesn't actually seem to
work: with the current configuration, commits fail if flake8 finds any problems. There
are some flake8 warnings that have not been fixed yet, so to get around this, flake8
checks can be disabled: `SKIP=flake8 git commit ...`

## Unit tests

Run the tests with `pytest`.

Some tests are quite slow, these tests are skipped by default. Set the
environment variable `PIPELINER_TEST_SLOW` to a non-empty string to run the slower
tests as well.

Some tests run actual programs, which can also make the test suite take much longer to
run. Set the environment variable `PIPELINER_TEST_LIVE` to a non-empty string to run
these tests as well. Live tests will only be run if the programs needed for them are
available.

## Contact us

For more information visit the [CCP-EM website](https://www.ccpem.ac.uk/).

The [CCP-EM mailing list](https://www.jiscmail.ac.uk/CCPEM) is a useful resource for questions about 
pipeliner, RELION, and other topics relating to CryoEM data processing.

For additional help and support contact CCP-EM directly at [ccpem@stfc.ac.uk](mailto:ccpem@stfc.ac.uk).

We welcome contributions from the community. This is a public repository so if you have 
feature requests or have made improvements please raise an issue or merge request.
