Metadata-Version: 2.4
Name: PyeScan
Version: 0.0.2
Summary: A Python library for working with retinal scans exported by Private Eye / Crystal Eye
Project-URL: Homepage, https://github.com/eye2gene/PyeScan
Project-URL: Repository, https://github.com/eye2gene/PyeScan
Project-URL: Issues, https://github.com/eye2gene/PyeScan/issues
Author-email: William Woof <William.woof@eye2gene.com>
Maintainer-email: William Woof <William.woof@eye2gene.com>
License-Expression: MIT
License-File: LICENSE
Keywords: OCT,fundus,medical imaging,ophthalmology,retina
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Python: <4.0,>=3.12
Requires-Dist: matplotlib
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pillow>=12
Requires-Dist: rich
Requires-Dist: tqdm
Requires-Dist: typer
Provides-Extra: all
Requires-Dist: ipython; extra == 'all'
Requires-Dist: ipywidgets; extra == 'all'
Requires-Dist: opencv-python-headless; extra == 'all'
Requires-Dist: retinalysis-registration; extra == 'all'
Requires-Dist: scikit-image; extra == 'all'
Requires-Dist: scipy; extra == 'all'
Provides-Extra: jupyter
Requires-Dist: ipython; extra == 'jupyter'
Requires-Dist: ipywidgets; extra == 'jupyter'
Provides-Extra: metrics
Requires-Dist: opencv-python-headless; extra == 'metrics'
Requires-Dist: scikit-image; extra == 'metrics'
Requires-Dist: scipy; extra == 'metrics'
Provides-Extra: projection
Requires-Dist: scikit-image; extra == 'projection'
Provides-Extra: registration
Requires-Dist: retinalysis-registration; extra == 'registration'
Requires-Dist: scipy; extra == 'registration'
Description-Content-Type: text/markdown

# PyeScan

PyeScan is a python library for streamlining the process of working with retinal scans in python and provide a common interface for various processes (e.g. running various models). The idea is for the main library to develop core, implementation-agnostic, functionality with various helpers/loaders/savers/views etc to deal with specific formats.

## Installation

Requires Python 3.12+.

### For users

Install the published package from PyPI (using [uv](https://docs.astral.sh/uv/) or pip):

```bash
uv pip install pyescan   # or: pip install pyescan

# With optional extras
uv pip install "pyescan[registration]"  # image registration (scipy)
uv pip install "pyescan[metrics]"       # metrics with OpenCV and scikit-image
uv pip install "pyescan[projection]"    # OCT/enface projection with scikit-image
uv pip install "pyescan[jupyter]"       # Jupyter notebook support
uv pip install "pyescan[all]"           # everything
```

### For developers

Clone the repo and use `uv sync` for a reproducible, lockfile-driven environment:

```bash
git clone https://github.com/eye2gene/PyeScan.git
cd PyeScan
uv sync --all-extras   # installs core + all optional extras + dev tools
```

This creates a `.venv`, installs pyescan in editable mode, and pins all dependencies to the exact versions in `uv.lock`.

## Releasing

Releases are built and published by GitHub Actions when a version tag is pushed.
PyPI authentication uses a Trusted Publisher, so no API token is stored in the
repository.

1. Update the version and commit it:

   ```bash
   uv version 0.1.0
   git add pyproject.toml uv.lock
   git commit -m "chore: release 0.1.0"
   ```

2. Tag that exact commit and push the tag:

   ```bash
   git tag -a v0.1.0 -m "PyeScan 0.1.0"
   git push origin v0.1.0
   ```

The release workflow runs quality checks and tests, builds and smoke-tests both
distribution formats, publishes them to PyPI, and creates a GitHub release.

## Project structure

```txt
PyeScan/
├── .github/
│   └── workflows/
│       ├── ci.yml          # Lint, type-check, test on every push/PR
│       └── release.yml     # Semantic release + publish to PyPI
├── src/
│   └── pyescan/
│       ├── __init__.py
│       ├── __main__.py
│       ├── CELoader.py
│       ├── annotation_loader.py
│       ├── py.typed
│       ├── core/           # Scan classes, metadata, visualisation
│       ├── metrics/        # Unified metrics system
│       └── tools/          # CLI, dataloading, dataset utilities
├── tests/
├── justfile
├── pyproject.toml
└── README.md
```

## Example usage

You can use `load_record_from_CE` to load a set of scans from a single PrivateEye/CrystalEye export
(N.B: This is currently only tested on Heidelberg scans):

```python
from pyescan.CELoader import load_record_from_CE

record_path = "/[PATH_TO_DATA]/[XXXXXXXX.pat]/[XXXXXXXX.sdb]/"
scans = load_record_from_CE(record_path)
```

`scans` is a list of scan instances of the various scan classes (all of which inherit from the `BaseScan`)

Alternatively you can load from a dataframe (restricted to a single `sdb`):

```python
from pyescan.CELoader import load_record_from_df

import pandas as pd

df = pd.read_csv("[PATH_to_CSV]")  # e.g. generated by get_pe_export_summary

df_scan = df.query("sdb == '01437803.sdb'")
scans = load_record_from_df(df_scan)
```

This is "configurable" to work with any data format, see `CrystalEyeParser` and `CrystalEyeParserCSV` in `CELoader.py` to see an example of how this is set up (N.B: These WILL be updated in future, hopefully to allow for proper configuration).

Data loading is done in a 'lazy' fashion where the actual images are only loaded when they are needed (however you can force load with `scan.preload()` if needed) meaning you can load scans for basic operations without having to load all the data - only the metadata is loaded.

You can get the raw data within the scan by calling `scan.image`, which will return a wrapper class for the image, or `scan.data` which will return a raw numpy array of the image. For OCT scans you can call `scan.enface` for the enface scan and `scan.bscans` to return the bscan array.

The library provides support for advanced indexing and slicing of the scans, for easy accessing of a certain region, e.g. `scan.bscans[:n,:100,:100]` (or `scan.data[:,:100,:100]` will return a numpy array of an `nx100x100` of the OCT volume.

You can also view a scan inside a jupyter notebook with `display(scan)` (or simply `scan` at the end of a cell), with support for interactive viewer for OCT.

You can access information through `scan.metadata.QUERY` which can be used to get various information about this scan. Only a few functions are currently implemented but these should be added by adding the required property to the relevant `MetadataView` class (in this case `MetadataViewCrystalEye`). This is hierarchical so scan/image specific information can be implemented in the `MetadataView` class, and then accessed seamlessly by the scan object.

### Annotations

Annotations support is still in development, but can be loaded from file and added to scan:

```python
# This loads a scan from a dataframe as long as the right columns are set
# The default mapping is: (this can be changed by supplying a dict as seen below)
#    {"group": "group", "source_id": "source_id", "modality": "modality", "image_location": "file_path", "n_images": "number_of_images" }
from pyescan.CELoader import load_record_from_df

scan, *_ = load_record_from_df(df_scan, {"image_location": "file_path_original"})

# This loads a scan from a dataframe, this one only needs file_path_col, bscan_index_col, and optionally feature_col
# If feature_col is supplied then it will automatically create a dict of all features, otherwise it will just turn a single (unnamed) annotation
from pyescan.annotation_loader import load_annotation_from_df

annotations = load_annotation_from_df(
    df_scan.query("modality == 'OCT'"), feature_col="feature"
)

# Add the loaded annotations to the scan
#  This will probably be update in future
scan.add_annotations(annotations)

# Then display the scan (can sometimes take a while as it needs to load all the images)
scan
```

Loading can also be done using `load_annotation_from_folder` which works in the same way but first indexes the folder structure according to a specified pattern using `summarise_dataset` and uses the resulting dataframe as an intermediate.

### Unified Metrics system

`pyescan.metrics` contains a bunch of useful metrics which can be run to get various statistics. This is built into a centralised dependency system so that any intermediate requirements are automatically computed (and cached). These used to all be set up to run directly on dataframes, however this was inflexible, and not really compatible with the idea of pyescan.

The new metric system is designed to be input agnostic, or rather they all simply take in a set of inputs (that they need to do the required calculations) and output a set of results. The clever part is that, by using a consistent internal naming scheme, the metric processor is able to work out what maps to what, which means that if we want an output which is dependent on a complex chain of inputs, the processor can find the relevant functions automatically and cache the intermediate results. This also saves overhead as it means we can run multiple metrics at once without having to recompute intermediate values (or reload files)!

Example running the new system on a dataframe:

```python
from pyescan.metrics.helpers import run_on_dataframe

import pandas as pd

df = pd.read_csv(YOUR_PATH_HERE)

stats = [
    "pixel_count_oct_4.0mm_superior",
    "pixel_count_oct_4.0mm_inferior",
    "pixel_count_oct_4.0mm_sinister",
    "pixel_count_oct_4.0mm_dexter",
]

col_map = {
    "file_path_mask": "file_path",
    "scan_width_px": "size_width",
    "scan_height_px": "size_height",
}
df_with_stats = run_on_dataframe(df, stats, col_map, auto_merge=True, named_only=False)

df_with_stats
```

New metrics can also be added - there is a function wrapper provided for doing this, see `pyescan.metrics.metrics` for examples (documentation to follow).

### Image registration

Basic image registration is supported via [retinalysis](https://github.com/Eyened/retinalysis-registration), in particular helper functions are added for computing registrations for all pairs of images, and converting into a set of "optimised" poses, making use of the cycle consistency property (i.e. any composition of homographies in a cycle should give the identity matrix). This is very much WIP and may be removed from PyeScan in future (hopefully into another repo), but for now the tools can be used as follows:

```python
from pyescan.tools.image_registration import (
    get_transforms,
    get_cleaned_poses,
    visualise_poses,
)

print("Registering pairs")
transforms = get_transforms(img_paths)

print("Inferring poses")
poses = get_cleaned_poses(transforms)

visualise_poses(poses)
```

(N.B: Suggest running with just a limited number of images, as currently registration is done for all pairs)

### Helper functions

In `pyescan.tools.dataset_utils` there are two useful dataset functions `summarise_dataset`, and `get_pe_export_summary`

`summarise_dataset` simply traverses a directory structure and finds all files matching a particular file structure, and summarises them in a single pandas dataframe.

For example:

```python
from pyescan.tools.dataset_utils import summarise_dataset

summarise_dataset(
    [ROOT_DIRECTORY], structure="{pat}/{sdb}/{source_id}_{bscan_index:\\d+}.png"
)
```

Will get all files that look like `{pat}/{sdb}/{source_id}_{bscan_index:\d+}.png` (compared to the `[ROOT_DIRECTORY]`) and list them in a table with columns for `pat`, `sdb`, `source_id`, and `bscan_index` (as well as `file_path`). As you can imagine this can be very useful for loading different types of input file structures.

The other function, `get_pe_export_summary`, is similar, but traverses through a Private/CrystalEye export and dives into the metadata, producing a dataframe with a full summary of all scans/images including images (there is some data still missing, in particular the image-level metadata is now included, but some fields are currently omitted for brevity). This is useful to make metadata summary CSVs which can be merged with other datasets (e.g. annotations/masks) as needed.

### CLI tools

All CLI tools are available under the unified `pyescan` command:

```bash
pyescan --help
pyescan summarise-export --help
pyescan summarise-dataset --help
pyescan run-function-on-csv --help
pyescan run-function-over-csv --help
pyescan run-metric --help
pyescan narrow-to-wide --help
```

You can also run via `python -m pyescan`.

## Architecture

The core object of PyeScan is the scan object which inherits from the `BaseScan` class (though some scan objects are also compositions of other BaseScan-derived instances, as well as being a `BaseScan` in their own right).

These scan classes provide a standard interface for various operations, along with various helper functionality.

A big part of how this is achieved is with `MetadataView` objects which (as the name suggests) provide a 'view' onto the raw metadata, translating from the original metadata structure (by providing various accessor properties), while also handling managing which part of the metadata the particular scan refers to.

Loading of actual images is currently done 'lazily' so data is only loaded when actually needed. This is intended to be largely seamless for the end user, but obviously there may be times where the user wants to 'precache' data, so `preload` and `unload` functions are included for forcing loading/unloading.

Currently only loading images directly from disk is supported, but it should be possible to implement other ways of loading (e.g. directly from e2e file or whatever) fairly transparently.

Annotations/masks can be loaded and added to scans. In future there will be a system to properly identify which model and which feature the annotation came from/refers to, and ways of automatically searching for annotations and linking them to certain scans.

During development of PyeScan it was found there was still a lot of working with dataframes, rather than loading in scans as PyeScan objects explicitly. The idea with the metric system is then to make it easy to work with either depending on the user needs.

## Development

```bash
# Quality checks (format, lint, type-check)
just qa

# Run tests
just test

# Build
just build

# See all available commands
just list
```

## How releases work

This project uses [conventional commits](https://www.conventionalcommits.org/) and [python-semantic-release](https://python-semantic-release.readthedocs.io/):

1. Develop on a feature branch with conventional commit messages (`feat:`, `fix:`, `docs:`, etc.)
2. Open a PR and merge to `main`
3. GitHub Actions automatically determines the next version, updates `CHANGELOG.md`, creates a tag/release, and publishes to PyPI

## Template updates

This project is linked to the [e2g-pypkg](https://github.com/eye2Gene/e2g-pypkg) template via [cruft](https://cruft.github.io/cruft/):

```bash
uv run cruft check    # check for template updates
uv run cruft diff     # preview changes
uv run cruft update   # apply updates
```

## Roadmap

- [ ] Full Ruff + Ty compliance pass
- [ ] Proper documentation (Sphinx or MkDocs)
- [ ] Comprehensive test suite with good coverage
- [ ] Develop and test running metrics on PyeScan scan objects directly
- [ ] Automatic annotation discovery and linking to scans
- [ ] Support loading images from formats other than disk (e.g. directly from .e2e files)
- [ ] Extract image registration tooling into a separate package

## License

MIT — see [LICENSE](LICENSE) for details.
