spacr.io
========

.. py:module:: spacr.io






Module Contents
---------------

.. py:function:: process_non_tif_non_2D_images(folder)

   Split multi-dimensional or non-TIFF images in ``folder`` into per-channel TIFFs.

   Grayscale non-TIFF images are converted to TIFF in place. Multi-
   dimensional images (3D/4D/5D) are split into one grayscale TIFF per
   ``(channel, Z, T)`` combination. Bit depth is preserved.

   :param folder: Directory containing the input images.
   :returns: None


.. py:class:: CombineLoaders(train_loaders)

   Round-robin iterator over multiple DataLoaders.

   Yields ``(loader_index, batch)`` pairs, drawing from a random loader
   each step and dropping loaders once exhausted.

   :param train_loaders: DataLoaders to combine.
   :raises StopIteration: when every wrapped loader is exhausted.


   .. py:attribute:: train_loaders


   .. py:attribute:: loader_iters


.. py:class:: CombinedDataset(datasets, shuffle=True)

   Bases: :py:obj:`torch.utils.data.Dataset`


   Concatenation of multiple ``Dataset`` objects behind a single index space.

   :param datasets: Datasets to concatenate; their samples must be
       index-compatible.
   :param shuffle: If True, index lookups are permuted once at
       construction time. Default ``True``.


   .. py:attribute:: datasets


   .. py:attribute:: lengths


   .. py:attribute:: total_length


   .. py:attribute:: shuffle
      :value: True



.. py:class:: NoClassDataset(data_dir, transform=None, shuffle=True, load_to_memory=False)

   Bases: :py:obj:`torch.utils.data.Dataset`


   Flat directory of unlabelled images returned alongside their file paths.

   :param data_dir: Directory containing image files.
   :param transform: Optional callable applied to each PIL image. If
       ``None``, images are converted with ``ToTensor``.
   :param shuffle: If True, shuffle filename list at construction.
       Default ``True``.
   :param load_to_memory: If True, decode all images once and hold them
       in RAM. Default ``False``.


   .. py:attribute:: data_dir


   .. py:attribute:: transform
      :value: None



   .. py:attribute:: shuffle
      :value: True



   .. py:attribute:: load_to_memory
      :value: False



   .. py:attribute:: filenames


   .. py:method:: load_image(img_path)

      Return the image at ``img_path`` decoded as RGB.

      :param img_path: Path to the image file.
      :returns: PIL ``Image`` in RGB mode.



   .. py:method:: shuffle_dataset()

      Shuffle the internal filename list in place.



.. py:class:: spacrDataset(data_dir, loader_classes, transform=None, shuffle=True, pin_memory=False, specific_files=None, specific_labels=None)

   Bases: :py:obj:`torch.utils.data.Dataset`


   Image classification dataset that reads class subfolders under ``data_dir``.

   :param data_dir: Root directory containing one subdirectory per class.
   :param loader_classes: Ordered list of class names — the index in
       this list becomes the integer label.
   :param transform: Optional callable applied to each PIL image.
   :param shuffle: If True, shuffle files+labels at construction.
   :param pin_memory: If True, eagerly load every image into RAM via a
       multiprocessing pool.
   :param specific_files: Optional explicit list of image paths. If
       supplied together with ``specific_labels``, directory scanning
       is skipped.
   :param specific_labels: Labels paired with ``specific_files``.


   .. py:attribute:: data_dir


   .. py:attribute:: classes


   .. py:attribute:: transform
      :value: None



   .. py:attribute:: shuffle
      :value: True



   .. py:attribute:: pin_memory
      :value: False



   .. py:attribute:: filenames
      :value: []



   .. py:attribute:: labels
      :value: []



   .. py:method:: load_image(img_path)

      Return the image at ``img_path`` decoded as RGB with EXIF orientation applied.



   .. py:method:: shuffle_dataset()

      Jointly shuffle ``filenames`` and ``labels`` in place.



   .. py:method:: get_plate(filepath)

      Return the plate identifier parsed from a filename (leading token before ``_``).

      :param filepath: Image path.
      :returns: Plate ID string.



.. py:class:: spacrDataLoader(*args, preload_batches=1, **kwargs)

   Bases: :py:obj:`torch.utils.data.DataLoader`


   DataLoader that pre-fetches batches into a queue on a background process.

   Wraps ``torch.utils.data.DataLoader`` and, when ``pin_memory`` is
   False, spawns a worker process that stays one or more batches ahead
   of consumption to hide I/O latency.

   :param preload_batches: Number of batches to keep queued ahead.
       Default ``1``.


   .. py:attribute:: preload_batches
      :value: 1



   .. py:attribute:: batch_queue


   .. py:attribute:: process
      :value: None



   .. py:attribute:: current_batch_index
      :value: 0



   .. py:attribute:: pin_memory


   .. py:method:: cleanup()

      Signal the preloader to stop and join the background process.



.. py:class:: TarImageDataset(tar_path, transform=None)

   Bases: :py:obj:`torch.utils.data.Dataset`


   Image dataset backed by a tar archive, decoded on demand.

   :param tar_path: Path to the tar archive.
   :param transform: Optional callable applied to each PIL image.


   .. py:attribute:: tar_path


   .. py:attribute:: transform
      :value: None



.. py:function:: load_images_from_paths(images_by_key)

   Load images grouped by key into NumPy arrays.

   :param images_by_key: Mapping of key -> list of image paths.
   :returns: Mapping of the same keys -> list of ``ndarray`` images.
       Paths that fail to load are skipped with a printed warning.


.. py:function:: concatenate_and_normalize(src, channels, save_dtype=np.float32, settings=None)

   Concatenate per-file channel arrays and normalise them into a single stack.

   :param src: Directory containing per-FOV ``.npy`` channel arrays.
   :param channels: Channel indices to keep in the output stack.
   :param save_dtype: NumPy dtype for the saved normalised arrays.
       Default ``np.float32``.
   :param settings: Preprocessing settings dict. Must contain the
       background, signal-to-noise, randomize, timelapse and plotting
       keys used elsewhere in preprocessing.
   :returns: Path to the directory where normalised arrays were saved.


.. py:function:: delete_empty_subdirectories(folder_path)

   Recursively delete every empty subdirectory under ``folder_path``.

   :param folder_path: Root directory to scan.
   :returns: None


.. py:function:: preprocess_img_data(settings)

   Preprocess raw microscopy images into normalised, channel-merged stacks.

   Converts z-stacks to MIPs, renames files into the spacr convention,
   merges per-channel folders into stacked ``.npy`` arrays, and writes
   normalised outputs. Behaviour is driven entirely by ``settings``
   (source path, channels, background, plotting, test mode, etc.).

   :param settings: Preprocessing settings dict. See
       ``settings.set_default_settings_preprocess_img_data`` for keys.
   :returns: ``(settings, src)`` when an existing ``masks`` folder is
       found and preprocessing is skipped; otherwise ``None``.


.. py:function:: read_plot_model_stats(train_file_path, val_file_path, save=False)

   Plot training vs. validation curves from a saved model's per-epoch CSVs.

   :param train_file_path: Path to the training stats CSV.
   :param val_file_path: Path to the validation stats CSV.
   :param save: If True, write PDFs next to the training CSV instead of
       showing them. Default ``False``.
   :returns: None


.. py:function:: convert_numpy_to_tiff(folder_path, limit=None)

   Convert every ``.npy`` array in ``folder_path`` to a TIFF under ``folder_path/tiff``.

   :param folder_path: Folder containing the ``.npy`` files.
   :param limit: If set, stop after processing this many files.
   :returns: None


.. py:function:: generate_cellpose_train_test(src, test_split=0.1)

   Split image/mask pairs in ``src`` into ``train`` and ``test`` sibling folders.

   Only images that have a corresponding mask in ``src/masks`` are
   considered.

   :param src: Folder containing images and a ``masks`` subfolder.
   :param test_split: Fraction of pairs to route into the test set.
       Default ``0.1``.
   :returns: None


.. py:function:: parse_gz_files(folder_path)

   Group ``.fastq.gz`` files in ``folder_path`` by sample name and read direction.

   :param folder_path: Directory containing gzipped FASTQ files named
       ``<sample>_R1_...`` / ``<sample>_R2_...``.
   :returns: Mapping ``{sample_name: {"R1": path, "R2": path}}``.


.. py:function:: generate_dataset(settings=None)

   Collect PNG crops referenced by one or more measurement databases into a tar archive.

   Selects paths from each source's ``measurements.db``, optionally
   subsamples, then bundles all images in parallel into a single
   dated tar written to the first source's ``datasets/`` folder.

   :param settings: Dataset-generation settings dict. See
       ``settings.set_generate_dataset_defaults`` for keys — notably
       ``src`` (str or list of str), ``file_metadata``, ``sample``
       (int / list) and ``experiment``.
   :returns: Path to the created tar archive.
   :raises RuntimeError: if ``src`` is malformed, no images are
       selected, or the destination folder cannot be resolved.


.. py:function:: generate_loaders(src, mode='train', image_size=224, batch_size=32, classes=None, n_jobs=None, validation_split=0.0, pin_memory=False, normalize=False, channels=None, augment=False, verbose=False)

   Build ``spacrDataLoader`` objects for training, validation, or testing.

   Reads class subfolders under ``src/<mode>``, applies the requested
   transforms (channel selection, optional normalisation, optional
   augmentation) and returns loaders sized to ``batch_size``.

   :param src: Root folder containing ``train``/``test`` subfolders.
   :param mode: Which split to load — ``'train'`` or ``'test'``.
   :param image_size: Square resize target in pixels. Default ``224``.
   :param batch_size: Loader batch size. Default ``32``.
   :param classes: Ordered class names. Default ``['nc', 'pc']``.
   :param n_jobs: DataLoader worker count. Default: derived from CPU count.
   :param validation_split: Fraction of the train split to hold out.
   :param pin_memory: If True, pin batches to page-locked memory.
   :param normalize: If True, apply per-channel normalisation.
   :param channels: Subset of RGB channels to keep, e.g. ``['r', 'g']``.
   :param augment: If True, apply the training augmentation pipeline.
   :param verbose: If True, log configuration to stdout.
   :returns: For ``mode='train'``, a tuple of loaders and a plot handle;
       for ``mode='test'``, the test loader (plus optional metadata).


.. py:function:: generate_training_dataset(settings)

   Build a balanced training/testing dataset from one of:
     - metadata rules (exact matches or compound 'where' rules)
     - annotation columns (each <col>_<value> is a standalone class)
     - measurement rules (numeric ranges/bins; supports multiple conditions per class)

   New behavior (annotation mode):
     - If a column has only one annotated value (e.g., only '1's), we add a
       '<column>_random' class using unannotated rows for that column (same size as positives).
     - Optional: persist that random selection into DB as a new INT column named '<column>_random' with 1's.

   Required helpers:
     - _read_and_merge_data, _read_db (from .io)
     - generate_dataset_from_lists(dst, class_data, classes, test_split)
     - save_settings (from .utils)


.. py:function:: training_dataset_from_annotation(db_path, dst, annotation_column='test', annotated_classes=(1, 2))

   Group PNG paths by annotation value, balancing class sizes when only one is annotated.

   :param db_path: SQLite database containing a ``png_list`` table with
       an ``png_path`` column plus ``annotation_column``.
   :param dst: Output root (currently unused; kept for API symmetry).
   :param annotation_column: Column in ``png_list`` holding class labels.
       Default ``'test'``.
   :param annotated_classes: Class values to pull from the annotation
       column. When length is 1, an equal-sized "other" class is
       sampled from unannotated rows.
   :returns: List of lists — one list of PNG paths per output class.


.. py:function:: training_dataset_from_annotation_metadata(db_path, dst, annotation_column='test', annotated_classes=(1, 2), metadata_type_by='columnID', class_metadata=None)

   Same as :func:`training_dataset_from_annotation` but pre-filtered by plate metadata.

   Restricts source rows to those whose ``rowID`` or ``columnID`` is in
   ``class_metadata`` before grouping by annotation value.

   :param db_path: SQLite database with a ``png_list`` table.
   :param dst: Output root (unused; kept for API symmetry).
   :param annotation_column: Column holding class labels.
   :param annotated_classes: Class values to pull.
   :param metadata_type_by: Which metadata column to filter on —
       ``'rowID'`` or ``'columnID'``.
   :param class_metadata: Allowed values for ``metadata_type_by``.
       Default ``['c1', 'c2']``.
   :returns: List of lists — one list of PNG paths per output class.
   :raises ValueError: if ``metadata_type_by`` is not ``'rowID'`` or
       ``'columnID'``.


.. py:function:: generate_dataset_from_lists(dst, class_data, classes, test_split=0.1)

   Copy files listed per-class into ``dst/train/<class>`` and ``dst/test/<class>`` folders.

   :param dst: Output root; ``train`` and ``test`` subfolders are created.
   :param class_data: Sequence of path lists, one per class.
   :param classes: Class names paired positionally with ``class_data``.
   :param test_split: Fraction of each class routed to ``test/``.
       Default ``0.1``.
   :returns: ``(train_dir, test_dir)`` tuple of the top-level split paths.
   :raises ValueError: if ``len(class_data) != len(classes)``.


.. py:function:: convert_separate_files_to_yokogawa(folder, regex)

   Rename per-slice TIFFs in ``folder`` into the Yokogawa CV filename convention.

   Files are grouped by ``(plateID, wellID, fieldID, timeID, chanID)``
   parsed from the regex. Groups with multiple Z-slices are max-
   projected before saving; each unique source well is mapped to the
   next unused Yokogawa well ID and the mapping is logged to
   ``rename_log.csv``.

   :param folder: Folder containing the source TIFFs.
   :param regex: Pattern with named groups ``wellID`` (required) plus
       optional ``plateID``, ``fieldID``, ``timeID``, ``chanID``,
       ``sliceID``.
   :returns: None


.. py:function:: convert_to_yokogawa(folder)

   Detects file type in the folder and converts them
   to Yokogawa-style naming with Maximum Intensity Projection (MIP).


.. py:function:: apply_augmentation(image, method)

   Return ``image`` transformed by a named geometric augmentation.

   :param image: NumPy image array.
   :param method: One of ``'rotate90'``, ``'rotate180'``, ``'rotate270'``,
       ``'flip_h'``, ``'flip_v'``; any other value returns the input
       unchanged.
   :returns: Augmented image array.


.. py:function:: process_instruction(entry)

   Copy one image/mask pair described by ``entry``, applying an optional augmentation.

   :param entry: Dict with keys ``src_img``, ``src_msk``, ``dst_img``,
       ``dst_msk`` and ``augment`` (augmentation name or falsy).
   :returns: ``1`` on success — used for progress counting.


.. py:function:: prepare_cellpose_dataset(input_root, augment_data=False, train_fraction=0.8, n_jobs=None)

   Aggregate image/mask pairs from sibling dataset folders into a Cellpose training split.

   Discovers ``<input_root>/*/masks`` layouts, balances datasets to a
   common size (with augmentations if requested) and copies the
   selected pairs into ``<input_root>/cellpose_dataset/train`` and
   ``.../test``.

   :param input_root: Directory containing one subfolder per dataset.
   :param augment_data: If True, expand under-sized datasets by
       applying geometric augmentations. Default ``False``.
   :param train_fraction: Fraction of pairs routed to the train split.
       Default ``0.8``.
   :param n_jobs: Worker count for parallel copies. Default: CPU count.
   :returns: None
   :raises ValueError: if no valid ``<subdir>/masks`` datasets are found.


