Metadata-Version: 2.4
Name: scigantic-headers
Version: 0.6.0
Summary: Decode scientific file headers (cryo-EM, light microscopy and light-sheet imaging incl. TIFF/OME-TIFF, CZI, HDF5/Imaris/BigDataViewer, LIF, ND2, Zarr/N5; tabular, flow cytometry, mass spec, sequencing, structural biology) into typed fields. Pure, zero-dependency, with bounded readers that follow internal pointers without downloading the file.
Author-email: Scigantic <support@scigantic.com>
License: MIT
Project-URL: Homepage, https://scigantic.com
Project-URL: Repository, https://github.com/Scigantic/scigantic-headers
Project-URL: Issues, https://github.com/Scigantic/scigantic-headers/issues
Keywords: cryo-em,mrc,header,schema-card,instrument,context,edge,light-sheet,microscopy,ome-tiff,czi,hdf5,imaris,bigdataviewer,ome-zarr,n5,nd2,lif
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# scigantic-headers

<p align="center">
    <a href="https://github.com/Scigantic/scigantic-headers/actions/workflows/ci.yml">
        <img alt="CI" src="https://github.com/Scigantic/scigantic-headers/actions/workflows/ci.yml/badge.svg" /></a>
    <a href="https://pypi.org/project/scigantic-headers/">
        <img alt="PyPI" src="https://img.shields.io/pypi/v/scigantic-headers" /></a>
    <a href="https://pypi.org/project/scigantic-headers/">
        <img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/scigantic-headers" /></a>
    <a href="https://github.com/Scigantic/scigantic-headers/blob/main/LICENSE">
        <img alt="License" src="https://img.shields.io/github/license/Scigantic/scigantic-headers" /></a>
</p>

Read the metadata of a scientific file and return its fields (dimensions, data
type, pixel size, columns, row count) as a dict, without reading the rest of the
file. Decodes formats across cryo-EM, light microscopy and light-sheet imaging,
arrays, tabular data, flow cytometry, mass spec, sequencing, genome annotation,
and structural biology (the full list is below). The decode functions have no
dependencies; separate reader functions fetch the bytes from a file or a URL:
the leading bytes for most formats, the trailing bytes for Parquet (whose schema
is in a footer), and a bounded handful of reads wherever the pointers lead for
TIFF, HDF5, CZI, LIF and ND2 (whose metadata sits behind internal offsets).

```python
from scigantic_headers import decode_file
hdr = decode_file("session/frame.mrc")
hdr.summary            # "MRC stack 4096x4096x16, float32"
hdr.fields["dtype"]    # "float32"
hdr.fields["nz"]       # 16   (frames)
```

Zero runtime dependencies and header-only reads, so it runs air-gapped, on
amd64 or arm64, and installs into a slim image with nothing else to pull.

## Install

    pip install scigantic-headers

Published to PyPI on a `v*` tag by `.github/workflows/publish.yml`. Pin the
version in anything that consumes it (`scigantic-headers==X.Y.Z`). With no
package index at build time, install from a wheel or a checkout instead:

    pip install ./scigantic-headers

## Why it exists

You often want to know what a file is, its shape, dtype, and key acquisition
fields, without reading the whole thing. A scientific file's first ~1 KiB is a
structured header that already says so. This reads that, and nothing more.

Decoders today:

- **MRC / MRCS**: cryo-EM micrographs, movie stacks, tilt series, EMDB maps.
- **NPY**: NumPy arrays. Cross-checked against numpy.
- **NIfTI-1**: neuroimaging volumes, `.nii` and `.nii.gz`. Cross-checked against
  nibabel. Big- and little-endian.
- **CryoSPARC `.cs`**: a structured-array dataset. Reports record count and the
  field schema.
- **Parquet**: reads the footer, not a leading header, and returns the column
  names, physical types, and row count. Cross-checked against pyarrow.
- **FCS**: flow cytometry (Flow Cytometry Standard). Returns parameter and event
  counts, data type, and per-channel names. Cross-checked against flowio.
- **mzML**: mass spectrometry. Returns the spectrum count, instrument model,
  software, source file, and run start time from the XML preamble. Cross-checked
  against a compliant XML parse.
- **FASTQ**: sequencing reads. Parses the Illumina read-name convention in the
  first record for instrument, run, flowcell, and lane, plus the read length.
- **Illumina run**: RunInfo.xml / RunParameters.xml. Instrument, flowcell, run
  id, date, and read structure. Dispatched by file name, not extension.
- **VCF**: variant calls. Version, sample names, reference, and meta-line counts
  from the header.
- **SAM**: alignment header. Version, sort order, reference sequences, read
  groups, and programs.
- **PDB**: structure header. PDB id, classification, deposition date, title,
  experiment method, and resolution.
- **mmCIF / CIF**: structures (the modern PDB format). Entry id, title,
  keywords, experiment method, deposition date, and resolution (including the
  cryo-EM resolution item).
- **GenBank**: annotated sequence records. LOCUS (name, length, molecule type,
  topology), definition, accession, version, and organism.
- **GFF3 / GTF**: genome annotation. Version and a preview of the sources and
  feature types.
- **BED**: genome intervals. BED flavor (column count) and track-line presence.
- **DICOM**: medical images. Modality, dimensions, bit depth, manufacturer, and
  study/series description. Reads technical fields only, never patient
  identifiers.

Light microscopy and light-sheet (added in 0.6.0; most of these follow internal
pointers, see "Pointer-based formats" below):

- **TIFF / BigTIFF / OME-TIFF** (`.tif`, `.tiff`, `.btf`, `.tf2`, `.tf8`):
  dimensions, dtype, compression, photometric, tiling, page count (chain
  followed up to a cap, with the truncation stated). The ImageDescription is
  read where it lives and classified: **OME-XML** (sizes X/Y/Z/C/T, physical
  sizes in um, channels, instrument, objective, acquisition date, via the
  shared OME-XML reader), **ImageJ** hyperstacks (slices/channels/frames,
  spacing and unit), **Micro-Manager** (summary JSON, private header, or tag
  51123), ImSpector/LaVision stacks by software name. Cross-checked against
  tifffile.
- **Zeiss CZI**: file header, metadata segment (XML, wherever the writer put it,
  including 490 MB into a light-sheet file), directory and attachment counts.
  Sizes X/Y/Z/C/T plus scenes, tiles, illuminations, views, rotations, phases;
  pixel type, scaling in um, channels and exposure, acquisition mode (SPIM,
  lattice light-sheet, widefield), microscope, objective, software. Verified on
  LSM 980, Axioscan, Lightsheet Z.1 and Lattice Lightsheet 7 files; cross-checked
  against czifile.
- **HDF5** and the imaging formats built on it, without libhdf5: superblocks
  v0 to v3, object headers v1 and v2, symbol tables and dense (fractal heap plus
  v2 B-tree) links and attributes, dataspace, datatype, layout, filters.
  Attributes are read from the object header the links point to, never scanned
  for, so a relocated header's stale copy cannot leak in.
  - **Imaris `.ims`**: X/Y/Z from DataSetInfo/Image, voxel size from the extents,
    channels with emission wavelengths, resolution levels, timepoints, the
    level-0 block dtype/chunks/compression, objective, recording date. An `.ims`
    that is a TIFF (Imaris IMS3) is reported as such through the TIFF decoder.
  - **BigDataViewer `.h5`**: setup and timepoint counts, the per-setup
    resolutions and subdivisions tables, the level-0 cells block shape (z, y,
    x), dtype and compression. Two Range requests describe a 51 GB file.
  - **Luxendo `.lux.h5`**: the generic structure labelled, with the `Data`
    block shape. No public sample was available; shape only.
  - **generic `.h5` / `.hdf5` / `.he5`**: root attributes (fixed and
    variable-length strings, numbers), groups, and datasets with shape, dtype,
    layout, chunks and filters, bounded and with the truncation stated.
- **KLB** (Keller Lab Block): the 319-byte header: X/Y/Z/C/T, dtype, bzip2 /
  zlib / none, block size, pixel size (KLB stores no unit; micrometres by
  convention), the 256-character metadata field.
- **ICS** (Image Cytometry Standard, `.ics` + `.ids`): version, layout order and
  sizes, dtype, byte order, compression, scale converted to um, units, labels,
  sensor/history entries (Huygens writes the objective NA there).
- **Leica LIF**: the UTF-16 XML header: image count and names, dimensions by
  DimID, dtype from channel resolution, voxel size from physical lengths,
  channels, objective, magnification, NA, software version. Cross-checked
  against readlif.
- **Nikon ND2**: the chunk map at the end of the file leads to the attribute,
  calibration, text-info and metadata chunks (both the Ver2 XML and the Ver3
  LiteVariant encodings are decoded for the needed values): width, height,
  components, bit depth, frame count, experiment loops (Z/T/positions), um per
  pixel, z step, channels, objective and NA, acquisition date, camera,
  compression. About 125 KB of a 218 MB file is read. Cross-checked against the
  nd2 package. Legacy JPEG 2000 ND2 is recognised only.
- **Imspector OBF / `.msr`** (Abberior, LaVision BioTec): file header, stack
  chain (count, names, rank, sizes, dtype, compression, physical lengths to um
  for the axes the footer marks as metres, axis labels). Verified on five real
  Abberior STED `.msr` files (hoerlteam/msr-reader test data) and cross-checked
  against obffile; no ImSpector-written LaVision `.msr` was available.
- **Zarr stores**: OME-Zarr / OME-NGFF 0.4 (`.zattrs`) and 0.5 (`zarr.json`,
  Zarr v3): axes, levels, level-0 scale to um, channels, plate/labels markers;
  plain `.zarray` / `.zgroup` arrays and groups (shape, chunks, dtype, codec);
  **N5** `attributes.json` (dimensions, block size, dtype, compression, pixel
  resolution). These are dispatched by file name. A store *directory* decodes
  through the sidecar it contains, and an OME-Zarr store also merges its level-0
  array's shape and dtype, so `decode_file("image.zarr")` reads like one image.
  Cross-checked against zarr on IDR data.
- **XML sidecars**: the generic `.xml` extension is now claimed by a dispatcher
  that sniffs the root element and returns None for anything it does not know:
  **BigDataViewer / BigStitcher `SpimData`** (loader format, data file, setups,
  timepoints, size, voxel size, channels, illuminations, angles, tiles,
  registrations), **OME-XML** sidecars and companion files (`.ome.xml`, same
  fields as OME-TIFF), and **EPU** `MicroscopeImage` (previously opt-in only).
  Illumina RunInfo.xml / RunParameters.xml keep their file-name dispatch, which
  wins over the extension.

`.gz` is transparent. The reader decompresses just the leading block, so a
gzipped `.nii.gz` or `.mrc.gz` decodes without inflating the whole file, and
dispatch sees the inner format.

Adding a format is a pure `bytes -> DecodedHeader | None` function plus one
`register_decoder` call. That call also declares how the bytes are fetched
(`read=Read(leading=...)`, `read=Read(footer=...)` for a schema in the tail like
Parquet, or `read=Read(seek=budget)` for a pointer-based format, whose decoder
then takes a `Source` instead of bytes), so the read strategy lives next to the
decoder, not in a table elsewhere. Nothing about the dispatch, the readers, or
the batch path is format-specific: the readers ask the registry how to read and
never name a format. NPY and NIfTI landed with zero plumbing changes.

## Pointer-based formats

TIFF keeps its image directories wherever the writer put them (libtiff and
Bio-Formats put them after the pixel data); HDF5 is a graph of object headers
reached from a superblock; a CZI's metadata segment can sit 490 MB into the
file; an ND2 starts from a chunk map at its end. No fixed leading or trailing
read decodes these, so a decoder for one of them takes a `Source`: a
random-access view with `read_at(offset, n)` and `read_tail(n)` that is total
(short reads instead of errors) and bounded three ways: a byte budget declared
with the decoder, a cap on reads, and, over HTTP, a cap on Range requests
(`URL_MAX_REQUESTS`). Reads from a URL are coalesced into 64 KiB cached chunks,
so a walk over neighbouring structures costs one round trip. The decode is
still a few KB of a file of any size; it is just not the first few KB. Measured
on the OME sample server:

    BDV drosophila.h5 (620 MB)                2 Range requests
    BDV WB.h5 (51 GB)                         2 Range requests
    Imaris Convallaria .ims (7 MB)            6 Range requests
    CZI MouseBrain light-sheet (492 MB)       5 Range requests (metadata at 492 MB)
    OME-TIFF tubhiswt (270 KB, 1 page)        3 Range requests
    OME-TIFF multi-channel .btf (21 pages)    3 Range requests (chain not chased)
    Leica LIF PR2729 (227 KB)                 2 Range requests
    Nikon ND2 sample_image (218 MB)           6 Range requests
    KLB img.klb                               2 Range requests

A plain TIFF whose directories trail each plane costs a request per page until
the chain cap stops it. An OME-TIFF states its plane count in the XML, so its
chain is not chased at all: `planeCount` comes from the OME block and
`pageCount` stops at the first directory, marked truncated.

`decode_bytes(key, data)` still works for these formats (the bytes become an
in-memory Source), and `BytesSource(ranges={offset: bytes})` replays a sparse
recording, which is how the golden fixtures hold a 500 MB file's few KB of
metadata (`tests/fixtures/*.sparse.json`).

## Pixel size for raw movies

### Cross-check the acquisition settings

Voltage, pixel size, dose and frame count usually arrive from several places typed
or exported by different people. A wrong one does not fail; it quietly caps the
resolution. `compare_acquisition` lines the sources up and reports disagreements:

```python
from scigantic_headers import compare_acquisition, decode_epu_xml, read_star_optics

epu = decode_epu_xml(open("FoilHole_..._Data_....xml", "rb").read()).fields
for f in compare_acquisition({
    "sheet": {"voltageKv": 300, "pixelSizeA": 0.84, "doseEA2": 42.4, "fractions": 40},
    "epu": epu,
    "relion": read_star_optics("Import/movies.star"),
}, target_resolution_a=3.3):
    print(f.severity, f.field, f.message)
```

It does not decide which source is right. EPU records the pixel size of the binned
readout grid, so `epu_pixel_size_candidates(epu)` lists the likely values for the
saved movie; compare with the movie's own dimensions. `decode_epu_xml` is not
auto-dispatched (`.xml` is too generic), so call it on a file you know is EPU's.

A raw MRC movie header has no pixel size: CELLA is 0, so `decode_*` returns
`pixelSizeA` as `None`. The value is in the RELION STAR or CryoSPARC `.cs` file
the workflow writes in the session, and the library reads it. `read_session_optics`
finds that file next to (or above) the data file and returns the optics:

```python
from scigantic_headers import decode_file, read_session_optics

hdr = decode_file("Movies/frame.mrc")       # header fields; pixelSizeA is None
opt = read_session_optics("Movies/frame.mrc")
# {'pixelSizeA': 1.05, 'voltageKv': 300.0, 'source': 'relion-star:particles.star'}
```

`read_star_optics(path)` and `read_cryosparc_optics(path)` read a specific optics
file when you already know which one. The RELION 3.1 `data_optics` loop, the
older key-value form, optics columns in the main data loop, and the legacy
detector-pixel-size / magnification pair are all handled.

## Layout

    src/scigantic_headers/
      decoders.py   pure core: registry + MRC / NPY / NIfTI / .cs decoders
      parquet.py    Parquet footer decode (small Thrift-compact reader)
      fcs.py        FCS (flow cytometry) header + TEXT-segment decode
      mzml.py       mzML (mass spec) XML-preamble metadata decode
      fastq.py      FASTQ first-record / read-name decode
      illumina.py   Illumina RunInfo.xml / RunParameters.xml decode
      vcf.py        VCF header decode
      sam.py        SAM alignment-header decode
      pdb.py        PDB structure-header decode
      mmcif.py      mmCIF / CIF structure-header decode
      genbank.py    GenBank flat-file header decode
      gff.py        GFF3 / GTF annotation-header decode
      bed.py        BED interval-file decode
      dicom.py      DICOM technical-metadata decode (no patient data)
      tiff.py       TIFF / BigTIFF / OME-TIFF / ImageJ / Micro-Manager (seeking)
      ome_xml.py    shared OME-XML reader (OME-TIFF, .ome.xml, companion files)
      czi.py        Zeiss CZI segments + metadata XML (seeking)
      hdf5.py       HDF5 walker; Imaris, BigDataViewer, Luxendo, generic (seeking)
      lif.py        Leica LIF XML header (seeking)
      nd2.py        Nikon ND2 chunk map + LiteVariant chunks (seeking)
      obf.py        Imspector OBF / .msr stack chain (seeking)
      klb.py        Keller Lab Block header
      ics.py        Image Cytometry Standard header
      zarr.py       OME-Zarr / Zarr v2+v3 / N5 sidecars (by file name)
      xml_sidecar.py  .xml dispatcher: BDV SpimData, OME-XML, EPU
      sources.py    read bytes from file or URL (leading, footer, or seeking); bounded parallel batch
      star.py       RELION STAR optics reader
      cryosparc.py  CryoSPARC .cs optics reader
      optics.py     read_session_optics: find a data file's optics file
      cli.py        scigantic-headers <file|url|--dir>
      benchmark.py  scigantic-headers-bench, measures the speed levers
    tests/          pytest (440 tests, incl. fuzz, golden and sparse real-file fixtures)

## Robustness

A decoder is handed the leading bytes of arbitrary files, so it must be total:
for any input it returns None or a valid, JSON-safe DecodedHeader. It never
raises and never emits NaN or inf. This is enforced, not assumed.

- **Fuzz** (`test_fuzz.py`): thousands of random and magic-seeded byte buffers
  through every decoder. Each result must serialize with `allow_nan=False`.
- **Golden fixtures** (`fixtures/mrc-cases.json`): hex input to exact decode. A
  change to a decoder that the fixture does not expect fails a test rather than
  passing silently.
- Non-finite floats (a garbage NIfTI pixdim, an inf MRC cell) sanitize to null.

## Speed

The decode is microseconds: a 1 KiB read and a few `struct` unpacks. Optimizing
that is a rounding error. All the speed is in I/O, and two things carry it. Both
are measured, not asserted (run `scigantic-headers-bench`).

**Read the header, not the file.** The read is bounded (`HEADER_BYTES`, 1 KiB, for
most formats; a few hundred KB for text formats with long headers, declared per
decoder), so its cost is independent of file size and a multi-GB movie decodes as
fast as a small one.

    header read + decode (1 KiB):        15 us
    full-file read + decode (200 MB):  22.7 ms      (about 1,500x cheaper)

**Parallelize the I/O, bounded.** Header reads across files are independent and
I/O-bound, so a thread pool overlaps their latency (the read releases the GIL).
Measured over 8 headers pulled by HTTP Range from a remote archive:

    serial   (1 worker):   25.7 s
    parallel (8 workers):   0.7 s        (about 35x)

That 35x is large because the server's per-request latency is about 3 s, so
overlapping eight requests recovers a lot of dead wait. On low-latency local
storage the speedup is smaller. The win scales with per-read latency, which is
why the pool is bounded and tunable (default 8; past a point more connections
hit server throttling or disk-queue thrash and get slower).

Two more, by construction:

- **Filter before I/O.** `has_decoder_for` is a cheap extension check. The batch
  and walk paths use it to skip a file before any syscall.
- **Zero deps, fast import.** The dtype table is a plain dict, not numpy. There
  is nothing to install and nothing to import, and it runs air-gapped.

The biggest system-level lever lives in the caller, not here: decode once when a
file lands, cache the small result, and answer every later query from it without
re-decoding.

## Use

    from scigantic_headers import decode_file, decode_paths, decode_urls, iter_decodable_files

    decode_file("s/frame.mrc")                              # one local file
    decode_file("acq/image.zarr")                           # a Zarr or N5 store directory
    decode_paths(iter_decodable_files("/data/sessions"))    # a tree, parallel
    decode_urls([...], workers=8)                           # remote, by Range

    # CLI
    scigantic-headers session/frame.mrc
    scigantic-headers https://example.org/path/img.mrcs
    scigantic-headers https://example.org/path/sample.ims     # HDF5 over Range
    scigantic-headers --dir /data/sessions --workers 8

    pytest
    scigantic-headers-bench            # reproduce the numbers above

## Limitations

- **HDF5 is walked, not fully parsed.** Groups and attribute sets are read
  from compact storage, symbol tables, and dense storage with v2 B-trees of
  depth 0 or 1 and fractal heaps with direct blocks; a group with thousands of
  links (a BDV file with thousands of timepoints) is listed up to a cap, with
  the truncation stated. Huge heap objects, virtual datasets, external links
  and soft links are not followed. Attribute values are decoded for strings and
  small numeric arrays; others are reported by type.
- **Vendor formats without a public sample** (Micro-Manager-native TIFF,
  ImSpector TIFF, Luxendo `.lux.h5`) were implemented from the published
  layouts and tested on constructed files; their tests say so. Olympus
  OIB (an OLE compound document) and OIR, Hamamatsu DCIMG, and raw `.bin` /
  `.dat` / `.raw` exports (which have no header to read) are not decoded.
- **Legacy ND2** (JPEG2000 based) is recognised but not described.
- BAM's BGZF framing and per-record structure are not handled; plain gzip is.
- **Output is not a standard.** The fields are an ad-hoc dict, not an
  interchange format such as Allotrope ASM. Map the dict to a standard if you
  need one.

## License

MIT. See LICENSE.
