Metadata-Version: 2.2
Name: dngparser
Version: 0.1.0
Summary: Fast DNG parser for numpy datasets (DJI Pocket 4 + Apple ProRAW)
Author: JonahZeng
License: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: C++
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Topic :: Multimedia :: Graphics
Classifier: Topic :: Scientific/Engineering :: Image Processing
Project-URL: Homepage, https://github.com/JonahZeng/dngparser
Project-URL: Repository, https://github.com/JonahZeng/dngparser
Project-URL: Issues, https://github.com/JonahZeng/dngparser/issues
Requires-Python: >=3.10
Requires-Dist: numpy>=2.0
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pybind11-stubgen>=2.5; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: scikit-build-core; extra == "dev"
Description-Content-Type: text/markdown

# dngparser

Fast DNG (Digital Negative) parser for machine learning datasets. Written in C++ with pybind11 bindings, returns numpy arrays with zero torch dependency.

Supports multiple DNG vendors:
- **DJI Pocket 4** — little-endian, uncompressed CFA Bayer (1ch uint16, strip-based)
- **Apple ProRAW** — big-endian, LJPEG-compressed LinearRaw (3ch uint16, tile-based)

## Features

- **C++ TIFF/DNG parser** - reads TIFF container structure, IFD chains, SubIFDs, and all DNG-specific tags directly (no libtiff dependency)
- **LJPEG decoder** - built-in lossless JPEG (SOF3) decoder with 16-bit Huffman lookup tables for fast Apple ProRAW tile decompression
- **Multi-format RAW read** - uncompressed CFA Bayer (DJI) and LJPEG-compressed tiled LinearRaw (Apple), returned as numpy uint16 arrays
- **Big-endian & little-endian** - handles both TIFF byte orders with automatic conversion
- **Full metadata extraction** - ColorMatrix (SRATIONAL-aware), BlackLevel, WhiteLevel, AsShotNeutral, NoiseProfile, ActiveArea, LinearizationTable, exposure info, opcodes
- **Numpy-first** - `load_dng()` returns plain numpy arrays and Python primitives
- **Type stubs** - ships `.pyi` files generated by pybind11-stubgen for IDE autocomplete

## Installation

```bash
pip install dngparser
```

Only depends on numpy. Install torch separately if you need it for dataset loading.

## Quick Start

```python
import dngparser

d = dngparser.load_dng("photo.dng")

# DJI Pocket 4 — CFA Bayer, 1 channel
raw = d["raw"]              # numpy.ndarray (H, W) uint16
print(d["cfa_pattern"])     # [0, 1, 1, 2] = RGGB
print(d["black_level"])     # [1024.0, 1024.0, 1024.0, 1024.0]
print(d["white_level"])     # 62400
print(d["compression"])     # 1 (uncompressed)

# Apple ProRAW — LinearRaw, 3 channels, LJPEG-compressed
d = dngparser.load_dng("IMG_5070.DNG")
raw = d["raw"]              # numpy.ndarray (H, W, 3) uint16
print(d["compression"])     # 7 (LJPEG)
print(d["bits_per_sample"]) # 12
print(d["is_tiled"])        # True
print(d["tile_width"])      # 504
print(d["linearization_table"])  # [0, 1, 2, ..., 4095]

# Shared metadata
print(d["color_matrix_1"])  # 3x3 float32, camera -> XYZ
print(d["as_shot_neutral"]) # [0.382, 1.0, 0.655]
print(d["noise_profile"])   # {"slope": 0.000258, "offset": 3.04e-08}
```

## PyTorch Dataset Example

```python
import random
import numpy as np
import torch
from torch.utils.data import Dataset
from dngparser import load_dng


class DngDataset(Dataset):
    """Paired RAW + JPG dataset for ISP training.

    Parameters
    ----------
    raw_dir : str or Path
        Directory containing .dng files.
    jpg_dir : str or Path, optional
        Directory containing corresponding .jpg files (same stem).
    crop_size : int, optional
        Random crop size for training. None = full resolution.
    normalize : bool
        If True, subtract black level and divide by white level.
    pack_bayer : bool
        If True, pack Bayer pattern into 4 channels (4, H/2, W/2).
    """

    def __init__(
        self,
        raw_dir,
        jpg_dir=None,
        crop_size=512,
        normalize=True,
        pack_bayer=True,
    ):
        from pathlib import Path

        self.raw_dir = Path(raw_dir)
        self.jpg_dir = Path(jpg_dir) if jpg_dir else None
        self.crop_size = crop_size
        self.normalize = normalize
        self.pack_bayer = pack_bayer
        self.raw_files = sorted(self.raw_dir.glob("*.dng"))

    def __len__(self):
        return len(self.raw_files)

    def __getitem__(self, idx):
        dng = load_dng(self.raw_files[idx])
        raw = torch.from_numpy(dng["raw"].copy())  # (H, W) uint16

        if self.normalize:
            raw = raw.to(torch.float32)
            black = torch.tensor(dng["black_level"], dtype=torch.float32)
            white = float(dng["white_level"])
            # Expand 2x2 black level to full image
            rows, cols = dng["cfa_repeat_rows"], dng["cfa_repeat_cols"]
            bl = black.view(rows, cols).repeat(
                raw.shape[0] // rows, raw.shape[1] // cols
            )[: raw.shape[0], : raw.shape[1]]
            raw = (raw - bl) / (white - bl.min().item())
            raw = raw.clamp(0.0, 1.0)
        else:
            raw = raw.to(torch.float32)

        if self.pack_bayer:
            # RGGB -> (4, H/2, W/2): [R, Gr, Gb, B]
            raw = torch.stack(
                [raw[0::2, 0::2], raw[0::2, 1::2],
                 raw[1::2, 0::2], raw[1::2, 1::2]],
                dim=0,
            )

        # Load JPG target if available
        if self.jpg_dir is not None:
            from PIL import Image

            jpg_path = self.jpg_dir / (self.raw_files[idx].stem + ".jpg")
            if jpg_path.exists():
                img = Image.open(jpg_path).convert("RGB")
                jpg = torch.from_numpy(np.array(img)).permute(2, 0, 1).float() / 255.0
            else:
                jpg = None
        else:
            jpg = None

        # Random crop
        if self.crop_size is not None and jpg is not None:
            h = raw.shape[-2]
            w = raw.shape[-1]
            y0 = random.randint(0, max(0, h - self.crop_size))
            x0 = random.randint(0, max(0, w - self.crop_size))
            raw = raw[..., y0:y0 + self.crop_size, x0:x0 + self.crop_size]
            jpg = jpg[..., y0:y0 + self.crop_size, x0:x0 + self.crop_size]

        result = {"raw": raw}
        if jpg is not None:
            result["jpg"] = jpg
        return result


# Usage
dataset = DngDataset(
    raw_dir="/path/to/dng_files",
    jpg_dir="/path/to/jpg_files",
    crop_size=512,
)
sample = dataset[0]
# sample["raw"]  -> (4, 512, 512) float32  (packed Bayer, normalized)
# sample["jpg"]  -> (3, 512, 512) float32  (sRGB target)
```

## Vendor Differences

| Feature | DJI Pocket 4 | Apple ProRAW |
|---------|-------------|-------------|
| Byte order | Little-endian (II) | Big-endian (MM) |
| Compression | Uncompressed (1) | LJPEG (7) |
| PhotometricInterpretation | CFA (32803) | LinearRaw (34892) |
| Channels | 1 (Bayer) | 3 (interleaved RGB) |
| Bit depth | 16 | 12 |
| Layout | Strips | Tiles (504x504) |
| LinearizationTable | absent | 4096-entry LUT |
| NoiseProfile location | Main IFD | Raw SubIFD |
| WhiteLevel type | LONG | SHORT[3] |
| Raw output shape | `(H, W)` uint16 | `(H, W, 3)` uint16 |

## What It Does NOT Do

The parser only reads and structures data. It does not apply:

- Black level subtraction
- Linearization
- Demosaicing
- White balance / color correction
- Tone mapping

These transforms belong to the differentiable ISP pipeline (planned as a separate project).

## Build from Source

Requires CMake 3.18+, a C++17 compiler, and Python 3.10+.

```powershell
python -m venv .venv
.\.venv\Scripts\pip install scikit-build-core build pybind11 numpy

# Build wheel
.\.venv\Scripts\python.exe -m build --wheel

# Install
.\.venv\Scripts\pip install --force-reinstall --no-deps dist\dngparser-0.1.0-*.whl

# Test
.\.venv\Scripts\python.exe tests\test_dng_parse.py
```

## Project Layout

```
cpp/
  include/dng/       # C++ headers: TIFF reader, LJPEG decoder, DNG parser, data structs
  src/               # C++ sources: tiff_reader.cpp, ljpeg_decoder.cpp, dng_parser.cpp
  binding/           # pybind11 module exposing load_dng()
python/dngparser/    # Python package: dng_loader.py, _dng.pyi
tests/               # End-to-end test with real DNG files (DJI + Apple)
```

## License

MIT
