Metadata-Version: 2.2
Name: lumif
Version: 1.0.0rc1
Summary: Read and write LUMIF-1 (.lum), the lossless Lumen astronomical image format
Keywords: astronomy,astrophotography,fits,lossless,lumen
License: Copyright (c) 2026 Delta3. All rights reserved.
         
         LIBLUMIF BINARY LICENSE AGREEMENT
         =================================
         
         This agreement governs the use of liblumif, the LUMIF-1 reading and writing
         library, supplied as a header file ("Header"), a pre-built library ("Library")
         and its documentation (together, the "Software"). The Software is the
         proprietary property of Delta3 ("Owner"). It is licensed, not
         sold. By installing, copying or using the Software you accept this agreement;
         if you do not accept it, do not use the Software.
         
         
         1. LICENSE GRANT
         
            The Owner grants you a non-exclusive, non-transferable, worldwide,
            royalty-free license to:
         
            a. Use the Header and the Library to build applications, plug-ins and
               scripts that read and write LUMIF-1 (.lum) files, including commercial
               and closed-source ones ("Your Application").
            b. Include the Library, unmodified and in the form received, in Your
               Application and distribute it to end users together with Your
               Application, provided that this agreement (or a notice that refers to
               it) accompanies the Library.
            c. Make copies of the Software needed for backup and for your internal
               development and testing.
         
            No fee is due for these rights. The Owner may offer separate written
            licenses for other uses.
         
         
         2. SOURCE CODE AND RESTRICTIONS
         
            The source code of the Software is a trade secret of the Owner and is not
            licensed to you. Except where section 3 or mandatory law allows it, you may
            not, and may not allow others to:
         
            a. Access, request, obtain or disclose the source code.
            b. Reverse engineer, decompile, disassemble, or otherwise attempt to
               derive the source code, algorithms, structure or internal data of the
               Library.
            c. Modify, patch, adapt, translate or create derivative works of the
               Library, or extract parts of it.
            d. Distribute the Library on its own, as a standalone product or as part of
               a library, SDK or toolkit that is offered as a substitute for it;
               sell, rent, lease, sublicense or lend the Library separately from Your
               Application.
            e. Use the Software to build a product whose main purpose is to provide a
               competing LUMIF-1 library to third parties by wrapping or repackaging
               the Library.
            f. Remove or alter copyright, license or attribution notices.
            g. Use the Software unlawfully, or to infringe the rights of others.
         
            The LUMIF-1 file format itself may be read and written by any software.
            Nothing in this agreement restricts you from using the Library to access
            your own .lum files and the .lum files of others.
         
         
         3. MANDATORY LAW
         
            Where applicable law (for example Article 6 of Directive 2009/24/EC) gives
            you a right that cannot be excluded by contract, such as observing,
            studying or decompiling the Library to the extent strictly necessary to
            achieve interoperability, this agreement does not remove that right. Any
            information obtained that way may only be used for that purpose and may not
            be disclosed or used to build software that is substantially similar to the
            Library.
         
         
         4. OWNERSHIP
         
            The Owner keeps all rights, title and interest in the Software, including
            all intellectual property rights. No right is granted except as stated in
            this agreement. Feedback you send to the Owner may be used by the Owner
            without restriction or compensation.
         
         
         5. THIRD-PARTY SOFTWARE
         
            The Library contains no third-party code.
         
         
         6. TERMINATION
         
            This agreement ends automatically if you breach it. On termination you
            must stop using the Software and delete all copies, except that end users
            who received the Library lawfully with Your Application may keep using it
            for as long as they comply with this agreement. Sections 2, 4, 7, 8 and 9
            survive termination.
         
         
         7. DISCLAIMER OF WARRANTY
         
            THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
            IMPLIED, INCLUDING THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
            PARTICULAR PURPOSE, NON-INFRINGEMENT AND ACCURACY OF DATA. YOU ARE
            RESPONSIBLE FOR KEEPING BACKUPS OF YOUR DATA.
         
         
         8. LIMITATION OF LIABILITY
         
            TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE OWNER IS NOT LIABLE FOR ANY
            INDIRECT, INCIDENTAL, SPECIAL OR CONSEQUENTIAL DAMAGES, OR FOR LOSS OF
            DATA, PROFITS OR BUSINESS, ARISING FROM THE USE OF OR INABILITY TO USE THE
            SOFTWARE, EVEN IF ADVISED OF THEIR POSSIBILITY. THE OWNER'S TOTAL LIABILITY
            IS LIMITED TO THE AMOUNT YOU PAID FOR THE SOFTWARE, WHICH IS ZERO. THIS
            SECTION DOES NOT LIMIT LIABILITY THAT CANNOT BE LIMITED BY LAW.
         
         
         9. GENERAL
         
            a. This agreement is governed by the laws of Italy, excluding its conflict-of-law rules, and the
               courts of Latina, Italy have exclusive
               jurisdiction, except where mandatory consumer law says otherwise.
            b. If a provision is found unenforceable, the rest remains in force.
            c. You may not assign this agreement without the Owner's written consent,
               except together with Your Application as a whole.
            d. This is the entire agreement about the Software. Changes must be in
               writing and signed by the Owner.
         
         Contact: Delta3, francescocogliandro@icloud.com
         
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: License :: Other/Proprietary License
Requires-Python: >=3.9
Requires-Dist: numpy
Description-Content-Type: text/markdown

# liblumif

Reference **C99** implementation of **LUMIF-1** (`.lum`), the lossless Lumen image format for
astronomical FITS data. It reads *and* writes the format, so Siril, PixInsight, Python (ctypes /
cffi) or any other program can use it. It has no dependency (revision 2 of the format dropped zstd).

* Format specification: [`docs/LUMIF-1-spec.md`](docs/LUMIF-1-spec.md)
* Rust reference: `crates/lumen-core/src/io/lum/`; golden files: `crates/lumen-core/tests/golden/`

## What it does

| | |
|---|---|
| decode | whole image, or any band of rows (only the touched tiles are read and decoded), from memory or streaming from a file; all payload modes, all predictors, range coder, palette, bit shift, CRC-32 checks |
| encode | planar `float32` (mono / RGB / CFA, NaN and every bit pattern preserved), or raw `u8/i16/u16/i32/f32/f64` buffers; `u16`-affine / palette / `f32` payload picked automatically and verified sample by sample |
| FITS | `lumif_encode_fits()` (keeps the original header) and `lumif_to_fits()` rebuild the source FITS **byte for byte** (BITPIX 8/16/32/−32/−64, 2-D or 3 planes) |
| errors | every function returns a `lumif_status`; nothing aborts, prints or leaves partial output |

## Build

CMake ≥ 3.16, any C99 compiler. Tested with MSVC 2022 (generator "Visual Studio 17 2022").

```bash
cmake -S libs/liblumif -B build -G "Visual Studio 17 2022" -A x64    # Windows
cmake --build build --config Release
# Linux / macOS (gcc or clang):
cmake -S libs/liblumif -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build
```

Options: `LUMIF_BUILD_TOOLS` (ON, builds `lumif-cat`), `LUMIF_SHARED` (OFF), `LUMIF_BUILD_FUZZ` (OFF).

**FMA warning for embedders.** The affine `u16` maps and the FITS scaling must be evaluated as
separate IEEE double operations. The code stores intermediates in `volatile` doubles, and the
CMake file adds `-ffp-contract=off`; if you compile the sources yourself, keep contraction off.

To use it from your own project: add `include/` to the include path and compile
`src/*.c`, or link the `lumif` CMake target.

## Example

```c
#include "lumif.h"

/* ---- read: whole image, then only rows 1000..1064 of plane 0 ---- */
lumif_reader *r;
lumif_status st = lumif_open_file("m33.lum", NULL, &r);          /* NULL = default limits */
if (st) { fprintf(stderr, "%s\n", lumif_strerror(st)); return 1; }
const lumif_info *in = lumif_get_info(r);
float *px = malloc((size_t)in->width * in->height * in->planes * sizeof *px);
st = lumif_read_all(r, px);                                       /* planar float32 */
st = lumif_read_rows(r, 0, 1000, 1064, px);                       /* partial: (64 * width) floats */
for (size_t i = 0; i < in->n_cards; i++)
    printf("%s = %s\n", in->cards[i].key, in->cards[i].value);
lumif_close(r);

/* ---- write: 16-bit sensor frame (raw u16, 6248 x 4176) ---- */
lumif_write_params p;
lumif_write_params_init(&p);
p.width = 6248; p.height = 4176; p.channels = LUMIF_CFA_RGGB;
uint8_t *out; size_t n;
st = lumif_encode_raw(&p, LUMIF_RAW_U16, raw_u16, 1.0, 0.0, &out, &n);
if (!st) { st = lumif_write_file("light.lum", out, n); lumif_free(out); }

/* ---- FITS in, FITS out (byte exact) ---- */
st = lumif_encode_fits(fits_bytes, fits_len, /*keep_origin*/ 1, /*effort*/ 1, &out, &n);
/* ... later ... */
lumif_open_memory(out, n, NULL, &r);
st = lumif_to_fits(r, &fits_again, &fits_again_len);              /* == fits_bytes */
```

Pixels are `float32` planar. Integer FITS data map to float exactly as the format defines
(`((raw*bscale)+bzero)/range`, see `lumif_raw_to_float`); `LUMIF_RAW_U16` is `v/65535`. Note that
`lumif_encode_raw(LUMIF_RAW_F32 …)` applies that FITS formula (`-0.0 + 0.0 = +0.0`); pass float
pixels to `lumif_encode()` directly when you need `-0.0` kept.

On Windows `lumif_open_file` / `lumif_write_file` take paths in the ANSI code page (`fopen`); for
Unicode paths read the file yourself (e.g. `_wfopen`) and use `lumif_open_memory`.
`lumif_set_threads(r, n)` (0 = one per CPU) decodes the tiles of a band on n threads (pthread / Win32; compile with `-DLUMIF_NO_THREADS` to remove): identical output, 16.8 MB M33 light 0.52 s -> 0.28 / 0.16 / 0.10 s with 2 / 4 / 8 threads. A reader is otherwise not thread-safe; use one reader per calling thread. All input is untrusted: sizes are
overflow-checked and bounded by the file size and by `lumif_limits` (default 2³⁴ samples, 2²⁴
samples per tile, 64 MiB of header). Hostile files can still make the decoder spend time on large
declared images; lower `max_samples` if that matters.

## `lumif-cat`

```
lumif-cat info    f.lum                   geometry, payload, header cards
lumif-cat check   f.lum                   decode whole + by row bands, print pixel CRC-32
lumif-cat dump    f.lum out.f32           planar little-endian float32
lumif-cat to-fits f.lum out.fits          byte-exact FITS rebuild
lumif-cat encode  in.fits out.lum [--no-origin] [--fast]
```

## Verification (no unit tests: real data)

```bash
python libs/liblumif/tools/make_synthetic.py D:/lumif_synth         # 15 small FITS covering every code path
python libs/liblumif/tools/verify.py --cat build/Release/lumif-cat.exe --lumen target/release/lumen.exe \
       --golden crates/lumen-core/tests/golden  D:/lumif_synth  D:/BenchData/fits --limit 8
```

For each FITS it checks that (1) the Rust encoder's `.lum` decodes in C to the identical FITS,
(2) the C encoder's `.lum` is decoded by the Rust CLI to the identical FITS, (3) C → C, (4) row-band
reads equal the whole-image read. Results of the last run: see "Results" below.

Fuzzing (`-DLUMIF_BUILD_FUZZ=ON`): `lumif-fuzz` mutates real `.lum` files (bit flips, interesting
32-bit values, truncation, chunk moves) and decodes them with small limits, the input sitting in
front of a guard page so any over-read faults; `lumif-fuzz-nocrc` is the same with the checksums
disabled so mutated payloads reach the range-coder / predictor paths. Run it with the Debug
build (MSVC debug CRT checks heap overruns) or with ASan/UBSan where available.

## Compression benchmark (public data)

47 public FITS images (744 MB; ESO 19, NOIRLab 16, MAST 12), one machine, external codecs on one thread. Every codec was decompressed and checked bit for bit; LUMIF-1 round-tripped 47/47 files. **Ratio** = FITS size / compressed size. **Size** = compressed size relative to LUMIF-1 on the same files (1.00 = equal, higher = larger). Speeds are MB/s of FITS data. Reproduce with `tools/bench_lossless/`.

### 16-bit integer frames (22 files, 12.7 MB average)

| Codec | Files | Ratio | Size | Compress MB/s | Decompress MB/s |
|---|--:|--:|--:|--:|--:|
| **LUMIF-1** | 22 | 2.95 | 1.000 | 23.0 | 19.8 |
| JPEG XL (lossless, effort 7) | 22 | 2.85 | 1.037 | 2.5 | 26.7 |
| JPEG XL (CFA planes split) | 22 | 2.82 | 1.047 | 1.8 | 26.4 |
| JPEG-LS (CharLS) | 22 | 2.80 | 1.056 | 93.7 | 116.3 |
| JPEG 2000 (lossless 5/3) | 22 | 2.79 | 1.059 | 19.9 | 22.8 |
| JPEG-LS (CFA planes split) | 22 | 2.76 | 1.069 | 91.8 | 117.7 |
| bzip2 -9 | 22 | 2.76 | 1.071 | 17.3 | 23.3 |
| xz -9e (LZMA) | 22 | 2.66 | 1.109 | 2.9 | 58.4 |
| XISF zstd-19 + shuffle | 22 | 2.63 | 1.123 | 7.4 | 871.3 |
| fpack HCOMPRESS lossless | 22 | 2.63 | 1.123 | 66.3 | 52.7 |
| brotli -q 11 | 22 | 2.54 | 1.162 | 0.7 | 87.9 |
| fpack Rice | 22 | 2.53 | 1.166 | 167.4 | 142.5 |
| XISF zlib-9 + shuffle | 22 | 2.50 | 1.181 | 48.5 | 556.5 |
| zstd -19 | 22 | 2.43 | 1.213 | 3.3 | 170.1 |
| fpack GZIP-2 (shuffle) | 22 | 2.24 | 1.319 | 47.1 | 116.3 |
| gzip -9 | 22 | 2.12 | 1.392 | 5.6 | 113.4 |
| XISF LZ4-HC + shuffle | 22 | 2.08 | 1.417 | 64.8 | 3628.1 |
| fpack GZIP-1 | 22 | 1.87 | 1.580 | 37.3 | 109.2 |


### Float (f32) images (21 files, 18.7 MB average)

| Codec | Files | Ratio | Size | Compress MB/s | Decompress MB/s |
|---|--:|--:|--:|--:|--:|
| **LUMIF-1** | 21 | 1.78 | 1.000 | 30.1 | 70.1 |
| JPEG XL (lossless, effort 7) | 17 | 1.80 | 1.011 | 18.0 | 77.4 |
| XISF zstd-19 + shuffle | 21 | 1.74 | 1.020 | 7.8 | 1939.7 |
| XISF zlib-9 + shuffle | 21 | 1.72 | 1.032 | 13.8 | 951.7 |
| xz -9e (LZMA) | 21 | 1.70 | 1.047 | 3.1 | 46.3 |
| bzip2 -9 | 21 | 1.66 | 1.074 | 14.2 | 18.2 |
| XISF LZ4-HC + shuffle | 21 | 1.62 | 1.099 | 18.8 | 4832.8 |
| brotli -q 11 | 20 | 1.51 | 1.114 | 0.6 | 90.9 |
| fpack GZIP-2 (shuffle) | 20 | 1.57 | 1.138 | 45.4 | 126.4 |
| zstd -19 | 21 | 1.51 | 1.174 | 4.9 | 244.7 |
| gzip -9 | 21 | 1.40 | 1.270 | 11.4 | 112.8 |
| fpack GZIP-1 | 20 | 1.36 | 1.311 | 32.3 | 105.4 |

Codecs that could not round-trip some files: fpack gzip-1 (1), fpack gzip-2 (shuffle) (1); lumif-1 is the smallest on 8 of 21 files; rows with fewer files are compared on their own subset.

### 32-bit integer frames (4 files, 17.9 MB average)

| Codec | Files | Ratio | Size | Compress MB/s | Decompress MB/s |
|---|--:|--:|--:|--:|--:|
| **LUMIF-1** | 4 | 4.35 | 1.000 | 42.9 | 40.5 |
| JPEG-LS (CharLS) | 4 | 4.24 | 1.028 | 159.3 | 184.7 |
| JPEG-LS (CFA planes split) | 4 | 4.22 | 1.031 | 159.6 | 188.2 |
| JPEG 2000 (lossless 5/3) | 4 | 4.22 | 1.032 | 34.4 | 38.2 |
| JPEG XL (lossless, effort 7) | 4 | 4.16 | 1.048 | 1.6 | 48.5 |
| JPEG XL (CFA planes split) | 4 | 4.14 | 1.052 | 1.5 | 47.8 |
| fpack HCOMPRESS lossless | 4 | 4.11 | 1.060 | 119.2 | 86.4 |
| bzip2 -9 | 4 | 4.11 | 1.061 | 27.8 | 30.6 |
| fpack Rice | 4 | 3.98 | 1.093 | 300.7 | 194.2 |
| XISF zstd-19 + shuffle | 4 | 3.91 | 1.115 | 11.9 | 2901.9 |
| XISF zlib-9 + shuffle | 4 | 3.89 | 1.119 | 18.7 | 1116.5 |
| xz -9e (LZMA) | 4 | 3.89 | 1.120 | 2.7 | 82.1 |
| brotli -q 11 | 4 | 3.70 | 1.177 | 0.8 | 168.6 |
| XISF LZ4-HC + shuffle | 4 | 3.65 | 1.191 | 33.3 | 10618.8 |
| zstd -19 | 4 | 3.52 | 1.238 | 3.6 | 296.9 |
| fpack GZIP-2 (shuffle) | 4 | 3.40 | 1.281 | 67.3 | 141.6 |
| gzip -9 | 4 | 2.87 | 1.520 | 1.3 | 152.1 |
| fpack GZIP-1 | 4 | 2.38 | 1.828 | 41.6 | 123.0 |


### By archive

| Archive | Files | Avg FITS MB | LUMIF-1 ratio | fpack size / LUMIF-1 | xz size / LUMIF-1 |
|---|--:|--:|--:|--:|--:|
| ESO | 19 | 19 | 2.02 | 1.15 | 1.07 |
| NOIRLab | 16 | 16 | 3.63 | 1.09 | 1.10 |
| MAST | 12 | 11 | 1.56 | 1.08 | 1.04 |

## What the encoder simplifies

The C encoder produces valid files that the Rust reader decodes identically, but is simpler:

| simplification | cost |
|---|---|
| predictor chosen by residual bit-length only (Rust also kept the runner-up in the past, +0.01 %) | ≈ 0.01 % |
| palette limited to 65536 distinct values (Rust 2²⁴) | rare |
| encoder single threaded | slower than the multi-threaded Rust encoder |

## Results

Measured before revision 2 (zstd removed); after it C and Rust files have identical sizes on the
verification corpus (ratio 1.000) and the encoders gained 7-40 % speed.

Last run (MSVC 2022 Release build vs Rust `lumen.exe` of the same checkout):

* **golden**: 5/5 `.lum` decode in C to the committed `.fits`, byte for byte.
* **FITS corpus**: 5 golden + 15 synthetic (`make_synthetic.py`) + 61 public files of
  `tools/bench_lossless` (ESO, NOIRLab, HST, JWST; u16 / i16 / f32) + 5 real M33 files (16 MB light,
  master dark, 97 MB f32 stack, 65 MB RGB result): in all four directions (Rust enc → C dec, C enc → Rust
  dec, C → C, row bands = whole image) the FITS is byte-identical. 86/86 OK.
* **Size, C encoder / Rust encoder (zstd level 3)**: median 1.002; camera frames 1.000–1.015; M33 light
  6.33 vs 6.34 MB; 97 MB f32 stack 70.91 vs 70.90 MB; four VIMOS files are 0.3 % *smaller*. Worst cases:
  the 65 MB RGB result 1.055 (Rust picks a 32-bit palette / zstd where C codes f32), a JWST f32 rate
  frame 1.12, files below 0.1 MB (header-dominated, up to 1.9×; f32_const 1.57×), and f64 / scaled-i32
  FITS whose origin *residual* is noisy (1.10×) or very regular (the synthetic `i32_ramp`: Rust's zstd
  makes the residual vanish, C stores it raw, 0.07 MB vs ~0 MB).
* **Speed** (MSVC /O2, M33 light 16.8 MB FITS): decode 0.5 s on 1 thread (26–38 MB/s of FITS-equivalent
  data), 0.08 s with `lumif_set_threads(r, 0)` on this machine, encode 1.5 s (5.6 s for 82 MB, 8.7 s for the 97 MB f32 stack). The Rust CLI is multi-threaded.
* **Robustness**: `lumif-fuzz` / `lumif-fuzz-nocrc` (50 000 mutated files of the golden + 30 real
  C/Rust-written `.lum`, clang 23 AddressSanitizer + UBSan, input before a guard page): no crash, no
  hang, no sanitizer report (the checksum-less variant decodes ~12 % of the mutants all the way through the
  decoders). 4 400 random-image encode → decode round trips (any float bit pattern, NaN, ±0, inf, palettes,
  shifts, CFA/RGB, odd sizes, random row bands) under the same sanitizers: 0 failures.
* **Compilers**: MSVC 2022 `/W4` and clang 23 `-Wall -Wextra`: no warnings in the library. gcc was not
  available on this machine (the code is plain C99, no extensions).

Not done / known limits: encoder is single threaded; no FITS regeneration for a `.lum` without stored origin
(`lumif_to_fits` returns `LUMIF_ERR_NO_ORIGIN`); FITS NaN payload round-trip assumes IEEE-754 NaN
propagation of the host (x86-64 SSE verified); palette > 65 536 values and zstd tile codec not
produced by the encoder (but both are decoded).

## Python

`python/lumif.py` is a ctypes binding (stdlib + numpy, no compiler needed at install time).

```bash
cmake -S libs/liblumif -B build-dll -DLUMIF_SHARED=ON -DLUMIF_BUILD_TOOLS=OFF && cmake --build build-dll --config Release
export LUMIF_LIB=build-dll/Release/lumif.dll      # or copy the library next to lumif.py
```

```python
import lumif
img = lumif.read("m33.lum", threads=0)            # img.pixels float32 (planes, h, w); img.info: cards, acq, ...
band = lumif.read("m33.lum", rows=(1000, 1064), plane=0)
lumif.write("copy.lum", img.pixels, channels=img.info["channels"], cards=img.info["cards"])
lumif.fits_to_lum("in.fits", "out.lum"); lumif.lum_to_fits("out.lum", "back.fits")   # byte-exact
```

`python/check_binding.py LUMEN_EXE files...` cross-checks the binding with the Rust CLI (pixels, byte-exact
FITS both ways, row bands, numpy write path).
