Metadata-Version: 2.4
Name: no-mersi
Version: 0.0.1
Summary: Processing tool for FengYun-3 satellite data
Project-URL: Documentation, https://github.com/pytroll/no-mersi#readme
Project-URL: Issues, https://github.com/pytroll/no-mersi/issues
Project-URL: Source, https://github.com/pytroll/no-mersi
Author-email: Martin Raspaud <martin.raspaud@smhi.se>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Python: >=3.14
Requires-Dist: hdf5plugin>=6.0.0
Requires-Dist: numba>=0.64.0
Requires-Dist: numpy>=2.0
Requires-Dist: platformdirs>=4.0
Requires-Dist: pycadu>=0.1.0
Requires-Dist: pyorbital>=1.12.1
Requires-Dist: pyspectral
Requires-Dist: tqdm>=4.0
Requires-Dist: xarray>=2024.1
Description-Content-Type: text/markdown

# no_mersi

## Status

What works:
- Processing FY3D and FY3F raw data to l1b for 1km and 250m data
- Generating the associated GEO files, navigation is using both GPS data and TLE if possible. Navigation has not been
carefully validated.
- CDF-based destriping for all TEB and RSB channels (see below)

What doesn’t (but we will work on it at some point):
- The thermal band calibration is very bad

## Installation

For now, the best is to clone the repo, and then use uv run as show below.


## Usage

```sh
uv run fy3-raw-to-l1b data/fy3d/station/rawdata_FY3D_MPT_f1_ch1_20260320T115601Z_20260320T121022Z_20260320T115601Z_zP4P11Ur3_vcdu2_43240.data --compression blosc generated --tle-file data/tle/tle-202603201215.txt
```

this will put the l1b and navigation files in the `generated` directory

I you want on visualise the results, you can use eg Satpy to generated some imagery.

For example, with:
```python
import hdf5plugin  # noqa, for blosc decompression
from satpy import Scene
scn = Scene(["generated/FY3D_20260320_115604_121013_43240_MERSI_1000M_L1B.HDF", 
             "generated/FY3D_20260320_115604_121013_43240_MERSI_GEO1K_L1B.HDF"], reader="mersi2_l1b")

comp = "true_color_raw"
comps = [comp]
scn.load(comps)
scn.show(comp)

lscn = scn.resample("euron1")
lscn.show(comp)
```

## Destriping

Destriping is **enabled by default** for all channels (TEB CH20–CH25 and RSB
CH01–CH19).  It uses CDF (Cumulative Distribution Function) matching: the
radiance histogram of each detector is compared to the ensemble of all healthy
detectors, and a Look-Up Table (LUT) is derived to equalise the distributions.

### LUT persistence

LUTs are derived from cumulative histogram statistics that are stored between
runs:

```
~/.local/share/no_mersi/destripe/
    fy3d_CH24_hist.npz      ← per-channel histogram state
    fy3d_CH24_ledger.json   ← deduplication log (prevents double-counting a pass)
    ...
```

After each processed granule the histograms are updated via an Exponential
Moving Average (α = 0.05), so the correction improves progressively as more
passes accumulate.

### First run — bootstrap

On the **first run** for a given channel there is no prior histogram, so the
processor performs an extra pre-pass through the full granule to build the
initial LUT.  This makes the first run noticeably slower (roughly double the
usual processing time).  Subsequent runs load the persisted state and are not
affected.

### Corrections (on by default) and how to skip them

The cleanest imagery is produced with no extra flags — CDF destriping, the
RVS scene correction, DCN (8 column bins), the per-detector non-linearity
correction, and an ADMM anisotropic-TV residual destripe (FY3D CH24/CH25;
FY3F CH23/CH24/CH25) are all applied by default.  Opt out individually:

```sh
uv run fy3-raw-to-l1b ... --no-destripe          # skip CDF destriping
uv run fy3-raw-to-l1b ... --no-dcn-correction    # skip DCN (fastest path)
uv run fy3-raw-to-l1b ... --no-rvs               # skip RVS scene correction
uv run fy3-raw-to-l1b ... --no-nonlinearity      # skip the non-linearity polynomial
uv run fy3-raw-to-l1b ... --no-admm-destripe     # skip the ADMM residual destripe (TEB)
```

`--all-corrections` is accepted as a legacy no-op alias (everything is
already default-on).

To reset the destripe LUT history for a specific channel (e.g. after an
instrument anomaly), delete the corresponding `.npz` and `.json` files in the
XDG data directory above.
