Metadata-Version: 2.4
Name: euler-preprocess
Version: 3.10.0
Summary: Physics-based preprocessing (fog, etc.) for RGB+depth datasets
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: Pillow
Requires-Dist: euler-loading
Provides-Extra: gpu
Requires-Dist: torch; extra == "gpu"
Provides-Extra: progress
Requires-Dist: tqdm; extra == "progress"

# euler-preprocess

Physics-based preprocessing transforms for multi-modal RGB+depth datasets. Built on top of [euler-loading](https://github.com/d-rothen/euler-loading) and [ds-crawler](https://github.com/d-rothen/ds-crawler).

Available transforms:

| Command | Description |
|---|---|
| `euler-preprocess fog` | Synthetic fog via the Koschmieder atmospheric scattering model |
| `euler-preprocess sky-depth` | Override depth values in sky regions with a constant |
| `euler-preprocess radial` | Convert planar (z-buffer) depth to radial (Euclidean) depth |

## Installation

```bash
uv pip install "euler-preprocess[gpu,progress] @ git+https://github.com/d-rothen/euler-fog"
```

## Usage

```bash
euler-preprocess fog       -c configs/example_dataset_config.json
euler-preprocess sky-depth -c configs/sky_depth_dataset_config.json
euler-preprocess radial    -c configs/radial_dataset_config.json
```

## Configuration

Every subcommand takes a **dataset config** JSON that points to the input data and a **transform config**. Each modality path must be a directory indexed by [ds-crawler](https://github.com/d-rothen/ds-crawler) with an `euler_loading` property that specifies the loader and function. This allows euler-loading to auto-select the correct dataset-specific loader.

### Dataset Config

```json
{
  "transform_config_path": "configs/run1.json",
  "output_path": "/path/to/output",
  "output_slot": "rgb",
  "sample": 42,
  "modalities": {
    "rgb": {"path": "/path/to/rgb", "split": "train"},
    "depth": "/path/to/depth",
    "semantic_segmentation": "/path/to/classSegmentation"
  },
  "hierarchical_modalities": {
    "intrinsics": {"path": "/path/to/intrinsics"}
  },
  "pipeline": {
    "output_root": "/pipeline/output",
    "outputs_manifest_path": "/pipeline/output/.euler_pipeline/pipeline_outputs.json",
    "output_targets": [
      {
        "slot": "rgb",
        "datasetType": "rgb",
        "relativePath": "foggy_rgb",
        "path": "/pipeline/output/foggy_rgb",
        "storage": "directory"
      }
    ]
  }
}
```

| Field | Description |
|---|---|
| `transform_config_path` | Path to the transform-specific config (see below). `fog_config_path` is also accepted for backward compatibility. |
| `output_path` | Output root used when no pipeline target overrides it. Optional if `pipeline.output_root` or `pipeline.output_targets[].path` supplies the destination. |
| `output_slot` | Optional slot selector when `pipeline.output_targets` contains multiple entries. Defaults to `rgb` for `fog`, `depth` for `sky-depth`, and `depth` for `radial`. |
| `sample` | Optional 0-based euler-loading dataset index. When set, only `dataset[sample]` is transformed, which is useful for small augmented benchmark slices from large datasets. |
| `samples` | Optional multi-sample selector. Use a list of 0-based indices (`[0, 10, 20]`) or a slice object such as `{"start": 0, "stop": 1000, "step": 2, "count": 100}`. `stop` is exclusive; `count` caps the selected indices after slicing. Do not set both `sample` and `samples`. |
| `modalities` | Regular modalities that participate in sample-ID intersection. Each value is either a plain path string or an object with a `path` key and an optional `split` key (see below). Which modalities are required depends on the transform (see table below). |
| `hierarchical_modalities` | Per-scene data (e.g. intrinsics). Same format as `modalities`. Loaded once per scene and cached. |
| `pipeline` | Optional runtime routing block compatible with `euler-inference` (`output_root`, `outputs_manifest_path`, `output_targets`). |

#### Inline splits

When a modality directory contains [ds-crawler](https://github.com/d-rothen/ds-crawler) split files (`.ds_crawler/split_<name>.json`), you can select a subset of the data by setting the `split` key on that modality. Sample IDs are matched by intersection across all modalities, so specifying a split on a single modality is sufficient to restrict the entire dataset.

For quick slices after euler-loading has matched modalities, set `samples`.
For example, `{"samples": {"step": 2}}` processes every second matched sample,
and `{"samples": {"start": 10, "step": 5, "count": 20}}` processes 20 samples
starting at index 10 with stride 5.

**Required modalities per transform:**

| Transform | `modalities` | `hierarchical_modalities` |
|---|---|---|
| `fog` | `rgb`, `depth`, `semantic_segmentation` | `intrinsics` when available; used for radial depth conversion and camera-profile optics |
| `sky-depth` | `depth`, `semantic_segmentation` | — |
| `radial` | `depth` | `intrinsics` |

#### Pipeline Runtime Block

`pipeline` follows the same shape as `euler-inference`:

```json
{
  "pipeline": {
    "output_root": "/pipeline/output",
    "outputs_manifest_path": "/pipeline/output/.euler_pipeline/pipeline_outputs.json",
    "output_targets": [
      {
        "slot": "depth",
        "datasetType": "depth",
        "relativePath": "radial_depth.zip",
        "path": "/pipeline/output/radial_depth.zip",
        "storage": "zip"
      }
    ]
  }
}
```

Notes:

- `output_root` is only a fallback when `output_path` is omitted.
- A matching `output_targets[].slot` overrides the write root for that run.
- `output_targets[].modelModalityId` is optional. When provided it is copied into the pipeline manifest; when omitted it is left out there as well.
- `storage: "directory"` writes a dataset directory and `storage: "zip"` writes a zip dataset.
- `storage: "file"` is parsed but rejected at runtime.
- When `outputs_manifest_path` is set and a pipeline target is matched, finalization writes `.euler_pipeline/pipeline_outputs.json` with the same manifest shape used by `euler-inference`.

---

## Fog Transform

### Fog Config

Controls the fog simulation.

```json
{
  "airlight": "from_sky",
  "seed": 1337,
  "depth_scale": 1.0,
  "resize_depth": true,
  "contrast_threshold": 0.05,
  "render_input_space": "srgb",
  "mode": "sample",
  "device": "cpu",
  "gpu_batch_size": 4,
  "capture": { "preset": "camera" },
  "camera_profile": "dashcam",
  "augmentations": { ... },
  "selection": { ... },
  "models": { ... }
}
```

| Field | Description |
|---|---|
| `airlight` | **Required.** Airlight estimation method: `"from_sky"` (mean sky colour), `"dcp"` (dark channel prior), or `"dcp_heuristic"` (robust DCP with sky-guided colouring when sky pixels exist). |
| `seed` | Random seed for reproducibility. `null` for non-deterministic. |
| `depth_scale` | Multiplier applied to depth values after loading. |
| `resize_depth` | Resize the depth map to match the RGB resolution (bilinear). |
| `contrast_threshold` | Threshold *C_t* used in the visibility-to-attenuation conversion (default `0.05`). |
| `render_input_space` | Colour space of the RGB supplied to the fog renderer. Use `"srgb"` for display-encoded dataset images so fog and airlight are mixed in scene-linear RGB; use `"linear"` for legacy configs or already-linear radiance. |
| `mode` | Optional scenario mode. Omit it or use `"sample"` for current one-scenario-per-image behavior; use `"progressive"` to render every scenario step for every image. |
| `device` | `"cpu"`, `"cuda"`, `"mps"`, or `"gpu"` (alias for cuda). |
| `gpu_batch_size` | Batch size when running on GPU. Uniform-model samples are batched; heterogeneous samples are processed individually. |
| `capture` / `capture_artifacts` | Optional post-fog camera artifact pipeline. Omit it or set `{"stages": []}` for the legacy no-op path. Set `true`, `{"preset": "camera"}`, or a custom `stages` list to enable optics, raw sensor, ISP, and compression artifacts. |
| `camera_profile` | Optional named or inline camera profile whose stage defaults are merged into the capture stack before per-stage overrides. Built-ins are `"default"`, `"generic"`, `"dashcam"`, and `"low_light_fog"`. |
| `camera_profiles` | Optional map of project-specific named profiles. Use this for calibrated lens, sensor, ISP, and transport settings. |
| `scenario_profiles` | Optional top-level correlated condition sampler. Each sampled scenario can choose the fog model, override model parameters, select the airlight method, switch camera profile/settings, and force named capture-stage condition profiles. |
| `augmentations` | Optional stepped augmentation set. When present, every input sample produces every configured augmentation and uses the file-id hierarchy output layout described below. |

### Processing Pipeline

Fog generation is split into two phases:

1. Ideal scene rendering: physics-based fog and auxiliary `scattering_coefficient`
   / `atmospheric_light` maps are computed.
2. Capture artifacts: camera-specific effects are applied to the rendered RGB
   only. Physical fog maps stay stable while the RGB output can receive
   exposure shifts, lens blur, vignetting, raw sensor noise, Bayer/demosaicing
   artifacts, ISP tone/gamma/sharpening, and JPEG/resize/quantization effects.

This keeps physical fog maps stable while making the RGB output extensible for
real-camera simulation.

### Capture Artifact Stack

Enable the recommended camera stack with:

```json
"capture": { "preset": "camera" }
```

or equivalently:

```json
"capture": true
```

For tighter control, provide explicit stages in camera order:

```json
"camera_profiles": {
  "real_drive_front": {
    "optics": {
      "lens_distortion": -0.012,
      "vignetting_strength": 0.18,
      "windshield_haze": {"enabled": true, "probability": 0.55}
    },
    "sensor": {
      "bayer_pattern": "RGGB",
      "iso": {"dist": "choice", "values": [200, 400, 800]},
      "base_iso": 100,
      "auto_exposure": {
        "enabled": true,
        "metering": "center_weighted",
        "target_luminance": {"dist": "uniform", "min": 0.16, "max": 0.24},
        "highlight_protection": 0.7,
        "resolve_iso": true,
        "max_iso": 1600
      },
      "full_well_electrons": [14000, 12000, 13000],
      "read_noise_electrons": {"dist": "uniform", "min": 2.0, "max": 6.0},
      "shadow_recovery_noise": {
        "enabled": true,
        "luminance_threshold": {"dist": "uniform", "min": 0.16, "max": 0.24},
        "luma_sigma": {"dist": "uniform", "min": 0.0, "max": 0.001},
        "chroma_sigma": {"dist": "uniform", "min": 0.012, "max": 0.028},
        "chroma_mode": "balanced",
        "red_chroma_gain": {"dist": "uniform", "min": 0.7, "max": 1.0},
        "blue_chroma_gain": {"dist": "uniform", "min": 1.7, "max": 2.7},
        "chroma_axis_correlation": {"dist": "uniform", "min": 0.05, "max": 0.25},
        "chroma_spatial_sigma": 0.0,
        "chroma_fine_fraction": 1.0,
        "chroma_luminance_preservation": 1.0,
        "black_noise_floor": 0.25,
        "black_suppression_luminance": 0.03,
        "black_suppression_softness": 0.08
      },
      "black_level": [0.003, 0.0035, 0.003],
      "white_level": [1.0, 0.995, 1.0],
      "adc_bit_depth": 12,
      "post_demosaic_bit_depth": 12
    },
    "isp": {
      "tone_map": "reinhard",
      "gamma": "srgb",
      "denoise_sigma": 0.2,
      "sharpen_amount": 0.2,
      "saturation": 0.9
    },
    "transport": {
      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
      "bit_depth": 8
    }
  }
},
"camera_profile": "real_drive_front",
"capture": {
  "stages": [
    {
      "type": "optics",
      "blur_sigma": {"dist": "uniform", "min": 0.2, "max": 0.8},
      "vignetting_strength": 0.15,
      "windshield_haze": {"enabled": true, "probability": 0.4},
      "droplets": {"enabled": false}
    },
    {
      "type": "sensor",
      "input_space": "srgb",
      "exposure_gain": {"dist": "uniform", "min": 0.85, "max": 1.2},
      "row_noise_sigma": 0.003
    },
    {
      "type": "isp",
      "tone_map": "reinhard",
      "gamma": "srgb",
      "denoise_sigma": 0.2,
      "sharpen_amount": 0.2,
      "saturation": 0.9
    },
    {
      "type": "transport",
      "jpeg": {"enabled": true, "quality": {"dist": "uniform", "min": 65, "max": 92}},
      "bit_depth": 8
    }
  ]
}
```

Supported stage types:

| Stage | Main effects |
|---|---|
| `optics` | Defocus/MTF blur, motion blur, bloom, veiling glare, vignetting, chromatic aberration, lens distortion, windshield haze, optional droplets. |
| `sensor` | Image-driven or sampled exposure, white balance, camera matrix, Bayer mosaic, shot/read noise, fixed-pattern noise, row/column banding, shadow-local recovery noise, hot/dead pixels, bilinear demosaic. |
| `isp` | Denoising, color correction, tone mapping, sRGB/gamma, local contrast, sharpening halos, saturation shifts. |
| `transport` | Crop/resize, bit-depth quantization, JPEG round-trip compression. |
| `exposure` | Lightweight standalone exposure and white-balance stage for simple custom chains. |

Set `sensor.auto_exposure.enabled` to meter the rendered image before raw sensor
sampling. `target_luminance`, `metering`, `highlight_*`, and gain bounds choose
the exposure; `resolve_iso` can raise ISO from the metering pressure, dark pixel
fraction, and fog opacity. When auto exposure is enabled, `exposure_gain` still
applies as scenario-specific exposure compensation.

Fog-aware metering modes use `CaptureContext.depth_m`, `k_map`, fog opacity, and
`attributes.sky_mask` when available. Use `"metering":
"fog_aware_center_weighted"` or `"sky_aware_center_weighted"` and tune
`sky_suppression`, `fog_meter_suppression`, `depth_meter_decay_m`, and
`min_meter_weight` to keep bright sky or dense far-field airlight from dominating
the exposure meter. Legacy metering modes are unchanged unless these suppression
keys are present.

Set `sensor.sensor_identity.enabled` for persistent sensor structure across
frames. The identity cache is deterministic for the same `sensor_id`, `seed`,
image shape, and Bayer pattern. `prnu_sigma` adds multiplicative pixel-response
non-uniformity before shot noise; `dsnu_sigma`, `persistent_row_sigma`, and
`persistent_column_sigma` add fixed raw-domain offsets; persistent hot/dead pixel
probabilities create stable bad-pixel masks that combine with the existing
per-image bad-pixel probabilities.

Set `sensor.shadow_recovery_noise.enabled` to add extra post-demosaic luma and
chroma corruption only where the pre-exposure rendered luminance was low. This
is useful for reducing broad global grain while keeping lifted shadows visibly
noisy. For dark high-ISO scenes, keep `luma_sigma` much lower than
`chroma_sigma`, use `chroma_mode: "balanced"`, and leave
`chroma_luminance_preservation` near `1.0`; this makes the local corruption read
as color noise instead of black luma speckles. `red_chroma_gain`,
`blue_chroma_gain`, and `chroma_axis_correlation` can match camera-specific
color noise, including blue/purple-biased high-ISO noise. Keep
`chroma_spatial_sigma` near `0` and avoid chroma subsampling when the target
noise is fine-grained rather than blocky. `black_noise_floor` with
`black_suppression_luminance`/`black_suppression_softness` reduces the extra
noise in near-clipped black regions, so the strongest visible noise sits in
dim-but-readable shadows.

Set `sensor.noise_adjustment` for relative, scenario-level noise controls on
top of the selected camera/condition profile. `level: 1.0` leaves the authored
profile unchanged; lower values suppress read/static/chroma noise and higher
values amplify it. `static_chroma_bias` ranges from `-1.0` for more fixed
pattern, row/column, banding, and bad-pixel noise to `1.0` for more
chromatic/high-ISO-looking shadow noise:

```json
{
  "capture_overrides": {
    "sensor": {
      "condition_profile": "nominal_gloom",
      "noise_adjustment": {
        "enabled": true,
        "level": 1.25,
        "static_chroma_bias": 0.35
      }
    }
  }
}
```

Any stage can define `condition_profiles` to sample coherent per-image settings
before the stage runs. This is useful for exposure states where ISO, exposure
gain, read noise, banding, and dark/fog noise modulation should move together:

```json
{
  "type": "sensor",
  "condition_profiles": [
    {"name": "clean_daylight", "weight": 0.25, "exposure_gain": 1.0, "iso": 100},
    {"name": "underexposed_noisy", "weight": 0.25, "exposure_gain": 0.65, "iso": 1600}
  ]
}
```

`isp.tone_map` supports `"reinhard"`, `"aces"`, `"clip"`, and `"lut"`. The LUT
mode uses a cheap interpolated 1D camera-response curve:

```json
{
  "type": "isp",
  "tone_map": "lut",
  "tone_map_strength": 1.0,
  "tone_map_lut": [0.0, 0.006, 0.014, 0.028, 0.052, 0.090, 0.145, 0.220, 0.320, 0.450, 0.610, 0.780, 0.900, 0.965, 0.995, 1.0],
  "tone_map_lut_domain": "linear",
  "gamma": "srgb"
}
```

Top-level `scenario_profiles` sample one latent scene/camera condition before
rendering. The selected scenario is merged over the root config, so it can drive
fog density, atmospheric light, camera profile, capture-stage overrides, ISP, and
compression together:

```json
"scenario_profiles": [
  {
    "name": "clean_low_noise_haze",
    "weight": 0.22,
    "model": "heterogeneous_k_ls",
    "airlight_method": "dcp_heuristic",
    "models": {
      "heterogeneous_k_ls": {
        "visibility_m": {"dist": "uniform", "min": 60.0, "max": 130.0}
      }
    },
    "capture_overrides": {
      "sensor": {"condition_profile": "clean_daylight"},
      "isp": {"denoise_sigma": {"dist": "uniform", "min": 0.08, "max": 0.32}}
    }
  },
  {
    "name": "underexposed_dense_gloom",
    "weight": 0.25,
    "model": "heterogeneous_k_ls",
    "airlight_method": "dcp_heuristic",
    "models": {
      "heterogeneous_k_ls": {
        "visibility_m": {"dist": "uniform", "min": 18.0, "max": 55.0},
        "scene_illumination": {
          "enabled": true,
          "global_ev": {"dist": "uniform", "min": 0.25, "max": 0.85},
          "near_ev": {"dist": "uniform", "min": 0.35, "max": 1.20},
          "near_decay_depth_m": {"dist": "uniform", "min": 10.0, "max": 22.0},
          "fog_coupled_ev": {"dist": "uniform", "min": 0.10, "max": 0.45},
          "sky_weight": 0.0
        }
      }
    },
    "capture_overrides": {
      "sensor": {
        "condition_profile": "underexposed_noisy",
        "auto_exposure": {
          "enabled": true,
          "metering": "fog_aware_center_weighted",
          "target_luminance": {"dist": "uniform", "min": 0.13, "max": 0.20},
          "highlight_protection": 0.78,
          "manual_gain_weight": 0.0,
          "sky_suppression": 0.85,
          "fog_meter_suppression": 0.65
        }
      },
      "transport": {"jpeg": {"quality": {"dist": "uniform", "min": 54, "max": 78}}}
    }
  }
]
```

`capture_overrides` is merged after camera-profile and stage settings. Use
`condition_profile` to force one named profile from a stage's
`condition_profiles`; if omitted, the stage continues sampling its own profile
weights locally.

Set top-level `"mode": "progressive"` to emit every configured scenario for
every input image instead of sampling one scenario. Each scenario accepts
`"steps"` and `"progressive_weight"` (or `"max_weight"` / `"weight"` as
aliases); the transform writes steps from weight `0` through the scenario's
configured weight, and weight `1` matches the original scenario. Fog density is
progressed in scattering-coefficient space, while numeric camera/config values
blend from the base config toward the scenario config. Progressive blends clamp
probability-like values and non-negative physical factors back into valid
mathematical domains so extrapolated weights above `1` do not create invalid
render parameters.
Source-backed outputs are written as `fog_progression` variants under each
source file id.

### Fog Model

The core equation is the **Koschmieder model** (atmospheric scattering):

```
I_fog(x) = I(x) * t(x)  +  L_s * (1 - t(x))
```

where:

- **I(x)** is the original RGB colour at pixel *x*
- **t(x) = exp(-k * d(x))** is the transmittance, which falls exponentially with depth *d* and attenuation coefficient *k*
- **L_s** is the atmospheric light (airlight), i.e. the colour of the fog/sky light scattered towards the camera
- **k** is derived from a meteorological visibility distance *V*: `k = -ln(C_t) / V`

Distant objects are attenuated more (`t` approaches 0) and replaced by airlight, just as in real fog.

For gloomy conditions, add `scene_illumination` inside a fog model config. This
darkens pre-fog scene radiance `I(x)` before the atmospheric scattering equation,
so near objects can become plausibly overcast or storm-lit instead of passing
through unchanged. `global_ev` applies to the whole non-sky scene, `near_ev` adds
extra near-field darkening with `near_decay_depth_m`, `fog_coupled_ev` adds a
term proportional to local fog opacity, and `sky_weight: 0.0` preserves sky
pixels when a sky mask is available.

### How Each Modality is Used

**RGB** — The clean scene image. Normalised to float32 in [0, 1]. This is the *I(x)* term in the fog equation -- it gets blended with the airlight according to transmittance.

**Depth** — A per-pixel depth map in **metres**. Provides the *d(x)* term in the transmittance calculation `t(x) = exp(-k * d(x))`. Pixels with greater depth receive more fog. Invalid values (NaN, inf, negative) are clamped to zero (treated as infinitely close, receiving no fog). By default, depth remains authoritative even where the semantic map says sky. For noisy real-world predictions, `sky_fog_path` can instead derive a density-weighted sky path through a bounded fog layer from the camera intrinsics and assumed orientation.

**Semantic Segmentation** — A per-pixel semantic segmentation map from which a boolean sky mask is derived, loaded via euler-loading's dataset-specific `semantic_segmentation` loader. The sky mask is used for airlight estimation when the `airlight` method is `"from_sky"`: the mean RGB of all sky pixels in the clean image is used as the airlight colour *L_s*. When `sky_fog_path` is configured, semantic sky pixels use the modeled fog-layer path while depth remains authoritative everywhere else.

**Intrinsics** *(optional)* — When present, planar (z-buffer) depth is converted to radial (Euclidean) depth before fog is applied.

For a bounded valley-fog treatment of sky, add this inside a model config:

```json
"sky_fog_path": {
  "mode": "layer",
  "camera_height_m": 1.6,
  "fog_valley_peak_m": 80.0,
  "camera_pitch_deg": 0.0,
  "camera_roll_deg": 0.0,
  "density_profile": "linear_fade",
  "transition_height_m": 10.0,
  "max_path_m": 1000.0
}
```

Pixel rays are reconstructed from the intrinsics. Near-horizon rays accumulate
long paths through fog, while upward rays exit the layer sooner. The default
`linear_fade` profile has full density below the transition and decreases
linearly to zero at `fog_valley_peak_m`; both `camera_height_m` and
`fog_valley_peak_m` are elevations in the same local vertical coordinate.
`uniform` keeps full density up to the top. Positive `camera_pitch_deg` points
the optical axis upward. `max_path_m` bounds horizon rays. The fog-layer path feeds the same fog
opacity, scene-illumination, and atmospheric-light-gradient heuristics as
ordinary depth. The original depth is retained for non-sky pixels and capture
effects. Enabling this option without intrinsics raises an error rather than
silently reverting to predicted sky depth.

### Airlight Estimation

The `airlight` config key selects how the atmospheric light *L_s* is estimated:

| Method | Description |
|---|---|
| `from_sky` | Mean RGB of sky pixels in the clean image. Falls back to white `[1, 1, 1]` when no sky pixels exist. |
| `dcp` | Dark Channel Prior — selects the brightest pixel (by channel sum) among the top 0.1% darkest-channel pixels. |
| `dcp_heuristic` | Robust DCP heuristic — pools the brighter half of the top 0.1% darkest-channel pixels, and when sky pixels exist it uses the brightest sky colours as the chromaticity prior while preserving DCP-derived luminance. Optional bias controls can nudge the result toward white or a cool fog tint. |

GPU-native implementations (`DCPAirlightTorch`, `DCPHeuristicAirlightTorch`) are used automatically when running on GPU.

When `airlight` is `"dcp_heuristic"`, you can optionally add:

```json
"dcp_heuristic": {
  "patch_size": 15,
  "top_percent": 0.001,
  "white_bias": 0.1,
  "cool_bias": 0.15,
  "cool_target": [0.93, 0.97, 1.0]
}
```

- `white_bias` mixes the final airlight toward neutral white.
- `cool_bias` mixes the final airlight toward a sky-relative cool target.
- `cool_target` is the cool-white anchor used to derive that sky-relative target. When sky pixels exist, the effective cool target is a blend of the estimated sky colour and `cool_target`; without sky pixels, it falls back to the airlight estimate and `cool_target`.
- `white_bias + cool_bias` must be `<= 1`.
- The tint bias preserves the estimated airlight luminance, so it shifts colour without silently changing fog density.

### Airlight Intensity Dampening

Estimated airlight is dampened by default as fog density increases. This keeps
strong fog closer to the low, grey lighting seen in real in-car fog footage
instead of letting DCP-style estimates wash dense fog toward white.

Each fog model can override the dampening curve:

```json
"airlight_dampening": {
  "enabled": true,
  "apply_to": "estimated",
  "reference_visibility_m": 80.0,
  "min_factor": 0.45,
  "max_factor": 1.0,
  "strength": 1.0
}
```

The factor is:
`min_factor + (max_factor - min_factor) / (1 + strength * beta / reference_beta)`.
`min_factor` and `max_factor` must be finite and non-negative. Values above
`1.0` are allowed when you intentionally want to brighten estimated airlight;
the final RGB output is still clamped to the valid image range. `strength`
must remain finite and non-negative.
`reference_beta` is either `reference_scattering_coefficient` /
`reference_beta`, or it is derived from `reference_visibility_m` using the
model's contrast threshold. The default applies only when `atmospheric_light`
uses an estimated airlight method (`"from_sky"`, `"dcp"`, or
`"dcp_heuristic"`); literal RGB `atmospheric_light` values stay exact unless
`apply_to` is set to `"all"`. Set `"enabled": false` or `apply_to: "none"` to
preserve the previous undampened behavior.

For `heterogeneous_ls` and `heterogeneous_k_ls`, the Perlin atmospheric-light
field is sampled around the dampened base airlight, so the spatial variation
does not reintroduce the old over-illuminated look.

### Model Selection

Each image is assigned a fog model via the `selection` block:

```json
"selection": {
  "mode": "weighted",
  "weights": {
    "uniform": 0.25,
    "heterogeneous_k": 0.35,
    "heterogeneous_ls": 0.25,
    "heterogeneous_k_ls": 0.15
  }
}
```

- **`fixed`** mode: always use a single named model.
- **`weighted`** mode: randomly select a model per image according to normalised weights.

Four models are available:

| Model | Description |
|---|---|
| `uniform` | Constant *k* and *L_s*. Standard homogeneous fog. |
| `heterogeneous_k` | Spatially-varying *k*, constant *L_s*. Simulates patchy fog / fog banks. |
| `heterogeneous_ls` | Constant *k*, spatially-varying *L_s*. Simulates scattered-light colour variation. |
| `heterogeneous_k_ls` | Both *k* and *L_s* vary spatially. Most expressive model. |

### Visibility Distribution

Each model specifies a `visibility_m` distribution from which a visibility distance (in metres) is sampled per image:

| `dist` | Parameters | Description |
|---|---|---|
| `constant` | `value` | Fixed value. |
| `uniform` | `min`, `max` | Uniform random in range. |
| `normal` | `mean`, `std`, optional `min`/`max` | Gaussian, optionally clamped. |
| `lognormal` | `mean`, `sigma`, optional `min`/`max` | Log-normal. |
| `choice` | `values`, optional `weights` | Discrete weighted choice. |

The sampled visibility *V* is converted to the attenuation coefficient:
**k = -ln(C_t) / V**. This happens once per output image. For
`heterogeneous_k` and `heterogeneous_k_ls`, that sampled value is the base
coefficient that is then spatially modulated by the noise field.

### Stepped Augmentations

For benchmark generation, set `augmentations` in the fog config. This switches
the fog transform from one sampled output per input to one output per configured
variant:

```json
{
  "airlight": "from_sky",
  "seed": 1337,
  "contrast_threshold": 0.05,
  "augmentations": {
    "file_id_hierarchy_name": "file_id",
    "attribute_key": "fog_augmentation",
    "models": ["uniform"],
    "visibility_m": [10, 20, 40, 70, 100],
    "airlight_methods": ["from_sky"]
  }
}
```

The matrix form above expands as the Cartesian product of `models`,
`visibility_m` (MOR in metres), optional `scattering_coefficients` / `beta`, and
airlight choices. `file_id_hierarchy_name` names the inserted hierarchy level
when the underlying ds-crawler writer has a hierarchy separator; the directory
name is the source file id in either case. For tighter control, use explicit
variants:

```json
"augmentations": {
  "variants": [
    {
      "id": "mor_010m_sky",
      "model": "uniform",
      "visibility_m": 10,
      "airlight_method": "from_sky"
    },
    {
      "id": "beta_0.15_white",
      "model": "heterogeneous_k",
      "scattering_coefficient": 0.15,
      "atmospheric_light": [1.0, 1.0, 1.0],
      "k_hetero": {
        "scales": "smooth_auto",
        "correlation_length_fraction": 0.25,
        "octaves": 3,
        "min_factor": 0.65,
        "max_factor": 1.45,
        "contrast": 0.65,
        "normalize_to_mean": true
      }
    }
  ]
}
```

Each output entry receives per-file ds-crawler attributes under
`fog_augmentation`, including the augmentation id, source id, source full id,
model, actual scattering coefficient, actual atmospheric light, and configured
MOR/beta descriptors when available. euler-loading exposes these as
`sample["attributes"]["rgb"]["fog_augmentation"]`.

### Heterogeneous Noise Fields

Both `k_hetero` and `ls_hetero` use Perlin FBM (fractional Brownian
motion) to generate spatially-varying factor fields. For realistic fog,
prefer the smooth mode: it keeps Perlin wavelengths tied to the image size,
then optionally reduces noise contrast and applies a final blur before mapping
the noise to physical factors.

```json
"k_hetero": {
  "scales": "smooth_auto",
  "correlation_length_fraction": 0.25,
  "octaves": 3,
  "max_scale": null,
  "min_factor": 0.65,
  "max_factor": 1.45,
  "contrast": 0.65,
  "smooth_sigma_fraction": 0.0,
  "normalize_to_mean": true
}
```

The noise field (values in [0, 1]) is mapped to a factor field:
`factor(x) = min_factor + (max_factor - min_factor) * noise(x)`.
`contrast < 1` compresses the noise around 0.5 before this mapping, avoiding
extreme local fog density. When `normalize_to_mean` is `true`, the factor field
is rescaled so its spatial mean equals 1.0, preserving the overall fog density
while introducing spatial variation. In other words, with heterogeneous `k`:
`k(x) = k_sampled * factor(x)`. If `visibility_m` / MOR was sampled from a
distribution, `k_sampled` is the coefficient derived from that one sampled MOR.
With `normalize_to_mean: true`, the arithmetic mean of the per-pixel `k` map
equals `k_sampled`; the median is not forced to match. With
`normalize_to_mean: false`, the map mean shifts by the mean of the factor field.

For `heterogeneous_ls` / `heterogeneous_k_ls`, `ls_hetero` can also include a
weak view-direction illumination prior. This modulates the atmospheric-light
field itself, so the rendered effect is still gated by fog transmittance:

```json
"ls_hetero": {
  "ls_gradient": {
    "enabled": true,
    "probability": 0.65,
    "axis": "vertical",
    "top_factor": {"dist": "uniform", "min": 1.03, "max": 1.14},
    "bottom_factor": {"dist": "uniform", "min": 0.88, "max": 0.99},
    "gamma": {"dist": "uniform", "min": 0.85, "max": 1.6},
    "normalize_to_mean": true,
    "fog_opacity_weight": 0.65
  }
}
```

| Parameter | Effect |
|---|---|
| `min_factor` / `max_factor` | Range of the multiplicative factor. |
| `normalize_to_mean` | Rescale factors so the image-wide mean equals the base value. Recommended for `k_hetero`. |
| `scales: "smooth_auto"` | Build low-frequency Perlin scales from the image size. |
| `correlation_length_fraction` | Approximate smallest fog feature size as a fraction of the shorter image side. Larger values create smoother gradients. |
| `octaves` / `lacunarity` / `max_scale` | Control how many increasingly broad Perlin components are mixed. |
| `contrast` | Compress or expand the Perlin range before mapping to factors. Values below 1 are recommended. |
| `smooth_sigma` / `smooth_sigma_fraction` | Optional final Gaussian blur in pixels or as a fraction of the shorter image side. |
| `ls_gradient` | Optional `L_s` top-to-bottom or left-to-right factor field. Keep it weak and probabilistic to avoid a deterministic image-position shortcut. |

### Fog Output

CLI runs write a source-backed RGB dataset. The output keeps the source RGB
dataset's relative paths, basenames, extensions, and ds-crawler metadata so the
result stays loadable by `euler-loading`:

```
<output_path>/
  .ds_crawler/dataset-head.json
  .ds_crawler/ds-crawler.json
  .ds_crawler/index.json
  Scene01/
    Camera_0/
      00000.png
```

When a pipeline target is present, `pipeline.output_targets[].path` replaces
`output_path` entirely. Standalone/direct `FogTransform(...)` usage without the
CLI still uses the legacy per-model layout with `config.json` sidecars.

With `augmentations` enabled, source-backed outputs are written one level below
the source file id instead:

```
<output_path>/
  .ds_crawler/dataset-head.json
  .ds_crawler/ds-crawler.json
  .ds_crawler/index.json
  Scene01/
    Camera_0/
      00000/
        mor_10m_airlight_from_sky.png
        mor_20m_airlight_from_sky.png
```

Auxiliary `scattering_coefficient` and `atmospheric_light` pipeline targets use
the same file-id hierarchy and write matching `.npy` augmentation files.

---

## Sky-Depth Transform

Overrides depth values in sky regions with a configurable constant. Useful for datasets where sky depth is encoded as zero or infinity and needs to be normalised to a large finite value.

### Sky-Depth Config

```json
{
  "sky_depth_value": 1000.0
}
```

| Field | Description |
|---|---|
| `sky_depth_value` | Depth value assigned to all sky pixels. Defaults to `1000.0`. |

### Sky-Depth Output

CLI runs write a source-backed depth dataset mirroring the input depth
modality's paths, filenames, extensions, and metadata. Standalone/direct
`SkyDepthTransform(...)` usage keeps the legacy `.npy` output behavior.

---

## Radial Transform

Converts planar (z-buffer) depth to radial (Euclidean) depth using camera intrinsics. For each pixel *(u, v)*:

```
d_radial(u, v) = d_planar(u, v) * sqrt(((u - cx)/fx)^2 + ((v - cy)/fy)^2 + 1)
```

### Radial Config

```json
{}
```

No special parameters are required. The transform reads intrinsics from the `intrinsics` hierarchical modality.

### Radial Output

CLI runs write a source-backed depth dataset mirroring the input depth
modality's layout and writer metadata. The emitted `index.json` also flips
`meta.radial_depth` to `true`. Standalone/direct `RadialTransform(...)` usage
keeps the legacy `.npy` output behavior.
