Metadata-Version: 2.4
Name: pysdsim
Version: 2026.11.1
Summary: An implementation of the Temperature-Dependent Multi-Relaxation Spectroscopic Dielectric Model for thawed and frozen organic soils.
Author-email: Morteza Khazaei <morteza.khazaei@usherbrooke.ca>
License: MIT License
        
        Copyright (c) 2023 Morteza Khazaei
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/Morteza-Khazaei/PySDS
Project-URL: Repository, https://github.com/Morteza-Khazaei/PySDS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rasterio
Requires-Dist: pyproj
Requires-Dist: numpy
Requires-Dist: tdmrsdm>=2026.10.0
Requires-Dist: tables
Requires-Dist: h5py
Requires-Dist: scipy
Requires-Dist: matplotlib
Requires-Dist: torch
Requires-Dist: opencv-python
Requires-Dist: segment-geospatial
Dynamic: license-file

# PySDS: Python Soil Dielectric Simulator

[![Python Version](https://img.shields.io/badge/python-%3E%3D3.8-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/github/license/Morteza-Khazaei/PySDS)](https://github.com/Morteza-Khazaei/PySDS/blob/main/LICENSE)

> An automatic framework for generating benchmark datasets of spatially coherent soil dielectric constant (DC) maps from soil organic carbon (SOC) maps, using the temperature-dependent multi-relaxation spectroscopic dielectric model (TD-MRSDM).

---

## About the project

Surface soil moisture (SSM) retrieval from synthetic aperture radar (SAR) imagery can be posed as a two-dimensional inverse scattering problem, which needs spatially coherent soil DC maps as priors, training targets or inputs. PySDS generates such maps automatically. It projects the point-scale TD-MRSDM, in the version of Mironov and Savin (2019), onto SOC maps, so that the spatial structure of SOC carries into the simulated real (ε′) and imaginary (ε″) parts of the soil DC.

<p align="center">
  <img src="docs/pysds_latex/figures/figure_1.png" alt="PySDS framework: SOC map to derived soil properties, TD-MRSDM simulation and soil dielectric constant maps" width="800">
</p>

The workflow (figure above):

1. **Inputs:** a SOC map (g/kg) as the only spatial input, plus user-defined ranges of SSM and soil surface temperature (SST) and a radar frequency.
2. **Derived soil properties:** soil organic matter (SOM) is derived from SOC with the van Bemmelen factor (SOM = 1.724 × SOC), and soil bulk density (SBD) with a SOC-based pedotransfer function (PTF).
3. **Simulation:** the TD-MRSDM, implemented in the companion [`tdmrsdm`](https://github.com/Morteza-Khazaei/TD-MRSDM) package (PyTD-MRSDM) with constants optimized against the dielectric database of organic Arctic soils (DDOAS), computes the soil DC for each SSM, SST and frequency scenario.
4. **Output:** soil DC maps (ε′ and ε″), saved as patches in HDF5 or NPZ datasets.

The underlying model covers radar frequencies from 10 MHz to 16 GHz, SST from −30 °C to +25 °C, and moisture up to 0.6 cm³/cm³. The accompanying paper evaluates its reliability across frequency, SSM, SST and SBD: ε′ is reliable under thawed conditions and at operational microwave bands, whereas ε″ is less certain at low frequencies, near saturation, in frozen soils and near the freeze–thaw transition. Because the model was developed for Arctic organic soils and SOM and SBD are derived from SOC through empirical relations, transferability to other soils is not guaranteed. PySDS does not retrieve SSM itself; it provides the dielectric data that SAR inversion methods need.

### Key features

* **Spatially coherent soil DC maps:** ε′ and ε″ maps that keep the SOC-driven soil heterogeneity of the input map.
* **Frozen and thawed soils:** bound, transient and unbound/ice water forms, through the TD-MRSDM.
* **Multi-condition datasets:** several (SST, SSM) conditions per geographic patch, at a chosen radar frequency.
* **Stratified sampling (optional):** balances the dataset across the ε′–ε″ space.
* **Satellite segmentation (optional):** segments each patch from satellite imagery with SAM (Segment Anything) through `segment-geospatial`.
* **Command-line tools:** `pysdsim` for dataset generation and `pysdsim-report` for statistical reports.

---

## Installation

PySDS requires Python ≥ 3.8.

```sh
pip install pysdsim
```

or the latest version from GitHub:

```sh
pip install "git+https://github.com/Morteza-Khazaei/PySDS.git"
```

PySDS requires `tdmrsdm` ≥ 2026.10.0, which ships the optimized `v3` TD-MRSDM constants; `pip` installs it automatically.

---

## Usage

### As a library

```python
import numpy as np
import rasterio
from rasterio.transform import from_origin
from pysdsim import Simulator

# 1. Create a small synthetic SOC map (g/kg); replace it with your own SOC raster
soc_map_path = "dummy_soc_map.tif"
soc = (10 + 50 * np.random.rand(256, 256)).astype(np.float32)
profile = {"driver": "GTiff", "height": 256, "width": 256, "count": 1, "dtype": "float32",
           "crs": "EPSG:32632", "transform": from_origin(500000, 5500000, 10, 10)}
with rasterio.open(soc_map_path, "w", **profile) as dst:
    dst.write(soc, 1)

# 2. Configure the simulator
sim = Simulator(
    workspace="pysds_output",       # folder created in your home directory (Google Drive on Colab)
    dataset_size=5,                 # number of patches
    patch_size=64,                  # 64 x 64 pixel patches
    soc_path=soc_map_path,
    freq_GHz=5.405,                 # C-band (Sentinel-1)
    sst_range=(-30, 30, 5),         # SST (°C): -30, -25, ..., 25 (stop value excluded)
    ssm_range=(0.05, 0.65, 0.05),   # SSM (cm³/cm³): 0.05, 0.10, ..., 0.60 (stop value excluded)
    no_data_threshold=0.1,          # maximum fraction of NoData pixels per patch
    enable_segmentation=False,      # True downloads satellite imagery and needs SAM
)

# 3. Generate the dataset
sim.run()
```

Ranges are given as `(start, stop, step)` and expanded with `numpy.arange`, so the stop value is **not** included.

The dataset is written to `~/pysds_output/PySDS_output/`, together with an HTML statistical report. In the library, `stratified_sampling` is off by default, so each patch gets one randomly drawn (SST, SSM) condition; set `stratified_sampling=True` to simulate several conditions per patch, as the command-line tool does by default.

### From the command line

```sh
pysdsim path/to/soc_map.tif \
    --workspace MyPySDS_Workspace \
    --nsize 1000 \
    --patch-size 128 \
    --frequency 5.405 \
    --sst_range -30 30 5 \
    --ssm_range 0.05 0.65 0.05
```

This simulates 1000 patches of 128 × 128 pixels at C-band (5.405 GHz), with SST from −30 °C to 25 °C in 5 °C steps and SSM from 0.05 to 0.60 cm³/cm³ in 0.05 steps. Several (SST, SSM) conditions are simulated per patch. By default, conditions are selected by stratified sampling, which fills the ε′–ε″ space evenly by keeping a quota of samples per dielectric bin (`--samples-per-bin`) and a maximum number of conditions per patch (`--max-per-patch`); use `--no-stratified` for uniform random sampling.

Full list of options (`pysdsim --help`):

```text
usage: pysdsim [-h] [-n DL_DB_NAME] [-w WORKSPACE] [-s NSIZE] [-p PATCH_SIZE]
               [-o OVERLAP OVERLAP] [-f FREQUENCY]
               [-t SST_RANGE SST_RANGE SST_RANGE]
               [-m SSM_RANGE SSM_RANGE SSM_RANGE] [-l NO_DATA_THRESHOLD]
               [-b MAX_SEARCH_DISTANCE] [-i SMOOTHING_ITERATIONS]
               [--save-format {hdf5,npz}] [--stratified] [--no-stratified]
               [--samples-per-bin SAMPLES_PER_BIN]
               [--max-per-patch MAX_PER_PATCH] [--enable-segmentation]
               [--sam-checkpoint SAM_CHECKPOINT]
               [--sam-model-type {vit_h,vit_l,vit_b}] [--gs-zoom GS_ZOOM]
               [--gs-source GS_SOURCE] [-v] [--version]
               soc_path
```

| Option | Default | Description |
|---|---|---|
| `soc_path` | (required) | SOC map (GeoTIFF, g/kg) |
| `-w`, `--workspace` | `PySDS_workspace` | Output folder, created in your home directory |
| `-s`, `--nsize` | `1000` | Number of patches |
| `-p`, `--patch-size` | `128` | Patch side length (pixels) |
| `-o`, `--overlap` | none | Patch stride in rows and columns |
| `-f`, `--frequency` | `1.41` | Radar frequency (GHz) |
| `-t`, `--sst_range` | `-25 25 5` | SST start, stop (excluded), step (°C, integers) |
| `-m`, `--ssm_range` | `0.05 0.55 0.1` | SSM start, stop (excluded), step (cm³/cm³) |
| `-l`, `--no_data_threshold` | `0.4` | Maximum fraction of NoData pixels per patch |
| `-b`, `--max_search_distance` | `10` | Search distance (pixels) for NoData filling |
| `-i`, `--smoothing_iterations` | `0` | Smoothing passes after NoData filling |
| `--save-format` | `hdf5` | `hdf5` or `npz` |
| `--stratified` / `--no-stratified` | stratified | Sampling of (SST, SSM) conditions |
| `--samples-per-bin` | `100` | Stratified quota per dielectric bin |
| `--max-per-patch` | `5` | Maximum conditions per patch (stratified) |
| `--enable-segmentation` | on | SAM segmentation of satellite imagery |
| `--sam-checkpoint`, `--sam-model-type` | `sam_vit_h_4b8939.pth`, `vit_h` | SAM model |
| `--gs-zoom`, `--gs-source` | `15`, `Satellite` | Satellite imagery zoom and source |
| `-v`, `--verbose` | off | Debug logging |

---

## Output files

Datasets are named automatically from their parameters:

```text
SynthSoilDiel_f<freq>GHz_p<size>x<count>_multiCond_SST<start><stop>s<step>C_SSMp<start>p<stop>s<step>_<seg>_v1.<ext>
```

| Component | Example | Meaning |
|---|---|---|
| `f<freq>GHz` | `f5p405GHz` | Frequency (decimal point written as `p`) |
| `p<size>x<count>` | `p128x1000` | Patch size × number of patches |
| `multiCond` | `multiCond` | Several (SST, SSM) conditions per patch |
| `SST<start><stop>s<step>C` | `SSTm30p30s5C` | SST arguments (`m` minus, `p` plus) |
| `SSMp<start>p<stop>s<step>` | `SSMp05p65s05` | SSM arguments × 100 |
| `withSeg` / `noSeg` | `withSeg` | Satellite segmentation on or off |
| `.h5` / `.npz` | `.h5` | HDF5 or NumPy archive |

The command-line example above produces `SynthSoilDiel_f5p405GHz_p128x1000_multiCond_SSTm30p30s5C_SSMp05p65s05_withSeg_v1.h5`. The SST and SSM parts record the arguments as given, including the excluded stop value.

Each patch stores the complex soil DC for every simulated condition, the SOC map, the SST and SSM values, the patch extent (WGS 84), summary statistics of the inputs and outputs, and the pedotransfer functions used, so that SOM and SBD can be reproduced from SOC.

---

## Statistical report

```sh
pysdsim-report <path_to_h5_file> [--bins <percentiles>] [--output-dir <directory>]
```

* `--bins`: percentile bins for the distribution report (default `0 10 25 50 75 90 100`).
* `--output-dir`: output directory (default `pysdsim_report_output`).

For example,

```sh
pysdsim-report ~/MyPySDS_Workspace/PySDS_output/SynthSoilDiel_f5p405GHz_p128x1000_multiCond_SSTm30p30s5C_SSMp05p65s05_withSeg_v1.h5 --output-dir my_report
```

writes `my_report/SynthSoilDiel_f5p405GHz_p128x1000_multiCond_SSTm30p30s5C_SSMp05p65s05_withSeg_v1_report.html`, with descriptive statistics, percentile analysis, outlier detection (IQR and z-score) and plots (histograms, scatter plots, Q–Q plots and a heatmap of the patch distribution).

Large SOC rasters are read in blocks of rows, so maps larger than the available memory can be processed.

---

## Example datasets

Two example soil DC datasets generated by PySDS, with patch sizes of 64 and 128 pixels, are available on Zenodo: [https://doi.org/10.5281/zenodo.15998448](https://doi.org/10.5281/zenodo.15998448).

---

## Contributing

Contributions are welcome:

1. Fork the project.
2. Create a feature branch (`git checkout -b feature/AmazingFeature`).
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`).
4. Push the branch (`git push origin feature/AmazingFeature`).
5. Open a pull request.

---

## License

Distributed under the MIT License. See `LICENSE` for details.

---

## Citation

If you use PySDS or the datasets it generates, please cite the accompanying paper (submitted to *Big Earth Data*):

```bibtex
@article{khazaei_pysds,
  title   = {An Automatic Framework for Generating Benchmark Datasets of Soil Dielectric Constant Maps},
  author  = {Khazaei, Morteza and Magagi, Ramata and Go{\"i}ta, Kalifa and Karavayskiy, Andrey Y.},
  journal = {Big Earth Data},
  note    = {Submitted}
}
```

---

## References

* Mironov, V., & Savin, I. (2015). A temperature-dependent multi-relaxation spectroscopic dielectric model for thawed and frozen organic soil at 0.05–15 GHz. *Physics and Chemistry of the Earth, Parts A/B/C*, 83, 57–64. https://doi.org/10.1016/j.pce.2015.02.011
* Mironov, V., & Savin, I. (2016). Temperature-dependent spectroscopic dielectric model at 0.05–16 GHz for a thawed and frozen Alaskan organic soil. In *Satellite Soil Moisture Retrieval* (pp. 169–186). Elsevier. https://doi.org/10.1016/B978-0-12-803388-3.00009-7
* Mironov, V., & Savin, I. (2019). Spectroscopic multirelaxation dielectric model of thawed and frozen Arctic soils considering the dependence on temperature and organic matter content. *Izvestiya, Atmospheric and Oceanic Physics*, 55, 986–995. https://doi.org/10.1134/S0001433819090305
* Savin, I., Mironov, V., Muzalevskiy, K., Fomin, S., Karavayskiy, A., Ruzicka, Z., & Lukin, Y. (2020). Dielectric database of organic Arctic soils (DDOAS). *Earth System Science Data*, 12, 3481–3487. https://doi.org/10.5194/essd-12-3481-2020
