Metadata-Version: 2.4
Name: tdmrsdm
Version: 2026.10.0
Summary: Temperature-Dependent Multi-Relaxation Spectroscopic Dielectric Model (TD-MRSDM) of soil permittivity, with the published, PySDS (v3) and organic-plus-mineral (allsoils_v1) constant sets.
Author-email: Morteza Khazaei <morteza.khazaei@usherbrooke.ca>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Morteza-Khazaei/TD-MRSDM
Project-URL: Issues, https://github.com/Morteza-Khazaei/TD-MRSDM/issues
Keywords: soil,dielectric,permittivity,microwave,remote sensing,TD-MRSDM
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Physics
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Provides-Extra: analysis
Requires-Dist: pandas>=2.0; extra == "analysis"
Requires-Dist: scipy>=1.10; extra == "analysis"
Requires-Dist: scikit-learn>=1.2; extra == "analysis"
Requires-Dist: xlrd>=2.0; extra == "analysis"
Provides-Extra: figures
Requires-Dist: tdmrsdm[analysis]; extra == "figures"
Requires-Dist: matplotlib>=3.7; extra == "figures"
Requires-Dist: seaborn>=0.12; extra == "figures"
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == "torch"
Provides-Extra: all
Requires-Dist: tdmrsdm[analysis,figures,torch]; extra == "all"
Provides-Extra: dev
Requires-Dist: tdmrsdm[all]; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: mypy>=1.5; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Dynamic: license-file

# TD-MRSDM: Temperature-Dependent Multi-Relaxation Spectroscopic Dielectric Model

[![Python Version](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![PyPI](https://img.shields.io/badge/PyPI-tdmrsdm-blue)](https://pypi.org/project/tdmrsdm/)

A Python implementation of the Temperature-Dependent Multi-Relaxation Spectroscopic Dielectric Model for characterizing the complex dielectric properties of organic Arctic soils across a wide range of frequencies, temperatures, and moisture conditions.

<p align="center">
  <img src="figures/TD-MRSDM.png" width="800" alt="TD-MRSDM Model Overview">
</p>

## Overview

**TD-MRSDM** is a physics-based dielectric model specifically designed for organic-rich soils in Arctic and permafrost regions. The model accurately predicts the complex permittivity (real and imaginary components) of soil across:

- **Frequency Range**: 0.05 to 15 GHz (VHF to Ku-band)
- **Temperature Range**: -40°C to +25°C (frozen to thawed states)
- **Soil Moisture**: Air-dry to field capacity
- **Organic Matter**: 30% to 90% by weight

This implementation is based on the peer-reviewed model published by [Mironov & Savin (2015)](https://doi.org/10.1016/j.pce.2015.02.011) and validated against the comprehensive **Dielectric Database of Organic Arctic Soils (DDOAS)** containing over 1.5 million measurements.

## Key Features

- **Physics-Based Multi-Relaxation Model**: Accounts for bound, transient, and unbound (or ice) water states
- **Wide Environmental Coverage**: Handles frozen and thawed soil conditions across Arctic temperature ranges
- **High Accuracy**: R² > 0.97 for refractive index against the DDOAS laboratory measurements
- **Easy to Use**: Simple Python API with just 5 parameters and only NumPy as a required dependency
- **Comprehensive Validation System**: Built-in tools for model evaluation against DDOAS database
- **Multiple Constant Sets**: Published 2019 constants or the re-tuned v1, v2 and v3 sets
- **Publication-Quality Outputs**: Generate high-resolution validation figures (800 DPI)
- **Fast Computation**: Vectorized NumPy operations for efficient batch processing

## Scientific Background

### The Model

The TD-MRSDM describes soil dielectric behavior by considering three distinct water phases:

1. **Bound Water**: Strongly bound to soil particles, limited mobility
2. **Transient Water**: Intermediate binding state
3. **Unbound Water**: Free water or ice (below 0°C)

The water forms are separated by two transition moistures, m<sub>g1</sub> (bound → transient) and m<sub>g2</sub> (transient → unbound or ice), which depend on organic matter and temperature. The refractive index and attenuation of the soil are piecewise-linear in gravimetric moisture between these thresholds, as in the published model.

### Applications

TD-MRSDM is particularly useful for:

- **Remote Sensing**: Soil moisture retrieval from satellite microwave observations (SMAP, SMOS, Sentinel-1)
- **Climate Modeling**: Permafrost dynamics and active layer monitoring
- **Ground-Penetrating Radar (GPR)**: Subsurface characterization in Arctic regions
- **Electromagnetic Modeling**: Antenna design and signal propagation in cold regions
- **Arctic Research**: Understanding soil-water-microwave interactions in permafrost environments

## Installation

### From PyPI (Recommended)

```bash
pip install tdmrsdm
```

### From Source

```bash
git clone https://github.com/Morteza-Khazaei/TD-MRSDM.git
cd TD-MRSDM
pip install -e .
```

### Dependencies

The forward model (`TDMRSDM`) needs only `numpy`. The other parts are optional extras:

| Extra | Installs | Needed for |
|-------|----------|------------|
| *(none)* | `numpy` | `TDMRSDM` forward model |
| `analysis` | `pandas`, `scipy`, `scikit-learn`, `xlrd` | `DDOASDataset`, `TrainableTDMRSDM`, `tdmrsdm.dependence`, `tdmrsdm.optimization` |
| `figures` | `analysis` + `matplotlib`, `seaborn` | `tdmrsdm.figure` |
| `torch` | `torch` | `TorchTDMRSDM` (differentiable port) and the GPU optimizer |
| `all` | all of the above | everything |

```bash
pip install "tdmrsdm[analysis]"   # or [figures], [torch], [all]
```

Python 3.8 or newer is required.

## Quick Start

### Basic Usage

```python
import numpy as np
from tdmrsdm import TDMRSDM

# Define soil and environmental conditions
frequency_GHz = 1.41        # Frequency in GHz (e.g., L-band)
temperature = 5.0           # Soil temperature in °C
organic_matter = 50.0       # Soil organic matter content in %
bulk_density = 0.95         # Soil bulk density in g/cm³
moisture = 0.30             # Volumetric soil moisture in m³/m³

# Initialize the model
model = TDMRSDM(
    freq_GHz=frequency_GHz,
    sst=temperature,
    som=organic_matter,
    sbd=bulk_density,
    ssm=moisture
)

# Compute complex permittivity
epsilon = model.run()

print(f"Complex Permittivity: {epsilon}")
print(f"Real Part (ε'): {epsilon.real}")
print(f"Imaginary Part (ε''): {epsilon.imag}")
```

### Computing Soil Dielectric Properties

```python
import numpy as np
import matplotlib.pyplot as plt
from tdmrsdm import TDMRSDM

# Define parameter ranges
temperatures = np.arange(-30, 26, 5)     # -30°C to +25°C
moistures = np.arange(0.05, 0.86, 0.05) # 5% to 85%
frequency = 1.41                          # L-band (SMAP frequency)

# Storage for results
real_parts = []
imag_parts = []

# Compute dielectric properties
for moisture in moistures:
    for temp in temperatures:
        model = TDMRSDM(
            freq_GHz=frequency,
            sst=temp,
            som=50.0,      # 50% organic matter
            sbd=0.95,      # 0.95 g/cm³
            ssm=moisture
        )
        eps = model.run()
        real_parts.append(eps.real[0])
        imag_parts.append(eps.imag[0])

# Visualization
plt.figure(figsize=(10, 6))
plt.scatter(moistures, real_parts, label="Real ε'", alpha=0.6)
plt.scatter(moistures, imag_parts, label="Imaginary ε''", alpha=0.6)
plt.xlabel('Volumetric Soil Moisture (m³/m³)')
plt.ylabel('Dielectric Constant')
plt.title('TD-MRSDM: Soil Dielectric Properties vs Moisture')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()
```

<p align="center">
  <img src="figures/soil_dc_vs_ssm.png" width="800" alt="Soil Dielectric Properties">
</p>

## Model Parameters

| Parameter | Symbol | Description | Unit | Typical Range |
|-----------|--------|-------------|------|---------------|
| `freq_GHz` | f | Electromagnetic frequency | GHz | 0.05 - 15 |
| `sst` | T | Soil surface temperature | °C | -40 to +25 |
| `som` | OM | Soil organic matter content | % | 30 - 90 |
| `sbd` | ρb | Soil bulk density | g/cm³ | 0.4 - 1.3 |
| `ssm` | θ | Volumetric soil moisture | cm³/cm³ | 0.0 - 0.85 |

### Optional Parameters

- `optimize` (str, optional): Choose the constant set
  - `None`: published constants of Mironov & Savin (2019) (default)
  - `'v1'`, `'v2'`: earlier re-tuned sets, kept for reproducibility
  - `'v3'`: constants re-tuned on the denoised DDOAS by differential evolution (NRMSE objective); this is the set used by [PySDS](https://github.com/Morteza-Khazaei/PySDS) and listed in Table A1 of the PySDS paper
  - `'allsoils_v1'`: constants refitted jointly on organic and mineral laboratory soils, for soils outside the organic range (see below)
- `constants_override` (dict, optional): use your own constants with the same structure as the JSON files

```python
# Use the v3 constants (as PySDS does)
model = TDMRSDM(
    freq_GHz=1.41,
    sst=5.0,
    som=50.0,
    sbd=0.95,
    ssm=0.30,
    optimize='v3'
)
```

The re-tuned dry-soil attenuation term of `v3` is slightly negative, so ε″ can be slightly negative (≥ −0.04) for nearly dry soil (SSM below about 0.05 cm³/cm³); clip it at zero if your application needs a passive medium.

### Mineral and organic soils (`allsoils_v1`)

The published, `v1`–`v3` constants were fitted on organic Arctic soils (30–90 % organic matter). `allsoils_v1` was refitted on the 7 organic DDOAS soils together with 7 mineral or low-organic laboratory soils measured at the Kirensky Institute of Physics (1.4–17 % organic matter, Karavayskiy, unpublished), in the tdmrsdm-allsoils project (not yet public). `TDMRSDM` and `TorchTDMRSDM` reproduce that project's model to about 1e-13 (`tests/test_allsoils_parity.py`).

Leave-one-soil-out error (range-normalized NRMSE, %, mean over soils; each soil scored by a fit that did not see it, or by the fixed constants), from the file's metadata:

| Constants | Organic soils RI / NAC | Mineral soils RI / NAC |
|-----------|------------------------|------------------------|
| published 2019 | 2.9 / 4.0 | 10.9 / 9.4 |
| `v3` | 2.7 / 3.3 | 11.0 / 9.1 |
| `allsoils_v1` | 3.4 / 4.3 | 4.1 / 5.4 |

Caveats recorded with the constants:
- no mineral soil was measured below 0 °C, so the frozen branch for mineral soils is constrained by the published values, not by data;
- three mineral soils have organic matter estimated by a pedotransfer function (±6.9 points) and were down-weighted;
- the salinity of the laboratory soils is unknown and is absorbed by the ionic-conductivity constants.

```python
model = TDMRSDM(freq_GHz=1.41, sst=20.0, som=3.0, sbd=1.4, ssm=0.25, optimize='allsoils_v1')
```

## Validation Against DDOAS Database

TD-MRSDM has been extensively validated against the **Dielectric Database of Organic Arctic Soils (DDOAS)**, which contains over 1.5 million measurements from seven Arctic soil samples with organic matter content ranging from 30% to 90%.

### Validation Performance

Training scores of the `v3` constants on the denoised DDOAS, as stored in the metadata of `td_mrsdm_constants_v2019_optimized_v3.json`:

| Metric | Refractive Index (n) | Norm. Attenuation Coeff. (κ) |
|--------|----------------------|------------------------------|
| **R²** | 0.976 | 0.917 |
| **NRMSE** | 6.43% | 30.52% |

### Running Validation Examples

The `examples/` folder contains ready-to-use Jupyter notebooks for validating TD-MRSDM against each DDOAS soil sample:

```bash
# Launch Jupyter and open a validation notebook
jupyter notebook examples/validate_BV.ipynb
```

**Available Validation Notebooks:**

| Notebook | Soil | Location | Organic Matter | Samples |
|----------|------|----------|----------------|---------|
| `validate_BV.ipynb` | Yamal Peninsula BV | Russia | 50% | ~15,000 |
| `validate_MS1.ipynb` | Yamal Peninsula MS1 | Russia | 61% | ~45,000 |
| `validate_MS2.ipynb` | Yamal Peninsula MS2 | Russia | 35% | ~38,000 |
| `validate_AK1.ipynb` | Alaska North Slope AK1 | USA | 80% | ~18,000 |
| `validate_AK2.ipynb` | Alaska North Slope AK2 | USA | 90% | ~16,000 |
| `validate_TM.ipynb` | Taimyr Peninsula TM | Russia | 38% | ~21,000 |

Each notebook generates:
- 4 publication-quality figures (800 DPI)
- Comprehensive validation metrics (R², NRMSE%, MBE)
- CSV files with detailed results
- Text reports

See [examples/README.md](examples/README.md) for detailed instructions.

### Using the Validation API

This API needs the `analysis` extra. The DDOAS files are in `data/ddoas/` of the repository and are not part of the PyPI package: run from a clone of the repository, or pass `data_dir=` to `DDOASDataset`.

```python
from tdmrsdm.trainable import TrainableTDMRSDM
from tdmrsdm.dataset.ddoas_dataset import DDOASDataset

# Load and filter the DDOAS benchmark manually
dataset = DDOASDataset(verbose=False)
bv_df = dataset.load_site("BV")
subset = bv_df[(bv_df["freq_GHz"] <= 5.0) & (bv_df["st"].between(-5, 5))]   # st: soil temperature (degC)

model = TrainableTDMRSDM(verbose=False)
ri_pred, nac_pred, metrics, df = model.predict(subset, return_dataframe=True)

print(f"Refractive Index R²: {metrics['ri_r2']:.4f}")
print(f"NAC NRMSE%: {metrics['nac_nrmse_pct']:.2f}%")
```

## DDOAS Database

The project includes the complete **Dielectric Database of Organic Arctic Soils (DDOAS)** in the `data/ddoas/` directory. This database contains:

- 7 organic-rich Arctic soil samples
- Measurements across ~10 MHz to ~16 GHz
- Temperature range: -40°C to +25°C
- Over 1.5 million individual measurements
- Coaxial-waveguide method under laboratory conditions

**Citation:**
> Savin, I., Mironov, V., Muzalevskiy, K., Fomin, S., Karavayskiy, A., Ruzicka, Z., and Lukin, Y.: Dielectric database of organic Arctic soils (DDOAS), Earth Syst. Sci. Data, 12, 3481–3487, https://doi.org/10.5194/essd-12-3481-2020, 2020.

**Data Access:** https://doi.org/10.5281/zenodo.3819912

See [data/ddoas/README.md](data/ddoas/README.md) for detailed database documentation.

## Project Structure

```
TD-MRSDM/
├── src/tdmrsdm/
│   ├── core.py               # TDMRSDM: NumPy forward model (numpy only)
│   ├── torch_core.py         # TorchTDMRSDM: differentiable PyTorch port   [torch]
│   ├── constants/            # constant sets: 2019 published, optimized v1, v2, v3, allsoils_v1
│   ├── dataset/              # DDOASDataset: DDOAS loading and filtering   [analysis]
│   ├── trainable.py          # TrainableTDMRSDM: prediction and metrics    [analysis]
│   ├── dependence/           # dependency analysis                         [analysis]
│   ├── optimization/         # constant re-tuning                          [analysis]
│   └── figure/               # validation figures                          [figures]
├── examples/                 # Jupyter notebooks (validation, heatmaps, SOC maps)
├── data/ddoas/               # DDOAS .xls files (repository only, not in the PyPI package)
├── figures/                  # README figures
├── tests/                    # pytest suite
└── pyproject.toml            # package configuration
```

## Advanced Features

### Batch Processing

Process multiple soil conditions efficiently:

```python
import numpy as np
from tdmrsdm import TDMRSDM

# Define arrays of conditions
frequencies = np.array([1.41, 6.9, 10.7])  # L, C, X bands
temperatures = np.array([-10, 0, 10, 20])
moistures = np.array([0.1, 0.3, 0.5])

results = []
for freq in frequencies:
    for temp in temperatures:
        for moist in moistures:
            model = TDMRSDM(freq, temp, som=50, sbd=0.95, ssm=moist)
            eps = model.run()
            results.append({
                'freq': freq,
                'temp': temp,
                'moisture': moist,
                'real': eps.real[0],
                'imag': eps.imag[0]
            })

# Convert to DataFrame for analysis
import pandas as pd
df = pd.DataFrame(results)
```

### Temperature Array Input

You can pass temperature arrays directly:

```python
import numpy as np
from tdmrsdm import TDMRSDM

# Array of temperatures
temps = np.array([-20, -10, 0, 10, 20])

# Single model call for multiple temperatures
model = TDMRSDM(
    freq_GHz=1.41,
    sst=temps,  # Pass array directly
    som=50,
    sbd=0.95,
    ssm=0.30
)

epsilon = model.run()
print(f"Permittivity at different temps: {epsilon}")
```

## Testing

```bash
pip install -e ".[dev]"
python -m pytest tests/
```

The DDOAS tests read `data/ddoas/` and the PyTorch parity tests need the `torch` extra.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

Please ensure your code:
- Follows PEP 8 style guidelines
- Includes appropriate documentation
- Passes all existing tests
- Adds new tests for new features

## Citation

If you use TD-MRSDM in your research, please cite both the original model paper and this implementation:

### Model Paper

```bibtex
@article{mironov2015tdmrsdm,
  title={A temperature-dependent multi-relaxation spectroscopic dielectric model for thawed and frozen organic soil at 0.05--15 GHz},
  author={Mironov, Valery and Savin, Igor},
  journal={Physics and Chemistry of the Earth, Parts A/B/C},
  volume={83},
  pages={57--64},
  year={2015},
  publisher={Elsevier},
  doi={10.1016/j.pce.2015.02.011}
}
```

### DDOAS Database

```bibtex
@article{savin2020ddoas,
  title={Dielectric database of organic Arctic soils (DDOAS)},
  author={Savin, Igor and Mironov, Valery and Muzalevskiy, Konstantin and Fomin, Sergey and Karavayskiy, Andrey and Ruzicka, Zdenek and Lukin, Yuriy},
  journal={Earth System Science Data},
  volume={12},
  number={4},
  pages={3481--3487},
  year={2020},
  publisher={Copernicus GmbH},
  doi={10.5194/essd-12-3481-2020}
}
```

### This Implementation

For the Python implementation:
```
Khazaei, M. (2025). TD-MRSDM: Temperature-Dependent Multi-Relaxation Spectroscopic
Dielectric Model - Python Implementation. GitHub repository.
https://github.com/Morteza-Khazaei/TD-MRSDM
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

### Copyright Notice

Copyright (c) 2023 Morteza Khazaei

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

## Author

**Morteza Khazaei**
- Email: [morteza.khazaei@usherbrooke.ca](mailto:morteza.khazaei@usherbrooke.ca)
- GitHub: [@Morteza-Khazaei](https://github.com/Morteza-Khazaei)
- Affiliation: Université de Sherbrooke

## Acknowledgments

- **Valery Mironov** and **Igor Savin** for developing the original TD-MRSDM model
- DDOAS team for providing the comprehensive validation database
- Arctic research community for valuable feedback and use cases

## References

1. **Mironov, V., & Savin, I. (2015)**. A temperature-dependent multi-relaxation spectroscopic dielectric model for thawed and frozen organic soil at 0.05–15 GHz. *Physics and Chemistry of the Earth, Parts A/B/C*, 83, 57-64. [https://doi.org/10.1016/j.pce.2015.02.011](https://doi.org/10.1016/j.pce.2015.02.011)

2. **Savin, I., Mironov, V., Muzalevskiy, K., Fomin, S., Karavayskiy, A., Ruzicka, Z., & Lukin, Y. (2020)**. Dielectric database of organic Arctic soils (DDOAS). *Earth System Science Data*, 12(4), 3481-3487. [https://doi.org/10.5194/essd-12-3481-2020](https://doi.org/10.5194/essd-12-3481-2020)

3. **DDOAS Database**: [https://doi.org/10.5281/zenodo.3819912](https://doi.org/10.5281/zenodo.3819912)

## Support

For questions, issues, or feature requests:
- Open an issue on [GitHub Issues](https://github.com/Morteza-Khazaei/TD-MRSDM/issues)
- Contact the author at [morteza.khazaei@usherbrooke.ca](mailto:morteza.khazaei@usherbrooke.ca)

## Changelog

### 2026.10.0
- Adds the `v3` constants (re-tuned on the denoised DDOAS) used by PySDS
- Adds the `allsoils_v1` constants (organic and mineral soils), with parity tests against tdmrsdm-allsoils
- Ships the constants JSON files in the wheel (previous releases could fail with "Cannot find constants folder")
- `import tdmrsdm` needs only NumPy; DDOAS, analysis, figure and PyTorch tools are optional extras
- Adds `TorchTDMRSDM`, a differentiable PyTorch port with parity tests against the NumPy model
- Supports Python 3.8+
- No warning is printed for the default `optimize=None`
- Fixes the dependence analysis for current pandas (empty bins no longer produce NaN weights)

### 2025.7.3
- Previous PyPI release

---

**Happy Modeling!** If you find TD-MRSDM useful for your research, please consider starring the repository and citing our work.
