# EDref v3 - Complete Manual and Reference

A comprehensive reference for the EDref crystallographic refinement engine.

---

## Table of Contents

1. [Overview](#1-overview)
2. [Installation & Setup](#2-installation--setup)
3. [Quick Start](#3-quick-start)
4. [Package Architecture](#4-package-architecture)
5. [Data Classes Reference](#5-data-classes-reference)
6. [Core Module Reference](#6-core-module-reference)
7. [I/O Module Reference](#7-io-module-reference)
8. [Refinement Module Reference](#8-refinement-module-reference)
9. [Analysis Module Reference](#9-analysis-module-reference)
10. [Mathematical Background](#10-mathematical-background)
11. [SHELXL Compatibility](#11-shelxl-compatibility)
12. [Code Examples](#12-code-examples)
13. [Troubleshooting](#13-troubleshooting)
14. [Validated Datasets](#14-validated-datasets)
15. [Resolved Issues History](#15-resolved-issues-history)
16. [Appendices](#16-appendices)

---

## 1. Overview

### 1.1 What is EDref?

EDref (Electron Diffraction Refinement Engine) is a Python library for crystallographic structure refinement. It implements full-matrix least-squares refinement on F² (intensity), designed to be compatible with SHELXL while being modular and extensible.

### 1.2 Key Features

- **SHELXL-compatible**: Reproduces SHELXL's kinematic refinement workflow
- **High performance**: Optional Numba parallel acceleration (7-8x speedup)
- **Modular design**: Use only the components you need
- **Type hints**: Full type annotations for better IDE support
- **Electron diffraction support**: Custom SFAC coefficients for electron scattering
- **Robust scaling**: Multiple outlier-resistant scaling methods
- **No hard-coded paths**: All file paths are parameters

### 1.3 Capabilities

| Feature | Status |
|---------|--------|
| Structure factor calculation | ✓ Complete |
| Isotropic temperature factors | ✓ Complete |
| Anisotropic temperature factors | ✓ Complete & Validated |
| Position refinement (x, y, z) | ✓ Complete & Validated |
| U_iso refinement | ✓ Complete & Validated |
| U_aniso refinement (6 params) | ✓ Complete & Validated |
| Scale factor refinement | ✓ Complete & Validated |
| SHELXL weighting scheme | ✓ Complete & Validated |
| Weight optimization | ✓ Complete & Validated |
| Reflection merging | ✓ Complete |
| Special position constraints | ✓ Complete |
| Robust scaling methods | ✓ Complete |
| Visualization & diagnostics | ✓ Complete |
| Dynamical correction (ED) | ✓ Complete |
| Occupancy refinement | ✗ Not implemented |
| Distance restraints (DFIX) | ✗ Not implemented |
| Hydrogen riding (AFIX) | ✗ Not implemented |
| Twin refinement | ✗ Not implemented |

### 1.4 Validation Summary

EDref has been validated against SHELXL/Olex2. See [Section 14](#14-validated-datasets) for detailed test results.

| Dataset | R1 Match | wR2 Match | Notes |
|---------|----------|-----------|-------|
| Aspirin | ✓ within 0.2% | ✓ within 0.01% | Primary validation |
| MFM-300 | ✓ within 1% | - | Electron diffraction |
| LTA (Pm-3m) | ✓ params match | - | High-symmetry ED, auto-constraints |

---

## 2. Installation & Setup

### 2.1 Requirements

**Required:**
- Python >= 3.10
- NumPy >= 1.21
- SciPy >= 1.7

**Optional (for 7-8x performance boost):**
- Numba >= 0.56

```bash
# Install Numba for parallel acceleration
pip install numba
```

When Numba is installed, `build_design_matrix()` automatically uses parallel
multi-threaded computation for significant speedup. Without Numba, EDref falls
back to sequential NumPy-based computation (still functional, just slower).

### 2.2 Installation Methods

```bash
cd EDref_v3

# Option 1: Add to PYTHONPATH (recommended for development)
export PYTHONPATH=$PWD/src:$PYTHONPATH

# Option 2: Install with pip
pip install .

# Option 3: Editable install for development
pip install -e .
```

### 2.3 Verify Installation

```python
import edref
print(edref.__version__)  # Should print "3.0.0"
```

### 2.4 Running Tests

```bash
cd EDref_v3
PYTHONPATH=src:$PYTHONPATH python3 -m pytest tests/ -v
```

**Note:** Use `python3` not `python` in this environment.

### 2.5 Test Data Location

**Core datasets** (in `example_data_do_not_modify/`):
- **Aspirin**: `example_data_do_not_modify/aspirin/Aspirin.ins`, `Aspirin.hkl`
- **MFM-300**: `example_data_do_not_modify/mfm300/2.ins`, `2.hkl`
- **LTA Zeolite**: `example_data_do_not_modify/LTA/LTA1/` and `LTA4/` (ED, Pm-3m high symmetry)

Do NOT modify these files - they are used for regression testing.

**Wisconsin Test Kit** (in `test_kit_wisc/`):

24 test structures from UW Madison Molecular Structure Laboratory covering:

| Crystal System | Count | Examples |
|---------------|-------|----------|
| Monoclinic | 10 | Aspirin, Structure1, Structure2, Structure5, Structure9 |
| Triclinic | 6 | Structure4, Structure7, Structure10, Structure14, Structure17, Structure29 |
| Orthorhombic | 7 | Structure6, Structure8, Structure15, Structure19, Structure24, Structure26, Structure30 |
| Tetragonal | 1 | Structure22 |

**Elements covered:** B, Br, C, Cd, Cl, Cr, F, H, Mn, N, Ni, O, P, Pd, Re, Ru, S, Si, Ti

**Source:** https://xray.chem.wisc.edu/crystallographic-problems/

See `test_kit_wisc/README.md` for complete documentation.

---

## 3. Quick Start

### 3.1 Basic Refinement Workflow

```python
from edref import (
    InsFileReader, HklFileReader,
    SpaceGroup, calculate_reciprocal_cell,
    merge_reflections, refine_structure
)

# 1. Load structure from SHELXL files
ins = InsFileReader("structure.ins")
ins.read()

hkl = HklFileReader("structure.hkl")
hkl.read()

# 2. Setup crystallographic objects
reciprocal = calculate_reciprocal_cell(ins.cell)
spacegroup = SpaceGroup(ins.latt, ins.symm)

# 3. Merge reflections (returns MergedReflection objects)
merged = merge_reflections(hkl.reflections, spacegroup)

# 4. Convert to tuple format for refine_structure()
# CRITICAL: refine_structure expects [(h, k, l, Fo², sigma), ...]
hkl_data = [(m.h, m.k, m.l, m.intensity, m.sigma) for m in merged]

# 5. Run refinement
atoms, history = refine_structure(
    atoms=ins.atoms,
    hkl_data=hkl_data,
    sfac_elements=ins.sfac_elements,
    spacegroup=spacegroup,
    reciprocal_cell=reciprocal,
    wavelength=ins.wavelength,
    refine_positions=True,
    refine_scale=True,
    max_cycles=20,
    verbose=True
)

# 6. Check results
final = history[-1]
print(f"R1 = {final.R1*100:.2f}%")
print(f"wR2 = {final.wR2*100:.2f}%")
print(f"GooF = {final.GooF:.3f}")
```

### 3.2 Calculate Structure Factors Only

```python
from edref import (
    InsFileReader, calculate_reciprocal_cell,
    SpaceGroup, calculate_structure_factor
)

ins = InsFileReader("structure.ins")
ins.read()

reciprocal = calculate_reciprocal_cell(ins.cell)
spacegroup = SpaceGroup(ins.latt, ins.symm)

# Calculate F(hkl) for a single reflection
Fc = calculate_structure_factor(
    h=1, k=0, l=0,
    atoms=ins.atoms,
    sfac_elements=ins.sfac_elements,
    spacegroup=spacegroup,
    reciprocal_cell=reciprocal,
    wavelength=ins.wavelength
)

print(f"|Fc|² = {abs(Fc)**2:.2f}")
```

### 3.3 Command-Line Interface (CLI)

EDref provides a command-line interface for running refinements directly from the terminal without writing Python scripts.

#### 3.3.1 Basic Usage

```bash
# Run EDref refinement only
python3 -m edref refine structure.ins --edref

# Run SHELXL refinement only
python3 -m edref refine structure.ins --shelxl

# Run comparative refinement (both SHELXL and EDref)
python3 -m edref refine structure.ins --compare
```

The CLI automatically finds the `.hkl` file based on the `.ins` file name.

#### 3.3.2 Options

| Option | Description |
|--------|-------------|
| `--edref` | Run EDref refinement only |
| `--shelxl` | Run SHELXL refinement only |
| `--compare` | Run both and compare results (default) |
| `--resolution MIN MAX` | Resolution limits in Ångströms (e.g., `--resolution 99 0.8`) |
| `--cycles N` | Total refinement cycles (default: 100) |
| `--batch N` | Batch size for weight updates (default: 10) |
| `--wght A B` | Initial weight parameters (default: 0.1 0.0) |
| `--exti` | Enable extinction correction (optimize EXTI every batch cycles) |
| `--exti-value X` | Initial extinction parameter value (default: 0.0 or from .ins file) |
| `--quiet` | Suppress verbose output |
| `--no-plot` | Disable automatic visualization (plots run by default) |
| `--save-plots PATH` | Save plots to files (e.g., `--save-plots output` creates `output_refine.png`) |
| `--scaling-comparison` | Run all 10 robust scaling methods and compare |
| `--iterative` | Enable iterative outlier removal (refine→analyze→omit→repeat) |
| `--iter-rounds N` | Maximum number of rounds (default: 20) |
| `--iter-omit N` | Reflections to omit per round (default: 5) |
| `--iter-percent P` | Stop after omitting P% of reflections (optional) |
| `--iter-threshold Z` | Only omit reflections with \|z_robust\| > Z (default: 5.0) |
| `--iter-cycles N` | Cycles for first round (default: 50) |
| `--iter-cycles-subsequent N` | Cycles for subsequent rounds (default: 20) |
| `--iter-output DIR` | Output directory (default: `<input>_iterative/`) |

#### 3.3.3 Examples

```bash
# Full anisotropic refinement with resolution cutoff
python3 -m edref refine Aspirin.ins --compare --resolution 99 0.8

# Quick 50-cycle EDref refinement
python3 -m edref refine structure.ins --edref --cycles 50

# SHELXL-only with custom initial weights
python3 -m edref refine structure.ins --shelxl --wght 0.05 1.0

# EDref with extinction correction (important for ED data!)
python3 -m edref refine structure.ins --edref --exti
```

#### 3.3.4 Output Format

The CLI reports comprehensive statistics including final weights:

**Single refinement output:**
```
PROGRAM  | R1(obs)=X.XX% | R1(all)=X.XX% | wR2=X.XX% | GooF=X.XXX | k=X.XXXX | WGHT X.XXXX X.XX | status
```

**Comparative refinement output:**
```
Metric               SHELXL       EDref            Δ
------------------------------------------------------------
Reflections            1662        1662           +0
R1(obs) %              5.95        4.72        -1.23
R1(all) %              6.42        5.18        -1.24
wR2 %                 17.27       10.87        -6.40
GooF                   1.404       0.888       -0.516
Scale k                0.7241      0.7189      -0.0052
Parameters              120         120           +0
Cycles                  100          30          -70
WGHT a                0.0707      0.0453      -0.0254
WGHT b                  0.84        0.27        -0.57
```

#### 3.3.5 CLI Classes (for programmatic use)

The CLI runners can also be used programmatically:

```python
from edref.cli import ShelxlRunner, EdrefRunner, ComparativeRunner

# Run SHELXL with batched weight updates
runner = ShelxlRunner(
    ins_path="structure.ins",
    total_cycles=100,
    batch_size=10,
    resolution_max=0.8
)
result = runner.run(verbose=True)
print(f"R1={result.R1_obs*100:.2f}%  WGHT {result.wght_a:.4f} {result.wght_b:.2f}")

# Run EDref
runner = EdrefRunner(
    ins_path="structure.ins",
    total_cycles=100,
    weight_opt_frequency=10
)
result = runner.run(verbose=True)

# Run comparative
runner = ComparativeRunner(
    ins_path="structure.ins",
    resolution_max=0.8
)
shelxl_result, edref_result = runner.run(verbose=True)
```

#### 3.3.6 RefinementResult Class

All runners return a `RefinementResult` object:

```python
@dataclass
class RefinementResult:
    program: str        # "SHELXL" or "EDref"
    cycles: int         # Number of cycles run
    converged: bool     # Whether refinement converged
    R1_obs: float       # R1 for observed data (Fo > threshold)
    R1_all: float       # R1 for all data
    wR2: float          # Weighted R2
    GooF: float         # Goodness of fit
    scale_k: float      # Final scale factor k
    n_reflections: int  # Total reflections
    n_obs: int          # Observed reflections
    n_params: int       # Number of parameters
    wght_a: float       # Final WGHT a parameter
    wght_b: float       # Final WGHT b parameter
    atoms: List[Atom]   # Refined atoms (optional)
    history: List       # Refinement history (EDref only)
```

---

## 4. Package Architecture

### 4.1 Directory Structure

```
src/edref/
├── __init__.py              # Main exports (36 functions/classes)
├── core/                    # Crystallographic fundamentals
│   ├── __init__.py
│   ├── crystallography.py   # Cell calculations, d-spacing
│   ├── symmetry.py          # Space group operations
│   ├── scattering.py        # Atomic scattering factors
│   └── structure_factors.py # F(hkl) calculation
├── io/                      # File I/O
│   ├── __init__.py
│   ├── formats.py           # Data classes
│   └── shelxl.py            # SHELXL file readers
├── refinement/              # Least-squares refinement
│   ├── __init__.py
│   ├── engine.py            # Main refine_structure()
│   ├── parameters.py        # Parameter management
│   ├── derivatives.py       # Analytical derivatives
│   ├── normal_equations.py  # Matrix assembly
│   ├── weighting.py         # SHELXL weighting
│   └── statistics.py        # R-factors, GooF
└── analysis/                # Data processing
    ├── __init__.py
    ├── merging.py           # Reflection merging
    └── robust_scaling.py    # Robust scaling methods
```

### 4.2 Module Dependencies

```
                    ┌─────────────┐
                    │   edref     │
                    │ (__init__)  │
                    └──────┬──────┘
           ┌───────────────┼───────────────┐
           ▼               ▼               ▼
    ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
    │    core/    │ │     io/     │ │ refinement/ │
    └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
           │               │               │
           ▼               ▼               ▼
    crystallography  formats.py     engine.py
    symmetry         shelxl.py      derivatives
    scattering                      parameters
    structure_factors               normal_equations
                                    weighting
                                    statistics
```

### 4.3 Import Conventions

All commonly-used classes and functions are exported from the package root:

```python
# Preferred: import from package root
from edref import InsFileReader, refine_structure, calculate_R1

# Also valid: import from submodules
from edref.io.shelxl import InsFileReader
from edref.refinement.engine import refine_structure
from edref.refinement.statistics import calculate_R1
```

---

## 5. Data Classes Reference

All data classes are defined in `edref/io/formats.py` and exported from the package root.

### 5.1 UnitCell

Represents a crystallographic unit cell.

```python
@dataclass
class UnitCell:
    a: float           # Cell length a (Å)
    b: float           # Cell length b (Å)
    c: float           # Cell length c (Å)
    alpha: float       # Angle α (degrees)
    beta: float        # Angle β (degrees)
    gamma: float       # Angle γ (degrees)
    wavelength: float = 0.71073  # X-ray wavelength (Å)
```

**Example:**
```python
from edref import UnitCell

cell = UnitCell(
    a=11.430, b=6.591, c=11.366,
    alpha=90.0, beta=95.68, gamma=90.0,
    wavelength=0.71073
)
```

### 5.2 ReciprocalCell

Reciprocal space cell parameters (computed from UnitCell).

```python
@dataclass
class ReciprocalCell:
    a_star: float      # Reciprocal length a* (Å⁻¹)
    b_star: float      # Reciprocal length b* (Å⁻¹)
    c_star: float      # Reciprocal length c* (Å⁻¹)
    alpha_star: float  # Reciprocal angle α* (RADIANS, not degrees!)
    beta_star: float   # Reciprocal angle β* (RADIANS)
    gamma_star: float  # Reciprocal angle γ* (RADIANS)
    volume: float      # Unit cell volume (Å³)
```

**CRITICAL:** Reciprocal cell angles are stored in **RADIANS**, not degrees. Do not convert them again:
```python
# CORRECT:
cos_gamma_star = np.cos(reciprocal_cell.gamma_star)

# WRONG - double conversion!
cos_gamma_star = np.cos(np.radians(reciprocal_cell.gamma_star))
```

### 5.3 Atom

Represents an atom in the structure.

```python
@dataclass
class Atom:
    label: str         # Atom label (e.g., "C1", "O2")
    sfac_num: int      # SFAC index (1-based)
    x: float           # Fractional coordinate x
    y: float           # Fractional coordinate y
    z: float           # Fractional coordinate z
    sof: float = 1.0   # Site occupancy factor (SHELXL encoding: 10*m + occ)
    U11: float = 0.05  # Anisotropic U₁₁ (Å²)
    U22: float = 0.05  # Anisotropic U₂₂ (Å²)
    U33: float = 0.05  # Anisotropic U₃₃ (Å²)
    U23: float = 0.0   # Anisotropic U₂₃ (Å²)
    U13: float = 0.0   # Anisotropic U₁₃ (Å²)
    U12: float = 0.0   # Anisotropic U₁₂ (Å²)
    aniso: bool = False  # Whether atom was defined with 6 U values

    def is_isotropic(self) -> bool:
        """Check if atom was defined as isotropic (returns `not self.aniso`)."""

    def U_iso(self) -> float:
        """Get isotropic U value (U_eq for anisotropic atoms)."""

    def occupancy(self) -> float:
        """Extract occupancy from SOF encoding."""

    def get_U_matrix(self) -> np.ndarray:
        """Return 3x3 U tensor matrix."""
```

**U parameter order:** U11, U22, U33, U23, U13, U12 (SHELXL convention)

**Note on `aniso` flag:** The `aniso` attribute tracks whether the atom was *defined* with anisotropic U parameters (6 values in .ins file), regardless of the actual U values. This is important because an atom can have U11=U22=U33 with off-diagonals=0 but still be intended for anisotropic refinement.

### 5.4 Reflection

Raw reflection data from HKL file.

```python
@dataclass
class Reflection:
    h: int             # Miller index h
    k: int             # Miller index k
    l: int             # Miller index l
    intensity: float   # Observed intensity (Fo²)
    sigma: float       # Standard uncertainty σ(Fo²)
    batch: int = 0     # Batch number (optional)
```

### 5.5 MergedReflection

Merged reflection after symmetry averaging.

```python
@dataclass
class MergedReflection:
    h: int             # Miller index h
    k: int             # Miller index k
    l: int             # Miller index l
    intensity: float   # Merged intensity
    sigma: float       # Merged uncertainty
    multiplicity: int  # Number of equivalents merged
```

### 5.6 ScatteringCoefficients

Custom scattering factor coefficients (for electron diffraction).

```python
@dataclass
class ScatteringCoefficients:
    element: str       # Element symbol
    a1: float          # Coefficient a₁
    b1: float          # Coefficient b₁
    a2: float          # Coefficient a₂
    b2: float          # Coefficient b₂
    a3: float          # Coefficient a₃
    b3: float          # Coefficient b₃
    a4: float          # Coefficient a₄
    b4: float          # Coefficient b₄
    c: float           # Constant term c
    f_prime: float = 0.0       # Anomalous f'
    f_double_prime: float = 0.0  # Anomalous f''
```

---

## 6. Core Module Reference

### 6.1 crystallography.py

**`calculate_reciprocal_cell(cell: UnitCell) -> ReciprocalCell`**

Calculate reciprocal cell parameters from unit cell.

```python
from edref import UnitCell, calculate_reciprocal_cell

cell = UnitCell(a=10.0, b=10.0, c=10.0, alpha=90, beta=90, gamma=90)
recip = calculate_reciprocal_cell(cell)
# recip.a_star, recip.b_star, recip.c_star in Å⁻¹
# recip.alpha_star, recip.beta_star, recip.gamma_star in RADIANS
```

**`calculate_d_spacing(h, k, l, reciprocal_cell) -> float`**

Calculate d-spacing for reflection (h, k, l) in Angstroms.

```python
from edref import calculate_d_spacing

d = calculate_d_spacing(1, 0, 0, recip)
print(f"d(1,0,0) = {d:.4f} Å")
```

**`calculate_sin_theta_over_lambda(h, k, l, reciprocal_cell) -> float`**

Calculate sin(θ)/λ (resolution parameter s = 1/(2d)).

```python
from edref import calculate_sin_theta_over_lambda

s = calculate_sin_theta_over_lambda(1, 0, 0, recip)
print(f"s = {s:.4f} Å⁻¹")
```

### 6.2 symmetry.py

**`class SpaceGroup`**

Represents a crystallographic space group.

```python
from edref import SpaceGroup

# Create from SHELXL LATT and SYMM cards
spacegroup = SpaceGroup(latt=-1, symm_cards=["-X, Y+1/2, -Z+1/2"])

# Properties
print(f"Number of operations: {len(spacegroup.operations)}")
print(f"Is centrosymmetric: {spacegroup.is_centrosymmetric}")
print(f"Centering type: {spacegroup.centering_type}")
```

**LATT Convention (SHELXL-standard):**

| LATT value | Meaning |
|------------|---------|
| Negative | Centrosymmetric (inversion added automatically) |
| Positive | Non-centrosymmetric |
| \|N\| = 1 | P (primitive) |
| \|N\| = 2 | I (body-centered) |
| \|N\| = 3 | R (rhombohedral) |
| \|N\| = 4 | F (face-centered) |
| \|N\| = 5-7 | A, B, C (face-centered) |

**Centrosymmetry Detection:**

SpaceGroup determines centrosymmetry from:
1. LATT sign (positive = centrosymmetric, adds inversion center)
2. Presence of inversion operation in SYMM cards

```python
# LATT 1: positive LATT adds inversion automatically (centrosymmetric)
sg1 = SpaceGroup(latt=1, symm_cards=["-X, Y+1/2, -Z+1/2"])
print(sg1.is_centrosymmetric)  # True

# LATT -1: negative LATT means no inversion added (non-centrosymmetric)
sg2 = SpaceGroup(latt=-1, symm_cards=["-X, Y+1/2, -Z+1/2"])
print(sg2.is_centrosymmetric)  # False (unless inversion in operations)
```

**`class SymmetryOperation`**

A single symmetry operation: r' = R·r + t

```python
from edref import SymmetryOperation
import numpy as np

symop = spacegroup.operations[0]
position = np.array([0.1, 0.2, 0.3])
new_position = symop.apply(position)

print(f"Rotation matrix:\n{symop.rotation}")
print(f"Translation: {symop.translation}")
```

### 6.3 scattering.py

**`get_scattering_factor(element, s, coefficients=None) -> complex`**

Calculate atomic scattering factor at resolution s = sin(θ)/λ.

```python
from edref import get_scattering_factor

# X-ray scattering (uses ITC coefficients)
f = get_scattering_factor("C", s=0.5)

# Electron scattering (with custom coefficients)
from edref import ScatteringCoefficients
coeffs = ScatteringCoefficients(
    element="C", a1=2.31, b1=20.84, a2=1.02, b2=10.21,
    a3=1.59, b3=0.569, a4=0.865, b4=51.65, c=0.216
)
f_electron = get_scattering_factor("C", s=0.5, coefficients=coeffs)
```

**`XRAY_SCATTERING_COEFFICIENTS`**

Dictionary of X-ray scattering coefficients from International Tables for Crystallography, Volume C.

**Supported elements (37 total):**
- Light elements: H, B, C, N, O, F
- Sulfur/halogens: S, Cl, Br, I
- Alkali/alkaline earth: Na, Mg, K, Ca
- Metalloids: Si, P
- First-row transition metals: Ti, V, Cr, Mn, Fe, Co, Ni, Cu, Zn
- Second/third-row transition metals: Mo, Ru, Pd, Ag, Cd, W, Re, Pt, Au
- Post-transition metals: Al, Sn, Pb, Bi

```python
from edref import XRAY_SCATTERING_COEFFICIENTS

c_coeffs = XRAY_SCATTERING_COEFFICIENTS["C"]
print(f"Carbon: a1={c_coeffs[0]}, b1={c_coeffs[1]}, ...")

# Check if an element is supported
"RU" in XRAY_SCATTERING_COEFFICIENTS  # True
```

**Note:** Anomalous dispersion corrections (f', f'') for Mo Kα radiation (λ=0.71073 Å) are also available in `ANOMALOUS_DISPERSION_MO_KA`.

### 6.4 structure_factors.py

**`calculate_structure_factor(h, k, l, atoms, sfac_elements, spacegroup, reciprocal_cell, wavelength, sfac_coefficients=None) -> complex`**

Calculate structure factor F(hkl) for a single reflection.

```python
from edref import calculate_structure_factor

Fc = calculate_structure_factor(
    h=1, k=0, l=0,
    atoms=atoms,
    sfac_elements=["C", "H", "O"],
    spacegroup=spacegroup,
    reciprocal_cell=reciprocal,
    wavelength=0.71073,
    sfac_coefficients=None  # Pass ins.sfac_coefficients for ED
)

Fc_sq = abs(Fc)**2
```

**`calculate_structure_factors_batch(hkl_list, atoms, sfac_elements, spacegroup, reciprocal_cell, wavelength, sfac_coefficients=None) -> NDArray`**

Calculate structure factors for multiple reflections (vectorized, much faster).

```python
from edref import calculate_structure_factors_batch

hkl_list = [(1, 0, 0), (0, 1, 0), (0, 0, 1), (1, 1, 0)]
Fc_array = calculate_structure_factors_batch(
    hkl_list, atoms, sfac_elements, spacegroup, reciprocal, wavelength
)
# Returns complex array of shape (n_reflections,)
```

---

## 7. I/O Module Reference

### 7.1 InsFileReader

Read SHELXL .ins/.res files.

```python
from edref import InsFileReader

ins = InsFileReader("structure.ins")
ins.read()

# Available attributes after reading:
ins.cell           # UnitCell object
ins.wavelength     # float
ins.latt           # int (LATT card value)
ins.symm           # List[str] (SYMM cards)
ins.sfac_elements  # List[str] (element symbols)
ins.sfac_coefficients  # Dict or None (custom scattering factors)
ins.atoms          # List[Atom]
ins.fvar           # float (overall scale factor = √k)
ins.wght_a         # float (WGHT parameter a)
ins.wght_b         # float (WGHT parameter b)
```

### 7.2 HklFileReader

Read SHELXL .hkl files (HKLF 4 format).

```python
from edref import HklFileReader

hkl = HklFileReader("structure.hkl")
hkl.read()

print(f"Reflections: {len(hkl.reflections)}")

for refl in hkl.reflections[:5]:
    print(f"({refl.h}, {refl.k}, {refl.l}): I={refl.intensity:.1f}, σ={refl.sigma:.1f}")
```

### 7.3 FcfFileReader

Read SHELXL .fcf files (calculated structure factors).

```python
from edref import FcfFileReader

fcf = FcfFileReader("structure.fcf")
fcf.read()

for refl in fcf.reflections[:5]:
    print(f"({refl.h}, {refl.k}, {refl.l}): Fo²={refl.Fo_sq:.1f}, Fc²={refl.Fc_sq:.2f}")
```

**CRITICAL:** All values in .fcf files are on the **ABSOLUTE scale**:
- Fc² is already on final scale (do NOT multiply by k)
- Fo² has been divided by the refined scale factor k
- σ(Fo²) is similarly scaled

### 7.4 File Formats

#### .ins/.res Format

```
TITL structure name
CELL λ a b c α β γ
ZERR Z esd_a esd_b esd_c esd_α esd_β esd_γ
LATT N           ! N>0: non-centrosym; N<0: centrosym; |N|=centering
SYMM x,y,z       ! Additional symmetry operations
SFAC C H N O     ! Element types (or custom coefficients)
UNIT n1 n2 n3 n4 ! Atoms per unit cell
L.S. 10          ! Refinement cycles
WGHT a b         ! Weight parameters
FVAR osf         ! Overall scale factor √k

! Atom lines: label sfac x y z sof U_iso
C1   1  0.1234  0.2345  0.3456  11.00000  0.025

! Or anisotropic: label sfac x y z sof U11 U22 U33 U23 U13 U12 =
! (= indicates continuation, anisotropic has 6 U values)

HKLF 4
END
```

**SOF encoding:** `SOF = 10*m + occ` where m is FVAR index, occ is occupancy

**Negative U convention:**
- Large negative U (e.g., -1.2): Riding H with U = 1.2 × U_eq(parent)
- Small negative U (e.g., -0.01): Legitimate refined value (use as-is)
- Threshold: U < -0.5 is treated as riding hydrogen

**Line length limit:** All lines must be ≤80 characters or SHELXL fails.

#### .hkl Format (HKLF 4)

```
   h    k    l       Fo²     σ(Fo²)
   1    0    0   1234.56     12.34
   0    0    0      0.00      0.00  ! Terminator
```

Format: h, k, l as I4; Fo² and σ as F8.2

#### File Encoding

All SHELXL files use **ISO-8859-1 (Latin-1)** encoding with CRLF line terminators:
```python
with open(filename, 'r', encoding='latin-1') as f:
```

---

## 8. Refinement Module Reference

### 8.1 engine.py

**`refine_structure(...) -> Tuple[List[Atom], List[RefinementCycle]]`**

Main refinement function.

```python
from edref import refine_structure

atoms, history = refine_structure(
    # Required parameters
    atoms=ins.atoms,                    # Starting model (modified in-place)
    hkl_data=merged_data,               # [(h, k, l, Fo², σ), ...]
    sfac_elements=ins.sfac_elements,    # ["C", "H", "O", ...]
    spacegroup=spacegroup,              # SpaceGroup object
    reciprocal_cell=reciprocal,         # ReciprocalCell object
    wavelength=ins.wavelength,          # Wavelength in Å

    # Refinement control
    max_cycles=20,                      # Maximum cycles (default: 20)
    convergence_threshold=0.05,         # max|shift/esd| threshold
    refine_positions=True,              # Refine x, y, z
    refine_Uiso=False,                  # Refine isotropic U
    refine_Uaniso=False,                # Refine anisotropic U
    refine_scale=True,                  # Refine scale factor

    # Initial values
    initial_scale=None,                 # Auto-calculated if None (RECOMMENDED)

    # Damping
    damping=0.0,                        # Marquardt-Levenberg damping λ

    # Weighting scheme (SHELXL-style is default)
    weighting_scheme='shelxl',          # 'shelxl', 'simple', 'biweight', or 'huber'
    shelxl_a=0.1,                       # WGHT parameter a
    shelxl_b=0.0,                       # WGHT parameter b
    optimize_weights=True,              # Optimize a, b during refinement
    weight_opt_frequency=10,            # Optimize every N cycles

    # Robust scaling for outlier resistance
    robust_scale_method=None,           # 'biweight', 'huber', 'sigma_clip_3', etc.

    # Extinction correction
    refine_extinction=False,            # Enable extinction optimization
    initial_exti=0.0,                   # Starting EXTI value
    exti_opt_frequency=10,              # Optimize every N cycles (like weights)

    # Electron diffraction
    sfac_coefficients=None,             # Custom scattering coeffs (REQUIRED for ED!)

    # Constraints
    constraints=None,                   # Position constraints
    u_constraints=None,                 # U tensor constraints

    # Output
    verbose=True                        # Print progress
)
```

**CRITICAL Parameters:**
- `initial_scale=None`: Let EDref auto-calculate. Do NOT pass `initial_scale=1.0`
- `sfac_coefficients`: MUST pass `ins.sfac_coefficients` for electron diffraction data

**Robust Scaling (for ED data with outliers):**

The `robust_scale_method` parameter enables outlier-resistant scale factor calculation:

| Method | Description | Best For |
|--------|-------------|----------|
| `None` | Standard SHELXL WLS (default) | Good quality data |
| `'biweight'` | Tukey biweight (c=6.0) | **Recommended for ED** |
| `'huber'` | Huber M-estimator | Moderate outliers |
| `'sigma_clip_3'` | Iterative 3σ clipping | Alternative to biweight |
| `'trimmed_15'` | Remove worst 15% | Use with caution |
| `'median'` | Median Fo²/Fc² ratio | Maximum robustness |

**Tested improvements on electron diffraction data:**
- MFM-300: biweight gives **2.2% lower R1** (30.66% vs 32.90%)
- LTA1: biweight gives 0.1% lower R1

```python
# Recommended for electron diffraction with outliers:
atoms, history = refine_structure(
    ...,
    robust_scale_method='biweight',  # Outlier-resistant scaling
)
```

**Extinction Correction:**

The `refine_extinction` parameter enables SHELXL-style extinction correction, which is particularly important for electron diffraction data.

SHELXL extinction formula:
```
Fc_corrected = Fc × [1 + 0.001 × x × Fc² × λ³ / sin(2θ)]^(-1/4)
```

**CRITICAL:** The search range for EXTI is automatically scaled based on wavelength:
- X-ray (λ ~ 0.7 Å): EXTI typically 0-10
- Electron diffraction (λ ~ 0.02 Å): EXTI can be 0-50,000+

This is because the formula contains λ³, so shorter wavelengths require proportionally larger EXTI values (~47,000× larger for ED vs X-ray).

**Tested improvements on electron diffraction data:**
- LTA1 (zeolite): extinction gives **4.5% lower R1** (23.1% vs 27.6% with EXTI ≈ 2800)

```python
# Recommended for electron diffraction:
atoms, history = refine_structure(
    ...,
    refine_extinction=True,      # Enable extinction
    initial_exti=0.0,            # Start from 0 (or use value from .ins file)
    exti_opt_frequency=10,       # Optimize every 10 cycles
)
```

**Weighting Schemes:**

| Scheme | Description |
|--------|-------------|
| `'shelxl'` | SHELXL-style w = 1/[σ² + (aP)² + bP] (default) |
| `'simple'` | Simple w = 1/σ² |
| `'biweight'` | SHELXL × biweight modulation |
| `'huber'` | SHELXL × Huber modulation |

**`class RefinementCycle`**

Results from a single refinement cycle.

```python
cycle = history[-1]  # Final cycle

cycle.cycle         # Cycle number
cycle.R1            # R1 for observed data (Fo² > 2σ)
cycle.R1_all        # R1 for all data
cycle.wR2           # Weighted R2
cycle.GooF          # Goodness of fit
cycle.max_shift     # Maximum |shift/esd|
cycle.scale_k       # Current scale factor k
cycle.n_reflections # Total number of reflections
cycle.n_obs         # Number of observed reflections (Fo² > 2σ)
cycle.n_params      # Number of refined parameters
cycle.weight_a      # Current WGHT a
cycle.weight_b      # Current WGHT b
cycle.exti          # Current extinction parameter (0.0 if not refining)

cycle.is_converged()  # True if max_shift < threshold
```

### 8.2 parameters.py

**`detect_special_positions(atoms, tolerance=0.001) -> List[PositionConstraint]`**

Automatically detect atoms on special positions based on coordinate values.

**Note:** The CLI (`python -m edref refine`) calls this automatically. You only need to call it manually when using the Python API directly.

```python
from edref.refinement.parameters import detect_special_positions

constraints = detect_special_positions(atoms, tolerance=0.001)

for c in constraints:
    print(f"Atom {c.atom_idx}: {c.constraint_type}")
    print(f"  Fixed coords: {c.fixed_coords}")
    print(f"  Coupled coords: {c.coupled_coords}")
```

**Detected constraint types:**

| Type | Position Constraint | Example |
|------|---------------------|---------|
| `4fold_axis` | x=0, y=0 fixed | Atom at (0, 0, z) |
| `mirror_diagonal_xy` | x=y coupled | Atom at (0.3, 0.3, z) |
| `mirror_diagonal_xy_z_fixed` | x=y coupled, z fixed | Atom at (0.3, 0.3, 0.5) |
| `mirror_yz_sum` | z = 1-y | Atom at (x, 0.6, 0.4) |
| `mirror_xz_sum` | z = 1-x | Atom at (0.3, y, 0.7) |
| `mirror_xy_sum` | y = 1-x | Atom at (0.3, 0.7, z) |
| `2fold_xz_fixed` | x, z fixed | Atom at (0, y, 0.5) |
| `z_fixed` | z fixed | Atom at (x, y, 0.5) |

**`detect_U_constraints(atoms, constraints, tolerance=0.001) -> List[UConstraint]`**

Detect U tensor constraints for special positions.

```python
from edref.refinement.parameters import detect_U_constraints

u_constraints = detect_U_constraints(atoms, constraints)
```

**U constraint mapping:**

| Position Type | U Constraints |
|--------------|---------------|
| `4fold_axis` | U11=U22, U12=U13=U23=0 |
| `mirror_diagonal_xy` | U11=U22, U13=U23 |
| `mirror_diagonal_xy_z_fixed` | U11=U22, U13=U23=0 |
| `mirror_yz_sum` | U22=U33, U12=U13 |
| `mirror_xz_sum` | U11=U33, U12=U23 |
| `mirror_xy_sum` | U11=U22, U13=U23 |
| `2fold_xz_fixed` | U12=U13=U23=0 |
| `z_fixed` | U13=U23=0 |

**`class PositionConstraint`**

```python
@dataclass
class PositionConstraint:
    atom_idx: int
    constraint_type: str      # '4fold_axis', 'mirror_diagonal_xy', 'z_fixed', etc.
    fixed_coords: Set[str]    # {'x', 'y'}, {'z'}, etc.
    coupled_coords: List[Tuple[str, str]]  # [('x', 'y')] for x=y or y+z=1
    fixed_values: dict        # {'x': 0.0, 'z': 0.5}, etc.
```

**Note:** For sum constraints (e.g., `mirror_yz_sum`), `coupled_coords=[('y', 'z')]` means `z = 1 - y`, not `z = y`.

### 8.3 weighting.py

**`calculate_shelxl_weight(Fo_sq, Fc_sq, sigma, a, b, scale_k) -> float`**

Calculate SHELXL weight for a single reflection.

```python
from edref import calculate_shelxl_weight

w = calculate_shelxl_weight(
    Fo_sq=100.0,    # Observed intensity (measurement scale)
    Fc_sq=95.0,     # Calculated intensity (absolute scale)
    sigma=5.0,      # Uncertainty (measurement scale)
    a=0.05,         # WGHT a
    b=0.0,          # WGHT b
    scale_k=1.0     # Scale factor
)
```

**`calculate_shelxl_weights_batch(Fo_sq, Fc_sq, sigma, a, b, scale_k) -> NDArray`**

Vectorized weight calculation for all reflections.

**`optimize_shelxl_weights(Fo_sq, Fc_sq, sigma, scale_k, current_a, current_b, n_params, sin_theta_over_lambda=None) -> Tuple[float, float]`**

Optimize SHELXL weight parameters using SHELXL/CAPOW algorithm.

```python
from edref import optimize_shelxl_weights

new_a, new_b = optimize_shelxl_weights(
    Fo_sq, Fc_sq, sigma,
    scale_k=1.0,
    current_a=0.05,
    current_b=0.0,
    n_params=100,
    sin_theta_over_lambda=sin_theta_lambda  # For resolution binning
)
```

**Algorithm:** Minimizes Σ(GooF_bin - 1.0)² across intensity AND resolution bins.

### 8.4 extinction.py

**`apply_extinction_correction(Fc_sq, sin_theta_over_lambda, wavelength, exti) -> NDArray`**

Apply SHELXL-style extinction correction to Fc².

```python
from edref import apply_extinction_correction

Fc_sq_corrected = apply_extinction_correction(
    Fc_sq=Fc_sq_array,              # Uncorrected Fc²
    sin_theta_over_lambda=stl,      # sin(θ)/λ for each reflection
    wavelength=0.01968,             # Wavelength in Å
    exti=2800.0                     # Extinction parameter
)
```

**SHELXL formula:**
```
Fc_corrected = Fc × [1 + 0.001 × x × Fc² × λ³ / sin(2θ)]^(-1/4)
Fc²_corrected = Fc² × [1 + 0.001 × x × Fc² × λ³ / sin(2θ)]^(-1/2)
```

**`optimize_extinction(Fo_sq, Fc_sq, sigma, sin_theta_over_lambda, wavelength, scale_k, shelxl_a, shelxl_b, current_exti, exti_min=None, exti_max=None) -> Tuple[float, float]`**

Find optimal EXTI parameter by minimizing weighted residual sum.

```python
from edref import optimize_extinction

optimal_exti, residual = optimize_extinction(
    Fo_sq, Fc_sq, sigma, sin_theta_over_lambda,
    wavelength=0.01968,
    scale_k=827.0,
    shelxl_a=0.1, shelxl_b=0.0,
    current_exti=0.0,
    # exti_min/max auto-calculated based on wavelength if None
)
```

**IMPORTANT:** The search range is automatically scaled based on wavelength:
- X-ray (λ ~ 0.7 Å): Search range [0, ~10]
- ED (λ ~ 0.02 Å): Search range [0, ~470,000]

This scaling factor is (0.71073 / λ)³, because the extinction formula contains λ³.

### 8.5 statistics.py

**`calculate_R1(Fo_sq, Fc_sq, scale_k, sigma=None, obs_only=False) -> Tuple[float, int]`**

Calculate amplitude R-factor.

```python
from edref import calculate_R1

# R1 for all data
R1_all, n_all = calculate_R1(Fo_sq, Fc_sq, scale_k=1.0)

# R1 for observed data only (Fo² > 2σ)
R1_obs, n_obs = calculate_R1(Fo_sq, Fc_sq, scale_k=1.0, sigma=sigma, obs_only=True)
```

**`calculate_wR2(Fo_sq, Fc_sq, weights, scale_k) -> float`**

Calculate weighted R-factor on F² (absolute scale).

**`calculate_GooF(Fo_sq, Fc_sq, weights, scale_k, n_params) -> float`**

Calculate goodness of fit. Target value is 1.0.

### 8.5 derivatives.py

Analytical derivatives for the design matrix.

**Scalar functions (single reflection):**
- `calculate_structure_factor_with_components()` - F(hkl) with intermediate values
- `calculate_position_derivative_dFsq_dx/dy/dz()` - Position derivatives
- `calculate_Uiso_derivative_dFsq_dU()` - Isotropic U derivative
- `calculate_Uij_derivative_dFsq_dU()` - Anisotropic U derivatives

**Batch functions (vectorized, use these for performance):**
- `calculate_structure_factor_with_components_batch()` - All reflections at once
- `calculate_position_derivatives_batch()` - Returns dx, dy, dz for all reflections
- `calculate_Uiso_derivative_batch()` - U_iso derivatives for all reflections
- `calculate_Uij_derivative_batch()` - Anisotropic U derivatives

**Parallel computation with Numba (v3.4.0+, best performance):**

```python
from edref.refinement.derivatives import (
    calculate_all_derivatives_parallel,
    FusedDerivativesResult,
    NUMBA_AVAILABLE,
    should_use_parallel,
    get_optimal_thread_count,
)

# Check if parallel computation is available
print(f"Numba available: {NUMBA_AVAILABLE}")

# Compute all derivatives in a single parallel pass
result = calculate_all_derivatives_parallel(
    hkl_array,           # Miller indices (n_refl, 3)
    atoms,               # Atom list
    sfac_elements,       # Element symbols
    spacegroup,          # Space group
    reciprocal_cell,     # Reciprocal cell
    wavelength,          # Wavelength
    sfac_coefficients,   # Optional scattering coefficients
    n_threads=None,      # None = auto-detect optimal thread count
)

# Result contains (FusedDerivativesResult dataclass):
result.A                # Real part of F, shape (n_refl,)
result.B                # Imaginary part of F, shape (n_refl,)
result.Fc_squared       # |F|² for all reflections, shape (n_refl,)
result.dFsq_dx          # ∂Fc²/∂x for all atoms, shape (n_atoms, n_refl)
result.dFsq_dy          # ∂Fc²/∂y for all atoms, shape (n_atoms, n_refl)
result.dFsq_dz          # ∂Fc²/∂z for all atoms, shape (n_atoms, n_refl)
result.dFsq_dUiso       # ∂Fc²/∂U_iso for all atoms, shape (n_atoms, n_refl)
result.dFsq_dUij        # ∂Fc²/∂U_ij, shape (n_atoms, 6, n_refl)
                        # Order: U11, U22, U33, U12, U13, U23
result.atom_is_aniso    # Boolean mask for anisotropic atoms
result.n_threads_used   # Number of threads actually used
```

The parallel implementation uses Numba's `prange` to parallelize over atoms, computing
Fc² and ALL derivatives in a single pass. This provides 2-8x speedup depending on the
dataset characteristics.

**Thread auto-tuning:**

The optimal thread count is automatically determined based on:
- Number of atoms (parallelization is over atoms)
- Available CPU cores (reserves 2 cores for system tasks)
- Environment variable `EDREF_NUM_THREADS` (optional override)

```python
# Check recommended thread count
n_threads = get_optimal_thread_count(n_atoms=14)  # Returns min(14, cpu_count-2)

# Check if parallel is beneficial for this workload
use_par = should_use_parallel(n_atoms=14, n_symops=16, n_refl=17404)  # True
use_par = should_use_parallel(n_atoms=21, n_symops=4, n_refl=9988)    # False
```

The heuristic uses a threshold of 100,000 operations per atom (n_symops × n_refl).
Below this threshold, the parallelization overhead exceeds the benefit.

**Performance comparison:**

| Dataset | Sequential | Parallel | Speedup |
|---------|------------|----------|---------|
| MFM-300 (14 atoms, 16 symops, 17k refl) | 246 ms | 29 ms | **8.4x** |
| Aspirin (21 atoms, 4 symops, 10k refl) | 63 ms | 9 ms | **7.0x** |

Note: First run includes JIT compilation overhead (~2-3 seconds). Subsequent runs
use cached compiled code (`cache=True` in Numba decorator).

### 8.6 normal_equations.py

**`build_design_matrix(hkl_data, atoms, ..., params, scale_k, return_Fc_squared=False, use_parallel=None, n_threads=None)`**

Build Jacobian matrix A[i,j] = ∂(k·Fc²[i])/∂p[j].

**Parameters:**
- `return_Fc_squared` (bool, default=False): If True, also return Fc² array
  to avoid redundant calculation. Provides ~35-40% speedup per cycle.
- `use_parallel` (bool | None, default=None): Controls parallel computation:
  - `None` (default): Auto-detect based on Numba availability and workload size
  - `True`: Force parallel computation (requires Numba)
  - `False`: Force sequential computation
- `n_threads` (int | None, default=None): Number of threads for parallel mode:
  - `None`: Auto-detect optimal count: `min(n_atoms, cpu_count - 2)`
  - Can be overridden via `EDREF_NUM_THREADS` environment variable

**Returns:**
- If `return_Fc_squared=False`: `NDArray` - Design matrix A
- If `return_Fc_squared=True`: `Tuple[NDArray, NDArray]` - (A, Fc_squared)

**Usage examples:**

```python
from edref.refinement.normal_equations import build_design_matrix

# Auto-detect (recommended) - uses parallel if beneficial
A, Fc_sq = build_design_matrix(
    hkl_data, atoms, sfac_elements, spacegroup, reciprocal_cell,
    wavelength, params, scale_k,
    return_Fc_squared=True
)

# Force parallel with specific thread count
A, Fc_sq = build_design_matrix(
    ..., use_parallel=True, n_threads=8
)

# Force sequential (useful for debugging or small structures)
A = build_design_matrix(..., use_parallel=False)
```

**Performance optimizations (v3.4.0+):**

1. **Return Fc² (`return_Fc_squared=True`):** Eliminates redundant call to
   `calculate_structure_factors_batch()` since design matrix computation
   already calculates Fc². Speedup: ~1.5x.

2. **Parallel computation (`use_parallel=True/None`):** When Numba is installed
   and workload is sufficient, uses parallel Numba implementation that computes
   Fc² and ALL derivatives in a single pass, parallelized over atoms.
   Speedup: **2-8x** depending on dataset.

3. **Auto-detection heuristic:** When `use_parallel=None`, automatically chooses
   the best path based on workload size (n_symops × n_refl per atom). Structures
   with high symmetry and many reflections benefit most from parallelization.

**Benchmark results:**

| Dataset | Sequential | Parallel | Speedup |
|---------|------------|----------|---------|
| MFM-300 (14 atoms, 16 symops, 17k refl) | 248 ms | 36 ms | **6.9x** |
| Aspirin (21 atoms, 4 symops, 10k refl) | 63 ms | 9 ms | **7.0x** |

**Auto-detection threshold:**

The parallel path is automatically chosen when:
- Numba is available, AND
- Work per atom ≥ 100,000 (n_symops × n_refl), AND
- At least 2 atoms

For small workloads, the parallelization overhead exceeds the benefit, so
sequential computation is preferred.

**`solve_normal_equations(A, W, Delta_y, damping=0.0) -> Tuple[NDArray, NDArray]`**

Solve (AᵀWA + λI)·Δp = AᵀW·Δy

Returns: (parameter_shifts, standard_uncertainties)

---

## 9. Analysis Module Reference

### 9.1 merging.py

**`merge_reflections(reflections, spacegroup, merge_friedel=True) -> List[MergedReflection]`**

Merge symmetry-equivalent reflections. **Automatically filters systematic absences.**

```python
from edref import merge_reflections

merged = merge_reflections(
    reflections=hkl.reflections,
    spacegroup=spacegroup,
    merge_friedel=True  # Merge Friedel pairs (for centrosymmetric)
)

print(f"Merged: {len(hkl.reflections)} → {len(merged)} reflections")
```

**SHELXL-compatible merging algorithm:**

The function uses formulas documented in SHELXL to match its behavior:

*Intensity:* Adaptive weighted mean
```
For positive-dominated data (≥50% positive): w = max(I, 0) / σ²
For weak/negative-dominated data:            w = 1 / σ²
I_merged = Σ(w × I) / Σw
```

*Sigma:* max(S1, S2) formula
```
S1 = Σ|I - ⟨I⟩| / [N × √(N-1)]   (esd from agreement of equivalents)
S2 = √(1 / Σ(1/σ²))              (esd from combined input sigmas)
sigma_merged = max(S1, S2)
```

The I/σ² weighting gives more weight to measurements with good signal-to-noise ratios. The max(S1, S2) formula ensures the merged sigma reflects both measurement precision and internal consistency.

**Systematic absence filtering:**

The function automatically excludes reflections that are systematically absent due to:
- Lattice centering (I, R, F, A, B, C)
- Screw axes (2₁ along a, b, or c)
- Glide planes (a-, b-, c-, n-glides perpendicular to each axis)

**`is_systematic_absence(h, k, l, spacegroup) -> bool`**

Check if a reflection is systematically absent.

```python
from edref.analysis.merging import is_systematic_absence

# P2₁/c: 2₁ screw along b, c-glide perpendicular to b
sg = SpaceGroup(latt=-1, symm_cards=["-X, Y+1/2, -Z+1/2"])

print(is_systematic_absence(0, 1, 0, sg))  # True (2₁ screw)
print(is_systematic_absence(0, 2, 0, sg))  # False
print(is_systematic_absence(1, 0, 1, sg))  # True (c-glide)
print(is_systematic_absence(1, 0, 2, sg))  # False
```

**`get_unique_hkl(h, k, l, spacegroup, merge_friedel=True) -> Tuple[int, int, int]`**

Get canonical representative for equivalence class.

**`apply_resolution_cutoff(reflections, cell, d_min=0.0, d_max=999.0) -> List[Reflection]`**

Apply SHEL-style resolution cutoff.

**`calculate_Rint(reflections, spacegroup, merge_friedel=True) -> float`**

Calculate internal agreement factor R_int.

---

**`select_rfree_reflections(hkl_data, d_values, fraction=0.05, n_shells=5, seed=None) -> Tuple[working_set, test_set, test_indices]`**

Select R-free test reflections using stratified sampling across resolution shells.

R-free is a cross-validation metric that detects overfitting by computing R1 on reflections
excluded from refinement. This function implements resolution-shell-stratified selection
to ensure uniform coverage across all resolution ranges.

**Parameters:**
- `hkl_data`: List of (h, k, l, Fo², sigma) tuples
- `d_values`: Array of d-spacings (same order as hkl_data)
- `fraction`: Fraction of reflections for test set (default: 0.05 = 5%)
- `n_shells`: Number of resolution shells (default: 5)
- `seed`: Random seed for reproducibility (default: None)

**Returns:**
- `working_set`: Reflections for refinement
- `test_set`: Reflections for R-free calculation
- `test_indices`: Original indices of test reflections

```python
from edref.analysis.merging import select_rfree_reflections
from edref.core.crystallography import calculate_d_spacing

# Calculate d-spacings
d_values = np.array([
    calculate_d_spacing(r.h, r.k, r.l, reciprocal_cell)
    for r in merged_reflections
])
hkl_data = [(r.h, r.k, r.l, r.intensity, r.sigma) for r in merged_reflections]

# Select R-free reflections (5% in 5 resolution shells)
working_set, test_set, test_indices = select_rfree_reflections(
    hkl_data,
    d_values,
    fraction=0.05,
    n_shells=5,
    seed=42  # For reproducibility
)

print(f"Working set: {len(working_set)} reflections")
print(f"Test set: {len(test_set)} reflections ({len(test_set)/len(hkl_data)*100:.1f}%)")

# Use working_set for refinement, then calculate R-free on test_set
```

**Shell binning:** Resolution shells are equidistant in 1/d² (reciprocal space), which
gives approximately equal volumes in reciprocal space and uniform statistical coverage.

**R-free correction handling:** When calculating R-free, the same corrections applied during
refinement are also applied to the test set:
- **Dynamical correction:** If using `--dynamical1`, `--dyn-power`, etc., the optimized correction
  parameters are applied to test reflections before R-free calculation
- **Extinction correction:** If using `--exti`, the refined EXTI parameter is applied to Fc²
  values for test reflections (critical for accurate R-free with extinction)

Without applying these corrections to the test set, R-free would be artificially inflated,
giving a false impression of overfitting.

### 9.2 robust_scaling.py

Provides outlier-resistant scale factor calculation methods. These are particularly important for electron diffraction data where dynamical scattering can cause severe outliers.

**Benchmark Results (electron diffraction data):**

| Dataset | Standard WLS | Biweight | Improvement |
|---------|--------------|----------|-------------|
| MFM-300 | 32.90% | **30.66%** | **-2.24%** |
| LTA1 | 27.64% | **27.55%** | -0.09% |
| LTA4 | 44.10% | 44.29% | +0.19% |

**`class ScalingResult`**

Data class returned by all scaling methods:

```python
@dataclass
class ScalingResult:
    k: float              # Optimal scale factor
    method: str           # Method name
    n_iterations: int     # Iterations to converge
    residual_std: float   # Residual standard deviation
    n_outliers: int       # Number of detected outliers
```

---

**`calculate_biweight_scale(Fo_sq, Fc_sq, sigma, c_biweight=6.0, ...) -> ScalingResult`**

Tukey biweight robust scaling. **Recommended for electron diffraction.**

Completely rejects extreme outliers (zero weight beyond threshold). Has ~50% breakdown point.

```python
from edref.analysis.robust_scaling import calculate_biweight_scale

result = calculate_biweight_scale(
    Fo_sq=Fo_sq_array,
    Fc_sq=Fc_sq_array,
    sigma=sigma_array,
    c_biweight=6.0  # Tukey threshold (optimal for ED data)
)

print(f"Scale k = {result.k:.4f}")
print(f"Outliers: {result.n_outliers}")
```

**Parameter tuning:** c=6.0 is optimal for MFM-300 (69 outliers). Lower c rejects more outliers.

---

**`calculate_huber_scale(Fo_sq, Fc_sq, sigma, k_huber=1.345, ...) -> ScalingResult`**

Huber M-estimator. Soft downweighting of outliers (not complete rejection).

```python
result = calculate_huber_scale(Fo_sq, Fc_sq, sigma, k_huber=1.345)
```

---

**`calculate_sigma_clip_scale(Fo_sq, Fc_sq, sigma, n_sigma=3.0, ...) -> ScalingResult`**

Iterative sigma-clipping (classic astronomy method). Rejects points > n_sigma from MAD-based estimate.

```python
result = calculate_sigma_clip_scale(Fo_sq, Fc_sq, sigma, n_sigma=5.0)
# 5σ optimal for MFM-300 (109 outliers rejected)
```

---

**`calculate_trimmed_scale(Fo_sq, Fc_sq, sigma, trim_fraction=0.15) -> ScalingResult`**

Trimmed mean. Excludes worst-fitting fraction before scaling. **Use with caution** - can remove too much data.

---

**`calculate_median_scale(Fo_sq, Fc_sq) -> ScalingResult`**

Median of Fo²/Fc² ratios. Maximum robustness (~50% breakdown) but lower efficiency.

---

**`calculate_winsorized_scale(Fo_sq, Fc_sq, sigma, winsor_fraction=0.10) -> ScalingResult`**

Winsorizes (caps) extreme values instead of removing them. Preserves sample size.

---

**`calculate_ransac_scale(Fo_sq, Fc_sq, sigma, n_samples=100, ...) -> ScalingResult`**

Random Sample Consensus. Fits on random subsets, identifies consensus. **Not recommended** - unstable for crystallographic data.

---

**`calculate_theil_sen_scale(Fo_sq, Fc_sq, max_pairs=10000) -> ScalingResult`**

Theil-Sen estimator. Median of pairwise slopes. 29% breakdown point.

---

**`compare_scaling_methods(Fo_sq, Fc_sq, sigma, verbose=True) -> dict`**

Compare all methods on the same data:

```python
from edref.analysis.robust_scaling import compare_scaling_methods

results = compare_scaling_methods(Fo_sq, Fc_sq, sigma)
# Prints comparison table and returns dict of ScalingResult objects
```

---

**Usage in Refinement:**

Pass `robust_scale_method` to `refine_structure()`:

```python
atoms, history = refine_structure(
    ...,
    robust_scale_method='biweight',  # Use biweight for scale calculation
)
```

---

### 9.3 visualization.py

Provides graphical analysis of refinement quality with three visualization types:
1. **Standard Refinement Plots** - Diagnostic plots after each refinement
2. **Comparative Plots** - SHELXL vs EDref comparison
3. **Scaling Methods Comparison** - All 10 robust scaling methods compared

**Data Classes:**

```python
@dataclass
class ReflectionAnalysis:
    """Per-reflection data recalculated for visualization."""
    hkl: np.ndarray           # (N, 3) Miller indices
    d_spacing: np.ndarray     # (N,) d-spacing in Angstroms
    Fo_sq: np.ndarray         # (N,) Observed intensities
    Fc_sq: np.ndarray         # (N,) Calculated intensities (absolute scale)
    sigma: np.ndarray         # (N,) Experimental uncertainties
    weights: np.ndarray       # (N,) SHELXL weights
    residuals: np.ndarray     # (N,) Fo²/k - Fc² (absolute scale)
    scale_k: float
    weight_a: float
    weight_b: float

@dataclass
class ScalingComparison:
    """Results from comparing all scaling methods."""
    methods: List[str]
    scale_factors: List[float]
    R1_values: List[float]
    n_outliers: List[int]
    results: Dict[str, ScalingResult]
```

---

**`calculate_reflection_analysis(atoms, hkl_data, sfac_elements, spacegroup, reciprocal_cell, wavelength, scale_k, weight_a, weight_b, sfac_coefficients=None) -> ReflectionAnalysis`**

Recalculate Fc², weights, and residuals for all reflections from refined atoms.

```python
from edref.analysis.visualization import calculate_reflection_analysis

analysis = calculate_reflection_analysis(
    atoms=refined_atoms,
    hkl_data=hkl_data,  # List of (h, k, l, Fo², sigma) tuples
    sfac_elements=ins.sfac_elements,
    spacegroup=spacegroup,
    reciprocal_cell=reciprocal_cell,
    wavelength=ins.wavelength,
    scale_k=result.scale_k,
    weight_a=result.wght_a,
    weight_b=result.wght_b,
    sfac_coefficients=ins.sfac_coefficients,
)
```

---

**`plot_refinement_summary(analysis, title="", history=None, save_path=None, show=True)`**

Generate 2x3 figure with standard diagnostic plots:
1. Fo² vs Fc² scatter (colored by residual)
2. Residuals vs Resolution (binned)
3. Residuals vs Intensity (binned)
4. Weight distribution histogram
5. Convergence plot (R1, wR2 vs cycle) - if history provided
6. GooF by resolution bin

```python
from edref.analysis.visualization import plot_refinement_summary

plot_refinement_summary(
    analysis,
    title="EDref Refinement Summary",
    history=result.history,  # Optional: for convergence plot
    save_path="refinement_summary.png",
    show=False,  # Set False for headless environments
)
```

---

**`plot_comparative_summary(shelxl_result, edref_result, shelxl_analysis, edref_analysis, title="", save_path=None, show=True)`**

Generate 2x3 comparison figure:
1. R-factor bar chart (R1, wR2, GooF side-by-side)
2. Fc² comparison scatter
3. Residual difference histogram
4. Resolution-dependent R1
5. Weight comparison scatter
6. Summary statistics table

---

**`run_scaling_comparison(atoms, hkl_data, sfac_elements, spacegroup, reciprocal_cell, wavelength, sfac_coefficients=None, verbose=True) -> ScalingComparison`**

Run all 10 scaling methods and compare outcomes.

```python
from edref.analysis.visualization import run_scaling_comparison, plot_scaling_comparison

comparison = run_scaling_comparison(
    atoms=refined_atoms,
    hkl_data=hkl_data,
    sfac_elements=ins.sfac_elements,
    spacegroup=spacegroup,
    reciprocal_cell=reciprocal_cell,
    wavelength=ins.wavelength,
    sfac_coefficients=ins.sfac_coefficients,
)

# Output (sorted by R1):
# Method          k         R1(obs) %  Outliers
# sigma_clip_3    0.6920    7.14       112
# biweight        0.6847    7.19       6
# ...
```

---

**`plot_scaling_comparison(comparison, title="", save_path=None, show=True)`**

Generate 2x2 figure comparing scaling methods:
1. Scale factor by method (bar chart)
2. R1 by method (sorted bar chart)
3. Outliers detected (bar chart)
4. Summary table of top methods

---

**CLI Integration:**

The visualization is integrated into the CLI with these options:

```bash
# Plots shown by default (use --no-plot to disable)
python3 -m edref refine structure.ins --edref

# Save plots to files
python3 -m edref refine structure.ins --edref --save-plots output_prefix
# Creates: output_prefix_refine.png

# Compare mode generates multiple plots
python3 -m edref refine structure.ins --compare --save-plots comparison
# Creates: comparison_shelxl.png, comparison_edref.png, comparison_compare.png

# Run scaling comparison
python3 -m edref refine structure.ins --edref --scaling-comparison --save-plots analysis
# Also creates: analysis_scaling.png
```

**Headless environments:** Set `MPLBACKEND=Agg` environment variable or use `--no-plot --save-plots path`.

---

## 10. Mathematical Background

### 10.1 Coordinate Systems

- **Fractional coordinates** (x, y, z): Position as fraction of unit cell edges, range [0, 1)
- **Cartesian coordinates**: Real-space position in Angstroms
- **Reciprocal space** (h, k, l): Miller indices for diffraction planes

### 10.2 Unit Cell and Reciprocal Cell

**Cell volume:**
```
V = a·b·c·√(1 - cos²α - cos²β - cos²γ + 2·cos(α)·cos(β)·cos(γ))
```

**Reciprocal cell parameters:**
```
a* = b·c·sin(α) / V
b* = a·c·sin(β) / V
c* = a·b·sin(γ) / V
```

**d-spacing (resolution):**
```
1/d² = h²a*² + k²b*² + l²c*² + 2hka*b*cos(γ*) + 2hla*c*cos(β*) + 2klb*c*cos(α*)
```

**Resolution parameter:**
```
s = sin(θ)/λ = 1/(2d)
```

### 10.3 Structure Factor Calculation

The structure factor F(hkl) is:
```
F(hkl) = Σⱼ Σₛ occⱼ · fⱼ(s) · Tⱼₛ · exp(2πi · φⱼₛ)
```

where:
- j sums over atoms in the asymmetric unit
- s sums over symmetry operations
- occⱼ is the site occupancy
- fⱼ(s) is the atomic scattering factor
- Tⱼₛ is the temperature factor
- φⱼₛ = h·xₛ + k·yₛ + l·zₛ is the phase

### 10.4 Atomic Scattering Factor

9-parameter Gaussian fit (Cromer-Mann):
```
f(s) = Σᵢ₌₁⁴ aᵢ·exp(-bᵢ·s²) + c + f' + if''

where s = sin(θ)/λ
```

For electron diffraction, custom coefficients are provided in the SFAC line.

### 10.5 Temperature Factors

**Isotropic (Debye-Waller):**
```
T_iso = exp(-8π² · U_iso · s²)
```

**Anisotropic:**
```
T_aniso = exp(-2π² · Σᵢⱼ Uᵢⱼ · hᵢ · hⱼ · aᵢ* · aⱼ*)

Expanded:
T = exp(-2π² · (U₁₁h²a*² + U₂₂k²b*² + U₃₃l²c*² + 2U₁₂hka*b* + 2U₁₃hla*c* + 2U₂₃klb*c*))
```

**Symmetry rotation of U tensor:**
```
U_rot = R · U · Rᵀ
```

### 10.6 Least-Squares Refinement

**Objective function (F² refinement):**
```
M = Σ w(hkl) · [Fo²(hkl) - k·Fc²(hkl)]²
```

**Scale factor parameterization (SHELXL-identical):**
```
k = FVAR²
Scaled Fc² = FVAR² · Fc²
Derivative: ∂(FVAR²·Fc²)/∂FVAR = 2·FVAR·Fc²
```

EDref refines FVAR = √k, not k directly.

**Normal equations:**
```
(AᵀWA + λI) · Δp = AᵀW · Δy

where:
  A = design matrix (Jacobian): Aᵢⱼ = ∂(k·Fc²ᵢ)/∂pⱼ
  W = diagonal weight matrix
  Δy = residual vector: Fo² - k·Fc²
  Δp = parameter shifts
  λ = Marquardt damping factor
```

**Convergence criterion:**
```
max|Δp/σ(p)| < 0.05
```

### 10.7 SHELXL Weighting Scheme

**CRITICAL:** SHELXL works on the **ABSOLUTE scale** internally.

**Convert to absolute scale:**
```
Fo²_abs = Fo² / k
σ_abs = σ / k
Fc² is already on absolute scale
```

**Weight formula (absolute scale):**
```
P = (max(Fo²_abs, 0) + 2·Fc²) / 3
w = 1 / [σ_abs² + (a·P)² + b·P]
```

**Parameter ranges:**
- a: typically 0.01 - 0.3
- b: typically 0 - 20

### 10.8 Derivatives for Design Matrix

**Scale factor (FVAR):**
```
∂(FVAR²·Fc²)/∂FVAR = 2·FVAR·Fc²
```

**Positions:**
```
Fc² = A² + B²  where F = A + iB

∂(Fc²)/∂x = 2·Re(F* · ∂F/∂x) = 2(A·∂A/∂x + B·∂B/∂x)

For atom j: ∂F/∂xⱼ = 2πi·h·termⱼ
           ∂F/∂yⱼ = 2πi·k·termⱼ
           ∂F/∂zⱼ = 2πi·l·termⱼ

where termⱼ = occ·f·T·exp(2πi·φⱼ)
```

**Isotropic U:**
```
T_iso = exp(-8π²·U·s²)
∂T/∂U = -8π²·s²·T

∂F/∂U = occ·f·(∂T/∂U)·exp(2πi·φ)
```

**Anisotropic U:**
```
Let: Q = h²a*², R = k²b*², S = l²c*²
     HK = hka*b*, HL = hla*c*, KL = klb*c*

∂T/∂U₁₁ = -2π²·Q·T
∂T/∂U₂₂ = -2π²·R·T
∂T/∂U₃₃ = -2π²·S·T
∂T/∂U₁₂ = -4π²·HK·T
∂T/∂U₁₃ = -4π²·HL·T
∂T/∂U₂₃ = -4π²·KL·T
```

### 10.9 Quality Statistics

**R1 (amplitude):**
```
R1 = Σ||Fo| - |Fc|| / Σ|Fo|
```
Scale cancels out in this ratio.

**R1(obs):** Only reflections with Fo² > 2·σ(Fo²)

**wR2 (intensity, absolute scale):**
```
wR2 = √[Σw(Fo²_abs - Fc²)² / Σw(Fo²_abs)²]
    = √[Σw(Fo²/k - Fc²)² / Σw(Fo²/k)²]
```

**GooF (absolute scale):**
```
GooF = √[Σw(Fo²_abs - Fc²)² / (N_obs - N_params)]
     = √[Σw(Fo²/k - Fc²)² / (N_obs - N_params)]
```

Target: GooF ≈ 1.0 (indicates proper weighting)

### 10.10 Reflection Merging

**Weighted mean:**
```
I_merged = Σ(wᵢ·Iᵢ) / Σ(wᵢ)  where wᵢ = 1/σᵢ²
```

**SHELXL merged sigma:**
```
σ_merged = max(S1, S2)

S1 = Σ|Iᵢ - I_merged| / (N·√(N-1))  [internal scatter]
S2 = √(1/Σwᵢ)                        [propagated error]
```

**R_int:**
```
R_int = Σ|Iᵢ - I_merged| / Σ|Iᵢ|
```

### 10.11 Systematic Absences

**Lattice centering:**
- I-centering: h+k+l = 2n (only even sums allowed)
- C-centering: h+k = 2n
- F-centering: h,k,l all even or all odd

**Screw axes:**
- 2₁ along b: (0,k,0) with k odd are absent
- 2₁ along c: (0,0,l) with l odd are absent

**Glide planes:**
- c-glide perpendicular to b: (h,0,l) with l odd are absent
- n-glide perpendicular to b: (h,0,l) with h+l odd are absent

---

## 11. SHELXL Compatibility

### 11.1 File Format Support

| Format | Read | Write |
|--------|------|-------|
| .ins | ✓ | ✗ |
| .res | ✓ | ✗ |
| .hkl (HKLF 4) | ✓ | ✗ |
| .fcf | ✓ | ✗ |
| .lst | ✗ | ✗ |

### 11.2 SHELXL Comparison Protocol

**CRITICAL:** When comparing EDref to SHELXL, both must use equivalent weight update protocols.

**SHELXL protocol (run in batches of 10 cycles):**

```bash
# Set L.S. 10 and WGHT 0.1 0 in .ins file initially

# Batch 1: Run 10 cycles
shelxl structure

# Extract CALCULATED optimal WGHT from .lst file
# See weight extraction priority below

# Copy .res to .ins
cp structure.res structure.ins
# Update WGHT line with the CALCULATED optimal values

# Batch 2-10: Repeat until 100 cycles total
```

**SHELXL weight output patterns in .lst file:**

SHELXL has two output patterns depending on whether calculated weights are "excessive":

**Pattern 1 - Excessive weights (e.g., ED data):**
```
Recommended weighting scheme:  WGHT    0.2000    0.0000
[Weight parameters refined to    0.3164        9.08  but appear excessive]
```
- "Recommended" = conservative fallback
- "Weight parameters refined to" = calculated optimal (USE THIS)

**Pattern 2 - Normal weights (e.g., good X-ray data):**
```
Recommended weighting scheme:  WGHT    0.0643    0.6730
```
- "Recommended" = calculated optimal (USE THIS - no bracketed line appears)

**Extraction priority:**
1. If "Weight parameters refined to X Y" present → use X, Y
2. If not present → use "Recommended weighting scheme" values
3. If neither found → keep current weights

**EDref equivalent:**
```python
refine_structure(
    ...,
    optimize_weights=True,        # DEFAULT
    weight_opt_frequency=10,      # DEFAULT
)
```

### 11.3 Reporting Requirements

Every refinement comparison must report START and END metrics:

```
START:  R1(obs)=X.XX%  wR2=X.XX%  GooF=X.XXX  k=X.XXXX  WGHT a=X.XXXX b=X.XXXX
END:    R1(obs)=X.XX%  wR2=X.XX%  GooF=X.XXX  k=X.XXXX  WGHT a=X.XXXX b=X.XXXX
Cycles: N (converged at max_shift/esd < 0.05)
```

**Scale factor reporting:**
- SHELXL: k = FVAR² (FVAR is √k, so square it)
- EDref: k is reported directly as `scale_k`

### 11.4 Critical Compatibility Notes

1. **Negative U values:** SHELXL sometimes produces negative (NPD) U values. Use them exactly as written:
   ```python
   # CORRECT: Use negative U directly
   T = np.exp(-8 * np.pi**2 * U_iso * s**2)

   # WRONG: Don't clamp or replace
   # U_iso = max(U_iso, 0.0)  # NO!
   ```

2. **Do NOT re-apply U constraints to SHELXL-refined structures:** SHELXL's refined U values already satisfy symmetry constraints. Re-applying can flip signs and break Fc calculation.

3. **SOF encoding:** `SOF = 10*m + occ` where m is FVAR index

4. **Anisotropic U order:** U11, U22, U33, U23, U13, U12

5. **AFIX not supported:** EDref does not implement SHELXL's AFIX riding hydrogen constraints:
   - SHELXL with AFIX: H positions calculated from parent (not refined)
   - EDref: H positions refined independently (+3 params per H)

### 11.5 Default Weighting Behavior

| Parameter | Default | Description |
|-----------|---------|-------------|
| `weighting_scheme` | 'shelxl' | SHELXL-style weights |
| `shelxl_a` | 0.1 | Starting value for a |
| `shelxl_b` | 0.0 | Starting value for b |
| `optimize_weights` | True | Optimize a, b every 10 cycles |
| `weight_opt_frequency` | 10 | Cycles between optimization |

---

## 12. Code Examples

### 12.1 Basic Structure Factor Calculation

```python
from edref import (
    InsFileReader, calculate_reciprocal_cell,
    SpaceGroup, calculate_structure_factor
)

# Load structure
ins = InsFileReader("aspirin.ins")
ins.read()

# Setup
recip = calculate_reciprocal_cell(ins.cell)
sg = SpaceGroup(ins.latt, ins.symm)

# Calculate F(1,0,0)
Fc = calculate_structure_factor(
    1, 0, 0, ins.atoms, ins.sfac_elements,
    sg, recip, ins.wavelength
)

print(f"F(1,0,0) = {Fc:.4f}")
print(f"|F(1,0,0)|² = {abs(Fc)**2:.2f}")
```

### 12.2 Full Refinement with Weight Optimization

```python
from edref import (
    InsFileReader, HklFileReader,
    SpaceGroup, calculate_reciprocal_cell,
    merge_reflections, refine_structure
)

# Load files
ins = InsFileReader("structure.ins")
ins.read()
hkl = HklFileReader("structure.hkl")
hkl.read()

# Setup
recip = calculate_reciprocal_cell(ins.cell)
sg = SpaceGroup(ins.latt, ins.symm)

# Merge reflections and convert format
merged = merge_reflections(hkl.reflections, sg)
data = [(m.h, m.k, m.l, m.intensity, m.sigma) for m in merged]

# Refine with SHELXL-style weights (defaults)
atoms, history = refine_structure(
    atoms=ins.atoms,
    hkl_data=data,
    sfac_elements=ins.sfac_elements,
    spacegroup=sg,
    reciprocal_cell=recip,
    wavelength=ins.wavelength,
    refine_positions=True,
    refine_scale=True,
    max_cycles=100,
    verbose=True
)

# Final statistics
final = history[-1]
print(f"\nFinal: R1={final.R1*100:.2f}%  wR2={final.wR2*100:.2f}%  GooF={final.GooF:.3f}")
print(f"Scale k={final.scale_k:.4f}  WGHT a={final.shelxl_a:.4f} b={final.shelxl_b:.2f}")
```

### 12.3 Anisotropic U Refinement

**Note:** The CLI auto-detects constraints. For Python API usage:

```python
from edref.refinement.parameters import detect_special_positions, detect_U_constraints

# Detect constraints for high-symmetry structures
# Note: No spacegroup parameter needed - detection is coordinate-based
constraints = detect_special_positions(ins.atoms)
u_constraints = detect_U_constraints(ins.atoms, constraints)

atoms, history = refine_structure(
    atoms=ins.atoms,
    hkl_data=data,
    sfac_elements=ins.sfac_elements,
    spacegroup=sg,
    reciprocal_cell=recip,
    wavelength=ins.wavelength,
    refine_positions=True,
    refine_Uiso=True,      # For isotropic atoms (e.g., H)
    refine_Uaniso=True,    # For anisotropic atoms
    refine_scale=True,
    constraints=constraints,
    u_constraints=u_constraints,
    max_cycles=100,
    verbose=True
)
```

### 12.4 Electron Diffraction with Custom Scattering Factors

```python
# Load structure with custom SFAC
ins = InsFileReader("electron_structure.ins")
ins.read()

# CRITICAL: Pass custom scattering coefficients for ED data!
atoms, history = refine_structure(
    atoms=ins.atoms,
    hkl_data=data,
    sfac_elements=ins.sfac_elements,
    spacegroup=sg,
    reciprocal_cell=recip,
    wavelength=ins.wavelength,
    sfac_coefficients=ins.sfac_coefficients,  # REQUIRED for ED!
    refine_positions=True,
    refine_scale=True,
    verbose=True
)
```

### 12.5 Robust Scaling for Outlier-Heavy Data

Electron diffraction data often has severe outliers due to dynamical scattering. Use robust scaling methods to improve R1.

**Option 1: Integrated into refinement (RECOMMENDED)**

```python
from edref import refine_structure

# Simply add robust_scale_method parameter
atoms, history = refine_structure(
    atoms=ins.atoms,
    hkl_data=hkl_data,
    sfac_elements=ins.sfac_elements,
    sfac_coefficients=ins.sfac_coefficients,  # Required for ED
    spacegroup=spacegroup,
    reciprocal_cell=reciprocal_cell,
    wavelength=ins.wavelength,
    refine_positions=True,
    refine_scale=True,
    robust_scale_method='biweight',  # <-- Outlier-resistant scaling
)

# MFM-300 result: R1 drops from 32.9% to 30.7% (-2.2%)
```

**Option 2: Manual comparison of methods**

```python
from edref.analysis.robust_scaling import compare_scaling_methods

# Compare all 10 scaling methods on your data
results = compare_scaling_methods(Fo_sq, Fc_sq, sigma, verbose=True)

# Output:
# Method          k       Iterations   Outliers   Std(resid)
# simple       4476.47            1          0       12.34
# biweight     3481.44            8         69        8.21  <-- best
# huber        2979.08           12         45        9.45
# ...
```

**Option 3: Specific method with custom parameters**

```python
from edref.analysis.robust_scaling import calculate_biweight_scale

result = calculate_biweight_scale(
    Fo_sq, Fc_sq, sigma,
    c_biweight=5.0,      # Lower c = more aggressive outlier rejection
    max_iterations=20,
    tolerance=1e-6
)
print(f"Scale k = {result.k:.4f}, outliers = {result.n_outliers}")
```

**Available robust scaling methods:**

| Method | Use Case |
|--------|----------|
| `'biweight'` | **Best for ED data** - Tukey biweight (c=6.0) |
| `'sigma_clip_3'` | Alternative - iterative 3σ clipping |
| `'huber'` | Moderate outliers - soft downweighting |
| `'median'` | Maximum robustness - median of ratios |
| `'trimmed_15'` | Remove worst 15% (use with caution) |

### 12.6 Using Position Constraints

**Note:** The CLI (`python -m edref refine`) auto-detects constraints. For Python API:

```python
from edref.refinement.parameters import (
    detect_special_positions,
    detect_U_constraints,
    PositionConstraint
)

# Automatic detection (no spacegroup needed - coordinate-based)
constraints = detect_special_positions(ins.atoms, tolerance=0.001)
u_constraints = detect_U_constraints(ins.atoms, constraints)

for c in constraints:
    atom = ins.atoms[c.atom_idx]
    print(f"{atom.label}: {c.constraint_type}, fixed={c.fixed_coords}")

# Manual constraint example (fix z coordinate)
manual_constraint = PositionConstraint(
    atom_idx=0,
    constraint_type='z_fixed',
    fixed_coords={'z'},
    coupled_coords=[],
    fixed_values={'z': 0.5}
)

# Refine with constraints
atoms, history = refine_structure(
    ...,
    constraints=constraints,
    u_constraints=u_constraints,
)
```

**Supported constraint types:** See Section 8.2 for the full list of auto-detected constraint types including diagonal mirrors (`x=y`), sum mirrors (`y+z=1`), 4-fold axes, and more.

### 12.7 Dynamical Correction for Electron Diffraction

Electron diffraction data suffers from dynamical scattering, where multiple scattering events redistribute intensity from strong to weak reflections. This causes:
- Negative ADPs (unphysical)
- High R-factors (30%+)
- High GooF (>>1)
- Non-convergent refinements

EDref provides empirical correction formulas to mitigate these effects.

#### 12.7.1 The Problem

Diagnostic analysis of uncorrected ED data shows systematic intensity errors:

| Reflection Type | Fo²/(k·Fc²) Ratio | Effect |
|-----------------|-------------------|--------|
| Weak (Q1) | ~25 | Strongly overestimated |
| Strong (Q4) | ~4 | Underestimated |

The effect is strongest in the inner shell (high d-spacing) and diminishes at high resolution.

#### 12.7.2 One-Parameter Correction (Recommended)

```python
import numpy as np
from edref import (
    InsFileReader, HklFileReader, SpaceGroup,
    calculate_reciprocal_cell, calculate_d_spacing,
    merge_reflections, refine_structure
)

def apply_dynamical_correction_1param(hkl_data, kappa=2.2):
    """
    Apply 1-parameter dynamical scattering correction.

    Formula:
        I_corr = I × (I / I_median)^(κ × d / d_max)
        σ_corr = σ × (1 + 10κ × d / d_max)

    Parameters
    ----------
    hkl_data : list of tuples
        Each tuple: (h, k, l, intensity, sigma, d_spacing)
    kappa : float
        Dynamical scattering strength (typically 1.0-3.0)
        - Thin crystals: κ = 1.0-1.5
        - Typical ED: κ = 2.0-2.2
        - Thick crystals: κ = 2.5-3.0

    Returns
    -------
    list of tuples
        Corrected data: (h, k, l, I_corr, sigma_corr)
    """
    intensities = np.array([row[3] for row in hkl_data])
    d_values = np.array([row[5] for row in hkl_data])

    I_median = np.median(intensities[intensities > 0])
    d_max = np.max(d_values)

    corrected = []
    for h, k, l, I, sigma, d in hkl_data:
        # Intensity correction: boost strong, reduce weak
        if I > 0:
            power = kappa * (d / d_max)
            I_corr = I * (I / I_median) ** power
        else:
            I_corr = 1.0

        # Sigma correction: inflate uncertainties in inner shell
        sigma_corr = sigma * (1 + 10 * kappa * d / d_max)

        corrected.append((h, k, l, max(I_corr, 1.0), max(sigma_corr, 1.0)))

    return corrected

# Usage example
ins = InsFileReader("structure.ins")
ins.read()
hkl = HklFileReader("structure.hkl")
hkl.read()

spacegroup = SpaceGroup(ins.latt, ins.symm)
reciprocal_cell = calculate_reciprocal_cell(ins.cell)
merged = merge_reflections(hkl.reflections, spacegroup)

# Prepare data with d-spacing
hkl_with_d = []
for r in merged:
    d = calculate_d_spacing(r.h, r.k, r.l, reciprocal_cell)
    hkl_with_d.append((r.h, r.k, r.l, r.intensity, r.sigma, d))

# Apply correction
hkl_corrected = apply_dynamical_correction_1param(hkl_with_d, kappa=2.2)

# Refine with corrected data
atoms, history = refine_structure(
    atoms=ins.atoms,
    hkl_data=hkl_corrected,
    sfac_elements=ins.sfac_elements,
    sfac_coefficients=ins.sfac_coefficients,
    spacegroup=spacegroup,
    reciprocal_cell=reciprocal_cell,
    wavelength=ins.wavelength,
    refine_positions=True,
    refine_scale=True,
    refine_Uaniso=True,
)
```

#### 12.7.3 Three-Parameter Correction (Best Results)

For optimal results, use the 3-parameter model with pedestal subtraction:

```python
def apply_dynamical_correction_3param(hkl_data, C=20000, alpha=1.6, beta=15):
    """
    Apply 3-parameter dynamical scattering correction.

    Formula:
        I_corr = (I - C) × (I / I_median)^(α × d / d_max)
        σ_corr = σ × (1 + β × d / d_max)

    Parameters
    ----------
    C : float
        Background pedestal to subtract (0-30000)
    alpha : float
        Power boost exponent (1.5-2.5)
    beta : float
        Sigma inflation factor (10-20)
    """
    intensities = np.array([row[3] for row in hkl_data])
    d_values = np.array([row[5] for row in hkl_data])

    I_median = np.median(intensities[intensities > 0])
    d_max = np.max(d_values)

    corrected = []
    for h, k, l, I, sigma, d in hkl_data:
        # Pedestal subtraction
        I_ped = I - C

        # Intensity correction
        if I_ped > 0:
            power = alpha * (d / d_max)
            I_corr = I_ped * (I_ped / I_median) ** power
        else:
            I_corr = 1.0

        # Sigma correction
        sigma_corr = sigma * (1 + beta * d / d_max)

        corrected.append((h, k, l, max(I_corr, 1.0), max(sigma_corr, 1.0)))

    return corrected

# Optimal parameters for MFM-300: C=20000, α=1.6, β=15
hkl_corrected = apply_dynamical_correction_3param(hkl_with_d, C=20000, alpha=1.6, beta=15)
```

#### 12.7.4 Command-Line Interface

EDref provides CLI flags for dynamical correction:

```bash
# Fixed 1-parameter correction (κ=2.2)
python3 -m edref refine structure.ins --edref --kappa 2.2 --aniso

# Fixed 3-parameter correction
python3 -m edref refine structure.ins --edref --dyn-params 20000 1.6 15 --aniso

# Automatic 1-parameter optimization (re-optimizes κ every 10 cycles)
python3 -m edref refine structure.ins --edref --dynamical1 --aniso

# Automatic 3-parameter optimization (re-optimizes C, α, β every 10 cycles)
python3 -m edref refine structure.ins --edref --dynamical3 --aniso

# With resolution cutoff
python3 -m edref refine structure.ins --edref --dynamical3 --resolution 1 0.5 --aniso
```

#### 12.7.5 wR2 Optimization Target (Default)

**IMPORTANT:** EDref uses wR2 (not R1) as the optimization target for dynamical parameter refinement.

**Why wR2?** The sigma inflation term changes which reflections count as "observed" (Fo² > 2σ). Using R1(obs) as the target allows the optimizer to "game" the metric by inflating sigmas so much that problematic reflections are excluded from the count.

| Optimization Target | Problem |
|---------------------|---------|
| R1(obs) | High β excludes reflections from n_obs, artificially lowering R1 |
| wR2 | Uses all reflections weighted by uncertainties, no gaming possible |

**Example of sigma gaming (MFM-300, β=15):**
- R1(obs) = 17.4% looks good, but only 1692/1894 reflections are "observed"
- R1(all) = 19.2% tells the true story

**With wR2-optimized parameters (β=5):**
- R1(obs) = 18.2%, with 1781/1894 reflections observed
- Honest improvement without excluding data

#### 12.7.6 Validated Results (MFM-300)

**Resolution 99-0.6 Å (1894 reflections):**

| Config | R1(obs) | n_obs | Params | Notes |
|--------|---------|-------|--------|-------|
| Baseline, opt weights | 27.25% | 1894 | 80 | Reference |
| κ=2.2 fixed, opt weights | **18.29%** | 1841 | 80 | Good improvement |
| 3-param fixed, opt weights | **17.36%** | 1692 | 80 | Best R1 but fewer obs |
| 3-param wR2-opt, opt weights | 19.29% | 1886 | 80 | Honest metric |

**Resolution 1-0.5 Å (high-resolution shell, 2786 reflections):**

| Config | R1(obs) | n_obs | Params | Notes |
|--------|---------|-------|--------|-------|
| Baseline, opt weights | 23.43% | 2786 | 80 | Reference |
| 3-param wR2-opt, opt weights | **15.39%** | 2770 | 80 | **Best result** |

**Key finding:** The wR2 optimizer found optimal parameters C=0, α=1.0, β=5.0 for the high-resolution shell - much gentler than the "standard" parameters. This suggests dynamical correction parameters should be **resolution-dependent**.

#### 12.7.7 Physical Interpretation

**Intensity correction** `(I/I_median)^(α × d/d_max)`:
- Strong reflections (I > I_median): Amplified to compensate for dynamical loss
- Weak reflections (I < I_median): Reduced to compensate for dynamical gain
- Resolution factor (d/d_max): Maximum effect in inner shell where dynamical scattering is strongest

**Sigma inflation** `σ × (1 + β × d/d_max)`:
- Inner shell: Larger inflation downweights unreliable data
- Outer shell: Minimal inflation preserves more kinematical data
- **Warning:** Excessive β (>15) can exclude too many reflections from n_obs

**Pedestal subtraction** (C):
- Removes constant background from incoherent scattering, detector noise, or sample holder
- Set C=0 for high-resolution shells where background is negligible

#### 12.7.8 Choosing Parameters

| Resolution Range | Recommended Approach |
|------------------|---------------------|
| Full (99-0 Å) | Use `--dynamical3` for automatic wR2 optimization |
| Inner shell (99-1 Å) | Fixed κ=2.2 or C=20000, α=1.6, β=15 may work |
| High-res (1-0.5 Å) | Use `--dynamical3` - optimizer finds gentler C=0, α=1.0, β=5 |

**Manual optimization approach:**
1. Start with `--dynamical3` (automatic wR2 optimization)
2. Check final parameters in output
3. If unstable, use fixed parameters with `--dyn-params C ALPHA BETA`
4. For best results, optimize separately for different resolution shells

#### 12.7.9 New Dynamical Correction Models (v3.2+)

The original 3-parameter model (C, α, β) has limitations: the pedestal C and sigma inflation β often converge to 0, leaving only α useful. New models improve on α-only by adding a shape parameter γ.

##### Power-Shape Model (RECOMMENDED)

```bash
python3 -m edref refine structure.ins --edref --dyn-power --aniso
```

**Formula:**
```
power = α × (d / d_max)^γ
I_corr = I × (I / I_median)^power
```

**Parameters:**
- **α** (0.1-5.0): Correction strength at d_max
- **γ** (0.5-3.0): Shape exponent
  - γ = 1.0: Linear (same as 1-param κ)
  - γ < 1.0: Faster onset at low d, plateaus toward high d (typical optimal: 0.2-0.7)
  - γ > 1.0: Slower onset, stronger effect confined to inner shell

**Why it works:** With γ ≈ 0.2-0.7, the correction drops off faster toward high resolution than linear, better matching the physical behavior where dynamical scattering diminishes gradually.

**Optimization note:** The α and γ ranges start at 0.1 and 0.5 (not 0.0) to prevent the optimizer from getting trapped at α=0. When α=0, any value of γ produces the same result (no correction), creating a "ridge" of local minima that can trap the optimizer.

##### Exponential-Decay Model

```bash
python3 -m edref refine structure.ins --edref --dyn-exp --aniso
```

**Formula:**
```
power = α × exp(-(d_max - d) / d_decay)
I_corr = I × (I / I_median)^power
```

**Parameters:**
- **α** (0.1-5.0): Maximum power at d_max
- **d_decay** (0.2-5.0 Å): Decay length scale

**Note:** Can be unstable with weight refinement; may find pathological minima. The α range starts at 0.1 (not 0.0) to prevent optimizer traps.

##### Absolute-Scale Model

```bash
python3 -m edref refine structure.ins --edref --dyn-abs --aniso
```

**Formula:**
```
power = α × (d / d_half)^γ / (1 + (d / d_half)^γ)
I_corr = I × (I / I_median)^power
```

**Parameters:**
- **α** (0.1-5.0): Asymptotic maximum power
- **d_half** (0.5-5.0 Å): D-spacing where power = α/2
- **γ** (0.5-4.0): Transition steepness

Uses absolute d-spacing scale rather than d/d_max ratio. The α range starts at 0.1 (not 0.0) to prevent optimizer traps.

##### Single-Parameter Variants

Optimize only one parameter of the 3-param model:
```bash
python3 -m edref refine structure.ins --edref --dyn3-alpha-only --aniso  # α only (C=0, β=0)
python3 -m edref refine structure.ins --edref --dyn3-C-only --aniso      # C only
python3 -m edref refine structure.ins --edref --dyn3-beta-only --aniso   # β only
```

##### Benchmark Results (MFM-300 at 1.5-0.5 Å)

| Model | CLI Option | R1(all) | wR2 | GooF | Best Parameters |
|-------|------------|---------|-----|------|-----------------|
| No correction | - | 27.39% | 62.33% | 7.072 | - |
| Alpha-only | `--dyn3-alpha-only` | 16.92% | 40.38% | 0.732 | α=1.22 |
| **Power-shape** | `--dyn-power` | **14.63%** | **36.06%** | **0.691** | **α=1.09, γ=0.28** |
| Exp-decay | `--dyn-exp` | DIVERGED | - | - | unstable |
| Absolute | `--dyn-abs` | 14.64% | 36.06% | 0.690 | α=2.67, d_half=3.66Å, γ=0.43 |

**Recommendation:** Use `--dyn-power` for best results. The power-shape model with optimized γ gives 2.3% R1 improvement over α-only.

---

### 12.8 Iterative Outlier Removal

The `--iterative` flag enables automatic iterative outlier removal. Each round: refine → analyze → identify worst outliers → omit → repeat. This is particularly useful for electron diffraction data with dynamical scattering outliers.

#### Basic Usage

```bash
# Run 10 rounds, omit 15 reflections per round with power-shape dynamical correction
python3 -m edref refine 2.ins --edref --dyn-power --iterative --iter-rounds 10 --iter-omit 15

# Stop after omitting up to 5% of reflections
python3 -m edref refine 2.ins --edref --iterative --iter-percent 5 --iter-omit 10

# With R-free cross-validation to monitor overfitting
python3 -m edref refine 2.ins --edref --dyn-power --iterative --rfree --iter-rounds 20
```

#### CLI Options

| Option | Default | Description |
|--------|---------|-------------|
| `--iterative` | - | Enable iterative outlier removal mode |
| `--iter-rounds N` | 20 | Maximum number of rounds |
| `--iter-omit N` | 5 | Reflections to omit per round |
| `--iter-percent P` | None | Stop after omitting P% of reflections |
| `--iter-threshold Z` | 5.0 | Only omit reflections with \|z_robust\| > Z. Use 0 to always take worst N. |
| `--iter-cycles N` | 50 | Cycles for first round |
| `--iter-cycles-subsequent N` | 20 | Cycles for subsequent rounds |
| `--iter-output DIR` | `<input>_iterative/` | Output directory |

#### Outlier Detection Algorithm

Outliers are identified using robust z-scores in log-space:

```
log_residual = log10(Fo²/k + 1) - log10(Fc² + 1)
z_robust = (log_residual - median) / (MAD / 0.6745)
```

Where MAD is the Median Absolute Deviation. Reflections with |z_robust| > threshold are candidates for omission. The N worst outliers above threshold are omitted each round.

**Special cases:**
- If `--iter-threshold 0`: Always omit the N worst-fitting reflections regardless of z-score
- If z_robust values are NaN (e.g., all residuals identical): Falls back to using raw log residuals for ranking

#### Output Structure

```
<input>_iterative/
├── round_001/
│   ├── refined.res           # Refined structure
│   ├── refined.fcf           # FCF file
│   ├── refinement.png        # 6-panel refinement summary
│   └── outliers.png          # 4-panel outlier analysis
├── round_002/
│   └── ...
├── final_report.png          # Parameter evolution plots
└── rounds_summary.csv        # Tabular data for all rounds
```

#### Final Report Contents

The `final_report.png` contains 6 panels:
1. **R1/wR2 vs Omitted** - Track R-factor improvement
2. **GooF vs Omitted** - Approach to ideal GooF=1.0
3. **Scale k vs Omitted** - Scale stability
4. **Dynamical params vs Omitted** - α, γ evolution (if applicable)
5. **R1 by Round** - Convergence plot
6. **Summary statistics** - Initial, best, and final values

#### Stopping Conditions

The iterative process stops when ANY condition is met:
1. Maximum rounds reached (`--iter-rounds`)
2. Maximum omit percentage reached (`--iter-percent`)
3. No outliers above threshold remain
4. R1 increasing for 3 consecutive rounds (divergence detection)

#### Programmatic Usage

```python
from edref.cli.runners import IterativeRunner

runner = IterativeRunner(
    ins_path="structure.ins",
    output_dir="my_iterative_output/",
    max_rounds=15,
    omit_per_round=10,
    max_omit_percent=5.0,
    outlier_threshold=5.0,
    cycles_first_round=50,
    cycles_subsequent=20,
    dynamical_mode="power_shape",
    convert_to_aniso=True,
    use_rfree=True,
)
round_results = runner.run(verbose=True)

# Access results
for result in round_results:
    print(f"Round {result.round_number}: R1={result.R1_obs*100:.2f}%, omitted={result.n_omitted_cumulative}")
```

#### Example Output

```
======================================================================
ITERATIVE OUTLIER REMOVAL REFINEMENT
======================================================================
Input: example_data/mfm300/2.ins
Output: 2_iterative
Max rounds: 10
Omit per round: 15
Outlier threshold: |z| > 5.0
Dynamical correction: power_shape

----------------------------------------------------------------------
ROUND 1
----------------------------------------------------------------------
EDref refinement: 50 cycles, weight update every 10, dynamical power-shape...

Round  1 | Omit=   0 | Refl= 3009 | R1=14.63% | wR2=36.06% | GooF=0.691 | α=1.09 γ=0.28

Found 156 reflections with |z| > 5.0
Top 15 outliers:
  (  0,  2,  6) z=+12.34
  (  1,  3,  5) z=-11.89
  ...
Omitting 15 reflections for next round

----------------------------------------------------------------------
ROUND 2
----------------------------------------------------------------------
...

======================================================================
ITERATIVE REFINEMENT COMPLETE
======================================================================
Rounds completed: 10
Total reflections omitted: 150
R1 improvement: 14.63% → 11.24%
Output directory: 2_iterative
```

---

## 13. Troubleshooting

### 13.1 Import Errors

**Problem:** `ModuleNotFoundError: No module named 'edref'`

**Solution:**
```bash
# Set PYTHONPATH
export PYTHONPATH=/path/to/EDref_v3/src:$PYTHONPATH

# Or install the package
pip install /path/to/EDref_v3
```

### 13.2 Refinement Diverges

**Symptoms:** R-factors increase, NaN values, or "singular matrix" errors

**Solutions:**
1. **Enable damping:**
   ```python
   refine_structure(..., damping=0.1)
   ```
2. **Stage the refinement:** First refine positions only, then add U parameters
3. **Check for special positions:** Atoms on mirrors/axes need constraints
4. **Don't pass initial_scale=1.0:** Let EDref auto-calculate

### 13.3 Matrix is Singular

**Cause:** Parameters are not independent (atoms on special positions)

**Solution:** The CLI auto-detects constraints. For Python API:
```python
from edref.refinement.parameters import detect_special_positions, detect_U_constraints

# Note: No spacegroup parameter needed
constraints = detect_special_positions(atoms)
u_constraints = detect_U_constraints(atoms, constraints)

refine_structure(..., constraints=constraints, u_constraints=u_constraints)
```

**Note:** Since 2026-01-19, the CLI automatically detects special position constraints. This fixed divergence issues with high-symmetry structures like LTA zeolite (Pm-3m).

### 13.4 Poor GooF (>> 1.0 or << 1.0)

**Solutions:**
1. Enable weight optimization: `optimize_weights=True` (default)
2. Adjust starting WGHT parameters
3. Check for systematic errors in data
4. For ED data: Use robust scaling methods

### 13.5 wR2 Much Higher Than SHELXL

**Cause:** Systematic absences not being filtered (fixed in 2026-01-19).

Reflections like (0,1,0) for P2₁/c have Fc²≈0 but Fo²≠0 (noise), causing massive residuals.

**Solution:** Use `merge_reflections()` which now filters absences automatically.

**Diagnostic:**
```python
# Check for reflections with small Fc² but large Fo²
for h, k, l, Fo_sq, sigma in hkl_data:
    Fc_sq = abs(calculate_structure_factor(h, k, l, ...))**2
    if Fc_sq < 1.0 and Fo_sq > 10:
        print(f"Potential absence: ({h},{k},{l}) Fc²={Fc_sq:.2f}, Fo²={Fo_sq:.1f}")
```

### 13.6 File Encoding Errors

**Problem:** `UnicodeDecodeError` when reading SHELXL files

**Solution:** Files are read with Latin-1 encoding automatically. Check file integrity.

### 13.7 Fc Values Don't Match SHELXL

**Checklist:**

1. **Check atom parsing:** Verify atoms with small negative U are parsed correctly:
   ```python
   for atom in ins.atoms:
       print(f"{atom.label}: is_iso={atom.is_isotropic()}, U11={atom.U11}")
   ```

2. **Don't re-apply U constraints to SHELXL structures:**
   ```python
   # CORRECT: Use atoms directly
   atoms = ins.atoms

   # WRONG for SHELXL-refined structures!
   # apply_U_constraints_to_atoms(ins.atoms, u_constraints)
   ```

3. **Pass custom SFAC for ED data:**
   ```python
   refine_structure(..., sfac_coefficients=ins.sfac_coefficients)
   ```

### 13.8 Resolution-Dependent Scale Factor Issues

**Problem:** 2-parameter scale (`refine_scale_B=True`) gives worse R1

**Explanation:** B and U are mathematically correlated:
- B contributes: `exp(-2B × s²)`
- U_iso contributes: `exp(-8π² × U × s²)`

When both refined simultaneously, the refinement oscillates.

**Solution:** Use 1-parameter scale (default) when refining U:
```python
refine_structure(
    ...,
    refine_scale=True,
    refine_scale_B=False,  # Don't use 2-param scale with U refinement
    refine_Uiso=True,
)
```

### 13.9 Scale Factor ~60% Wrong

**Cause:** Using simple `w=1/σ²` weights instead of SHELXL weights (fixed 2026-01-19).

**Solution:** Use default `weighting_scheme='shelxl'`.

### 13.10 High R1 on Electron Diffraction Data with Outliers

**Symptoms:** R1 is 2-5% higher than expected, even with correct structure.

**Cause:** Dynamical scattering creates severe outliers that dominate the scale factor calculation. Standard weighted least-squares gives too much weight to strong low-sigma outliers.

**Solution:** Use robust scaling methods:

```python
atoms, history = refine_structure(
    ...,
    robust_scale_method='biweight',  # Recommended for ED data
)
```

**Tested improvements:**
- MFM-300: R1 drops from 32.9% → 30.7% (2.2% improvement)
- LTA1: R1 drops from 27.6% → 27.5% (0.1% improvement)

**Alternative methods:** `'sigma_clip_3'`, `'huber'`, `'median'`

See Section 9.2 for details on all available robust scaling methods.

---

## 14. Validated Datasets

### 14.1 Aspirin (Primary Validation)

**Location:** `example_data_do_not_modify/aspirin/`

| Property | Value |
|----------|-------|
| Space group | P2₁/c (centrosymmetric) |
| LATT | 1 (with inversion in SYMM) |
| Atoms | 21 (13 non-H, 8 H) |
| Reflections | 2148 merged (from 9988 unmerged) |

**Statistics-only comparison (from SHELXL .res, no H atoms):**

| Metric | SHELXL | EDref | Δ |
|--------|--------|-------|---|
| R1(obs) | 8.48% | 8.32% | -0.16% |
| wR2 | 24.75% | 24.76% | **+0.01%** ✓ |
| GooF | 1.118 | 0.945 | -0.173 |
| Scale k | 0.686 | 0.686 | +0.000 |

**Anisotropic perturbation test (±0.02 frac, 100 cycles):**

| Metric | SHELXL | EDref | Δ |
|--------|--------|-------|---|
| R1(obs) | 6.62% | 5.99% | -0.63% |
| wR2 | 16.99% | 15.95% | -1.04% |
| GooF | 1.20 | 1.07 | -0.13 |
| Scale k | 0.572 | 0.704 | +0.13* |

*Scale difference due to SHELXL refining occupancies (not implemented in EDref)

**Performance:**
- EDref: ~99ms/cycle (anisotropic, 118 params)
- SHELXL: ~6ms/cycle
- EDref converges in fewer cycles (37 vs 100)

### 14.2 MFM-300 (Electron Diffraction)

**Location:** `example_data_do_not_modify/mfm300/`

| Property | Value |
|----------|-------|
| Space group | I4₁22 (high symmetry) |
| LATT | -2 (I-centered, centrosymmetric) |
| Atoms | 13 |
| Reflections | ~17000 raw, ~6000+ merged |
| Data type | Electron diffraction |

**Comprehensive test (99-0.6 Å, 1894 reflections):**

| Config | R1(obs) | wR2 | GooF | k | Params | EXTI |
|--------|---------|-----|------|---|--------|------|
| **Isotropic** | | | | | | |
| EDref+EXTI | 22.64% | 49.50% | 0.979 | 2579 | 39 | 12236 |
| EDref (no EXTI) | 28.09% | 60.17% | 0.987 | 1465 | 39 | - |
| SHELXL (no EXTI) | 45.25%* | 64.01% | 0.975 | 1173 | 39 | - |
| **Anisotropic** | | | | | | |
| EDref+EXTI | 21.28% | 47.27% | 1.043 | 2947 | 69 | 15972 |
| EDref (no EXTI) | 27.43% | 59.27% | 0.996 | 1467 | 69 | - |
| SHELXL (no EXTI) | 28.50% | 60.14% | 1.177 | 1305 | 79 | - |

*SHELXL isotropic diverges at 0.6 Å resolution (R1 increases from 34% → 45%)

**Key findings:**
- **EXTI gives major improvement:** R1 drops from ~28% to ~21-22%
- **EDref without EXTI matches SHELXL:** 27.43% vs 28.50% (within 1%)
- **Anisotropic EDref uses fewer params:** 69 vs SHELXL's 79 (better constraint handling)

**B-factor analysis (known limitation):**

Both EDref and SHELXL produce **negative B factors** for most atoms after refinement:

| Atom | Start | EDref+EXTI | EDref | SHELXL |
|------|-------|------------|-------|--------|
| V1 | 0.08 | -0.63 | -1.23 | -1.51 |
| O1 | 0.24 | 0.20 | -0.17 | 0.33 |
| C3 | 0.03 | -0.73 | -1.20 | -0.08 |
| O4 | 2.32 | 5.54 | 4.17 | 3.99 |

This is a **fundamental limitation of kinematic refinement** on electron diffraction data - the model tries to compensate for dynamical scattering effects by pushing ADPs negative. EXTI partially compensates but doesn't fully solve this issue.

**Notes:**
- High R1 (~20-30%) is inherent to ED data quality and kinematic approximation
- Requires custom SFAC coefficients for electron scattering
- Requires special position constraints (7 position + 7 U constraints)
- Negative B factors indicate dynamical scattering effects not captured by kinematic model

### 14.3 LTA Zeolite (High-Symmetry Electron Diffraction)

**Location:** `example_data_do_not_modify/LTA/`

| Property | Value |
|----------|-------|
| Space group | Pm-3m (cubic, #221) |
| LATT | 1 (P-centered, centrosymmetric) |
| Atoms | 4 (1 Si, 3 O) |
| Reflections | ~872 merged (from ~14,000 raw) |
| Data type | Electron diffraction |
| Symmetry | Very high (requires special position constraints) |

**Detected special positions:**

| Atom | Coordinates | Constraint Type | Position | U Params |
|------|-------------|-----------------|----------|----------|
| SI01 | (0.87, 0.68, 0.50) | z_fixed | 2 | 4 |
| O002 | (1.00, 0.72, 0.50) | 2fold_xz_fixed | 1 | 3 |
| O003 | (0.80, 0.80, 0.50) | mirror_diagonal_xy_z_fixed | 1 | 3 |
| O004 | (0.84, 0.61, 0.39) | mirror_yz_sum | 2 | 4 |

**Total parameters:** 6 position + 14 U + 1 scale = **21** (matches SHELXL exactly)

**LTA1 Results (100 cycles):**

| Metric | SHELXL | EDref | Δ |
|--------|--------|-------|---|
| R1(obs) | 27.82% | 22.57% | -5.25% |
| wR2 | 62.86% | 50.44% | -12.42% |
| GooF | 1.105 | 0.900 | -0.205 |
| Parameters | 21 | **21** | 0 ✓ |

**LTA4 Results (100 cycles):**

| Metric | SHELXL | EDref | Δ |
|--------|--------|-------|---|
| R1(obs) | 35.47% | 37.35% | +1.88% |
| wR2 | 71.40% | 67.65% | -3.75% |
| GooF | 1.148 | 1.047 | -0.101 |
| Parameters | 21 | **21** | 0 ✓ |

**Key validation:** Without constraints, EDref would refine 37 parameters and completely diverge (LTA4 showed R1 ~ 30 million %). The automatic constraint detection now ensures parameter count matches SHELXL.

---

## 15. Resolved Issues History

### 15.1 MFM-300 Scale Factor Divergence (Resolved 2026-01-19)

**Problem:** EDref scale factor diverged from SHELXL, R1 was ~10% higher.

**Root causes and fixes:**

1. **Custom SFAC coefficients not used**
   - EDref was using X-ray scattering factors instead of electron
   - Fix: Pass `sfac_coefficients=ins.sfac_coefficients`

2. **Initial scale defaulted to 1.0**
   - With k=1 instead of k~2600, cycle 1 had massive errors
   - Fix: Auto-calculate initial scale from data

3. **U_iso minimum too restrictive**
   - EDref clamped U_iso to 0.001; SHELXL allows -0.001
   - Fix: Changed minimum to -0.001

### 15.2 Absolute Scale Convention (Fixed 2026-01-19)

**Problem:** GooF was ~3× too high, wR2 was ~13% too high.

**Cause:** Mixed measurement-scale Fo² with scaled Fc² in P formula.

**Fix:** All weights, residuals, and statistics now use absolute scale:
```python
Fo_sq_abs = Fo_sq / scale_k
sigma_abs = sigma / scale_k
P = (max(Fo_sq_abs, 0) + 2 * Fc_sq) / 3
```

### 15.3 Scale Factor Calculation (Fixed 2026-01-19)

**Problem:** Simple `w=1/σ²` weights gave 60% error in scale factor.

**Cause:** Strong reflections dominated due to Fc⁴ term.

**Fix:** Use SHELXL-style iterative weights on absolute scale:
```
Before: k = 992, error = -60%
After:  k = 2503, error = +1%
```

### 15.4 Riding Hydrogen Detection (Fixed 2026-01-19)

**Problem:** Small negative U values (-0.01) incorrectly treated as riding H.

**Fix:** Changed threshold from `U < 0` to `U < -0.5`:
- Large negative (e.g., -1.2): Riding H indicator
- Small negative (e.g., -0.01): Legitimate refined value

### 15.5 Systematic Absence Filtering (Fixed 2026-01-19)

**Problem:** wR2 was ~2× too high for P2₁/c structures.

**Cause:** Systematic absences (Fc²≈0) with noise in Fo² dominated residuals.

**Fix:** `merge_reflections()` now automatically filters:
- Screw axis absences (2₁ along a, b, c)
- Glide plane absences (a-, b-, c-, n-glides)

### 15.6 Weight Optimization Algorithm (Fixed 2026-01-19)

**Problem:** Weight optimization targeted flat GooF rather than GooF=1.0.

**Old algorithm:**
```python
score = cv(bin_goofs) + 0.5 * |log(mean_goof)|  # Wrong
```

**New algorithm (SHELXL/CAPOW):**
```python
score = Σ(GooF_bin - 1.0)²  # Correct
```

### 15.7 Special Position Constraints Not Auto-Detected (Fixed 2026-01-19)

**Problem:** High-symmetry structures (e.g., LTA zeolite in Pm-3m) diverged or gave wrong parameter counts.

**Symptoms:**
- EDref refining 37 parameters vs SHELXL's 21 for LTA
- LTA4 completely diverged (R1 = 30 million %)
- Extremely high max_shift/esd values (millions)

**Root cause:** Constraint detection functions existed but were never called automatically.

**Fixes implemented:**

1. **Improved `detect_special_position()`** - Now detects:
   - Diagonal mirrors with fixed z (`x=y, z=0.5`)
   - Sum-to-one mirrors (`y+z=1`, `x+z=1`, `x+y=1`)
   - Face-edge intersections (`x=0, z=0.5`)

2. **Improved U constraint mapping:**
   - `2fold_xz_fixed`: U12=U13=U23=0 (was missing U13)
   - `z_fixed`: U13=U23=0 (was returning None)
   - New types: `mirror_diagonal_xy_z_fixed`, `mirror_yz_sum`, etc.

3. **CLI auto-detection** - `EdrefRunner` now automatically:
   - Calls `detect_special_positions()`
   - Calls `detect_U_constraints()`
   - Passes constraints to `refine_structure()`

4. **Sum constraint handling** - `apply_constraints_to_atoms()` now handles `z = 1 - y` type relationships correctly.

**Result:** LTA parameter count matches SHELXL exactly (21 vs 21), and refinement no longer diverges.

---

## 16. Appendices

### Appendix A: Quick Reference Card

**Most Used Imports:**
```python
from edref import (
    # I/O
    InsFileReader, HklFileReader, FcfFileReader,

    # Data types
    UnitCell, Atom, Reflection, SpaceGroup,

    # Core functions
    calculate_reciprocal_cell, calculate_structure_factor,
    merge_reflections, refine_structure,

    # Statistics
    calculate_R1, calculate_wR2, calculate_GooF,
)
```

**Minimal Workflow:**
```python
ins = InsFileReader("file.ins"); ins.read()
hkl = HklFileReader("file.hkl"); hkl.read()
recip = calculate_reciprocal_cell(ins.cell)
sg = SpaceGroup(ins.latt, ins.symm)
merged = merge_reflections(hkl.reflections, sg)
data = [(m.h, m.k, m.l, m.intensity, m.sigma) for m in merged]
atoms, history = refine_structure(
    ins.atoms, data, ins.sfac_elements, sg, recip, ins.wavelength,
    refine_positions=True, refine_scale=True
)
print(f"R1 = {history[-1].R1*100:.2f}%")
```

**Key Formulas:**

| Quantity | Formula |
|----------|---------|
| s (resolution) | sin(θ)/λ = 1/(2d) |
| T_iso | exp(-8π²Us²) |
| k (scale) | FVAR² |
| ∂(k·Fc²)/∂FVAR | 2·FVAR·Fc² |
| P (weight) | (Fo²/k + 2Fc²)/3 |
| w (SHELXL) | 1/[σ²/k² + (aP)² + bP] |
| R1 | Σ\|\|Fo\|-\|Fc\|\| / Σ\|Fo\| |
| wR2 | √[Σw(Fo²/k-Fc²)² / Σw(Fo²/k)²] |
| GooF | √[Σw(Fo²/k-Fc²)² / (N-P)] |

### Appendix B: Common SHELXL Instructions

| Instruction | Purpose |
|-------------|---------|
| L.S. n | Run n refinement cycles |
| WGHT a b | Set weight parameters |
| FVAR osf | Set scale factor √k |
| ANIS | Make atoms anisotropic |
| DFIX d σ A B | Restrain distance A-B |
| SIMU | Similar U restraint |
| HFIX mn C | Generate H on atom C |

### Appendix C: External References

**SHELXL Documentation:** `/home/agent/claude/shelxlbackground/`
- `SHELXL_Complete_Reference.md`
- `reference_materials/SHELXL_Formulas_and_Equations.md`
- `reference_materials/SHELXL_Instruction_Reference.md`
- `reference_materials/SHELXL_File_Formats.md`

**SHELXL Binary:** `/home/agent/claude/wilson/benchmark/shelxl`

Usage: `shelxl <basename>` reads `<basename>.ins` and `<basename>.hkl`

---

## Changelog

**2026-01-22 (v3.4.0):**
- PERFORMANCE: Parallel Numba implementation with multi-threading
  - New `calculate_all_derivatives_parallel()` function computes Fc² and ALL derivatives
    in a single pass, parallelized over atoms using Numba `prange`
  - **7-8x speedup** over sequential batch implementation
  - Benchmarked: MFM-300 248→36ms (6.9x), Aspirin 63→9ms (7.0x)
  - Auto-detection heuristic chooses best path based on workload size
  - Thread auto-tuning: `min(n_atoms, cpu_count - 2)`
  - Environment variable `EDREF_NUM_THREADS` for manual override
  - New parameters in `build_design_matrix()`:
    - `use_parallel` (bool | None): Control parallel vs sequential path
    - `n_threads` (int | None): Manual thread count override
  - New exports from `derivatives.py`:
    - `NUMBA_AVAILABLE` - Check if Numba is installed
    - `FusedDerivativesResult` - Dataclass with all computed values
    - `get_optimal_thread_count()` - Thread auto-tuning helper
    - `should_use_parallel()` - Workload heuristic helper
    - `calculate_all_derivatives_parallel()` - Main parallel function
  - Results verified to machine precision (max diff < 10⁻⁹)
  - First run has JIT compilation overhead (~2-3s), subsequent runs use cache

**2026-01-22 (v3.3.1):**
- PERFORMANCE: Numba JIT acceleration for fused derivative computation
  - Compiles inner symmetry loops to native code when Numba is installed
  - Additional **1.6-2.2x speedup** on top of fused computation
  - Benchmarked: MFM-300 isotropic 91→57ms (1.59x), anisotropic 245→110ms (2.22x)
  - Falls back gracefully to pure NumPy when Numba is unavailable
  - Results verified to machine precision (10⁻¹¹ or better)
  - Install Numba with: `pip install numba`
  - Superseded by v3.4.0 parallel implementation

**2026-01-22 (v3.3.0):**
- PERFORMANCE: Fused Fc² + derivative computation (15-50% additional speedup)
  - `calculate_fused_derivatives_batch()` computes Fc² and all derivatives in one pass
  - Eliminates ~115 MB intermediate array storage by accumulating on-the-fly
  - `build_design_matrix()` now uses fused computation by default (`use_fused=True`)
  - Benchmarked additional speedup: **1.12x Aspirin, 1.49x MFM-300**
  - Design matrix matches to machine precision (10⁻¹² or better)
  - See Section 8.5 (derivatives.py) and 8.6 (normal_equations.py) for API details

- PERFORMANCE: Eliminated redundant structure factor calculation in refinement loop
  - `build_design_matrix()` now accepts `return_Fc_squared=True` parameter
  - Returns tuple `(A, Fc_squared)` to avoid calling `calculate_structure_factors_batch()` separately
  - Benchmarked speedup: **1.53x for Aspirin (35%), 1.63x for MFM-300 (39%)**
  - Fc² values are numerically identical (verified to 10⁻¹² precision)

- **Combined optimization impact:**
  - Aspirin (21 atoms, mixed iso/aniso): ~1.7x total speedup
  - MFM-300 (13 atoms, isotropic): ~2.4x total speedup
  - MFM-300 (13 atoms, anisotropic): ~2.0x total speedup

**2026-01-21:**
- Added dynamical correction formulas for electron diffraction data (Section 12.7)
- 1-parameter model: `I_corr = I × (I/I_median)^(κ × d/d_max)`
- 3-parameter model: adds pedestal subtraction and separate sigma inflation
- Validated on MFM-300: R1 reduced from 32.4% to 17.8%, all ADPs positive
- Created comparison report: `MFM300_Correction_Comparison.pdf`

**2026-01-19:**
- Validated against Olex2/SHELXL on full aspirin (21 atoms with H)
- EDref R1=6.02%, wR2=15.25%, GooF=1.029 vs Olex2/SHELXL R1~6.1%, wR2~15.4%
- Documented AFIX limitation (EDref refines H positions independently)
- CRITICAL FIX: Systematic absence filtering for screw axes and glide planes
- CRITICAL FIX: Absolute scale convention for weighting and statistics
- CRITICAL FIX: Scale factor calculation with SHELXL-style iterative weights
- CRITICAL FIX: Weight optimization algorithm (SHELXL/CAPOW)
- FIX: SpaceGroup LATT sign convention
- FIX: Riding hydrogen detection threshold (U < -0.5)
- FIX: Initial scale auto-calculation
- FIX: U_iso minimum changed to -0.001

**Previous:**
- Anisotropic refinement validated
- FVAR parameterization (k = FVAR²)
- Performance optimization: ~99ms/cycle anisotropic

---

*EDref v3.4.0 - Manual last updated: 2026-01-22 (Parallel multi-threaded computation)*
