# EDref v3 - Development Guide

Python implementation of SHELXL-style least-squares refinement on F². **Read this before writing code.**

**ALWAYS check `manual.md` for:** API signatures, mathematical derivations, code examples, troubleshooting details.

---

## Table of Contents

1. [Critical Rules](#1-critical-rules) - Prevent common mistakes
2. [When to Consult manual.md](#2-when-to-consult-manualmd) - Reference guide
3. [Project Overview](#3-project-overview) - Package structure, test data
4. [Running EDref](#4-running-edref) - CLI and Python API
5. [Key Algorithms](#5-key-algorithms) - How refinement works
6. [File Formats](#6-file-formats) - SHELXL file specs
7. [Troubleshooting](#7-troubleshooting) - Common issues
8. [Validated Results](#8-validated-results) - Reference benchmarks
9. [External References](#9-external-references) - SHELXL docs, literature
10. [Releasing](#10-releasing) - PyPI publishing

---

## 1. CRITICAL RULES

These prevent incorrect results and divergent refinements.

### 1.0 Update Documentation When Changing Code

**⚠️ CRITICAL - ALWAYS UPDATE DOCS AFTER CODE CHANGES ⚠️**

When you modify EDref code (new features, CLI options, API changes), you MUST:
1. Update `manual.md` with new function signatures, parameters, examples
2. Update `CLAUDE.md` if it affects development workflow or critical rules
3. Add appropriate docstrings to new functions
4. Update `--help` text for new CLI options

**DO NOT** push code changes without corresponding documentation updates.

### 1.1 Always Check manual.md First

**Before debugging or implementing new features:**
1. Search manual.md for relevant keywords, function names, or error types
2. Check Section 13 (Troubleshooting) for known issues
3. Check Section 12 (Code Examples) for working patterns

### 1.2 SHELXL Comparison Protocol

SHELXL does NOT update weights during `L.S. N` cycles. For valid comparisons:

```bash
# Run SHELXL in batches of 10 cycles
# Extract CALCULATED weights from .lst: "Weight parameters refined to X.XXXX Y.YY"
# Do NOT use "Recommended weighting scheme" (conservative fallback)
# Copy .res to .ins, update WGHT, repeat
```

**EDref default:** `optimize_weights=True`, `weight_opt_frequency=10`

See manual.md Section 11.2 for full protocol.

### 1.3 Always Report Complete Metrics

# ╔══════════════════════════════════════════════════════════════════════════════╗
# ║  STOP! READ THIS BEFORE CREATING ANY COMPARISON TABLE OR REPORTING RESULTS  ║
# ╚══════════════════════════════════════════════════════════════════════════════╝

**THIS IS THE MOST VIOLATED RULE. DO NOT SKIP ANY METRICS. EVER.**

When you show refinement results, you MUST include ALL of the following.
If you create a table missing ANY of these, you have failed.

## MANDATORY METRICS FOR EVERY SINGLE REFINEMENT RESULT:

```
R1(obs)=XX.XX% | R1(all)=XX.XX% | wR2=XX.XX% | GooF=X.XXX | k=XXXX.XX | WGHT X.XXXX X.XX | Refl=N (M obs) | Params=N | Cycles=N
```

Plus if applicable:
- R-free: `R-free=XX.XX% (N)`
- EXTI: `EXTI=X.X`
- Dynamical: `α=X.XX | γ=X.XX` (or κ, C, β, d_half, d_decay as appropriate)

Plus in table header:
- Dataset name
- Resolution range (e.g., "2.0-0.5Å")
- Refinement type (iso/aniso)

## ❌ BAD - NEVER DO THIS:

```
| Model | α | R1 | R-free |
|-------|------|------|--------|
| wR2 target | 1.52 | 20.34% | 16.92% |
| Residual target | 1.67 | 20.45% | 22.09% |
```

THIS TABLE IS MISSING: wR2, GooF, k, WGHT, Refl, Params, Cycles, resolution, dataset, γ

## ✓ GOOD - ALWAYS DO THIS:

**MFM-300, 2.0-0.5Å, isotropic, 50 cycles, alpha-only dynamical correction:**

| Target | α | R1(obs) | R1(all) | wR2 | GooF | k | WGHT | Refl | Params | R-free (n) |
|--------|------|---------|---------|------|------|------|------|------|--------|------------|
| wR2 | 1.52 | 20.34% | 20.34% | 48.95% | 0.794 | 24081.7 | 0.3158 2.11 | 3009 (3009 obs) | 39 | 16.92% (159) |
| Residual | 1.67 | 20.45% | 20.45% | 49.30% | 0.737 | 28847.0 | 0.3158 2.11 | 3009 (3009 obs) | 39 | 22.09% (159) |

## CHECKLIST BEFORE SHOWING ANY TABLE:

- [ ] R1(obs) present?
- [ ] R1(all) present?
- [ ] wR2 present?
- [ ] GooF present?
- [ ] Scale k present?
- [ ] WGHT a b present?
- [ ] Reflection count present?
- [ ] Parameter count present?
- [ ] Cycle count present?
- [ ] Resolution range in header?
- [ ] Dataset name in header?
- [ ] Iso/aniso stated?
- [ ] R-free present (if calculated)?
- [ ] Dynamical params present (if used)?
- [ ] EXTI present (if refined)?

**If ANY checkbox is unchecked, DO NOT SHOW THE TABLE.**

The user cannot interpret results without complete information. A table missing wR2 and GooF is USELESS. A table missing WGHT is USELESS. Partial information is worse than no information.

### 1.3.1 Always Provide Output Files and Plots

**When running refinements, ALWAYS:**
1. Use `--save-plots PATH` to save visualization plots (refinement summary + outlier analysis)
2. List all output files generated (`.res`, `.fcf`, `.png` files)
3. Show the plots inline in the response
4. Specify the output directory used

**Example CLI with all outputs:**
```bash
mkdir -p output_dir && PYTHONPATH=src:$PYTHONPATH python3 -m edref refine input.ins \
    --edref --dyn-power --aniso --cycles 100 \
    --output output_dir/refined \
    --save-plots output_dir/refined \
    --no-plot  # Save without displaying (for batch runs)
```

**Output files to report:**
- `refined.res` - Refined structure (SHELXL format)
- `refined.fcf` - Structure factors (LIST 6 format)
- `refined_refine.png` - 6-panel refinement summary
- `refined_outliers.png` - 4-panel outlier analysis

### 1.4 Pass SFAC Coefficients for Electron Diffraction

**CRITICAL:** ED data requires custom scattering factors:
```python
refined_atoms, history = refine_structure(
    ...,
    sfac_coefficients=ins.sfac_coefficients,  # REQUIRED for ED!
)
```
Without this, Fc² values are ~5× wrong for heavy atoms.

### 1.5 Never Pass initial_scale=1.0

Let EDref auto-calculate the initial scale from data. Passing `initial_scale=1.0` causes catastrophic divergence.

### 1.6 Convert MergedReflection Objects to Tuples

```python
merged = merge_reflections(hkl.reflections, spacegroup)
# MUST convert for refine_structure():
hkl_data = [(r.h, r.k, r.l, r.intensity, r.sigma) for r in merged]
```

### 1.7 Do NOT Re-Apply U Constraints to SHELXL Structures

SHELXL-refined atoms already satisfy constraints. Re-applying can flip signs and increase R1 by 10%+.

### 1.8 Negative U Values

- Small negative U (e.g., -0.01): Legitimate refined value, use as-is
- Large negative U (e.g., -1.2): Riding hydrogen indicator (parser threshold: U < -0.5)

**Never clamp U to positive values.**

### 1.9 ReciprocalCell Angles are in RADIANS

```python
# CORRECT
cos_gamma_star = np.cos(reciprocal_cell.gamma_star)
# WRONG - don't convert!
# cos_gamma_star = np.cos(np.radians(reciprocal_cell.gamma_star))
```

### 1.10 SHELXL Weighting Uses Absolute Scale

All weights, residuals, and statistics use absolute scale:
```python
Fo²_abs = Fo² / k
σ_abs = σ / k
P = (Fo²_abs + 2·Fc²) / 3
w = 1 / [σ_abs² + (a·P)² + b·P]
```

### 1.11 Do NOT Refine B and U Simultaneously

Resolution-dependent B scale (`k(s) = k₀ × exp(-2B × s²)`) is correlated with U. Use `refine_scale_B=False` when refining U.

### 1.12 Extinction Scaling for ED

EXTI scales with λ³. For ED (λ≈0.02 Å), typical EXTI is 0-50,000+ vs 0-1 for X-rays. EDref auto-scales the search range.

### 1.13 Use Robust Scaling for ED Data

For dynamical scattering outliers:
```python
robust_scale_method='biweight'  # ~2% R1 improvement
```
See manual.md Section 9.2 for all methods.

### 1.14 Dynamical Correction for ED

Apply intensity corrections to mitigate dynamical effects:
```bash
python3 -m edref refine structure.ins --edref --dynamical1      # 1-param (κ)
python3 -m edref refine structure.ins --edref --dynamical3      # 3-param (C, α, β) - legacy
python3 -m edref refine structure.ins --edref --dyn3-alpha-only # α only (C=0, β=0)
python3 -m edref refine structure.ins --edref --dyn-power       # Power-shape (α, γ) - RECOMMENDED
python3 -m edref refine structure.ins --edref --dyn-exp         # Exp-decay (α, d_decay)
python3 -m edref refine structure.ins --edref --dyn-abs         # Absolute (α, d_half, γ)
```

**Recommended model: `--dyn-power`** (power-shape with α and γ parameters)
- Formula: `power = α × (d/d_max)^γ`, then `I_corr = I × (I/I_median)^power`
- γ < 1 applies more correction to outer shell, better matching physical behavior
- MFM-300 at 1.5-0.5Å: R1 = 14.63% (vs 16.92% for α-only, 27% without correction)

See manual.md Section 12.7 for parameter guidelines.

### 1.15 Weight Optimization Differs from SHELXL

EDref uses documented CAPOW algorithm: `Σ(GooF_bin - 1.0)²`

SHELXL uses an undocumented variant. Expect different optimal weights but matching R1 values (~1-2%). This is an expected difference, not a bug.

### 1.16 Use --aniso Flag for Anisotropic Refinement

**⚠️ CRITICAL - ALWAYS USE --aniso FOR ANISOTROPIC REFINEMENT ⚠️**

The `--aniso` CLI flag converts isotropic atoms to anisotropic. Without it, refinement remains isotropic even if you expect anisotropic refinement.

```bash
python3 -m edref refine structure.ins --edref --aniso
```

**How to verify:** Check "Params=N" in output. For 13 atoms:
- Isotropic: ~39 params (3 pos + 1 U_iso per atom + scale)
- Anisotropic: ~80 params (3 pos + 6 U_aniso per atom + scale)

### 1.17 Weight Parameter a=0.0 Can Be Legitimate

The SHELXL weighting scheme `w = 1 / [σ² + (aP)² + bP]` has two parameters. The optimizer may legitimately find `a=0.0` as optimal, meaning only the `bP` term is needed to achieve GooF≈1.0.

**Do NOT assume a=0.0 is a bug.** Check:
- Is GooF close to 1.0? If yes, optimization worked correctly
- Compare with other refinements using same data

### 1.18 R-free Cross-Validation

Use `--rfree` to calculate R-free, a cross-validation statistic that detects overfitting:

```bash
python3 -m edref refine structure.ins --edref --rfree
python3 -m edref refine structure.ins --edref --rfree --rfree-seed 42  # Reproducible
```

**How R-free works:**
- 5% of reflections are randomly selected in 5 resolution shells (equidistant in 1/d²)
- Test reflections are excluded from refinement
- R-free is calculated using the refined model on the test set
- If R-free >> R1(obs), the model may be overfitting

**When to use R-free:**
- Comparing dynamical correction models
- Validating complex refinement strategies
- Detecting over-parameterization

### 1.19 Run Lint and Tests Before Every Push

**ALWAYS run these checks before `git push`:**

```bash
python3 -m ruff check src/edref/        # Lint check
python3 -m ruff check src/edref/ --fix  # Auto-fix lint issues
python3 -m pytest tests/ -v             # Run tests
```

CI will reject pushes with lint errors. Fix all issues locally first.

---

## 2. WHEN TO CONSULT manual.md

| Need | manual.md Section |
|------|-------------------|
| Function signatures, parameters | 5-9 (Data Classes, Core, I/O, Refinement, Analysis) |
| Mathematical derivations | 10 (Crystallographic Formulas) |
| Working code examples | 12 (Code Examples) |
| SHELXL comparison protocol | 11 (SHELXL Compatibility) |
| Troubleshooting | 13 (Common Issues) |
| Key formulas table | Appendix A |
| SHELXL instructions | Appendix B |

---

## 3. PROJECT OVERVIEW

### 3.1 What EDref Does

Minimizes: `M = Σ w(hkl) × [Fo²(hkl) - k·Fc²(hkl)]²`

**Capabilities:** Position, scale, U_iso, U_aniso refinement; SHELXL weighting; special position constraints; extinction correction; dynamical correction; robust scaling.

**Not implemented:** DFIX/DANG/SADI restraints, AFIX riding H, twin refinement, occupancy refinement.

### 3.2 Package Structure

```
EDref_v3/
├── src/edref/
│   ├── core/           # crystallography, symmetry, scattering, structure_factors
│   ├── io/             # formats (data classes), shelxl (.ins/.res/.hkl/.fcf)
│   ├── refinement/     # engine, parameters, derivatives, weighting, statistics, extinction
│   ├── analysis/       # merging, robust_scaling, visualization
│   └── cli/            # runners, refine (argparse)
├── tests/
├── example_data_do_not_modify/   # DO NOT MODIFY - regression testing
│   ├── aspirin/        # Primary: Aspirin.ins, Aspirin.hkl
│   ├── mfm300/         # ED: 2.ins, 2.hkl
│   └── LTA/            # High-symmetry ED: LTA1/, LTA4/
├── test_kit_wisc/      # 24 structures, all crystal systems
└── manual.md           # Complete API reference
```

### 3.3 Installation & Testing

```bash
cd EDref_v3
export PYTHONPATH=$PWD/src:$PYTHONPATH
python3 -m pytest tests/ -v
```

**⚠️ IMPORTANT: Installed Package Takes Precedence ⚠️**

If EDref is installed in `~/.local/lib/python3.10/site-packages/edref/`, it will take precedence over local source code. This causes confusing behavior where your code changes don't take effect.

**Always set PYTHONPATH when testing local changes:**
```bash
PYTHONPATH=/home/agent/claude/EDref_v3/src:$PYTHONPATH python3 -m edref refine ...
```

**To check which edref is being used:**
```bash
python3 -c "import edref; print(edref.__file__)"
# Should show: /home/agent/claude/EDref_v3/src/edref/__init__.py
# NOT: ~/.local/lib/python3.10/site-packages/edref/__init__.py
```

---

## 4. RUNNING EDref

### 4.1 CLI (Recommended)

```bash
# EDref only
python3 -m edref refine structure.ins --edref

# Compare EDref vs SHELXL
python3 -m edref refine structure.ins --compare

# With options
python3 -m edref refine structure.ins --edref --resolution 99 0.6 --cycles 50 --exti --dynamical1
```

**CLI Options:** `--edref`, `--shelxl`, `--compare`, `--resolution MIN MAX`, `--cycles N`, `--batch N`, `--wght A B`, `--quiet`, `--no-plot`, `--save-plots PATH`, `--exti`, `--aniso`, `--no-afix`, `--no-rigu`, `--fix-weights`, `--rfree`, `--rfree-seed N`, `--dynamical1`, `--dynamical3`, `--dyn3-alpha-only`, `--dyn-power`, `--dyn-exp`, `--dyn-abs`, `--output PATH`, `--no-output`, `--no-fcf`

**Output Files:** By default, EDref writes two output files after refinement:

1. **`.res` file** (e.g., `structure_edref.res`): Refined structure in SHELXL format. Olex2-compatible with refinement statistics in REM comments.

2. **`.fcf` file** (e.g., `structure_edref.fcf`): SHELXL LIST 4 format containing:
   - Merged Fo² and σ on absolute scale (divided by k)
   - Final Fc² on absolute scale (with extinction correction applied if refined)
   - If dynamical correction was used, Fo² values are the corrected intensities

Use `--output PATH` to specify a different output basename. Use `--no-output` to disable .res output, or `--no-fcf` to disable .fcf output.

**Saving Plots:** Use `--save-plots PATH` to save visualization plots to PNG files (e.g., `--save-plots myrun` creates `myrun_refine.png`). By default, plots are displayed but not saved. Combine with `--no-plot` to save without displaying.

### 4.2 Python API

```python
from edref import (
    InsFileReader, HklFileReader, SpaceGroup,
    calculate_reciprocal_cell, merge_reflections, refine_structure
)

ins = InsFileReader(ins_path); ins.read()
hkl = HklFileReader(hkl_path); hkl.read()
sg = SpaceGroup(ins.latt, ins.symm)
recip = calculate_reciprocal_cell(ins.cell)
merged = merge_reflections(hkl.reflections, sg, merge_friedel=sg.is_centrosymmetric)
hkl_data = [(r.h, r.k, r.l, r.intensity, r.sigma) for r in merged]

refined_atoms, history = refine_structure(
    atoms=ins.atoms,
    hkl_data=hkl_data,
    sfac_elements=ins.sfac_elements,
    sfac_coefficients=ins.sfac_coefficients,  # REQUIRED for ED!
    spacegroup=sg,
    reciprocal_cell=recip,
    wavelength=ins.wavelength,
    refine_positions=True,
    refine_scale=True,
)

final = history[-1]
print(f"R1={final.R1*100:.2f}% wR2={final.wR2*100:.2f}% GooF={final.GooF:.3f}")
```

### 4.3 Key Parameters

| Parameter | Default | Purpose |
|-----------|---------|---------|
| `refine_positions` | True | x, y, z |
| `refine_scale` | False | Scale k |
| `refine_Uiso` | False | Isotropic U |
| `refine_Uaniso` | False | Anisotropic U |
| `sfac_coefficients` | None | **REQUIRED for ED** |
| `robust_scale_method` | None | 'biweight', 'huber', etc. |
| `refine_extinction` | False | EXTI optimization |
| `damping` | 0.0 | Marquardt-Levenberg |

Full parameter list: manual.md Section 8.

### 4.4 Special Position Constraints

CLI auto-detects constraints. For Python API:
```python
from edref.refinement.parameters import detect_special_positions, detect_U_constraints
constraints = detect_special_positions(ins.atoms)
u_constraints = detect_U_constraints(ins.atoms, constraints)
refine_structure(..., constraints=constraints, u_constraints=u_constraints)
```

Supported types: `4fold_axis`, `mirror_diagonal_xy`, `mirror_yz_sum`, `2fold_xz_fixed`, `z_fixed`, etc.

See manual.md Section 8.4 for full list.

---

## 5. KEY ALGORITHMS

**Objective:** `M = Σ w(hkl) × [Fo²(hkl) - k·Fc²(hkl)]²`

**Scale:** k = FVAR² (same as SHELXL)

**Weighting (absolute scale):**
```
P = (Fo²/k + 2Fc²) / 3
w = 1 / [σ²/k² + (aP)² + bP]
```

**Statistics:**
```
R1 = Σ||Fo| - |Fc|| / Σ|Fo|  (obs: Fo² > 2σ)
wR2 = √[Σw(Fo²/k - Fc²)² / Σw(Fo²/k)²]
GooF = √[Σw(Fo²/k - Fc²)² / (N - P)]
```

**Convergence:** max|Δp/σ(p)| < 0.05

**Normal equations:** (AᵀWA + λI)·Δp = AᵀW·Δy

Full derivations: manual.md Section 10.

---

## 6. FILE FORMATS

### .ins/.res

```
CELL λ a b c α β γ
LATT N        # N<0: non-centro; |N|: 1=P, 2=I, 3=R, 4=F
SFAC C H N O  # Or custom coefficients for ED
FVAR osf      # √k
atom sfac x y z sof U_iso
```

**Key:** Line length ≤80 chars. SOF = 10m + occ. Encoding: Latin-1.

### .hkl (HKLF 4)

Format: h,k,l as I4; Fo², σ as F8.2. Terminator: 0 0 0 0 0

### .fcf

All values on **absolute scale** (Fo²/k, σ/k, Fc²).

Full specs: manual.md Section 7.

---

## 7. TROUBLESHOOTING

| Problem | Solution |
|---------|----------|
| Refinement diverges | Enable damping=0.1; stage refinement (positions first, then U); check for constraints |
| Singular matrix | Add position/U constraints for special positions |
| wR2 much higher than SHELXL | Use `merge_reflections()` (filters absences automatically) |
| Fc values don't match | Pass `sfac_coefficients` for ED; don't re-apply U constraints |
| Scale 60% wrong | Use `weighting_scheme='shelxl'` (default) |
| U not changing | Check normal equations use absolute scale (fixed 2026-01-20) |

Full troubleshooting: manual.md Section 13.

---

## 8. VALIDATED RESULTS

### Aspirin (X-ray)

| Metric | SHELXL | EDref |
|--------|--------|-------|
| R1(obs) | 8.48% | 8.32% |
| wR2 | 24.75% | 24.76% |

### MFM-300 (Electron Diffraction)

| Metric | SHELXL | EDref | EDref + dyn-power |
|--------|--------|-------|-------------------|
| R1(obs) | 32.31% | 31.27% | **14.63%** |
| wR2 | - | - | 36.06% |
| GooF | - | - | 0.691 |

Best dynamical correction: `--dyn-power` with α=1.09, γ=0.28 (at 1.5-0.5Å resolution)

### LTA Zeolite (High-Symmetry ED)

Parameter count matches SHELXL exactly (21) with auto-detected constraints.

Full benchmark tables: manual.md Section 11.

---

## 9. EXTERNAL REFERENCES

### SHELXL Documentation

Location: `/home/agent/claude/shelxlbackground/`
- `SHELXL_Complete_Reference.md`
- `reference_materials/SHELXL_Formulas_and_Equations.md`

### SHELXL Binary

`/home/agent/claude/wilson/benchmark/shelxl`

### Literature

- CAPOW: Johnson et al., *J. Appl. Cryst.* 2018, 51, 304-307
- SHELXL-97 manual: Sheldrick, G.M.

---

## 10. RELEASING

### Process

1. Update version in `src/edref/__init__.py` and `pyproject.toml`
2. Update version test in `tests/test_refinement.py`
3. Run: `python3 -m ruff check src/edref/ && python3 -m pytest tests/ -v`
4. Commit: `git commit -m "Release vX.Y.Z"`
5. Tag: `git tag -a vX.Y.Z -m "Version X.Y.Z"`
6. Push: `git push origin main && git push origin vX.Y.Z`
7. Trigger `publish-pypi` job in GitLab CI

**Versioning:** MAJOR.MINOR.PATCH (semver)

---

*Repository: https://gitlab.com/crystaldiffractor/edref*

*PyPI: https://pypi.org/project/edref/*

*For complete API reference, mathematical derivations, code examples, and troubleshooting: see manual.md*
