Nelder-Mead is a classic derivative-free optimization method that uses a simplex (n+1 vertices in n dimensions) to search the parameter space. It's remarkably robust and intuitive, using geometric operations like reflection, expansion, and contraction to navigate toward the optimum.
Implementation Details
| Component | Details | Links |
|---|---|---|
| Original Algorithm |
John Nelder & Roger Mead Simplex-based direct search method Uses geometric transformations of n+1 vertices Published: 1965 |
📄 Paper |
| Humpday Python |
Python Implementation Uses SciPy's optimize.minimize with method='Nelder-Mead' Well-tested and numerically stable implementation File: humpday/optimizers/scipycube.py |
🐍 Python Code |
| Humpday JavaScript |
Browser Implementation Pure JavaScript port with simplex operations Maintains geometric transformation logic Class: NelderMead |
🌐 JavaScript Code |
| Library Dependencies |
Python: scipy.optimize JavaScript: None (pure JS implementation) |
📖 SciPy Docs |
🏁 Performance Characteristics
- Best for: Low to medium dimensions (2-20D), robust across function types
- Dimensions: Performance degrades significantly above 10-20 dimensions
- Function evaluations: Typically O(n) evaluations per iteration
- Convergence: Reliable but can be slow on ill-conditioned problems
- Robustness: Excellent on noisy, discontinuous, and multimodal functions
📚 Educational Resources
🔍 Algorithm Visualization
The Nelder-Mead method is one of the most visually intuitive optimization algorithms. The simplex (triangle in 2D) moves through the parameter space by:
- Reflection: Mirror the worst point across the centroid
- Expansion: If reflection improves, try going further
- Contraction: If reflection fails, contract toward the best point
- Shrinkage: If all else fails, shrink the entire simplex