NelderMead is a faithful pure-Python port of scipy's implementation — same canonical coefficients (ρ=1, χ=2, ψ=σ=½), same simplex-initialization rules, same convergence test on simplex spread — wrapped in a restart layer that handles the well-known simplex-collapse failure mode described by Kelley (1999).
HumpDay's Nelder-Mead at a glance
- Coefficients: ρ=1 (reflection), χ=2 (expansion), ψ=½ (contraction), σ=½ (shrinkage) — scipy's defaults.
- Simplex init: from a seed point x0, the remaining vertices perturb one coordinate at a time by 1 + nonzdelt (or zdelt if the coordinate is zero).
- Convergence:
xatol = fatol = 1e-12, tightened from scipy's default 1e-4 so the algorithm uses the full budget where the landscape permits. - Restart on collapse: when the simplex meets the convergence test, instead of terminating we reseed and continue (Kelley 1999). Restart 0 uses a fresh uniform draw; subsequent restarts alternate between intensification (reseed around the current best) and diversification (fresh uniform). Each restart uses a different
nonzdeltfrom the schedule[0.05, 0.15, 0.30, 0.10, 0.50, 0.20]so the new simplex isn't a scaled copy of the collapsed one.
Why restart matters
Kelley (1999, SIAM J. Optim. 10(1), DOI) proved that vanilla Nelder-Mead can converge to a non-stationary point when the simplex collapses into a degenerate shape. On smooth landscapes the practical consequence is that the algorithm terminates well before the budget is exhausted — on a sphere objective with a 200-eval budget, the tightened tolerance still triggers around eval ~45, leaving most of the budget unused. The restart layer reseeds when convergence is detected and continues until the budget is consumed, dramatically improving global performance on multimodal surfaces and refining further on smooth ones.
Interactive 3D Visualization
See Nelder-Mead in action on 3D optimization surfaces:
Loading 3D visualization...
Requires WebGL support
Instructions: Choose a test function and algorithm, then click Start to watch the step-by-step optimization process.
Implementation Details
| Component | Details | Links |
|---|---|---|
| Original Algorithm |
John Nelder & Roger Mead Simplex-based direct search method Uses geometric transformations of n+1 vertices Published: 1965 |
Paper |
| Reference Implementation |
scipy.optimize Nelder-Mead scipy's Fortran-backed simplex method. Reference: scipy.optimize.minimize(method='Nelder-Mead')
|
SciPy Docs |
| HumpDay Python |
HumpDay NelderMeadPure-Python simplex method, faithful to scipy's parameters and convergence test. No required dependencies. File: humpday/optimizers/scipy_algorithms.py
|
Source |
| HumpDay JavaScript |
Browser NelderMeadMirrors the Python port. Class: NelderMead
|
JS Port |
Performance Characteristics
- Best for: Low-dimensional black-box objectives where the simplex moves can navigate the surface without curvature information.
- Worst for: High dimensions — simplex moves degenerate past ~10-20 dimensions because the relevant geometry becomes too thin.
- Per iteration: 1 reflection eval; possibly 1 expansion or 1-2 contraction evals; rarely n+1 shrinkage evals.
- Relative to scipy.optimize Nelder-Mead at
n_trials=200, n_dim=2: ties across sphere, Rosenbrock, and Ackley — algorithmically identical, differs only in the interpreter tax. See SOTA status for the live numbers.
Educational Resources
Algorithm Visualization
The Nelder-Mead method is one of the most visually intuitive optimization algorithms. The simplex (triangle in 2D) moves through the parameter space by:
- Reflection: Mirror the worst point across the centroid
- Expansion: If reflection improves, try going further
- Contraction: If reflection fails, contract toward the best point
- Shrinkage: If all else fails, shrink the entire simplex