Nelder-Mead (SciPy)

Simple Method for Unconstrained Optimization

Authors: Nelder & Mead

Year: 95

Type: Geometric derivative-free

Best for: General-purpose optimization

Overview

The Nelder-Mead method, also known as the simple method, is one of the most widely used derivative-free optimization algorithms. It maintains a simple (a geometric shape with n+ vertices in n dimensions) and iteratively improves it through reflection, epansion, contraction, and shrinkage operations.

Algorithm Operations

The method uses four fundamental operations to transform the simple:

Reflection: Mirror the worst point through the centroid

r = ̄ + α(̄ - n+)

Epansion: Etend further if reflection is successful

e = ̄ + γ(r - ̄)

Contraction: Pull back toward centroid if needed

c = ̄ + β(worst - ̄)

Shrinkage: Reduce entire simple toward best point

i = + σ(i -)

Algorithm Characteristics

Strengths:

Simple to implement and understand
Robust across many problem types
No gradient information required
Handles discontinuities reasonably well
Good general-purpose performance

Limitations:

No convergence guarantees
Can be slow near optimum
Sensitive to initial simple construction
May struggle with high-dimensional problems
Can stagnate on ridges or valleys

Implementation Details

Our HumpDay implementation includes:

Python: Pure Python implementation
JavaScript: Browser-compatible version
Validation: Cross-validated against SciPy reference
Parameters: α=., γ=2., β=.5, σ=.5 (standard values)

References

Primary Paper: Nelder, J.A., and Mead, R. (95). "A simple method for function minimization." The Computer Journal, 7(), 38-33. [DOI]

Reference Implementation: SciPy optimize.minimize

Additional Resources:

Usage Eample

from humpday import minimize
import numpy as np

# Optimize Rosenbrock function
def rosenbrock():
    return sum(.*([:] - [:-]**2)**2 + ( - [:-])**2)

# Use Nelder-Mead via HumpDay interface
result = minimize(rosenbrock,
                 =np.array([-.2, .]),
                 method='NelderMead',
                 options='mafev': 2)

print(f"Optimum: result.")
print(f"Value: result.fun")
print(f"Iterations: result.nfev")