import numpy as np
from math import inf
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
from scipy.optimize import shgo
from scipy.optimize import direct
from scipy.optimize import differential_evolution
from scipy.optimize import dual_annealing
from scipy.optimize import basinhopping
import matplotlib.pyplot as plt
5 Sequential Parameter Optimization: Using scipy
Optimizers
This notebook describes how different optimizers form the scipy optimize
package can be used on the surrogate. The optimization algorithms are available from https://docs.scipy.org/doc/scipy/reference/optimize.html
5.1 The Objective Function Branin
The
spotPython
package provides several classes of objective functions.We will use an analytical objective function, i.e., a function that can be described by a (closed) formula.
Here we will use the Branin function. The 2-dim Branin function is
\[y = a * (x2 - b * x1**2 + c * x1 - r) ** 2 + s * (1 - t) * cos(x1) + s,\] where values of a, b, c, r, s and t are: \(a = 1, b = 5.1 / (4*pi**2), c = 5 / pi, r = 6, s = 10\) and \(t = 1 / (8*pi)\).
It has three global minima:
\(f(x) = 0.397887\) at \((-\pi, 12.275)\), \((\pi, 2.275)\), and \((9.42478, 2.475)\).
Input Domain: This function is usually evaluated on the square x1 in [-5, 10] x x2 in [0, 15].
from spotPython.fun.objectivefunctions import analytical
= np.array([-5,-0])
lower = np.array([10,15]) upper
= analytical(seed=123).fun_branin fun
5.2 The Optimizer
Differential Evalution from the
scikit.optimize
package, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution is the default optimizer for the search on the surrogate.Other optimiers that are available in
spotPython
:dual_annealing
direct
shgo
basinhopping
, see https://docs.scipy.org/doc/scipy/reference/optimize.html#global-optimization.
These can be selected as follows:
surrogate_control = "model_optimizer": differential_evolution
We will use
differential_evolution
.The optimizer can use
1000
evaluations. This value will be passed to thedifferential_evolution
method, which has the argumentmaxiter
(int). It defines the maximum number of generations over which the entire differential evolution population is evolved, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution
= spot.Spot(fun=fun,
spot_de = lower,
lower = upper,
upper = 20,
fun_evals = inf,
max_time =125,
seed=False,
noise= False,
show_models={"init_size": 10},
design_control={"n_theta": 2,
surrogate_control"model_optimizer": differential_evolution,
"model_fun_evals": 1000,
}) spot_de.run()
<spotPython.spot.spot.Spot at 0x109275cf0>
5.3 Print the Results
spot_de.print_results()
min y: 0.39954452181606825
x0: -3.1572063272190256
x1: 12.290492397853292
[['x0', -3.1572063272190256], ['x1', 12.290492397853292]]
5.4 Show the Progress
=True) spot_de.plot_progress(log_y
spot_de.surrogate.plot()
5.5 Exercises
5.5.1 dual_annealing
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.2 direct
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.3 shgo
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.4 basinhopping
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.5 Performance Comparison
Compare the performance and run time of the 5 different optimizers:
* `differential_evolution`
* `dual_annealing`
* `direct`
* `shgo`
* `basinhopping`.
The Branin function has three global minima:
- \(f(x) = 0.397887\) at
- \((-\pi, 12.275)\),
- \((\pi, 2.275)\), and
- \((9.42478, 2.475)\).
- Which optima are found by the optimizers? Does the
seed
change this behavior?