import numpy as np
from math import inf
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
from scipy.optimize import shgo
from scipy.optimize import direct
from scipy.optimize import differential_evolution
from scipy.optimize import dual_annealing
from scipy.optimize import basinhopping
5 Sequential Parameter Optimization: Using scipy
Optimizers
This notebook describes how different optimizers form the scipy optimize
package can be used on the surrogate. The optimization algorithms are available from https://docs.scipy.org/doc/scipy/reference/optimize.html
5.1 The Objective Function Branin
The
spotPython
package provides several classes of objective functions.We will use an analytical objective function, i.e., a function that can be described by a (closed) formula.
Here we will use the Branin function. The 2-dim Branin function is
\[y = a * (x2 - b * x1**2 + c * x1 - r) ** 2 + s * (1 - t) * cos(x1) + s,\] where values of a, b, c, r, s and t are: \(a = 1, b = 5.1 / (4*pi**2), c = 5 / pi, r = 6, s = 10\) and \(t = 1 / (8*pi)\).
It has three global minima:
\(f(x) = 0.397887\) at \((-\pi, 12.275)\), \((\pi, 2.275)\), and \((9.42478, 2.475)\).
Input Domain: This function is usually evaluated on the square x1 in [-5, 10] x x2 in [0, 15].
from spotPython.fun.objectivefunctions import analytical
= np.array([-5,-0])
lower = np.array([10,15]) upper
= analytical(seed=123).fun_branin fun
5.2 The Optimizer
Differential Evalution from the
scikit.optimize
package, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution is the default optimizer for the search on the surrogate.Other optimiers that are available in
spotPython
:dual_annealing
direct
shgo
basinhopping
, see https://docs.scipy.org/doc/scipy/reference/optimize.html#global-optimization.
These can be selected as follows:
surrogate_control = "model_optimizer": differential_evolution
We will use
differential_evolution
.The optimizer can use
1000
evaluations. This value will be passed to thedifferential_evolution
method, which has the argumentmaxiter
(int). It defines the maximum number of generations over which the entire differential evolution population is evolved, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution
Similar to the one-dimensional case, which was introduced in Section Section 1.7, we can use TensorBoard to monitor the progress of the optimization. We will use the same code, only the prefix is different:
from spotPython.utils.file import get_experiment_name
from spotPython.utils.init import fun_control_init
from spotPython.utils.file import get_spot_tensorboard_path
= "05_DE_"
PREFIX = get_experiment_name(prefix=PREFIX)
experiment_name print(experiment_name)
= fun_control_init(
fun_control =get_spot_tensorboard_path(experiment_name)) spot_tensorboard_path
05_DE__bartz08-2_2023-07-10_00-20-52
= spot.Spot(fun=fun,
spot_de = lower,
lower = upper,
upper = 20,
fun_evals = inf,
max_time =125,
seed=False,
noise= False,
show_models={"init_size": 10},
design_control={"n_theta": len(lower),
surrogate_control"model_optimizer": differential_evolution,
"model_fun_evals": 1000,
},=fun_control)
fun_control spot_de.run()
<spotPython.spot.spot.Spot at 0x107a1eb30>
5.2.1 TensorBoard
Now we can start TensorBoard in the background with the following command:
tensorboard --logdir="./runs"
We can access the TensorBoard web server with the following URL:
http://localhost:6006/
The TensorBoard plot illustrates how spotPython
can be used as a microscope for the internal mechanisms of the surrogate-based optimization process. Here, one important parameter, the learning rate \(\theta\) of the Kriging surrogate is plotted against the number of optimization steps.
5.3 Print the Results
spot_de.print_results()
min y: 0.39969164980122684
x0: -3.158224446089584
x1: 12.293182279400076
[['x0', -3.158224446089584], ['x1', 12.293182279400076]]
5.4 Show the Progress
=True) spot_de.plot_progress(log_y
spot_de.surrogate.plot()
5.5 Exercises
5.5.1 dual_annealing
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.2 direct
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.3 shgo
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.4 basinhopping
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
5.5.5 Performance Comparison
Compare the performance and run time of the 5 different optimizers:
* `differential_evolution`
* `dual_annealing`
* `direct`
* `shgo`
* `basinhopping`.
The Branin function has three global minima:
- \(f(x) = 0.397887\) at
- \((-\pi, 12.275)\),
- \((\pi, 2.275)\), and
- \((9.42478, 2.475)\).
- Which optima are found by the optimizers? Does the
seed
change this behavior?