list | grep "spot[RiverPython]" pip
spotPython 0.2.33
spotRiver 0.0.93
Note: you may need to restart the kernel to use updated packages.
spotPython
and Ray Tune on CIFAR10In this tutorial, we will show how spotPython
can be integrated into the PyTorch
training workflow. It is based on the tutorial “Hyperparameter Tuning with Ray Tune” from the PyTorch
documentation (PyTorch 2023a), which is an extension of the tutorial “Training a Classifier” (PyTorch 2023b) for training a CIFAR10 image classifier.
This document refers to the following software versions:
python
: 3.10.10torch
: 2.0.1torchvision
: 0.15.0list | grep "spot[RiverPython]" pip
spotPython 0.2.33
spotRiver 0.0.93
Note: you may need to restart the kernel to use updated packages.
spotPython
can be installed via pip1.
!pip install spotPython
spotPython
from gitHub.# import sys
# !{sys.executable} -m pip install --upgrade build
# !{sys.executable} -m pip install --upgrade --force-reinstall spotPython
Results that refer to the Ray Tune
package are taken from https://PyTorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html2.
Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.
= 30
MAX_TIME = 10
INIT_SIZE = "cpu" # "cuda:0" DEVICE
from spotPython.utils.device import getDevice
= getDevice(DEVICE)
DEVICE print(DEVICE)
cpu
import os
import copy
import socket
import warnings
from datetime import datetime
from dateutil.tz import tzlocal
= datetime.now(tzlocal())
start_time = socket.gethostname().split(".")[0]
HOSTNAME = '14-torch' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
experiment_name = experiment_name.replace(':', '-')
experiment_name print(experiment_name)
if not os.path.exists('./figures'):
'./figures')
os.makedirs("ignore") warnings.filterwarnings(
14-torch_p040025_30min_10init_2023-06-17_00-07-28
fun_control
DictionaryspotPython
uses a Python dictionary for storing the information required for the hyperparameter tuning process. This dictionary is called fun_control
and is initialized with the function fun_control_init
. The function fun_control_init
returns a skeleton dictionary. The dictionary is filled with the required information for the hyperparameter tuning process. It stores the hyperparameter tuning settings, e.g., the deep learning network architecture that should be tuned, the classification (or regression) problem, and the data that is used for the tuning. The dictionary is used as an input for the SPOT function.
from spotPython.utils.init import fun_control_init
= fun_control_init(task="classification",
fun_control ="runs/14_spot_ray_hpt_torch_cifar10",
tensorboard_path=DEVICE,) device
The data loading process is implemented in the same manner as described in the Section “Data loaders” in PyTorch (2023a). The data loaders are wrapped into the function load_data_cifar10
which is identical to the function load_data
in PyTorch (2023a). A global data directory is used, which allows sharing the data directory between different trials. The method load_data_cifar10
is part of the spotPython
package and can be imported from spotPython.data.torchdata
.
In the following step, the test and train data are added to the dictionary fun_control
.
from spotPython.data.torchdata import load_data_cifar10
= load_data_cifar10()
train, test = len(train)
n_samples # add the dataset to the fun_control
fun_control.update({"train": train,
"test": test,
"n_samples": n_samples})
Files already downloaded and verified
Files already downloaded and verified
After the training and test data are specified and added to the fun_control
dictionary, spotPython
allows the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables. The preprocessing model is called prep_model
(“preparation” or pre-processing) and includes steps that are not subject to the hyperparameter tuning process. The preprocessing model is specified in the fun_control
dictionary. The preprocessing model can be implemented as a sklearn
pipeline. The following code shows a typical preprocessing pipeline:
categorical_columns = ["cities", "colors"]
one_hot_encoder = OneHotEncoder(handle_unknown="ignore",
sparse_output=False)
prep_model = ColumnTransformer(
transformers=[
("categorical", one_hot_encoder, categorical_columns),
],
remainder=StandardScaler(),
)
Because the Ray Tune (ray[tune]
) hyperparameter tuning as described in PyTorch (2023a) does not use a preprocessing model, the preprocessing model is set to None
here.
= None
prep_model "prep_model": prep_model}) fun_control.update({
algorithm
and core_model_hyper_dict
The same neural network model as implemented in the section “Configurable neural network” of the PyTorch
tutorial (PyTorch 2023a) is used here. We will show the implementation from PyTorch (2023a) in Section 13.4.2.1 first, before the extended implementation with spotPython
is shown in ?sec-implementation-with-spotpython.
We used the same hyperparameters that are implemented as configurable in the PyTorch
tutorial. We specify the layer sizes, namely l1
and l2
, of the fully connected layers:
class Net(nn.Module):
def __init__(self, l1=120, l2=84):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, l1)
self.fc2 = nn.Linear(l1, l2)
self.fc3 = nn.Linear(l2, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x
The learning rate, i.e., lr
, of the optimizer is made configurable, too:
optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)
spotPython
implements a class which is similar to the class described in the PyTorch
tutorial. The class is called Net_CIFAR10
and is implemented in the file netcifar10.py
.
from torch import nn
import torch.nn.functional as F
import spotPython.torch.netcore as netcore
class Net_CIFAR10(netcore.Net_Core):
def __init__(self, l1, l2, lr_mult, batch_size, epochs, k_folds, patience,
optimizer, sgd_momentum):
super(Net_CIFAR10, self).__init__(
lr_mult=lr_mult,
batch_size=batch_size,
epochs=epochs,
k_folds=k_folds,
patience=patience,
optimizer=optimizer,
sgd_momentum=sgd_momentum,
)
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, l1)
self.fc2 = nn.Linear(l1, l2)
self.fc3 = nn.Linear(l2, 10)
def forward(self, x):
x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = self.fc3(x)
return x
Net_Core
classNet_CIFAR10
inherits from the class Net_Core
which is implemented in the file netcore.py
. It implements the additional attributes that are common to all neural network models. The Net_Core
class is implemented in the file netcore.py
. It implements hyperparameters as attributes, that are not used by the core_model
, e.g.:
optimizer
),lr
),batch_size
),epochs
),k_folds
), andpatience
).Users can add further attributes to the class. The class Net_Core
is shown below.
from torch import nn
class Net_Core(nn.Module):
def __init__(self, lr_mult, batch_size, epochs, k_folds, patience,
optimizer, sgd_momentum):
super(Net_Core, self).__init__()
self.lr_mult = lr_mult
self.batch_size = batch_size
self.epochs = epochs
self.k_folds = k_folds
self.patience = patience
self.optimizer = optimizer
self.sgd_momentum = sgd_momentum
Comparing the class Net
from the PyTorch
tutorial and the class Net_CIFAR10
from spotPython
, we see that the class Net_CIFAR10
has additional attributes and does not inherit from nn
directly. It adds an additional class, Net_core
, that takes care of additional attributes that are common to all neural network models, e.g., the learning rate multiplier lr_mult
or the batch size batch_size
.
spotPython
’s core_model
implements an instance of the Net_CIFAR10
class. In addition to the basic neural network model, the core_model
can use these additional attributes. spotPython
provides methods for handling these additional attributes to guarantee 100% compatibility with the PyTorch
classes. The method add_core_model_to_fun_control
adds the hyperparameters and additional attributes to the fun_control
dictionary. The method is shown below.
from spotPython.torch.netcifar10 import Net_CIFAR10
from spotPython.data.torch_hyper_dict import TorchHyperDict
from spotPython.hyperparameters.values import add_core_model_to_fun_control
= Net_CIFAR10
core_model = add_core_model_to_fun_control(core_model=core_model,
fun_control =fun_control,
fun_control=TorchHyperDict,
hyper_dict=None) filename
In Section 13.5.1, we first describe how to configure the search space with ray[tune]
(as shown in PyTorch (2023a)) and then how to configure the search space with spotPython
in ?sec-configuring-the-search-space-with-spotpython.
Ray Tune’s search space can be configured as follows (PyTorch 2023a):
config = {
"l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
"lr": tune.loguniform(1e-4, 1e-1),
"batch_size": tune.choice([2, 4, 8, 16])
}
The tune.sample_from()
function enables the user to define sample methods to obtain hyperparameters. In this example, the l1
and l2
parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The lr
(learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.
At each trial, ray[tune]
will randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. ray[tune]
uses the ASHAScheduler
which will terminate bad performing trials early.
hyper_dict
Hyperparameters for the Selected AlgorithmspotPython
uses JSON
files for the specification of the hyperparameters. Users can specify their individual JSON
files, or they can use the JSON
files provided by spotPython
. The JSON
file for the core_model
is called torch_hyper_dict.json
.
In contrast to ray[tune]
, spotPython
can handle numerical, boolean, and categorical hyperparameters. They can be specified in the JSON
file in a similar way as the numerical hyperparameters as shown below. Each entry in the JSON
file represents one hyperparameter with the following structure: type
, default
, transform
, lower
, and upper
.
"factor_hyperparameter": {
"levels": ["A", "B", "C"],
"type": "factor",
"default": "B",
"transform": "None",
"core_model_parameter_type": "str",
"lower": 0,
"upper": 2},
The corresponding entries for the Net_CIFAR10
class are shown below.
{"Net_CIFAR10":
{
"l1": {
"type": "int",
"default": 5,
"transform": "transform_power_2_int",
"lower": 2,
"upper": 9},
"l2": {
"type": "int",
"default": 5,
"transform": "transform_power_2_int",
"lower": 2,
"upper": 9},
"lr_mult": {
"type": "float",
"default": 1.0,
"transform": "None",
"lower": 0.1,
"upper": 10},
"batch_size": {
"type": "int",
"default": 4,
"transform": "transform_power_2_int",
"lower": 1,
"upper": 4},
"epochs": {
"type": "int",
"default": 3,
"transform": "transform_power_2_int",
"lower": 1,
"upper": 4},
"k_folds": {
"type": "int",
"default": 2,
"transform": "None",
"lower": 2,
"upper": 3},
"patience": {
"type": "int",
"default": 5,
"transform": "None",
"lower": 2,
"upper": 10},
"optimizer": {
"levels": ["Adadelta",
"Adagrad",
"Adam",
"AdamW",
"SparseAdam",
"Adamax",
"ASGD",
"LBFGS",
"NAdam",
"RAdam",
"RMSprop",
"Rprop",
"SGD"],
"type": "factor",
"default": "SGD",
"transform": "None",
"class_name": "torch.optim",
"core_model_parameter_type": "str",
"lower": 0,
"upper": 12},
"sgd_momentum": {
"type": "float",
"default": 0.0,
"transform": "None",
"lower": 0.0,
"upper": 1.0}
}
}
Ray tune (PyTorch 2023a) does not provide a way to change the specified hyperparameters without re-compilation. However, spotPython
provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions are described in the following.
hyper_dict
Hyperparameters for the Selected Algorithm aka core_model
After specifying the model, the corresponding hyperparameters, their types and bounds are loaded from the JSON
file torch_hyper_dict.json
. After loading, the user can modify the hyperparameters, e.g., the bounds. spotPython
provides a simple rule for de-activating hyperparameters: If the lower and the upper bound are set to identical values, the hyperparameter is de-activated. This is useful for the hyperparameter tuning, because it allows to specify a hyperparameter in the JSON
file, but to de-activate it in the fun_control
dictionary. This is done in the next step.
Since the hyperparameter k_folds
is not used in the PyTorch
tutorial, it is de-activated here by setting the lower and upper bound to the same value. Note, k_folds
is of type “integer”.
from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
= modify_hyper_parameter_bounds(fun_control,
fun_control "batch_size", bounds=[1, 5])
= modify_hyper_parameter_bounds(fun_control,
fun_control "k_folds", bounds=[0, 0])
= modify_hyper_parameter_bounds(fun_control,
fun_control "patience", bounds=[3, 3])
In a similar manner as for the numerical hyperparameters, the categorical hyperparameters can be modified. New configurations can be chosen by adding or deleting levels. For example, the hyperparameter optimizer
can be re-configured as follows:
In the following setting, two optimizers ("SGD"
and "Adam"
) will be compared during the spotPython
hyperparameter tuning. The hyperparameter optimizer
is active.
from spotPython.hyperparameters.values import modify_hyper_parameter_levels
= modify_hyper_parameter_levels(fun_control,
fun_control "optimizer", ["SGD", "Adam"])
The hyperparameter optimizer
can be de-activated by choosing only one value (level), here: "SGD"
.
= modify_hyper_parameter_levels(fun_control, "optimizer", ["SGD"]) fun_control
As discussed in Section 13.6, there are some issues with the LBFGS optimizer. Therefore, the usage of the LBFGS optimizer is not deactivated in spotPython
by default. However, the LBFGS optimizer can be activated by adding it to the list of optimizers. Rprop
was removed, because it does perform very poorly (as some pre-tests have shown). However, it can also be activated by adding it to the list of optimizers. Since SparseAdam
does not support dense gradients, Adam
was used instead. Therefore, there are 10 default optimizers:
= modify_hyper_parameter_levels(fun_control, "optimizer",
fun_control "Adadelta", "Adagrad", "Adam", "AdamW", "Adamax", "ASGD",
["NAdam", "RAdam", "RMSprop", "SGD"])
Table 13.1 shows some of the optimizers available in PyTorch
:
\(a\) denotes (0.9,0.999), \(b\) (0.5,1.2), and \(c\) (1e-6, 50), respectively. \(R\) denotes required, but unspecified
. “m” denotes momentum
, “w_d” weight_decay
, “d” dampening
, “n” nesterov
, “r” rho
, “l_s” learning rate for scaling delta
, “l_d” lr_decay
, “b” betas
, “l” lambd
, “a” alpha
, “m_d” for momentum_decay
, “e” etas
, and “s_s” for step_sizes
.
Optimizer | lr | m | w_d | d | n | r | l_s | l_d | b | l | a | m_d | e | s_s |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Adadelta | - | - | 0. | - | - | 0.9 | 1. | - | - | - | - | - | - | - |
Adagrad | 1e-2 | - | 0. | - | - | - | - | 0. | - | - | - | - | - | - |
Adam | 1e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | - | - | - |
AdamW | 1e-3 | - | 1e-2 | - | - | - | - | - | \(a\) | - | - | - | - | - |
SparseAdam | 1e-3 | - | - | - | - | - | - | - | \(a\) | - | - | - | - | - |
Adamax | 2e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | - | - | - |
ASGD | 1e-2 | .9 | 0. | - | F | - | - | - | - | 1e-4 | .75 | - | - | - |
LBFGS | 1. | - | - | - | - | - | - | - | - | - | - | - | - | - |
NAdam | 2e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | 0 | - | - |
RAdam | 1e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | - | - | - |
RMSprop | 1e-2 | 0. | 0. | - | - | - | - | - | \(a\) | - | - | - | - | - |
Rprop | 1e-2 | - | - | - | - | - | - | - | - | - | \(b\) | \(c\) | - | - |
SGD | \(R\) | 0. | 0. | 0. | F | - | - | - | - | - | - | - | - | - |
spotPython
implements an optimization
handler that maps the optimizer names to the corresponding PyTorch
optimizers.
We recommend deactivating PyTorch
’s LBFGS optimizer, because it does not perform very well. The PyTorch
documentation, see https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html#torch.optim.LBFGS, states:
This is a very memory intensive optimizer (it requires additional
param_bytes * (history_size + 1)
bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.
Furthermore, the LBFGS optimizer is not compatible with the PyTorch
tutorial. The reason is that the LBFGS optimizer requires the closure
function, which is not implemented in the PyTorch
tutorial. Therefore, the LBFGS
optimizer is recommended here. Since there are ten optimizers in the portfolio, it is not recommended tuning the hyperparameters that effect one single optimizer only.
spotPython
provides a multiplier for the default learning rates, lr_mult
, because optimizers use different learning rates. Using a multiplier for the learning rates might enable a simultaneous tuning of the learning rates for all optimizers. However, this is not recommended, because the learning rates are not comparable across optimizers. Therefore, we recommend fixing the learning rate for all optimizers if multiple optimizers are used. This can be done by setting the lower and upper bounds of the learning rate multiplier to the same value as shown below.
Thus, the learning rate, which affects the SGD
optimizer, will be set to a fixed value. We choose the default value of 1e-3
for the learning rate, because it is used in other PyTorch
examples (it is also the default value used by spotPython
as defined in the optimizer_handler()
method). We recommend tuning the learning rate later, when a reduced set of optimizers is fixed. Here, we will demonstrate how to select in a screening phase the optimizers that should be used for the hyperparameter tuning.
For the same reason, we will fix the sgd_momentum
to 0.9
.
= modify_hyper_parameter_bounds(fun_control,
fun_control "lr_mult", bounds=[1.0, 1.0])
= modify_hyper_parameter_bounds(fun_control,
fun_control "sgd_momentum", bounds=[0.9, 0.9])
The evaluation procedure requires the specification of the way how the data is split into a train and a test set and the loss function (and a metric). As a default, spotPython
provides a standard hold-out data split and cross validation.
If a hold-out data split is used, the data will be partitioned into a training, a validation, and a test data set. The split depends on the setting of the eval
parameter. If eval
is set to train_hold_out
, one data set, usually the original training data set, is split into a new training and a validation data set. The training data set is used for training the model. The validation data set is used for the evaluation of the hyperparameter configuration and early stopping to prevent overfitting. In this case, the original test data set is not used.
spotPython
returns the hyperparameters of the machine learning and deep learning models, e.g., number of layers, learning rate, or optimizer, but not the model weights. Therefore, after the SPOT run is finished, the corresponding model with the optimized architecture has to be trained again with the best hyperparameter configuration. The training is performed on the training data set. The test data set is used for the final evaluation of the model.
Summarizing, the following splits are performed in the hold-out setting:
spotPython
with eval
set to train_hold_out
to determine the best hyperparameter configuration.train_tuned(model_spot, train, "model_spot.pt")
.test_tuned(model_spot, test, "model_spot.pt")
These steps will be exemplified in the following sections.
In addition to this hold-out
setting, spotPython
provides another hold-out setting, where an explicit test data is specified by the user that will be used as the validation set. To choose this option, the eval
parameter is set to test_hold_out
. In this case, the training data set is used for the model training. Then, the explicitly defined test data set is used for the evaluation of the hyperparameter configuration (the validation).
The cross validation setting is used by setting the eval
parameter to train_cv
or test_cv
. In both cases, the data set is split into \(k\) folds. The model is trained on \(k-1\) folds and evaluated on the remaining fold. This is repeated \(k\) times, so that each fold is used exactly once for evaluation. The final evaluation is performed on the test data set. The cross validation setting is useful for small data sets, because it allows to use all data for training and evaluation. However, it is computationally expensive, because the model has to be trained \(k\) times.
Combinations of the above settings are possible, e.g., cross validation can be used for training and hold-out for evaluation or vice versa. Also, cross validation can be used for training and testing. Because cross validation is not used in the PyTorch
tutorial (PyTorch 2023a), it is not considered further here.
An overview of the training evaluations is shown in Table 13.2. "train_cv"
and "test_cv"
use sklearn.model_selection.KFold()
internally. More details on the data splitting are provided in Section 20.14 (in the Appendix).
eval |
train |
test |
function | comment |
---|---|---|---|---|
"train_hold_out" |
\(\checkmark\) | train_one_epoch() , validate_one_epoch() for early stopping |
splits the train data set internally |
|
"test_hold_out" |
\(\checkmark\) | \(\checkmark\) | train_one_epoch() , validate_one_epoch() for early stopping |
use the test data set for validate_one_epoch() |
"train_cv" |
\(\checkmark\) | evaluate_cv(net, train) |
CV using the train data set |
|
"test_cv" |
\(\checkmark\) | evaluate_cv(net, test) |
CV using the test data set . Identical to "train_cv" , uses only test data. |
train_tuned(model, train)
: train the model with the best hyperparameter configuration (or simply the default) on the training data set. It splits the train
data into new train
and validation
sets using create_train_val_data_loaders()
, which calls torch.utils.data.random_split()
internally. Currently, 60% of the data is used for training and 40% for validation. The train
data is used for training the model with train_hold_out()
. The validation
data is used for early stopping using validate_fold_or_hold_out()
on the validation
data set.
test_tuned(model, test)
: test the model on the test data set. No data splitting is performed. The (trained) model is evaluated using the validate_fold_or_hold_out()
function. Note: During training, "shuffle"
is set to True
, whereas during testing, "shuffle"
is set to False
.
Section 20.14.1.4 describes the final evaluation of the tuned architecture.
fun_control.update({"eval": "train_hold_out",
"path": "torch_model.pt",
"shuffle": True})
The key "loss_function"
specifies the loss function which is used during the optimization. There are several different loss functions under PyTorch
’s nn
package. For example, a simple loss is MSELoss
, which computes the mean-squared error between the output and the target. In this tutorial we will use CrossEntropyLoss
, because it is also used in the PyTorch
tutorial.
from torch.nn import CrossEntropyLoss
= CrossEntropyLoss()
loss_function "loss_function": loss_function}) fun_control.update({
In addition to the loss functions, spotPython
provides access to a large number of metrics.
"metric_sklearn"
is used for metrics that follow the scikit-learn
conventions."river_metric"
is used for the river based evaluation (Montiel et al. 2021) via eval_oml_iter_progressive
, and"metric_torch"
is used for the metrics from TorchMetrics
.TorchMetrics
is a collection of more than 90 PyTorch metrics, see https://torchmetrics.readthedocs.io/en/latest/. Because the PyTorch
tutorial uses the accuracy as metric, we use the same metric here. Currently, accuracy is computed in the tutorial’s example code. We will use TorchMetrics
instead, because it offers more flexibilty, e.g., it can be used for regression and classification. Furthermore, TorchMetrics
offers the following advantages:
* A standardized interface to increase reproducibility
* Reduces Boilerplate
* Distributed-training compatible
* Rigorously tested
* Automatic accumulation over batches
* Automatic synchronization between multiple devices
Therefore, we set
import torchmetrics
= torchmetrics.Accuracy(task="multiclass", num_classes=10).to(fun_control["device"])
metric_torch "metric_torch": metric_torch}) fun_control.update({
The following code passes the information about the parameter ranges and bounds to spot
.
from spotPython.hyperparameters.values import (
get_var_type,
get_var_name,
get_bound_values
)= get_var_type(fun_control)
var_type = get_var_name(fun_control)
var_name "var_type": var_type,
fun_control.update({"var_name": var_name})
= get_bound_values(fun_control, "lower")
lower = get_bound_values(fun_control, "upper") upper
Now, the dictionary fun_control
contains all information needed for the hyperparameter tuning. Before the hyperparameter tuning is started, it is recommended to take a look at the experimental design. The method gen_design_table
generates a design table as follows:
from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control))
| name | type | default | lower | upper | transform |
|--------------|--------|-----------|---------|---------|-----------------------|
| l1 | int | 5 | 2 | 9 | transform_power_2_int |
| l2 | int | 5 | 2 | 9 | transform_power_2_int |
| lr_mult | float | 1.0 | 1 | 1 | None |
| batch_size | int | 4 | 1 | 5 | transform_power_2_int |
| epochs | int | 3 | 3 | 4 | transform_power_2_int |
| k_folds | int | 1 | 0 | 0 | None |
| patience | int | 5 | 3 | 3 | None |
| optimizer | factor | SGD | 0 | 9 | None |
| sgd_momentum | float | 0.0 | 0.9 | 0.9 | None |
This allows to check if all information is available and if the information is correct. ?tbl-design shows the experimental design for the hyperparameter tuning. The table shows the hyperparameters, their types, default values, lower and upper bounds, and the transformation function. The transformation function is used to transform the hyperparameter values from the unit hypercube to the original domain. The transformation function is applied to the hyperparameter values before the evaluation of the objective function. Hyperparameter transformations are shown in the column “transform”, e.g., the l1
default is 5
, which results in the value \(2^5 = 32\) for the network, because the transformation transform_power_2_int
was selected in the JSON
file. The default value of the batch_size
is set to 4
, which results in a batch size of \(2^4 = 16\).
fun_torch
The objective function fun_torch
is selected next. It implements an interface from PyTorch
’s training, validation, and testing methods to spotPython
.
from spotPython.fun.hypertorch import HyperTorch
= HyperTorch().fun_torch fun
We add the default setting to the initial design:
from spotPython.hyperparameters.values import get_default_hyperparameters_as_array
=TorchHyperDict().load()
hyper_dict= get_default_hyperparameters_as_array(fun_control, hyper_dict) X_start
The spotPython
hyperparameter tuning is started by calling the Spot
function. Here, we will run the tuner for approximately 30 minutes (max_time
). Note: the initial design is always evaluated in the spotPython
run. As a consequence, the run may take longer than specified by max_time
, because the evaluation time of initial design (here: init_size
, 10 points) is performed independently of max_time
. During the run, results from the training is shown. These results can be visualized with Tensorboard as will be shown in Section 13.13.
from spotPython.spot import spot
from math import inf
import numpy as np
= spot.Spot(fun=fun,
spot_tuner = lower,
lower = upper,
upper = inf,
fun_evals = 1,
fun_repeats = MAX_TIME,
max_time = False,
noise = np.sqrt(np.spacing(1)),
tolerance_x = var_type,
var_type = var_name,
var_name = "y",
infill_criterion = 1,
n_points =123,
seed= 50,
log_level = False,
show_models= True,
show_progress= fun_control,
fun_control ={"init_size": INIT_SIZE,
design_control"repeats": 1},
={"noise": True,
surrogate_control"cod_type": "norm",
"min_theta": -4,
"max_theta": 3,
"n_theta": len(var_name),
"model_fun_evals": 10_000,
"log_level": 50
})=X_start) spot_tuner.run(X_start
config: {'l1': 64, 'l2': 16, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.4531873696327209
Accuracy on hold-out set: 0.465
MulticlassAccuracy value on hold-out data: 0.4650000035762787
Epoch: 2
Loss on hold-out set: 1.325196225118637
Accuracy on hold-out set: 0.525
MulticlassAccuracy value on hold-out data: 0.5249999761581421
Epoch: 3
Loss on hold-out set: 1.3048064527988434
Accuracy on hold-out set: 0.53355
MulticlassAccuracy value on hold-out data: 0.5335500240325928
Epoch: 4
Loss on hold-out set: 1.2734438054800035
Accuracy on hold-out set: 0.55855
MulticlassAccuracy value on hold-out data: 0.5585500001907349
Epoch: 5
Loss on hold-out set: 1.2120844751596451
Accuracy on hold-out set: 0.5749
MulticlassAccuracy value on hold-out data: 0.5748999714851379
Epoch: 6
Loss on hold-out set: 1.2068462555170059
Accuracy on hold-out set: 0.5833
MulticlassAccuracy value on hold-out data: 0.583299994468689
Epoch: 7
Loss on hold-out set: 1.1836825958490371
Accuracy on hold-out set: 0.58985
MulticlassAccuracy value on hold-out data: 0.5898500084877014
Epoch: 8
Loss on hold-out set: 1.1850792266607284
Accuracy on hold-out set: 0.59325
MulticlassAccuracy value on hold-out data: 0.5932499766349792
Epoch: 9
Loss on hold-out set: 1.166281735277176
Accuracy on hold-out set: 0.60015
MulticlassAccuracy value on hold-out data: 0.6001499891281128
Epoch: 10
Loss on hold-out set: 1.1835454508304597
Accuracy on hold-out set: 0.5997
MulticlassAccuracy value on hold-out data: 0.5996999740600586
Epoch: 11
Loss on hold-out set: 1.2132742559909822
Accuracy on hold-out set: 0.5947
MulticlassAccuracy value on hold-out data: 0.5946999788284302
Epoch: 12
Loss on hold-out set: 1.2120959482431413
Accuracy on hold-out set: 0.5944
MulticlassAccuracy value on hold-out data: 0.5943999886512756
Early stopping at epoch 11
Returned to Spot: Validation loss: 1.2120959482431413
----------------------------------------------
config: {'l1': 16, 'l2': 128, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5005870576888323
Accuracy on hold-out set: 0.4507
MulticlassAccuracy value on hold-out data: 0.4507000148296356
Epoch: 2
Loss on hold-out set: 1.4895953057467937
Accuracy on hold-out set: 0.46875
MulticlassAccuracy value on hold-out data: 0.46875
Epoch: 3
Loss on hold-out set: 1.5067340997964145
Accuracy on hold-out set: 0.4771
MulticlassAccuracy value on hold-out data: 0.4771000146865845
Epoch: 4
Loss on hold-out set: 1.420065576042235
Accuracy on hold-out set: 0.5134
MulticlassAccuracy value on hold-out data: 0.5134000182151794
Epoch: 5
Loss on hold-out set: 1.3807816077925266
Accuracy on hold-out set: 0.5275
MulticlassAccuracy value on hold-out data: 0.5274999737739563
Epoch: 6
Loss on hold-out set: 1.3997260218821466
Accuracy on hold-out set: 0.53845
MulticlassAccuracy value on hold-out data: 0.5384500026702881
Epoch: 7
Loss on hold-out set: 1.358016163313389
Accuracy on hold-out set: 0.544
MulticlassAccuracy value on hold-out data: 0.5440000295639038
Epoch: 8
Loss on hold-out set: 1.4457263150051236
Accuracy on hold-out set: 0.5355
MulticlassAccuracy value on hold-out data: 0.5354999899864197
Epoch: 9
Loss on hold-out set: 1.4476015083182603
Accuracy on hold-out set: 0.53715
MulticlassAccuracy value on hold-out data: 0.5371500253677368
Epoch: 10
Loss on hold-out set: 1.3770421450687573
Accuracy on hold-out set: 0.54405
MulticlassAccuracy value on hold-out data: 0.5440499782562256
Early stopping at epoch 9
Returned to Spot: Validation loss: 1.3770421450687573
----------------------------------------------
config: {'l1': 32, 'l2': 8, 'lr_mult': 1.0, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'ASGD', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 2.0151158308029173
Accuracy on hold-out set: 0.2361
MulticlassAccuracy value on hold-out data: 0.2361000031232834
Epoch: 2
Loss on hold-out set: 1.8380549493789673
Accuracy on hold-out set: 0.29645
MulticlassAccuracy value on hold-out data: 0.2964499890804291
Epoch: 3
Loss on hold-out set: 1.7380830037593842
Accuracy on hold-out set: 0.34725
MulticlassAccuracy value on hold-out data: 0.3472500145435333
Epoch: 4
Loss on hold-out set: 1.6875056188583375
Accuracy on hold-out set: 0.3618
MulticlassAccuracy value on hold-out data: 0.3617999851703644
Epoch: 5
Loss on hold-out set: 1.6507797204971313
Accuracy on hold-out set: 0.3751
MulticlassAccuracy value on hold-out data: 0.3750999867916107
Epoch: 6
Loss on hold-out set: 1.6054888113737107
Accuracy on hold-out set: 0.38645
MulticlassAccuracy value on hold-out data: 0.38644999265670776
Epoch: 7
Loss on hold-out set: 1.5978620321035386
Accuracy on hold-out set: 0.3944
MulticlassAccuracy value on hold-out data: 0.3944000005722046
Epoch: 8
Loss on hold-out set: 1.5594049923419953
Accuracy on hold-out set: 0.41145
MulticlassAccuracy value on hold-out data: 0.41144999861717224
Returned to Spot: Validation loss: 1.5594049923419953
----------------------------------------------
config: {'l1': 8, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adamax', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5543288917303086
Accuracy on hold-out set: 0.42405
MulticlassAccuracy value on hold-out data: 0.4240500032901764
Epoch: 2
Loss on hold-out set: 1.4726357238292693
Accuracy on hold-out set: 0.4561
MulticlassAccuracy value on hold-out data: 0.4560999870300293
Epoch: 3
Loss on hold-out set: 1.4089747812896967
Accuracy on hold-out set: 0.48535
MulticlassAccuracy value on hold-out data: 0.48535001277923584
Epoch: 4
Loss on hold-out set: 1.3494931949332356
Accuracy on hold-out set: 0.51255
MulticlassAccuracy value on hold-out data: 0.5125499963760376
Epoch: 5
Loss on hold-out set: 1.3851889771848918
Accuracy on hold-out set: 0.50685
MulticlassAccuracy value on hold-out data: 0.506850004196167
Epoch: 6
Loss on hold-out set: 1.323475112516433
Accuracy on hold-out set: 0.5249
MulticlassAccuracy value on hold-out data: 0.5249000191688538
Epoch: 7
Loss on hold-out set: 1.3506805676035583
Accuracy on hold-out set: 0.5229
MulticlassAccuracy value on hold-out data: 0.5228999853134155
Epoch: 8
Loss on hold-out set: 1.3404605934880673
Accuracy on hold-out set: 0.5318
MulticlassAccuracy value on hold-out data: 0.5317999720573425
Returned to Spot: Validation loss: 1.3404605934880673
----------------------------------------------
config: {'l1': 128, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 8, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.4604634573936461
Accuracy on hold-out set: 0.46545
MulticlassAccuracy value on hold-out data: 0.4654499888420105
Epoch: 2
Loss on hold-out set: 1.3089509534597397
Accuracy on hold-out set: 0.5315
MulticlassAccuracy value on hold-out data: 0.531499981880188
Epoch: 3
Loss on hold-out set: 1.2708063730597496
Accuracy on hold-out set: 0.5548
MulticlassAccuracy value on hold-out data: 0.5547999739646912
Epoch: 4
Loss on hold-out set: 1.2297203123867513
Accuracy on hold-out set: 0.57435
MulticlassAccuracy value on hold-out data: 0.5743499994277954
Epoch: 5
Loss on hold-out set: 1.1867765383303166
Accuracy on hold-out set: 0.58975
MulticlassAccuracy value on hold-out data: 0.5897499918937683
Epoch: 6
Loss on hold-out set: 1.2137519815444946
Accuracy on hold-out set: 0.58415
MulticlassAccuracy value on hold-out data: 0.5841500163078308
Epoch: 7
Loss on hold-out set: 1.24783908636868
Accuracy on hold-out set: 0.57795
MulticlassAccuracy value on hold-out data: 0.5779500007629395
Epoch: 8
Loss on hold-out set: 1.2409861245393754
Accuracy on hold-out set: 0.5781
MulticlassAccuracy value on hold-out data: 0.5781000256538391
Early stopping at epoch 7
Returned to Spot: Validation loss: 1.2409861245393754
----------------------------------------------
config: {'l1': 512, 'l2': 16, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'RMSprop', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 2.304224283838272
Accuracy on hold-out set: 0.10365
MulticlassAccuracy value on hold-out data: 0.10365000367164612
Epoch: 2
Loss on hold-out set: 2.3086710446357728
Accuracy on hold-out set: 0.0982
MulticlassAccuracy value on hold-out data: 0.0982000008225441
Epoch: 3
Loss on hold-out set: 2.3041055164337156
Accuracy on hold-out set: 0.09975
MulticlassAccuracy value on hold-out data: 0.09974999725818634
Epoch: 4
Loss on hold-out set: 2.3042114679813386
Accuracy on hold-out set: 0.10005
MulticlassAccuracy value on hold-out data: 0.10005000233650208
Epoch: 5
Loss on hold-out set: 2.3042567992210388
Accuracy on hold-out set: 0.10005
MulticlassAccuracy value on hold-out data: 0.10005000233650208
Epoch: 6
Loss on hold-out set: 2.305777049589157
Accuracy on hold-out set: 0.10005
MulticlassAccuracy value on hold-out data: 0.10005000233650208
Early stopping at epoch 5
Returned to Spot: Validation loss: 2.305777049589157
----------------------------------------------
config: {'l1': 8, 'l2': 8, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adagrad', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.8623228480339051
Accuracy on hold-out set: 0.285
MulticlassAccuracy value on hold-out data: 0.2849999964237213
Epoch: 2
Loss on hold-out set: 1.7958615369796753
Accuracy on hold-out set: 0.3275
MulticlassAccuracy value on hold-out data: 0.32749998569488525
Epoch: 3
Loss on hold-out set: 1.7516172283172606
Accuracy on hold-out set: 0.3393
MulticlassAccuracy value on hold-out data: 0.3393000066280365
Epoch: 4
Loss on hold-out set: 1.7269452848434448
Accuracy on hold-out set: 0.34945
MulticlassAccuracy value on hold-out data: 0.3494499921798706
Epoch: 5
Loss on hold-out set: 1.7147653727531433
Accuracy on hold-out set: 0.3551
MulticlassAccuracy value on hold-out data: 0.35510000586509705
Epoch: 6
Loss on hold-out set: 1.6960768105506896
Accuracy on hold-out set: 0.36215
MulticlassAccuracy value on hold-out data: 0.36215001344680786
Epoch: 7
Loss on hold-out set: 1.6849525526046754
Accuracy on hold-out set: 0.3691
MulticlassAccuracy value on hold-out data: 0.36910000443458557
Epoch: 8
Loss on hold-out set: 1.6722080268859862
Accuracy on hold-out set: 0.3775
MulticlassAccuracy value on hold-out data: 0.3774999976158142
Returned to Spot: Validation loss: 1.6722080268859862
----------------------------------------------
config: {'l1': 256, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'RMSprop', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 2.033727491092682
Accuracy on hold-out set: 0.23535
MulticlassAccuracy value on hold-out data: 0.23534999787807465
Epoch: 2
Loss on hold-out set: 1.9742233956336974
Accuracy on hold-out set: 0.25375
MulticlassAccuracy value on hold-out data: 0.2537499964237213
Epoch: 3
Loss on hold-out set: 2.0164827750205996
Accuracy on hold-out set: 0.247
MulticlassAccuracy value on hold-out data: 0.24699999392032623
Epoch: 4
Loss on hold-out set: 1.9548802864074708
Accuracy on hold-out set: 0.26835
MulticlassAccuracy value on hold-out data: 0.2683500051498413
Epoch: 5
Loss on hold-out set: 1.9147050152778626
Accuracy on hold-out set: 0.27635
MulticlassAccuracy value on hold-out data: 0.2763499915599823
Epoch: 6
Loss on hold-out set: 1.880613294696808
Accuracy on hold-out set: 0.27755
MulticlassAccuracy value on hold-out data: 0.27755001187324524
Epoch: 7
Loss on hold-out set: 1.8822319697380065
Accuracy on hold-out set: 0.28125
MulticlassAccuracy value on hold-out data: 0.28125
Epoch: 8
Loss on hold-out set: 1.9520383694648742
Accuracy on hold-out set: 0.28685
MulticlassAccuracy value on hold-out data: 0.2868500053882599
Returned to Spot: Validation loss: 1.9520383694648742
----------------------------------------------
config: {'l1': 256, 'l2': 512, 'lr_mult': 1.0, 'batch_size': 2, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.714820458814455
Accuracy on hold-out set: 0.4314
MulticlassAccuracy value on hold-out data: 0.43140000104904175
Epoch: 2
Loss on hold-out set: 1.6406829629886859
Accuracy on hold-out set: 0.47825
MulticlassAccuracy value on hold-out data: 0.47824999690055847
Epoch: 3
Loss on hold-out set: 1.545428125515567
Accuracy on hold-out set: 0.4975
MulticlassAccuracy value on hold-out data: 0.4975000023841858
Epoch: 4
Loss on hold-out set: 1.5244613147333956
Accuracy on hold-out set: 0.51565
MulticlassAccuracy value on hold-out data: 0.5156499743461609
Epoch: 5
Loss on hold-out set: 1.6418606465695462
Accuracy on hold-out set: 0.52855
MulticlassAccuracy value on hold-out data: 0.5285500288009644
Epoch: 6
Loss on hold-out set: 1.6219294878519634
Accuracy on hold-out set: 0.5234
MulticlassAccuracy value on hold-out data: 0.5234000086784363
Epoch: 7
Loss on hold-out set: 1.5469318324963246
Accuracy on hold-out set: 0.53335
MulticlassAccuracy value on hold-out data: 0.5333499908447266
Early stopping at epoch 6
Returned to Spot: Validation loss: 1.5469318324963246
----------------------------------------------
config: {'l1': 4, 'l2': 256, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adadelta', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.6368618047714234
Accuracy on hold-out set: 0.3795
MulticlassAccuracy value on hold-out data: 0.37950000166893005
Epoch: 2
Loss on hold-out set: 1.4890002604484558
Accuracy on hold-out set: 0.44565
MulticlassAccuracy value on hold-out data: 0.44565001130104065
Epoch: 3
Loss on hold-out set: 1.4198544501304626
Accuracy on hold-out set: 0.4787
MulticlassAccuracy value on hold-out data: 0.47870001196861267
Epoch: 4
Loss on hold-out set: 1.3945291288375854
Accuracy on hold-out set: 0.47445
MulticlassAccuracy value on hold-out data: 0.4744499921798706
Epoch: 5
Loss on hold-out set: 1.3828875508308411
Accuracy on hold-out set: 0.48545
MulticlassAccuracy value on hold-out data: 0.48544999957084656
Epoch: 6
Loss on hold-out set: 1.3790054025650025
Accuracy on hold-out set: 0.4917
MulticlassAccuracy value on hold-out data: 0.4916999936103821
Epoch: 7
Loss on hold-out set: 1.34652443151474
Accuracy on hold-out set: 0.5028
MulticlassAccuracy value on hold-out data: 0.5027999877929688
Epoch: 8
Loss on hold-out set: 1.4034742686271668
Accuracy on hold-out set: 0.48495
MulticlassAccuracy value on hold-out data: 0.4849500060081482
Epoch: 9
Loss on hold-out set: 1.342092141532898
Accuracy on hold-out set: 0.50445
MulticlassAccuracy value on hold-out data: 0.5044500231742859
Epoch: 10
Loss on hold-out set: 1.4018613757133485
Accuracy on hold-out set: 0.4942
MulticlassAccuracy value on hold-out data: 0.4941999912261963
Epoch: 11
Loss on hold-out set: 1.3043518905639648
Accuracy on hold-out set: 0.52165
MulticlassAccuracy value on hold-out data: 0.5216500163078308
Epoch: 12
Loss on hold-out set: 1.3959884852409363
Accuracy on hold-out set: 0.50015
MulticlassAccuracy value on hold-out data: 0.5001500248908997
Epoch: 13
Loss on hold-out set: 1.3304384766578674
Accuracy on hold-out set: 0.52045
MulticlassAccuracy value on hold-out data: 0.5204499959945679
Epoch: 14
Loss on hold-out set: 1.3407153537750245
Accuracy on hold-out set: 0.5124
MulticlassAccuracy value on hold-out data: 0.5123999714851379
Early stopping at epoch 13
Returned to Spot: Validation loss: 1.3407153537750245
----------------------------------------------
config: {'l1': 64, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5650458437919617
Accuracy on hold-out set: 0.41825
MulticlassAccuracy value on hold-out data: 0.4182499945163727
Epoch: 2
Loss on hold-out set: 1.445731195640564
Accuracy on hold-out set: 0.4699
MulticlassAccuracy value on hold-out data: 0.4699000120162964
Epoch: 3
Loss on hold-out set: 1.3417516043663025
Accuracy on hold-out set: 0.52355
MulticlassAccuracy value on hold-out data: 0.5235499739646912
Epoch: 4
Loss on hold-out set: 1.262966798520088
Accuracy on hold-out set: 0.55425
MulticlassAccuracy value on hold-out data: 0.5542500019073486
Epoch: 5
Loss on hold-out set: 1.2543963306427002
Accuracy on hold-out set: 0.5588
MulticlassAccuracy value on hold-out data: 0.5587999820709229
Epoch: 6
Loss on hold-out set: 1.2052456276416779
Accuracy on hold-out set: 0.57825
MulticlassAccuracy value on hold-out data: 0.578249990940094
Epoch: 7
Loss on hold-out set: 1.225034781074524
Accuracy on hold-out set: 0.5767
MulticlassAccuracy value on hold-out data: 0.57669997215271
Epoch: 8
Loss on hold-out set: 1.1629827931880952
Accuracy on hold-out set: 0.5963
MulticlassAccuracy value on hold-out data: 0.5963000059127808
Epoch: 9
Loss on hold-out set: 1.1475618918180466
Accuracy on hold-out set: 0.6036
MulticlassAccuracy value on hold-out data: 0.603600025177002
Epoch: 10
Loss on hold-out set: 1.1584332782506943
Accuracy on hold-out set: 0.60115
MulticlassAccuracy value on hold-out data: 0.6011499762535095
Epoch: 11
Loss on hold-out set: 1.1803555101156236
Accuracy on hold-out set: 0.59495
MulticlassAccuracy value on hold-out data: 0.5949500203132629
Epoch: 12
Loss on hold-out set: 1.19343827252388
Accuracy on hold-out set: 0.5925
MulticlassAccuracy value on hold-out data: 0.5924999713897705
Early stopping at epoch 11
Returned to Spot: Validation loss: 1.19343827252388
----------------------------------------------
spotPython tuning: 1.19343827252388 [#---------] 8.48%
config: {'l1': 64, 'l2': 16, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5412216246128083
Accuracy on hold-out set: 0.4335
MulticlassAccuracy value on hold-out data: 0.4334999918937683
Epoch: 2
Loss on hold-out set: 1.4014608226299285
Accuracy on hold-out set: 0.4952
MulticlassAccuracy value on hold-out data: 0.4952000081539154
Epoch: 3
Loss on hold-out set: 1.3415150342941284
Accuracy on hold-out set: 0.5222
MulticlassAccuracy value on hold-out data: 0.5221999883651733
Epoch: 4
Loss on hold-out set: 1.2791368619918824
Accuracy on hold-out set: 0.5424
MulticlassAccuracy value on hold-out data: 0.5424000024795532
Epoch: 5
Loss on hold-out set: 1.2638525828123093
Accuracy on hold-out set: 0.557
MulticlassAccuracy value on hold-out data: 0.5569999814033508
Epoch: 6
Loss on hold-out set: 1.2445933714866637
Accuracy on hold-out set: 0.5644
MulticlassAccuracy value on hold-out data: 0.5644000172615051
Epoch: 7
Loss on hold-out set: 1.1995793781757356
Accuracy on hold-out set: 0.57665
MulticlassAccuracy value on hold-out data: 0.5766500234603882
Epoch: 8
Loss on hold-out set: 1.1943189901113511
Accuracy on hold-out set: 0.58475
MulticlassAccuracy value on hold-out data: 0.5847499966621399
Epoch: 9
Loss on hold-out set: 1.1913710834741593
Accuracy on hold-out set: 0.5865
MulticlassAccuracy value on hold-out data: 0.5864999890327454
Epoch: 10
Loss on hold-out set: 1.1839374267339706
Accuracy on hold-out set: 0.5901
MulticlassAccuracy value on hold-out data: 0.5900999903678894
Epoch: 11
Loss on hold-out set: 1.1812278437137604
Accuracy on hold-out set: 0.5866
MulticlassAccuracy value on hold-out data: 0.5866000056266785
Epoch: 12
Loss on hold-out set: 1.2444978132724762
Accuracy on hold-out set: 0.58325
MulticlassAccuracy value on hold-out data: 0.5832499861717224
Epoch: 13
Loss on hold-out set: 1.1696878748416901
Accuracy on hold-out set: 0.59405
MulticlassAccuracy value on hold-out data: 0.5940499901771545
Epoch: 14
Loss on hold-out set: 1.2090726840496062
Accuracy on hold-out set: 0.5943
MulticlassAccuracy value on hold-out data: 0.5942999720573425
Epoch: 15
Loss on hold-out set: 1.2110995768785477
Accuracy on hold-out set: 0.59075
MulticlassAccuracy value on hold-out data: 0.590749979019165
Epoch: 16
Loss on hold-out set: 1.2590364922046662
Accuracy on hold-out set: 0.5813
MulticlassAccuracy value on hold-out data: 0.5813000202178955
Early stopping at epoch 15
Returned to Spot: Validation loss: 1.2590364922046662
----------------------------------------------
spotPython tuning: 1.19343827252388 [##--------] 19.93%
config: {'l1': 64, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.4611024236679078
Accuracy on hold-out set: 0.46365
MulticlassAccuracy value on hold-out data: 0.4636499881744385
Epoch: 2
Loss on hold-out set: 1.3886946356773378
Accuracy on hold-out set: 0.5037
MulticlassAccuracy value on hold-out data: 0.5037000179290771
Epoch: 3
Loss on hold-out set: 1.2549228223323823
Accuracy on hold-out set: 0.56355
MulticlassAccuracy value on hold-out data: 0.5635499954223633
Epoch: 4
Loss on hold-out set: 1.2044146399497986
Accuracy on hold-out set: 0.57655
MulticlassAccuracy value on hold-out data: 0.5765500068664551
Epoch: 5
Loss on hold-out set: 1.2458909042119979
Accuracy on hold-out set: 0.56375
MulticlassAccuracy value on hold-out data: 0.5637500286102295
Epoch: 6
Loss on hold-out set: 1.2286671370744706
Accuracy on hold-out set: 0.57875
MulticlassAccuracy value on hold-out data: 0.5787500143051147
Epoch: 7
Loss on hold-out set: 1.2020091131210326
Accuracy on hold-out set: 0.5877
MulticlassAccuracy value on hold-out data: 0.5877000093460083
Epoch: 8
Loss on hold-out set: 1.2225105068445206
Accuracy on hold-out set: 0.5869
MulticlassAccuracy value on hold-out data: 0.586899995803833
Epoch: 9
Loss on hold-out set: 1.2825752641677857
Accuracy on hold-out set: 0.56875
MulticlassAccuracy value on hold-out data: 0.5687500238418579
Epoch: 10
Loss on hold-out set: 1.2709592715501785
Accuracy on hold-out set: 0.58095
MulticlassAccuracy value on hold-out data: 0.5809500217437744
Early stopping at epoch 9
Returned to Spot: Validation loss: 1.2709592715501785
----------------------------------------------
spotPython tuning: 1.19343827252388 [###-------] 27.16%
config: {'l1': 64, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5346863508224486
Accuracy on hold-out set: 0.4341
MulticlassAccuracy value on hold-out data: 0.4341000020503998
Epoch: 2
Loss on hold-out set: 1.4146504063606262
Accuracy on hold-out set: 0.49215
MulticlassAccuracy value on hold-out data: 0.4921500086784363
Epoch: 3
Loss on hold-out set: 1.3082896121263503
Accuracy on hold-out set: 0.533
MulticlassAccuracy value on hold-out data: 0.5329999923706055
Epoch: 4
Loss on hold-out set: 1.210405622291565
Accuracy on hold-out set: 0.5708
MulticlassAccuracy value on hold-out data: 0.5708000063896179
Epoch: 5
Loss on hold-out set: 1.2332082440137864
Accuracy on hold-out set: 0.5648
MulticlassAccuracy value on hold-out data: 0.5648000240325928
Epoch: 6
Loss on hold-out set: 1.1999848236083985
Accuracy on hold-out set: 0.57735
MulticlassAccuracy value on hold-out data: 0.5773500204086304
Epoch: 7
Loss on hold-out set: 1.1633819435358048
Accuracy on hold-out set: 0.59525
MulticlassAccuracy value on hold-out data: 0.5952500104904175
Epoch: 8
Loss on hold-out set: 1.2047866644620895
Accuracy on hold-out set: 0.58395
MulticlassAccuracy value on hold-out data: 0.5839499831199646
Epoch: 9
Loss on hold-out set: 1.1512282137155534
Accuracy on hold-out set: 0.6
MulticlassAccuracy value on hold-out data: 0.6000000238418579
Epoch: 10
Loss on hold-out set: 1.1410926800966263
Accuracy on hold-out set: 0.60695
MulticlassAccuracy value on hold-out data: 0.6069499850273132
Epoch: 11
Loss on hold-out set: 1.178787278699875
Accuracy on hold-out set: 0.59355
MulticlassAccuracy value on hold-out data: 0.5935500264167786
Epoch: 12
Loss on hold-out set: 1.2072421681880952
Accuracy on hold-out set: 0.5944
MulticlassAccuracy value on hold-out data: 0.5943999886512756
Epoch: 13
Loss on hold-out set: 1.173101697587967
Accuracy on hold-out set: 0.60135
MulticlassAccuracy value on hold-out data: 0.6013500094413757
Early stopping at epoch 12
Returned to Spot: Validation loss: 1.173101697587967
----------------------------------------------
spotPython tuning: 1.173101697587967 [####------] 36.62%
config: {'l1': 4, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.6226150213241577
Accuracy on hold-out set: 0.39105
MulticlassAccuracy value on hold-out data: 0.3910500109195709
Epoch: 2
Loss on hold-out set: 1.5156013435363769
Accuracy on hold-out set: 0.4313
MulticlassAccuracy value on hold-out data: 0.43130001425743103
Epoch: 3
Loss on hold-out set: 1.4325028928756713
Accuracy on hold-out set: 0.463
MulticlassAccuracy value on hold-out data: 0.46299999952316284
Epoch: 4
Loss on hold-out set: 1.3903947087287902
Accuracy on hold-out set: 0.4792
MulticlassAccuracy value on hold-out data: 0.47920000553131104
Epoch: 5
Loss on hold-out set: 1.3774570764541625
Accuracy on hold-out set: 0.4875
MulticlassAccuracy value on hold-out data: 0.48750001192092896
Epoch: 6
Loss on hold-out set: 1.3664662029266357
Accuracy on hold-out set: 0.49485
MulticlassAccuracy value on hold-out data: 0.4948500096797943
Epoch: 7
Loss on hold-out set: 1.3878898769378663
Accuracy on hold-out set: 0.4792
MulticlassAccuracy value on hold-out data: 0.47920000553131104
Epoch: 8
Loss on hold-out set: 1.344132685279846
Accuracy on hold-out set: 0.49905
MulticlassAccuracy value on hold-out data: 0.49904999136924744
Epoch: 9
Loss on hold-out set: 1.3085883013725281
Accuracy on hold-out set: 0.51515
MulticlassAccuracy value on hold-out data: 0.5151500105857849
Epoch: 10
Loss on hold-out set: 1.3518012718200683
Accuracy on hold-out set: 0.49655
MulticlassAccuracy value on hold-out data: 0.4965499937534332
Epoch: 11
Loss on hold-out set: 1.303925577068329
Accuracy on hold-out set: 0.5153
MulticlassAccuracy value on hold-out data: 0.5152999758720398
Epoch: 12
Loss on hold-out set: 1.2804302824020386
Accuracy on hold-out set: 0.52605
MulticlassAccuracy value on hold-out data: 0.5260499715805054
Epoch: 13
Loss on hold-out set: 1.2829505469322204
Accuracy on hold-out set: 0.52715
MulticlassAccuracy value on hold-out data: 0.5271499752998352
Epoch: 14
Loss on hold-out set: 1.2925728231430054
Accuracy on hold-out set: 0.5252
MulticlassAccuracy value on hold-out data: 0.5252000093460083
Epoch: 15
Loss on hold-out set: 1.2516523637771606
Accuracy on hold-out set: 0.53405
MulticlassAccuracy value on hold-out data: 0.5340499877929688
Epoch: 16
Loss on hold-out set: 1.2586891078948974
Accuracy on hold-out set: 0.5389
MulticlassAccuracy value on hold-out data: 0.5389000177383423
Returned to Spot: Validation loss: 1.2586891078948974
----------------------------------------------
spotPython tuning: 1.173101697587967 [#####-----] 46.41%
config: {'l1': 512, 'l2': 512, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.3859094167709352
Accuracy on hold-out set: 0.4948
MulticlassAccuracy value on hold-out data: 0.49480000138282776
Epoch: 2
Loss on hold-out set: 1.3084599166870117
Accuracy on hold-out set: 0.53885
MulticlassAccuracy value on hold-out data: 0.5388500094413757
Epoch: 3
Loss on hold-out set: 1.2569380239248276
Accuracy on hold-out set: 0.56075
MulticlassAccuracy value on hold-out data: 0.5607500076293945
Epoch: 4
Loss on hold-out set: 1.2008638921260835
Accuracy on hold-out set: 0.58195
MulticlassAccuracy value on hold-out data: 0.5819500088691711
Epoch: 5
Loss on hold-out set: 1.1793278839111327
Accuracy on hold-out set: 0.5943
MulticlassAccuracy value on hold-out data: 0.5942999720573425
Epoch: 6
Loss on hold-out set: 1.1956181304454803
Accuracy on hold-out set: 0.609
MulticlassAccuracy value on hold-out data: 0.609000027179718
Epoch: 7
Loss on hold-out set: 1.2681621943235397
Accuracy on hold-out set: 0.59915
MulticlassAccuracy value on hold-out data: 0.5991500020027161
Epoch: 8
Loss on hold-out set: 1.4033361210107804
Accuracy on hold-out set: 0.5902
MulticlassAccuracy value on hold-out data: 0.5902000069618225
Early stopping at epoch 7
Returned to Spot: Validation loss: 1.4033361210107804
----------------------------------------------
spotPython tuning: 1.173101697587967 [#####-----] 53.37%
config: {'l1': 16, 'l2': 128, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.50163823056221
Accuracy on hold-out set: 0.45285
MulticlassAccuracy value on hold-out data: 0.45285001397132874
Epoch: 2
Loss on hold-out set: 1.4629781801223756
Accuracy on hold-out set: 0.4827
MulticlassAccuracy value on hold-out data: 0.482699990272522
Epoch: 3
Loss on hold-out set: 1.3330827356815338
Accuracy on hold-out set: 0.51915
MulticlassAccuracy value on hold-out data: 0.5191500186920166
Epoch: 4
Loss on hold-out set: 1.3042912577152252
Accuracy on hold-out set: 0.5294
MulticlassAccuracy value on hold-out data: 0.5293999910354614
Epoch: 5
Loss on hold-out set: 1.3167204762935638
Accuracy on hold-out set: 0.52805
MulticlassAccuracy value on hold-out data: 0.5280500054359436
Epoch: 6
Loss on hold-out set: 1.2452145859718322
Accuracy on hold-out set: 0.5545
MulticlassAccuracy value on hold-out data: 0.5544999837875366
Epoch: 7
Loss on hold-out set: 1.2154634838581084
Accuracy on hold-out set: 0.56645
MulticlassAccuracy value on hold-out data: 0.5664499998092651
Epoch: 8
Loss on hold-out set: 1.2069166351795197
Accuracy on hold-out set: 0.5699
MulticlassAccuracy value on hold-out data: 0.5698999762535095
Epoch: 9
Loss on hold-out set: 1.2354827906370163
Accuracy on hold-out set: 0.57335
MulticlassAccuracy value on hold-out data: 0.5733500123023987
Epoch: 10
Loss on hold-out set: 1.1957695059537887
Accuracy on hold-out set: 0.5793
MulticlassAccuracy value on hold-out data: 0.5792999863624573
Epoch: 11
Loss on hold-out set: 1.1772377661705018
Accuracy on hold-out set: 0.5871
MulticlassAccuracy value on hold-out data: 0.5871000289916992
Epoch: 12
Loss on hold-out set: 1.1675749730825424
Accuracy on hold-out set: 0.58645
MulticlassAccuracy value on hold-out data: 0.5864499807357788
Epoch: 13
Loss on hold-out set: 1.1668046678304673
Accuracy on hold-out set: 0.5931
MulticlassAccuracy value on hold-out data: 0.5931000113487244
Epoch: 14
Loss on hold-out set: 1.2392135566949845
Accuracy on hold-out set: 0.56945
MulticlassAccuracy value on hold-out data: 0.5694500207901001
Epoch: 15
Loss on hold-out set: 1.143719405269623
Accuracy on hold-out set: 0.597
MulticlassAccuracy value on hold-out data: 0.597000002861023
Epoch: 16
Loss on hold-out set: 1.1767093319892883
Accuracy on hold-out set: 0.58885
MulticlassAccuracy value on hold-out data: 0.5888500213623047
Returned to Spot: Validation loss: 1.1767093319892883
----------------------------------------------
spotPython tuning: 1.173101697587967 [######----] 64.66%
config: {'l1': 32, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5119601490974426
Accuracy on hold-out set: 0.4423
MulticlassAccuracy value on hold-out data: 0.4422999918460846
Epoch: 2
Loss on hold-out set: 1.3535101760864259
Accuracy on hold-out set: 0.51035
MulticlassAccuracy value on hold-out data: 0.5103499889373779
Epoch: 3
Loss on hold-out set: 1.2928748548507691
Accuracy on hold-out set: 0.5385
MulticlassAccuracy value on hold-out data: 0.5385000109672546
Epoch: 4
Loss on hold-out set: 1.2824455682992935
Accuracy on hold-out set: 0.5497
MulticlassAccuracy value on hold-out data: 0.5497000217437744
Epoch: 5
Loss on hold-out set: 1.208276400732994
Accuracy on hold-out set: 0.5728
MulticlassAccuracy value on hold-out data: 0.5727999806404114
Epoch: 6
Loss on hold-out set: 1.1729432007312774
Accuracy on hold-out set: 0.58785
MulticlassAccuracy value on hold-out data: 0.5878499746322632
Epoch: 7
Loss on hold-out set: 1.1892239879369735
Accuracy on hold-out set: 0.57775
MulticlassAccuracy value on hold-out data: 0.577750027179718
Epoch: 8
Loss on hold-out set: 1.164001667714119
Accuracy on hold-out set: 0.58995
MulticlassAccuracy value on hold-out data: 0.5899500250816345
Epoch: 9
Loss on hold-out set: 1.144117252755165
Accuracy on hold-out set: 0.5968
MulticlassAccuracy value on hold-out data: 0.5968000292778015
Epoch: 10
Loss on hold-out set: 1.1417374807834626
Accuracy on hold-out set: 0.6019
MulticlassAccuracy value on hold-out data: 0.6018999814987183
Epoch: 11
Loss on hold-out set: 1.1576101055979728
Accuracy on hold-out set: 0.59125
MulticlassAccuracy value on hold-out data: 0.5912500023841858
Epoch: 12
Loss on hold-out set: 1.1774953495502472
Accuracy on hold-out set: 0.593
MulticlassAccuracy value on hold-out data: 0.5929999947547913
Epoch: 13
Loss on hold-out set: 1.1590280144691467
Accuracy on hold-out set: 0.5987
MulticlassAccuracy value on hold-out data: 0.5986999869346619
Early stopping at epoch 12
Returned to Spot: Validation loss: 1.1590280144691467
----------------------------------------------
spotPython tuning: 1.1590280144691467 [#######---] 74.05%
config: {'l1': 32, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5969278493881225
Accuracy on hold-out set: 0.41915
MulticlassAccuracy value on hold-out data: 0.4191499948501587
Epoch: 2
Loss on hold-out set: 1.4429768196105957
Accuracy on hold-out set: 0.47605
MulticlassAccuracy value on hold-out data: 0.4760499894618988
Epoch: 3
Loss on hold-out set: 1.359341883277893
Accuracy on hold-out set: 0.5094
MulticlassAccuracy value on hold-out data: 0.5094000101089478
Epoch: 4
Loss on hold-out set: 1.3306642253875733
Accuracy on hold-out set: 0.52815
MulticlassAccuracy value on hold-out data: 0.5281500220298767
Epoch: 5
Loss on hold-out set: 1.29293729763031
Accuracy on hold-out set: 0.5385
MulticlassAccuracy value on hold-out data: 0.5385000109672546
Epoch: 6
Loss on hold-out set: 1.2112073586463927
Accuracy on hold-out set: 0.56945
MulticlassAccuracy value on hold-out data: 0.5694500207901001
Epoch: 7
Loss on hold-out set: 1.2407475979804992
Accuracy on hold-out set: 0.56155
MulticlassAccuracy value on hold-out data: 0.5615500211715698
Epoch: 8
Loss on hold-out set: 1.1786227509498597
Accuracy on hold-out set: 0.58535
MulticlassAccuracy value on hold-out data: 0.585349977016449
Epoch: 9
Loss on hold-out set: 1.1752745832443237
Accuracy on hold-out set: 0.5876
MulticlassAccuracy value on hold-out data: 0.5875999927520752
Epoch: 10
Loss on hold-out set: 1.1731052838802338
Accuracy on hold-out set: 0.59365
MulticlassAccuracy value on hold-out data: 0.5936499834060669
Epoch: 11
Loss on hold-out set: 1.1381336972236633
Accuracy on hold-out set: 0.6019
MulticlassAccuracy value on hold-out data: 0.6018999814987183
Epoch: 12
Loss on hold-out set: 1.159678161048889
Accuracy on hold-out set: 0.59665
MulticlassAccuracy value on hold-out data: 0.5966500043869019
Epoch: 13
Loss on hold-out set: 1.1422260280609131
Accuracy on hold-out set: 0.5971
MulticlassAccuracy value on hold-out data: 0.597100019454956
Epoch: 14
Loss on hold-out set: 1.1201010601997377
Accuracy on hold-out set: 0.61055
MulticlassAccuracy value on hold-out data: 0.6105499863624573
Epoch: 15
Loss on hold-out set: 1.1607105719089508
Accuracy on hold-out set: 0.5942
MulticlassAccuracy value on hold-out data: 0.5942000150680542
Epoch: 16
Loss on hold-out set: 1.1112337918281556
Accuracy on hold-out set: 0.61195
MulticlassAccuracy value on hold-out data: 0.6119499802589417
Returned to Spot: Validation loss: 1.1112337918281556
----------------------------------------------
spotPython tuning: 1.1112337918281556 [########--] 83.88%
config: {'l1': 32, 'l2': 128, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5478447868347167
Accuracy on hold-out set: 0.4323
MulticlassAccuracy value on hold-out data: 0.43230000138282776
Epoch: 2
Loss on hold-out set: 1.4247279371261596
Accuracy on hold-out set: 0.47955
MulticlassAccuracy value on hold-out data: 0.47955000400543213
Epoch: 3
Loss on hold-out set: 1.336058818912506
Accuracy on hold-out set: 0.5218
MulticlassAccuracy value on hold-out data: 0.5217999815940857
Epoch: 4
Loss on hold-out set: 1.2712731137275697
Accuracy on hold-out set: 0.55135
MulticlassAccuracy value on hold-out data: 0.5513499975204468
Epoch: 5
Loss on hold-out set: 1.2226607837677002
Accuracy on hold-out set: 0.56815
MulticlassAccuracy value on hold-out data: 0.568149983882904
Epoch: 6
Loss on hold-out set: 1.2355902368545533
Accuracy on hold-out set: 0.56365
MulticlassAccuracy value on hold-out data: 0.5636500120162964
Epoch: 7
Loss on hold-out set: 1.1664306200027466
Accuracy on hold-out set: 0.58865
MulticlassAccuracy value on hold-out data: 0.5886499881744385
Epoch: 8
Loss on hold-out set: 1.199628316783905
Accuracy on hold-out set: 0.5743
MulticlassAccuracy value on hold-out data: 0.5742999911308289
Epoch: 9
Loss on hold-out set: 1.1859040815353394
Accuracy on hold-out set: 0.58725
MulticlassAccuracy value on hold-out data: 0.5872499942779541
Epoch: 10
Loss on hold-out set: 1.1459507223129273
Accuracy on hold-out set: 0.6044
MulticlassAccuracy value on hold-out data: 0.6043999791145325
Epoch: 11
Loss on hold-out set: 1.1218714394569398
Accuracy on hold-out set: 0.6064
MulticlassAccuracy value on hold-out data: 0.6064000129699707
Epoch: 12
Loss on hold-out set: 1.1523362829208375
Accuracy on hold-out set: 0.60055
MulticlassAccuracy value on hold-out data: 0.6005499958992004
Epoch: 13
Loss on hold-out set: 1.11508677110672
Accuracy on hold-out set: 0.61255
MulticlassAccuracy value on hold-out data: 0.6125500202178955
Epoch: 14
Loss on hold-out set: 1.150922962474823
Accuracy on hold-out set: 0.603
MulticlassAccuracy value on hold-out data: 0.6029999852180481
Epoch: 15
Loss on hold-out set: 1.1595768379211426
Accuracy on hold-out set: 0.601
MulticlassAccuracy value on hold-out data: 0.6010000109672546
Epoch: 16
Loss on hold-out set: 1.1175544973373412
Accuracy on hold-out set: 0.6176
MulticlassAccuracy value on hold-out data: 0.6176000237464905
Early stopping at epoch 15
Returned to Spot: Validation loss: 1.1175544973373412
----------------------------------------------
spotPython tuning: 1.1112337918281556 [#########-] 93.81%
config: {'l1': 64, 'l2': 4, 'lr_mult': 1.0, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 2.3026317724227905
Accuracy on hold-out set: 0.1013
MulticlassAccuracy value on hold-out data: 0.10130000114440918
Epoch: 2
Loss on hold-out set: 2.3027376378059388
Accuracy on hold-out set: 0.09795
MulticlassAccuracy value on hold-out data: 0.09794999659061432
Epoch: 3
Loss on hold-out set: 2.3029168005943297
Accuracy on hold-out set: 0.09815
MulticlassAccuracy value on hold-out data: 0.09814999997615814
Epoch: 4
Loss on hold-out set: 2.3031554979324342
Accuracy on hold-out set: 0.0995
MulticlassAccuracy value on hold-out data: 0.09950000047683716
Early stopping at epoch 3
Returned to Spot: Validation loss: 2.3031554979324342
----------------------------------------------
spotPython tuning: 1.1112337918281556 [##########] 96.13%
config: {'l1': 8, 'l2': 512, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.443802201795578
Accuracy on hold-out set: 0.46585
MulticlassAccuracy value on hold-out data: 0.46584999561309814
Epoch: 2
Loss on hold-out set: 1.3793369000911713
Accuracy on hold-out set: 0.4983
MulticlassAccuracy value on hold-out data: 0.4982999861240387
Epoch: 3
Loss on hold-out set: 1.321274983072281
Accuracy on hold-out set: 0.51835
MulticlassAccuracy value on hold-out data: 0.5183500051498413
Epoch: 4
Loss on hold-out set: 1.3934255885124207
Accuracy on hold-out set: 0.50095
MulticlassAccuracy value on hold-out data: 0.5009499788284302
Epoch: 5
Loss on hold-out set: 1.3057110221385957
Accuracy on hold-out set: 0.5352
MulticlassAccuracy value on hold-out data: 0.5351999998092651
Epoch: 6
Loss on hold-out set: 1.2474714047431945
Accuracy on hold-out set: 0.55105
MulticlassAccuracy value on hold-out data: 0.5510500073432922
Epoch: 7
Loss on hold-out set: 1.2338773973941803
Accuracy on hold-out set: 0.5607
MulticlassAccuracy value on hold-out data: 0.560699999332428
Epoch: 8
Loss on hold-out set: 1.2474857745409011
Accuracy on hold-out set: 0.55915
MulticlassAccuracy value on hold-out data: 0.559149980545044
Epoch: 9
Loss on hold-out set: 1.2384405407428742
Accuracy on hold-out set: 0.56505
MulticlassAccuracy value on hold-out data: 0.5650500059127808
Epoch: 10
Loss on hold-out set: 1.2279422499895096
Accuracy on hold-out set: 0.5678
MulticlassAccuracy value on hold-out data: 0.567799985408783
Epoch: 11
Loss on hold-out set: 1.2342207060098649
Accuracy on hold-out set: 0.57065
MulticlassAccuracy value on hold-out data: 0.5706499814987183
Epoch: 12
Loss on hold-out set: 1.215912817621231
Accuracy on hold-out set: 0.56915
MulticlassAccuracy value on hold-out data: 0.5691499710083008
Epoch: 13
Loss on hold-out set: 1.2128731129407884
Accuracy on hold-out set: 0.5781
MulticlassAccuracy value on hold-out data: 0.5781000256538391
Epoch: 14
Loss on hold-out set: 1.2394779655933381
Accuracy on hold-out set: 0.57315
MulticlassAccuracy value on hold-out data: 0.5731499791145325
Epoch: 15
Loss on hold-out set: 1.256415672135353
Accuracy on hold-out set: 0.57335
MulticlassAccuracy value on hold-out data: 0.5733500123023987
Epoch: 16
Loss on hold-out set: 1.2154606033802033
Accuracy on hold-out set: 0.57725
MulticlassAccuracy value on hold-out data: 0.5772500038146973
Early stopping at epoch 15
Returned to Spot: Validation loss: 1.2154606033802033
----------------------------------------------
spotPython tuning: 1.1112337918281556 [##########] 100.00% Done...
<spotPython.spot.spot.Spot at 0x28e513460>
The textual output shown in the console (or code cell) can be visualized with Tensorboard.
Start TensorBoard through the command line to visualize data you logged. Specify the root log directory as used in fun_control = fun_control_init(task="regression", tensorboard_path="runs/24_spot_torch_regression")
as the tensorboard_path
. The argument logdir points to directory where TensorBoard will look to find event files that it can display. TensorBoard will recursively walk the directory structure rooted at logdir, looking for .tfevents. files.
tensorboard --logdir=runs
Go to the URL it provides or to http://localhost:6006/. The following figures show some screenshots of Tensorboard.
The state of the notebook can be saved and reloaded as follows:
import pickle
= False
SAVE = False
LOAD
if SAVE:
= "res_" + experiment_name + ".pkl"
result_file_name with open(result_file_name, 'wb') as f:
pickle.dump(spot_tuner, f)
if LOAD:
= "add_the_name_of_the_result_file_here.pkl"
result_file_name with open(result_file_name, 'rb') as f:
= pickle.load(f) spot_tuner
After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The following code generates the progress plot from ?fig-progress.
=False,
spot_tuner.plot_progress(log_y="./figures/" + experiment_name+"_progress.png") filename
?fig-progress shows a typical behaviour that can be observed in many hyperparameter studies (Bartz et al. 2022): the largest improvement is obtained during the evaluation of the initial design. The surrogate model based optimization-optimization with the surrogate refines the results. ?fig-progress also illustrates one major difference between ray[tune]
as used in PyTorch (2023a) and spotPython
: the ray[tune]
uses a random search and will generate results similar to the black dots, whereas spotPython
uses a surrogate model based optimization and presents results represented by red dots in ?fig-progress. The surrogate model based optimization is considered to be more efficient than a random search, because the surrogate model guides the search towards promising regions in the hyperparameter space.
In addition to the improved (“optimized”) hyperparameter values, spotPython
allows a statistical analysis, e.g., a sensitivity analysis, of the results. We can print the results of the hyperparameter tuning, see ?tbl-results. The table shows the hyperparameters, their types, default values, lower and upper bounds, and the transformation function. The column “tuned” shows the tuned values. The column “importance” shows the importance of the hyperparameters. The column “stars” shows the importance of the hyperparameters in stars. The importance is computed by the SPOT software.
from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control=fun_control, spot=spot_tuner))
| name | type | default | lower | upper | tuned | transform | importance | stars |
|--------------|--------|-----------|---------|---------|---------|-----------------------|--------------|---------|
| l1 | int | 5 | 2.0 | 9.0 | 5.0 | transform_power_2_int | 5.62 | * |
| l2 | int | 5 | 2.0 | 9.0 | 6.0 | transform_power_2_int | 6.13 | * |
| lr_mult | float | 1.0 | 1.0 | 1.0 | 1.0 | None | 0.00 | |
| batch_size | int | 4 | 1.0 | 5.0 | 5.0 | transform_power_2_int | 0.63 | . |
| epochs | int | 3 | 3.0 | 4.0 | 4.0 | transform_power_2_int | 100.00 | *** |
| k_folds | int | 1 | 0.0 | 0.0 | 0.0 | None | 0.00 | |
| patience | int | 5 | 3.0 | 3.0 | 3.0 | None | 0.00 | |
| optimizer | factor | SGD | 0.0 | 9.0 | 3.0 | None | 0.05 | |
| sgd_momentum | float | 0.0 | 0.9 | 0.9 | 0.9 | None | 0.00 | |
To visualize the most important hyperparameters, spotPython
provides the function plot_importance
. The following code generates the importance plot from ?fig-importance.
=0.025,
spot_tuner.plot_importance(threshold="./figures/" + experiment_name+"_importance.png") filename
The architecture of the spotPython
model can be obtained as follows. First, the numerical representation of the hyperparameters are obtained, i.e., the numpy array X
is generated. This array is then used to generate the model model_spot
by the function get_one_core_model_from_X
. The model model_spot
has the following architecture:
from spotPython.hyperparameters.values import get_one_core_model_from_X
= spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
X = get_one_core_model_from_X(X, fun_control)
model_spot model_spot
Net_CIFAR10(
(conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
(pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
(fc1): Linear(in_features=400, out_features=32, bias=True)
(fc2): Linear(in_features=32, out_features=64, bias=True)
(fc3): Linear(in_features=64, out_features=10, bias=True)
)
In a similar manner as in ?sec-get-spot-results, the default hyperparameters can be obtained.
# fun_control was modified, we generate a new one with the original
# default hyperparameters
from spotPython.hyperparameters.values import get_one_core_model_from_X
= fun_control
fc "core_model_hyper_dict":
fc.update({"core_model"].__name__]})
hyper_dict[fun_control[= get_one_core_model_from_X(X_start, fun_control=fc)
model_default model_default
Net_CIFAR10(
(conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
(pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
(fc1): Linear(in_features=400, out_features=32, bias=True)
(fc2): Linear(in_features=32, out_features=32, bias=True)
(fc3): Linear(in_features=32, out_features=10, bias=True)
)
The method train_tuned
takes a model architecture without trained weights and trains this model with the train data. The train data is split into train and validation data. The validation data is used for early stopping. The trained model weights are saved as a dictionary.
This evaluation is similar to the final evaluation in PyTorch (2023a).
from spotPython.torch.traintest import (
train_tuned,
test_tuned,
)=model_default, train_dataset=train, shuffle=True,
train_tuned(net=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric= fun_control["device"], show_batch_interval=1_000_000,
device =None,
path=fun_control["task"],)
task
=model_default, test_dataset=test,
test_tuned(net=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric=False,
shuffle= fun_control["device"],
device =fun_control["task"],) task
Epoch: 1
Loss on hold-out set: 2.3059939056396486
Accuracy on hold-out set: 0.1143
MulticlassAccuracy value on hold-out data: 0.11429999768733978
Epoch: 2
Loss on hold-out set: 2.3010946350097656
Accuracy on hold-out set: 0.1101
MulticlassAccuracy value on hold-out data: 0.11010000109672546
Epoch: 3
Loss on hold-out set: 2.292931368255615
Accuracy on hold-out set: 0.11565
MulticlassAccuracy value on hold-out data: 0.1156499981880188
Epoch: 4
Loss on hold-out set: 2.2728519828796387
Accuracy on hold-out set: 0.14455
MulticlassAccuracy value on hold-out data: 0.14454999566078186
Epoch: 5
Loss on hold-out set: 2.2270529048919676
Accuracy on hold-out set: 0.147
MulticlassAccuracy value on hold-out data: 0.1469999998807907
Epoch: 6
Loss on hold-out set: 2.1787823093414307
Accuracy on hold-out set: 0.17165
MulticlassAccuracy value on hold-out data: 0.1716500073671341
Epoch: 7
Loss on hold-out set: 2.143781774520874
Accuracy on hold-out set: 0.20205
MulticlassAccuracy value on hold-out data: 0.20205000042915344
Epoch: 8
Loss on hold-out set: 2.1199397923469543
Accuracy on hold-out set: 0.21655
MulticlassAccuracy value on hold-out data: 0.21654999256134033
Returned to Spot: Validation loss: 2.1199397923469543
----------------------------------------------
Loss on hold-out set: 2.1146215215682984
Accuracy on hold-out set: 0.22
MulticlassAccuracy value on hold-out data: 0.2199999988079071
Final evaluation: Validation loss: 2.1146215215682984
Final evaluation: Validation metric: 0.2199999988079071
----------------------------------------------
(2.1146215215682984, nan, tensor(0.2200))
The following code trains the model model_spot
.
If path
is set to a filename, e.g., path = "model_spot_trained.pt"
, the weights of the trained model will be saved to this file.
If path
is set to a filename, e.g., path = "model_spot_trained.pt"
, the weights of the trained model will be loaded from this file.
=model_spot, train_dataset=train,
train_tuned(net=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric=True,
shuffle= fun_control["device"],
device =None,
path=fun_control["task"],)
task=model_spot, test_dataset=test,
test_tuned(net=False,
shuffle=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric= fun_control["device"],
device =fun_control["task"],) task
Epoch: 1
Loss on hold-out set: 1.5367841676712035
Accuracy on hold-out set: 0.4464
MulticlassAccuracy value on hold-out data: 0.446399986743927
Epoch: 2
Loss on hold-out set: 1.4369130460739137
Accuracy on hold-out set: 0.48825
MulticlassAccuracy value on hold-out data: 0.4882499873638153
Epoch: 3
Loss on hold-out set: 1.322485485935211
Accuracy on hold-out set: 0.5274
MulticlassAccuracy value on hold-out data: 0.527400016784668
Epoch: 4
Loss on hold-out set: 1.2750712456703186
Accuracy on hold-out set: 0.54975
MulticlassAccuracy value on hold-out data: 0.5497499704360962
Epoch: 5
Loss on hold-out set: 1.278616949081421
Accuracy on hold-out set: 0.54925
MulticlassAccuracy value on hold-out data: 0.5492500066757202
Epoch: 6
Loss on hold-out set: 1.2618403136253358
Accuracy on hold-out set: 0.55655
MulticlassAccuracy value on hold-out data: 0.5565500259399414
Epoch: 7
Loss on hold-out set: 1.218701737689972
Accuracy on hold-out set: 0.5767
MulticlassAccuracy value on hold-out data: 0.57669997215271
Epoch: 8
Loss on hold-out set: 1.1871091922760009
Accuracy on hold-out set: 0.58135
MulticlassAccuracy value on hold-out data: 0.5813500285148621
Epoch: 9
Loss on hold-out set: 1.211964674282074
Accuracy on hold-out set: 0.5793
MulticlassAccuracy value on hold-out data: 0.5792999863624573
Epoch: 10
Loss on hold-out set: 1.1984187335968017
Accuracy on hold-out set: 0.5831
MulticlassAccuracy value on hold-out data: 0.5831000208854675
Epoch: 11
Loss on hold-out set: 1.168288896369934
Accuracy on hold-out set: 0.5915
MulticlassAccuracy value on hold-out data: 0.5914999842643738
Epoch: 12
Loss on hold-out set: 1.1712540884971618
Accuracy on hold-out set: 0.58955
MulticlassAccuracy value on hold-out data: 0.5895500183105469
Epoch: 13
Loss on hold-out set: 1.1461831785202026
Accuracy on hold-out set: 0.59495
MulticlassAccuracy value on hold-out data: 0.5949500203132629
Epoch: 14
Loss on hold-out set: 1.136024131679535
Accuracy on hold-out set: 0.6035
MulticlassAccuracy value on hold-out data: 0.6035000085830688
Epoch: 15
Loss on hold-out set: 1.1744485748291016
Accuracy on hold-out set: 0.59685
MulticlassAccuracy value on hold-out data: 0.5968499779701233
Epoch: 16
Loss on hold-out set: 1.151477359199524
Accuracy on hold-out set: 0.5935
MulticlassAccuracy value on hold-out data: 0.593500018119812
Returned to Spot: Validation loss: 1.151477359199524
----------------------------------------------
Loss on hold-out set: 1.1276507280505121
Accuracy on hold-out set: 0.6006
MulticlassAccuracy value on hold-out data: 0.600600004196167
Final evaluation: Validation loss: 1.1276507280505121
Final evaluation: Validation metric: 0.600600004196167
----------------------------------------------
(1.1276507280505121, nan, tensor(0.6006))
The contour plots in this section visualize the interactions of the three most important hyperparameters. Since some of these hyperparameters take fatorial or integer values, sometimes step-like fitness landcapes (or response surfaces) are generated. SPOT draws the interactions of the main hyperparameters by default. It is also possible to visualize all interactions.
= "./figures/" + experiment_name
filename =filename) spot_tuner.plot_important_hyperparameter_contour(filename
l1: 5.61786788811698
l2: 6.134164172253628
batch_size: 0.6312892162051759
epochs: 100.0
optimizer: 0.048227370836895544
The figures (?fig-contour) show the contour plots of the loss as a function of the hyperparameters. These plots are very helpful for benchmark studies and for understanding neural networks. spotPython
provides additional tools for a visual inspection of the results and give valuable insights into the hyperparameter tuning process. This is especially useful for model explainability, transparency, and trustworthiness. In addition to the contour plots, ?fig-parallel shows the parallel plot of the hyperparameters.
spot_tuner.parallel_plot()
Parallel coordinates plots
This tutorial presents the hyperparameter tuning open source software spotPython
for PyTorch
. To show its basic features, a comparison with the “official” PyTorch
hyperparameter tuning tutorial (PyTorch 2023a) is presented. Some of the advantages of spotPython
are:
tensorboard
.Currently, only rudimentary parallel and distributed neural network training is possible, but these capabilities will be extended in the future. The next version of spotPython
will also include a more detailed documentation and more examples.
Important: This tutorial does not present a complete benchmarking study (Bartz-Beielstein et al. 2020). The results are only preliminary and highly dependent on the local configuration (hard- and software). Our goal is to provide a first impression of the performance of the hyperparameter tuning package spotPython
. To demonstrate its capabilities, a quick comparison with ray[tune]
was performed. ray[tune]
was chosen, because it is presented as “an industry standard tool for distributed hyperparameter tuning.” The results should be interpreted with care.
The output from ray[tune]
could look like this (PyTorch 2023b):
Number of trials: 10 (10 TERMINATED)
------+------+-------------+--------------+---------+------------+--------------------+
| l1 | l2 | lr | batch_size | loss | accuracy | training_iteration |
+------+------+-------------+--------------+---------+------------+--------------------|
| 64 | 4 | 0.00011629 | 2 | 1.87273 | 0.244 | 2 |
| 32 | 64 | 0.000339763 | 8 | 1.23603 | 0.567 | 8 |
| 8 | 16 | 0.00276249 | 16 | 1.1815 | 0.5836 | 10 |
| 4 | 64 | 0.000648721 | 4 | 1.31131 | 0.5224 | 8 |
| 32 | 16 | 0.000340753 | 8 | 1.26454 | 0.5444 | 8 |
| 8 | 4 | 0.000699775 | 8 | 1.99594 | 0.1983 | 2 |
| 256 | 8 | 0.0839654 | 16 | 2.3119 | 0.0993 | 1 |
| 16 | 128 | 0.0758154 | 16 | 2.33575 | 0.1327 | 1 |
| 16 | 8 | 0.0763312 | 16 | 2.31129 | 0.1042 | 4 |
| 128 | 16 | 0.000124903 | 4 | 2.26917 | 0.1945 | 1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+
Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806
Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.↩︎
We were not able to install Ray Tune
on our system. Therefore, we used the results from the PyTorch
tutorial.↩︎