14  HPT: PyTorch With spotPython and Ray Tune on CIFAR10

In this tutorial, we will show how spotPython can be integrated into the PyTorch training workflow. It is based on the tutorial “Hyperparameter Tuning with Ray Tune” from the PyTorch documentation (PyTorch 2023a), which is an extension of the tutorial “Training a Classifier” (PyTorch 2023b) for training a CIFAR10 image classifier.

This document refers to the following software versions:

pip list | grep  "spot[RiverPython]"
spotPython                 0.2.34
spotRiver                  0.0.93
Note: you may need to restart the kernel to use updated packages.

spotPython can be installed via pip1.

!pip install spotPython
# import sys
# !{sys.executable} -m pip install --upgrade build
# !{sys.executable} -m pip install --upgrade --force-reinstall spotPython

Results that refer to the Ray Tune package are taken from https://PyTorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html2.

14.1 Setup

Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.

MAX_TIME = 30
INIT_SIZE = 10
DEVICE = "cpu" # "cuda:0"
from spotPython.utils.device import getDevice
DEVICE = getDevice(DEVICE)
print(DEVICE)
cpu
import os
import copy
import socket
import warnings
from datetime import datetime
from dateutil.tz import tzlocal
start_time = datetime.now(tzlocal())
HOSTNAME = socket.gethostname().split(".")[0]
experiment_name = '14-torch' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
experiment_name = experiment_name.replace(':', '-')
print(experiment_name)
if not os.path.exists('./figures'):
    os.makedirs('./figures')
warnings.filterwarnings("ignore")
14-torch_p040025_30min_10init_2023-06-17_15-09-43

14.2 Initialization of the fun_control Dictionary

spotPython uses a Python dictionary for storing the information required for the hyperparameter tuning process. This dictionary is called fun_control and is initialized with the function fun_control_init. The function fun_control_init returns a skeleton dictionary. The dictionary is filled with the required information for the hyperparameter tuning process. It stores the hyperparameter tuning settings, e.g., the deep learning network architecture that should be tuned, the classification (or regression) problem, and the data that is used for the tuning. The dictionary is used as an input for the SPOT function.

from spotPython.utils.init import fun_control_init
fun_control = fun_control_init(task="classification",
    tensorboard_path="runs/14_spot_ray_hpt_torch_cifar10",
    device=DEVICE,)

14.3 PyTorch Data Loading

The data loading process is implemented in the same manner as described in the Section “Data loaders” in PyTorch (2023a). The data loaders are wrapped into the function load_data_cifar10 which is identical to the function load_data in PyTorch (2023a). A global data directory is used, which allows sharing the data directory between different trials. The method load_data_cifar10 is part of the spotPython package and can be imported from spotPython.data.torchdata.

In the following step, the test and train data are added to the dictionary fun_control.

from spotPython.data.torchdata import load_data_cifar10
train, test = load_data_cifar10()
n_samples = len(train)
# add the dataset to the fun_control
fun_control.update({
    "train": train,
    "test": test,
    "n_samples": n_samples})
Files already downloaded and verified
Files already downloaded and verified

14.4 The Model (Algorithm) to be Tuned

14.4.1 Specification of the Preprocessing Model

After the training and test data are specified and added to the fun_control dictionary, spotPython allows the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables. The preprocessing model is called prep_model (“preparation” or pre-processing) and includes steps that are not subject to the hyperparameter tuning process. The preprocessing model is specified in the fun_control dictionary. The preprocessing model can be implemented as a sklearn pipeline. The following code shows a typical preprocessing pipeline:

categorical_columns = ["cities", "colors"]
one_hot_encoder = OneHotEncoder(handle_unknown="ignore",
                                    sparse_output=False)
prep_model = ColumnTransformer(
        transformers=[
             ("categorical", one_hot_encoder, categorical_columns),
         ],
         remainder=StandardScaler(),
     )

Because the Ray Tune (ray[tune]) hyperparameter tuning as described in PyTorch (2023a) does not use a preprocessing model, the preprocessing model is set to None here.

prep_model = None
fun_control.update({"prep_model": prep_model})

14.4.2 Select algorithm and core_model_hyper_dict

The same neural network model as implemented in the section “Configurable neural network” of the PyTorch tutorial (PyTorch 2023a) is used here. We will show the implementation from PyTorch (2023a) in Section 14.4.2.1 first, before the extended implementation with spotPython is shown in Section 14.4.2.2.

14.4.2.1 Implementing a Configurable Neural Network With Ray Tune

We used the same hyperparameters that are implemented as configurable in the PyTorch tutorial. We specify the layer sizes, namely l1 and l2, of the fully connected layers:

class Net(nn.Module):
    def __init__(self, l1=120, l2=84):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The learning rate, i.e., lr, of the optimizer is made configurable, too:

optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

14.4.2.2 Implementing a Configurable Neural Network With spotPython

spotPython implements a class which is similar to the class described in the PyTorch tutorial. The class is called Net_CIFAR10 and is implemented in the file netcifar10.py.

from torch import nn
import torch.nn.functional as F
import spotPython.torch.netcore as netcore


class Net_CIFAR10(netcore.Net_Core):
    def __init__(self, l1, l2, lr_mult, batch_size, epochs, k_folds, patience,
    optimizer, sgd_momentum):
        super(Net_CIFAR10, self).__init__(
            lr_mult=lr_mult,
            batch_size=batch_size,
            epochs=epochs,
            k_folds=k_folds,
            patience=patience,
            optimizer=optimizer,
            sgd_momentum=sgd_momentum,
        )
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

14.4.3 The Net_Core class

Net_CIFAR10 inherits from the class Net_Core which is implemented in the file netcore.py. It implements the additional attributes that are common to all neural network models. The Net_Core class is implemented in the file netcore.py. It implements hyperparameters as attributes, that are not used by the core_model, e.g.:

  • optimizer (optimizer),
  • learning rate (lr),
  • batch size (batch_size),
  • epochs (epochs),
  • k_folds (k_folds), and
  • early stopping criterion “patience” (patience).

Users can add further attributes to the class. The class Net_Core is shown below.

from torch import nn


class Net_Core(nn.Module):
    def __init__(self, lr_mult, batch_size, epochs, k_folds, patience,
        optimizer, sgd_momentum):
        super(Net_Core, self).__init__()
        self.lr_mult = lr_mult
        self.batch_size = batch_size
        self.epochs = epochs
        self.k_folds = k_folds
        self.patience = patience
        self.optimizer = optimizer
        self.sgd_momentum = sgd_momentum

14.4.4 Comparison of the Approach Described in the PyTorch Tutorial With spotPython

Comparing the class Net from the PyTorch tutorial and the class Net_CIFAR10 from spotPython, we see that the class Net_CIFAR10 has additional attributes and does not inherit from nn directly. It adds an additional class, Net_core, that takes care of additional attributes that are common to all neural network models, e.g., the learning rate multiplier lr_mult or the batch size batch_size.

spotPython’s core_model implements an instance of the Net_CIFAR10 class. In addition to the basic neural network model, the core_model can use these additional attributes. spotPython provides methods for handling these additional attributes to guarantee 100% compatibility with the PyTorch classes. The method add_core_model_to_fun_control adds the hyperparameters and additional attributes to the fun_control dictionary. The method is shown below.

from spotPython.torch.netcifar10 import Net_CIFAR10
from spotPython.data.torch_hyper_dict import TorchHyperDict
from spotPython.hyperparameters.values import add_core_model_to_fun_control
core_model = Net_CIFAR10
fun_control = add_core_model_to_fun_control(core_model=core_model,
                              fun_control=fun_control,
                              hyper_dict=TorchHyperDict,
                              filename=None)

14.5 The Search Space: Hyperparameters

In Section 14.5.1, we first describe how to configure the search space with ray[tune] (as shown in PyTorch (2023a)) and then how to configure the search space with spotPython in -14.

14.5.1 Configuring the Search Space With Ray Tune

Ray Tune’s search space can be configured as follows (PyTorch 2023a):

config = {
    "l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "lr": tune.loguniform(1e-4, 1e-1),
    "batch_size": tune.choice([2, 4, 8, 16])
}

The tune.sample_from() function enables the user to define sample methods to obtain hyperparameters. In this example, the l1 and l2 parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.

At each trial, ray[tune] will randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. ray[tune] uses the ASHAScheduler which will terminate bad performing trials early.

14.5.2 Configuring the Search Space With spotPython

14.5.2.1 The hyper_dict Hyperparameters for the Selected Algorithm

spotPython uses JSON files for the specification of the hyperparameters. Users can specify their individual JSON files, or they can use the JSON files provided by spotPython. The JSON file for the core_model is called torch_hyper_dict.json.

In contrast to ray[tune], spotPython can handle numerical, boolean, and categorical hyperparameters. They can be specified in the JSON file in a similar way as the numerical hyperparameters as shown below. Each entry in the JSON file represents one hyperparameter with the following structure: type, default, transform, lower, and upper.

"factor_hyperparameter": {
    "levels": ["A", "B", "C"],
    "type": "factor",
    "default": "B",
    "transform": "None",
    "core_model_parameter_type": "str",
    "lower": 0,
    "upper": 2},

The corresponding entries for the Net_CIFAR10 class are shown below.

{"Net_CIFAR10":
    {
        "l1": {
            "type": "int",
            "default": 5,
            "transform": "transform_power_2_int",
            "lower": 2,
            "upper": 9},
        "l2": {
            "type": "int",
            "default": 5,
            "transform": "transform_power_2_int",
            "lower": 2,
            "upper": 9},
        "lr_mult": {
            "type": "float",
            "default": 1.0,
            "transform": "None",
            "lower": 0.1,
            "upper": 10},
        "batch_size": {
            "type": "int",
            "default": 4,
            "transform": "transform_power_2_int",
            "lower": 1,
            "upper": 4},
        "epochs": {
            "type": "int",
            "default": 3,
            "transform": "transform_power_2_int",
            "lower": 1,
            "upper": 4},
        "k_folds": {
            "type": "int",
            "default": 2,
            "transform": "None",
            "lower": 2,
            "upper": 3},
        "patience": {
            "type": "int",
            "default": 5,
            "transform": "None",
            "lower": 2,
            "upper": 10},
        "optimizer": {
            "levels": ["Adadelta",
                       "Adagrad",
                       "Adam",
                       "AdamW",
                       "SparseAdam",
                       "Adamax",
                       "ASGD",
                       "LBFGS",
                       "NAdam",
                       "RAdam",
                       "RMSprop",
                       "Rprop",
                       "SGD"],
            "type": "factor",
            "default": "SGD",
            "transform": "None",
            "class_name": "torch.optim",
            "core_model_parameter_type": "str",
            "lower": 0,
            "upper": 12},
        "sgd_momentum": {
            "type": "float",
            "default": 0.0,
            "transform": "None",
            "lower": 0.0,
            "upper": 1.0}
    }
}

14.5.3 Modifying the Hyperparameters

Ray tune (PyTorch 2023a) does not provide a way to change the specified hyperparameters without re-compilation. However, spotPython provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions are described in the following.

14.5.3.1 Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model

After specifying the model, the corresponding hyperparameters, their types and bounds are loaded from the JSON file torch_hyper_dict.json. After loading, the user can modify the hyperparameters, e.g., the bounds. spotPython provides a simple rule for de-activating hyperparameters: If the lower and the upper bound are set to identical values, the hyperparameter is de-activated. This is useful for the hyperparameter tuning, because it allows to specify a hyperparameter in the JSON file, but to de-activate it in the fun_control dictionary. This is done in the next step.

14.5.3.2 Modify Hyperparameters of Type numeric and integer (boolean)

Since the hyperparameter k_folds is not used in the PyTorch tutorial, it is de-activated here by setting the lower and upper bound to the same value. Note, k_folds is of type “integer”.

from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
fun_control = modify_hyper_parameter_bounds(fun_control, 
    "batch_size", bounds=[1, 5])
fun_control = modify_hyper_parameter_bounds(fun_control, 
    "k_folds", bounds=[0, 0])
fun_control = modify_hyper_parameter_bounds(fun_control, 
    "patience", bounds=[3, 3])

14.5.3.3 Modify Hyperparameter of Type factor

In a similar manner as for the numerical hyperparameters, the categorical hyperparameters can be modified. New configurations can be chosen by adding or deleting levels. For example, the hyperparameter optimizer can be re-configured as follows:

In the following setting, two optimizers ("SGD" and "Adam") will be compared during the spotPython hyperparameter tuning. The hyperparameter optimizer is active.

from spotPython.hyperparameters.values import modify_hyper_parameter_levels
fun_control = modify_hyper_parameter_levels(fun_control,
     "optimizer", ["SGD", "Adam"])

The hyperparameter optimizer can be de-activated by choosing only one value (level), here: "SGD".

fun_control = modify_hyper_parameter_levels(fun_control, "optimizer", ["SGD"])

As discussed in Section 14.6, there are some issues with the LBFGS optimizer. Therefore, the usage of the LBFGS optimizer is not deactivated in spotPython by default. However, the LBFGS optimizer can be activated by adding it to the list of optimizers. Rprop was removed, because it does perform very poorly (as some pre-tests have shown). However, it can also be activated by adding it to the list of optimizers. Since SparseAdam does not support dense gradients, Adam was used instead. Therefore, there are 10 default optimizers:

fun_control = modify_hyper_parameter_levels(fun_control, "optimizer",
    ["Adadelta", "Adagrad", "Adam", "AdamW", "Adamax", "ASGD", 
    "NAdam", "RAdam", "RMSprop", "SGD"])

14.6 Optimizers

Table 14.1 shows some of the optimizers available in PyTorch:

\(a\) denotes (0.9,0.999), \(b\) (0.5,1.2), and \(c\) (1e-6, 50), respectively. \(R\) denotes required, but unspecified. “m” denotes momentum, “w_d” weight_decay, “d” dampening, “n” nesterov, “r” rho, “l_s” learning rate for scaling delta, “l_d” lr_decay, “b” betas, “l” lambd, “a” alpha, “m_d” for momentum_decay, “e” etas, and “s_s” for step_sizes.

Table 14.1: Optimizers available in PyTorch (selection). The default values are shown in the table.
Optimizer lr m w_d d n r l_s l_d b l a m_d e s_s
Adadelta - - 0. - - 0.9 1. - - - - - - -
Adagrad 1e-2 - 0. - - - - 0. - - - - - -
Adam 1e-3 - 0. - - - - - \(a\) - - - - -
AdamW 1e-3 - 1e-2 - - - - - \(a\) - - - - -
SparseAdam 1e-3 - - - - - - - \(a\) - - - - -
Adamax 2e-3 - 0. - - - - - \(a\) - - - - -
ASGD 1e-2 .9 0. - F - - - - 1e-4 .75 - - -
LBFGS 1. - - - - - - - - - - - - -
NAdam 2e-3 - 0. - - - - - \(a\) - - 0 - -
RAdam 1e-3 - 0. - - - - - \(a\) - - - - -
RMSprop 1e-2 0. 0. - - - - - \(a\) - - - - -
Rprop 1e-2 - - - - - - - - - \(b\) \(c\) - -
SGD \(R\) 0. 0. 0. F - - - - - - - - -

spotPython implements an optimization handler that maps the optimizer names to the corresponding PyTorch optimizers.

A note on LBFGS

We recommend deactivating PyTorch’s LBFGS optimizer, because it does not perform very well. The PyTorch documentation, see https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html#torch.optim.LBFGS, states:

This is a very memory intensive optimizer (it requires additional param_bytes * (history_size + 1) bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.

Furthermore, the LBFGS optimizer is not compatible with the PyTorch tutorial. The reason is that the LBFGS optimizer requires the closure function, which is not implemented in the PyTorch tutorial. Therefore, the LBFGS optimizer is recommended here. Since there are ten optimizers in the portfolio, it is not recommended tuning the hyperparameters that effect one single optimizer only.

A note on the learning rate

spotPython provides a multiplier for the default learning rates, lr_mult, because optimizers use different learning rates. Using a multiplier for the learning rates might enable a simultaneous tuning of the learning rates for all optimizers. However, this is not recommended, because the learning rates are not comparable across optimizers. Therefore, we recommend fixing the learning rate for all optimizers if multiple optimizers are used. This can be done by setting the lower and upper bounds of the learning rate multiplier to the same value as shown below.

Thus, the learning rate, which affects the SGD optimizer, will be set to a fixed value. We choose the default value of 1e-3 for the learning rate, because it is used in other PyTorch examples (it is also the default value used by spotPython as defined in the optimizer_handler() method). We recommend tuning the learning rate later, when a reduced set of optimizers is fixed. Here, we will demonstrate how to select in a screening phase the optimizers that should be used for the hyperparameter tuning.

For the same reason, we will fix the sgd_momentum to 0.9.

fun_control = modify_hyper_parameter_bounds(fun_control,
    "lr_mult", bounds=[1.0, 1.0])
fun_control = modify_hyper_parameter_bounds(fun_control,
    "sgd_momentum", bounds=[0.9, 0.9])

14.7 Evaluation: Data Splitting

The evaluation procedure requires the specification of the way how the data is split into a train and a test set and the loss function (and a metric). As a default, spotPython provides a standard hold-out data split and cross validation.

14.7.1 Hold-out Data Split

If a hold-out data split is used, the data will be partitioned into a training, a validation, and a test data set. The split depends on the setting of the eval parameter. If eval is set to train_hold_out, one data set, usually the original training data set, is split into a new training and a validation data set. The training data set is used for training the model. The validation data set is used for the evaluation of the hyperparameter configuration and early stopping to prevent overfitting. In this case, the original test data set is not used.

Note

spotPython returns the hyperparameters of the machine learning and deep learning models, e.g., number of layers, learning rate, or optimizer, but not the model weights. Therefore, after the SPOT run is finished, the corresponding model with the optimized architecture has to be trained again with the best hyperparameter configuration. The training is performed on the training data set. The test data set is used for the final evaluation of the model.

Summarizing, the following splits are performed in the hold-out setting:

  1. Run spotPython with eval set to train_hold_out to determine the best hyperparameter configuration.
  2. Train the model with the best hyperparameter configuration (“architecture”) on the training data set: train_tuned(model_spot, train, "model_spot.pt").
  3. Test the model on the test data: test_tuned(model_spot, test, "model_spot.pt")

These steps will be exemplified in the following sections.

In addition to this hold-out setting, spotPython provides another hold-out setting, where an explicit test data is specified by the user that will be used as the validation set. To choose this option, the eval parameter is set to test_hold_out. In this case, the training data set is used for the model training. Then, the explicitly defined test data set is used for the evaluation of the hyperparameter configuration (the validation).

14.7.2 Cross-Validation

The cross validation setting is used by setting the eval parameter to train_cv or test_cv. In both cases, the data set is split into \(k\) folds. The model is trained on \(k-1\) folds and evaluated on the remaining fold. This is repeated \(k\) times, so that each fold is used exactly once for evaluation. The final evaluation is performed on the test data set. The cross validation setting is useful for small data sets, because it allows to use all data for training and evaluation. However, it is computationally expensive, because the model has to be trained \(k\) times.

Note

Combinations of the above settings are possible, e.g., cross validation can be used for training and hold-out for evaluation or vice versa. Also, cross validation can be used for training and testing. Because cross validation is not used in the PyTorch tutorial (PyTorch 2023a), it is not considered further here.

14.7.3 Overview of the Evaluation Settings

14.7.3.1 Settings for the Hyperparameter Tuning

An overview of the training evaluations is shown in Table 14.2. "train_cv" and "test_cv" use sklearn.model_selection.KFold() internally. More details on the data splitting are provided in Section 21.14 (in the Appendix).

Table 14.2: Overview of the evaluation settings.
eval train test function comment
"train_hold_out" \(\checkmark\) train_one_epoch(), validate_one_epoch() for early stopping splits the train data set internally
"test_hold_out" \(\checkmark\) \(\checkmark\) train_one_epoch(), validate_one_epoch() for early stopping use the test data set for validate_one_epoch()
"train_cv" \(\checkmark\) evaluate_cv(net, train) CV using the train data set
"test_cv" \(\checkmark\) evaluate_cv(net, test) CV using the test data set . Identical to "train_cv", uses only test data.

14.7.3.2 Settings for the Final Evaluation of the Tuned Architecture

14.7.3.2.1 Training of the Tuned Architecture

train_tuned(model, train): train the model with the best hyperparameter configuration (or simply the default) on the training data set. It splits the traindata into new train and validation sets using create_train_val_data_loaders(), which calls torch.utils.data.random_split() internally. Currently, 60% of the data is used for training and 40% for validation. The train data is used for training the model with train_hold_out(). The validation data is used for early stopping using validate_fold_or_hold_out() on the validation data set.

14.7.3.2.2 Testing of the Tuned Architecture

test_tuned(model, test): test the model on the test data set. No data splitting is performed. The (trained) model is evaluated using the validate_fold_or_hold_out() function. Note: During training, "shuffle" is set to True, whereas during testing, "shuffle" is set to False.

Section 21.14.1.4 describes the final evaluation of the tuned architecture.

fun_control.update({
    "eval": "train_hold_out",
    "path": "torch_model.pt",
    "shuffle": True})

14.8 Evaluation: Loss Functions and Metrics

The key "loss_function" specifies the loss function which is used during the optimization. There are several different loss functions under PyTorch’s nn package. For example, a simple loss is MSELoss, which computes the mean-squared error between the output and the target. In this tutorial we will use CrossEntropyLoss, because it is also used in the PyTorch tutorial.

from torch.nn import CrossEntropyLoss
loss_function = CrossEntropyLoss()
fun_control.update({"loss_function": loss_function})

In addition to the loss functions, spotPython provides access to a large number of metrics.

  • The key "metric_sklearn" is used for metrics that follow the scikit-learn conventions.
  • The key "river_metric" is used for the river based evaluation (Montiel et al. 2021) via eval_oml_iter_progressive, and
  • the key "metric_torch" is used for the metrics from TorchMetrics.

TorchMetrics is a collection of more than 90 PyTorch metrics, see https://torchmetrics.readthedocs.io/en/latest/. Because the PyTorch tutorial uses the accuracy as metric, we use the same metric here. Currently, accuracy is computed in the tutorial’s example code. We will use TorchMetrics instead, because it offers more flexibilty, e.g., it can be used for regression and classification. Furthermore, TorchMetrics offers the following advantages:

* A standardized interface to increase reproducibility
* Reduces Boilerplate
* Distributed-training compatible
* Rigorously tested
* Automatic accumulation over batches
* Automatic synchronization between multiple devices

Therefore, we set

import torchmetrics
metric_torch = torchmetrics.Accuracy(task="multiclass", num_classes=10).to(fun_control["device"])
fun_control.update({"metric_torch": metric_torch})

14.9 Preparing the SPOT Call

The following code passes the information about the parameter ranges and bounds to spot.

from spotPython.hyperparameters.values import (
    get_var_type,
    get_var_name,
    get_bound_values    
    )
var_type = get_var_type(fun_control)
var_name = get_var_name(fun_control)
fun_control.update({"var_type": var_type,
                    "var_name": var_name})

lower = get_bound_values(fun_control, "lower")
upper = get_bound_values(fun_control, "upper")

Now, the dictionary fun_control contains all information needed for the hyperparameter tuning. Before the hyperparameter tuning is started, it is recommended to take a look at the experimental design. The method gen_design_table generates a design table as follows:

from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control))
| name         | type   | default   |   lower |   upper | transform             |
|--------------|--------|-----------|---------|---------|-----------------------|
| l1           | int    | 5         |     2   |     9   | transform_power_2_int |
| l2           | int    | 5         |     2   |     9   | transform_power_2_int |
| lr_mult      | float  | 1.0       |     1   |     1   | None                  |
| batch_size   | int    | 4         |     1   |     5   | transform_power_2_int |
| epochs       | int    | 3         |     3   |     4   | transform_power_2_int |
| k_folds      | int    | 1         |     0   |     0   | None                  |
| patience     | int    | 5         |     3   |     3   | None                  |
| optimizer    | factor | SGD       |     0   |     9   | None                  |
| sgd_momentum | float  | 0.0       |     0.9 |     0.9 | None                  |

This allows to check if all information is available and if the information is correct. ?tbl-design shows the experimental design for the hyperparameter tuning. The table shows the hyperparameters, their types, default values, lower and upper bounds, and the transformation function. The transformation function is used to transform the hyperparameter values from the unit hypercube to the original domain. The transformation function is applied to the hyperparameter values before the evaluation of the objective function. Hyperparameter transformations are shown in the column “transform”, e.g., the l1 default is 5, which results in the value \(2^5 = 32\) for the network, because the transformation transform_power_2_int was selected in the JSON file. The default value of the batch_size is set to 4, which results in a batch size of \(2^4 = 16\).

14.10 The Objective Function fun_torch

The objective function fun_torch is selected next. It implements an interface from PyTorch’s training, validation, and testing methods to spotPython.

from spotPython.fun.hypertorch import HyperTorch
fun = HyperTorch().fun_torch

14.11 Using Default Hyperparameters or Results from Previous Runs

We add the default setting to the initial design:

from spotPython.hyperparameters.values import get_default_hyperparameters_as_array
hyper_dict=TorchHyperDict().load()
X_start = get_default_hyperparameters_as_array(fun_control, hyper_dict)

14.12 Starting the Hyperparameter Tuning

The spotPython hyperparameter tuning is started by calling the Spot function. Here, we will run the tuner for approximately 30 minutes (max_time). Note: the initial design is always evaluated in the spotPython run. As a consequence, the run may take longer than specified by max_time, because the evaluation time of initial design (here: init_size, 10 points) is performed independently of max_time. During the run, results from the training is shown. These results can be visualized with Tensorboard as will be shown in Section 14.13.

from spotPython.spot import spot
from math import inf
import numpy as np
spot_tuner = spot.Spot(fun=fun,
                   lower = lower,
                   upper = upper,
                   fun_evals = inf,
                   fun_repeats = 1,
                   max_time = MAX_TIME,
                   noise = False,
                   tolerance_x = np.sqrt(np.spacing(1)),
                   var_type = var_type,
                   var_name = var_name,
                   infill_criterion = "y",
                   n_points = 1,
                   seed=123,
                   log_level = 50,
                   show_models= False,
                   show_progress= True,
                   fun_control = fun_control,
                   design_control={"init_size": INIT_SIZE,
                                   "repeats": 1},
                   surrogate_control={"noise": True,
                                      "cod_type": "norm",
                                      "min_theta": -4,
                                      "max_theta": 3,
                                      "n_theta": len(var_name),
                                      "model_fun_evals": 10_000,
                                      "log_level": 50
                                      })
spot_tuner.run(X_start=X_start)

config: {'l1': 64, 'l2': 16, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.4466147364616393
Accuracy on hold-out set: 0.4714
MulticlassAccuracy value on hold-out data: 0.4713999927043915
Epoch: 2
Loss on hold-out set: 1.3187020245075225
Accuracy on hold-out set: 0.52965
MulticlassAccuracy value on hold-out data: 0.5296499729156494
Epoch: 3
Loss on hold-out set: 1.2365772798776626
Accuracy on hold-out set: 0.5666
MulticlassAccuracy value on hold-out data: 0.5666000247001648
Epoch: 4
Loss on hold-out set: 1.2172871111154557
Accuracy on hold-out set: 0.57205
MulticlassAccuracy value on hold-out data: 0.5720499753952026
Epoch: 5
Loss on hold-out set: 1.2040423777341842
Accuracy on hold-out set: 0.5837
MulticlassAccuracy value on hold-out data: 0.5837000012397766
Epoch: 6
Loss on hold-out set: 1.204949404501915
Accuracy on hold-out set: 0.5865
MulticlassAccuracy value on hold-out data: 0.5864999890327454
Epoch: 7
Loss on hold-out set: 1.1944059766292572
Accuracy on hold-out set: 0.5885
MulticlassAccuracy value on hold-out data: 0.5885000228881836
Epoch: 8
Loss on hold-out set: 1.1817050766229629
Accuracy on hold-out set: 0.6029
MulticlassAccuracy value on hold-out data: 0.6029000282287598
Epoch: 9
Loss on hold-out set: 1.2363497236967087
Accuracy on hold-out set: 0.58715
MulticlassAccuracy value on hold-out data: 0.587149977684021
Epoch: 10
Loss on hold-out set: 1.2241075073719025
Accuracy on hold-out set: 0.589
MulticlassAccuracy value on hold-out data: 0.5889999866485596
Epoch: 11
Loss on hold-out set: 1.2310864565610886
Accuracy on hold-out set: 0.59385
MulticlassAccuracy value on hold-out data: 0.5938500165939331
Early stopping at epoch 10
Returned to Spot: Validation loss: 1.2310864565610886
----------------------------------------------

config: {'l1': 16, 'l2': 128, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.6160703612476588
Accuracy on hold-out set: 0.43005
MulticlassAccuracy value on hold-out data: 0.43004998564720154
Epoch: 2
Loss on hold-out set: 1.4658435981988907
Accuracy on hold-out set: 0.47165
MulticlassAccuracy value on hold-out data: 0.47165000438690186
Epoch: 3
Loss on hold-out set: 1.4115622333094477
Accuracy on hold-out set: 0.4885
MulticlassAccuracy value on hold-out data: 0.4884999990463257
Epoch: 4
Loss on hold-out set: 1.421499628649652
Accuracy on hold-out set: 0.49305
MulticlassAccuracy value on hold-out data: 0.4930500090122223
Epoch: 5
Loss on hold-out set: 1.4507728659309447
Accuracy on hold-out set: 0.5048
MulticlassAccuracy value on hold-out data: 0.504800021648407
Epoch: 6
Loss on hold-out set: 1.3837634062178432
Accuracy on hold-out set: 0.52795
MulticlassAccuracy value on hold-out data: 0.5279499888420105
Epoch: 7
Loss on hold-out set: 1.4347644170668907
Accuracy on hold-out set: 0.5195
MulticlassAccuracy value on hold-out data: 0.5195000171661377
Epoch: 8
Loss on hold-out set: 1.3805725740564057
Accuracy on hold-out set: 0.5316
MulticlassAccuracy value on hold-out data: 0.5315999984741211
Epoch: 9
Loss on hold-out set: 1.4081523430068046
Accuracy on hold-out set: 0.5201
MulticlassAccuracy value on hold-out data: 0.5200999975204468
Epoch: 10
Loss on hold-out set: 1.421753317871131
Accuracy on hold-out set: 0.53665
MulticlassAccuracy value on hold-out data: 0.5366500020027161
Epoch: 11
Loss on hold-out set: 1.5013980769732036
Accuracy on hold-out set: 0.5229
MulticlassAccuracy value on hold-out data: 0.5228999853134155
Early stopping at epoch 10
Returned to Spot: Validation loss: 1.5013980769732036
----------------------------------------------

config: {'l1': 32, 'l2': 8, 'lr_mult': 1.0, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'ASGD', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.9073260845184326
Accuracy on hold-out set: 0.2799
MulticlassAccuracy value on hold-out data: 0.2799000144004822
Epoch: 2
Loss on hold-out set: 1.7863998437404633
Accuracy on hold-out set: 0.33835
MulticlassAccuracy value on hold-out data: 0.33834999799728394
Epoch: 3
Loss on hold-out set: 1.7207605367898942
Accuracy on hold-out set: 0.36495
MulticlassAccuracy value on hold-out data: 0.3649500012397766
Epoch: 4
Loss on hold-out set: 1.6622491683721543
Accuracy on hold-out set: 0.38795
MulticlassAccuracy value on hold-out data: 0.38795000314712524
Epoch: 5
Loss on hold-out set: 1.6118217293739319
Accuracy on hold-out set: 0.4072
MulticlassAccuracy value on hold-out data: 0.40720000863075256
Epoch: 6
Loss on hold-out set: 1.568985506439209
Accuracy on hold-out set: 0.4193
MulticlassAccuracy value on hold-out data: 0.41929998993873596
Epoch: 7
Loss on hold-out set: 1.5368788693904876
Accuracy on hold-out set: 0.42715
MulticlassAccuracy value on hold-out data: 0.42715001106262207
Epoch: 8
Loss on hold-out set: 1.5136046350240708
Accuracy on hold-out set: 0.44125
MulticlassAccuracy value on hold-out data: 0.4412499964237213
Returned to Spot: Validation loss: 1.5136046350240708
----------------------------------------------

config: {'l1': 8, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adamax', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.636824743950367
Accuracy on hold-out set: 0.4149
MulticlassAccuracy value on hold-out data: 0.414900004863739
Epoch: 2
Loss on hold-out set: 1.4592909926623105
Accuracy on hold-out set: 0.47365
MulticlassAccuracy value on hold-out data: 0.4736500084400177
Epoch: 3
Loss on hold-out set: 1.3327275277674198
Accuracy on hold-out set: 0.5267
MulticlassAccuracy value on hold-out data: 0.5267000198364258
Epoch: 4
Loss on hold-out set: 1.2955283494904637
Accuracy on hold-out set: 0.54505
MulticlassAccuracy value on hold-out data: 0.5450500249862671
Epoch: 5
Loss on hold-out set: 1.3228704055428504
Accuracy on hold-out set: 0.5397
MulticlassAccuracy value on hold-out data: 0.5396999716758728
Epoch: 6
Loss on hold-out set: 1.259785818330571
Accuracy on hold-out set: 0.56455
MulticlassAccuracy value on hold-out data: 0.56454998254776
Epoch: 7
Loss on hold-out set: 1.2894477102706208
Accuracy on hold-out set: 0.56135
MulticlassAccuracy value on hold-out data: 0.5613499879837036
Epoch: 8
Loss on hold-out set: 1.2455974631845952
Accuracy on hold-out set: 0.56895
MulticlassAccuracy value on hold-out data: 0.5689499974250793
Returned to Spot: Validation loss: 1.2455974631845952
----------------------------------------------

config: {'l1': 128, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 8, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.444605352306366
Accuracy on hold-out set: 0.47655
MulticlassAccuracy value on hold-out data: 0.47655001282691956
Epoch: 2
Loss on hold-out set: 1.3022559710025787
Accuracy on hold-out set: 0.5368
MulticlassAccuracy value on hold-out data: 0.5368000268936157
Epoch: 3
Loss on hold-out set: 1.2439701816678048
Accuracy on hold-out set: 0.5581
MulticlassAccuracy value on hold-out data: 0.5580999851226807
Epoch: 4
Loss on hold-out set: 1.2341607529520988
Accuracy on hold-out set: 0.56225
MulticlassAccuracy value on hold-out data: 0.562250018119812
Epoch: 5
Loss on hold-out set: 1.2272565495789052
Accuracy on hold-out set: 0.57375
MulticlassAccuracy value on hold-out data: 0.5737500190734863
Epoch: 6
Loss on hold-out set: 1.1893009765654803
Accuracy on hold-out set: 0.59
MulticlassAccuracy value on hold-out data: 0.5899999737739563
Epoch: 7
Loss on hold-out set: 1.2082740735113622
Accuracy on hold-out set: 0.59235
MulticlassAccuracy value on hold-out data: 0.5923500061035156
Epoch: 8
Loss on hold-out set: 1.2378235348641873
Accuracy on hold-out set: 0.59105
MulticlassAccuracy value on hold-out data: 0.5910500288009644
Epoch: 9
Loss on hold-out set: 1.2819154630593956
Accuracy on hold-out set: 0.5865
MulticlassAccuracy value on hold-out data: 0.5864999890327454
Early stopping at epoch 8
Returned to Spot: Validation loss: 1.2819154630593956
----------------------------------------------

config: {'l1': 512, 'l2': 16, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'RMSprop', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 2.305287952852249
Accuracy on hold-out set: 0.0974
MulticlassAccuracy value on hold-out data: 0.09740000218153
Epoch: 2
Loss on hold-out set: 2.3063367626190185
Accuracy on hold-out set: 0.1001
MulticlassAccuracy value on hold-out data: 0.10010000318288803
Epoch: 3
Loss on hold-out set: 2.3056188706874847
Accuracy on hold-out set: 0.1002
MulticlassAccuracy value on hold-out data: 0.10019999742507935
Epoch: 4
Loss on hold-out set: 2.30639640212059
Accuracy on hold-out set: 0.09735
MulticlassAccuracy value on hold-out data: 0.09735000133514404
Early stopping at epoch 3
Returned to Spot: Validation loss: 2.30639640212059
----------------------------------------------

config: {'l1': 8, 'l2': 8, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adagrad', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.8104798585891724
Accuracy on hold-out set: 0.3167
MulticlassAccuracy value on hold-out data: 0.3167000114917755
Epoch: 2
Loss on hold-out set: 1.7184131657600403
Accuracy on hold-out set: 0.35295
MulticlassAccuracy value on hold-out data: 0.35295000672340393
Epoch: 3
Loss on hold-out set: 1.680698029232025
Accuracy on hold-out set: 0.3691
MulticlassAccuracy value on hold-out data: 0.36910000443458557
Epoch: 4
Loss on hold-out set: 1.6480772290229797
Accuracy on hold-out set: 0.38445
MulticlassAccuracy value on hold-out data: 0.3844499886035919
Epoch: 5
Loss on hold-out set: 1.616037343788147
Accuracy on hold-out set: 0.39835
MulticlassAccuracy value on hold-out data: 0.3983500003814697
Epoch: 6
Loss on hold-out set: 1.5998824858665466
Accuracy on hold-out set: 0.4011
MulticlassAccuracy value on hold-out data: 0.4011000096797943
Epoch: 7
Loss on hold-out set: 1.5861845971107482
Accuracy on hold-out set: 0.4107
MulticlassAccuracy value on hold-out data: 0.4106999933719635
Epoch: 8
Loss on hold-out set: 1.5785142189502717
Accuracy on hold-out set: 0.4119
MulticlassAccuracy value on hold-out data: 0.41190001368522644
Returned to Spot: Validation loss: 1.5785142189502717
----------------------------------------------

config: {'l1': 256, 'l2': 64, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'RMSprop', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 2.3064946407318114
Accuracy on hold-out set: 0.09805
MulticlassAccuracy value on hold-out data: 0.09804999828338623
Epoch: 2
Loss on hold-out set: 2.304391358757019
Accuracy on hold-out set: 0.0999
MulticlassAccuracy value on hold-out data: 0.09989999979734421
Epoch: 3
Loss on hold-out set: 2.304707049560547
Accuracy on hold-out set: 0.10025
MulticlassAccuracy value on hold-out data: 0.1002499982714653
Epoch: 4
Loss on hold-out set: 2.3052314863204955
Accuracy on hold-out set: 0.1024
MulticlassAccuracy value on hold-out data: 0.10239999741315842
Epoch: 5
Loss on hold-out set: 2.30669695892334
Accuracy on hold-out set: 0.0971
MulticlassAccuracy value on hold-out data: 0.09709999710321426
Early stopping at epoch 4
Returned to Spot: Validation loss: 2.30669695892334
----------------------------------------------

config: {'l1': 256, 'l2': 512, 'lr_mult': 1.0, 'batch_size': 2, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.671713733038353
Accuracy on hold-out set: 0.4238
MulticlassAccuracy value on hold-out data: 0.423799991607666
Epoch: 2
Loss on hold-out set: 1.5574801034649677
Accuracy on hold-out set: 0.4824
MulticlassAccuracy value on hold-out data: 0.48240000009536743
Epoch: 3
Loss on hold-out set: 1.5389536535870652
Accuracy on hold-out set: 0.51615
MulticlassAccuracy value on hold-out data: 0.5161499977111816
Epoch: 4
Loss on hold-out set: 1.44540891609514
Accuracy on hold-out set: 0.51255
MulticlassAccuracy value on hold-out data: 0.5125499963760376
Epoch: 5
Loss on hold-out set: 1.517581852355556
Accuracy on hold-out set: 0.53585
MulticlassAccuracy value on hold-out data: 0.5358499884605408
Epoch: 6
Loss on hold-out set: 1.507274435602174
Accuracy on hold-out set: 0.53325
MulticlassAccuracy value on hold-out data: 0.5332499742507935
Epoch: 7
Loss on hold-out set: 1.5716324373546902
Accuracy on hold-out set: 0.5369
MulticlassAccuracy value on hold-out data: 0.536899983882904
Early stopping at epoch 6
Returned to Spot: Validation loss: 1.5716324373546902
----------------------------------------------

config: {'l1': 4, 'l2': 256, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adadelta', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.7436754522323608
Accuracy on hold-out set: 0.3456
MulticlassAccuracy value on hold-out data: 0.3456000089645386
Epoch: 2
Loss on hold-out set: 1.5545325801849366
Accuracy on hold-out set: 0.4121
MulticlassAccuracy value on hold-out data: 0.4120999872684479
Epoch: 3
Loss on hold-out set: 1.4470017765045167
Accuracy on hold-out set: 0.45425
MulticlassAccuracy value on hold-out data: 0.4542500078678131
Epoch: 4
Loss on hold-out set: 1.411888943862915
Accuracy on hold-out set: 0.47335
MulticlassAccuracy value on hold-out data: 0.47334998846054077
Epoch: 5
Loss on hold-out set: 1.4242105187416076
Accuracy on hold-out set: 0.47055
MulticlassAccuracy value on hold-out data: 0.470550000667572
Epoch: 6
Loss on hold-out set: 1.382616423034668
Accuracy on hold-out set: 0.489
MulticlassAccuracy value on hold-out data: 0.48899999260902405
Epoch: 7
Loss on hold-out set: 1.3809644070625304
Accuracy on hold-out set: 0.48615
MulticlassAccuracy value on hold-out data: 0.48614999651908875
Epoch: 8
Loss on hold-out set: 1.3573257621765136
Accuracy on hold-out set: 0.49735
MulticlassAccuracy value on hold-out data: 0.4973500072956085
Epoch: 9
Loss on hold-out set: 1.349511796951294
Accuracy on hold-out set: 0.50385
MulticlassAccuracy value on hold-out data: 0.503849983215332
Epoch: 10
Loss on hold-out set: 1.4214812358856201
Accuracy on hold-out set: 0.48305
MulticlassAccuracy value on hold-out data: 0.48304998874664307
Epoch: 11
Loss on hold-out set: 1.3630828218460083
Accuracy on hold-out set: 0.49525
MulticlassAccuracy value on hold-out data: 0.49524998664855957
Epoch: 12
Loss on hold-out set: 1.3295877265930176
Accuracy on hold-out set: 0.5141
MulticlassAccuracy value on hold-out data: 0.5141000151634216
Epoch: 13
Loss on hold-out set: 1.3434017603874207
Accuracy on hold-out set: 0.49995
MulticlassAccuracy value on hold-out data: 0.49994999170303345
Epoch: 14
Loss on hold-out set: 1.3819206321716309
Accuracy on hold-out set: 0.4948
MulticlassAccuracy value on hold-out data: 0.49480000138282776
Epoch: 15
Loss on hold-out set: 1.335169330406189
Accuracy on hold-out set: 0.5084
MulticlassAccuracy value on hold-out data: 0.508400022983551
Early stopping at epoch 14
Returned to Spot: Validation loss: 1.335169330406189
----------------------------------------------

config: {'l1': 64, 'l2': 8, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.566739748287201
Accuracy on hold-out set: 0.4265
MulticlassAccuracy value on hold-out data: 0.42649999260902405
Epoch: 2
Loss on hold-out set: 1.4413416323661805
Accuracy on hold-out set: 0.4762
MulticlassAccuracy value on hold-out data: 0.47620001435279846
Epoch: 3
Loss on hold-out set: 1.3442577162742615
Accuracy on hold-out set: 0.5162
MulticlassAccuracy value on hold-out data: 0.5162000060081482
Epoch: 4
Loss on hold-out set: 1.2730355889320373
Accuracy on hold-out set: 0.5464
MulticlassAccuracy value on hold-out data: 0.5464000105857849
Epoch: 5
Loss on hold-out set: 1.2325679732322692
Accuracy on hold-out set: 0.56315
MulticlassAccuracy value on hold-out data: 0.5631499886512756
Epoch: 6
Loss on hold-out set: 1.234675616312027
Accuracy on hold-out set: 0.56685
MulticlassAccuracy value on hold-out data: 0.5668500065803528
Epoch: 7
Loss on hold-out set: 1.2309454638957977
Accuracy on hold-out set: 0.56415
MulticlassAccuracy value on hold-out data: 0.5641499757766724
Epoch: 8
Loss on hold-out set: 1.1936278371334077
Accuracy on hold-out set: 0.58005
MulticlassAccuracy value on hold-out data: 0.580049991607666
Epoch: 9
Loss on hold-out set: 1.1985017212629319
Accuracy on hold-out set: 0.57875
MulticlassAccuracy value on hold-out data: 0.5787500143051147
Epoch: 10
Loss on hold-out set: 1.156120140528679
Accuracy on hold-out set: 0.59375
MulticlassAccuracy value on hold-out data: 0.59375
Epoch: 11
Loss on hold-out set: 1.188929581952095
Accuracy on hold-out set: 0.5899
MulticlassAccuracy value on hold-out data: 0.589900016784668
Epoch: 12
Loss on hold-out set: 1.1851380628108978
Accuracy on hold-out set: 0.5955
MulticlassAccuracy value on hold-out data: 0.5954999923706055
Epoch: 13
Loss on hold-out set: 1.1414653688192367
Accuracy on hold-out set: 0.60405
MulticlassAccuracy value on hold-out data: 0.6040499806404114
Epoch: 14
Loss on hold-out set: 1.1418095175027847
Accuracy on hold-out set: 0.60275
MulticlassAccuracy value on hold-out data: 0.6027500033378601
Epoch: 15
Loss on hold-out set: 1.1521970530152321
Accuracy on hold-out set: 0.6067
MulticlassAccuracy value on hold-out data: 0.6067000031471252
Epoch: 16
Loss on hold-out set: 1.1751966602563857
Accuracy on hold-out set: 0.60045
MulticlassAccuracy value on hold-out data: 0.6004499793052673
Early stopping at epoch 15
Returned to Spot: Validation loss: 1.1751966602563857
----------------------------------------------
spotPython tuning: 1.1751966602563857 [#---------] 11.56% 

config: {'l1': 64, 'l2': 4, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.643492722606659
Accuracy on hold-out set: 0.3976
MulticlassAccuracy value on hold-out data: 0.397599995136261
Epoch: 2
Loss on hold-out set: 1.5325763413906097
Accuracy on hold-out set: 0.4368
MulticlassAccuracy value on hold-out data: 0.4368000030517578
Epoch: 3
Loss on hold-out set: 1.4571219738960266
Accuracy on hold-out set: 0.474
MulticlassAccuracy value on hold-out data: 0.4740000069141388
Epoch: 4
Loss on hold-out set: 1.4097870640277863
Accuracy on hold-out set: 0.48805
MulticlassAccuracy value on hold-out data: 0.48805001378059387
Epoch: 5
Loss on hold-out set: 1.3428638056278228
Accuracy on hold-out set: 0.517
MulticlassAccuracy value on hold-out data: 0.5170000195503235
Epoch: 6
Loss on hold-out set: 1.3394954214096069
Accuracy on hold-out set: 0.5164
MulticlassAccuracy value on hold-out data: 0.5163999795913696
Epoch: 7
Loss on hold-out set: 1.2683543021678925
Accuracy on hold-out set: 0.5499
MulticlassAccuracy value on hold-out data: 0.5498999953269958
Epoch: 8
Loss on hold-out set: 1.3157669149875642
Accuracy on hold-out set: 0.54065
MulticlassAccuracy value on hold-out data: 0.5406500101089478
Epoch: 9
Loss on hold-out set: 1.2618958990097047
Accuracy on hold-out set: 0.5559
MulticlassAccuracy value on hold-out data: 0.555899977684021
Epoch: 10
Loss on hold-out set: 1.2395635766744613
Accuracy on hold-out set: 0.5692
MulticlassAccuracy value on hold-out data: 0.5691999793052673
Epoch: 11
Loss on hold-out set: 1.2327296003580093
Accuracy on hold-out set: 0.5759
MulticlassAccuracy value on hold-out data: 0.5759000182151794
Epoch: 12
Loss on hold-out set: 1.2726782939910888
Accuracy on hold-out set: 0.56905
MulticlassAccuracy value on hold-out data: 0.5690500140190125
Epoch: 13
Loss on hold-out set: 1.2548043754816056
Accuracy on hold-out set: 0.57555
MulticlassAccuracy value on hold-out data: 0.5755500197410583
Epoch: 14
Loss on hold-out set: 1.2296062014341353
Accuracy on hold-out set: 0.57665
MulticlassAccuracy value on hold-out data: 0.5766500234603882
Epoch: 15
Loss on hold-out set: 1.2613706868171692
Accuracy on hold-out set: 0.5696
MulticlassAccuracy value on hold-out data: 0.569599986076355
Epoch: 16
Loss on hold-out set: 1.2483285284280776
Accuracy on hold-out set: 0.5758
MulticlassAccuracy value on hold-out data: 0.5758000016212463
Returned to Spot: Validation loss: 1.2483285284280776
----------------------------------------------
spotPython tuning: 1.1751966602563857 [##--------] 22.94% 

config: {'l1': 64, 'l2': 256, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.4363937804698943
Accuracy on hold-out set: 0.4712
MulticlassAccuracy value on hold-out data: 0.47119998931884766
Epoch: 2
Loss on hold-out set: 1.3496123288631439
Accuracy on hold-out set: 0.51095
MulticlassAccuracy value on hold-out data: 0.5109500288963318
Epoch: 3
Loss on hold-out set: 1.2807581854581833
Accuracy on hold-out set: 0.5398
MulticlassAccuracy value on hold-out data: 0.5397999882698059
Epoch: 4
Loss on hold-out set: 1.2335183374643326
Accuracy on hold-out set: 0.55795
MulticlassAccuracy value on hold-out data: 0.5579500198364258
Epoch: 5
Loss on hold-out set: 1.1995039861917496
Accuracy on hold-out set: 0.5776
MulticlassAccuracy value on hold-out data: 0.5776000022888184
Epoch: 6
Loss on hold-out set: 1.2130537831783295
Accuracy on hold-out set: 0.57745
MulticlassAccuracy value on hold-out data: 0.5774499773979187
Epoch: 7
Loss on hold-out set: 1.2435360367059707
Accuracy on hold-out set: 0.5599
MulticlassAccuracy value on hold-out data: 0.5598999857902527
Epoch: 8
Loss on hold-out set: 1.1640983190774918
Accuracy on hold-out set: 0.58975
MulticlassAccuracy value on hold-out data: 0.5897499918937683
Epoch: 9
Loss on hold-out set: 1.231044849896431
Accuracy on hold-out set: 0.5741
MulticlassAccuracy value on hold-out data: 0.5741000175476074
Epoch: 10
Loss on hold-out set: 1.1696051315188407
Accuracy on hold-out set: 0.59635
MulticlassAccuracy value on hold-out data: 0.5963500142097473
Epoch: 11
Loss on hold-out set: 1.2444086892366408
Accuracy on hold-out set: 0.58395
MulticlassAccuracy value on hold-out data: 0.5839499831199646
Early stopping at epoch 10
Returned to Spot: Validation loss: 1.2444086892366408
----------------------------------------------
spotPython tuning: 1.1751966602563857 [###-------] 30.88% 

config: {'l1': 256, 'l2': 512, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adagrad', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.4386033380508423
Accuracy on hold-out set: 0.46975
MulticlassAccuracy value on hold-out data: 0.46974998712539673
Epoch: 2
Loss on hold-out set: 1.3450033884048462
Accuracy on hold-out set: 0.51245
MulticlassAccuracy value on hold-out data: 0.5124499797821045
Epoch: 3
Loss on hold-out set: 1.3187074427604675
Accuracy on hold-out set: 0.51945
MulticlassAccuracy value on hold-out data: 0.5194500088691711
Epoch: 4
Loss on hold-out set: 1.2667100133895874
Accuracy on hold-out set: 0.5443
MulticlassAccuracy value on hold-out data: 0.5443000197410583
Epoch: 5
Loss on hold-out set: 1.2486800297737122
Accuracy on hold-out set: 0.5536
MulticlassAccuracy value on hold-out data: 0.553600013256073
Epoch: 6
Loss on hold-out set: 1.2323607445716858
Accuracy on hold-out set: 0.5671
MulticlassAccuracy value on hold-out data: 0.5670999884605408
Epoch: 7
Loss on hold-out set: 1.2147771913528442
Accuracy on hold-out set: 0.5693
MulticlassAccuracy value on hold-out data: 0.5692999958992004
Epoch: 8
Loss on hold-out set: 1.197420945262909
Accuracy on hold-out set: 0.57925
MulticlassAccuracy value on hold-out data: 0.5792499780654907
Epoch: 9
Loss on hold-out set: 1.2009706085205079
Accuracy on hold-out set: 0.57985
MulticlassAccuracy value on hold-out data: 0.5798500180244446
Epoch: 10
Loss on hold-out set: 1.1854215562820434
Accuracy on hold-out set: 0.58255
MulticlassAccuracy value on hold-out data: 0.5825499892234802
Epoch: 11
Loss on hold-out set: 1.1850558595657348
Accuracy on hold-out set: 0.5874
MulticlassAccuracy value on hold-out data: 0.5874000191688538
Epoch: 12
Loss on hold-out set: 1.175267757320404
Accuracy on hold-out set: 0.5901
MulticlassAccuracy value on hold-out data: 0.5900999903678894
Epoch: 13
Loss on hold-out set: 1.1720441947937013
Accuracy on hold-out set: 0.59635
MulticlassAccuracy value on hold-out data: 0.5963500142097473
Epoch: 14
Loss on hold-out set: 1.1703355357170104
Accuracy on hold-out set: 0.5949
MulticlassAccuracy value on hold-out data: 0.5949000120162964
Epoch: 15
Loss on hold-out set: 1.1717702922821045
Accuracy on hold-out set: 0.5978
MulticlassAccuracy value on hold-out data: 0.5978000164031982
Epoch: 16
Loss on hold-out set: 1.1664946717262268
Accuracy on hold-out set: 0.59895
MulticlassAccuracy value on hold-out data: 0.5989500284194946
Returned to Spot: Validation loss: 1.1664946717262268
----------------------------------------------
spotPython tuning: 1.1664946717262268 [####------] 40.96% 

config: {'l1': 128, 'l2': 4, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adagrad', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.866480973625183
Accuracy on hold-out set: 0.25195
MulticlassAccuracy value on hold-out data: 0.2519499957561493
Epoch: 2
Loss on hold-out set: 1.7948342315673829
Accuracy on hold-out set: 0.2874
MulticlassAccuracy value on hold-out data: 0.2874000072479248
Epoch: 3
Loss on hold-out set: 1.7742672409057618
Accuracy on hold-out set: 0.2945
MulticlassAccuracy value on hold-out data: 0.2944999933242798
Epoch: 4
Loss on hold-out set: 1.7477629230499268
Accuracy on hold-out set: 0.31325
MulticlassAccuracy value on hold-out data: 0.31325000524520874
Epoch: 5
Loss on hold-out set: 1.7320594287872315
Accuracy on hold-out set: 0.31665
MulticlassAccuracy value on hold-out data: 0.31665000319480896
Epoch: 6
Loss on hold-out set: 1.7180017408370971
Accuracy on hold-out set: 0.3193
MulticlassAccuracy value on hold-out data: 0.31929999589920044
Epoch: 7
Loss on hold-out set: 1.7102912202835083
Accuracy on hold-out set: 0.3284
MulticlassAccuracy value on hold-out data: 0.32839998602867126
Epoch: 8
Loss on hold-out set: 1.697158182144165
Accuracy on hold-out set: 0.33075
MulticlassAccuracy value on hold-out data: 0.3307499885559082
Epoch: 9
Loss on hold-out set: 1.6870841979980469
Accuracy on hold-out set: 0.3362
MulticlassAccuracy value on hold-out data: 0.3361999988555908
Epoch: 10
Loss on hold-out set: 1.6802684423446654
Accuracy on hold-out set: 0.34155
MulticlassAccuracy value on hold-out data: 0.34154999256134033
Epoch: 11
Loss on hold-out set: 1.6742459674835206
Accuracy on hold-out set: 0.34105
MulticlassAccuracy value on hold-out data: 0.34104999899864197
Epoch: 12
Loss on hold-out set: 1.6662571050643922
Accuracy on hold-out set: 0.3439
MulticlassAccuracy value on hold-out data: 0.34389999508857727
Epoch: 13
Loss on hold-out set: 1.6622610288619994
Accuracy on hold-out set: 0.34955
MulticlassAccuracy value on hold-out data: 0.3495500087738037
Epoch: 14
Loss on hold-out set: 1.6554066762924193
Accuracy on hold-out set: 0.34985
MulticlassAccuracy value on hold-out data: 0.34984999895095825
Epoch: 15
Loss on hold-out set: 1.653537935256958
Accuracy on hold-out set: 0.359
MulticlassAccuracy value on hold-out data: 0.35899999737739563
Epoch: 16
Loss on hold-out set: 1.6472090824127197
Accuracy on hold-out set: 0.3552
MulticlassAccuracy value on hold-out data: 0.35519999265670776
Returned to Spot: Validation loss: 1.6472090824127197
----------------------------------------------
spotPython tuning: 1.1664946717262268 [#####-----] 50.73% 

config: {'l1': 4, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.7094561862945556
Accuracy on hold-out set: 0.35015
MulticlassAccuracy value on hold-out data: 0.3501499891281128
Epoch: 2
Loss on hold-out set: 1.6287086835861206
Accuracy on hold-out set: 0.38705
MulticlassAccuracy value on hold-out data: 0.38705000281333923
Epoch: 3
Loss on hold-out set: 1.5933373462677003
Accuracy on hold-out set: 0.39645
MulticlassAccuracy value on hold-out data: 0.396450012922287
Epoch: 4
Loss on hold-out set: 1.5190346406936646
Accuracy on hold-out set: 0.41995
MulticlassAccuracy value on hold-out data: 0.419950008392334
Epoch: 5
Loss on hold-out set: 1.488317742919922
Accuracy on hold-out set: 0.43415
MulticlassAccuracy value on hold-out data: 0.43415001034736633
Epoch: 6
Loss on hold-out set: 1.4708645124435424
Accuracy on hold-out set: 0.44365
MulticlassAccuracy value on hold-out data: 0.4436500072479248
Epoch: 7
Loss on hold-out set: 1.465621766281128
Accuracy on hold-out set: 0.44965
MulticlassAccuracy value on hold-out data: 0.44964998960494995
Epoch: 8
Loss on hold-out set: 1.4434933650970458
Accuracy on hold-out set: 0.44965
MulticlassAccuracy value on hold-out data: 0.44964998960494995
Epoch: 9
Loss on hold-out set: 1.4157191271781921
Accuracy on hold-out set: 0.464
MulticlassAccuracy value on hold-out data: 0.46399998664855957
Epoch: 10
Loss on hold-out set: 1.437671357345581
Accuracy on hold-out set: 0.45545
MulticlassAccuracy value on hold-out data: 0.45544999837875366
Epoch: 11
Loss on hold-out set: 1.4159093366622926
Accuracy on hold-out set: 0.4676
MulticlassAccuracy value on hold-out data: 0.4675999879837036
Epoch: 12
Loss on hold-out set: 1.3909984421730042
Accuracy on hold-out set: 0.47545
MulticlassAccuracy value on hold-out data: 0.4754500091075897
Epoch: 13
Loss on hold-out set: 1.3982753993034363
Accuracy on hold-out set: 0.47335
MulticlassAccuracy value on hold-out data: 0.47334998846054077
Epoch: 14
Loss on hold-out set: 1.3893168983459472
Accuracy on hold-out set: 0.4792
MulticlassAccuracy value on hold-out data: 0.47920000553131104
Epoch: 15
Loss on hold-out set: 1.3777840379714965
Accuracy on hold-out set: 0.4863
MulticlassAccuracy value on hold-out data: 0.486299991607666
Epoch: 16
Loss on hold-out set: 1.3935101521492004
Accuracy on hold-out set: 0.4791
MulticlassAccuracy value on hold-out data: 0.47909998893737793
Returned to Spot: Validation loss: 1.3935101521492004
----------------------------------------------
spotPython tuning: 1.1664946717262268 [######----] 60.45% 

config: {'l1': 32, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.6025432550430299
Accuracy on hold-out set: 0.4149
MulticlassAccuracy value on hold-out data: 0.414900004863739
Epoch: 2
Loss on hold-out set: 1.4515989580154418
Accuracy on hold-out set: 0.4684
MulticlassAccuracy value on hold-out data: 0.4684000015258789
Epoch: 3
Loss on hold-out set: 1.3799453285217285
Accuracy on hold-out set: 0.5046
MulticlassAccuracy value on hold-out data: 0.5045999884605408
Epoch: 4
Loss on hold-out set: 1.3721969090461732
Accuracy on hold-out set: 0.50745
MulticlassAccuracy value on hold-out data: 0.5074499845504761
Epoch: 5
Loss on hold-out set: 1.275218014717102
Accuracy on hold-out set: 0.54205
MulticlassAccuracy value on hold-out data: 0.5420500040054321
Epoch: 6
Loss on hold-out set: 1.2408557297706604
Accuracy on hold-out set: 0.56475
MulticlassAccuracy value on hold-out data: 0.5647500157356262
Epoch: 7
Loss on hold-out set: 1.217976010131836
Accuracy on hold-out set: 0.5657
MulticlassAccuracy value on hold-out data: 0.5656999945640564
Epoch: 8
Loss on hold-out set: 1.2025233350753783
Accuracy on hold-out set: 0.5818
MulticlassAccuracy value on hold-out data: 0.5817999839782715
Epoch: 9
Loss on hold-out set: 1.1648075584411621
Accuracy on hold-out set: 0.59075
MulticlassAccuracy value on hold-out data: 0.590749979019165
Epoch: 10
Loss on hold-out set: 1.1898816374778747
Accuracy on hold-out set: 0.58215
MulticlassAccuracy value on hold-out data: 0.5821499824523926
Epoch: 11
Loss on hold-out set: 1.1876069959640503
Accuracy on hold-out set: 0.5836
MulticlassAccuracy value on hold-out data: 0.5835999846458435
Epoch: 12
Loss on hold-out set: 1.153373938179016
Accuracy on hold-out set: 0.5995
MulticlassAccuracy value on hold-out data: 0.5995000004768372
Epoch: 13
Loss on hold-out set: 1.1510844681739807
Accuracy on hold-out set: 0.5977
MulticlassAccuracy value on hold-out data: 0.5976999998092651
Epoch: 14
Loss on hold-out set: 1.1490264488220214
Accuracy on hold-out set: 0.60155
MulticlassAccuracy value on hold-out data: 0.6015499830245972
Epoch: 15
Loss on hold-out set: 1.1566388602256774
Accuracy on hold-out set: 0.6024
MulticlassAccuracy value on hold-out data: 0.602400004863739
Epoch: 16
Loss on hold-out set: 1.1548289288520812
Accuracy on hold-out set: 0.60085
MulticlassAccuracy value on hold-out data: 0.600849986076355
Returned to Spot: Validation loss: 1.1548289288520812
----------------------------------------------
spotPython tuning: 1.1548289288520812 [#######---] 70.21% 

config: {'l1': 16, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adamax', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.565166037005186
Accuracy on hold-out set: 0.4338
MulticlassAccuracy value on hold-out data: 0.43380001187324524
Epoch: 2
Loss on hold-out set: 1.407238422137499
Accuracy on hold-out set: 0.4962
MulticlassAccuracy value on hold-out data: 0.49619999527931213
Epoch: 3
Loss on hold-out set: 1.3353772922366858
Accuracy on hold-out set: 0.52635
MulticlassAccuracy value on hold-out data: 0.5263500213623047
Epoch: 4
Loss on hold-out set: 1.2500376120194794
Accuracy on hold-out set: 0.55645
MulticlassAccuracy value on hold-out data: 0.5564500093460083
Epoch: 5
Loss on hold-out set: 1.2211679279513656
Accuracy on hold-out set: 0.5721
MulticlassAccuracy value on hold-out data: 0.5720999836921692
Epoch: 6
Loss on hold-out set: 1.226639595225826
Accuracy on hold-out set: 0.57695
MulticlassAccuracy value on hold-out data: 0.5769500136375427
Epoch: 7
Loss on hold-out set: 1.2179959335636348
Accuracy on hold-out set: 0.5766
MulticlassAccuracy value on hold-out data: 0.5766000151634216
Epoch: 8
Loss on hold-out set: 1.2281979153301568
Accuracy on hold-out set: 0.5817
MulticlassAccuracy value on hold-out data: 0.5817000269889832
Epoch: 9
Loss on hold-out set: 1.2719298387154936
Accuracy on hold-out set: 0.579
MulticlassAccuracy value on hold-out data: 0.5789999961853027
Epoch: 10
Loss on hold-out set: 1.219429868082516
Accuracy on hold-out set: 0.59685
MulticlassAccuracy value on hold-out data: 0.5968499779701233
Early stopping at epoch 9
Returned to Spot: Validation loss: 1.219429868082516
----------------------------------------------
spotPython tuning: 1.1548289288520812 [########--] 77.15% 

config: {'l1': 64, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5942431531906127
Accuracy on hold-out set: 0.40595
MulticlassAccuracy value on hold-out data: 0.40595000982284546
Epoch: 2
Loss on hold-out set: 1.4340217566490174
Accuracy on hold-out set: 0.4813
MulticlassAccuracy value on hold-out data: 0.4812999963760376
Epoch: 3
Loss on hold-out set: 1.3440862701416016
Accuracy on hold-out set: 0.5134
MulticlassAccuracy value on hold-out data: 0.5134000182151794
Epoch: 4
Loss on hold-out set: 1.2724248729705812
Accuracy on hold-out set: 0.54425
MulticlassAccuracy value on hold-out data: 0.5442500114440918
Epoch: 5
Loss on hold-out set: 1.2218507262229918
Accuracy on hold-out set: 0.5642
MulticlassAccuracy value on hold-out data: 0.5641999840736389
Epoch: 6
Loss on hold-out set: 1.183581609916687
Accuracy on hold-out set: 0.58515
MulticlassAccuracy value on hold-out data: 0.5851500034332275
Epoch: 7
Loss on hold-out set: 1.2213542881965638
Accuracy on hold-out set: 0.5706
MulticlassAccuracy value on hold-out data: 0.5705999732017517
Epoch: 8
Loss on hold-out set: 1.2021549138069152
Accuracy on hold-out set: 0.5764
MulticlassAccuracy value on hold-out data: 0.5763999819755554
Epoch: 9
Loss on hold-out set: 1.1232080152511597
Accuracy on hold-out set: 0.6063
MulticlassAccuracy value on hold-out data: 0.6062999963760376
Epoch: 10
Loss on hold-out set: 1.1270282478809357
Accuracy on hold-out set: 0.60285
MulticlassAccuracy value on hold-out data: 0.6028500199317932
Epoch: 11
Loss on hold-out set: 1.1303316102981567
Accuracy on hold-out set: 0.59775
MulticlassAccuracy value on hold-out data: 0.5977500081062317
Epoch: 12
Loss on hold-out set: 1.0843656971931457
Accuracy on hold-out set: 0.616
MulticlassAccuracy value on hold-out data: 0.6159999966621399
Epoch: 13
Loss on hold-out set: 1.1646690001487732
Accuracy on hold-out set: 0.60295
MulticlassAccuracy value on hold-out data: 0.6029499769210815
Epoch: 14
Loss on hold-out set: 1.1350477192401887
Accuracy on hold-out set: 0.60805
MulticlassAccuracy value on hold-out data: 0.6080499887466431
Epoch: 15
Loss on hold-out set: 1.1388844965934752
Accuracy on hold-out set: 0.60815
MulticlassAccuracy value on hold-out data: 0.6081500053405762
Early stopping at epoch 14
Returned to Spot: Validation loss: 1.1388844965934752
----------------------------------------------
spotPython tuning: 1.1388844965934752 [#########-] 86.51% 

config: {'l1': 512, 'l2': 256, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adam', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.4503090155005456
Accuracy on hold-out set: 0.47725
MulticlassAccuracy value on hold-out data: 0.47725000977516174
Epoch: 2
Loss on hold-out set: 1.465384444616735
Accuracy on hold-out set: 0.51715
MulticlassAccuracy value on hold-out data: 0.5171499848365784
Epoch: 3
Loss on hold-out set: 1.3661446234055794
Accuracy on hold-out set: 0.54795
MulticlassAccuracy value on hold-out data: 0.547950029373169
Epoch: 4
Loss on hold-out set: 1.2786859003394841
Accuracy on hold-out set: 0.57105
MulticlassAccuracy value on hold-out data: 0.5710499882698059
Epoch: 5
Loss on hold-out set: 1.3653297372542417
Accuracy on hold-out set: 0.57705
MulticlassAccuracy value on hold-out data: 0.577049970626831
Epoch: 6
Loss on hold-out set: 1.4640028541607797
Accuracy on hold-out set: 0.57755
MulticlassAccuracy value on hold-out data: 0.5775499939918518
Epoch: 7
Loss on hold-out set: 1.4530144686490865
Accuracy on hold-out set: 0.5928
MulticlassAccuracy value on hold-out data: 0.5928000211715698
Early stopping at epoch 6
Returned to Spot: Validation loss: 1.4530144686490865
----------------------------------------------
spotPython tuning: 1.1388844965934752 [#########-] 94.23% 

config: {'l1': 64, 'l2': 4, 'lr_mult': 1.0, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.7160225823402404
Accuracy on hold-out set: 0.32635
MulticlassAccuracy value on hold-out data: 0.32635000348091125
Epoch: 2
Loss on hold-out set: 1.493948697566986
Accuracy on hold-out set: 0.4523
MulticlassAccuracy value on hold-out data: 0.4523000121116638
Epoch: 3
Loss on hold-out set: 1.3879366069197654
Accuracy on hold-out set: 0.50215
MulticlassAccuracy value on hold-out data: 0.5021499991416931
Epoch: 4
Loss on hold-out set: 1.3560745789170265
Accuracy on hold-out set: 0.51215
MulticlassAccuracy value on hold-out data: 0.51214998960495
Epoch: 5
Loss on hold-out set: 1.2768551769018173
Accuracy on hold-out set: 0.54655
MulticlassAccuracy value on hold-out data: 0.5465499758720398
Epoch: 6
Loss on hold-out set: 1.2863004852354527
Accuracy on hold-out set: 0.54875
MulticlassAccuracy value on hold-out data: 0.5487499833106995
Epoch: 7
Loss on hold-out set: 1.3319906251192093
Accuracy on hold-out set: 0.5289
MulticlassAccuracy value on hold-out data: 0.5289000272750854
Epoch: 8
Loss on hold-out set: 1.24964777251482
Accuracy on hold-out set: 0.56875
MulticlassAccuracy value on hold-out data: 0.5687500238418579
Returned to Spot: Validation loss: 1.24964777251482
----------------------------------------------
spotPython tuning: 1.1388844965934752 [##########] 98.84% 

config: {'l1': 64, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1
Loss on hold-out set: 1.5556772226333617
Accuracy on hold-out set: 0.4349
MulticlassAccuracy value on hold-out data: 0.4348999857902527
Epoch: 2
Loss on hold-out set: 1.443882141494751
Accuracy on hold-out set: 0.4764
MulticlassAccuracy value on hold-out data: 0.4763999879360199
Epoch: 3
Loss on hold-out set: 1.3536285732269286
Accuracy on hold-out set: 0.50545
MulticlassAccuracy value on hold-out data: 0.5054500102996826
Epoch: 4
Loss on hold-out set: 1.2988192339897155
Accuracy on hold-out set: 0.5387
MulticlassAccuracy value on hold-out data: 0.5386999845504761
Epoch: 5
Loss on hold-out set: 1.2503215166091919
Accuracy on hold-out set: 0.554
MulticlassAccuracy value on hold-out data: 0.5540000200271606
Epoch: 6
Loss on hold-out set: 1.2349485918998717
Accuracy on hold-out set: 0.56155
MulticlassAccuracy value on hold-out data: 0.5615500211715698
Epoch: 7
Loss on hold-out set: 1.205826044178009
Accuracy on hold-out set: 0.57505
MulticlassAccuracy value on hold-out data: 0.5750499963760376
Epoch: 8
Loss on hold-out set: 1.1841682409286498
Accuracy on hold-out set: 0.5823
MulticlassAccuracy value on hold-out data: 0.5823000073432922
Returned to Spot: Validation loss: 1.1841682409286498
----------------------------------------------
spotPython tuning: 1.1388844965934752 [##########] 100.00% Done...
<spotPython.spot.spot.Spot at 0x2985a6b30>

14.13 Tensorboard

The textual output shown in the console (or code cell) can be visualized with Tensorboard.

14.13.1 Tensorboard: Start Tensorboard

Start TensorBoard through the command line to visualize data you logged. Specify the root log directory as used in fun_control = fun_control_init(task="regression", tensorboard_path="runs/24_spot_torch_regression") as the tensorboard_path. The argument logdir points to directory where TensorBoard will look to find event files that it can display. TensorBoard will recursively walk the directory structure rooted at logdir, looking for .tfevents. files.

tensorboard --logdir=runs

Go to the URL it provides or to http://localhost:6006/. The following figures show some screenshots of Tensorboard.

Figure 14.1: Tensorboard

Figure 14.2: Tensorboard

14.13.2 Saving the State of the Notebook

The state of the notebook can be saved and reloaded as follows:

import pickle
SAVE = False
LOAD = False

if SAVE:
    result_file_name = "res_" + experiment_name + ".pkl"
    with open(result_file_name, 'wb') as f:
        pickle.dump(spot_tuner, f)

if LOAD:
    result_file_name = "add_the_name_of_the_result_file_here.pkl"
    with open(result_file_name, 'rb') as f:
        spot_tuner =  pickle.load(f)

14.14 Results

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The following code generates the progress plot from ?fig-progress.

spot_tuner.plot_progress(log_y=False, 
    filename="./figures/" + experiment_name+"_progress.png")

Progress plot. Black dots denote results from the initial design. Red dots illustrate the improvement found by the surrogate model based optimization.

?fig-progress shows a typical behaviour that can be observed in many hyperparameter studies (Bartz et al. 2022): the largest improvement is obtained during the evaluation of the initial design. The surrogate model based optimization-optimization with the surrogate refines the results. ?fig-progress also illustrates one major difference between ray[tune] as used in PyTorch (2023a) and spotPython: the ray[tune] uses a random search and will generate results similar to the black dots, whereas spotPython uses a surrogate model based optimization and presents results represented by red dots in ?fig-progress. The surrogate model based optimization is considered to be more efficient than a random search, because the surrogate model guides the search towards promising regions in the hyperparameter space.

In addition to the improved (“optimized”) hyperparameter values, spotPython allows a statistical analysis, e.g., a sensitivity analysis, of the results. We can print the results of the hyperparameter tuning, see ?tbl-results. The table shows the hyperparameters, their types, default values, lower and upper bounds, and the transformation function. The column “tuned” shows the tuned values. The column “importance” shows the importance of the hyperparameters. The column “stars” shows the importance of the hyperparameters in stars. The importance is computed by the SPOT software.

from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control=fun_control, spot=spot_tuner))
| name         | type   | default   |   lower |   upper |   tuned | transform             |   importance | stars   |
|--------------|--------|-----------|---------|---------|---------|-----------------------|--------------|---------|
| l1           | int    | 5         |     2.0 |     9.0 |     6.0 | transform_power_2_int |       100.00 | ***     |
| l2           | int    | 5         |     2.0 |     9.0 |     5.0 | transform_power_2_int |        21.53 | *       |
| lr_mult      | float  | 1.0       |     1.0 |     1.0 |     1.0 | None                  |         0.00 |         |
| batch_size   | int    | 4         |     1.0 |     5.0 |     5.0 | transform_power_2_int |         0.00 |         |
| epochs       | int    | 3         |     3.0 |     4.0 |     4.0 | transform_power_2_int |         0.05 |         |
| k_folds      | int    | 1         |     0.0 |     0.0 |     0.0 | None                  |         0.00 |         |
| patience     | int    | 5         |     3.0 |     3.0 |     3.0 | None                  |         0.00 |         |
| optimizer    | factor | SGD       |     0.0 |     9.0 |     3.0 | None                  |         0.06 |         |
| sgd_momentum | float  | 0.0       |     0.9 |     0.9 |     0.9 | None                  |         0.00 |         |

To visualize the most important hyperparameters, spotPython provides the function plot_importance. The following code generates the importance plot from ?fig-importance.

spot_tuner.plot_importance(threshold=0.025,
    filename="./figures/" + experiment_name+"_importance.png")

Variable importance plot, threshold 0.025.

14.14.1 Get the Tuned Architecture (SPOT Results)

The architecture of the spotPython model can be obtained as follows. First, the numerical representation of the hyperparameters are obtained, i.e., the numpy array X is generated. This array is then used to generate the model model_spot by the function get_one_core_model_from_X. The model model_spot has the following architecture:

from spotPython.hyperparameters.values import get_one_core_model_from_X
X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
model_spot = get_one_core_model_from_X(X, fun_control)
model_spot
Net_CIFAR10(
  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=64, bias=True)
  (fc2): Linear(in_features=64, out_features=32, bias=True)
  (fc3): Linear(in_features=32, out_features=10, bias=True)
)

14.14.2 Get Default Hyperparameters

In a similar manner as in ?sec-get-spot-results, the default hyperparameters can be obtained.

# fun_control was modified, we generate a new one with the original 
# default hyperparameters
from spotPython.hyperparameters.values import get_one_core_model_from_X
fc = fun_control
fc.update({"core_model_hyper_dict":
    hyper_dict[fun_control["core_model"].__name__]})
model_default = get_one_core_model_from_X(X_start, fun_control=fc)
model_default
Net_CIFAR10(
  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=32, bias=True)
  (fc2): Linear(in_features=32, out_features=32, bias=True)
  (fc3): Linear(in_features=32, out_features=10, bias=True)
)

14.14.3 Evaluation of the Default Architecture

The method train_tuned takes a model architecture without trained weights and trains this model with the train data. The train data is split into train and validation data. The validation data is used for early stopping. The trained model weights are saved as a dictionary.

This evaluation is similar to the final evaluation in PyTorch (2023a).

from spotPython.torch.traintest import (
    train_tuned,
    test_tuned,
    )
train_tuned(net=model_default, train_dataset=train, shuffle=True,
        loss_function=fun_control["loss_function"],
        metric=fun_control["metric_torch"],
        device = fun_control["device"], show_batch_interval=1_000_000,
        path=None,
        task=fun_control["task"],)

test_tuned(net=model_default, test_dataset=test, 
        loss_function=fun_control["loss_function"],
        metric=fun_control["metric_torch"],
        shuffle=False, 
        device = fun_control["device"],
        task=fun_control["task"],)        
Epoch: 1
Loss on hold-out set: 2.3072064586639405
Accuracy on hold-out set: 0.10675
MulticlassAccuracy value on hold-out data: 0.1067499965429306
Epoch: 2
Loss on hold-out set: 2.303994348716736
Accuracy on hold-out set: 0.10635
MulticlassAccuracy value on hold-out data: 0.10634999722242355
Epoch: 3
Loss on hold-out set: 2.3012452766418456
Accuracy on hold-out set: 0.10855
MulticlassAccuracy value on hold-out data: 0.10854999721050262
Epoch: 4
Loss on hold-out set: 2.2982975690841676
Accuracy on hold-out set: 0.12
MulticlassAccuracy value on hold-out data: 0.11999999731779099
Epoch: 5
Loss on hold-out set: 2.294377840232849
Accuracy on hold-out set: 0.14495
MulticlassAccuracy value on hold-out data: 0.1449500024318695
Epoch: 6
Loss on hold-out set: 2.2877407669067384
Accuracy on hold-out set: 0.15335
MulticlassAccuracy value on hold-out data: 0.15334999561309814
Epoch: 7
Loss on hold-out set: 2.270779088020325
Accuracy on hold-out set: 0.1859
MulticlassAccuracy value on hold-out data: 0.1859000027179718
Epoch: 8
Loss on hold-out set: 2.215397458076477
Accuracy on hold-out set: 0.21195
MulticlassAccuracy value on hold-out data: 0.21195000410079956
Returned to Spot: Validation loss: 2.215397458076477
----------------------------------------------
Loss on hold-out set: 2.214467565917969
Accuracy on hold-out set: 0.2169
MulticlassAccuracy value on hold-out data: 0.21690000593662262
Final evaluation: Validation loss: 2.214467565917969
Final evaluation: Validation metric: 0.21690000593662262
----------------------------------------------
(2.214467565917969, nan, tensor(0.2169))

14.14.4 Evaluation of the Tuned Architecture

The following code trains the model model_spot.

If path is set to a filename, e.g., path = "model_spot_trained.pt", the weights of the trained model will be saved to this file.

If path is set to a filename, e.g., path = "model_spot_trained.pt", the weights of the trained model will be loaded from this file.

train_tuned(net=model_spot, train_dataset=train,
        loss_function=fun_control["loss_function"],
        metric=fun_control["metric_torch"],
        shuffle=True,
        device = fun_control["device"],
        path=None,
        task=fun_control["task"],)
test_tuned(net=model_spot, test_dataset=test,
            shuffle=False,
            loss_function=fun_control["loss_function"],
            metric=fun_control["metric_torch"],
            device = fun_control["device"],
            task=fun_control["task"],)
Epoch: 1
Loss on hold-out set: 1.5381738952636719
Accuracy on hold-out set: 0.43645
MulticlassAccuracy value on hold-out data: 0.4364500045776367
Epoch: 2
Loss on hold-out set: 1.4230143032073974
Accuracy on hold-out set: 0.48145
MulticlassAccuracy value on hold-out data: 0.48144999146461487
Epoch: 3
Loss on hold-out set: 1.3266478410720826
Accuracy on hold-out set: 0.5233
MulticlassAccuracy value on hold-out data: 0.5232999920845032
Epoch: 4
Loss on hold-out set: 1.2738251304626464
Accuracy on hold-out set: 0.54255
MulticlassAccuracy value on hold-out data: 0.5425500273704529
Epoch: 5
Loss on hold-out set: 1.260533795928955
Accuracy on hold-out set: 0.5562
MulticlassAccuracy value on hold-out data: 0.5562000274658203
Epoch: 6
Loss on hold-out set: 1.2176606630325317
Accuracy on hold-out set: 0.56935
MulticlassAccuracy value on hold-out data: 0.569350004196167
Epoch: 7
Loss on hold-out set: 1.1835321116447448
Accuracy on hold-out set: 0.58395
MulticlassAccuracy value on hold-out data: 0.5839499831199646
Epoch: 8
Loss on hold-out set: 1.195100956439972
Accuracy on hold-out set: 0.5806
MulticlassAccuracy value on hold-out data: 0.5806000232696533
Epoch: 9
Loss on hold-out set: 1.2124000025749206
Accuracy on hold-out set: 0.577
MulticlassAccuracy value on hold-out data: 0.5770000219345093
Epoch: 10
Loss on hold-out set: 1.1470019676208496
Accuracy on hold-out set: 0.5971
MulticlassAccuracy value on hold-out data: 0.597100019454956
Epoch: 11
Loss on hold-out set: 1.1803068031311035
Accuracy on hold-out set: 0.5901
MulticlassAccuracy value on hold-out data: 0.5900999903678894
Epoch: 12
Loss on hold-out set: 1.1421940431594848
Accuracy on hold-out set: 0.60315
MulticlassAccuracy value on hold-out data: 0.6031500101089478
Epoch: 13
Loss on hold-out set: 1.1567931615829468
Accuracy on hold-out set: 0.5959
MulticlassAccuracy value on hold-out data: 0.5958999991416931
Epoch: 14
Loss on hold-out set: 1.145644029521942
Accuracy on hold-out set: 0.60195
MulticlassAccuracy value on hold-out data: 0.6019499897956848
Epoch: 15
Loss on hold-out set: 1.1257336742401123
Accuracy on hold-out set: 0.6082
MulticlassAccuracy value on hold-out data: 0.6082000136375427
Epoch: 16
Loss on hold-out set: 1.1485932287216187
Accuracy on hold-out set: 0.61135
MulticlassAccuracy value on hold-out data: 0.6113499999046326
Returned to Spot: Validation loss: 1.1485932287216187
----------------------------------------------
Loss on hold-out set: 1.1416264067823514
Accuracy on hold-out set: 0.6139
MulticlassAccuracy value on hold-out data: 0.6139000058174133
Final evaluation: Validation loss: 1.1416264067823514
Final evaluation: Validation metric: 0.6139000058174133
----------------------------------------------
(1.1416264067823514, nan, tensor(0.6139))

14.14.5 Detailed Hyperparameter Plots

The contour plots in this section visualize the interactions of the three most important hyperparameters. Since some of these hyperparameters take fatorial or integer values, sometimes step-like fitness landcapes (or response surfaces) are generated. SPOT draws the interactions of the main hyperparameters by default. It is also possible to visualize all interactions.

filename = "./figures/" + experiment_name
spot_tuner.plot_important_hyperparameter_contour(filename=filename)
l1:  100.0
l2:  21.526495340452232
epochs:  0.04625101267179485
optimizer:  0.0641650757520071

Contour plots.

The figures (?fig-contour) show the contour plots of the loss as a function of the hyperparameters. These plots are very helpful for benchmark studies and for understanding neural networks. spotPython provides additional tools for a visual inspection of the results and give valuable insights into the hyperparameter tuning process. This is especially useful for model explainability, transparency, and trustworthiness. In addition to the contour plots, ?fig-parallel shows the parallel plot of the hyperparameters.

spot_tuner.parallel_plot()

Parallel coordinates plots

14.15 Summary and Outlook

This tutorial presents the hyperparameter tuning open source software spotPython for PyTorch. To show its basic features, a comparison with the “official” PyTorch hyperparameter tuning tutorial (PyTorch 2023a) is presented. Some of the advantages of spotPython are:

  • Numerical and categorical hyperparameters.
  • Powerful surrogate models.
  • Flexible approach and easy to use.
  • Simple JSON files for the specification of the hyperparameters.
  • Extension of default and user specified network classes.
  • Noise handling techniques.
  • Interaction with tensorboard.

Currently, only rudimentary parallel and distributed neural network training is possible, but these capabilities will be extended in the future. The next version of spotPython will also include a more detailed documentation and more examples.

Important

Important: This tutorial does not present a complete benchmarking study (Bartz-Beielstein et al. 2020). The results are only preliminary and highly dependent on the local configuration (hard- and software). Our goal is to provide a first impression of the performance of the hyperparameter tuning package spotPython. To demonstrate its capabilities, a quick comparison with ray[tune] was performed. ray[tune] was chosen, because it is presented as “an industry standard tool for distributed hyperparameter tuning.” The results should be interpreted with care.

14.16 Appendix

14.16.1 Sample Output From Ray Tune’s Run

The output from ray[tune] could look like this (PyTorch 2023b):

Number of trials: 10 (10 TERMINATED)
------+------+-------------+--------------+---------+------------+--------------------+
|   l1 |   l2 |          lr |   batch_size |    loss |   accuracy | training_iteration |
+------+------+-------------+--------------+---------+------------+--------------------|
|   64 |    4 | 0.00011629  |            2 | 1.87273 |     0.244  |                  2 |
|   32 |   64 | 0.000339763 |            8 | 1.23603 |     0.567  |                  8 |
|    8 |   16 | 0.00276249  |           16 | 1.1815  |     0.5836 |                 10 |
|    4 |   64 | 0.000648721 |            4 | 1.31131 |     0.5224 |                  8 |
|   32 |   16 | 0.000340753 |            8 | 1.26454 |     0.5444 |                  8 |
|    8 |    4 | 0.000699775 |            8 | 1.99594 |     0.1983 |                  2 |
|  256 |    8 | 0.0839654   |           16 | 2.3119  |     0.0993 |                  1 |
|   16 |  128 | 0.0758154   |           16 | 2.33575 |     0.1327 |                  1 |
|   16 |    8 | 0.0763312   |           16 | 2.31129 |     0.1042 |                  4 |
|  128 |   16 | 0.000124903 |            4 | 2.26917 |     0.1945 |                  1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+
Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806

  1. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.↩︎

  2. We were not able to install Ray Tune on our system. Therefore, we used the results from the PyTorch tutorial.↩︎