list | grep "spot[RiverPython]" pip
spotPython 0.2.37
spotRiver 0.0.93
Note: you may need to restart the kernel to use updated packages.
In this tutorial, we will show how spotPython
can be integrated into the PyTorch
training workflow.
This document refers to the following software versions:
python
: 3.10.10torch
: 2.0.1torchvision
: 0.15.0list | grep "spot[RiverPython]" pip
spotPython 0.2.37
spotRiver 0.0.93
Note: you may need to restart the kernel to use updated packages.
spotPython
can be installed via pip. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.
!pip install spotPython
spotPython
from gitHub.# import sys
# !{sys.executable} -m pip install --upgrade build
# !{sys.executable} -m pip install --upgrade --force-reinstall spotPython
Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.
DEVICE
."cpu"
is preferred (on Mac)."cuda:0"
instead.None
, spotPython
will automatically select the device.
"mps"
on Macs, which is not the best choice for simple neural nets.= 1
MAX_TIME = 5
INIT_SIZE = "cpu" # "cuda:0" DEVICE
from spotPython.utils.device import getDevice
= getDevice(DEVICE)
DEVICE print(DEVICE)
cpu
import os
import copy
import socket
from datetime import datetime
from dateutil.tz import tzlocal
= datetime.now(tzlocal())
start_time = socket.gethostname().split(".")[0]
HOSTNAME = '11-torch' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
experiment_name = experiment_name.replace(':', '-')
experiment_name print(experiment_name)
if not os.path.exists('./figures'):
'./figures') os.makedirs(
11-torch_bartz09_1min_5init_2023-06-18_18-26-09
fun_control
DictionaryspotPython
uses a Python dictionary for storing the information required for the hyperparameter tuning process, which was described in Section 14.2.
from spotPython.utils.init import fun_control_init
= fun_control_init(task="classification",
fun_control ="runs/11_spot_hpt_torch_fashion_mnist",
tensorboard_path=DEVICE) device
from torchvision import datasets, transforms
from torchvision.transforms import ToTensor
def load_data(data_dir="./data"):
# Download training data from open datasets.
= datasets.FashionMNIST(
training_data =data_dir,
root=True,
train=True,
download=ToTensor(),
transform
)# Download test data from open datasets.
= datasets.FashionMNIST(
test_data =data_dir,
root=False,
train=True,
download=ToTensor(),
transform
)return training_data, test_data
= load_data()
train, test train.data.shape, test.data.shape
(torch.Size([60000, 28, 28]), torch.Size([10000, 28, 28]))
= len(train)
n_samples # add the dataset to the fun_control
"data": None,
fun_control.update({"train": train,
"test": test,
"n_samples": n_samples,
"target_column": None})
After the training and test data are specified and added to the fun_control
dictionary, spotPython
allows the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables, see Section 14.4. This feature is not used here, so we do not change the default value (which is None
).
algorithm
) and core_model_hyper_dict
spotPython
implements a class which is similar to the class described in the PyTorch tutorial. The class is called Net_fashionMNIST
and is implemented in the file netfashionMNIST.py
. The class is imported here.
from torch import nn
import spotPython.torch.netcore as netcore
class Net_fashionMNIST(netcore.Net_Core):
def __init__(self, l1, l2, lr_mult, batch_size, epochs, k_folds, patience, optimizer, sgd_momentum):
super(Net_fashionMNIST, self).__init__(
=lr_mult,
lr_mult=batch_size,
batch_size=epochs,
epochs=k_folds,
k_folds=patience,
patience=optimizer,
optimizer=sgd_momentum,
sgd_momentum
)self.flatten = nn.Flatten()
self.linear_relu_stack = nn.Sequential(
28 * 28, l1),
nn.Linear(
nn.ReLU(),
nn.Linear(l1, l2),
nn.ReLU(),10)
nn.Linear(l2,
)
def forward(self, x):
= self.flatten(x)
x = self.linear_relu_stack(x)
logits return logits
This class inherits from the class Net_Core
which is implemented in the file netcore.py
, see Section 14.5.1.
from spotPython.data.torch_hyper_dict import TorchHyperDict
from spotPython.torch.netfashionMNIST import Net_fashionMNIST
from spotPython.hyperparameters.values import add_core_model_to_fun_control
= add_core_model_to_fun_control(core_model=Net_fashionMNIST,
fun_control =fun_control,
fun_control=TorchHyperDict,
hyper_dict=None) filename
hyper_dict
Hyperparameters for the Selected AlgorithmspotPython
uses JSON
files for the specification of the hyperparameters, which were described in Section 14.5.5.
The corresponding entries for the Net_fashionMNIST
class are shown below.
"Net_fashionMNIST":
{
"l1": {
"type": "int",
"default": 5,
"transform": "transform_power_2_int",
"lower": 2,
"upper": 9},
"l2": {
"type": "int",
"default": 5,
"transform": "transform_power_2_int",
"lower": 2,
"upper": 9},
"lr_mult": {
"type": "float",
"default": 1.0,
"transform": "None",
"lower": 0.1,
"upper": 10.0},
"batch_size": {
"type": "int",
"default": 4,
"transform": "transform_power_2_int",
"lower": 1,
"upper": 4},
"epochs": {
"type": "int",
"default": 3,
"transform": "transform_power_2_int",
"lower": 3,
"upper": 4},
"k_folds": {
"type": "int",
"default": 1,
"transform": "None",
"lower": 1,
"upper": 1},
"patience": {
"type": "int",
"default": 5,
"transform": "None",
"lower": 2,
"upper": 10
},
"optimizer": {
"levels": ["Adadelta",
"Adagrad",
"Adam",
"AdamW",
"SparseAdam",
"Adamax",
"ASGD",
"NAdam",
"RAdam",
"RMSprop",
"Rprop",
"SGD"],
"type": "factor",
"default": "SGD",
"transform": "None",
"core_model_parameter_type": "str",
"lower": 0,
"upper": 12},
"sgd_momentum": {
"type": "float",
"default": 0.0,
"transform": "None",
"lower": 0.0,
"upper": 1.0}
},
hyper_dict
Hyperparameters for the Selected Algorithm aka core_model
spotPython
provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 14.6.
The hyperparameter k_folds
is not used, it is de-activated here by setting the lower and upper bound to the same value.
l1
and l2
as well as epochs
and patience
are set to small values for demonstration purposes. These values are too small for a real application.fun_control = modify_hyper_parameter_bounds(fun_control, "l1", bounds=[2, 7])
fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[7, 9])
andfun_control = modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 7])
from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
= modify_hyper_parameter_bounds(fun_control, "k_folds", bounds=[0, 0])
fun_control = modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 2])
fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[2, 3])
fun_control = modify_hyper_parameter_bounds(fun_control, "l1", bounds=[2, 5])
fun_control = modify_hyper_parameter_bounds(fun_control, "l2", bounds=[2, 5]) fun_control
from spotPython.hyperparameters.values import modify_hyper_parameter_levels
= modify_hyper_parameter_levels(fun_control, "optimizer",["Adam", "AdamW", "Adamax", "NAdam"]) fun_control
Optimizers are described in Section 14.6.1.
= modify_hyper_parameter_bounds(fun_control,
fun_control "lr_mult", bounds=[1e-3, 1e-3])
= modify_hyper_parameter_bounds(fun_control,
fun_control "sgd_momentum", bounds=[0.9, 0.9])
The evaluation procedure requires the specification of two elements:
These are described in Section 19.7.1.
The key "loss_function"
specifies the loss function which is used during the optimization, see Section 14.7.5.
We will use CrossEntropy loss for the multiclass-classification task.
from torch.nn import CrossEntropyLoss
= CrossEntropyLoss()
loss_function
fun_control.update({"loss_function": loss_function,
"shuffle": True,
"eval": "train_hold_out"
})
from torchmetrics import Accuracy
= Accuracy(task="multiclass", num_classes=10).to(fun_control["device"])
metric_torch "metric_torch": metric_torch}) fun_control.update({
The following code passes the information about the parameter ranges and bounds to spot
.
# extract the variable types, names, and bounds
from spotPython.hyperparameters.values import (get_bound_values,
get_var_name,
get_var_type,)= get_var_type(fun_control)
var_type = get_var_name(fun_control)
var_name "var_type": var_type,
fun_control.update({"var_name": var_name})
= get_bound_values(fun_control, "lower")
lower = get_bound_values(fun_control, "upper") upper
from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control))
| name | type | default | lower | upper | transform |
|--------------|--------|-----------|---------|---------|-----------------------|
| l1 | int | 5 | 2 | 5 | transform_power_2_int |
| l2 | int | 5 | 2 | 5 | transform_power_2_int |
| lr_mult | float | 1.0 | 0.001 | 0.001 | None |
| batch_size | int | 4 | 1 | 4 | transform_power_2_int |
| epochs | int | 3 | 2 | 3 | transform_power_2_int |
| k_folds | int | 1 | 0 | 0 | None |
| patience | int | 5 | 2 | 2 | None |
| optimizer | factor | SGD | 0 | 3 | None |
| sgd_momentum | float | 0.0 | 0.9 | 0.9 | None |
fun_torch
The objective function fun_torch
is selected next. It implements an interface from PyTorch
’s training, validation, and testing methods to spotPython
.
from spotPython.fun.hypertorch import HyperTorch
= HyperTorch().fun_torch fun
import numpy as np
from spotPython.spot import spot
from math import inf
= spot.Spot(fun=fun,
spot_tuner = lower,
lower = upper,
upper = inf,
fun_evals = 1,
fun_repeats = MAX_TIME,
max_time = False,
noise = np.sqrt(np.spacing(1)),
tolerance_x = var_type,
var_type = var_name,
var_name = "y",
infill_criterion = 1,
n_points =123,
seed= 50,
log_level = False,
show_models= True,
show_progress= fun_control,
fun_control ={"init_size": INIT_SIZE,
design_control"repeats": 1},
={"noise": True,
surrogate_control"cod_type": "norm",
"min_theta": -4,
"max_theta": 3,
"n_theta": len(var_name),
"model_fun_evals": 10_000,
"log_level": 50
})=X_start) spot_tuner.run(X_start
config: {'l1': 16, 'l2': 8, 'lr_mult': 0.001, 'batch_size': 16, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1 |
MulticlassAccuracy: 0.1741250008344650 | Loss: 2.3109110913276671 | Acc: 0.1741250000000000.
Epoch: 2 |
MulticlassAccuracy: 0.1715833395719528 | Loss: 2.2910392692883810 | Acc: 0.1715833333333333.
Epoch: 3 |
MulticlassAccuracy: 0.1730833351612091 | Loss: 2.2703858129183452 | Acc: 0.1730833333333333.
Epoch: 4 |
MulticlassAccuracy: 0.1771250069141388 | Loss: 2.2447114669481913 | Acc: 0.1771250000000000.
Epoch: 5 |
MulticlassAccuracy: 0.1965000033378601 | Loss: 2.2187511997222900 | Acc: 0.1965000000000000.
Epoch: 6 |
MulticlassAccuracy: 0.2387916594743729 | Loss: 2.1936514188448588 | Acc: 0.2387916666666667.
Epoch: 7 |
MulticlassAccuracy: 0.2870000004768372 | Loss: 2.1691776240666707 | Acc: 0.2870000000000000.
Epoch: 8 |
MulticlassAccuracy: 0.3211250007152557 | Loss: 2.1450629870096845 | Acc: 0.3211250000000000.
Returned to Spot: Validation loss: 2.1450629870096845
config: {'l1': 8, 'l2': 8, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 4, 'k_folds': 0, 'patience': 2, 'optimizer': 'Adamax', 'sgd_momentum': 0.9}
Epoch: 1 |
MulticlassAccuracy: 0.1249166652560234 | Loss: 2.2959158184528352 | Acc: 0.1249166666666667.
Epoch: 2 |
MulticlassAccuracy: 0.1256249994039536 | Loss: 2.2819335967699685 | Acc: 0.1256250000000000.
Epoch: 3 |
MulticlassAccuracy: 0.1418333351612091 | Loss: 2.2667567798296608 | Acc: 0.1418333333333333.
Epoch: 4 |
MulticlassAccuracy: 0.1923750042915344 | Loss: 2.2518876403967538 | Acc: 0.1923750000000000.
Returned to Spot: Validation loss: 2.251887640396754
config: {'l1': 32, 'l2': 16, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1 |
MulticlassAccuracy: 0.2989583313465118 | Loss: 2.1220241287549335 | Acc: 0.2989583333333333.
Epoch: 2 |
MulticlassAccuracy: 0.4264166653156281 | Loss: 1.9027830239633718 | Acc: 0.4264166666666667.
Epoch: 3 |
MulticlassAccuracy: 0.4466249942779541 | Loss: 1.6658251576721668 | Acc: 0.4466250000000000.
Epoch: 4 |
MulticlassAccuracy: 0.4940416812896729 | Loss: 1.4468630692561468 | Acc: 0.4940416666666667.
Epoch: 5 |
MulticlassAccuracy: 0.5954583287239075 | Loss: 1.2635337122765680 | Acc: 0.5954583333333333.
Epoch: 6 |
MulticlassAccuracy: 0.6301666498184204 | Loss: 1.1272197062385578 | Acc: 0.6301666666666667.
Epoch: 7 |
MulticlassAccuracy: 0.6424166560173035 | Loss: 1.0278998744903753 | Acc: 0.6424166666666666.
Epoch: 8 |
MulticlassAccuracy: 0.6501666903495789 | Loss: 0.9577597779563317 | Acc: 0.6501666666666667.
Returned to Spot: Validation loss: 0.9577597779563317
config: {'l1': 4, 'l2': 8, 'lr_mult': 0.001, 'batch_size': 4, 'epochs': 4, 'k_folds': 0, 'patience': 2, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1 |
MulticlassAccuracy: 0.0986249968409538 | Loss: 2.3216527020335196 | Acc: 0.0986250000000000.
Epoch: 2 |
MulticlassAccuracy: 0.0994166657328606 | Loss: 2.3144613005518915 | Acc: 0.0994166666666667.
Epoch: 3 |
MulticlassAccuracy: 0.1108750030398369 | Loss: 2.3077332268158597 | Acc: 0.1108750000000000.
Epoch: 4 |
MulticlassAccuracy: 0.1649166643619537 | Loss: 2.2978090685009955 | Acc: 0.1649166666666667.
Returned to Spot: Validation loss: 2.2978090685009955
config: {'l1': 16, 'l2': 32, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'Adam', 'sgd_momentum': 0.9}
Epoch: 1 |
MulticlassAccuracy: 0.1350833326578140 | Loss: 2.2859470740954082 | Acc: 0.1350833333333333.
Epoch: 2 |
MulticlassAccuracy: 0.1572916656732559 | Loss: 2.2582766885757448 | Acc: 0.1572916666666667.
Epoch: 3 |
MulticlassAccuracy: 0.2042083293199539 | Loss: 2.2206204667091369 | Acc: 0.2042083333333333.
Epoch: 4 |
MulticlassAccuracy: 0.2462916672229767 | Loss: 2.1780837457180025 | Acc: 0.2462916666666667.
Epoch: 5 |
MulticlassAccuracy: 0.3529166579246521 | Loss: 2.1308557176987328 | Acc: 0.3529166666666667.
Epoch: 6 |
MulticlassAccuracy: 0.4033749997615814 | Loss: 2.0807629495859148 | Acc: 0.4033750000000000.
Epoch: 7 |
MulticlassAccuracy: 0.4294166564941406 | Loss: 2.0288179952700931 | Acc: 0.4294166666666667.
Epoch: 8 |
MulticlassAccuracy: 0.4579583406448364 | Loss: 1.9751288238366445 | Acc: 0.4579583333333334.
Returned to Spot: Validation loss: 1.9751288238366445
config: {'l1': 8, 'l2': 16, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1 |
MulticlassAccuracy: 0.1192083358764648 | Loss: 2.2743606491883597 | Acc: 0.1192083333333333.
Epoch: 2 |
MulticlassAccuracy: 0.1605000048875809 | Loss: 2.2305470862785977 | Acc: 0.1605000000000000.
Epoch: 3 |
MulticlassAccuracy: 0.1834166646003723 | Loss: 2.1861692521969478 | Acc: 0.1834166666666667.
Epoch: 4 |
MulticlassAccuracy: 0.2228333353996277 | Loss: 2.1395828361511229 | Acc: 0.2228333333333333.
Epoch: 5 |
MulticlassAccuracy: 0.2688750028610229 | Loss: 2.0910682301521302 | Acc: 0.2688750000000000.
Epoch: 6 |
MulticlassAccuracy: 0.3116250038146973 | Loss: 2.0412952810923257 | Acc: 0.3116250000000000.
Epoch: 7 |
MulticlassAccuracy: 0.3353333473205566 | Loss: 1.9898198271592458 | Acc: 0.3353333333333333.
Epoch: 8 |
MulticlassAccuracy: 0.3470416665077209 | Loss: 1.9367182476520539 | Acc: 0.3470416666666667.
Returned to Spot: Validation loss: 1.936718247652054
spotPython tuning: 0.9577597779563317 [####------] 44.71%
config: {'l1': 16, 'l2': 16, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 4, 'k_folds': 0, 'patience': 2, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1 |
MulticlassAccuracy: 0.1799166649580002 | Loss: 2.2612573777635894 | Acc: 0.1799166666666667.
Epoch: 2 |
MulticlassAccuracy: 0.2322083264589310 | Loss: 2.2095232503712179 | Acc: 0.2322083333333333.
Epoch: 3 |
MulticlassAccuracy: 0.2866249978542328 | Loss: 2.1415398869315783 | Acc: 0.2866250000000000.
Epoch: 4 |
MulticlassAccuracy: 0.3446666598320007 | Loss: 2.0612082856297493 | Acc: 0.3446666666666667.
Returned to Spot: Validation loss: 2.0612082856297493
spotPython tuning: 0.9577597779563317 [##########] 100.00% Done...
<spotPython.spot.spot.Spot at 0x2942afaf0>
The textual output shown in the console (or code cell) can be visualized with Tensorboard as described in Section 14.9, see also the description in the documentation: Tensorboard.
After the hyperparameter tuning run is finished, the results can be analyzed as described in Section 14.10.
= False
SAVE = False
LOAD
if SAVE:
= "res_" + experiment_name + ".pkl"
result_file_name with open(result_file_name, 'wb') as f:
pickle.dump(spot_tuner, f)
if LOAD:
= "ADD THE NAME here, e.g.: res_ch10-friedman-hpt-0_maans03_60min_20init_1K_2023-04-14_10-11-19.pkl"
result_file_name with open(result_file_name, 'rb') as f:
= pickle.load(f) spot_tuner
After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The following code generates the progress plot from ?fig-progress.
=False,
spot_tuner.plot_progress(log_y="./figures/" + experiment_name+"_progress.png") filename
print(gen_design_table(fun_control=fun_control,
=spot_tuner)) spot
| name | type | default | lower | upper | tuned | transform | importance | stars |
|--------------|--------|-----------|---------|---------|---------|-----------------------|--------------|---------|
| l1 | int | 5 | 2.0 | 5.0 | 5.0 | transform_power_2_int | 8.64 | * |
| l2 | int | 5 | 2.0 | 5.0 | 4.0 | transform_power_2_int | 0.00 | |
| lr_mult | float | 1.0 | 0.001 | 0.001 | 0.001 | None | 0.00 | |
| batch_size | int | 4 | 1.0 | 4.0 | 1.0 | transform_power_2_int | 0.05 | |
| epochs | int | 3 | 2.0 | 3.0 | 3.0 | transform_power_2_int | 0.00 | |
| k_folds | int | 1 | 0.0 | 0.0 | 0.0 | None | 0.00 | |
| patience | int | 5 | 2.0 | 2.0 | 2.0 | None | 0.00 | |
| optimizer | factor | SGD | 0.0 | 3.0 | 3.0 | None | 100.00 | *** |
| sgd_momentum | float | 0.0 | 0.9 | 0.9 | 0.9 | None | 0.00 | |
=0.025, filename="./figures/" + experiment_name+"_importance.png") spot_tuner.plot_importance(threshold
The architecture of the spotPython
model can be obtained by the following code:
from spotPython.hyperparameters.values import get_one_core_model_from_X
= spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
X = get_one_core_model_from_X(X, fun_control)
model_spot model_spot
Net_fashionMNIST(
(flatten): Flatten(start_dim=1, end_dim=-1)
(linear_relu_stack): Sequential(
(0): Linear(in_features=784, out_features=32, bias=True)
(1): ReLU()
(2): Linear(in_features=32, out_features=16, bias=True)
(3): ReLU()
(4): Linear(in_features=16, out_features=10, bias=True)
)
)
= fun_control
fc "core_model_hyper_dict":
fc.update({"core_model"].__name__]})
hyper_dict[fun_control[= get_one_core_model_from_X(X_start, fun_control=fc)
model_default model_default
Net_fashionMNIST(
(flatten): Flatten(start_dim=1, end_dim=-1)
(linear_relu_stack): Sequential(
(0): Linear(in_features=784, out_features=32, bias=True)
(1): ReLU()
(2): Linear(in_features=32, out_features=32, bias=True)
(3): ReLU()
(4): Linear(in_features=32, out_features=10, bias=True)
)
)
The method train_tuned
takes a model architecture without trained weights and trains this model with the train data. The train data is split into train and validation data. The validation data is used for early stopping. The trained model weights are saved as a dictionary.
from spotPython.torch.traintest import train_tuned
=model_default, train_dataset=train, shuffle=True,
train_tuned(net=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric= fun_control["device"],
device =1_000_000,
show_batch_interval=None,
path=fun_control["task"]) task
Epoch: 1 |
MulticlassAccuracy: 0.4554583430290222 | Loss: 2.0432237401803333 | Acc: 0.4554583333333334.
Epoch: 2 |
MulticlassAccuracy: 0.5426250100135803 | Loss: 1.5820121876398723 | Acc: 0.5426250000000000.
Epoch: 3 |
MulticlassAccuracy: 0.6052083373069763 | Loss: 1.3004284824927648 | Acc: 0.6052083333333333.
Epoch: 4 |
MulticlassAccuracy: 0.6294999718666077 | Loss: 1.1371406654914220 | Acc: 0.6294999999999999.
Epoch: 5 |
MulticlassAccuracy: 0.6450833082199097 | Loss: 1.0319149243036907 | Acc: 0.6450833333333333.
Epoch: 6 |
MulticlassAccuracy: 0.6532083153724670 | Loss: 0.9602405934532483 | Acc: 0.6532083333333333.
Epoch: 7 |
MulticlassAccuracy: 0.6605416536331177 | Loss: 0.9097962180574735 | Acc: 0.6605416666666667.
Epoch: 8 |
MulticlassAccuracy: 0.6669999957084656 | Loss: 0.8731928959290186 | Acc: 0.6670000000000000.
Returned to Spot: Validation loss: 0.8731928959290186
from spotPython.torch.traintest import test_tuned
=model_default, test_dataset=test,
test_tuned(net=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric=False,
shuffle= fun_control["device"],
device =fun_control["task"]) task
MulticlassAccuracy: 0.6626999974250793 | Loss: 0.8856547701835632 | Acc: 0.6627000000000000.
Final evaluation: Validation loss: 0.8856547701835632
Final evaluation: Validation metric: 0.6626999974250793
----------------------------------------------
(0.8856547701835632, nan, tensor(0.6627))
The following code trains the model model_spot
. If path
is set to a filename, e.g., path = "model_spot_trained.pt"
, the weights of the trained model will be saved to this file.
=model_spot, train_dataset=train,
train_tuned(net=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric=True,
shuffle= fun_control["device"],
device =None,
path=fun_control["task"]) task
Epoch: 1 |
Batch: 10000. Batch Size: 2. Training Loss (running): 2.261
MulticlassAccuracy: 0.2789166569709778 | Loss: 2.1177865044077238 | Acc: 0.2789166666666666.
Epoch: 2 |
Batch: 10000. Batch Size: 2. Training Loss (running): 2.054
MulticlassAccuracy: 0.3697499930858612 | Loss: 1.9005305001537005 | Acc: 0.3697500000000000.
Epoch: 3 |
Batch: 10000. Batch Size: 2. Training Loss (running): 1.831
MulticlassAccuracy: 0.4807916581630707 | Loss: 1.6832537238399188 | Acc: 0.4807916666666667.
Epoch: 4 |
Batch: 10000. Batch Size: 2. Training Loss (running): 1.619
MulticlassAccuracy: 0.5228750109672546 | Loss: 1.4908735026891033 | Acc: 0.5228750000000000.
Epoch: 5 |
Batch: 10000. Batch Size: 2. Training Loss (running): 1.436
MulticlassAccuracy: 0.5329166650772095 | Loss: 1.3368805952395002 | Acc: 0.5329166666666667.
Epoch: 6 |
Batch: 10000. Batch Size: 2. Training Loss (running): 1.287
MulticlassAccuracy: 0.5986250042915344 | Loss: 1.2163718137256800 | Acc: 0.5986250000000000.
Epoch: 7 |
Batch: 10000. Batch Size: 2. Training Loss (running): 1.180
MulticlassAccuracy: 0.6298333406448364 | Loss: 1.1232892836707955 | Acc: 0.6298333333333334.
Epoch: 8 |
Batch: 10000. Batch Size: 2. Training Loss (running): 1.088
MulticlassAccuracy: 0.6455000042915344 | Loss: 1.0465920382387315 | Acc: 0.6455000000000000.
Returned to Spot: Validation loss: 1.0465920382387315
=model_spot, test_dataset=test,
test_tuned(net=False,
shuffle=fun_control["loss_function"],
loss_function=fun_control["metric_torch"],
metric= fun_control["device"],
device =fun_control["task"]) task
MulticlassAccuracy: 0.6462000012397766 | Loss: 1.0492378783531486 | Acc: 0.6462000000000000.
Final evaluation: Validation loss: 1.0492378783531486
Final evaluation: Validation metric: 0.6462000012397766
----------------------------------------------
(1.0492378783531486, nan, tensor(0.6462))
= "./figures/" + experiment_name
filename =filename) spot_tuner.plot_important_hyperparameter_contour(filename
l1: 8.64447667113925
batch_size: 0.04876078606450302
optimizer: 100.0
spot_tuner.parallel_plot()
Parallel coordinates plots
= False
PLOT_ALL if PLOT_ALL:
= spot_tuner.k
n for i in range(n-1):
for j in range(i+1, n):
=i, j=j, min_z=min_z, max_z = max_z) spot_tuner.plot_contour(i