aiml.evaluation package
Submodules
aiml.evaluation.check_accuracy module
check_accuracy.py
This module defines functions for calculating the accuracy of a given dataset on a model.
- aiml.evaluation.check_accuracy.check_accuracy(model, dataloader, device)[source]
Calculate the accuracy of a dataset using a pre-trained machine learning model.
- Parameters:
model – The pre-trained machine learning model.
dataloader – The dataloader for the dataset to be tested.
device (str) – The device to use, either ‘cpu’ or ‘gpu’.
- Returns:
The accuracy of the dataset when tested on the provided model.
- Return type:
float
- aiml.evaluation.check_accuracy.check_accuracy_with_flags(model, dataloader, device)[source]
Calculate the accuracy of a dataset using a machine learning model.
- Parameters:
model – The machine learning model for evaluation.
dataloader – The dataloader for the dataset to be tested.
device (str) – The device to use, either ‘cpu’ or ‘gpu’.
- Returns:
The accuracy of the dataset when tested on the provided model. list: A list showing which images were correctly recognized (True) or not (False).
- Return type:
float
aiml.evaluation.dynamic module
dynamic.py
This module provides the decide_attack function which will decide the next attack to be applied and its parameter.
- aiml.evaluation.dynamic.decide_attack(result_list, attack_para_list=[[[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[50], [100], [150]], [[1], [16], [32]], [[1], [16], [32]]])[source]
Write the results of the previous attack to a text file and determine the next attack and its parameters based on attack history.
- Parameters:
result_list – A list where the first element is the overall mark, and the subsequent elements are lists containing the history of previous attacks. Sublists stores the attack number, parameter number, and accuracy.
- Returns:
- The number of the next attack
(could be the same or the next one in the attack_method_list).
next_parameter_number (int): The number of the next parameter. continue_testing (bool): Whether to continue testing attacks or not. overall_mark (int): A score summarizing the previous performance.
- Return type:
next_attack_number (int)
aiml.evaluation.evaluate module
evaluate.py
This module provides the evaluate function which will evaluate the model with the given data and attack methods.
- aiml.evaluation.evaluate.evaluate(input_model, input_test_data, input_train_data=None, input_shape=None, clip_values=None, nb_classes=None, batch_size_attack=64, num_threads_attack=0, batch_size_train=64, batch_size_test=64, num_workers=4, dry=False, attack_para_list=[[[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[1], [16], [32]], [[50], [100], [150]], [[1], [16], [32]], [[1], [16], [32]]])[source]
Evaluate the model’s performance using the provided data and attack methods.
- Parameters:
input_model (str or model) – A string of the name of the machine learning model or the machine learning model itself.
input_test_data (str or dataset) – A string of the name of the testing dataset or the testing dataset itself.
input_train_data (str or dataset, optional) – A string of the name of the training dataset or the training dataset itself (default is None).
input_shape (tuple, optional) – Shape of input data (default is None).
clip_values (tuple, optional) – Range of input data values (default is None).
nb_classes (int, optional) – Number of classes in the dataset (default is None).
batch_size_attack (int, optional) – Batch size for attack testing (default is 64).
num_threads_attack (int, optional) – Number of threads for attack testing (default is 0).
batch_size_train (int, optional) – Batch size for training data (default is 64).
batch_size_test (int, optional) – Batch size for test data (default is 64).
num_workers (int, optional) – Number of workers to use for data loading (default is half of the available CPU cores).
dry (bool, optional) – When True, the code should only test one example.
attack_para_list (list, optional) – List of parameter combinations for the attack.
- Returns:
None.