Module src.PyOghma_ML.Training

This module provides functionality for training machine learning models using TensorFlow/Keras.

It includes classes and utilities for configuring, training, and saving models. The module supports various model architectures, including residual models and multi-input models, and allows for customization of hyperparameters such as learning rates, activation functions, and layer configurations.

Classes

class Model_Settings (initializer: str = 'he_normal',
activation: str = 'silu',
regularization=None,
layer_nodes: list = None,
dropout: list = None,
batch_norm: bool = True,
epochs: int = 256,
inital_learning_rate: float = 1e-06,
gamma_learning_rate: float = 0.001,
power_learning_rate: float = 3,
batch_size: int = 2024,
patience: int = 16,
loss_function: str = 'mse',
metrics: list = None,
training_percentage: float = 0.8,
validation_percentage: float = 0.2,
permutations_limit: int = 1000,
ensemble_presample: int = 1,
ensemble_maximum: int = 2,
ensemble_tollerance: float = 0.001,
ensemble_patience: int = 10,
inputs: int = 1,
decay_rate: int = 32)
Expand source code
class Model_Settings:
    """
    A comprehensive configuration class for neural network model settings.

    This class encapsulates all hyperparameters and configuration options needed
    for training machine learning models in the PyOghma_ML framework. It provides
    sensible defaults while allowing full customization of the training process.

    Architecture Settings:
        - Layer configuration (nodes, dropout, normalization)
        - Activation functions and weight initialization
        - Regularization techniques

    Training Parameters:
        - Learning rate scheduling with exponential decay
        - Batch size and epoch configuration
        - Early stopping with patience

    Ensemble Configuration:
        - Multiple model training and aggregation
        - Performance tolerance and optimization limits
        - Data sampling strategies

    Attributes:
        initializer (str): Weight initialization method for model layers.
            Common options: 'he_normal', 'glorot_uniform', 'random_normal'.
        activation (str): Activation function for hidden layers.
            Recommended: 'silu', 'relu', 'tanh', 'gelu'.
        regularization: Regularization technique (L1, L2, or custom).
        layer_nodes (list): Number of neurons in each hidden layer.
        dropout (list): Dropout rates for each layer (prevents overfitting).
        batch_norm (bool): Whether to apply batch normalization.
        epochs (int): Maximum number of training epochs.
        inital_learning_rate (float): Starting learning rate for optimization.
        gamma_learning_rate (float): Decay factor for learning rate schedule.
        power_learning_rate (float): Power parameter for learning rate decay.
        batch_size (int): Number of samples per training batch.
        patience (int): Early stopping patience (epochs without improvement).
        loss_function (str): Loss function for training ('mse', 'mae', etc.).
        metrics (list): Additional metrics to monitor during training.
        training_percentage (float): Fraction of data used for training.
        validation_percentage (float): Fraction of data used for validation.
        permutations_limit (int): Maximum permutations for data augmentation.
        ensemble_presample (int): Number of models to pre-train for ensemble.
        ensemble_maximum (int): Maximum number of models in ensemble.
        ensemble_tollerance (float): Performance tolerance for ensemble inclusion.
        ensemble_patience (int): Patience for ensemble optimization.
        inputs (int): Number of input branches for multi-input models.
        decay_rate (float): Additional decay parameter for learning rate.

    Example:
        >>> settings = Model_Settings(
        ...     layer_nodes=[128, 64, 32],
        ...     dropout=[0.2, 0.3, 0.4],
        ...     epochs=500,
        ...     batch_size=1024
        ... )
    """
    
    def __init__(self, 
                 initializer: str = 'he_normal',
                 activation: str = 'silu',
                 regularization = None,
                 layer_nodes: list = None,
                 dropout: list = None,
                 batch_norm: bool = True,
                 epochs: int = 256,
                 inital_learning_rate: float = 1e-6,
                 gamma_learning_rate: float = 0.001,
                 power_learning_rate: float = 3,
                 batch_size: int = 2024,
                 patience: int = 16,
                 loss_function: str = 'mse',
                 metrics: list = None,
                 training_percentage: float = 0.8,
                 validation_percentage: float = 0.2,
                 permutations_limit: int = 1000,
                 ensemble_presample: int = 1,
                 ensemble_maximum: int = 2,
                 ensemble_tollerance: float = 1e-3,
                 ensemble_patience: int = 10,
                 inputs: int = 1,
                 decay_rate: int = 32) -> None:
        
        self.initializer = initializer
        self.activation = activation
        self.regularization = regularization
        self.layer_nodes = layer_nodes if layer_nodes is not None else [128, 128, 128, 128]
        self.dropout = dropout if dropout is not None else [0.01, 0.01, 0.01, 0.1]
        self.batch_norm = batch_norm
        self.epochs = epochs
        self.inital_learning_rate = inital_learning_rate
        self.gamma_learning_rate = gamma_learning_rate
        self.power_learning_rate = power_learning_rate
        self.batch_size = batch_size
        self.patience = patience
        self.loss_function = loss_function
        self.metrics = metrics if metrics is not None else [tf.keras.metrics.MeanAbsoluteError()]
        self.training_percentage = training_percentage
        self.validation_percentage = validation_percentage
        self.permutations_limit = permutations_limit
        self.ensemble_presample = ensemble_presample
        self.ensemble_maximum = ensemble_maximum
        self.ensemble_tollerance = ensemble_tollerance
        self.ensemble_patience = ensemble_patience
        self.inputs = inputs
        self.decay_rate = decay_rate

A comprehensive configuration class for neural network model settings.

This class encapsulates all hyperparameters and configuration options needed for training machine learning models in the PyOghma_ML framework. It provides sensible defaults while allowing full customization of the training process.

Architecture Settings: - Layer configuration (nodes, dropout, normalization) - Activation functions and weight initialization - Regularization techniques

Training Parameters: - Learning rate scheduling with exponential decay - Batch size and epoch configuration - Early stopping with patience

Ensemble Configuration: - Multiple model training and aggregation - Performance tolerance and optimization limits - Data sampling strategies

Attributes

initializer : str
Weight initialization method for model layers. Common options: 'he_normal', 'glorot_uniform', 'random_normal'.
activation : str
Activation function for hidden layers. Recommended: 'silu', 'relu', 'tanh', 'gelu'.
regularization
Regularization technique (L1, L2, or custom).
layer_nodes : list
Number of neurons in each hidden layer.
dropout : list
Dropout rates for each layer (prevents overfitting).
batch_norm : bool
Whether to apply batch normalization.
epochs : int
Maximum number of training epochs.
inital_learning_rate : float
Starting learning rate for optimization.
gamma_learning_rate : float
Decay factor for learning rate schedule.
power_learning_rate : float
Power parameter for learning rate decay.
batch_size : int
Number of samples per training batch.
patience : int
Early stopping patience (epochs without improvement).
loss_function : str
Loss function for training ('mse', 'mae', etc.).
metrics : list
Additional metrics to monitor during training.
training_percentage : float
Fraction of data used for training.
validation_percentage : float
Fraction of data used for validation.
permutations_limit : int
Maximum permutations for data augmentation.
ensemble_presample : int
Number of models to pre-train for ensemble.
ensemble_maximum : int
Maximum number of models in ensemble.
ensemble_tollerance : float
Performance tolerance for ensemble inclusion.
ensemble_patience : int
Patience for ensemble optimization.
inputs : int
Number of input branches for multi-input models.
decay_rate : float
Additional decay parameter for learning rate.

Example

>>> settings = Model_Settings(
...     layer_nodes=[128, 64, 32],
...     dropout=[0.2, 0.3, 0.4],
...     epochs=500,
...     batch_size=1024
... )
class Training (training_features: numpy.ndarray,
training_targets: numpy.ndarray,
validation_features: numpy.ndarray,
validation_targets: numpy.ndarray,
dir: str,
model_settings:  | None = None,
model_type: str | None = None,
existing: bool | None = False)
Expand source code
class Training:
    """
    A class for training machine learning models using TensorFlow/Keras.

    This class supports different model architectures, including residual
    models, and provides methods for training, saving, and configuring models.
    """

    def __init__(self, training_features: np.ndarray, training_targets: np.ndarray, validation_features: np.ndarray, validation_targets: np.ndarray, dir: str, model_settings: Optional[dataclass] = None, model_type: Optional[str] = None, existing: Optional[bool] = False) -> None:
        """
        Initialize a Training instance and train the model.

        This method sets up and trains a neural network model using the provided
        training and validation data. It supports both creating new models and
        continuing training from existing models.

        Args:
            training_features (numpy.ndarray): Input features for training the model.
            training_targets (numpy.ndarray): Target values for training the model.
            validation_features (numpy.ndarray): Input features for model validation.
            validation_targets (numpy.ndarray): Target values for model validation.
            dir (str): Directory path where the trained model will be saved.
            model_settings (Model_Settings, optional): Configuration settings for the model
                including architecture, hyperparameters, and training parameters.
            model_type (str, optional): Type of model architecture to use. 
                Supported types include 'Residual' and default Sequential models.
            existing (bool, optional): Whether to load and continue training an existing
                model (True) or create a new model (False). Defaults to False.

        Note:
            - For 'Residual' models, the number of inputs is doubled during fitting
            - Learning rate scheduling with exponential decay is applied for existing models
            - The model is automatically saved after training completion
        """
        #if model_settings is not None:
        self.model_settings = model_settings
        # else:
        #     self.model_settings = Model_Settings()
        self.input_dim = len(training_features[0])
        self.output_dim = np.shape(training_targets)[1]
        # self.validation_features = validation_features.astype(np.float16)
        # self.validation_targets = validation_targets.astype(np.float16)
        if existing == False:
            match model_type:
                case 'Residual':
                    print('Residual')
                    self.residual_model(self.model_settings.inputs)
                    self.fitting(training_features, training_targets, validation_features, validation_targets, self.model_settings.inputs*2)
                case _:
                    print('Sequential')
                    self.model = tf.keras.models.Sequential()
                    self.setup_model()
                    self.fitting(training_features, training_targets, validation_features, validation_targets, self.model_settings.inputs)
            self.saveing(dir)
        else:
            lr_schedule = keras.optimizers.schedules.ExponentialDecay(
            initial_learning_rate=self.model_settings.inital_learning_rate,
            decay_steps=1e6,
            decay_rate=self.model_settings.decay_rate,
            staircase=True
            )
            self.dir = dir
            match model_type:
                case 'Residual':
                    print('Residual')
                    #self.residual_model_existing(self.model_settings.inputs)
                    self.model = self.load_existing_model(dir)
                    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
                    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
                    self.fitting(training_features, training_targets, validation_features, validation_targets, self.model_settings.inputs*2)
                case _:
                    print('Sequential')
                    #self.model = tf.keras.models.Sequential()
                    #self.setup_model_existing()
                    self.model = self.load_existing_model(dir)
                    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
                    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
                    self.fitting(training_features, training_targets, validation_features, validation_targets, self.model_settings.inputs)
            self.saveing_existing(dir)

    def load_existing_model(self, dir: str) -> tf.keras.Model:
        """
        Load an existing trained model from the specified directory.

        This method loads a previously saved Keras model, allowing for 
        continued training or inference. It handles the model loading
        process and returns the loaded model instance.

        Args:
            dir (str): Directory path where the model files are stored.
                The model should be saved in Keras format (.keras file).

        Returns:
            tf.keras.Model: The loaded Keras model ready for training or inference.

        Raises:
            FileNotFoundError: If the model file is not found in the specified directory.
            ValueError: If the model file is corrupted or incompatible.
        """
        model_path = os.path.join(dir, 'model.keras')
        if os.path.exists(model_path):
            m = keras.models.load_model(model_path, compile=False)
            #m.save_weights(os.path.join(dir, 'model_weights.weights.h5'))
            return m
        else:
            raise FileNotFoundError(f"Model file not found at {model_path}")

    def setup_model(self) -> None:
        """
        Set up a standard sequential model based on the provided model settings.

        This method adds layers to the model, including dense layers, activation
        functions, batch normalization, and dropout layers.
        """
        for i, nodes in enumerate(self.model_settings.layer_nodes):
            if i == 0:
                self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
                self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
                if self.model_settings.batch_norm:
                    self.model.add(tf.keras.layers.BatchNormalization())
                if len(self.model_settings.dropout) != 0:
                    self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))
            else:
                self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
                self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
                if self.model_settings.batch_norm:
                    self.model.add(tf.keras.layers.BatchNormalization())
                if len(self.model_settings.dropout) != 0:
                    self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))

        self.model.add(tf.keras.layers.Dense(self.output_dim, activation='tanh'))

        lr_schedule = keras.optimizers.schedules.ExponentialDecay(
            initial_learning_rate=self.model_settings.inital_learning_rate,
            decay_steps=1e6,
            decay_rate=self.model_settings.decay_rate,
            staircase=True
        )

        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

    def setup_model_existing(self) -> None:
        """
        Set up a standard sequential model based on the provided model settings.

        This method adds layers to the model, including dense layers, activation
        functions, batch normalization, and dropout layers.
        """
        for i, nodes in enumerate(self.model_settings.layer_nodes):
            if i == 0:
                self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
                self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
                if self.model_settings.batch_norm:
                    self.model.add(tf.keras.layers.BatchNormalization())
                if len(self.model_settings.dropout) != 0:
                    self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))
            else:
                self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
                self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
                if self.model_settings.batch_norm:
                    self.model.add(tf.keras.layers.BatchNormalization())
                if len(self.model_settings.dropout) != 0:
                    self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))

        self.model.add(tf.keras.layers.Dense(self.output_dim, activation='tanh'))

        lr_schedule = keras.optimizers.schedules.ExponentialDecay(
            initial_learning_rate=self.model_settings.inital_learning_rate,
            decay_steps=1e6,
            decay_rate=self.model_settings.decay_rate,
            staircase=True
        )

        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
        self.model.load_weights(os.path.join(self.dir, 'model_weights.weights.h5'))

    def residual_block(self, inputs: tf.Tensor, n_layers: int, nodes: int, activation: str, dropout: Optional[float] = None) -> tf.Tensor:
        """
        Create a residual block for the model.

        Args:
            inputs (tensorflow.Tensor): Input tensor for the block.
            n_layers (int): Number of layers in the block.
            nodes (int): Number of nodes in each layer.
            activation (str): Activation function to use.
            dropout (float, optional): Dropout rate.

        Returns:
            tensorflow.Tensor: Output tensor of the residual block.
        """
        if self.model_settings.batch_norm:
            x = layers.BatchNormalization()(inputs)
        x = layers.Dense(nodes, activation=activation)(x)

        for idx in range(n_layers):
            x = layers.Dense(nodes, activation=activation)(x)
            if dropout is not None:
                x = layers.Dropout(dropout)(x)

        x = layers.Add()([inputs, x])
        return x

    def setup_residual_model(self) -> None:
        """
        Set up a residual model with a single input.

        This method creates a model with multiple residual blocks and compiles it.
        """
        inputs = keras.Input(shape=(self.input_dim,))

        x = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(inputs)

        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        outputs = layers.Dense(self.output_dim, activation=None)(x)

        self.model = keras.Model(inputs=inputs, outputs=outputs)

        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

    def setup_2input_residual_model(self) -> None:
        """
        Set up a residual model with two inputs.

        This method creates a model with two input branches, each with residual
        blocks, and combines them using subtraction.
        """
        input_1 = keras.Input(shape=(int(self.input_dim/2),))
        input_2 = keras.Input(shape=(int(self.input_dim/2),))

        x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)
        x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)

        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        x = layers.Subtract()([x_1, x_2])

        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        outputs = layers.Dense(self.output_dim, activation=None)(x)

        self.model = keras.Model(inputs=[input_1,input_2], outputs=outputs)

        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

    def residual_model(self, inputs: int =1) -> None:
        print('inputs: ', inputs)
        match inputs*2:
            case 1:
                self.residual_model_1_input()
            case 2:
                self.residual_model_2_input()
            case 4:
                self.residual_model_4_input()
            case 8:
                self.residual_model_8_input()
            case _:
                self.residual_model_1_input()

    def residual_model_existing(self, inputs: int =1) -> None:
        print('inputs: ', inputs)
        match inputs*2:
            case 1:
                self.residual_model_1_input()
            case 2:
                self.residual_model_2_input_existing()
            case 4:
                self.residual_model_4_input()
            case 8:
                self.residual_model_8_input()
            case _:
                self.residual_model_1_input()
    
    def residual_model_1_input(self) -> None:
        """
        Set up a residual model with a single input.

        This method creates a model with multiple residual blocks and compiles it.
        """
        inputs = keras.Input(shape=(self.input_dim,))

        x = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(inputs)

        for i in range(len(self.model_settings.layer_nodes)):
            x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        outputs = layers.Dense(self.output_dim, activation='tanh')(x)

        self.model = keras.Model(inputs=inputs, outputs=outputs)

        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

    def residual_model_2_input(self) -> None:
        """
        Set up a residual model with two inputs.

        This method creates a model with two input branches, each with residual
        blocks, and combines them using subtraction.
        """
        input_1 = keras.Input(shape=(int(self.input_dim/2),),name='input_1')
        input_2 = keras.Input(shape=(int(self.input_dim/2),),name='input_2')

        x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)#
        x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)

        for i in range(len(self.model_settings.layer_nodes)):
            x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        x = layers.Subtract()([x_1, x_2])

        for i in range(len(self.model_settings.layer_nodes)):
            x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        #O_A = keras.actiavation.tanh()
        outputs = layers.Dense(self.output_dim, activation='tanh')(x)

        self.model = keras.Model(inputs=(input_1,input_2), outputs=outputs)

        lr_schedule = keras.optimizers.schedules.ExponentialDecay(
            initial_learning_rate=self.model_settings.inital_learning_rate,
            decay_steps=1e6,
            decay_rate=self.model_settings.decay_rate,
            staircase=True
        )

        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
    
    def residual_model_2_input_existing(self) -> None:
        """
        Set up a residual model with two inputs.

        This method creates a model with two input branches, each with residual
        blocks, and combines them using subtraction.
        """
        input_1 = keras.Input(shape=(int(self.input_dim/2),),name='input_1')
        input_2 = keras.Input(shape=(int(self.input_dim/2),),name='input_2')

        x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)#
        x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)

        for i in range(len(self.model_settings.layer_nodes)):
            x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        x = layers.Subtract()([x_1, x_2])

        for i in range(len(self.model_settings.layer_nodes)):
            x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        #O_A = keras.actiavation.tanh()
        outputs = layers.Dense(self.output_dim, activation='tanh')(x)

        self.model = keras.Model(inputs=(input_1,input_2), outputs=outputs)

        lr_schedule = keras.optimizers.schedules.ExponentialDecay(
            initial_learning_rate=self.model_settings.inital_learning_rate,
            decay_steps=1e6,
            decay_rate=self.model_settings.decay_rate,
            staircase=True
        )

        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
        self.model.load_weights(os.path.join(self.dir, 'model_weights.weights.h5'))

    def residual_model_4_input(self) -> None:
        """
        Set up a residual model with four inputs.

        This method creates a model with four input branches, each with residual
        blocks, and combines them using subtraction.
        """
        input_1 = keras.Input(shape=(int(self.input_dim/4),))
        input_2 = keras.Input(shape=(int(self.input_dim/4),))
        input_3 = keras.Input(shape=(int(self.input_dim/4),))
        input_4 = keras.Input(shape=(int(self.input_dim/4),))

        x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)
        x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)
        x_3 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_3)
        x_4 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_4)

        for i in range(len(self.model_settings.layer_nodes)):
            x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_4 = self.residual_block(x_4, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        x_1 = layers.Subtract()([x_1, x_2])
        x_3 = layers.Subtract()([x_3, x_4])

        for i in range(len(self.model_settings.layer_nodes)):
            x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        
        x = layers.Subtract()([x_1, x_3])

        for i in range(len(self.model_settings.layer_nodes)):
            x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        outputs = layers.Dense(self.output_dim, activation=None)(x)

        self.model = keras.Model(inputs=[input_1,input_2,input_3,input_4], outputs=outputs)
        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
    
    def residual_model_8_input(self) -> None:
        """
        Set up a residual model with eight inputs.

        This method creates a model with eight input branches, each with residual
        blocks, and combines them using subtraction.
        """
        input_1 = keras.Input(shape=(int(self.input_dim/8),))
        input_2 = keras.Input(shape=(int(self.input_dim/8),))
        input_3 = keras.Input(shape=(int(self.input_dim/8),))
        input_4 = keras.Input(shape=(int(self.input_dim/8),))
        input_5 = keras.Input(shape=(int(self.input_dim/8),))
        input_6 = keras.Input(shape=(int(self.input_dim/8),))
        input_7 = keras.Input(shape=(int(self.input_dim/8),))
        input_8 = keras.Input(shape=(int(self.input_dim/8),))

        x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)
        x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)
        x_3 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_3)
        x_4 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_4)
        x_5 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_5)
        x_6 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_6)
        x_7 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_7)
        x_8 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_8)

        for i in range(len(self.model_settings.layer_nodes)):
            x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_4 = self.residual_block(x_4, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_5 = self.residual_block(x_5, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_6 = self.residual_block(x_6, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_7 = self.residual_block(x_7, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_8 = self.residual_block(x_8, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        
        x_1 = layers.Subtract()([x_1, x_2])
        x_3 = layers.Subtract()([x_3, x_4])
        x_5 = layers.Subtract()([x_5, x_6])
        x_7 = layers.Subtract()([x_7, x_8])

        for i in range(len(self.model_settings.layer_nodes)):
            x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_5 = self.residual_block(x_5, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_7 = self.residual_block(x_7, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

        x_1 = layers.Subtract()([x_1, x_3])
        x_5 = layers.Subtract()([x_5, x_7])

        for i in range(len(self.model_settings.layer_nodes)):
            x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
            x_5 = self.residual_block(x_5, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        
        x = layers.Subtract()([x_1, x_5])
        for i in range(len(self.model_settings.layer_nodes)):
            x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        
        outputs = layers.Dense(self.output_dim, activation=None)(x)
        self.model = keras.Model(inputs=[input_1,input_2,input_3,input_4,input_5,input_6,input_7,input_8], outputs=outputs)
        model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
        self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

    def fitting(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray, inputs: int) -> float:
        print('fitting:', inputs)
        match inputs:
            case 1:
                self.fitting_1_input(x, y, val_x, val_y)
            case 2:
                self.fitting_2_input(x, y, val_x, val_y)
            case 4:
                self.fitting_4_input(x, y, val_x, val_y)
            case 8:
                self.fitting_8_input(x, y, val_x, val_y)
            case _:
                self.fitting_1_input(x, y, val_x, val_y)
        
    def fitting_1_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
        """
        Train the model using the provided training and validation data.

        Args:
            x (numpy.ndarray): Training features.
            y (numpy.ndarray): Training targets.
            val_x (numpy.ndarray): Validation features.
            val_y (numpy.ndarray): Validation targets.

        Returns:
            float: The final validation mean absolute error.
        """
        callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
        history = self.model.fit(
            x=x,
            y=y,
            validation_data=(val_x, val_y),
            epochs=self.model_settings.epochs,
            callbacks=callbacks,
            shuffle=True,
            batch_size=int(self.model_settings.batch_size),
        )
        return history.history['val_mean_absolute_error'][-1]
        # """
        # Train the model using the provided training and validation data.

        # Args:
        #     x (numpy.ndarray): Training features.
        #     y (numpy.ndarray): Training targets.
        #     val_x (numpy.ndarray): Validation features.
        #     val_y (numpy.ndarray): Validation targets.

        # Returns:
        #     float: The final validation mean absolute error.
        # """
        # print('fitting_2_input')
        # callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
        # history = self.model.fit(
        #     x=(x[:, :int(len(x[0])/2)], x[:, int(len(x[0])/2):]),
        #     y=y,
        #     validation_data=([ val_x[:, :int(len(x[0])/2)], val_x[:, int(len(x[0])/2):]], val_y),
        #     epochs=self.model_settings.epochs,
        #     callbacks=callbacks,
        #     shuffle=True,
        #     batch_size=int(self.model_settings.batch_size),
        # )
        # return history.history['val_mean_absolute_error'][-1]
    
    def fitting_2_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
        """
        Train the model using the provided training and validation data.

        Args:
            x (numpy.ndarray): Training features.
            y (numpy.ndarray): Training targets.
            val_x (numpy.ndarray): Validation features.
            val_y (numpy.ndarray): Validation targets.

        Returns:
            float: The final validation mean absolute error.
        """
        print('fitting_2_input')
        callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
        history = self.model.fit(
            x=(x[:, :int(len(x[0])/2)], x[:, int(len(x[0])/2):]),
            y=y,
            validation_data=([ val_x[:, :int(len(x[0])/2)], val_x[:, int(len(x[0])/2):]], val_y),
            epochs=self.model_settings.epochs,
            callbacks=callbacks,
            shuffle=True,
            batch_size=int(self.model_settings.batch_size),
        )
        return history.history['val_mean_absolute_error'][-1]
    
    def fitting_4_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
        """
        Train the model using the provided training and validation data.

        Args:
            x (numpy.ndarray): Training features.
            y (numpy.ndarray): Training targets.
            val_x (numpy.ndarray): Validation features.
            val_y (numpy.ndarray): Validation targets.

        Returns:
            float: The final validation mean absolute error.
        """
        callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
        history = self.model.fit(
            x=[x[:, :int(len(x[0])*1/4)], x[:, int(len(x[0])*2/4):int(len(x[0])*3/4)], x[:, int(len(x[0])*1/4):int(len(x[0])*2/4)], x[:, int(len(x[0])*3/4):]],
            y=y,
            validation_data=([ val_x[:, :int(len(x[0])*1/4)], val_x[:, int(len(x[0])*2/4):int(len(x[0])*3/4)], val_x[:, int(len(x[0])*1/4):int(len(x[0])*2/4)], val_x[:, int(len(x[0])*3/4):]], val_y),
            epochs=self.model_settings.epochs,
            callbacks=callbacks,
            shuffle=True,
            batch_size=int(self.model_settings.batch_size),
        )
        return history.history['val_mean_absolute_error'][-1]
    
    def fitting_8_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
        """
        Train the model using the provided training and validation data.

        Args:
            x (numpy.ndarray): Training features.
            y (numpy.ndarray): Training targets.
            val_x (numpy.ndarray): Validation features.
            val_y (numpy.ndarray): Validation targets.

        Returns:
            float: The final validation mean absolute error.
        """
        callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
        print(np.shape(x))
        print(np.shape(val_x))
        print(np.shape(x[0]))
        print(np.shape(val_x[0]))
        history = self.model.fit(
            x=[x[:, :int(len(x[0]) * 1/8)], x[:, int(len(x[0]) * 4/8):int(len(x[0])* 5/8)], x[:, int(len(x[0]) * 1/8):int(len(x[0])* 2/8)], x[:, int(len(x[0])* 5/8):int(len(x[0])* 6/8)], x[:, int(len(x[0])* 2/8):int(len(x[0])* 3/8)], x[:, int(len(x[0])*6/8):int(len(x[0])* 7/8)], x[:, int(len(x[0])*3/8):int(len(x[0])*4/8)], x[:, int(len(x[0])*7/8):]],
            y=y,
            validation_data=([val_x[:, :int(len(x[0])* 1/8)], val_x[:, int(len(x[0]) * 4/8):int(len(x[0]) * 5/8)], val_x[:, int(len(x[0])*1/8):int(len(x[0])*2/8)], val_x[:, int(len(x[0])*5/8):int(len(x[0])*6/8)], val_x[:, int(len(x[0])*2/8):int(len(x[0])*3/8)], val_x[:, int(len(x[0])*6/8):int(len(x[0])*7/8)], val_x[:, int(len(x[0])*3/8):int(len(x[0])*4/8)], val_x[:, int(len(x[0])*7/8):]], val_y),
            epochs=self.model_settings.epochs,
            callbacks=callbacks,
            shuffle=True,
            batch_size=int(self.model_settings.batch_size),
        )
        return history.history['val_mean_absolute_error'][-1]

    def saveing(self, dir: str) -> None:
        """
        Save the trained model to the specified directory.

        Args:
            dir (str): Directory to save the model.
        """
        dir = os.path.join(dir, 'model.keras')
        self.model.save(dir)

    def saveing_existing(self, dir: str) -> None:
        """
        Save the trained model to the specified directory.

        Args:
            dir (str): Directory to save the model.
        """
        files = str(len(os.listdir(dir)) + 1)
        print('Saving model to:', dir)
        dir = os.path.join(dir, 'model'+files+'.keras')
        self.model.save(dir)

A class for training machine learning models using TensorFlow/Keras.

This class supports different model architectures, including residual models, and provides methods for training, saving, and configuring models.

Initialize a Training instance and train the model.

This method sets up and trains a neural network model using the provided training and validation data. It supports both creating new models and continuing training from existing models.

Args

training_features : numpy.ndarray
Input features for training the model.
training_targets : numpy.ndarray
Target values for training the model.
validation_features : numpy.ndarray
Input features for model validation.
validation_targets : numpy.ndarray
Target values for model validation.
dir : str
Directory path where the trained model will be saved.
model_settings : Model_Settings, optional
Configuration settings for the model including architecture, hyperparameters, and training parameters.
model_type : str, optional
Type of model architecture to use. Supported types include 'Residual' and default Sequential models.
existing : bool, optional
Whether to load and continue training an existing model (True) or create a new model (False). Defaults to False.

Note

  • For 'Residual' models, the number of inputs is doubled during fitting
  • Learning rate scheduling with exponential decay is applied for existing models
  • The model is automatically saved after training completion

Methods

def fitting(self,
x: numpy.ndarray,
y: numpy.ndarray,
val_x: numpy.ndarray,
val_y: numpy.ndarray,
inputs: int) ‑> float
Expand source code
def fitting(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray, inputs: int) -> float:
    print('fitting:', inputs)
    match inputs:
        case 1:
            self.fitting_1_input(x, y, val_x, val_y)
        case 2:
            self.fitting_2_input(x, y, val_x, val_y)
        case 4:
            self.fitting_4_input(x, y, val_x, val_y)
        case 8:
            self.fitting_8_input(x, y, val_x, val_y)
        case _:
            self.fitting_1_input(x, y, val_x, val_y)
def fitting_1_input(self,
x: numpy.ndarray,
y: numpy.ndarray,
val_x: numpy.ndarray,
val_y: numpy.ndarray) ‑> float
Expand source code
def fitting_1_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
    """
    Train the model using the provided training and validation data.

    Args:
        x (numpy.ndarray): Training features.
        y (numpy.ndarray): Training targets.
        val_x (numpy.ndarray): Validation features.
        val_y (numpy.ndarray): Validation targets.

    Returns:
        float: The final validation mean absolute error.
    """
    callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
    history = self.model.fit(
        x=x,
        y=y,
        validation_data=(val_x, val_y),
        epochs=self.model_settings.epochs,
        callbacks=callbacks,
        shuffle=True,
        batch_size=int(self.model_settings.batch_size),
    )
    return history.history['val_mean_absolute_error'][-1]
    # """
    # Train the model using the provided training and validation data.

    # Args:
    #     x (numpy.ndarray): Training features.
    #     y (numpy.ndarray): Training targets.
    #     val_x (numpy.ndarray): Validation features.
    #     val_y (numpy.ndarray): Validation targets.

    # Returns:
    #     float: The final validation mean absolute error.
    # """
    # print('fitting_2_input')
    # callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
    # history = self.model.fit(
    #     x=(x[:, :int(len(x[0])/2)], x[:, int(len(x[0])/2):]),
    #     y=y,
    #     validation_data=([ val_x[:, :int(len(x[0])/2)], val_x[:, int(len(x[0])/2):]], val_y),
    #     epochs=self.model_settings.epochs,
    #     callbacks=callbacks,
    #     shuffle=True,
    #     batch_size=int(self.model_settings.batch_size),
    # )
    # return history.history['val_mean_absolute_error'][-1]

Train the model using the provided training and validation data.

Args

x : numpy.ndarray
Training features.
y : numpy.ndarray
Training targets.
val_x : numpy.ndarray
Validation features.
val_y : numpy.ndarray
Validation targets.

Returns

float
The final validation mean absolute error.
def fitting_2_input(self,
x: numpy.ndarray,
y: numpy.ndarray,
val_x: numpy.ndarray,
val_y: numpy.ndarray) ‑> float
Expand source code
def fitting_2_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
    """
    Train the model using the provided training and validation data.

    Args:
        x (numpy.ndarray): Training features.
        y (numpy.ndarray): Training targets.
        val_x (numpy.ndarray): Validation features.
        val_y (numpy.ndarray): Validation targets.

    Returns:
        float: The final validation mean absolute error.
    """
    print('fitting_2_input')
    callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
    history = self.model.fit(
        x=(x[:, :int(len(x[0])/2)], x[:, int(len(x[0])/2):]),
        y=y,
        validation_data=([ val_x[:, :int(len(x[0])/2)], val_x[:, int(len(x[0])/2):]], val_y),
        epochs=self.model_settings.epochs,
        callbacks=callbacks,
        shuffle=True,
        batch_size=int(self.model_settings.batch_size),
    )
    return history.history['val_mean_absolute_error'][-1]

Train the model using the provided training and validation data.

Args

x : numpy.ndarray
Training features.
y : numpy.ndarray
Training targets.
val_x : numpy.ndarray
Validation features.
val_y : numpy.ndarray
Validation targets.

Returns

float
The final validation mean absolute error.
def fitting_4_input(self,
x: numpy.ndarray,
y: numpy.ndarray,
val_x: numpy.ndarray,
val_y: numpy.ndarray) ‑> float
Expand source code
def fitting_4_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
    """
    Train the model using the provided training and validation data.

    Args:
        x (numpy.ndarray): Training features.
        y (numpy.ndarray): Training targets.
        val_x (numpy.ndarray): Validation features.
        val_y (numpy.ndarray): Validation targets.

    Returns:
        float: The final validation mean absolute error.
    """
    callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
    history = self.model.fit(
        x=[x[:, :int(len(x[0])*1/4)], x[:, int(len(x[0])*2/4):int(len(x[0])*3/4)], x[:, int(len(x[0])*1/4):int(len(x[0])*2/4)], x[:, int(len(x[0])*3/4):]],
        y=y,
        validation_data=([ val_x[:, :int(len(x[0])*1/4)], val_x[:, int(len(x[0])*2/4):int(len(x[0])*3/4)], val_x[:, int(len(x[0])*1/4):int(len(x[0])*2/4)], val_x[:, int(len(x[0])*3/4):]], val_y),
        epochs=self.model_settings.epochs,
        callbacks=callbacks,
        shuffle=True,
        batch_size=int(self.model_settings.batch_size),
    )
    return history.history['val_mean_absolute_error'][-1]

Train the model using the provided training and validation data.

Args

x : numpy.ndarray
Training features.
y : numpy.ndarray
Training targets.
val_x : numpy.ndarray
Validation features.
val_y : numpy.ndarray
Validation targets.

Returns

float
The final validation mean absolute error.
def fitting_8_input(self,
x: numpy.ndarray,
y: numpy.ndarray,
val_x: numpy.ndarray,
val_y: numpy.ndarray) ‑> float
Expand source code
def fitting_8_input(self, x: np.ndarray, y: np.ndarray, val_x: np.ndarray, val_y: np.ndarray) -> float:
    """
    Train the model using the provided training and validation data.

    Args:
        x (numpy.ndarray): Training features.
        y (numpy.ndarray): Training targets.
        val_x (numpy.ndarray): Validation features.
        val_y (numpy.ndarray): Validation targets.

    Returns:
        float: The final validation mean absolute error.
    """
    callbacks = [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=self.model_settings.patience, verbose=1)]
    print(np.shape(x))
    print(np.shape(val_x))
    print(np.shape(x[0]))
    print(np.shape(val_x[0]))
    history = self.model.fit(
        x=[x[:, :int(len(x[0]) * 1/8)], x[:, int(len(x[0]) * 4/8):int(len(x[0])* 5/8)], x[:, int(len(x[0]) * 1/8):int(len(x[0])* 2/8)], x[:, int(len(x[0])* 5/8):int(len(x[0])* 6/8)], x[:, int(len(x[0])* 2/8):int(len(x[0])* 3/8)], x[:, int(len(x[0])*6/8):int(len(x[0])* 7/8)], x[:, int(len(x[0])*3/8):int(len(x[0])*4/8)], x[:, int(len(x[0])*7/8):]],
        y=y,
        validation_data=([val_x[:, :int(len(x[0])* 1/8)], val_x[:, int(len(x[0]) * 4/8):int(len(x[0]) * 5/8)], val_x[:, int(len(x[0])*1/8):int(len(x[0])*2/8)], val_x[:, int(len(x[0])*5/8):int(len(x[0])*6/8)], val_x[:, int(len(x[0])*2/8):int(len(x[0])*3/8)], val_x[:, int(len(x[0])*6/8):int(len(x[0])*7/8)], val_x[:, int(len(x[0])*3/8):int(len(x[0])*4/8)], val_x[:, int(len(x[0])*7/8):]], val_y),
        epochs=self.model_settings.epochs,
        callbacks=callbacks,
        shuffle=True,
        batch_size=int(self.model_settings.batch_size),
    )
    return history.history['val_mean_absolute_error'][-1]

Train the model using the provided training and validation data.

Args

x : numpy.ndarray
Training features.
y : numpy.ndarray
Training targets.
val_x : numpy.ndarray
Validation features.
val_y : numpy.ndarray
Validation targets.

Returns

float
The final validation mean absolute error.
def load_existing_model(self, dir: str) ‑> keras.src.models.model.Model
Expand source code
def load_existing_model(self, dir: str) -> tf.keras.Model:
    """
    Load an existing trained model from the specified directory.

    This method loads a previously saved Keras model, allowing for 
    continued training or inference. It handles the model loading
    process and returns the loaded model instance.

    Args:
        dir (str): Directory path where the model files are stored.
            The model should be saved in Keras format (.keras file).

    Returns:
        tf.keras.Model: The loaded Keras model ready for training or inference.

    Raises:
        FileNotFoundError: If the model file is not found in the specified directory.
        ValueError: If the model file is corrupted or incompatible.
    """
    model_path = os.path.join(dir, 'model.keras')
    if os.path.exists(model_path):
        m = keras.models.load_model(model_path, compile=False)
        #m.save_weights(os.path.join(dir, 'model_weights.weights.h5'))
        return m
    else:
        raise FileNotFoundError(f"Model file not found at {model_path}")

Load an existing trained model from the specified directory.

This method loads a previously saved Keras model, allowing for continued training or inference. It handles the model loading process and returns the loaded model instance.

Args

dir : str
Directory path where the model files are stored. The model should be saved in Keras format (.keras file).

Returns

tf.keras.Model
The loaded Keras model ready for training or inference.

Raises

FileNotFoundError
If the model file is not found in the specified directory.
ValueError
If the model file is corrupted or incompatible.
def residual_block(self,
inputs: tensorflow.python.framework.tensor.Tensor,
n_layers: int,
nodes: int,
activation: str,
dropout: float | None = None) ‑> tensorflow.python.framework.tensor.Tensor
Expand source code
def residual_block(self, inputs: tf.Tensor, n_layers: int, nodes: int, activation: str, dropout: Optional[float] = None) -> tf.Tensor:
    """
    Create a residual block for the model.

    Args:
        inputs (tensorflow.Tensor): Input tensor for the block.
        n_layers (int): Number of layers in the block.
        nodes (int): Number of nodes in each layer.
        activation (str): Activation function to use.
        dropout (float, optional): Dropout rate.

    Returns:
        tensorflow.Tensor: Output tensor of the residual block.
    """
    if self.model_settings.batch_norm:
        x = layers.BatchNormalization()(inputs)
    x = layers.Dense(nodes, activation=activation)(x)

    for idx in range(n_layers):
        x = layers.Dense(nodes, activation=activation)(x)
        if dropout is not None:
            x = layers.Dropout(dropout)(x)

    x = layers.Add()([inputs, x])
    return x

Create a residual block for the model.

Args

inputs : tensorflow.Tensor
Input tensor for the block.
n_layers : int
Number of layers in the block.
nodes : int
Number of nodes in each layer.
activation : str
Activation function to use.
dropout : float, optional
Dropout rate.

Returns

tensorflow.Tensor
Output tensor of the residual block.
def residual_model(self, inputs: int = 1) ‑> None
Expand source code
def residual_model(self, inputs: int =1) -> None:
    print('inputs: ', inputs)
    match inputs*2:
        case 1:
            self.residual_model_1_input()
        case 2:
            self.residual_model_2_input()
        case 4:
            self.residual_model_4_input()
        case 8:
            self.residual_model_8_input()
        case _:
            self.residual_model_1_input()
def residual_model_1_input(self) ‑> None
Expand source code
def residual_model_1_input(self) -> None:
    """
    Set up a residual model with a single input.

    This method creates a model with multiple residual blocks and compiles it.
    """
    inputs = keras.Input(shape=(self.input_dim,))

    x = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(inputs)

    for i in range(len(self.model_settings.layer_nodes)):
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    outputs = layers.Dense(self.output_dim, activation='tanh')(x)

    self.model = keras.Model(inputs=inputs, outputs=outputs)

    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

Set up a residual model with a single input.

This method creates a model with multiple residual blocks and compiles it.

def residual_model_2_input(self) ‑> None
Expand source code
def residual_model_2_input(self) -> None:
    """
    Set up a residual model with two inputs.

    This method creates a model with two input branches, each with residual
    blocks, and combines them using subtraction.
    """
    input_1 = keras.Input(shape=(int(self.input_dim/2),),name='input_1')
    input_2 = keras.Input(shape=(int(self.input_dim/2),),name='input_2')

    x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)#
    x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)

    for i in range(len(self.model_settings.layer_nodes)):
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    x = layers.Subtract()([x_1, x_2])

    for i in range(len(self.model_settings.layer_nodes)):
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    #O_A = keras.actiavation.tanh()
    outputs = layers.Dense(self.output_dim, activation='tanh')(x)

    self.model = keras.Model(inputs=(input_1,input_2), outputs=outputs)

    lr_schedule = keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=self.model_settings.inital_learning_rate,
        decay_steps=1e6,
        decay_rate=self.model_settings.decay_rate,
        staircase=True
    )

    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

Set up a residual model with two inputs.

This method creates a model with two input branches, each with residual blocks, and combines them using subtraction.

def residual_model_2_input_existing(self) ‑> None
Expand source code
def residual_model_2_input_existing(self) -> None:
    """
    Set up a residual model with two inputs.

    This method creates a model with two input branches, each with residual
    blocks, and combines them using subtraction.
    """
    input_1 = keras.Input(shape=(int(self.input_dim/2),),name='input_1')
    input_2 = keras.Input(shape=(int(self.input_dim/2),),name='input_2')

    x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)#
    x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)

    for i in range(len(self.model_settings.layer_nodes)):
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    x = layers.Subtract()([x_1, x_2])

    for i in range(len(self.model_settings.layer_nodes)):
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    #O_A = keras.actiavation.tanh()
    outputs = layers.Dense(self.output_dim, activation='tanh')(x)

    self.model = keras.Model(inputs=(input_1,input_2), outputs=outputs)

    lr_schedule = keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=self.model_settings.inital_learning_rate,
        decay_steps=1e6,
        decay_rate=self.model_settings.decay_rate,
        staircase=True
    )

    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
    self.model.load_weights(os.path.join(self.dir, 'model_weights.weights.h5'))

Set up a residual model with two inputs.

This method creates a model with two input branches, each with residual blocks, and combines them using subtraction.

def residual_model_4_input(self) ‑> None
Expand source code
def residual_model_4_input(self) -> None:
    """
    Set up a residual model with four inputs.

    This method creates a model with four input branches, each with residual
    blocks, and combines them using subtraction.
    """
    input_1 = keras.Input(shape=(int(self.input_dim/4),))
    input_2 = keras.Input(shape=(int(self.input_dim/4),))
    input_3 = keras.Input(shape=(int(self.input_dim/4),))
    input_4 = keras.Input(shape=(int(self.input_dim/4),))

    x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)
    x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)
    x_3 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_3)
    x_4 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_4)

    for i in range(len(self.model_settings.layer_nodes)):
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_4 = self.residual_block(x_4, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    x_1 = layers.Subtract()([x_1, x_2])
    x_3 = layers.Subtract()([x_3, x_4])

    for i in range(len(self.model_settings.layer_nodes)):
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    
    x = layers.Subtract()([x_1, x_3])

    for i in range(len(self.model_settings.layer_nodes)):
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    outputs = layers.Dense(self.output_dim, activation=None)(x)

    self.model = keras.Model(inputs=[input_1,input_2,input_3,input_4], outputs=outputs)
    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

Set up a residual model with four inputs.

This method creates a model with four input branches, each with residual blocks, and combines them using subtraction.

def residual_model_8_input(self) ‑> None
Expand source code
def residual_model_8_input(self) -> None:
    """
    Set up a residual model with eight inputs.

    This method creates a model with eight input branches, each with residual
    blocks, and combines them using subtraction.
    """
    input_1 = keras.Input(shape=(int(self.input_dim/8),))
    input_2 = keras.Input(shape=(int(self.input_dim/8),))
    input_3 = keras.Input(shape=(int(self.input_dim/8),))
    input_4 = keras.Input(shape=(int(self.input_dim/8),))
    input_5 = keras.Input(shape=(int(self.input_dim/8),))
    input_6 = keras.Input(shape=(int(self.input_dim/8),))
    input_7 = keras.Input(shape=(int(self.input_dim/8),))
    input_8 = keras.Input(shape=(int(self.input_dim/8),))

    x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)
    x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)
    x_3 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_3)
    x_4 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_4)
    x_5 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_5)
    x_6 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_6)
    x_7 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_7)
    x_8 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_8)

    for i in range(len(self.model_settings.layer_nodes)):
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_4 = self.residual_block(x_4, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_5 = self.residual_block(x_5, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_6 = self.residual_block(x_6, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_7 = self.residual_block(x_7, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_8 = self.residual_block(x_8, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    
    x_1 = layers.Subtract()([x_1, x_2])
    x_3 = layers.Subtract()([x_3, x_4])
    x_5 = layers.Subtract()([x_5, x_6])
    x_7 = layers.Subtract()([x_7, x_8])

    for i in range(len(self.model_settings.layer_nodes)):
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_3 = self.residual_block(x_3, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_5 = self.residual_block(x_5, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_7 = self.residual_block(x_7, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    x_1 = layers.Subtract()([x_1, x_3])
    x_5 = layers.Subtract()([x_5, x_7])

    for i in range(len(self.model_settings.layer_nodes)):
        x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
        x_5 = self.residual_block(x_5, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    
    x = layers.Subtract()([x_1, x_5])
    for i in range(len(self.model_settings.layer_nodes)):
        x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    
    outputs = layers.Dense(self.output_dim, activation=None)(x)
    self.model = keras.Model(inputs=[input_1,input_2,input_3,input_4,input_5,input_6,input_7,input_8], outputs=outputs)
    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

Set up a residual model with eight inputs.

This method creates a model with eight input branches, each with residual blocks, and combines them using subtraction.

def residual_model_existing(self, inputs: int = 1) ‑> None
Expand source code
def residual_model_existing(self, inputs: int =1) -> None:
    print('inputs: ', inputs)
    match inputs*2:
        case 1:
            self.residual_model_1_input()
        case 2:
            self.residual_model_2_input_existing()
        case 4:
            self.residual_model_4_input()
        case 8:
            self.residual_model_8_input()
        case _:
            self.residual_model_1_input()
def saveing(self, dir: str) ‑> None
Expand source code
def saveing(self, dir: str) -> None:
    """
    Save the trained model to the specified directory.

    Args:
        dir (str): Directory to save the model.
    """
    dir = os.path.join(dir, 'model.keras')
    self.model.save(dir)

Save the trained model to the specified directory.

Args

dir : str
Directory to save the model.
def saveing_existing(self, dir: str) ‑> None
Expand source code
def saveing_existing(self, dir: str) -> None:
    """
    Save the trained model to the specified directory.

    Args:
        dir (str): Directory to save the model.
    """
    files = str(len(os.listdir(dir)) + 1)
    print('Saving model to:', dir)
    dir = os.path.join(dir, 'model'+files+'.keras')
    self.model.save(dir)

Save the trained model to the specified directory.

Args

dir : str
Directory to save the model.
def setup_2input_residual_model(self) ‑> None
Expand source code
def setup_2input_residual_model(self) -> None:
    """
    Set up a residual model with two inputs.

    This method creates a model with two input branches, each with residual
    blocks, and combines them using subtraction.
    """
    input_1 = keras.Input(shape=(int(self.input_dim/2),))
    input_2 = keras.Input(shape=(int(self.input_dim/2),))

    x_1 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_1)
    x_2 = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(input_2)

    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_1 = self.residual_block(x_1, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x_2 = self.residual_block(x_2, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    x = layers.Subtract()([x_1, x_2])

    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    outputs = layers.Dense(self.output_dim, activation=None)(x)

    self.model = keras.Model(inputs=[input_1,input_2], outputs=outputs)

    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

Set up a residual model with two inputs.

This method creates a model with two input branches, each with residual blocks, and combines them using subtraction.

def setup_model(self) ‑> None
Expand source code
def setup_model(self) -> None:
    """
    Set up a standard sequential model based on the provided model settings.

    This method adds layers to the model, including dense layers, activation
    functions, batch normalization, and dropout layers.
    """
    for i, nodes in enumerate(self.model_settings.layer_nodes):
        if i == 0:
            self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
            self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
            if self.model_settings.batch_norm:
                self.model.add(tf.keras.layers.BatchNormalization())
            if len(self.model_settings.dropout) != 0:
                self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))
        else:
            self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
            self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
            if self.model_settings.batch_norm:
                self.model.add(tf.keras.layers.BatchNormalization())
            if len(self.model_settings.dropout) != 0:
                self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))

    self.model.add(tf.keras.layers.Dense(self.output_dim, activation='tanh'))

    lr_schedule = keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=self.model_settings.inital_learning_rate,
        decay_steps=1e6,
        decay_rate=self.model_settings.decay_rate,
        staircase=True
    )

    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

Set up a standard sequential model based on the provided model settings.

This method adds layers to the model, including dense layers, activation functions, batch normalization, and dropout layers.

def setup_model_existing(self) ‑> None
Expand source code
def setup_model_existing(self) -> None:
    """
    Set up a standard sequential model based on the provided model settings.

    This method adds layers to the model, including dense layers, activation
    functions, batch normalization, and dropout layers.
    """
    for i, nodes in enumerate(self.model_settings.layer_nodes):
        if i == 0:
            self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
            self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
            if self.model_settings.batch_norm:
                self.model.add(tf.keras.layers.BatchNormalization())
            if len(self.model_settings.dropout) != 0:
                self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))
        else:
            self.model.add(tf.keras.layers.Dense(nodes, kernel_initializer=self.model_settings.initializer, kernel_regularizer=self.model_settings.regularization))
            self.model.add(tf.keras.layers.Activation(self.model_settings.activation))
            if self.model_settings.batch_norm:
                self.model.add(tf.keras.layers.BatchNormalization())
            if len(self.model_settings.dropout) != 0:
                self.model.add(tf.keras.layers.Dropout(self.model_settings.dropout[i]))

    self.model.add(tf.keras.layers.Dense(self.output_dim, activation='tanh'))

    lr_schedule = keras.optimizers.schedules.ExponentialDecay(
        initial_learning_rate=self.model_settings.inital_learning_rate,
        decay_steps=1e6,
        decay_rate=self.model_settings.decay_rate,
        staircase=True
    )

    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)
    self.model.load_weights(os.path.join(self.dir, 'model_weights.weights.h5'))

Set up a standard sequential model based on the provided model settings.

This method adds layers to the model, including dense layers, activation functions, batch normalization, and dropout layers.

def setup_residual_model(self) ‑> None
Expand source code
def setup_residual_model(self) -> None:
    """
    Set up a residual model with a single input.

    This method creates a model with multiple residual blocks and compiles it.
    """
    inputs = keras.Input(shape=(self.input_dim,))

    x = layers.Dense(self.model_settings.layer_nodes[0], activation=None)(inputs)

    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])
    x = self.residual_block(x, nodes=self.model_settings.layer_nodes[0], n_layers=len(self.model_settings.layer_nodes), activation=self.model_settings.activation, dropout=self.model_settings.dropout[0])

    outputs = layers.Dense(self.output_dim, activation=None)(x)

    self.model = keras.Model(inputs=inputs, outputs=outputs)

    model_optimiser = tf.keras.optimizers.Adam(learning_rate=lr(self.model_settings.inital_learning_rate, self.model_settings.gamma_learning_rate, self.model_settings.power_learning_rate))
    self.model.compile(optimizer=model_optimiser, loss=self.model_settings.loss_function, metrics=self.model_settings.metrics)

Set up a residual model with a single input.

This method creates a model with multiple residual blocks and compiles it.

class lr (initial_learning_rate: float, gamma: float, power: float)
Expand source code
@src.saving.register_keras_serializable()
class lr(tf.keras.optimizers.schedules.LearningRateSchedule):
    """
    A custom learning rate schedule for TensorFlow/Keras.

    This schedule adjusts the learning rate based on the training step using
    a power decay formula.
    """

    def __init__(self, initial_learning_rate: float, gamma: float, power: float) -> None:
        """
        Initialize the learning rate schedule.

        Args:
            initial_learning_rate (float): Initial learning rate.
            gamma (float): Decay rate.
            power (float): Power for the decay formula.
        """
        self.initial_learning_rate = initial_learning_rate
        self.gamma = gamma
        self.power = power

    def __call__(self, step: int) -> float:
        """
        Calculate the learning rate for a given training step.

        Args:
            step (int): The current training step.

        Returns:
            float: The calculated learning rate.
        """
        return self.initial_learning_rate * tf.pow((step * self.gamma +1), -self.power)

    def get_config(self) -> dict:
        """
        Get the configuration of the learning rate schedule.

        Returns:
            dict: A dictionary containing the configuration.
        """
        config = {
            'initial_learning_rate': self.initial_learning_rate,
            'gamma': self.gamma,
            'power': self.power
        }
        return config

A custom learning rate schedule for TensorFlow/Keras.

This schedule adjusts the learning rate based on the training step using a power decay formula.

Initialize the learning rate schedule.

Args

initial_learning_rate : float
Initial learning rate.
gamma : float
Decay rate.
power : float
Power for the decay formula.

Ancestors

  • keras.src.optimizers.schedules.learning_rate_schedule.LearningRateSchedule

Methods

def get_config(self) ‑> dict
Expand source code
def get_config(self) -> dict:
    """
    Get the configuration of the learning rate schedule.

    Returns:
        dict: A dictionary containing the configuration.
    """
    config = {
        'initial_learning_rate': self.initial_learning_rate,
        'gamma': self.gamma,
        'power': self.power
    }
    return config

Get the configuration of the learning rate schedule.

Returns

dict
A dictionary containing the configuration.