Ensemble

Module

Prosper_nn provides implementations for specialized time series forecasting neural networks and related utility functions.

Copyright (C) 2022 Nico Beck, Julia Schemm, Henning Frechen, Jacob Fidorra,: Denni Schmidt, Sai Kiran Srivatsav Gollapalli

This file is part of Propser_nn.

Propser_nn is free software: you can redistribute it and/or modify

it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

class prosper_nn.models.ensemble.ensemble.Ensemble(model: ~torch.nn.modules.module.Module, n_models: int, sparsity: int = 0.0, keep_pruning_mask: bool = False, initializer: ~typing.Callable = <function kaiming_uniform_>, parallel: bool = False, combination_type: str = 'mean')[source]

Bases: Module

Ensemble module to train multiple instances of a model. The different instances are initialized and saved as a class attribute model.

The forward pass of this module returns one tensor that contains the outputs of the individual models at the beginning and their mean/median in the last index.

To train the models iterate over the model output tensor and calculate and sum up the loss for each output. Call backward() for the summed up loss.

The mean/median output can be used when the model is trained and in normal use. The mean/median output should not be used during training.

Parameters:

model (torch.nn.Module) – PyTorch Module or Sequential that is duplicated to an ensemble
n_models (int) – number of model instances that are created and trained.
sparsity (float) – If model has pruned weights and the pruning mask should not be kept (see keep_pruning_mask), this value should be the amount the weight is pruned. The weights that were pruned in the original model are also pruned in the individual copies of the model in the ensemble. Each model in the ensemble is pruned independently.
keep_pruning_mask (bool) – If True the pruning masks of the original model will be used for all the models in the ensemble.
initializer (torch.nn.init) – Initializer used to initialize the different model instances. If None, all model instances are initialized the same.
parallel (bool) – If True the forward path of the models are calculated in parallel. It is possible that this is not faster than without parallelization.
combination_type (str) – In the ensemble the outputs of the individual models are combined. The final output can be the median or mean of the submodels outputs.

Return type:

None

forward(*input: Tensor | Tuple[Tensor]) → Tensor[source]

Forward passing function of the module. Passes input through all instances of the given model and returns a torch.Tensor with an additional dimension. It contains the individual outputs and as last entry their mean/median.

Parameters:: input (Union[torch.Tensor, Tuple[torch.Tensor]]) – Input vector as torch.Tensor or a Tuple of torch.Tensor
Returns:: Tensor containing the individual model outputs in the first n_models entries of the first dimension and their mean/median in the last entry.
Return type:: torch.Tensor

get_final_output(outs)[source]

set(variable: str, value) → None[source]

Set for all models in the ensemble a variable to the given value. :param variable: The name of the variable in the submodels that should be set. :type variable: str :param value: The value the variable is set to.

Return type:: None

prosper_nn.models.ensemble.ensemble.init_model(model: Module, init_func: Callable, *params, **kwargs) → None[source]

Method to initialize all parameter of a given model.

Parameters:

model (torch.nn.Module) – PyTorch Module or Sequential that is duplicated to an ensemble
init_func (Callable) – Function with which the weights of the model are initialized.
*params (list, optional) – List with params for init_func.
**kwargs (dict, optional) – Dict with kwargs for init_func.

Return type:

None

Example

# replace simple model with your architecture
ensemble = Ensemble(simple_model, n_models)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(ensemble.parameters())
for t in range(epochs):
    # Forward pass: Compute predicted y by passing x to the ensemble
    y_preds, mean_y = torch.split(ensemble(x), n_models)

    optimizer.zero_grad()
    # compute loss for each model instance
    loss = sum([criterion(y_pred, y) for y_pred in y_preds])
    loss.backward()
    optimizer.step()