pyMAISE.Tuner

class pyMAISE.Tuner(xtrain, ytrain, model_settings)[source]

Bases: object

Hyperparameter tuning object.

Supported Models

Supported models include

from [PVG+11] and sequential neural networks from [C+15].

Supported Neural Network Layers and Optimizers

pyMAISE supports the following neural network layers using [C+15]:

and the following optimizers:

Note

For additional layer or optimizer support, submit a detailed issue at the pyMAISE GitHub repository outlining the layer or optimizer required.

Parameters:

  • xtrain (xarray.DataArray) – Input training data.

  • ytrain (xarray.DataArray) – Output training data.

  • model_settings (dict of int, float, str, or pyMAISE.HyperParameters) –

    This dictionary specifies the name of the models of interest, which are assigned as a list to the models key. The model names are provided in the Supported Models section; all names that do not match those keys are assumed to be neural network models. For specific hyperparameters, please refer to the links provided for the models.

    For classical models, sklearn models [PVG+11], this dictionary specifies the hyperparameters different from default but remain constant throughout the hyperparameter tuning process. This is done by assigning a sub-dictionary under the key of the model’s name.

    For neural network models [C+15], this dictionary specifies both hyperparameters that remain constant throughout tuning and the tuning space using pyMAISE.Int, pyMAISE.Float, pyMAISE.Choice, pyMAISE.Boolean, and pyMAISE.Fixed. This is done in the same way as classical models, where hyperparameters and their values are specified in sub-dictionaries under their model’s key. In addition, number of layers, optimizers, wrappers, and sublayers can be specified.

Warning

When hyperparameter tuning a neural network with multiple of the same layer in one model, ensure the names of the layers are different, but the keywords are still present. For example, a dense sequential neural network with multiple dense layers can use names like Dense_input, Dense_hidden, and Dense_output.

Examples

Given 2D input and output training data (xtrain, ytrain) an example using linear and random forest models.

import pyMAISE as mai

model_settings = {
    "models": ["Linear", "RF"],
    "RF": {
        "n_estimators": 150,
    },
}
tuner = mai.Tuner(xtrain, ytrain, model_settings)

From the above, we see we specify a linear model with default hyperparameters and a random forest model with all default hyperparameters except for 150 estimators.

Given 3D input and 2D output time series data (xtrain, ytrain) from pyMAISE.preprocessing.SplitSequence, we can define a CNN-LSTM.

import pyMAISE as mai
from tensorflow.keras.layers import TimeDistributed
from tensorflow.keras.callbacks import ReduceLROnPlateau

cnn_lstm_structure = {
    "Reshape_input": {
        "target_shape": (2, 2, xtrain.shape[-1])
    },
    "Conv1D": {
        "filters": mai.Int(min_value=50, max_value=150),
        "kernel_size": 1,
        "activation": "relu",
        "wrapper": (
            TimeDistributed, {
                "input_shape": (None, 2, xtrain.shape[-1])
            },
        ),
    },
    "MaxPooling1D": {
        "pool_size": 2,
        "wrapper": TimeDistributed,
    },
    "Flatten": {
        "wrapper": TimeDistributed,
    },
    "LSTM": {
        "num_layers": mai.Int(min_value=0, max_value=3),
        "units": mai.Int(min_value=20, max_value=100),
        "activation": "tanh",
        "recurrent_activation": "sigmoid",
        "recurrent_dropout": mai.Choice([0.0, 0.2, 0.4, 0.6]),
        "return_sequences": True,
    },
    "LSTM_output": {
        "units": mai.Int(min_value=20, max_value=100),
        "activation": "tanh",
        "recurrent_activation": "sigmoid",
    },
    "Dense": {
        "units": ytrain.shape[-1],
        "activation": "linear",
    },
}
fitting = {
    "batch_size": 512,
    "epochs": 5,
    "validation_split":0.15,
    "callbacks": [
        ReduceLROnPlateau(
            monitor='val_mean_absolute_error',
            factor=0.8,
            patience=2,
            min_lr=0,
            verbose=1,
        ),
        EarlyStopping(
            monitor="val_mean_absolute_error",
            patience=3,
        )
    ]
}
adam = {
    "learning_rate": mai.Float(min_value=0.00001, max_value=0.001),
    "clipnorm": mai.Float(min_value=0.8, max_value=1.2),
    "clipvalue": mai.Float(min_value=0.3, max_value=0.7),
}
compiling = {
    "loss": "mean_absolute_error",
    "metrics": ["mean_absolute_error"],
}

model_settings = {
    "models": ["CNN-LSTM"],
    "CNN-LSTM": {
        "structural_params": cnn_lstm_structure,
        "optimizer": "Adam",
        "Adam": adam,
        "compile_params": compiling,
        "fitting_params": fitting,
    },
}
tuner = mai.Tuner(xtrain, ytrain, model_settings=model_settings)

We see that we defined a neural network with 7 layers with the following tuning space:

  • 1D convolutional layer filters,

  • hidden LSTM number of layers,

  • hidden LSTM units,

  • hidden LSTM recurrent dropout,

  • output LSTM units,

  • Adam learning rate,

  • Adam clipnorm,

  • Adam clipvalue.

Additionally, the Conv1D, MaxPooling1D, and Flatten layers use the keras.layers.TimeDistributed wrapper to accommodate the temporal dimension.

__init__(xtrain, ytrain, model_settings)[source]

Methods

bayesian_search(param_spaces[, models, ...])

Bayesian search over hyperparameter space for classical models.

convergence_plot([ax, model_types])

Create a convergence plot for search using pyMAISE.Tuner.cv_performance_data.

grid_search(param_spaces[, models, scoring, ...])

Grid search over hyperparameter space for classical models.

manual_search([models, model_settings])

Fit a single hyperparameter configuration.

nn_bayesian_search([models, objective, ...])

Bayesian search for neural networks.

nn_grid_search([models, objective, ...])

Grid search for neural networks.

nn_hyperband_search([models, objective, ...])

Hyperband search for neural networks.

nn_random_search([models, objective, ...])

Random search for neural networks.

random_search(param_spaces[, models, ...])

Random search over hyperparameter space for classical models.

Attributes

cv_performance_data

Cross-validation performance, mean and standard deviation of the test score, for each model.

supported_classical_models

Dictionary of supported models.

Bayesian search over hyperparameter space for classical models. This function uses skopt.BayesSearchCV [sko16].

Parameters:

  • param_spaces (dict of dict of skopt.space.Dimension instance) – The parameters which will be tuned through a Bayesian search over every configuration of hyperparameter in each model dictionary. Each parameter is defined using skopt.space.Dimension instances (Real, Integer, or Categorical).

  • models (list of str or None, default=None) – A list of model names defined in the initialization of pyMAISE.Tuner. If None then all classical models are subject to Bayesian search.

Note

For information on scoring, n_iter, optimizer_kwargs, fit_params, n_jobs, n_points, refit, cv, and pre_dispatch refer to skopt’s documentation.

Returns:

data – The hyperparameters and models for the top pyMAISE.Settings.num_configs_saved for each model. If fewer configurations are provided than pyMAISE.Settings.num_configs_saved then all are taken.

Return type:

dict of tuple(pd.DataFrame, model object)

convergence_plot(ax=None, model_types=None)[source]

Create a convergence plot for search using pyMAISE.Tuner.cv_performance_data.

Parameters:

  • ax (matplotlib.pyplot.axis or None, default=None) – Axis object. If None then one is created.

  • model_types (list of str or None, default=None) – List of model names to add to the convergence plot. If None then all are added.

Returns:

ax – Axis object.

Return type:

matplotlib.pyplot.axis or None, default=None

property cv_performance_data

Cross-validation performance, mean and standard deviation of the test score, for each model.

Type:

list of float

Grid search over hyperparameter space for classical models. This function uses sklearn.model_selection.GridSearchCV [PVG+11].

Parameters:

  • param_spaces (dict of dict of list) – The parameters which will be tuned through an exhaustive search over every configuration of hyperparameter in each model dictionary. Each parameter is defined as a dictionary key and assigned a list.

  • models (list of str or None, default=None) – A list of model names that were defined in the initialization of pyMAISE.Tuner. If None then all classical models are subject to grid search.

Note

For information on scoring, n_jobs, refit, cv, and pre_dispatch refer to sklearn’s documentation.

Returns:

data – The hyperparameters and models for the top pyMAISE.Settings.num_configs_saved for each model. If fewer configurations are provided than pyMAISE.Settings.num_configs_saved then all are taken.

Return type:

dict of tuple(pd.DataFrame, model object)

Fit a single hyperparameter configuration.

Parameters:

  • models (list of str or None, default=None) – The names of the models to be fit using manual search. If None then all the models specified in the initialization of the pyMAISE.Tuner are fit.

  • model_settings (dict of int, float, or str) – The model settings for the models which are sub-dictionaries under the model key. If None then the hyperparameter configurations specified in the initialization of the pyMAISE.Tuner are used.

Returns:

data – The hyperparameters and models for each model type.

Return type:

dict of tuple(pd.DataFrame, model object)

Bayesian search for neural networks. This function uses keras_tuner.oracles.BayesianOptimizationOracle with pyMAISE.CVTuner for cross-validation. Iterate over sampled hyperparameter space using Bayesian optimization and return the top models for each model type.

Parameters:

  • models (list of string or None, default=None) – The names of the neural network models for Bayesian search. If None, then all neural networks are fit with Bayesian search.

  • objective (str or keras_tuner.Objective, default=None) – The objective of the search. If the objective is a str of a sklearn.metrics, then that is used as the objective. Otherwise the builtin objectives within KerasTuner are used.

  • cv (int or cross-validation generator, default=5) – If an int then either sklearn.model_selection.StratifiedKFold or sklearn.model_selection.KFold are used depending on the pyMAISE.Settings.problem_type and output data type. If the problem is a classification problem and the output data is either binary or multiclass, then sklearn.model_selection.StratifiedKFold is used.

  • shuffle (bool, default=False) – Whether to shuffle the data before cross-validation split.

Note

For information on max_trials, num_initial_points, alpha, beta, hyperparameters, tune_new_entries, max_retries_per_trial, max_consecutive_failed_trials, overwrite, directory, and project_name refer to KerasTuner documentation.

Returns:

data – The hyperparameters and models for the top pyMAISE.Settings.num_configs_saved for each model. If fewer configurations are provided, than pyMAISE.Settings.num_configs_saved then all are taken.

Return type:

dict of tuple(pd.DataFrame, model object)

Grid search for neural networks. This function uses keras_tuner.oracles.GridSearchOracle with pyMAISE.CVTuner for cross validation. Iterate over the defined search space and return the top models for each model type.

Parameters:

  • models (list of string or None, default=None) – The names of the neural network models for grid search. If None, then all neural networks are fit with grid search.

  • objective (str or keras_tuner.Objective, default=None) – The objective of the search. If the objective is a str of a sklearn.metrics, then that is used as the objective. Otherwise, the built-in objectives within KerasTuner are used.

  • cv (int or cross-validation generator, default=5) –

    If an int, then either sklearn.model_selection.StratifiedKFold or sklearn.model_selection.KFold are used depending on the pyMAISE.Settings.problem_type and output data type. If the problem is a classification problem and the output data is either binary or multiclass, then sklearn.model_selection.StratifiedKFold is used.

  • shuffle (bool, default=False) – Whether to shuffle the data before cross-validation split.

Note

For information on max_trials, hyperparameters, allow_new_entries, tune_new_entries, max_consecutive_failed_trials, overwrite, directory, and project_name refer to the KerasTuner documentation.

Returns:

data – The hyperparameters and models for the top pyMAISE.Settings.num_configs_saved for each model. If fewer configurations are provided, than pyMAISE.Settings.num_configs_saved then all are taken.

Return type:

dict of tuple(pd.DataFrame, model object)

Hyperband search for neural networks. This function uses keras_tuner.oracles.HyperbandOracle with pyMAISE.CVTuner for cross validation.

Parameters:

  • models (list of string or None, default=None) – The names of the neural network models for grid search. If None, then all neural networks are fit with grid search.

  • objective (str or keras_tuner.Objective, default=None) – The objective of the search. If the objective is a str of a sklearn.metrics then that is used as the objective. Otherwise the builtin objectives within KerasTuner are used.

  • cv (int or cross-validation generator, default=5) –

    If an int, then either sklearn.model_selection.StratifiedKFold or sklearn.model_selection.KFold are used depending on the pyMAISE.Settings.problem_type and output data type. If the problem is a classification problem and the output data is either binary or multiclass then sklearn.model_selection.StratifiedKFold is used.

  • shuffle (bool, default=False) – Whether to shuffle the data before cross-validation split.

Note

For information on max_epochs, factor, hyperband_iterations, hyperparameters, tune_new_entries, allow_new_entries, max_retries_per_trial, max_consecutive_failed_trials, overwrite, directory, and project_name refer to KerasTuner documentation.

Returns:

data – The hyperparameters and models for the top pyMAISE.Settings.num_configs_saved for each model. If fewer configurations are provided, than pyMAISE.Settings.num_configs_saved then all are taken.

Return type:

dict of tuple(pd.DataFrame, model object)

Random search for neural networks. This function uses keras_tuner.oracles.RandomSearchOracle with pyMIASE.CVTuner for cross validation. Sample the defined search space based on a random distribution for each model type.

Parameters:

  • models (list of string or None, default=None) – The names of the neural network models for random search. If None, then all neural networks are fit with random search.

  • objective (str or keras_tuner.Objective, default=None) – The objective of the search. If the objective is a str of a sklearn.metrics, then that is used as the objective. Otherwise, the built-in objectives within KerasTuner are used.

  • cv (int or cross-validation generator, default=5) –

    If an int, then either sklearn.model_selection.StratifiedKFold or sklearn.model_selection.KFold are used depending on the pyMAISE.Settings.problem_type and output data type. If the problem is a classification problem and the output data is either binary or multiclass, then sklearn.model_selection.StratifiedKFold is used.

  • shuffle (bool, default=False) – Whether to shuffle the data before cross-validation split.

Note

For information on max_trials, hyperparameters, allow_new_entries, tune_new_entries, max_retries_per_trial, max_consecutive_failed_trials, overwrite, directory, and project_name refer to KerasTuner documentation.

Returns:

data – The hyperparameters and models for the top pyMAISE.Settings.num_configs_saved for each model. If fewer configurations are provided, than pyMAISE.Settings.num_configs_saved then all are taken.

Return type:

dict of tuple(pd.DataFrame, model object)

Random search over hyperparameter space for classical models. This function uses sklearn.model_selection.RandomizedSearchCV [PVG+11].

Parameters:

  • param_spaces (dict of dict of list or distributions) – The parameters which will be tuned through a random search over every configuration of hyperparameter in each model dictionary. Each parameter is defined as a dictionary key and assigned a list or distribution with an rvs method.

  • models (list of str or None, default=None) – A list of model names defined in the initialization of pyMAISE.Tuner. If None then all classical models are subject to grid search.

Note

For information on scoring, n_iter, n_jobs, refit, cv, and pre_dispatch refer to sklearn’s documentation.

Returns:

data – The hyperparameters and models for the top pyMAISE.Settings.num_configs_saved for each model. If fewer configurations are provided than pyMAISE.Settings.num_configs_saved then all are taken.

Return type:

dict of tuple(pd.DataFrame, model object)

supported_classical_models = {'AB': <class 'pyMAISE.methods._adaboost.AdaBoost'>, 'DT': <class 'pyMAISE.methods._dtree.DecisionTree'>, 'EN': <class 'pyMAISE.methods._elastic.ElasticNet'>, 'ET': <class 'pyMAISE.methods._extra_trees.ExtraTrees'>, 'GB': <class 'pyMAISE.methods._gradient_boosting.GradientBoosting'>, 'GP': <class 'pyMAISE.methods._gaussian_process.GaussianProcess'>, 'KN': <class 'pyMAISE.methods._kneighbors.KNeighbors'>, 'Lasso': <class 'pyMAISE.methods._lasso.LassoRegression'>, 'Linear': <class 'pyMAISE.methods._linear.LinearRegression'>, 'Logistic': <class 'pyMAISE.methods._logistic_regression.LogisticRegression'>, 'MultiOutput': <class 'pyMAISE.methods._multi_output.MultiOutput'>, 'RD': <class 'pyMAISE.methods._ridge.RidgeRegression'>, 'RF': <class 'pyMAISE.methods._rforest.RandomForest'>, 'SVM': <class 'pyMAISE.methods._svm.SVM'>, 'Stacking': <class 'pyMAISE.methods._stacking.Stacking'>}

Dictionary of supported models.

Type:

dict of pyMAISE.methods