Heat Conduction

Intputs

qprime: Linear heat generation rate (\(\frac{W}{m}\))
mdot: Mass flow rate (\(\frac{g}{s}\))
Tin: Temperature of the fuel boundary (\(K\))
R: Fuel radius (\(m\))
L: Fuel length (\(m\))
Cp: Heat capacity (\(\frac{J}{g \cdot K}\))
k: Thermal conductivity (\(\frac{W}{m \cdot K}\))

Output

T: Fuel centerline temperature (\(K\))

This data set consists of 1000 points with seven inputs and one output. The data set was constructed through Latin hypercube sampling of the seven input parameters for heat conduction through a fuel rod. These samples were then used to solve for the fuel centerline temperature analytically. The geometry of the problem is illustrated in the figure below, and it is assumed volumetric heat generation is uniform radially. The problem is defined by

\begin{equation} \frac{1}{r}\frac{d}{dr}(kr\frac{dT}{dr}) + q''' = 0, \end{equation}

with two boundary conditions: \(\frac{dT}{dr}|_{r = 0} = 0\) and \(T(R) = T_{in}\). Therefore, the temperature profile in the fuel is

\begin{equation} T(r) = \frac{q'}{4\pi k}(1 - (r / R)^2) + T_{in}. \end{equation}

The following are a few standard packages and functions that will prove helpful while using pyMAISE along with pyMAISE-specific functionality.

[1]:

# Importing Packages
import time
import cv2
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import uniform, randint
from sklearn.preprocessing import MinMaxScaler

# pyMAISE specific imports
import pyMAISE as mai
from pyMAISE.datasets import load_heat
from pyMAISE.preprocessing import scale_data, train_test_split, correlation_matrix

# Plot settings
matplotlib_settings = {
    "font.size": 12,
    "legend.fontsize": 11,
    "figure.figsize": (8, 8)
}
plt.rcParams.update(**matplotlib_settings)

Model Initialization

We will examine the performance of 7 models in this data set:

Linear regression: Linear,
Lasso regression: Lasso,
Support vector regression: SVM,
Decision tree regression: DT,
Random forest regression: RF,
K-nearest neighbors regression: KN,
Sequential dense neural networks: FNN.

For hyperparameter tuning each model, we must initialize the architecture and optimize search spaces. For all the classical models we initialize them using the scikit-learn default configurations. The FNNs are defined as sequential dense neural networks. For the input and hidden dense layers we are hyperparameter tuning the number of nodes, sublayer, and sublayer dropout rate. We are also tuning the number of hidden layers, the Adam learning rate, and batch size.

[7]:

# Initializing all the models wanted in the model)settings variable along with neurel network archetecture/optimization hps
model_settings = {
    "models": ["Linear", "Lasso", "SVM", "DT", "RF", "KN", "FNN"],
    "FNN": {
        "structural_params": {
            "Dense_hidden": {
                "num_layers": mai.Int(min_value=0, max_value=3),
                "units": mai.Int(min_value=25, max_value=400),
                "activation": "relu",
                "kernel_initializer": "normal",
                "sublayer": mai.Choice(["Dropout_hidden", "None"]),
                "Dropout_hidden": {
                    "rate": mai.Float(min_value=0.4, max_value=0.6),
                },
            },
            "Dense_output": {
                "units": ytrain.shape[-1],
                "activation": "linear",
                "kernel_initializer": "normal",
            },
        },
        "optimizer": "Adam",
        "Adam": {
            "learning_rate": mai.Float(min_value=1e-5, max_value=0.001),
        },
        "compile_params": {
            "loss": "mean_absolute_error",
            "metrics": ["mean_absolute_error"],
        },
        "fitting_params": {
            "batch_size": mai.Choice([8, 16, 32]),
            "epochs": 50,
            "validation_split": 0.15,
        },
    },
}
# Constructing Tuner object for the search space above
tuner = mai.Tuner(xtrain, ytrain, model_settings=model_settings)

Hyperparameter Tuning

For hyperparameter tuning the classical models, we use pyMAISE.Tuner.random_search with 300 iterations and 5-fold cross-validation. We do this because the classical models are relatively cheap to train, so we can better cover the hyperparameter search space. For the FNNs, we use pyMAISE.Tuner.nn_bayesian_search with 50 iterations and 5-fold cross-validation. We use Bayesian search with fewer configurations as the FNNs are more computationally expensive. The Bayesian search should converge on an optimal configuration within 30-50 iterations. The cross-validation ensures there is no bias in the configurations saved. The hyperparameter search spaces are outlined below.

[8]:

# Classical Model search space
random_search_spaces = {
    "Lasso": {
        "alpha": uniform(loc=0.0001, scale=0.0099), # 0.0001 - 0.01
    },
    "DT": {
        "max_depth": randint(low=5, high=50), # 5 - 50
        "max_features": [None, "sqrt", "log2", 2, 4, 6],
        "min_samples_split": randint(low=2, high=20), # 2 - 20
        "min_samples_leaf": randint(low=1, high=20), # 1 - 20
    },
    "SVM": {
        "kernel": ["linear", "poly", "rbf", "sigmoid"],
        "degree": randint(low=1, high=5),
        "gamma": ["scale", "auto"],
    },
    "RF": {
        "n_estimators": randint(low=50, high=200), # 50 - 200
        "criterion": ["squared_error", "absolute_error", "poisson"],
        "min_samples_split": randint(low=2, high=20), # 2 - 20
        "min_samples_leaf": randint(low=1, high=20), # 1 - 20
        "max_features": [None, "sqrt", "log2", 2, 4, 6],
    },
    "KN": {
        "n_neighbors": randint(low=1, high=20), # 1 - 20
        "weights": ["uniform", "distance"],
        "leaf_size": randint(low=1, high=30), # 1 - 30
        "p": randint(low=1, high=10), # 1 - 10
    },
}

# Running the tuner for with the search algorithm desired
start = time.time()
random_search_configs = tuner.random_search(
    param_spaces=random_search_spaces,
    n_iter=300,
    n_jobs=6,
    cv=5,
)
bayesian_search_configs = tuner.nn_bayesian_search(
    objective="r2_score",
    max_trials=50,
    cv=5,
)
print("Hyperparameter tuning took " + str((time.time() - start) / 60) + " minutes to process.")

Hyperparameter tuning took 29.920362746715547 minutes to process.

With the conclusion of hyper-parameter tuning we can see the training results of each iteration of the bayesian search.

[9]:

ax = tuner.convergence_plot(model_types="FNN")
ax.set_ylim([0, 1])
plt.show()

../_images/benchmarks_heat_conduction_17_0.png

After about ten iterations, the Bayesian search converges on the optimal solution for this parameter space.

Model Postprocessing

With the models tuned and the top pyMAISE.Settings.num_configs_saved saved, we can now pass these models to the PostProcessor for model comparison and analysis. We will increase the FNN models epochs for better performance.

[10]:

postprocessor = mai.PostProcessor(
    data=(xtrain, xtest, ytrain, ytest),
    model_configs=[random_search_configs, bayesian_search_configs],
    new_model_settings={
        "FNN": {"fitting_params": {"epochs": 200}},
    },
    yscaler=yscaler,
)

The regression performance metrics on fuel centerline temperature are shown below.

[11]:

postprocessor.metrics().drop("Parameter Configurations", axis=1)

[11]:

	Model Types	Train R2	Train MAE	Train MAPE	Train RMSE	Train RMSPE	Test R2	Test MAE	Test MAPE	Test RMSE	Test RMSPE
20	RF	0.997154	4.986508	0.386106	8.052169	0.631222	0.994233	7.923253	0.616351	11.824369	0.918915
18	RF	0.997033	5.139566	0.397534	8.221685	0.642415	0.994039	8.098878	0.630486	12.021820	0.936141
17	RF	0.998024	4.427935	0.342536	6.709407	0.533691	0.993746	8.205877	0.637626	12.314054	0.956969
19	RF	0.997136	5.263513	0.406251	8.077905	0.635408	0.993728	8.350372	0.647859	12.331529	0.954462
16	RF	0.998684	3.552243	0.275988	5.476834	0.438886	0.993029	8.551964	0.666155	13.000574	1.011017
14	DT	0.993846	8.446816	0.650752	11.841301	0.915109	0.985220	12.827502	1.009586	18.930021	1.501253
13	DT	0.993846	8.446816	0.650752	11.841301	0.915109	0.985220	12.827502	1.009586	18.930021	1.501253
15	DT	0.996577	6.133811	0.476671	8.831676	0.699559	0.984935	12.667243	0.999214	19.111302	1.520477
11	DT	0.995777	6.739139	0.520000	9.808999	0.759180	0.984636	12.876282	1.016210	19.300072	1.535464
12	DT	0.994587	7.802882	0.601207	11.106283	0.863034	0.984309	13.410189	1.055767	19.504322	1.550309
28	FNN	0.990703	4.107766	0.322267	14.554742	1.100226	0.974040	7.219007	0.559021	25.087926	1.846443
30	FNN	0.989318	8.569111	0.674231	15.601300	1.283347	0.973228	11.522918	0.926079	25.476912	2.078072
26	FNN	0.984739	4.631119	0.358272	18.647505	1.472439	0.969190	7.493166	0.586952	27.330885	2.143828
27	FNN	0.981239	11.610469	0.900106	20.675833	1.634222	0.967317	14.332504	1.134863	28.149507	2.258070
29	FNN	0.940880	26.184468	1.958679	36.702944	2.808690	0.932436	28.152282	2.155911	40.473259	3.180488
10	SVM	0.895306	42.695727	3.314075	48.841919	3.815481	0.857033	50.145252	3.946441	58.874553	4.650346
8	SVM	0.895306	42.695727	3.314075	48.841919	3.815481	0.857033	50.145252	3.946441	58.874553	4.650346
7	SVM	0.895306	42.695727	3.314075	48.841919	3.815481	0.857033	50.145252	3.946441	58.874553	4.650346
6	SVM	0.895306	42.695727	3.314075	48.841919	3.815481	0.857033	50.145252	3.946441	58.874553	4.650346
9	SVM	0.895306	42.695727	3.314075	48.841919	3.815481	0.857033	50.145252	3.946441	58.874553	4.650346
5	Lasso	0.839945	52.227829	4.062700	60.390247	4.745456	0.850541	52.731179	4.168716	60.196391	4.797726
4	Lasso	0.839980	52.218883	4.062042	60.383614	4.744978	0.850534	52.729709	4.168467	60.197942	4.797652
3	Lasso	0.840101	52.185042	4.059570	60.360787	4.743353	0.850487	52.726402	4.167712	60.207380	4.797624
2	Lasso	0.840165	52.163806	4.058018	60.348782	4.742518	0.850443	52.726202	4.167401	60.216178	4.797823
1	Lasso	0.840206	52.148545	4.056897	60.340929	4.741985	0.850402	52.726049	4.167163	60.224350	4.798088
0	Linear	0.840331	52.061837	4.050545	60.317326	4.740691	0.849952	52.749655	4.167604	60.315047	4.802679
21	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.765921	66.107932	5.263937	75.333995	6.082310
24	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.762813	65.333817	5.200211	75.832477	6.125086
23	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.762162	67.368345	5.362470	75.936563	6.129685
22	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.762162	67.368345	5.362470	75.936563	6.129685
25	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.742511	64.803547	5.156468	79.011268	6.367217

This data set represents radial heat conduction, so we do not expect linear models to perform well. This is shown by the relatively poor performance of Linear and Lasso. The best-performing models were RF, DT, and FNN with test and train r-squared above 0.99.

Using the pyMAISE.PostProcessor.print_model, we can see the optimal hyperparameter configurations for each model.

[12]:

for model in ["Lasso", "DT", "RF", "KN", "FNN"]:
    postprocessor.print_model(model_type=model)
    print()

Model Type: Lasso
  alpha: 0.00028379044515601027

Model Type: DT
  max_depth: 40
  max_features: None
  min_samples_leaf: 6
  min_samples_split: 4

Model Type: RF
  criterion: poisson
  max_features: 6
  min_samples_leaf: 3
  min_samples_split: 2
  n_estimators: 138

Model Type: KN
  leaf_size: 12
  n_neighbors: 10
  p: 1
  weights: distance

Model Type: FNN
  Structural Hyperparameters
    Layer: Dense_hidden_0
      units: 251
      sublayer: None
    Layer: Dense_hidden_1
      units: 184
      sublayer: None
    Layer: Dense_hidden_2
      units: 47
      sublayer: None
    Layer: Dense_output_0
  Compile/Fitting Hyperparameters
    Adam_learning_rate: 0.0008821712781015931
    batch_size: 8
Model: "FNN"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 Dense_hidden_0 (Dense)      (None, 251)               2008

 Dense_hidden_1 (Dense)      (None, 184)               46368

 Dense_hidden_2 (Dense)      (None, 47)                8695

 Dense_output_0 (Dense)      (None, 1)                 48

=================================================================
Total params: 57119 (223.12 KB)
Trainable params: 57119 (223.12 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

Below is the network plot for the FNN.

[13]:

postprocessor.nn_network_plot(
    to_file="./supporting/heat_conduction_network.png",
    show_shapes=True,
    show_layer_names=True,
    expand_nested=True,
    show_layer_activations=True,
)

[13]:

../_images/benchmarks_heat_conduction_25_0.png

To visualize the performance of these models, we can use the pyMAISE.PostProcessor.diagonal_validation_plot functions to produce diagonal validation plots.

[14]:

models = np.array([["Linear", "Lasso"], ["DT", "KN"], ["RF", "SVM"]])
fig = plt.figure(figsize=(15, 25))
gs = fig.add_gridspec(4, 2)
for i in range(models.shape[0]):
    for j in range(models.shape[1]):
        ax = fig.add_subplot(gs[i, j])
        ax = postprocessor.diagonal_validation_plot(model_type=models[i, j])
        ax.set_title(models[i, j])
ax = fig.add_subplot(gs[3, :])
ax = postprocessor.diagonal_validation_plot(model_type="FNN")
ax.set_title("FNN")
plt.show()

../_images/benchmarks_heat_conduction_27_0.png

The performance differences between RF/DT with the other models are apparent along with the overfitting of KN. The RF and DT predictions are closely spread along \(y=x\) while the KN test predictions are over-predicted at lower temperatures and under-predicted at higher temperatures.

Similarly, the pyMAISE.PostProcessor.validation_plot function produces validation plots showing each prediction’s absolute relative error.

[15]:

fig = plt.figure(figsize=(15, 25))
gs = fig.add_gridspec(4, 2)
for i in range(models.shape[0]):
    for j in range(models.shape[1]):
        ax = fig.add_subplot(gs[i, j])
        ax = postprocessor.validation_plot(model_type=models[i, j])
        ax.set_title(models[i, j])
ax = fig.add_subplot(gs[3, :])
ax = postprocessor.validation_plot(model_type="FNN")
ax.set_title("FNN")
plt.show()

../_images/benchmarks_heat_conduction_29_0.png

The performance of the models is best represented by the magnitudes observed on the y-axis; however, even RF gets as high as \(>10.0\%\) error.

Finally, the most performant FNN learning curve is shown by pyMAISE.PostProcessor.nn_learning_plot.

[16]:

postprocessor.nn_learning_plot()
plt.show()

../_images/benchmarks_heat_conduction_31_0.png

From the learning curve we can see the top performing neural network based on test \(R^2\) is slightly overfit with validation above training.

Heat Conduction

pyMAISE Initialization

Model Initialization

Hyperparameter Tuning

Model Postprocessing