BWR Micro Core

Inputs

PSZ: Fuel bundle region Power Shaping Zone (PSZ).
DOM: Fuel bundle region Dominant zone (DOM).
vanA: Fuel bundle region vanishing zone A (VANA).
vanB: Fuel bundle region vanishing zone B (VANB)
subcool: Represents moderator inlet conditions. Core inlet subcooling is interpreted as at the steam dome pressure (i.e., not core-averaged pressure). The input value for subcooling will automatically be increased to account for this fact. (Btu/lb)?
CRD: Defines the position of all control rod groups (banks).
flow_rate: Defines essential global design data for rated coolant mass flux for the active core, \(\frac{kg}{(cm^{2}-hr)}\). Coolant mass flux equals active core flow divided by core cross-section area. The core cross-section area is DXA 2 times the number of assemblies.
power_density: Defines essential global design data for rated power density using cold dimensions, \((\frac{kw}{liter})\).
VFNGAP: Defines the narrow water gap width ratio to the sum of the narrow and wide water gap widths.

Output

K-eff: Reactivity coefficient k-effective, the effective neutron multiplication factor.
Max3Pin: Maximum planar-averaged pin power peaking factor.
Max4Pin: maximum pin-power peaking factor, \(F_{q}\), (which includes axial intranodal peaking).
F-delta-H: Ratio of max-to-average enthalpy rises in a channel.
Max-Fxy: Maximum radial pin-power peaking factor.

The data set consists of 2000 data points with nine inputs and five outputs. This data set was constructed through uniform and normal sampling of the 9 input parameters for a boiling water reactor (BWR) micro-core. These samples were then used to solve for reactor characteristic changes in heat distribution and neutron flux. This BWR micro-core consists of 4 radially and axially heterogeneous assemblies of the same type constructed in a 2x2 grid with a control blade placed in the center. A single assembly composition can be seen in the figure below. A single assembly was broken into seven zones where each zone’s 2D radial cross-sectional information was constructed using CASMO-4. These cross sectional libraries were then processed through CMSLINK for SIMULATE-3 to interpret. The core geometry and physics were implemented and modeled using SIMULATE-3 [RFS21].

Note: The reference uses the same model as this example, but the data generated differs from the paper.

To start, we import all the needed packages.

[1]:

from pyMAISE.datasets import load_BWR
from pyMAISE.preprocessing import correlation_matrix, train_test_split, scale_data
import pyMAISE as mai

import time
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import uniform, randint
from sklearn.preprocessing import MinMaxScaler

# Plot settings
matplotlib_settings = {
    "font.size": 12,
    "legend.fontsize": 11,
    "figure.figsize": (8, 8)
}
plt.rcParams.update(**matplotlib_settings)

Model Initialization

Given this data set has a multi-dimensional output we will compare the performance of 6 machine learning (ML) models:

Linear regression: Linear,
Lasso regression: Lasso,
Decision tree regression: DT,
Random forest regression: RF,
K-nearest neighbors regression: KN,
Sequential dense neural networks: FNN.

For hyperparameter tuning, we initialize all classical models as scikit-learn defaults. For the FNN we define input and output layers with possible dropout layers. These layers include hyperparameter tuning of their number of nodes, use of sublayers, and the rate of dropout. The dense hidden layers include tuning of their depth.

[7]:

model_settings = {
    "models": ["Linear", "Lasso", "DT", "RF", "KN", "FNN"],
    "FNN": {
        "structural_params": {
            "Dense_hidden": {
                "num_layers": mai.Int(min_value=0, max_value=5),
                "units": mai.Int(min_value=25, max_value=600),
                "activation": "relu",
                "kernel_initializer": "normal",
                "sublayer": mai.Choice(["Dropout_hidden", "None"]),
                "Dropout_hidden": {
                    "rate": mai.Float(min_value=0.2, max_value=0.6),
                },
            },
            "Dense_output": {
                "units": ytrain.shape[-1],
                "activation": "linear",
                "kernel_initializer": "normal",
            },
        },
        "optimizer": "Adam",
        "Adam": {
            "learning_rate": mai.Float(min_value=1e-5, max_value=0.001),
        },
        "compile_params": {
            "loss": "mean_absolute_error",
            "metrics": ["mean_absolute_error"],
        },
        "fitting_params": {
            "batch_size": mai.Choice([8, 16, 32, 64]),
            "epochs": 100,
            "validation_split": 0.15,
        },
    },
}
tuner = mai.Tuner(xtrain, ytrain, model_settings=model_settings)

Hyper-parameter Tuning

We use a random search with 200 iterations and 5 cross-validation splits (1000 fits per model) for the classical models. The hyperparameter search space is defined for all but linear regression in which the default scikit-learn configuration will be tested. For the FNN, we use Bayesian search with 150 iterations and 5 cross-validation splits (750 fits total). This offers possible convergence on an optimal configuration without taking excessive time. Classical models tend to be simpler than neural networks, so many random search iterations are possible.

[8]:

random_search_spaces = {
    "Lasso": {
        "alpha": uniform(loc=0.0001, scale=0.0099), # 0.0001 - 0.01
    },
    "DT": {
        "max_depth": randint(low=5, high=50), # 5 - 50
        "max_features": [None, "sqrt", "log2", 2, 4, 6],
        "min_samples_split": randint(low=2, high=20), # 2 - 20
        "min_samples_leaf": randint(low=1, high=20), # 1 - 20
    },
    "RF": {
        "n_estimators": randint(low=50, high=200), # 50 - 200
        "criterion": ["squared_error", "absolute_error", "poisson"],
        "min_samples_split": randint(low=2, high=20), # 2 - 20
        "min_samples_leaf": randint(low=1, high=20), # 1 - 20
        "max_features": [None, "sqrt", "log2", 2, 4, 6],
    },
    "KN": {
        "n_neighbors": randint(low=1, high=20), # 1 - 20
        "weights": ["uniform", "distance"],
        "leaf_size": randint(low=1, high=30), # 1 - 30
        "p": randint(low=1, high=10), # 1 - 10
    },
}
start = time.time()
random_search_configs = tuner.random_search(
    param_spaces=random_search_spaces,
    n_iter=200,
    n_jobs=6,
    cv=5,
)
bayesian_search_configs = tuner.nn_bayesian_search(
    objective="r2_score",
    max_trials=150,
    cv=5,
)
print("Hyperparameter tuning took " + str((time.time() - start) / 60) + " minutes to process.")

Hyperparameter tuning took 356.09841007788975 minutes to process.

Below is the convergence plot for the FNN Bayesian search.

[9]:

ax = tuner.convergence_plot(model_types="FNN")
ax.set_ylim([0, 1])
plt.show()

The search could not produce an FNN configuration with performance better than ~0.55 validation \(R^2\).

Model Post-processing

With the models tuned and the top pyMAISE.Settings.num_configs_saved saved, we can now pass these models to the pyMAISE.PostProcessor for model comparison and analysis. We will increase the `FNN’ models ‘epochs’ for better performance.

[10]:

postprocessor = mai.PostProcessor(
    data=(xtrain, xtest, ytrain, ytest),
    model_configs=[random_search_configs, bayesian_search_configs],
    new_model_settings={
        "FNN": {"fitting_params": {"epochs": 500}},
    },
    yscaler=yscaler,
)

To compare the performance of these models, we will compute five metrics for both the training and testing data:

mean absolute percentage error: MAPE \(=\frac{100}{n} \sum_{i = 1}^n \frac{|y_i - \hat{y}_i|}{\text{max}(\epsilon, |y_i|)}\),
root mean squared error RMSE \(=\sqrt{\frac{1}{n}\sum^n_{i = 1}(y_i - \hat{y}_i)^2}\),
root mean squared percentage error RMSPE \(= \sqrt{\frac{1}{n}\sum_{i = 1}^n\Big(\frac{y_i - \hat{y}_i}{\text{max}(\epsilon, |y_i|)}\Big)^2}\),
mean absolute error MAE = \(=\frac{1}{n}\sum^n_{i = 1}|y_i - \hat{y}_i|\),
and r-squared R2 \(=1 - \frac{\sum^n_{i = 1}(y_i - \hat{y}_i)^2}{\sum^n_{i = 1}(y_i - \bar{y}_i)^2}\),

where \(y\) is the actual outcome, \(\bar{y}\) is the average outcome, \(\hat{y}\) is the model predicted outcome, and \(n\) is the number of observations. The performance metrics for each model are evaluated below for each output starting with K-eff.

[11]:

postprocessor.metrics(y="K-eff").drop("Parameter Configurations", axis=1)

[11]:

	Model Types	Train R2	Train MAE	Train MAPE	Train RMSE	Train RMSPE	Test R2	Test MAE	Test MAPE	Test RMSE	Test RMSPE
23	FNN	0.998773	0.001366	0.159733	0.003153	0.391195	0.996537	0.002056	0.247558	0.005048	0.641347
13	RF	0.997293	0.002379	0.277732	0.004683	0.588231	0.994875	0.003065	0.355261	0.006141	0.747542
15	RF	0.996531	0.002675	0.313134	0.005301	0.666018	0.994532	0.003235	0.375361	0.006343	0.772951
6	DT	0.997931	0.001813	0.210028	0.004094	0.525573	0.994320	0.003028	0.347965	0.006465	0.776412
11	RF	0.997722	0.002118	0.246561	0.004296	0.539748	0.994165	0.003109	0.360690	0.006552	0.798555
14	RF	0.995979	0.002822	0.331567	0.005707	0.719813	0.994097	0.003282	0.381155	0.006590	0.806652
25	FNN	0.997856	0.003129	0.345450	0.004168	0.485280	0.993797	0.003896	0.441898	0.006756	0.830296
7	DT	0.997386	0.002163	0.250615	0.004602	0.583470	0.993789	0.003166	0.363948	0.006760	0.808992
22	FNN	0.998955	0.001116	0.129889	0.002910	0.367927	0.993190	0.002494	0.296647	0.007078	0.878726
8	DT	0.998563	0.001511	0.174124	0.003412	0.427937	0.992987	0.003176	0.365299	0.007183	0.865607
12	RF	0.999016	0.001389	0.165696	0.002824	0.355474	0.992841	0.003463	0.409076	0.007257	0.896557
24	FNN	0.997302	0.002804	0.317049	0.004675	0.552303	0.991486	0.003694	0.427629	0.007915	0.976416
21	FNN	0.998260	0.002198	0.249648	0.003754	0.462599	0.991150	0.003482	0.407892	0.008069	1.010183
9	DT	0.994976	0.003008	0.351389	0.006380	0.802997	0.990316	0.003773	0.438014	0.008441	1.024557
10	DT	0.994976	0.003008	0.351389	0.006380	0.802997	0.990316	0.003773	0.438014	0.008441	1.024557
18	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.824066	0.019533	2.388407	0.035979	4.737138
19	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.823633	0.018537	2.296553	0.036023	4.782349
17	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.823399	0.018694	2.319553	0.036047	4.798674
16	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.822128	0.018935	2.347963	0.036176	4.818303
20	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.819803	0.019067	2.368093	0.036412	4.875045
5	Lasso	0.644822	0.044448	5.102120	0.053642	6.645229	0.615436	0.044679	5.048666	0.053193	6.461506
4	Lasso	0.644943	0.044455	5.102349	0.053633	6.642841	0.615330	0.044696	5.050085	0.053200	6.460920
3	Lasso	0.645001	0.044457	5.102447	0.053629	6.641681	0.615272	0.044704	5.050828	0.053204	6.460678
1	Lasso	0.645191	0.044468	5.102834	0.053614	6.637730	0.615053	0.044736	5.053618	0.053219	6.459947
2	Lasso	0.645256	0.044471	5.102943	0.053609	6.636330	0.614972	0.044748	5.054660	0.053225	6.459675
0	Linear	0.645844	0.044644	5.113036	0.053565	6.601073	0.610855	0.045158	5.091797	0.053509	6.460909

[12]:

postprocessor.metrics(y="Max3Pin").drop("Parameter Configurations", axis=1)

[12]:

	Model Types	Train R2	Train MAE	Train MAPE	Train RMSE	Train RMSPE	Test R2	Test MAE	Test MAPE	Test RMSE	Test RMSPE
13	RF	0.991386	0.067592	1.441376	0.148336	3.349790	0.985910	0.085857	1.771153	0.182047	3.608965
11	RF	0.993123	0.058166	1.245006	0.132547	3.051272	0.985888	0.083662	1.731283	0.182192	3.683613
23	FNN	0.996665	0.046158	1.140229	0.092296	2.340877	0.985802	0.081495	1.867539	0.182743	3.993780
15	RF	0.988637	0.078723	1.670788	0.170372	3.805057	0.983573	0.093657	1.927749	0.196567	3.910202
14	RF	0.986899	0.085257	1.868505	0.182939	4.177348	0.982549	0.096739	2.051692	0.202599	4.123379
6	DT	0.996258	0.046695	1.027971	0.097772	2.014262	0.982124	0.088756	1.895437	0.205051	3.697189
12	RF	0.996584	0.042010	0.895571	0.093410	2.079312	0.982099	0.095073	2.010146	0.205198	4.178493
7	DT	0.995048	0.055587	1.221065	0.112476	2.308505	0.981479	0.088564	1.878852	0.208717	3.793688
25	FNN	0.997058	0.046623	1.004573	0.086694	1.658235	0.981335	0.085710	1.823149	0.209530	4.652214
8	DT	0.997906	0.035586	0.794106	0.073142	1.528311	0.979938	0.087007	1.840447	0.217228	3.790687
22	FNN	0.996165	0.031768	0.745065	0.098973	2.532103	0.978773	0.077249	1.670651	0.223446	5.152048
9	DT	0.987734	0.081175	1.773881	0.177013	3.622204	0.973566	0.104973	2.218803	0.249349	4.747194
10	DT	0.987734	0.081175	1.773881	0.177013	3.622204	0.973566	0.104973	2.218803	0.249349	4.747194
24	FNN	0.990440	0.069954	1.679439	0.156273	3.964677	0.964843	0.105073	2.412319	0.287564	6.997278
21	FNN	0.993225	0.059250	1.449736	0.131555	2.636976	0.963982	0.109250	2.505515	0.291064	6.604596
17	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.859805	0.341124	7.470042	0.574243	11.961763
20	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.858092	0.343483	7.544345	0.577741	12.008309
16	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.857907	0.343529	7.519161	0.578117	12.042405
19	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.857297	0.340908	7.451508	0.579357	12.093224
5	Lasso	0.848930	0.457794	12.066209	0.621220	16.508182	0.849171	0.434871	11.761782	0.595623	15.950458
4	Lasso	0.848981	0.457609	12.064935	0.621114	16.510371	0.849144	0.434811	11.762585	0.595677	15.957545
3	Lasso	0.849005	0.457521	12.064359	0.621065	16.511503	0.849130	0.434788	11.763140	0.595705	15.961062
1	Lasso	0.849083	0.457217	12.062437	0.620905	16.515819	0.849073	0.434709	11.765168	0.595817	15.973744
2	Lasso	0.849109	0.457111	12.061795	0.620852	16.517497	0.849051	0.434689	11.766057	0.595861	15.978429
0	Linear	0.849794	0.455700	12.052001	0.619440	16.525914	0.848033	0.435595	11.809027	0.597866	16.119386
18	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.847448	0.380890	8.679914	0.599016	12.985283

[13]:

postprocessor.metrics(y="Max4Pin").drop("Parameter Configurations", axis=1)

[13]:

	Model Types	Train R2	Train MAE	Train MAPE	Train RMSE	Train RMSPE	Test R2	Test MAE	Test MAPE	Test RMSE	Test RMSPE
23	FNN	0.997350	0.036132	0.785741	0.088658	2.000391	0.985869	0.074636	1.563013	0.196140	4.091016
13	RF	0.991490	0.070690	1.440539	0.158885	3.340799	0.985561	0.091236	1.816571	0.198267	3.786319
11	RF	0.993156	0.061513	1.257432	0.142491	3.043308	0.985210	0.089573	1.792643	0.200664	3.896805
15	RF	0.988848	0.081675	1.660086	0.181881	3.802734	0.983549	0.098679	1.966861	0.211633	4.070294
14	RF	0.987118	0.089381	1.873946	0.195483	4.177504	0.982331	0.103412	2.122786	0.219327	4.322753
12	RF	0.996615	0.044723	0.910009	0.100209	2.080990	0.981386	0.102940	2.095839	0.225114	4.386684
6	DT	0.996019	0.049726	1.046821	0.108673	2.133910	0.981158	0.095063	1.956473	0.226489	3.893733
7	DT	0.995007	0.059814	1.254301	0.121707	2.383996	0.980897	0.094389	1.933704	0.228052	3.943680
25	FNN	0.996956	0.048649	1.021222	0.095023	1.722230	0.980440	0.090605	1.872811	0.230765	4.918177
8	DT	0.997929	0.036962	0.791367	0.078385	1.563494	0.979580	0.092866	1.889772	0.235781	3.907639
22	FNN	0.996350	0.034193	0.774589	0.104056	2.445643	0.977732	0.082222	1.745725	0.246223	5.526027
9	DT	0.987523	0.085062	1.780158	0.192384	3.774353	0.973475	0.109372	2.241698	0.268725	4.907399
10	DT	0.987523	0.085062	1.780158	0.192384	3.774353	0.973475	0.109372	2.241698	0.268725	4.907399
24	FNN	0.990354	0.073433	1.680703	0.169157	4.089031	0.960108	0.114190	2.525845	0.329555	7.774914
21	FNN	0.992917	0.062721	1.581082	0.144956	2.862251	0.959812	0.120268	2.754126	0.330774	7.280688
17	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.849547	0.371623	7.715947	0.640005	12.573974
20	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.847674	0.374279	7.806270	0.643977	12.619909
16	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.847420	0.374969	7.791527	0.644514	12.661445
19	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.847153	0.371910	7.717963	0.645078	12.690818
5	Lasso	0.842487	0.504971	12.825745	0.683561	17.383599	0.840374	0.481494	12.547761	0.659227	16.933634
4	Lasso	0.842541	0.504758	12.824177	0.683445	17.386680	0.840345	0.481393	12.547970	0.659288	16.941840
3	Lasso	0.842566	0.504659	12.823573	0.683391	17.388251	0.840329	0.481344	12.548071	0.659320	16.945906
1	Lasso	0.842647	0.504316	12.821511	0.683215	17.394128	0.840269	0.481212	12.549501	0.659445	16.960536
2	Lasso	0.842674	0.504194	12.820817	0.683156	17.396377	0.840245	0.481180	12.550409	0.659495	16.965928
0	Linear	0.843329	0.502262	12.822763	0.681734	17.450752	0.839692	0.480674	12.568713	0.660634	17.102810
18	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.838965	0.413559	8.973902	0.662130	13.547422

[14]:

postprocessor.metrics(y="F-delta-H").drop("Parameter Configurations", axis=1)

[14]:

	Model Types	Train R2	Train MAE	Train MAPE	Train RMSE	Train RMSPE	Test R2	Test MAE	Test MAPE	Test RMSE	Test RMSPE
12	RF	0.998482	0.004523	0.279244	0.007829	0.464364	0.990578	0.011899	0.735244	0.019983	1.199901
23	FNN	0.996727	0.006562	0.417757	0.011494	0.707052	0.989159	0.012994	0.812683	0.021435	1.306208
14	RF	0.992627	0.010610	0.660450	0.017252	1.039509	0.987413	0.014070	0.874196	0.023097	1.401953
25	FNN	0.995615	0.007943	0.491951	0.013304	0.800692	0.987326	0.013927	0.867201	0.023178	1.418400
11	RF	0.995669	0.007979	0.495274	0.013223	0.796832	0.987247	0.013545	0.836791	0.023250	1.413180
13	RF	0.994066	0.009523	0.592782	0.015477	0.933024	0.986768	0.014063	0.872984	0.023682	1.440716
15	RF	0.992410	0.010851	0.676160	0.017504	1.054572	0.986681	0.014666	0.909797	0.023760	1.440807
22	FNN	0.997174	0.005791	0.366185	0.010680	0.658142	0.986425	0.013579	0.849232	0.023987	1.470234
21	FNN	0.995293	0.009791	0.657091	0.013784	0.891407	0.985418	0.015506	1.012139	0.024861	1.534868
24	FNN	0.993806	0.009475	0.605964	0.015813	0.978422	0.982883	0.015317	0.959202	0.026935	1.619930
6	DT	0.996436	0.005876	0.366133	0.011995	0.710766	0.977895	0.014694	0.902003	0.030609	1.830049
8	DT	0.997739	0.004290	0.266598	0.009555	0.558579	0.977649	0.013863	0.856208	0.030779	1.852160
7	DT	0.994930	0.007537	0.466821	0.014306	0.844629	0.977543	0.015256	0.933814	0.030852	1.828449
9	DT	0.990461	0.011317	0.700394	0.019623	1.165194	0.977182	0.017000	1.044667	0.031099	1.872723
10	DT	0.990461	0.011317	0.700394	0.019623	1.165194	0.977182	0.017000	1.044667	0.031099	1.872723
17	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.822414	0.064461	4.001694	0.086758	5.201311
16	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.821844	0.064547	4.003877	0.086897	5.198466
20	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.820912	0.065513	4.065113	0.087124	5.201340
19	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.819930	0.064339	3.996437	0.087363	5.261993
18	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.803493	0.068264	4.260931	0.091263	5.491031
5	Lasso	0.498160	0.114155	7.138919	0.142331	8.551027	0.497752	0.116100	7.190258	0.145903	8.675038
4	Lasso	0.498282	0.114117	7.136380	0.142314	8.550270	0.497570	0.116103	7.190272	0.145929	8.676701
3	Lasso	0.498339	0.114100	7.135163	0.142306	8.549930	0.497479	0.116104	7.190323	0.145943	8.677537
1	Lasso	0.498525	0.114039	7.131015	0.142279	8.548884	0.497147	0.116114	7.190761	0.145991	8.680605
2	Lasso	0.498593	0.114017	7.129541	0.142270	8.548524	0.497024	0.116118	7.190951	0.146009	8.681712
0	Linear	0.500209	0.113364	7.082101	0.142040	8.535937	0.490688	0.116487	7.210541	0.146925	8.736598

[15]:

postprocessor.metrics(y="Max-Fxy").drop("Parameter Configurations", axis=1)

[15]:

	Model Types	Train R2	Train MAE	Train MAPE	Train RMSE	Train RMSPE	Test R2	Test MAE	Test MAPE	Test RMSE	Test RMSPE
8	DT	0.995129	0.002350	0.127048	0.003566	0.192220	0.984784	0.004376	0.236856	0.006454	0.347122
6	DT	0.993032	0.002740	0.148071	0.004265	0.228157	0.983128	0.004274	0.230556	0.006797	0.358349
7	DT	0.991140	0.003222	0.174433	0.004809	0.258195	0.981793	0.004500	0.243409	0.007060	0.373805
9	DT	0.977551	0.004266	0.230594	0.007654	0.406376	0.951994	0.005614	0.302294	0.011464	0.595937
10	DT	0.977551	0.004266	0.230594	0.007654	0.406376	0.951994	0.005614	0.302294	0.011464	0.595937
14	RF	0.966465	0.004901	0.263158	0.009355	0.486589	0.943567	0.005764	0.308609	0.012430	0.638504
12	RF	0.991682	0.002126	0.114296	0.004659	0.242042	0.941876	0.005460	0.292668	0.012615	0.649056
11	RF	0.978290	0.004184	0.225011	0.007527	0.391691	0.941754	0.005814	0.311582	0.012628	0.646624
13	RF	0.972246	0.004827	0.259801	0.008511	0.443496	0.938264	0.006109	0.327455	0.013001	0.666358
15	RF	0.962596	0.005564	0.299410	0.009880	0.514410	0.928830	0.006691	0.358840	0.013959	0.715766
22	FNN	0.912521	0.003690	0.209051	0.015110	0.887237	0.626553	0.009574	0.540140	0.031976	1.861733
21	FNN	0.695916	0.006726	0.395103	0.028171	1.744604	0.490332	0.010399	0.603870	0.037355	2.278705
23	FNN	0.792850	0.006350	0.360289	0.023252	1.373419	0.438234	0.012453	0.703560	0.039218	2.294565
24	FNN	0.474672	0.009105	0.536088	0.037028	2.294589	0.398160	0.011165	0.650115	0.040592	2.487946
25	FNN	0.460635	0.007524	0.455184	0.037519	2.351462	0.363795	0.010510	0.617507	0.041735	2.566917
18	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.341078	0.023635	1.317588	0.042474	2.512828
20	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.322885	0.023965	1.337774	0.043056	2.567247
17	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.321626	0.023894	1.333911	0.043096	2.564022
19	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.321059	0.023508	1.312540	0.043114	2.558481
16	KN	1.000000	0.000000	0.000000	0.000000	0.000000	0.317456	0.023942	1.336695	0.043228	2.574517
2	Lasso	0.296247	0.027661	1.531472	0.042857	2.529053	0.287355	0.028119	1.560361	0.044171	2.613463
1	Lasso	0.296197	0.027651	1.530955	0.042858	2.529233	0.287305	0.028108	1.559769	0.044173	2.613641
3	Lasso	0.296045	0.027625	1.529532	0.042863	2.529754	0.287152	0.028079	1.558143	0.044178	2.614156
4	Lasso	0.295998	0.027618	1.529128	0.042865	2.529908	0.287106	0.028070	1.557680	0.044179	2.614308
5	Lasso	0.295897	0.027603	1.528297	0.042868	2.530232	0.287006	0.028053	1.556750	0.044182	2.614628
0	Linear	0.300056	0.027768	1.537516	0.042741	2.520600	0.286307	0.028432	1.577655	0.044204	2.613712

[16]:

postprocessor.metrics().drop(
    ["Parameter Configurations", "Train MAE", "Test MAE", "Train RMSE", "Test RMSE"],
    axis=1,
)

[16]:

	Model Types	Train R2	Train MAPE	Train RMSPE	Test R2	Test MAPE	Test RMSPE
6	DT	0.995935	0.559805	1.122534	0.983725	1.066487	2.111147
7	DT	0.994702	0.673447	1.275759	0.983100	1.070745	2.149723
8	DT	0.997453	0.430649	0.854108	0.982988	1.037716	2.152643
11	RF	0.991592	0.693857	1.564570	0.978853	1.006598	2.087755
13	RF	0.989296	0.802446	1.731068	0.978276	1.028685	2.049980
14	RF	0.985818	0.999525	2.120153	0.977991	1.147687	2.258648
12	RF	0.996476	0.472963	1.044436	0.977756	1.108595	2.262138
15	RF	0.985804	0.923916	1.968558	0.975433	1.107722	2.182004
9	DT	0.987649	0.967283	1.954225	0.973307	1.249095	2.629562
10	DT	0.987649	0.967283	1.954225	0.973307	1.249095	2.629562
22	FNN	0.980233	0.444956	1.378210	0.912535	1.020479	2.977754
23	FNN	0.956473	0.572750	1.362587	0.879120	1.038871	2.465383
21	FNN	0.935122	0.866532	1.719567	0.878139	1.456709	3.741808
25	FNN	0.889624	0.663676	1.403580	0.861338	1.124513	2.877201
24	FNN	0.889315	0.963849	2.375805	0.859496	1.395022	3.971297
17	KN	1.000000	0.000000	0.000000	0.735358	4.568229	7.419949
20	KN	1.000000	0.000000	0.000000	0.733873	4.624319	7.454370
19	KN	1.000000	0.000000	0.000000	0.733814	4.555000	7.477373
16	KN	1.000000	0.000000	0.000000	0.733351	4.599845	7.459027
18	KN	1.000000	0.000000	0.000000	0.731010	5.124148	7.854740
5	Lasso	0.626059	7.732258	10.323653	0.617948	7.621043	10.127053
4	Lasso	0.626149	7.731394	10.324014	0.617899	7.621718	10.130263
3	Lasso	0.626191	7.731015	10.324224	0.617872	7.622101	10.131868
1	Lasso	0.626328	7.729750	10.325159	0.617769	7.623763	10.137695
2	Lasso	0.626376	7.729314	10.325556	0.617729	7.624488	10.139842
0	Linear	0.627846	7.721483	10.326855	0.615115	7.651547	10.206683

Based on the performance metrics presented above, random forest and decision tree are the top-performing models. FNN did well on all but Max-Fxy. K-nearest neighbors overfit to each output. Linear and lasso regression failed to capture any of the outputs, indicating that this data set is nonlinear.

Next, the best-performing model hyperparameters are shown.

[17]:

for model in ["Lasso", "DT", "RF", "KN", "FNN"]:
    postprocessor.print_model(model_type=model)
    print()

Model Type: Lasso
  alpha: 0.0007368296399123653

Model Type: DT
  max_depth: 27
  max_features: None
  min_samples_leaf: 3
  min_samples_split: 5

Model Type: RF
  criterion: poisson
  max_features: None
  min_samples_leaf: 3
  min_samples_split: 4
  n_estimators: 143

Model Type: KN
  leaf_size: 22
  n_neighbors: 10
  p: 2
  weights: distance

Model Type: FNN
  Structural Hyperparameters
    Layer: Dense_hidden_0
      units: 511
      sublayer: None
    Layer: Dense_hidden_1
      units: 367
      sublayer: None
    Layer: Dense_hidden_2
      units: 563
      sublayer: None
    Layer: Dense_hidden_3
      units: 441
      sublayer: None
    Layer: Dense_hidden_4
      units: 162
      sublayer: None
    Layer: Dense_output_0
  Compile/Fitting Hyperparameters
    Adam_learning_rate: 0.0009660778027367906
    batch_size: 8
Model: "FNN"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 Dense_hidden_0 (Dense)      (None, 511)               5110

 Dense_hidden_1 (Dense)      (None, 367)               187904

 Dense_hidden_2 (Dense)      (None, 563)               207184

 Dense_hidden_3 (Dense)      (None, 441)               248724

 Dense_hidden_4 (Dense)      (None, 162)               71604

 Dense_output_0 (Dense)      (None, 5)                 815

=================================================================
Total params: 721341 (2.75 MB)
Trainable params: 721341 (2.75 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

Below is the network plot for the FNN.

[18]:

postprocessor.nn_network_plot(
    to_file="./supporting/bwr_network.png",
    show_shapes=True,
    show_layer_names=True,
    expand_nested=True,
    show_layer_activations=True,
)

[18]:

To visualize the performance of these models we can use the pyMAISE.PostProcessor.diagonal_validation_plot functions to produce diagonal validation plots.

[19]:

def performance_plot(meth, output):
    models = np.array([["Linear", "Lasso"], ["DT", "KN"], ["RF", "FNN"]])
    fig, axarr = plt.subplots(models.shape[0], models.shape[1], figsize=(15,20))
    for i in range(models.shape[0]):
        for j in range(models.shape[1]):
            plt.sca(axarr[i, j])
            axarr[i, j] = meth(model_type=models[i, j], y=[output])
            axarr[i, j].set_title(models[i, j])

performance_plot(postprocessor.diagonal_validation_plot, "K-eff")
plt.show()

[20]:

performance_plot(postprocessor.diagonal_validation_plot, "Max3Pin")
plt.show()

[21]:

performance_plot(postprocessor.diagonal_validation_plot, "Max4Pin")
plt.show()

[22]:

performance_plot(postprocessor.diagonal_validation_plot, "F-delta-H")
plt.show()

[23]:

performance_plot(postprocessor.diagonal_validation_plot, "Max-Fxy")
plt.show()

The performance differences between random forest, decision tree, and FNN are minimal except on Max-Fxy. For Max-Fxy, all models showed considerable spread, with random forest and decision tree showing the tightest spread. In all the distributions, the clear winner is the random forest model. K-nearest neighbors are overfitting to the training data.

Similarly, the pyMAISE.PostProcessor.validation_plot function produces validation plots showing each prediction’s absolute relative error.

[24]:

# Output Features and models to plot
models = np.array([["Linear", "Lasso"], ["DT", "KN"], ["RF", "FNN"]])

# Plot info
fig, axarr = plt.subplots(models.shape[0], models.shape[1], figsize=(17,22))

# Iterate and plot
for i in range(models.shape[0]):
    for j in range(models.shape[1]):
        plt.sca(axarr[i, j])
        axarr[i, j] = postprocessor.validation_plot(model_type=models[i, j])
        axarr[i, j].set_title(models[i, j])
        axarr[i, j].get_legend().remove()

fig.legend(["K-eff", "Max3Pin", "Max4Pin", "F-delta-H", "Max-Fxy"], loc="upper center", ncol=5)
plt.show()

The performance of the models is best represented by the magnitudes observed on the y-axis; however, even RF and DT get as high as \(>35\%\) error on Max3Pin and Max4Pin.

Finally, the learning curve of the most performant FNN is shown by pyMAISE.PostProcessor.nn_learning_plot.

[25]:

postprocessor.nn_learning_plot()
plt.show()

The FNN is slightly overfitting with the validation curve below the training curve.

BWR Micro Core

pyMAISE Initialization

Model Initialization

Hyper-parameter Tuning

Model Post-processing