Check the effect of varying one hyperparameter

To see the effect of varying one hyperparameter on the model performance we can use the function gridSearch. The function iterates through a set of predefined hyperparameter values, train the model and displays in real-time the evaluation metric in the RStudio viewer pane (hover over the points to get a tooltip with extra information). Let’s see how the AUC changes varying the regularization multiplier. First we have to define the values for the hyperparameter that we want to test. For that we create a named list that we will use as an argument for the function gridSearch:

# Define the values for bg
h <- list(reg = seq(0.2, 1, 0.1))
# Call the gridSearch function
exp_1 <- gridSearch(model, hypers = h, metric = "auc", test = val)

As you noticed we used the validation dataset as test argument. The output of the function is an object of class SDMtune. Let’s print it:

exp_1

When you print the output, the text contains the models configuration that have been used during the execution of the function. In our case, only the regularization multiplier reg has multiple values. You can plot the SDMtune object:

plot(exp_1, title = "Experiment 1")

and you can also recreate the interactive chart using:

plot(exp_1, title = "Experiment 1", interactive = TRUE)

The SDMtune object stores the results in the slot @results:

exp_1@results

You can order them with:

exp_1@results[order(-exp_1@results$test_AUC), ]

In the next example we check how the TSS changes varying the regularization multiplier from 1 to 4:

# Define the values for reg
h <- list(reg = 1:4)
# Call the gridSearch function
exp_2 <- gridSearch(model, hypers = h, metric = "tss", test = val)

and how AUC changes varying the feature combinations using the following values: l, lq, lh, lqp, lqph and lqpht:

# Define the values for fc
h <- list(fc = c("l", "lq", "lh", "lqp", "lqph", "lqpht"))
# Call the gridSearch function
exp_3 <- gridSearch(model, hypers = h, metric = "auc", test = val)

Train a Maxent model and see how the AUC changes varying the number of iterations from 300 to 1100 with increments of 200 (highlight to see the solution):

maxent_model <- train("Maxent", data = data)
# Define the values for fc
h <- list("iter" = seq(300, 1100, 200))
# Call the gridSearch function
exp_4 <- gridSearch(maxent_model, hypers = h, metric = "auc", test = val)

To see which hyperparameters can be tuned in a given model use the function getTunableArgs. For example:

getTunableArgs(model)

Random search

The function randomSearch trains models taking a random sample of the predefined configurations. In the next example we select 10 random configurations:

h <- list(reg = seq(0.2, 5, 0.2), fc = c("l", "lq", "lh", "lp", "lqp", "lqph"))
exp_6 <- randomSearch(model, hypers = h, metric = "auc", test = val, pop = 10,
                      seed = 65466)

The real-time chart plots two different graphs, one with the chosen metric for each trained model and one with the evaluation metric for the starting and the best found model. As you can see, the function is able to find a better combination of the model hyperparameters compared to the starting model; and this training only 10 instead of 150 models. The results includes the 10 trained model. If you are not happy with the solution, you can check the best hyperparameter combinations and this gives you an intuition of which ones are the hyperparameters to “refine” using the function gridSearch. The SDMtune object stores the results in a data.frame than can be accessed with the following command:

exp_6@results

Optimize Model

The previous function doesn’t learn anything from the trained models, it just selects n random combinations of hyperparameters. The function optimizeModel instead uses a genetic algorithm to find an optimum or near optimum solution. Check the function documentation to understand how it works, here we provide the code to execute it:

exp_7 <- optimizeModel(model, hypers = h, metric = "auc", test = val, pop = 15,
                       gen = 2, seed = 798)

Evaluate final model

Let’s say we want to use the best tuned model found by the randomSearch function. Before evaluating the model using the testing dataset, we can merge the training and the validation datasets together to increase the number of locations and train a new model with the merged observations and the tuned configuration. At this point we may have removed variables using the varSel or reduceVar function. If this is the case, we cannot merge directly the initial datasets which contain all the environmental variables. We can extract the train dataset with the selected variables from the output of the experiment and merge it with the validation dataset using the function mergeSWD:

index <- which.max(exp_6@results$test_AUC)  # Index of the best model in the experiment
new_train <- exp_6@models[[index]]@data  # New train dataset containing only the selected variables
merged_data <- mergeSWD(new_train, val, only_presence = TRUE) # Merge only presence data

The val dataset contains all the initial environmental variables but the mergeSWD function will merge only those that are present in both datasets (in case you have performed variable selection).

Then we get the model configuration from the experiment 6:

final_model <- train("Maxnet", data = merged_data, fc = exp_6@results[index, 1],
                     reg = exp_6@results[index, 2])

Now we can evaluate the final model using the held apart testing dataset:

auc(final_model, test = test)