Sampling Local Optimums

Sometimes we may want to sample multiple promising parameter sets at the same time. This is especially effective if the process is being run in parallel. The bayesOpt function always samples the global optimum of the acquisition function, however it is also possible to tell it to sample local optimums of the acquisition function at the same time.

Using the acqThresh parameter, you can specify the minimum percentage utility of the global optimum required for a different local optimum to be considered. As an example, let’s say we are optimizing 1 hyperparameter min_child_weight, which is bounded between [0,5]. Our acquisition function may look like the following:

In this case, there are 3 promising candidate parameters. We may want to run our scoring function on all 3. If acqThresh is set to be below ~0.95, and iters.k is set to at least 3, the process would use all 3 of the local maximums as candidate parameter sets in the next round of scoring function runs.

Adding Noise

If there are only 2 local optimums, and iters.k is 3, we still need to obtain another parameter set to run. We could choose a random parameter set within the bounds, however it is usually more worthwhile to decrease uncertainty around the promising points. Therefore, bayesOpt will randomly select points around our local optimums to sample if there aren’t enough local optimums to satisfy iters.k.