MCBoost - Basics and Extensions

1.1 Preprocessing

Then we do basic preprocessing:

• Collapse rarest factors according to their prevalence
• Drop missing factor levels
• One-hot encode categorical variables
• Impute NA’s using a histogram approach

library(mlr3pipelines)
pipe = po("collapsefactors", no_collapse_above_prevalence = 0.0006) %>>%
  po("fixfactors") %>>%
  po("encode") %>>%
  po("imputehist")
prep_task = pipe$train(train_tsk)[[1]]

In order to simulate settings where a sensitive feature is not available, we remove the (dummy encoded) feature Race from the training task.

prep_task$set_col_roles(c("Race.Amer.Indian.Eskimo", "Race.Asian.Pac.Islander", "Race.Black", "Race.Other", "Race.White"), remove_from = "feature")

Now we fit a random forest.

library(mlr3learners)
l = lrn("classif.ranger", num.trees = 10L, predict_type = "prob")
l$train(prep_task)

1.2 MCBoost

A simple way to use the predictions from any model in mcboost is to wrap the predict function and provide it as an initial predictor. This can be done from any model / any library. Note, that we have to make sure, that our init_predictor returns a numeric vector of predictions.

init_predictor = function(data) {
  l$predict_newdata(data)$prob[, 2]
}

As mcboost requires the data to be provided in X, y format (a data.table or data.frame of features and a vector of labels), we create those two objects.

data = prep_task$data(cols = prep_task$feature_names)
labels = 1 - one_hot(prep_task$data(cols = prep_task$target_names)[[1]])

We use a ridge regularized linear regression model as the auditor.

mc = MCBoost$new(auditor_fitter = "RidgeAuditorFitter", init_predictor = init_predictor)
mc$multicalibrate(data, labels)

The print method additionally lists the average auditor values in the different buckets in each iteration:

mc

1.3 Evaluation on Test Data

adult_test = fread(
  "https://raw.githubusercontent.com/Yorko/mlcourse.ai/master/data/adult_test.csv",
  stringsAsFactors = TRUE
)
adult_test$Country = NULL
adult_test$fnlwgt = NULL

# The first row seems to have an error
adult_test = adult_test[Target != "",]
adult_test$Target = droplevels(adult_test$Target)

# Note, that we have to convert columns from numeric to integer here:
sdc = train_tsk$feature_types[type == "integer", id]
adult_test[, (sdc) := lapply(.SD, as.integer), .SDcols = sdc]

test_tsk = TaskClassif$new("adult_test", adult_test, target = "Target")
prep_test = pipe$predict(test_tsk)[[1]]

Now, we can again extract X, y.

test_data = prep_test$data(cols = prep_test$feature_names)
test_labels = 1 - one_hot(prep_test$data(cols = prep_test$target_names)[[1]])

and predict.

prs = mc$predict_probs(test_data)

The accuracy of the multi-calibrated model

mean(round(prs) == test_labels)

is similar to the non-calibrated model.

mean(round(init_predictor(test_data)) == test_labels)

But if we have a look at the bias for the different subpopulations of feature Race, we can see that the predictions got more calibrated. Note that we did not explicitly give neither the initial model nor the auditor access to the feature Race.

# Get bias per subgroup for multi-calibrated predictor
adult_test$biasmc = (prs - test_labels)
adult_test[, .(abs(mean(biasmc)), .N), by = .(Race)]
# Get bias per subgroup for initial predictor
adult_test$biasinit = (init_predictor(test_data) - test_labels)
adult_test[, .(abs(mean(biasinit)), .N), by = .(Race)]

1.4 The Auditor Effect

We can also obtain the auditor effect after multicalibration. This indicates “how much” each observation has been affected by multi-calibration (on average across iterations).

ae = mc$auditor_effect(test_data)
hist(ae)

We can see that there are a few instances with more pronounced effects, while most have actually only a low effect.

In order to get more insights, we compute quantiles of the less and more effected population (median as cut-point) and analyze differences.

effect = apply(test_data[ae >= median(ae[ae > 0]),], 2, quantile)
no_effect  = apply(test_data[ae < median(ae[ae>0]),], 2, quantile)
difference = apply((effect-no_effect), 2, mean)
difference[difference > 0.1]

There seems to be a difference in some variables like Education and Marital_Status.

We can further analyze the individuals:

test_data[ae >= median(ae[ae>0]), names(which(difference > 0.1)), with = FALSE]

Predicting using only the first ‘n’ iterations

Multi-calibration is an iterative procedure. The t parameter can be used to predict using only the first t iterations. This then predicts using only the first t iterations of the multi-calibration procedure.

prs = mc$predict_probs(test_data, t = 3L)

3.1 CVLearner

In this section we use a Cross-Validated learner that predicts on held-out data during the training phase. This idea is based on Wolpert (1992)’s Stacked Generalization. Other, simpler methods include choosing a smaller step size eta or reducing the number of iters.

tsk = tsk("sonar")

As an init_predictor we again use a ranger model from mlr3 and construct an init predictor using the convenience function provided by mcboost.

learner = lrn("classif.ranger", predict_type = "prob")
learner$train(tsk)
init_predictor = mlr3_init_predictor(learner)

… and we can calibrate this predictor. This time, we use a CVTreeAuditorFitter instead of a TreeAuditorFitter. This allows us to avoid overfitting similar to a technique coined stacked generalization first described by Wolpert in 1992. Note, that this can sometimes take a little longer since each learner is cross-validated using 3 folds (default).

d = data[, -1]
l = data$Class
mc = MCBoost$new(init_predictor = init_predictor, auditor_fitter=CVTreeAuditorFitter$new(), max_iter = 2L)
mc$multicalibrate(d[1:200,], l[1:200])
mc$predict_probs(d[201:208,])

3.2 Data Splitting

We can also use a fresh chunk of the validation data in each iteration. mcboost implements two strategies, "bootstrap" and "split". While "split" simply splits up the data, "bootstrap" draws a new bootstrap sample of the data in each iteration.

tsk = tsk("sonar")

Again, we use a ranger mlr3 model as our initial predictor:

learner = lrn("classif.ranger", predict_type = "prob")
learner$train(tsk)
init_predictor = mlr3_init_predictor(learner)

and we can now calibrate:

d = data[, -1]
l = data$Class
mc = MCBoost$new(
  init_predictor = init_predictor,
  auditor_fitter= TreeAuditorFitter$new(),
  iter_sampling = "bootstrap"
)
mc$multicalibrate(d[1:200,], l[1:200])
mc$predict_probs(d[201:208,])

4.1 LearnerAuditorFitter

The Subpop-fitter can be easily adjusted by constructing it from a LearnerAuditorFitter. This allows for using any mlr3 learner. See here for a list of available learners.

rf = LearnerAuditorFitter$new(lrn("regr.rpart", minsplit = 10L))
mc = MCBoost$new(auditor_fitter = rf)
mc$multicalibrate(data, labels)

The TreeAuditorFitter and RidgeAuditorFitter are two instantiations of this Fitter with pre-defined learners. By providing their character strings the fitter could be automatically constructed.

4.2 SubpopAuditorFitter & SubgroupAuditorFitter

In some occasions, instead of using a Learner, we might want to use a fixed set of subgroups. Those can either be defined from the data itself or provided from the outside.

Splitting via the dataset

In order to split the data into groups according to a set of columns, we use a SubpopAuditorFitter together with a list of subpops. Those define the group splits to multi-calibrate on. These splits can be either a character string, referencing a binary variable in the data or a function that, when evaluated on the data, returns a binary vector.

In order to showcase both options, we add a binary variable to our data:

data[, Bin := sample(c(1, 0), nrow(data), replace = TRUE)]

rf = SubpopAuditorFitter$new(list(
  "Bin",
  function(data) {data[["V1"]] > 0.2},
  function(data) {data[["V1"]] > 0.2 | data[["V3"]] < 0.29}
))

mc = MCBoost$new(auditor_fitter = rf)
mc$multicalibrate(data, labels)

And we can again apply it to predict on new data:

mc$predict_probs(data)

Manually defined masks

If we want to add the splitting from the outside, by supplying binary masks for the rows of the data, we can provide manually defined masks. Note, that the masks have to correspond with the number of rows in the dataset.

rf = SubgroupAuditorFitter$new(list(
  rep(c(0, 1), 104),
  rep(c(1, 1, 1, 0), 52)
))

mc = MCBoost$new(auditor_fitter = rf)
mc$multicalibrate(data, labels)

During prediction, we now have to supply a set of masks for the prediction data.

predict_masks = list(
  rep(c(0, 1), 52),
  rep(c(1, 1, 1, 0), 26)
)

mc$predict_probs(data[1:104,], subgroup_masks = predict_masks)

6.1 Preprocessing

Then we do basic preprocessing, since we do not have any categorical variables, we only impute NA’s using a histogram approach.

library(mlr3pipelines)
pipe =  po("imputehist")
prep_task = pipe$train(list(tsk))[[1]]

prep_task$set_col_roles(c("racepctblack", "racePctWhite", "racePctAsian", "racePctHisp", "community"), remove_from = "feature")

Now we fit our first Learner: A random forest.

library(mlr3learners)
l = lrn("regr.ranger", num.trees = 10L)
l$train(prep_task)

6.2 MCBoost

A simple way to use the predictions from any Model in mcboost is to wrap the predict function and provide it as an initial predictor. This can be done from any model / any library. Note, that we have to make sure, that our init_predictor returns a numeric vector of predictions.

init_predictor = function(data) {
  l$predict_newdata(data)$response
}

As mcboost requires the data to be provided in X, y format (a data.table or data.frame of features and a vector of labels), we create those two objects.

data = prep_task$data(cols = prep_task$feature_names)
labels = prep_task$data(cols = prep_task$target_names)[[1]]

mc = MCBoost$new(auditor_fitter = "RidgeAuditorFitter", init_predictor = init_predictor, eta = 0.1)
mc$multicalibrate(data, labels)

6.3 Evaluation on Test Data

We first create the test task by setting the holdout rows to use, and then use our preprocessing pipe's predict function to also impute missing values for the validation data. Then we again extract features X and target y.

test_task = tsk$clone()
test_task$row_roles$use = test_task$row_roles$holdout
test_task = pipe$predict(list(test_task))[[1]]
test_data = test_task$data(cols = tsk$feature_names)
test_labels = test_task$data(cols = tsk$target_names)[[1]]

and predict.

prs = mc$predict_probs(test_data)

Now we can compute the MSE of the multi-calibrated model

mean((prs - test_labels)^2)

and compare to the non-calibrated version:

mean((init_predictor(test_data) - test_labels)^2)

But looking at sub-populations we can see that the predictions got more calibrated. Since we cannot show all subpopulations we only show the MSE for the feature racepctblack.

test_data$se_mcboost = (prs - test_labels)^2
test_data$se_init = (init_predictor(test_data) - test_labels)^2

test_data[, .(mcboost = mean(se_mcboost), initial = mean(se_init), .N), by = .(racepctblack > 0.5)]