TOCPOs are more complicated than FOCPOs, since they potentially need to operate on data at three different points: During initial training, during the re-transformation for new prediction data, and during the inversion of predictions made by a model trained on transformed data. Similarly to makeCPO(), makeCPOTargetOp() splits these operations up into functions that create “control” objects, and functions that do the actual transformation. makeCPOExtendedTargetOp(), on the other hand, gives the user more flexibility at the price of the user having to make sure that transformation and retransformation perform the same operation–similarly to makeCPOExtendedTrafo() for FOCPOs.
makeCPOTargetOp()
makeCPOTargetOp() has a cpo.train and cpo.retrafo function parameter that work similarly to the ones of makeCPO(). In contrast to makeCPO(), however, cpo.retrafo must return the target data instead of the feature data. The data and target parameters of cpo.retrafo get the same data as they get in a FOCPO created with makeCPO(), with the exception that if dataformat is "task" or "df.all", the target parameter will receive the whole input data in form of a Task or data.frame (while the data argument, as in a FOCPO, will receive only the feature data.frame). The return value of cpo.retrafo for a TOCPO must always be in the same format as the input target value: a data.frame with the manipulated target values when dataformat is anything besides "task" or "df.all", or a Task or data.frame of all data (with non-target columns unmodified) otherwise.
Inversion of predictions is performed using the functions cpo.train.invert and cpo.invert. cpo.train.invert takes a data and a control argument, and any arguments declared in the par.set. It is called whenever new data is fed into the CPO or its retrafo CPOTrained, and creates a CPOTrained state that is used to invert the prediction done on this new data. The control argument takes the value returned by the cpo.train function upon initial training, and the data argument is the new data for which to prepare the CPOTrained inverter. It has the form dictated by dataformat, with the exception that "task" and "df.all" dataformat are handled as "df.feature"; this is necessary since the new data could be a data.frame of data with unknown target.
The following is an example of a TOCPO that trains a classification Learner on a binary classification Task and changes it to a Task of whether or not the Learner predicted the truth for a given data line correctly. (Real-world applications would probably need to take some precautions against overfitting.) In its cpo.train step, the given Learner is trained on the incoming data and the resulting WrappedModel object is returned as the “control” object. This is given to the cpo.retrafo function, which performs prediction and creates a new classification Task with the match / mismatch between model prediction and ground truth as target. When an external Learner is trained on data that was preprocessed like this, its prediction will be whether the CPO-internal Learner can be trusted to predict a given data row. To “invert” this, i.e. to get the actual prediction, the cpo.invert function needs to have the internal Learner’s prediction as well as the prediction made by the external Learner. The former is provided by cpo.train.invert, which uses the WrappedModel to make a prediction on the new data, and given as control.invert to cpo.invert. The latter is the target data given to cpo.invert. This example CPO supports inverting both "response" and "prob" predict.type predictions, as declared in the predict.type.map argument. The actual predict.type to invert is given to cpo.invert as an argument.
xmpMetaLearn = makeCPOTargetOp("xmp.meta", # nolint
pSS(lrn: untyped),
dataformat = "task",
properties.target = c("classif", "twoclass"),
predict.type.map = c(response = "response", prob = "prob"),
cpo.train = function(data, target, lrn) {
cat("*** cpo.train ***\n")
lrn = setPredictType(lrn, "prob")
train(lrn, data)
},
cpo.retrafo = function(data, target, control, lrn) {
cat("*** cpo.retrafo ***\n")
prediction = predict(control, target)
tname = getTaskTargetNames(target)
tdata = getTaskData(target)
tdata[[tname]] = factor(prediction$data$response == prediction$data$truth)
makeClassifTask(getTaskId(target), tdata, tname, positive = "TRUE",
fixup.data = "no", check.data = FALSE)
},
cpo.train.invert = function(data, control, lrn) {
cat("*** cpo.train.invert ***\n")
predict(control, newdata = data)$data
},
cpo.invert = function(target, control.invert, predict.type, lrn) {
cat("*** cpo.invert ***\n")
if (predict.type == "prob") {
outmat = as.matrix(control.invert[grep("^prob\\.", names(control.invert))])
revmat = outmat[, c(2, 1)]
outmat * target[, "prob.TRUE", drop = TRUE] +
revmat * target[, "prob.FALSE", drop = TRUE]
} else {
stopifnot(levels(target) == c("FALSE", "TRUE"))
numeric.prediction = as.numeric(control.invert$response)
numeric.res = ifelse(target == "TRUE",
numeric.prediction,
3 - numeric.prediction)
factor(levels(control.invert$response)[numeric.res],
levels(control.invert$response))
}
})
cpo = xmpMetaLearn(makeLearner("classif.logreg"))
To show the inner workings of this CPO, the following example data is used.
set.seed(12)
split = makeResampleInstance(hout, pid.task)
train.task = subsetTask(pid.task, split$train.inds[[1]])
test.task = subsetTask(pid.task, split$predict.inds[[1]])
It can be instructive to watch the cat() output of this CPO to see which function gets called at what point in the lifecycle. The cpo.train function is called first to create the control object. The Task is transformed in cpo.retrafo. Also cpo.train.invert is called, since an inverter attribute is attached to the returned trafo.
trafd = train.task %>>% cpo
#> *** cpo.train ***
#> *** cpo.train.invert ***
#> *** cpo.retrafo ***
attributes(trafd)
#> $names
#> [1] "type" "env" "weights" "blocking" "coordinates"
#> [6] "task.desc"
#>
#> $class
#> [1] "ClassifTask" "SupervisedTask" "Task"
#>
#> $retrafo
#> CPO Retrafo chain {type:classif}
#> [RETRAFO xmp.meta(lrn = <classif.log...)]
#>
#> $inverter
#> CPO Inverter chain {type:classif} (able to predict 'response', 'prob')
#> [INVERTER xmp.meta(lrn = <classif.log...){type:classif}]
The values of the target column (“diabetes”) of the result can be compared with the prediction of a "classif.logreg" Learner on the same data:
head(getTaskData(trafd))
#> pregnant glucose pressure triceps insulin mass pedigree age diabetes
#> 450 0 120 74 18 63 30.5 0.285 26 TRUE
#> 346 8 126 88 36 108 38.5 0.349 49 FALSE
#> 336 0 165 76 43 255 47.9 0.259 26 FALSE
#> 247 10 122 68 0 0 31.2 0.258 41 TRUE
#> 174 1 79 60 42 48 43.5 0.678 23 TRUE
#> 453 0 91 68 32 210 39.9 0.381 25 TRUE
model = train(makeLearner("classif.logreg", predict.type = "prob"), train.task)
head(predict(model, train.task)$data[c("truth", "response")])
#> truth response
#> 450 neg neg
#> 346 neg pos
#> 336 neg pos
#> 247 neg neg
#> 174 neg neg
#> 453 neg neg
When new data is transformed using the retrafo CPOTrained, another inverter attribute is created, and hence cpo.train.invert is called again. Since the target column of the test.task in the following example is also transformed, the cpo.retrafo function is called.
retr = test.task %>>% retrafo(trafd)
#> *** cpo.retrafo ***
#> *** cpo.train.invert ***
attributes(retr)
#> $names
#> [1] "type" "env" "weights" "blocking" "coordinates"
#> [6] "task.desc"
#>
#> $class
#> [1] "ClassifTask" "SupervisedTask" "Task"
#>
#> $inverter
#> CPO Inverter chain {type:classif} (able to predict 'response', 'prob')
#> [INVERTER xmp.meta(lrn = <classif.log...){type:classif}]
In a real world application, it would be possible for the new incoming data to have unknown target values. In that case, no target column would need to be changed, and cpo.retrafo is not called. The resulting data, retr.df, equals the input data with a retrafo attribute added.
retr.df = getTaskData(test.task, target.extra = TRUE)$data %>>% retrafo(trafd)
#> *** cpo.train.invert ***
names(attributes(retr.df))
#> [1] "names" "class" "row.names" "inverter"
The invert functionality can be demonstrated by making a prediction with an external model.
ext.model = train("classif.svm", trafd)
ext.pred = predict(ext.model, retr)
newpred = invert(inverter(retr), ext.pred)
#> *** cpo.invert ***
performance(newpred)
#> mmce
#> 0.2200521
It may also be instructive to attach the xmpMetaLearn CPO to a Learner to see which functions get called during training and prediction of a TOCPO-Learner. Since the Learner does not do inversion of the training data, a CPOTrained for inversion is not created during training, and cpo.train.invert is hence not called. Only cpo.train (for control object creation) and cpo.retrafo (target value change) are called. During prediction, the input data is used to create an (internally used) inversion CPOTrained which promptly gets used by the prediction made by "classif.svm". Hence both cpo.train.invert and cpo.invert are called in succession.
cpo.learner = cpo %>>% makeLearner("classif.svm")
cpo.model = train(cpo.learner, train.task)
#> *** cpo.train ***
#> *** cpo.retrafo ***
lrnpred = predict(cpo.model, test.task)
#> *** cpo.train.invert ***
#> *** cpo.invert ***
performance(lrnpred)
#> mmce
#> 0.2200521
See Postscriptum for an evaluation of xmpMeatLearn’s performance.
Functional TOCPO
Just like for FOCPOs, it is possible to create functional TOCPOs. In the case of makeCPOTargetOp(), it is possible to have cpo.train create cpo.retrafo and cpo.train.invert, instead of giving them to makeCPOTargetOp() directly. Just as in makeCPO, these functions can then access the state of their environment in the cpo.train call and hence have neither a control argument, nor any arguments for the par.set parameters. Since cpo.train must in this case create two functions, these functions only need to be defined within cpo.train, the return value is ignored.
Note that cpo.retrafo and cpo.train.invert must either be both functional or both object based.
It is furthermore possible to return a cpo.invert function by cpo.train.invert, instead of giving it to makeCPOTargetOp(). As above, the returned function should not have any parameters for the ones given in par.set, and should not have a control.invert. cpo.invert can be functional or not, independently of whether cpo.retrafo and cpo.train.invert are functional.
As in makeCPO(), all functions that are given functionally must be explicitly set to NULL in the makeCPOTargetOp() call.
The xmpMetaLearn example above with functional cpo.retrafo, cpo.train.invert and cpo.invert would look like the following:
xmpMetaLearn = makeCPOTargetOp("xmp.meta.fnc", # nolint
pSS(lrn: untyped),
dataformat = "task",
properties.target = c("classif", "twoclass"),
predict.type.map = c(response = "response", prob = "prob"),
# set the cpo.* parameters not needed to NULL:
cpo.retrafo = NULL, cpo.train.invert = NULL, cpo.invert = NULL,
cpo.train = function(data, target, lrn) {
cat("*** cpo.train ***\n")
lrn = setPredictType(lrn, "prob")
model = train(lrn, data)
cpo.retrafo = function(data, target) {
cat("*** cpo.retrafo ***\n")
prediction = predict(model, target)
tname = getTaskTargetNames(target)
tdata = getTaskData(target)
tdata[[tname]] = factor(prediction$data$response == prediction$data$truth)
makeClassifTask(getTaskId(target), tdata, tname, positive = "TRUE",
fixup.data = "no", check.data = FALSE)
}
cpo.train.invert = function(data) {
cat("*** cpo.train.invert ***\n")
prediction = predict(model, newdata = data)$data
function(target, predict.type) { # this is returned as cpo.invert
cat("*** cpo.invert ***\n")
if (predict.type == "prob") {
outmat = as.matrix(prediction[grep("^prob\\.", names(prediction))])
revmat = outmat[, c(2, 1)]
outmat * target[, "prob.TRUE", drop = TRUE] +
revmat * target[, "prob.FALSE", drop = TRUE]
} else {
stopifnot(levels(target) == c("FALSE", "TRUE"))
numeric.prediction = as.numeric(prediction$response)
numeric.res = ifelse(target == "TRUE",
numeric.prediction,
3 - numeric.prediction)
factor(levels(prediction$response)[numeric.res],
levels(prediction$response))
}
}
}
})
Constant Invert TOCPOs
The example given above is a relatively elaborate TOCPO which needs information from the prediction data to perform inversion. Many simpler applications of target transformation do not need this information if their inversion step is independent of this data. It is possible to declare such a TOCPO using the constant.invert flag in makeCPOTargetOp(). If constant.invert is set to TRUE, the cpo.train.invert argument must be explicitly set to NULL. cpo.train still needs to have a control.invert argument; it is set to the value returned by cpo.train.
The following example is a TOCPO for regression Tasks that centers target values during training. After prediction, the data is inverted by adding the original mean of the training data to the predictions. This inversion operation does not need any information about the prediction data going in, so the TOCPO can be declared constant.invert.
The cpo.retrafo function is also called when new prediction data with a target column is transformed (as during model validation). In that case, the mean of the training data column is subtracted. Therefore the mean generated by cpo.train needs to be used in cpo.retrafo (i.e. the control value), not the mean of the target data present.
xmpRegCenter = makeCPOTargetOp("xmp.center", # nolint
constant.invert = TRUE,
cpo.train.invert = NULL, # necessary for constant.invert = TRUE
dataformat = "df.feature",
properties.target = "regr",
cpo.train = function(data, target) {
# control value is just the mean of the target column
mean(target[[1]])
},
cpo.retrafo = function(data, target, control) {
# subtract mean from target column in retrafo
target[[1]] = target[[1]] - control
target
},
cpo.invert = function(target, predict.type, control.invert) {
target + control.invert
})
cpo = xmpRegCenter()
To illustrate this CPO, the following data is used:
train.task = subsetTask(bh.task, 150:155)
getTaskTargets(train.task)
#> [1] 15.4 21.5 19.6 15.3 19.4 17.0
predict.task = subsetTask(bh.task, 156:160)
getTaskTargets(predict.task)
#> [1] 15.6 13.1 41.3 24.3 23.3
The target column of the task after transformation has a mean of 0.
trafd = train.task %>>% cpo
getTaskTargets(trafd)
#> [1] -2.633333 3.466667 1.566667 -2.733333 1.366667 -1.033333
When applying the retrafo CPOTrained to a new task, the mean of the training task target column is subtracted.
getTaskTargets(predict.task)
#> [1] 15.6 13.1 41.3 24.3 23.3
retr = retrafo(trafd)
predict.traf = predict.task %>>% retr
getTaskTargets(predict.traf)
#> [1] -2.433333 -4.933333 23.266667 6.266667 5.266667
When inverting a regression prediction, the mean of the training data target column is added to the prediction.
model = train("regr.lm", trafd)
pred = predict(model, predict.traf)
#> Warning in predict.lm(.model$learner.model, newdata = .newdata, se.fit =
#> FALSE, : prediction from a rank-deficient fit may be misleading
pred
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 -2.433333 -2.854142
#> 157 2 -4.933333 5.802064
#> 158 3 23.266667 14.442103
#> 159 4 6.266667 8.302782
#> 160 5 5.266667 8.633308
invert(inverter(predict.traf), pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 15.6 15.17919
#> 157 2 13.1 23.83540
#> 158 3 41.3 32.47544
#> 159 4 24.3 26.33611
#> 160 5 23.3 26.66664
Since "regr.lm" is translation invariant and deterministic, the prediction equals the prediction made without centering the target:
model = train("regr.lm", train.task)
predict(model, predict.task)
#> Warning in predict.lm(.model$learner.model, newdata = .newdata, se.fit =
#> FALSE, : prediction from a rank-deficient fit may be misleading
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 15.6 15.17919
#> 157 2 13.1 23.83540
#> 158 3 41.3 32.47544
#> 159 4 24.3 26.33611
#> 160 5 23.3 26.66664
A special property of constant.invert TOCPOs is that their retrafo CPOTrained can also be used for inversion. This is the case since the tight coupling of inversion operation to the data used to create the prediction is not necessary when the inversion is actually independent of this data. This is indicated by getCPOTrainedCapability() returning a vector with the "invert" capability set to 1. However, when using the retrafo CPOTrained for inversion, the “truth” column is absent from the inverted prediction.
getCPOTrainedCapability(retr)
#> retrafo invert
#> 1 1
invert(retr, pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id response
#> 156 1 15.17919
#> 157 2 23.83540
#> 158 3 32.47544
#> 159 4 26.33611
#> 160 5 26.66664
Functional Constant Invert TOCPO
Just as above, constant.invert TOCPOs can be functional. For this, the cpo.train function must declare both a cpo.retrafo and a cpo.invert variable which perform the requested operations. These functions have no control or control.invert parameter, and no parameters pertaining to par.set.
Stateless TOCPO
Very simple target column operations that operate on a row-by-row basis without needing information e.g. from training data, can be declared as “stateless”. Similarly to makeCPO(), when cpo.train parameter is set to NULL, no control object is created for a CPOTrained. Furthermore, a stateless TOCPO must always have constant.invert set as well. Therefore, only cpo.retrafo and cpo.invert are given as functions, both without a control or control.invert argument. One example is a TOCPO that log-transforms the target column of a regression task, and exponentiates the predictions made from this during inversion. (A better inversion would take the "se" prediction into account, see cpoLogTrafoRegr.)
xmpLogRegr = makeCPOTargetOp("log.regr", # nolint
constant.invert = TRUE,
properties.target = "regr",
cpo.train = NULL, cpo.train.invert = NULL,
cpo.retrafo = function(data, target) {
target[[1]] = log(target[[1]])
target
},
cpo.invert = function(target, predict.type) {
exp(target)
})
cpo = xmpLogRegr()
The CPO takes the logarithm of the task target column both during training and when using the retrafo CPOTrained.
trafd = train.task %>>% cpo
getTaskTargets(trafd)
#> [1] 2.734368 3.068053 2.975530 2.727853 2.965273 2.833213
retr = retrafo(trafd)
predict.traf = predict.task %>>% retr
getTaskTargets(predict.traf)
#> [1] 2.747271 2.572612 3.720862 3.190476 3.148453
model = train("regr.lm", trafd)
pred = predict(model, predict.traf)
#> Warning in predict.lm(.model$learner.model, newdata = .newdata, se.fit =
#> FALSE, : prediction from a rank-deficient fit may be misleading
pred
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 2.747271 2.726065
#> 157 2 2.572612 3.224557
#> 158 3 3.720862 3.685047
#> 159 4 3.190476 3.353014
#> 160 5 3.148453 3.365374
Note that both the inverter and the retrafo CPOTrained can be used for inversion, since a stateless TOCPO also has constant.invert set. As above, when using the retrafo CPOTrained, the truth column is absent from the result.
invert(inverter(predict.traf), pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id truth response
#> 156 1 15.6 15.27267
#> 157 2 13.1 25.14244
#> 158 3 41.3 39.84699
#> 159 4 24.3 28.58876
#> 160 5 23.3 28.94433
invert(retr, pred)
#> Prediction: 5 observations
#> predict.type: response
#> threshold:
#> time: 0.00
#> id response
#> 156 1 15.27267
#> 157 2 25.14244
#> 158 3 39.84699
#> 159 4 28.58876
#> 160 5 28.94433
makeCPOExtendedTargetOp()
Just as for FOCPOs, it is possible to declare a TOCPO while having more direct control over what happens at which stage of training, re-transformation, or inversion. In a TOCPO defined with makeCPOTargetOp(), the cpo.retrafo and cpo.train.invert functions are called automatically when necessary during training and re-transformation. makeCPOExtendedTargetOp() instead has a cpo.trafo and a cpo.retrafo parameter, which get called during the respective operation.
cpo.trafo must be a function taking the same parameters as cpo.train in makeCPOTargetOp(). Instead of returning a control object, it must define a variable named “control”, and a variable named “control.invert”. The former is used as the control argument of cpo.retrafo, the latter is used as control.invert for cpo.invert when using the inverter CPOTrained created during training. The return value of cpo.trafo must be similar to the value returned by cpo.retrafo in makeCPOTargetOp(): it must be the modified data set or target, depending on dataformat.
cpo.retrafo must take the same parameters as in makeCPOTargetOp(). It must declare a control.invert variable that will be given to cpo.retrafo when using the inverter CPOTrained created during retransformation. Since cpo.retrafo is always called during retrafo CPOTrained application, a “target” column may or may not be present. If a target column is not present, the target parameter of cpo.retrafo is NULL and the return value of cpo.retrafo is ignored; otherwise it must be the transformed target value (which, as in makeCPOTargetOp(), can be a Task or data.frame of all data if dataformat is "task" or "df.all").
cpo.invert works just as in makeCPOTargetOp().
The following is a nonsensical, synthetic example that adds 1 to the target column of a regression Task during initial training, subtracts 1 during retrafo re-application and is a no-op during inversion.
xmpSynCPO = makeCPOExtendedTargetOp("syn.cpo", # nolint
properties.target = "regr",
cpo.trafo = function(data, target) {
cat("*** cpo.trafo ***\n")
target[[1]] = target[[1]] + 1
control = "control created in cpo.trafo"
control.invert = "control.invert created in cpo.trafo"
target
},
cpo.retrafo = function(data, target, control) {
cat("*** cpo.retrafo ***", "control is:", deparse(control), sep = "\n")
control.invert = "control.invert created in cpo.retrafo"
if (!is.null(target)) {
cat("target is non-NULL, performing transformation\n")
target[[1]] = target[[1]] - 1
return(target)
} else {
cat("target is NULL, no transformation (but control.invert was created)\n")
return(NULL) # is ignored.
}
},
cpo.invert = function(target, control.invert, predict.type) {
cat("*** invert ***", "control.invert is:", deparse(control.invert),
sep = "\n")
target
})
cpo = xmpSynCPO()
For an “extended” TOCPO, only one of the transformation functions is called in each invocation. Initial transformation calls cpo.trafo and adds 1 to the targets; using the CPOTrained for re-transformation calls cpo.retrafo and subtracts 1.
trafd = train.task %>>% cpo
#> *** cpo.trafo ***
getTaskTargets(trafd)
#> [1] 16.4 22.5 20.6 16.3 20.4 18.0
retrafd = train.task %>>% retrafo(trafd)
#> *** cpo.retrafo ***
#> control is:
#> "control created in cpo.trafo"
#> target is non-NULL, performing transformation
getTaskTargets(retrafd)
#> [1] 14.4 20.5 18.6 14.3 18.4 16.0
It is also possible to perform re-transformation with a data.frame that does not include the target column. In that case the target value given to cpo.retrafo will be NULL, as reported by that function in this example:
retrafd = getTaskData(train.task, target.extra = TRUE)$data %>>% retrafo(trafd)
#> *** cpo.retrafo ***
#> control is:
#> "control created in cpo.trafo"
#> target is NULL, no transformation (but control.invert was created)
The trafd object has an inverter CPOTrained attribute that was created by cpo.trafo, the retrafd object has an inverter CPOTrained attribute created by cpo.retrafo (necessarily). This is made visible by the given example inverter function:
inv = invert(inverter(trafd), 1:6)
#> *** invert ***
#> control.invert is:
#> "control.invert created in cpo.trafo"
inv = invert(inverter(retrafd), 1:6)
#> *** invert ***
#> control.invert is:
#> "control.invert created in cpo.retrafo"
