1. Diagnostics of predicted probability scores
1.1 plotROC
The plotROC uses the ggplot2 framework to create the ROC curve and prints the AUROC inside. It comes with an option to display the change points of the prediction probability scores on the graph if you set the Show.labels = T.
> data('ActualsAndScores')
> plotROC(actuals=ActualsAndScores$Actuals, predictedScores=ActualsAndScores$PredictedScores)
You can also get the sensitivity matrix used to make the plot by turning on returnSensitivityMat = TRUE.
> sensMat <- plotROC(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores,
+ returnSensitivityMat = TRUE)1.2. sensitivity or recall
Sensitivity, also considered as the ‘True Positive Rate’ or ‘recall’ is the proportion of ‘Events’ (or ‘Ones’) correctly predicted by the model, for a given prediction probability cutoff score. The default cutoff score for the specificity function unless specified by the threshold argument is taken as 0.5.
> sensitivity(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores)[1] 1
If the objective of your problem is to maximize the ability of your model to detect the ‘Events’ (or ‘Ones’), even at the cost of wrongly predicting the non-events (‘Zeros’) as an event (‘One’), then you could set the threshold as determined by the optimalCutoff() with optimiseFor='Ones'.
NOTE: This may not be the best example, because we are able to achieve the maximum sensitivity of 1 with the default cutoff of 0.5. However, I am showing this example just to understand how this could be implemented in real projects.
> max_sens_cutoff <- optimalCutoff(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores,
+ optimiseFor = "Ones") # determine cutoff to maximize sensitivity.
>
> print(max_sens_cutoff) # This would be cut-off score that achieved maximum sensitivity.[1] 0.5531893
> sensitivity(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores,
+ threshold = max_sens_cutoff)[1] 1
1.3. specificity
For a given probability score cutoff (threshold), specificity computes what proportion of the total non-events (zeros) were predicted accurately. It can alo be computed as 1 - False Positive Rate. If unless specified, the default threshold value is set as 0.5, which means, the values of ActualsAndScores$PredictedScores above 0.5 is considered as events (Ones).
> specificity(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores)[1] 0.1411765
If you wish to know what proportion of non-events could be detected by lowering the threshold:
> specificity(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores,
+ threshold = 0.35)[1] 0.01176471
1.4. precision
For a given probability score cutoff (threshold), precision or ‘positive predictive value’ computes the proportion of the total events (ones) out of the total that were predicted to be events (ones).
> precision(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores)[1] 0.5379747
1.5. npv
For a given probability score cutoff (threshold), npv or ‘negative predictive value’ computes the proportion of the total non-events (zeros) out of the total that were predicted to be non-events (zeros).
> npv(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores)[1] 1
1.6. youdensIndex
Youden’s J Index (Youden 1950), calculated as \[J = Sensitivity + Specificity − 1\] represents the proportions of correctly predicted observations for both the events (Ones) and nonevents (Zeros). It is particularly useful if you want a single measure that accounts for both false-positive and false-negative rates
> youdensIndex(actuals = ActualsAndScores$Actuals, predictedScores = ActualsAndScores$PredictedScores)[1] 0.1411765
