Fitting

To fit a model, we give it the data set and the columns to use. In this case, the 2nd column, Dropout_Rate, is the criterion to be predicted. The cues are the following columns, percent of Low_Income_Students and percent of Limited_English_Students. They are at indexes 3 and 4.

Let’s fit two models: * ttbModel, Take The Best, which uses the highest-validity cue that discriminates (more details below). * regModel, a version of R’s “lm” function for linear regression wrapped to fit into heurstica’s interface.

criterion_col <- 2
ttb <- ttbModel(schools, criterion_col, c(3:4))
reg <- regModel(schools, criterion_col, c(3:4))

What do the fits look like? We can examine Take The Best’s cue validities and the regression coefficients.

ttb$cue_validities

##      Low_Income_Students Limited_English_Students 
##                0.6000000                0.5555556

coef(reg)

##      Low_Income_Students Limited_English_Students 
##               0.24985315               0.07322294

Both Take The Best and regression give a higher weight to Low_Income_Students than Limited_English_Students, although of course how they use the weights differs. Take The Best will use a lexicographic order, making its prediction based solely on Low_Income_Students as long as the schools have differing values– which they do for all 5 schools in this data set. That means it will ignore Limited_English_Students when predicting on this data set. In contrast, regression will use a weighted sum of both cues, but with the most important cues weighted more.

Predicting the fitted data

To see a model’s predictions, we use the predictPair function. It takes two rows of data– which together comprise a “row pair”– and the fitted model. predictPair outputs three possible values:

• 1 predicts the first row passed to it.
• -1 predicts the second row passed to it.
• 0 is a guess.

In Bowen vs. Collins, it outputs 1, meaning it predicts Bowen has a higher dropout rate. In Bowen vs. Fenger, it outputs -1, meaning it predicts Fenger has a higher dropout rate.

predictPair(subset(schools, Name=="Bowen"), subset(schools, Name=="Collins"), ttb)

## [1] 1

predictPair(subset(schools, Name=="Bowen"), subset(schools, Name=="Fenger"), ttb)

## [1] -1

Note that the output depends on the order of the rows. In the reversed pair of Collins vs. Bowen, the output is -1. This is consistent because it still picks Bowen, regardless of order.

predictPair(subset(schools, Name=="Collins"), subset(schools, Name=="Bowen"), ttb)

## [1] -1

All rows

It is tedious to predict one row pair at a time, so let’s use heurstica’s predictPairSummary function instead. We simply pass it the data and the heuristics whose predictions we are interested in. It produces a matrix with all row pairs, which in this case is 10 (5 * 4 / 2).

out <- predictPairSummary(schools, ttb, reg)
# See the first row: It has row indexes.
out[1,]

##           Row1           Row2 CorrectGreater       ttbModel       regModel 
##              1              2              1              1             -1

# Convert indexes to school names for easier interpretation
out_df <- data.frame(out)
out_df$Row1 <- schools$Name[out_df$Row1]
out_df$Row2 <- schools$Name[out_df$Row2]
out_df

##       Row1    Row2 CorrectGreater ttbModel regModel
## 1    Bowen Collins              1        1       -1
## 2    Bowen  Fenger             -1       -1        1
## 3    Bowen  Juarez              1        1       -1
## 4    Bowen   Young              1       -1        1
## 5  Collins  Fenger             -1       -1        1
## 6  Collins  Juarez             -1       -1       -1
## 7  Collins   Young              1       -1        1
## 8   Fenger  Juarez              1        1       -1
## 9   Fenger   Young              1       -1        1
## 10  Juarez   Young              1       -1        1

The first row shows the Bowen vs. Collins example we considered above. Because CorrectGreater is 1, that means TTB predicted it correctly– Bowen really does have a higher drop-out rate. But regression predicted -1 for this row pair, which is incorrect.

predictPairSummary is for beginners. heuristica offers full flexibility in output with the rowPairApply function. After passing it the data, you can pass it any number of generators to make the columns you want. Some examples are below, where we print only the first row.

# Same as predictPairSummary.
out_same <- rowPairApply(schools, rowIndexes(), correctGreater(criterion_col), heuristics(ttb, reg))
out_same[1,]

##           Row1           Row2 CorrectGreater       ttbModel       regModel 
##              1              2              1              1             -1

# Show first the heuristic predictions, then CorrectGreater.  No row indexes.
out_simple <- rowPairApply(schools, heuristics(ttb, reg), correctGreater(criterion_col))
out_simple[1,]

##       ttbModel       regModel CorrectGreater 
##              1             -1              1

Assessing Overall Performance

For an overall measure of performance, we can measure the percent of correct inferences for all pairs of schools in the data with percentCorrect, namely the number of correct predictions divided by the total number of predictions. We give the function the data to be predicted (in this case the same as what was fit) and the fitted models to assess.

percentCorrect(schools, ttb, reg)

##   ttbModel regModel
## 1       60       50

Take The Best got 60% correct and regression got 50% correct, which is the same as chance.

Regression is the best linear unbiased model for the data. But this data had a very small sample size of just 5 schools, and good estimates require more data.

This is an unusual case where TTB actually beat regression in a fitting task. Usually ttb only wins in out-of-sample performance, e.g. fitting 5 schools and then predicting on other schools not used in the fit.

For a more realistic example, see the vignette with cross-validated out-of-sample performance on a complete data set.