This short vignette illustrates basic usage of the OutlierTree library for outlier detection, using the hypothyroid dataset which is bundled with it.
This is a library which flags suspicious values within an observation, contrasting them against the normal values in a human-readable format and potentially adding conditions within the data that make the observation more suspicious; and does so in a similar way as one would do it manually, by checking extreme values in sorted order and filtering observations according to the values of other variables (e.g. if some other variable is TRUE or FALSE).
For a full description of the procedure see Explainable outlier detection through decision tree conditioning.
This is a dataset about hospital patients who might potentially have hypo- or hyperthyroidism problems. The observations are about anonymous people whose demographic characteristics, drug intake, and hormone indicators were recorded, along with the judgement about their condition.
It contains many interesting outliers which have something obviously wrong when examined visually, but which would nevertheless be missed by other outlier detection methods.
library(outliertree)
data(hypothyroid)
summary(hypothyroid)
age sex on.thyroxine query.on.thyroxine
Min. : 1.00 F :1817 Mode :logical Mode :logical
1st Qu.: 36.00 M : 849 FALSE:2442 FALSE:2733
Median : 54.00 NA's: 106 TRUE :330 TRUE :39
Mean : 51.75
3rd Qu.: 67.00
Max. :455.00
NA's :1
on.antithyroid.medication sick pregnant thyroid.surgery
Mode :logical Mode :logical Mode :logical Mode :logical
FALSE:2738 FALSE:2663 FALSE:1882 FALSE:2734
TRUE :34 TRUE :109 TRUE :41 TRUE :38
NA's :849
I131.treatment query.hypothyroid query.hyperthyroid lithium
Mode :logical Mode :logical Mode :logical Mode :logical
FALSE:2724 FALSE:2611 FALSE:2600 FALSE:2758
TRUE :48 TRUE :161 TRUE :172 TRUE :14
tumor goitre hypopituitary psych
Mode :logical Mode :logical Mode :logical Mode :logical
FALSE:2747 FALSE:2701 FALSE:2771 FALSE:2638
TRUE :25 TRUE :71 TRUE :1 TRUE :134
TSH T3 TT4 T4U
Min. : 0.005 Min. : 0.05 Min. : 2.0 Min. :0.3100
1st Qu.: 0.430 1st Qu.: 1.60 1st Qu.: 88.0 1st Qu.:0.8800
Median : 1.400 Median : 2.00 Median :104.0 Median :0.9800
Mean : 4.509 Mean : 2.03 Mean :109.1 Mean :0.9986
3rd Qu.: 2.600 3rd Qu.: 2.40 3rd Qu.:125.0 3rd Qu.:1.0800
Max. :478.000 Max. :10.60 Max. :430.0 Max. :2.1200
NA's :277 NA's :576 NA's :179 NA's :291
referral.source diagnosis FTI
other:1615 compensated: 154 Min. : 2.381
STMW : 91 negative :2553 1st Qu.: 92.857
SVHC : 275 primary : 63 Median :107.175
SVHD : 31 secondary : 2 Mean :110.791
SVI : 760 3rd Qu.:124.272
Max. :394.495
NA's :292 otree <- outlier.tree(hypothyroid, nthreads=1)
Reporting top 8 outliers [out of 8 found]
row [531] - suspicious column: [hypopituitary] - suspicious value: [TRUE]
distribution: 99.964% different [norm. obs: 2772]
row [623] - suspicious column: [age] - suspicious value: [455.00]
distribution: 99.964% <= 94.00 - [mean: 51.60] - [sd: 18.98] - [norm. obs: 2770]
row [2230] - suspicious column: [T3] - suspicious value: [10.60]
distribution: 99.951% <= 7.10 - [mean: 1.98] - [sd: 0.75] - [norm. obs: 2050]
given:
[query.hyperthyroid] = [FALSE]
row [1138] - suspicious column: [age] - suspicious value: [75.00]
distribution: 95.122% <= 42.00 - [mean: 31.46] - [sd: 5.28] - [norm. obs: 39]
given:
[pregnant] = [TRUE]
row [2211] - suspicious column: [age] - suspicious value: [73.00]
distribution: 95.122% <= 42.00 - [mean: 31.46] - [sd: 5.28] - [norm. obs: 39]
given:
[pregnant] = [TRUE]
row [1438] - suspicious column: [FTI] - suspicious value: [394.50]
distribution: 99.618% <= 232.08 - [mean: 132.68] - [sd: 28.23] - [norm. obs: 261]
given:
[TT4] > [123.00] (value: 430.00)
[referral.source] != [other] (value: STMW)
row [745] - suspicious column: [TT4] - suspicious value: [239.00]
distribution: 98.571% <= 177.00 - [mean: 135.23] - [sd: 12.57] - [norm. obs: 69]
given:
[FTI] between (97.96, 128.12] (value: 112.74)
[T4U] > [1.12] (value: 2.12)
[age] > [55.00] (value: 87.00)
row [1412] - suspicious column: [TT4] - suspicious value: [430.00]
distribution: 99.762% <= 230.00 - [mean: 111.88] - [sd: 31.88] - [norm. obs: 420]
given:
[T3] is NA(i.e. it’s saying that it’s abnormal to be pregnant at the age of 75, or to not be classified as hyperthyroidal when having very high thyroid hormone levels)
A look at the distributions within the clusters in which some outliers were flagged:
pregnant <- hypothyroid[hypothyroid$pregnant,]
hist(pregnant$age, breaks=50, col="navy",
main="Age distribution among pregnant patients",
xlab="Age")
non.hyperthyr <- hypothyroid[!hypothyroid$query.hyperthyroid,]
hist(non.hyperthyr$T3, breaks=50, col="darkred",
main="T3 hormone levels\n(Non-hyperthyroidal patients)",
xlab="T3 blood concentration")
The identified outliers, along with all the relevant information, are returned as a list of lists, which can be inspected manually and the exact conditions extracted from them (see documentation for more details).
They are nevertheless returned as a class of its own in order to provide pretty-printing and slicing:
outliers <- predict(otree, hypothyroid, outliers_print=FALSE)
outliers[1:700]
Reporting top 2 outliers [out of 2 found]
row [531] - suspicious column: [hypopituitary] - suspicious value: [TRUE]
distribution: 99.964% different [norm. obs: 2772]
row [623] - suspicious column: [age] - suspicious value: [455.00]
distribution: 99.964% <= 94.00 - [mean: 51.60] - [sd: 18.98] - [norm. obs: 2770]outliers[1138]
Reporting top 1 outliers [out of 1 found]
row [1138] - suspicious column: [age] - suspicious value: [75.00]
distribution: 95.122% <= 42.00 - [mean: 31.46] - [sd: 5.28] - [norm. obs: 39]
given:
[pregnant] = [TRUE]outliers[[1138]]
$suspicous_value
$suspicous_value$column
[1] "age"
$suspicous_value$value
[1] 75
$suspicous_value$decimals
[1] 0
$group_statistics
$group_statistics$upper_thr
[1] 42
$group_statistics$pct_below
[1] 0.9512195
$group_statistics$mean
[1] 31.46154
$group_statistics$sd
[1] 5.28078
$group_statistics$n_obs
[1] 39
$conditions
$conditions[[1]]
$conditions[[1]]$column
[1] "pregnant"
$conditions[[1]]$value_this
[1] TRUE
$conditions[[1]]$comparison
[1] "="
$conditions[[1]]$value_comp
[1] TRUE
$tree_depth
[1] 1
$uses_NA_branch
[1] FALSE
$outlier_score
[1] 0.01297346