prcbench

The aim of the prcbench package is to provide a testing workbench for evaluating precision-recall curves under various conditions. It contains integrated interfaces for the following five tools. It also contains predefined test data sets.

Tool	Link
ROCR	Tool web site, CRAN
AUCCalculator	Tool web site
PerfMeas	CRAN
PRROC	CRAN
precrec	Tool web site, CRAN

Documentation

Introduction to prcbench – a package vignette that contains the descriptions of the functions with several useful examples. View the vignette with vignette("introduction", package = "prcbench") in R. The HTML version is also available on the GitHub Pages.
Help pages – all the functions including the S3 generics have their own help pages with plenty of examples. View the main help page with help(package = "prcbench") in R. The HTML version is also available on the GitHub Pages.

Dependencies

Java

AUCCalculator requires a Java runtime (>= 6).

Installation

Install the release version of prcbench from CRAN with install.packages("prcbench").
Alternatively, you can install a development version of prcbench from our GitHub repository. To install it:
1. Make sure you have a working development environment.
  - Windows: Install Rtools (available on the CRAN website).
  - Mac: Install Xcode from the Mac App Store.
  - Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).
2. Install devtools from CRAN with install.packages("devtools").
3. Install prcbench from the GitHub repository with devtools::install_github("evalclass/prcbench").

Troubleshooting

microbenchmark

microbenchmark does not work on some OSs. prcbench uses system.time when microbenchmark is not available.

rJava

Some OSs require further configuration for rJava.

sudo R CMD javareconf

JDKs for macOS Big Sur.

JRI support on macOS Big Sur – see this Stack Overflow thread.

install.packages("rJava", configure.args="--disable-jri")

Examples

Following two examples show the basic usage of prcbench functions.

Benchmarking

The run_benchmark function outputs the result of microbenchmark for specified tools.

## Load library
library(prcbench)

## Run microbenchmark for auc5 (five tools) on b10 (balanced 5 Ps and 5 Ns)
testset <- create_testset("bench", "b10")
toolset <- create_toolset(set_names = "auc5")
res <- run_benchmark(testset, toolset)

## Use knitr::kable to show the result in a table format
knitr::kable(res$tab, digits = 2)

testset	toolset	toolname	min	lq	mean	median	uq	max	neval
b10	auc5	AUCCalculator	2.65	4.01	13.82	4.63	6.55	51.27	5
b10	auc5	PerfMeas	0.08	0.08	0.34	0.09	0.18	1.26	5
b10	auc5	precrec	6.08	6.10	28.07	6.43	6.51	115.22	5
b10	auc5	PRROC	0.28	0.33	3.53	0.44	0.48	16.14	5
b10	auc5	ROCR	2.29	2.59	16.48	3.72	18.69	55.12	5

Evaluation of precision-recall curves

The run_evalcurve function evaluates precision-recall curves with predefined test datasets. The autoplot shows a plot with the result of the run_evalcurve function.

## ggplot2 is necessary to use autoplot
library(ggplot2)

## Plot base points and the result of precrec on c1, c2, and c3 test sets
testset <- create_testset("curve", c("c1", "c2", "c3"))
toolset <- create_toolset("precrec")
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1)

## Plot the results of PerfMeas and PRROC on c1, c2, and c3 test sets
toolset <- create_toolset(c("PerfMeas", "PRROC"))
scores2 <- run_evalcurve(testset, toolset)
autoplot(scores2, base_plot = FALSE)

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2017; 33 (1): 145-147.

doi: 10.1093/bioinformatics/btw570

External links

Classifier evaluation with imbalanced datasets – our web site that contains several pages with useful tips for performance evaluation on binary classifiers.
The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets – our paper that summarized potential pitfalls of ROC plots with imbalanced datasets and advantages of using precision-recall plots instead.