1. Finding a source for package information

First we need to identify a source of package metadata. There are a number of places one may want to look for this information, be it a source code directory, local package library or remote package repository. Once we find a source of package data, we begin to collect it in a package reference (pkg_ref) object.

Learn more: ?pkg_ref

2. Caching package metadata

If more information is needed to perform a given risk assessment, we will use what metadata we already have to continue to search for more fine-grained information about the package. For example, if we have a location of a locally installed package, we can use that path to search for that package’s DESCRIPTION file, and from there read in the DESCRIPTION contents. To avoid repeatedly processing the same metadata, these intermediate results are cached within the pkg_ref object so that they can be used in the derivation of mulitple risk metrics.

Learn more: ?pkg_ref_cache

3. Assess this metadata against a risk criterion

For each measure of risk, we first try to boil down that measure into some fundamental nugget of the package metadata that is comparable across packages and sources of information. The cross-comparable result of assessing a package in this way is what we refer to as a package metric (pkg_metric).

For example, with that DESCRIPTION file content, we might look at whether a maintainer is identified in the authors list. To ensure we can easily compare this information between packages that use the Authors field and the Authors@R field, we would boil this information down to just a single logical value indicating whether or not a maintainer was identified.

Learn more: ?pkg_assess

4. Score our metrics

After we have these atomic representations of metrics, we want to score them so that they can be meaningfully compared to one another. In practice this just embeds a means of converting from the datatype of the metric to a numeric value on a fixed scale from 0 (worst) to 1 (best).

Given our maintainer metric example, we might rate a package as 1 (great) if no maintainer is identified or 0 (poor) if a maintainer is found.

Learn more: ?pkg_score

5. Summarizing across metric scores

Finally, we may want to look at these scores of individual metrics in some sort of aggregate risk score. Naturally, not all metric scores may warrant the same weight. Having scores normalized to a fixed range allows us to define a summarizing algorithm to consistently assess and compare packages.

Notably, risk is an inverse scale from metric scores. High metric scores are favorable, whereas high risk scores are unfavorable.

Learn more: ?summarize_scores

Creating a package reference object

First, we create a package reference class object using the pkg_ref constructor function. This object will contain metadata as it’s collected in the various risk assessments.

library(riskmetric)
riskmetric_pkg_ref <- pkg_ref("riskmetric")
print(riskmetric_pkg_ref)

#> <pkg_install, pkg_ref> riskmetric v0.1.2
#> $path
#>   [1] "/home/user/username/R/3.6/Resources/library/riskmetric"
#> $source
#>   [1] "pkg_install"
#> $version
#>   [1] '0.1.2'
#> $name
#>   [1] "riskmetric"
#> $bug_reports...
#> $bug_reports_host...
#> $bug_reports_url...
#> $description...
#> $downloads...
#> $help...
#> $help_aliases...
#> $license...
#> $maintainer...
#> $name_first_letter...
#> $news...
#> $r_cmd_check...
#> $release_date...
#> $remote_checks...
#> $source_control_url...
#> $vignettes...
#> $website_urls...

Here we see that the riskmetric pkg_ref object is actually subclassed as a pkg_install. There is a hierarchy of pkg_ref object classes including pkg_source for source code directories, pkg_install for locally installed packages and pkg_remote for references to package information pulled from the internet including pkg_cran_remote and pkg_bioc_remote for CRAN and Bioconductor hosted packages respectively.

Throughout all of riskmetric, S3 classes are used extensively to make use of generic functions with divergent, reference mechanism dependent behaviors for caching metadata, assessing packages and scoring metrics.

Likewise, some fields have a trailing ... indicating that they haven’t yet been computed, but that the reference type has knowledge of how to go out and grab that information if the field is requested. Behind the scenes, this is done using the pkg_ref_cache function, which itself is an S3 generic, using the name of the field and pkg_ref class to dispatch to appropriate functions for retrieving metadata.

Assessing a package

There are a number of prespecified assessments, all prefixed by convention with assess_*. Every assessment function takes a single argument, a pkg_ref object and produces a pkg_metric object corresponding to the assess_* function that was applied.

riskmetric_export_help_metric <- assess_export_help(riskmetric_pkg_ref)
print(riskmetric_export_help_metric[1:5])

#>        all_assessments    assess_news_current assessment_error_throw 
#>                   TRUE                   TRUE                   TRUE 
#>   assess_has_vignettes             pkg_assess 
#>                   TRUE                   TRUE

Every function in the assess_* family of functions is expected to return basic measure of a package. In this case, we return a named logical vector indicating whether each export function has an associated help document.

The return type also leaves a trail of what assessment produced this metric. In addition to the pkg_metric class, we now have a pkg_metric_export_help subclass which is used for dispatching to an appropriate scoring method.

It’s worth pointing out that the act of calling this function has had the side-effect of mutating our riskmetric_pkg_ref object.

riskmetric_pkg_ref

#> <pkg_install, pkg_ref> riskmetric v0.1.2
#> $help_aliases
#>                            %||%                        .tools
#>              "if_not_null_else"                   "dot-tools"
#>                  allow_mutation               all_assessments
#>                "allow_mutation"             "all_assessments"
#>     assessment_error_as_warning        assessment_error_empty
#>   "assessment_error_as_warning"      "assessment_error_empty"
#>   <continued>
#> $path
#>   [1] "/home/user/username/R/3.6/Resources/library/riskmetric"
#> $source
#>   [1] "pkg_install"
#> $version
#>   [1] '0.1.2'
#> $name
#>   [1] "riskmetric"
#> $bug_reports...
#> $bug_reports_host...
#> $bug_reports_url...
#> $description...
#> $downloads...
#> $help...
#> $license...
#> $maintainer...
#> $name_first_letter...
#> $news...
#> $r_cmd_check...
#> $release_date...
#> $remote_checks...
#> $source_control_url...
#> $vignettes...
#> $website_urls...

Here riskmetric_pkg_ref$help_aliases has a known value because it was needed to asses whether the package has documentation for its exports.

a note on caching

This happens because pkg_ref objects are really just environments with some syntactic sugar, and environments in R are always modified by-reference. This globally mutable behavior is used so that operations performed by one assessment can be reused by others. Likewise, computing one field may require that a previous field has been computed first, triggering a chain of metadata retrieval. In this case, $help_aliases required that $path be available.

This chaining behavior comes for free by implementing the pkg_ref_cache caching function for each field. For contributors, this alleviates the need to remember an order of operations, and for users this behavior means that subsets of assessments can be run in an arbitrary order without pulling superfluous metadata, keeping track of every-growing objects or ensuring certain assessments get called before others.

In addition to the metric-specific assess_* family of functions, a more comprehensive pkg_assess function is provided. Notably, pkg_assess accepts a pkg_ref object and list of assessments to apply, defaulting to all_assessments(), which returns a list of all assess_* functions in the riskmetric namespace.

pkg_assess(riskmetric_pkg_ref)

#> <list_of<pkg_metric>[15]>
#> $news_current
#> [1] TRUE
#> attr(,"class")
#> [1] "pkg_metric_news_current" "pkg_metric"             
#> [3] "logical"                
#> attr(,"label")
#> [1] "NEWS file contains entry for current version number"
#> 
#> $has_vignettes
#> [1] 0
#> 
#> $has_bug_reports_url
#> [1] "https://github.com/pharmaR/riskmetric/issues"
#> 
#> $bugs_status
#> <simpleError in FUN(X[[i]], ...): subscript out of bounds>
#> 
#> $license
#> [1] "MIT + file LICENSE"
#> 
#> $export_help
#>             all_assessments         assess_news_current 
#>                        TRUE                        TRUE 
#>      assessment_error_throw        assess_has_vignettes 
#>                        TRUE                        TRUE 
#>                  pkg_assess  assess_has_bug_reports_url 
#>                        TRUE                        TRUE 
#>            score_error_zero  assess_last_30_bugs_status 
#>                        TRUE                        TRUE 
#>                metric_score              assess_license 
#>                        TRUE                        TRUE 
#>      assessment_error_empty                  as_pkg_ref 
#>                        TRUE                        TRUE 
#>                   pkg_score          assess_export_help 
#>                        TRUE                        TRUE 
#>        assess_downloads_1yr              score_error_NA 
#>                        TRUE                        TRUE 
#> assessment_error_as_warning          assess_has_website 
#>                        TRUE                        TRUE 
#>            summarize_scores          assess_r_cmd_check 
#>                        TRUE                        TRUE 
#>               as_pkg_metric        assess_remote_checks 
#>                        TRUE                        TRUE 
#>         score_error_default       assess_has_maintainer 
#>                        TRUE                        TRUE 
#>   assess_exported_namespace             assess_has_news 
#>                        TRUE                        TRUE 
#>                  pkg_metric   assess_has_source_control 
#>                        TRUE                        TRUE 
#>                     pkg_ref        assess_covr_coverage 
#>                        TRUE                        TRUE 
#> 
#> $downloads_1yr
#> [1] 1823
#> 
#> $has_website
#> [1] "https://pharmar.github.io/riskmetric/"
#> [2] "https://github.com/pharmaR/riskmetric"
#> 
#> $r_cmd_check
#> [1] NA
#> attr(,"class")
#> [1] "pkg_metric_na"          "pkg_metric_condition"   "pkg_metric_r_cmd_check"
#> [4] "pkg_metric"             "logical"               
#> attr(,"label")
#> [1] "Package check results"
#> 
#> $remote_checks
#> [1] NA
#> attr(,"class")
#> [1] "pkg_metric_na"            "pkg_metric_condition"    
#> [3] "pkg_metric_remote_checks" "pkg_metric"              
#> [5] "logical"                 
#> attr(,"label")
#> [1] "Number of OS flavors that passed/warned/errored on R CMD check"
#> 
#> $has_maintainer
#> [1] "Eli Miller <eli.miller@atorusresearch.com>"
#> 
#> $exported_namespace
#>  [1] "all_assessments"             "assess_news_current"        
#>  [3] "assessment_error_throw"      "assess_has_vignettes"       
#>  [5] "pkg_assess"                  "assess_has_bug_reports_url" 
#>  [7] "score_error_zero"            "assess_last_30_bugs_status" 
#>  [9] "metric_score"                "assess_license"             
#> [11] "assessment_error_empty"      "as_pkg_ref"                 
#> [13] "pkg_score"                   "assess_export_help"         
#> [15] "assess_downloads_1yr"        "score_error_NA"             
#> [17] "assessment_error_as_warning" "assess_has_website"         
#> [19] "summarize_scores"            "assess_r_cmd_check"         
#> [21] "as_pkg_metric"               "assess_remote_checks"       
#> [23] "score_error_default"         "assess_has_maintainer"      
#> [25] "assess_exported_namespace"   "assess_has_news"            
#> [27] "pkg_metric"                  "assess_has_source_control"  
#> [29] "pkg_ref"                     "assess_covr_coverage"       
#> 
#> $has_news
#> [1] 1
#> attr(,"class")
#> [1] "pkg_metric_has_news" "pkg_metric"          "integer"            
#> attr(,"label")
#> [1] "number of discovered NEWS files"
#> 
#> $has_source_control
#> [1] "https://github.com/pharmaR/riskmetric"
#> 
#> $covr_coverage
#> [1] NA
#> attr(,"class")
#> [1] "pkg_metric_na"            "pkg_metric_condition"    
#> [3] "pkg_metric_covr_coverage" "pkg_metric"              
#> [5] "logical"                 
#> attr(,"label")
#> [1] "Package unit test coverage"

Since that is a lot to take in, pkg_assess also operates on tibbles, returning a cleaner output that might be easier to sort through when assessing a package.

pkg_assess(as_tibble(riskmetric_pkg_ref))

#> # A tibble: 1 x 18
#>   package    version pkg_ref             news_current       has_vignettes     
#>   <chr>      <chr>   <list<pkg_ref>>     <list<pkg_metric>> <list<pkg_metric>>
#> 1 riskmetric 0.1.2   riskmetric<install> TRUE               0                 
#> # … with 13 more variables: has_bug_reports_url <list<pkg_metric>>,
#> #   bugs_status <list<pkg_metric>>, license <list<pkg_metric>>,
#> #   export_help <list<pkg_metric>>, downloads_1yr <list<pkg_metric>>,
#> #   has_website <list<pkg_metric>>, r_cmd_check <list<pkg_metric>>,
#> #   remote_checks <list<pkg_metric>>, has_maintainer <list<pkg_metric>>,
#> #   exported_namespace <list<pkg_metric>>, has_news <list<pkg_metric>>,
#> #   has_source_control <list<pkg_metric>>, covr_coverage <list<pkg_metric>>

Scoring package metrics

After a metric has been collected, we “score” the metric to convert it to a quantified representation of risk.

There is a single scoring function, metric_score, that dispatches based on the class of the metric that is passed to it to interpret the atomic metric result.

metric_score(riskmetric_export_help_metric)
#> [1] 1

For convenience, pkg_score is provided as a convenience to operate on pkg_ref objects directly. It can also operate on the tibble produced by pkg_assess applied to a pkg_ref tibble, providing a new tibble with scored metrics.

pkg_score(pkg_assess(as_tibble(pkg_ref("riskmetric"))))
#> # A tibble: 1 x 19
#>   package    version pkg_ref             pkg_score news_current has_vignettes
#>   <chr>      <chr>   <list<pkg_ref>>         <dbl> <pkg_scor>   <pkg_scor>   
#> 1 riskmetric 0.1.2   riskmetric<install>     -1.47 1            0            
#> # … with 13 more variables: has_bug_reports_url <pkg_scor>,
#> #   bugs_status <pkg_scor>, license <pkg_scor>, export_help <pkg_scor>,
#> #   downloads_1yr <pkg_scor>, has_website <pkg_scor>, r_cmd_check <pkg_scor>,
#> #   remote_checks <pkg_scor>, has_maintainer <pkg_scor>,
#> #   exported_namespace <pkg_scor>, has_news <pkg_scor>,
#> #   has_source_control <pkg_scor>, covr_coverage <pkg_scor>

Note that pkg_assess and pkg_score accepts an error_handler argument which determines how errors are escalated for communication. We’ve chosen to default to being cautious, displaying warnings liberally to ensure thorough documentation of the risk assessment process. If these warnings are bothersome, there are alternative reporting schemes in the assessment_error_* and score_error_* families of functions.

Creating a tibble from pkg_refs

We start by calling our pkg_ref constructor function with a list or vector. Doing so will return a list of pkg_ref objects. With this list, we can use tibble::as_tibble to convert the pkg_ref list into a tibble, automatically populating some useful index columns like package and version. To clean things up further we can use the magrittr pipe (%>%) to chain these commands together.

package_tbl <- pkg_ref(c("riskmetric", "utils", "tools")) %>%
  as_tibble()

The riskmetric workflow on multiple packages

pkg_assess and pkg_score can operate on tibbles, making it easy to simultaneously test an entire cohort of packages at once.

package_tbl %>%
  pkg_assess() %>%
  pkg_score()
#> # A tibble: 3 x 19
#>   package    version pkg_ref             pkg_score news_current has_vignettes
#>   <chr>      <chr>   <list<pkg_ref>>         <dbl> <pkg_scor>   <pkg_scor>   
#> 1 riskmetric 0.1.2   riskmetric<install>     -1.47 1            0            
#> 2 utils      3.6.2   utils<install>         -13.8  0            1            
#> 3 tools      3.6.2   tools<install>          -7.07 0            0            
#> # … with 13 more variables: has_bug_reports_url <pkg_scor>,
#> #   bugs_status <pkg_scor>, license <pkg_scor>, export_help <pkg_scor>,
#> #   downloads_1yr <pkg_scor>, has_website <pkg_scor>, r_cmd_check <pkg_scor>,
#> #   remote_checks <pkg_scor>, has_maintainer <pkg_scor>,
#> #   exported_namespace <pkg_scor>, has_news <pkg_scor>,
#> #   has_source_control <pkg_scor>, covr_coverage <pkg_scor>

Notice that a summary column, pkg_score, is included in addition to our metric scores. This value is a shorthand for aggregating a weighted average of risk scores across tibble columns using summarize_scores.

package_tbl %>%
  pkg_assess() %>%
  pkg_score() %>%
  summarize_scores()
#> [1]  -1.467467 -13.799696  -7.066667