An introduction to linelist

library(linelist)

Motivations

Outbreak analytics pipelines often start with case line lists, which are data tables in which every line is a different case/patient, and columns record different variables of potential epidemiological interest such as date of events (e.g. onset of symptom, case notification), disease outcome, or patient data (e.g. age, sex, occupation). Such data is typically held in a data.frame (or a tibble) and used in various downstream analysis. While this approach is functional, it often means that each analysis step will:

  1. need to check the required inputs are present in the data, and for the user to specify where (e.g. ‘This is the column where dates of onset are stored.’)

  2. need to validate the required data (e.g. ‘Check that the field storing dates of onset are indeed dates, and not a character.’)

The aim of linelist is to take care of these pre-requisites once and for all before downstream analyses, thus helping to make data pipelines more robust and straightforward.

linelist in a nutshell

Outline

linelist is an R package which implements basic data representation for case line lists, alongside accessors and basic methods. It essentially provides three types of functionalities:

  1. tagging: a tags system permits to pre-identify key epidemiological variables needed in downstream analyses (e.g. dates of case notification, symptom onset, age, gender, disease outcome)

  2. validation: functions checking that tagged variables are indeed present in the data.frame/tibble, and that they have the expected type (e.g. checking that dates are Date, integer or numeric)

  3. secured methods: generic functions which could lead to the loss of tagged variables have dedicated methods for linelist objects with adapted behaviours, either updating tags as needed (e.g. rename, names() <- ...) or issuing warnings/errors when tagged variables are lost (e.g. select, x[], x[[]])

Should I use linelist?

linelist is designed to add a robust, foundational layer to your data pipelines, but it might add unnecessary complexity to your analysis scripts. Here are a few hints to gauge if you should consider using the package.

You may have use for linelist if …:

Conversely, you probably do not need it if …:

Getting started

Installation

Our stable versions are released periodically on CRAN, and can be installed using:

If you prefer using the latest features and bug fixes, you can alternatively install the development version of linelist from GitHub using the following commands:

Once installed, you can load the package in your R session using:

Key functionalities

A linelist object is an instance of a data.frame or a tibble in which key epidemiological variables have been tagged. The main features of the packages are broken down into the 3 categories outlined above.

Tagging system

Tags are paired keys pointing a reference epidemiological variables to the name of a column in a data.frame or tibble. The tagging system permits to construct linelist objects, modify tags in existing objects, check and access existing tags and the corresponding variables.

  • make_linelist(): to create a linelist object by tagging key epi variables in a data.frame or a tibble

  • set_tags(): to add, remove, or modify tags in a linelist

  • tags(): to list variables which have been tagged in a linelist

  • tags_names(): to list all recognized tag names; details on what the tags represent can be found at ?make_linelist

  • select_tags(): to select columns of a linelist based on tags using dplyr compatible syntax

  • tags_df(): to obtain a data.frame of all the tagged variables in a linelist

Validation

Basic routines are provided to validate linelist objects. More advanced validation e.g. looking at compatibility of dated events will be implemented in a separate package.

  • validate_tags(): check that tagged variables are present in the dataset, that tags match the pre-defined list of tagged variables

  • validate_types(): check that tagged variables have an acceptable class, as defined in tags_types()

  • validate_linelist(): general validation of linelist objects, equivalent to running both validate_tags() and validate_types(), and checking the class of the object

Secured methods

These are dedicated S3 methods for existing generics which can be used to prevent the loss of tagged variables.

  • lost_tags_actions(): to set the behaviour to adopt when tagged variables would be lost by an operation: issue a warning (default), an error, or ignore

  • get_lost_tags_actions(): to check the current behaviour for lost tagged variables

  • rename: adapted from dplyr::rename, to rename columns of a linelist; will rename tags as needed to match the new column names

  • names(x) <-: same as rename, but using the ‘base R’ approach to renaming columns

  • select(): adapted from dplyr::select, for subsetting columns; an additional argument can be used to subset tagged variables as well; will behave according to get_lost_tags_actions() if tagged variables are lost

  • x[] and x[[]]: same as dplyr::select but using ‘base R’ syntax; will behave according to get_lost_tags_actions() if tagged variables are lost

Worked example

Example dataset

In this example, we use the case line list of the Hagelloch 1861 measles outbreak, distributed by the outbreaks pacakge as measles_hagelloch_1861 .

Creating a linelist object

Let us assume we want to tag the following variables to facilitate downstream analyses, after having checked their tag name in ?make_linelist:

We first load a few useful packages, and create a linelist with the above information:

The printing of the object confirms that the tags have been added. If we want to double-check which variables have been tagged:

Changing tags

Tags can be added / removed / changed using set_tags().

Let us assume we also want to record the outcome: it is currently missing, but can be built from dates of deaths (missing date = survived). This can be done by using mutate on x to create the new variable (remember x is not only a linelist, but also a regular tibble and this compatible with dplyr verbs), and setting up a new tag using set_tags:

If we wanted to undo the above, i.e. remove the outcome tag, we only need set it to NULL:

Accessing tagged variables

Now that key variables have been tagged in x, we can used these pre-defined fields in downstream analyses, without having to worry about variable names and types. We could access tagged variables using any of the following means:

Using safeguards

Because x remains a valid tibble, we can use any data handling operations implemented in dplyr. However, some of these operations may cause accidental removal of key tagged variables. linelist provides a safeguard mechanism against this. For instance, let’s assume we want to select only some columns of x:

Here, the above command gave a meaningful warning, in which select removes some of the variables that were tagged. This is because linelist implements a specific select method for linelist objects - see ?select.linelist. In fact, looking at the documentation, we can see that select() can be used to select variables both from the dataset and from the tags, using the tags = ... argument. For instance, to retain the first 2 variables, and the gender tag:

Again, we observe a warning as before due to the loss of tagged variables in the operation. This behaviour can be silenced if needed, or could be changed to issue an error (for stronger pipelines for instance):