Crunch is designed for survey data, and it includes native support
for two data structures common in survey data: categorical arrays (or
“grids”) and multiple response. In R, these would typically be
represented either as multiple, separate columns in a
data.frame or as some other data structure that needs
processing before you can analyze it. Crunch makes it so you don’t have
to think about the underlying data structures in order to do exploratory
analysis.
Categorical Arrays
Surveys commonly ask sets of related questions. In the Stack Overflow
survey, there are several questions about the importance of various
factors for hiring. Conceptually these questions go together, but
because of the limitations of storing data in a table they are presented
as separate variables.
ds$ImportantHiringCompanies
## ImportantHiringCompanies (categorical)
## Congratulations! You've just been put in charge of technical recruiting at Globex, a multinational high- tech firm. This job comes with a corner office, and you have an experienced staff of recruiters at your disposal. They want to know what they should prioritize when recruiting software developers. How important should each of the following be in Globex's hiring process? Previous companies worked at
##
## Count
## Somewhat important 511
## Not very important 316
## Important 230
## Not at all important 63
## Very important 43
ds$ImportantHiringAlgorithms
## ImportantHiringAlgorithms (categorical)
## Congratulations! You've just been put in charge of technical recruiting at Globex, a multinational high- tech firm. This job comes with a corner office, and you have an experienced staff of recruiters at your disposal. They want to know what they should prioritize when recruiting software developers. How important should each of the following be in Globex's hiring process? Knowledge of algorithms and data structures
##
## Count
## Important 502
## Very important 320
## Somewhat important 281
## Not very important 50
## Not at all important 14
Crunch allows you to bundle these related questions together into a
Categorical Array. This is immensely useful when you want to
analyze or communicate information about the set of questions. You can
use the whole array in a cross-tab without having to process dozens of
individual variables.
To create a Categorical Array variable from R, we can use the
makeArray function. We’ll pass to that function a selection
of variables to bind together into an array, along with any additional
metadata we want to provide. Fortunately here, the variables we want to
select have a common naming convention, so we can grep for
them:
imphire <- grep("^ImportantHiring", names(ds), value = TRUE)
imphire
## [1] "ImportantHiringAlgorithms" "ImportantHiringTechExp" "ImportantHiringCommunication"
## [4] "ImportantHiringOpenSource" "ImportantHiringPMExp" "ImportantHiringCompanies"
## [7] "ImportantHiringTitles" "ImportantHiringEducation" "ImportantHiringRep"
## [10] "ImportantHiringGettingThingsDone"
Now we can use that selection in makeArray:
ds$ImportantHiring <- makeArray(ds[imphire], name = "Importance in Hiring Process")
The variables we included in the array have been converted into
subvariables of the categorical array, and are still accessible using
the subvariables() function.
subvariables(ds$ImportantHiring)
## Subvariables:
## $ImportantHiringAlgorithms | ImportantHiringAlgorithms
## $ImportantHiringTechExp | ImportantHiringTechExp
## $ImportantHiringCommunication | ImportantHiringCommunication
## $ImportantHiringOpenSource | ImportantHiringOpenSource
## $ImportantHiringPMExp | ImportantHiringPMExp
## $ImportantHiringCompanies | ImportantHiringCompanies
## $ImportantHiringTitles | ImportantHiringTitles
## $ImportantHiringEducation | ImportantHiringEducation
## $ImportantHiringRep | ImportantHiringRep
## $ImportantHiringGettingThingsDone | ImportantHiringGettingThingsDone
Those names aren’t that pretty. As we saw above, though, we have
better names for them appended to the question wording we read in from
the schema. Each subvariable description has the full
question wording, which really should be the description of
the parent array variable, and then after the question mark, the
response label.
Let’s use some regular expressions to clean that up. We can pull the
part before the question mark from one of the subvariables and add that
to the parent, and then prune the question wording from all and use that
as the “names”:
description(ds$ImportantHiring) <- sub("^(.*\\?).*$", "\\1",
descriptions(subvariables(ds$ImportantHiring))[1])
names(subvariables(ds$ImportantHiring)) <- sub("^.*\\? (.*)$", "\\1",
descriptions(subvariables(ds$ImportantHiring)))
subvariables(ds$ImportantHiring)
## Subvariables:
## $ImportantHiringAlgorithms | Knowledge of algorithms and data structures
## $ImportantHiringTechExp | Experience with specific tools (libraries, frameworks, etc.) used by the employer
## $ImportantHiringCommunication | Communication skills
## $ImportantHiringOpenSource | Contributions to open source projects
## $ImportantHiringPMExp | Experience with specific project management tools & techniques
## $ImportantHiringCompanies | Previous companies worked at
## $ImportantHiringTitles | Previous job titles held
## $ImportantHiringEducation | Educational credentials (e.g. schools attended, specific field of study, grade...
## $ImportantHiringRep | Stack Overflow reputation
## $ImportantHiringGettingThingsDone | Track record of getting things done
webApp(ds$ImportantHiring)
Multiple Response
Crunch also has a special variable type for multiple-response data.
These are survey questions that give users a set of options and allow
them to select more than one of them. An example of this from the Stack
Overflow survey is asking which language a respondent wanted to work
with. In this dataset, the responses are stored in a single column with
each language delimited by semicolon:
knitr::kable(SO_survey[1:5, "HaveWorkedLanguage", drop = FALSE], row.names = FALSE)
| Matlab; Python; R; SQL |
| R; SQL |
| C#; JavaScript; Matlab; Python; R |
| Java; JavaScript; Lua; PHP; R; SQL |
| C++; Matlab; Python; R; VBA |
To analyze this data without Crunch you would usually go through an
elaborate data cleaning process to split the delimited variable up into
its constituent parts, and then reshape it into a format which is more
amenable to analysis. With Crunch, you can make use of multiple response
variables to store it in a compact form, and analyze it just like any
other variable. Since delimited text is such a common way of storing
this kind of answer, Crunch allows you to create a variable directly
from the text. This function creates a derived variable, which
means that if you make changes to the data in the initial variable, the
derived variable will change along with it.
ds$WantWorkLanguageMR <- makeMRFromText(ds$WantWorkLanguage,
delim = "; ",
name = "Languages Desired for Work",
description = description(ds$WantWorkLanguage))
To see what this looks like, let’s do a simple table()
of the new variable and see the counts:
table(ds$WantWorkLanguageMR)
## WantWorkLanguageMR
## VBA TypeScript Java Scala JavaScript Perl Lua Matlab
## 44 170 418 257 591 65 69 120
## Erlang Assembly VB.NET Python Common Lisp Dart CoffeeScript Elixir
## 67 89 24 915 44 22 51 70
## Ruby Groovy C F# Hack C++ Objective-C R
## 143 43 255 88 18 420 65 815
## Clojure Rust Swift Julia Haskell Visual Basic 6 SQL Go
## 82 105 148 155 150 23 594 289
## PHP Smalltalk
## 149 24
Since this is multiple-response data, the sum of these numbers may be
bigger than the number of respondents because people were allowed to
select more than one.
We can reorder the subvariables to make this card easier to read.
Let’s sort them by frequency.
counts <- sort(table(ds$WantWorkLanguageMR), decreasing = TRUE)
subvariables(ds$WantWorkLanguageMR) <- subvariables(ds$WantWorkLanguageMR)[names(counts)]
