Installing the Arrow Package on Linux
TL;DR
In most cases, install.packages("arrow") should just
work. There are things you can do to make the installation faster (see
below). If, for any reason, it doesn’t work, set the environment
variable ARROW_R_DEV=true, retry, and share the logs with
us.
The Apache Arrow project is implemented in multiple languages, and
the R package depends on the Arrow C++ library (referred to from here on
as libarrow). This means that when you install arrow, you need both the
R and C++ versions. If you install arrow from CRAN on a machine running
Windows or MacOS, when you call install.packages("arrow"),
a precompiled binary containing both the R package and libarrow will be
downloaded. However, CRAN does not host R package binaries for Linux,
and so you must choose from one of the alternative approaches.
This vignette outlines the recommend approaches to installing arrow
on Linux, starting from the simplest and least customisable to the most
complex but with more flexbility to customise your installation.
The intended audience for this document is arrow R package
users on Linux, and not Arrow developers. If you’re
contributing to the Arrow project, see
vignette("developing", package = "arrow") for resources to
help you on set up your development environment. You can also find a
more detailed discussion of the code run during the installation process
in the developers’
installation docs
Having trouble installing arrow? See the “Troubleshooting” section
below.
Installing a release version (the easy way)
Method 1 - Installation with a precompiled libarrow binary
As mentioned above, on macOS and Windows, when you run
install.packages("arrow"), and install arrow from CRAN, you
get an R binary package that contains a precompiled version of libarrow,
though CRAN does not host binary packages for Linux. This means that the
default behaviour when you run install.packages() on Linux
is to retrieve the source version of the R package that has to be
compiled locally, including building libarrow from source. See method 2
below for details of this.
For a faster installation, we recommend that you instead use one of
the methods below for installing arrow with a precompiled libarrow
binary.
Method 1a - Binary R package containing libarrow binary via
RSPM/conda
If you want a quicker installation process, and by default a more
fully-featured build, you could install arrow from RStudio’s public
package manager, which hosts binaries for both Windows and
Linux.
For example, if you are using Ubuntu 20.04 (Focal):
options(
HTTPUserAgent =
sprintf(
"R/%s R (%s)",
getRversion(),
paste(getRversion(), R.version["platform"], R.version["arch"], R.version["os"])
)
)
install.packages("arrow", repos = "https://packagemanager.rstudio.com/all/__linux__/focal/latest")
Note that the User Agent header must be specified as in the example
above. Please check the
RStudio Package Manager: Admin Guide for more details.
For other Linux distributions, to get the relevant URL, you can visit
the
RSPM site, click on ‘binary’, and select your preferred
distribution.
Similarly, if you use conda to manage your R
environment, you can get the latest official release of the R package
including libarrow via:
conda install -c conda-forge --strict-channel-priority r-arrow
Method 1b - R source package with libarrow binary
Another way of achieving faster installation with all key features
enabled is to use static libarrow binaries we host. These are used
automatically on many Linux distributions (x86_64 architecture only),
according to the allowlist.
If your distribution isn’t in the list, you can opt-in by setting the
NOT_CRAN environment variable before you call
install.packages():
Sys.setenv("NOT_CRAN" = TRUE)
install.packages("arrow")
This installs the source version of the R package, but during the
installation process will check for compatible libarrow binaries that we
host and use those if available. If no binary is available or can’t be
found, then this option falls back onto method 2 below (full source
build), but setting the environment variable results in a more
fully-featured build than default.
Except for the those built for gcc 4.8 (default on CentOS 7), the
binaries include support for AWS S3 and Google Cloud Storage (GCS).
These features require libcurl and openssl libraries installed
separately; see below on how to install them. If you don’t have these
installed, the libarrow binary won’t be used, and you will fall back to
the full source build.
Installing a release version (the less easy way)
Method 2 - Installing an R source package and building libarrow from
source
Generally, compiling and installing R packages with C++ dependencies
requires either installing system packages, which you may not have
privileges to do, or building the C++ dependencies separately, which
introduces all sorts of additional ways for things to go wrong.
The full source build of arrow, compiling both C++ and R bindings,
does handle most of the dependency management for you, but it is much
slower. However, if using binaries isn’t an option for you, or you wish
to fine-tune or customize your Linux installation, the instructions in
this section explain how to do that.
Basic configuration for building from source with fully featured
installation
If you wish to install libarrow from source instead of looking for
pre-compiled binaries, you can set the LIBARROW_BINARY
variable.
Sys.setenv("LIBARROW_BINARY" = FALSE)
By default, this is set to TRUE, and so libarrow will
only be built from source if this environment variable is set to
FALSE or no compatible binary for your OS can be found.
When compiling libarrow from source, you have the power to really
fine-tune which features to install. You can set the environment
variable LIBARROW_MINIMAL to FALSE to enable a
more full-featured build including S3 support and alternative memory
allocators.
Sys.setenv("LIBARROW_MINIMAL" = FALSE)
By default this variable is unset, which builds many commonly used
features such as Parquet support but disables some features that are
more costly to build, like S3 and GCS support. If set to
TRUE, a trimmed-down version of arrow is installed with all
optional features disabled.
Note that in this guide, you will have seen us mention the
environment variable NOT_CRAN - this is a convenience
variable, which when set to TRUE, automatically sets
LIBARROW_MINIMAL to FALSE and
LIBARROW_BINARY to TRUE.
Building libarrow from source requires more time and resources than
installing a binary. We recommend that you set the environment variable
ARROW_R_DEV to TRUE for more verbose output
during the installation process if anything goes wrong.
Sys.setenv("ARROW_R_DEV" = TRUE)
Once you have set these variables, call
install.packages() to install arrow using this
configuration.
install.packages("arrow")
The section below discusses environment variables you can set before
calling install.packages("arrow") to build from source and
customise your configuration.
Handling libarrow dependencies
When you build libarrow from source, its dependencies will be
automatically downloaded. The environment variable
ARROW_DEPENDENCY_SOURCE controls whether the libarrow
installation also downloads or installs all dependencies (when set to
BUNDLED), uses only system-installed dependencies (when set
to SYSTEM) or checks system-installed dependencies first
and only installs dependencies which aren’t already present (when set to
AUTO, the default).
These dependencies vary by platform; however, if you wish to install
these yourself prior to libarrow installation, we recommend that you
take a look at the docker file
for whichever of our CI builds (the ones ending in “cpp” are for
building Arrow’s C++ libaries, aka libarrow) corresponds most closely to
your setup. This will contain the most up-to-date information about
dependencies and minimum versions.
If downloading dependencies at build time is not an option, as when
building on a system that is disconnected or behind a firewall, there
are a few options. See “Offline builds” below.
Dependencies for S3 and GCS support
The arrow package allows you to work with data in AWS S3 or in other
cloud storage system that emulate S3, as well as Google Cloud Storage.
However, support for working with S3 and GCS is not enabled in the
default source build, and it has additional system requirements. To
enable it, set the environment variable
LIBARROW_MINIMAL=false or NOT_CRAN=true to
choose the full-featured build, or more selectively set
ARROW_S3=ON and/or ARROW_GCS=ON. You also need
the following system dependencies:
gcc >= 4.9 or clang >= 3.3; note
that the default compiler on CentOS 7 is gcc 4.8.5, which is not
sufficient- CURL: install
libcurl-devel (rpm) or
libcurl4-openssl-dev (deb)
- OpenSSL >= 1.0.2: install
openssl-devel (rpm) or
libssl-dev (deb)
The prebuilt libarrow binaries come with S3 and GCS support enabled,
so you will need to meet these system requirements in order to use them.
If you’re building everything from source, the install script will check
for the presence of these dependencies and turn off S3 and GCS support
in the build if the prerequisites are not met–installation will succeed
but without S3 or GCS functionality. If afterwards you install the
missing system requirements, you’ll need to reinstall the package in
order to enable S3 and GCS support.
Advanced configuration for building from source
In this section, we describe how to fine-tune your installation at a
more granular level.
libarrow configuration
Some features are optional when you build Arrow from source - you can
configure whether these components are built via the use of environment
variables. The names of the environment variables which control these
features and their default values are shown below.
ARROW_S3 |
S3 support (if dependencies are met)* |
OFF |
ARROW_GCS |
GCS support (if dependencies are met)* |
OFF |
ARROW_JEMALLOC |
The jemalloc memory allocator |
ON |
ARROW_MIMALLOC |
The mimalloc memory allocator |
ON |
ARROW_PARQUET |
|
ON |
ARROW_DATASET |
|
ON |
ARROW_JSON |
The JSON parsing library |
ON |
ARROW_WITH_RE2 |
The RE2 regular expression library, used in some string compute
functions |
ON |
ARROW_WITH_UTF8PROC |
The UTF8Proc string library, used in many other string compute
functions |
ON |
ARROW_WITH_BROTLI |
Compression algorithm |
ON |
ARROW_WITH_BZ2 |
Compression algorithm |
ON |
ARROW_WITH_LZ4 |
Compression algorithm |
ON |
ARROW_WITH_SNAPPY |
Compression algorithm |
ON |
ARROW_WITH_ZLIB |
Compression algorithm |
ON |
ARROW_WITH_ZSTD |
Compression algorithm |
ON |
R package configuration
There are a number of other variables that affect the
configure script and the bundled build script. All boolean
variables are case-insensitive.
LIBARROW_BUILD |
Allow building from source |
true |
LIBARROW_BINARY |
Try to install libarrow binary instead of building from
source |
(unset) |
LIBARROW_MINIMAL |
Build with minimal features enabled |
(unset) |
NOT_CRAN |
Set LIBARROW_BINARY=true and
LIBARROW_MINIMAL=false |
false |
ARROW_R_DEV |
More verbose messaging and regenerates some code |
false |
ARROW_USE_PKG_CONFIG |
Use pkg-config to search for libarrow
install |
true |
LIBARROW_DEBUG_DIR |
Directory to save source build logs |
(unset) |
CMAKE |
Alternative CMake path |
(unset) |
See below for more in-depth explanations of these environment
variables.
LIBARROW_BINARY : By default on many distributions, or
if explicitly set to true, the script will determine
whether there is a prebuilt libarrow that will work with your system.
You can set it to false to skip this option altogether, or
you can specify a string “distro-version” that corresponds to a binary
that is available, to override what this function may discover by
default. Possible values are: “centos-7” (gcc 4.8, no AWS/GCS support);
“ubuntu-18.04” (gcc 8, openssl 1); “ubuntu-22.04” (openssl 3).LIBARROW_BUILD : If set to false, the
build script will not attempt to build the C++ from source. This means
you will only get a working arrow R package if a prebuilt binary is
found. Use this if you want to avoid compiling the C++ library, which
may be slow and resource-intensive, and ensure that you only use a
prebuilt binary.LIBARROW_MINIMAL : If set to false, the
build script will enable some optional features, including S3 support
and additional alternative memory allocators. This will increase the
source build time but results in a more fully functional library. If set
to true turns off Parquet, Datasets, compression libraries,
and other optional features. This is not commonly used but may be
helpful if needing to compile on a platform that does not support these
features, e.g. Solaris.NOT_CRAN : If this variable is set to
true, as the devtools package does, the build
script will set LIBARROW_BINARY=true and
LIBARROW_MINIMAL=false unless those environment variables
are already set. This provides for a more complete and fast installation
experience for users who already have NOT_CRAN=true as part
of their workflow, without requiring additional environment variables to
be set.ARROW_R_DEV : If set to true, more verbose
messaging will be printed in the build script.
arrow::install_arrow(verbose = TRUE) sets this. This
variable also is needed if you’re modifying C++ code in the package: see
the developer guide vignette.ARROW_USE_PKG_CONFIG: If set to false, the
configure script won’t look for Arrow libraries on your system and
instead will look to download/build them. Use this if you have a version
mismatch between installed system libraries and the version of the R
package you’re installing.LIBARROW_DEBUG_DIR : If the C++ library building from
source fails (cmake), there may be messages telling you to
check some log file in the build directory. However, when the library is
built during R package installation, that location is in a temp
directory that is already deleted. To capture those logs, set this
variable to an absolute (not relative) path and the log files will be
copied there. The directory will be created if it does not exist.CMAKE : When building the C++ library from source, you
can specify a /path/to/cmake to use a different version
than whatever is found on the $PATH.
Install the nightly build
Daily development builds, which are not official releases, can be
installed from the Ursa Labs repository:
Sys.setenv(NOT_CRAN = TRUE)
install.packages("arrow", repos = c(arrow = "https://nightlies.apache.org/arrow/r", getOption("repos")))
or for conda users via:
conda install -c arrow-nightlies -c conda-forge --strict-channel-priority r-arrow
Install from git repo
You can also install the R package from a git checkout:
git clone https://github.com/apache/arrow
cd arrow/r
R CMD INSTALL .
If you don’t already have libarrow on your system, when installing
the R package from source, it will also download and build libarrow for
you. See the section above on build environment variables for options
for configuring the build source and enabled features.
Installation using install_arrow()
The previous instructions are useful for a fresh arrow installation,
but arrow provides the function install_arrow(), which you
can use if you:
- already have arrow installed and want to upgrade to a different
version
- want to install a development build
- want to try to reinstall and fix issues with Linux C++ binaries
install_arrow() provides some convenience wrappers
around the various environment variables described below.
Although this function is part of the arrow package, it is also
available as a standalone script, so you can access it for convenience
without first installing the package:
source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")
Install the latest release
Install the nightly build
install_arrow(nightly = TRUE)
Install with more verbose output for debugging errors
install_arrow(verbose = TRUE)
install_arrow() does not require environment variables
to be set in order to satisfy C++ dependencies.
Note that, unlike packages like tensorflow,
blogdown, and others that require external dependencies,
you do not need to run install_arrow() after a successful
arrow installation.
Offline installation
The install-arrow.R file also includes the
create_package_with_all_dependencies() function. Normally,
when installing on a computer with internet access, the build process
will download third-party dependencies as needed. This function provides
a way to download them in advance.
Doing so may be useful when installing Arrow on a computer without
internet access. Note that Arrow can be installed on a computer
without internet access without doing this, but many useful features
will be disabled, as they depend on third-party components. More
precisely, arrow::arrow_info()$capabilities() will be
FALSE for every capability. One approach to add more
capabilities in an offline install is to prepare a package with
pre-downloaded dependencies. The
create_package_with_all_dependencies() function does this
preparation.
If you’re using binary packages you shouldn’t need to follow these
steps. You should download the appropriate binary from your package
repository, transfer that to the offline computer, and install that. Any
OS can create the source bundle, but it cannot be installed on Windows.
(Instead, use a standard Windows binary package.)
Note if you’re using RStudio Package Manager on Linux: If you still
want to make a source bundle with this function, make sure to set the
first repo in options("repos") to be a mirror that contains
source packages (that is: something other than the RSPM binary mirror
URLs).
Step 1 - Using a computer with internet access, pre-download the
dependencies:
- Install the arrow package or run
source("https://raw.githubusercontent.com/apache/arrow/master/r/R/install-arrow.R")
- Run
create_package_with_all_dependencies("my_arrow_pkg.tar.gz")
- Copy the newly created
my_arrow_pkg.tar.gz to the
computer without internet access
Step 2 - On the computer without internet access, install the
prepared package:
- Install the arrow package from the copied file
install.packages("my_arrow_pkg.tar.gz", dependencies = c("Depends", "Imports", "LinkingTo"))- This installation will build from source, so
cmake must
be available
- Run
arrow_info() to check installed capabilities
Alternative, hands-on approach
- Download the dependency files
(
cpp/thirdparty/download_dependencies.sh may be
helpful)
- Copy the directory of dependencies to the offline computer
- Create the environment variable
ARROW_THIRDPARTY_DEPENDENCY_DIR on the offline computer,
pointing to the copied directory.
- Install the arrow package as usual.
Troubleshooting
The intent is that install.packages("arrow") will just
work and handle all C++ dependencies, but depending on your system, you
may have better results if you tune one of several parameters. Here are
some known complications and ways to address them.
Package failed to build C++ dependencies
If you see a message like
------------------------- NOTE ---------------------------
There was an issue preparing the Arrow C++ libraries.
See https://arrow.apache.org/docs/r/articles/install.html
---------------------------------------------------------
in the output when the package fails to install, that means that
installation failed to retrieve or build the libarrow version compatible
with the current version of the R package.
Please check the “Known installation issues” below to see if any
apply, and if none apply, set the environment variable
ARROW_R_DEV=TRUE for more verbose output and try installing
again. Then, please report an
issue and include the full installation output.
Using system libraries
If a system library or other installed Arrow is found but it doesn’t
match the R package version (for example, you have libarrow 1.0.0 on
your system and are installing R package 2.0.0), it is likely that the R
bindings will fail to compile. Because the Apache Arrow project is under
active development, it is essential that versions of libarrow and the R
package matches. When install.packages("arrow") has to
download libarrow, the install script ensures that you fetch the
libarrow version that corresponds to your R package version. However, if
you are using a version of libarrow already on your system, version
match isn’t guaranteed.
To fix version mismatch, you can either update your libarrow system
packages to match the R package version, or set the environment variable
ARROW_USE_PKG_CONFIG=FALSE to tell the configure script not
to look for system version of libarrow. (The latter is the default of
install_arrow().) System libarrow versions are available
corresponding to all CRAN releases but not for nightly or dev versions,
so depending on the R package version you’re installing, system libarrow
version may not be an option.
Note also that once you have a working R package installation based
on system (shared) libraries, if you update your system libarrow
installation, you’ll need to reinstall the R package to match its
version. Similarly, if you’re using libarrow system libraries, running
update.packages() after a new release of the arrow package
will likely fail unless you first update the libarrow system
packages.
Using prebuilt binaries
If the R package finds and downloads a prebuilt binary of libarrow,
but then the arrow package can’t be loaded, perhaps with “undefined
symbols” errors, please report an
issue. This is likely a compiler mismatch and may be resolvable by
setting some environment variables to instruct R to compile the packages
to match libarrow.
A workaround would be to set the environment variable
LIBARROW_BINARY=FALSE and retry installation: this value
instructs the package to build libarrow from source instead of
downloading the prebuilt binary. That should guarantee that the compiler
settings match.
If a prebuilt libarrow binary wasn’t found for your operating system
but you think it should have been, please report an
issue and share the console output. You may also set the environment
variable ARROW_R_DEV=TRUE for additional debug
messages.
Building libarrow from source
If building libarrow from source fails, check the error message. (If
you don’t see an error message, only the ----- NOTE -----,
set the environment variable ARROW_R_DEV=TRUE to increase
verbosity and retry installation.) The install script should work
everywhere, so if libarrow fails to compile, please report an
issue so that we can improve the script.
Known installation issues
On CentOS, if you are using a more modern
devtoolset, you may need to set the environment variables
CC and CXX either in the shell or in R’s
Makeconf. For CentOS 7 and above, both the Arrow system
packages and the C++ binaries for R are built with the default system
compilers. If you want to use either of these and you have a
devtoolset installed, set
CC=/usr/bin/gcc CXX=/usr/bin/g++ to use the system
compilers instead of the devtoolset. Alternatively, if you
want to build arrow with the newer devtoolset compilers,
set both ARROW_USE_PKG_CONFIG and
LIBARROW_BINARY to false so that you build the
Arrow C++ from source using those compilers. Compiler mismatch between
the arrow system libraries and the R package may cause R to segfault
when arrow package functions are used. See discussions here and here.
If you have multiple versions of zstd installed on
your system, installation by building libarrow from source may fail with
an “undefined symbols” error. Workarounds include (1) setting
LIBARROW_BINARY to use a C++ binary; (2) setting
ARROW_WITH_ZSTD=OFF to build without zstd; or
(3) uninstalling the conflicting zstd. See discussion here.
Contributing
As mentioned above, please report an
issue if you encounter ways to improve this. If you find that your
Linux distribution or version is not supported, we welcome the
contribution of Docker images (hosted on Docker Hub) that we can use in
our continuous integration. These Docker images should be minimal,
containing only R and the dependencies it requires. (For reference, see
the images that R-hub uses.)
You can test the arrow R package installation using the
docker-compose setup included in the
apache/arrow git repository. For example,
R_ORG=rhub R_IMAGE=ubuntu-gcc-release R_TAG=latest docker-compose build r
R_ORG=rhub R_IMAGE=ubuntu-gcc-release R_TAG=latest docker-compose run r
installs the arrow R package, including libarrow, on the rhub/ubuntu-gcc-release
image.