Working with Cloud Storage (S3, GCS)

Creating a FileSystem object

One way of working with filesystems is to create ?FileSystem objects. ?S3FileSystem objects can be created with the s3_bucket() function, which automatically detects the bucket’s AWS region. Similarly, ?GcsFileSystem objects can be created with the gs_bucket() function. The resulting FileSystem will consider paths relative to the bucket’s path (so for example you don’t need to prefix the bucket path when listing a directory).

With a FileSystem object, you can point to specific files in it with the $path() method and pass the result to file readers and writers (read_parquet(), write_feather(), et al.). For example, to read a parquet file from the example NYC taxi data (used in vignette("dataset", package = "arrow")):

bucket <- s3_bucket("voltrondata-labs-datasets")
# Or in GCS (anonymous = TRUE is required if credentials are not configured):
bucket <- gs_bucket("voltrondata-labs-datasets", anonymous = TRUE)
df <- read_parquet(bucket$path("nyc-taxi/year=2019/month=6/data.parquet"))

Note that this will be slower to read than if the file were local, though if you’re running on a machine in the same AWS region as the file in S3, the cost of reading the data over the network should be much lower.

You can list the files and/or directories in a bucket or subdirectory using the $ls() method:

bucket$ls("nyc-taxi")
# Or recursive:
bucket$ls("nyc-taxi", recursive = TRUE)

NOTE: in GCS, you should always use recursive = TRUE as directories often don’t appear in $ls() results.

See help(FileSystem) for a list of options that s3_bucket()/S3FileSystem$create() and gs_bucket()/GcsFileSystem$create() can take.

The object that s3_bucket() and gs_bucket() return is technically a SubTreeFileSystem, which holds a path and a file system to which it corresponds. SubTreeFileSystems can be useful for holding a reference to a subdirectory somewhere (on S3, GCS, or elsewhere).

One way to get a subtree is to call the $cd() method on a FileSystem

june2019 <- bucket$cd("2019/06")
df <- read_parquet(june2019$path("data.parquet"))

SubTreeFileSystem can also be made from a URI:

june2019 <- SubTreeFileSystem$create("s3://voltrondata-labs-datasets/nyc-taxi/2019/06")

URIs

File readers and writers (read_parquet(), write_feather(), et al.) also accept a URI as the source or destination file, as do open_dataset() and write_dataset(). An S3 URI looks like:

s3://[access_key:secret_key@]bucket/path[?region=]

A GCS URI looks like:

gs://[access_key:secret_key@]bucket/path
gs://anonymous@bucket/path

For example, one of the NYC taxi data files used in vignette("dataset", package = "arrow") is found at

s3://voltrondata-labs-datasets/nyc-taxi/year=2019/month=6/data.parquet
# Or in GCS (anonymous required on public buckets):
gs://anonymous@voltrondata-labs-datasets/nyc-taxi/year=2019/month=6/data.parquet

Given this URI, you can pass it to read_parquet() just as if it were a local file path:

df <- read_parquet("s3://voltrondata-labs-datasets/nyc-taxi/year=2019/month=6/data.parquet")
# Or in GCS:
df <- read_parquet("gs://anonymous@voltrondata-labs-datasets/nyc-taxi/year=2019/month=6/data.parquet")

s3://voltrondata-labs-datasets/?endpoint_override=https%3A%2F%2Fstorage.googleapis.com&allow_bucket_creation=true

fs <- S3FileSystem$create(
  endpoint_override="https://storage.googleapis.com",
  allow_bucket_creation=TRUE
)
fs$path("voltrondata-labs-datasets/")

gs://anonymous@voltrondata-labs-datasets/nyc-taxi/?retry_limit_seconds=10

Authentication

gcloud auth application-default login

bucket <- gs_bucket("voltrondata-labs-datasets", anonymous = TRUE)
fs <- GcsFileSystem$create(anonymous = TRUE)
df <- read_parquet("gs://anonymous@voltrondata-labs-datasets/nyc-taxi/year=2019/month=6/data.parquet")

Using a proxy server

If you need to use a proxy server to connect to an S3 bucket, you can provide a URI in the form http://user:password@host:port to proxy_options. For example, a local proxy server running on port 1316 can be used like this:

bucket <- s3_bucket("voltrondata-labs-datasets", proxy_options = "http://localhost:1316")

File systems that emulate S3

The S3FileSystem machinery enables you to work with any file system that provides an S3-compatible interface. For example, MinIO is and object-storage server that emulates the S3 API. If you were to run minio server locally with its default settings, you could connect to it with arrow using S3FileSystem like this:

minio <- S3FileSystem$create(
  access_key = "minioadmin",
  secret_key = "minioadmin",
  scheme = "http",
  endpoint_override = "localhost:9000"
)

or, as a URI, it would be

s3://minioadmin:minioadmin@?scheme=http&endpoint_override=localhost%3A9000

(note the URL escaping of the : in endpoint_override).

Among other applications, this can be useful for testing out code locally before running on a remote S3 bucket.

Disabling the use of environment variables

As mentioned above, it is possible to make use of environment variables to configure access. However, if you wish to pass in connection details via a URI or alternative methods but also have existing AWS environment variables defined, these may interfere with your session. For example, you may see an error message like:

Error: IOError: When resolving region for bucket 'analysis': AWS Error [code 99]: curlCode: 6, Couldn't resolve host name 

You can unset these environment variables using Sys.unsetenv(), for example:

Sys.unsetenv("AWS_DEFAULT_REGION")
Sys.unsetenv("AWS_S3_ENDPOINT")

By default, the AWS SDK tries to retrieve metadata about user configuration, which can cause conficts when passing in connection details via URI (for example when accessing a MINIO bucket). To disable the use of AWS environment variables, you can set environment variable AWS_EC2_METADATA_DISABLED to TRUE.

Sys.setenv(AWS_EC2_METADATA_DISABLED = TRUE)