AWS S3 Backend

Dyfan Jones

AWS Athena allows SQL querying to be performed on AWS S3 buckets. To gain access to this, the correct permission level needs to be enabled.

AWS S3 Structures

noctua uploads the data into AWS S3, then registers the table in AWS Athena. When appending data to an existing AWS Athena table, noctua adds the data in the specified AWS S3 partition and then repairs the AWS Athena table.

noctua uses the parameter: s3.location from the function dbWriteTable for the AWS S3 location. If s3.location isn’t specified then the location is taken from the initial connection (dbConnect).

noctua aligns the s3.location to the following AWS S3 structure: {s3.location}/{schema}/{table_name}/{partition}/{file} (remember that s3.location has to be in s3 uri format: “s3://bucket-name/key-name”). This is to allow tables with same name to be uploaded to different schemas.

NOTE: noctua won’t duplicate the table name or schema if they have been provided in s3.location. For example:

dbWriteTable(con, "myschema.table", table, 
             s3.location = "s3://mybucket/myschema/table",
             partition = c("year" = "2020"))

# AWS S3 location
"s3://mybucket/myschema/table/year=2020/table.tsv"

File Types

Currently noctua supports the following file types [".tsv", ".csv", ".parquet"]. For parquet files, the package arrow is used. This package will have to be installed, before data can be sent to AWS S3 in parquet format.

noctua also supports compression when uploading data to AWS S3. For delimited files (".tsv" and ".csv"), gunzip compression is used. When using gunzip compression, noctua will split the zipped file into a maximum of 20 equal parts. This is to speed up how AWS Athena queries gunzip compressed files (Default Compression Method for Flat Files). Snappy compression is used for compressing parquet files.