Unicode: Emoji, accents, and international text

ASCII

The smallest unit of data transfer on modern computers is the byte, a sequence of eight ones and zeros that can encode a number between 0 and 255 (hexadecimal 0x00 and 0xff). In the earliest character encodings, the numbers from 0 to 127 (hexadecimal 0x00 to 0x7f) were standardized in an encoding known as ASCII, the American Standard Code for Information Interchange. Here are the characters corresponding to these codes:

codes <- matrix(0:127, 8, 16, byrow = TRUE,
                dimnames = list(0:7, c(0:9, letters[1:6])))
ascii <- apply(codes, c(1, 2), intToUtf8)

# replace control codes with ""
ascii["0", c(0:6, "e", "f")] <- ""
ascii["1",] <- ""
ascii["7", "f"] <- ""

utf8_print(ascii, quote = FALSE)

  0 1 2 3 4 5 6 7  8  9  a  b  c  d  e f
0               \a \b \t \n \v \f \r    
1                                       
2   ! " # $ % & '  (  )  *  +  ,  -  . /
3 0 1 2 3 4 5 6 7  8  9  :  ;  <  =  > ?
4 @ A B C D E F G  H  I  J  K  L  M  N O
5 P Q R S T U V W  X  Y  Z  [  \\ ]  ^ _
6 ` a b c d e f g  h  i  j  k  l  m  n o
7 p q r s t u v w  x  y  z  {  |  }  ~  

The first 32 codes (the first two rows of the table) are special control codes, the most common of which, 0x0a denotes a new line (\n). The special code 0x00 often denotes the end of the input, and R does not allow this value in character strings. Code 0x7f corresponds to a “delete” control.

When you call utf8_print, it uses the low level utf8_encode subroutine format control codes; they format as \uXXXX for four hexadecimal digits XXXX or as \UXXXXYYYY for eight hexadecimal digits XXXXYYYY:

utf8_print(intToUtf8(1:0x0f), quote = FALSE)

[1] \u0001\u0002\u0003\u0004\u0005\u0006\a\b\t\n\v\f\r\u000e\u000f

Compare utf8_print output with the output with the base R print function:

print(intToUtf8(1:0x0f), quote = FALSE)

[1] \001\002\003\004\005\006\a\b\t\n\v\f\r\016\017

Base R format control codes below 128 using octal escapes. There are some other differences between the function which we will highlight below.

Latin-1

ASCII works fine for most text in English, but not for other languages. The Latin-1 encoding extends ASCII to Latin languages by assigning the numbers 128 to 255 (hexadecimal 0x80 to 0xff) to other common characters in Latin languages. We can see these characters below.

codes <- matrix(128:255, 8, 16, byrow = TRUE,
                dimnames = list(c(8:9, letters[1:6]), c(0:9, letters[1:6])))
latin1 <- apply(codes, c(1, 2), intToUtf8)

# replace control codes with ""
latin1[c("8", "9"),] <- ""

utf8_print(latin1, quote = FALSE)

  0 1 2 3 4 5 6 7 8 9 a b c d e f
8                                
9                                
a   ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬   ® ¯
b ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿
c À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï
d Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß
e à á â ã ä å æ ç è é ê ë ì í î ï
f ð ñ ò ó ô õ ö ÷ ø ù ú û ü ý þ ÿ

As with ASCII, the first 32 numbers are control codes. The others are characters common in Latin languages. Note that 0xa3, the invalid byte from Mansfield Park, corresponds to a pound sign in the Latin-1 encoding. Given the context of the byte:

lines[15252]

[1] "the command of her beauty, and her \xa320,000, any one who could satisfy the"

this is probably the right symbol. The text is probably encoded in Latin-1, not UTF-8 or ASCII as claimed in the file.

If you run into an error while reading text that claims to be ASCII, it is probably encoded as Latin-1. Note, however, that this is not the only possibility, and there are many other encodings. The iconvlist function will list the ones that R knows how to process:

head(iconvlist(), n = 20)

 [1] "437"            "850"            "852"            "855"           
 [5] "857"            "860"            "861"            "862"           
 [9] "863"            "865"            "866"            "869"           
[13] "ANSI_X3.4-1968" "ANSI_X3.4-1986" "ARABIC"         "ARMSCII-8"     
[17] "ASCII"          "ASMO-708"       "ATARI"          "ATARIST"       

UTF-8

With only 256 unique values, a single byte is not enough to encode every character. Multi-byte encodings allow for encoding more. UTF-8 encodes characters using between 1 and 4 bytes each and allows for up to 1,112,064 character codes. Most of these codes are currently unassigned, but every year the Unicode consortium meets and adds new characters. You can find a list of all of the characters in the Unicode Character Database. A listing of the Emoji characters is available separately.

Say you want to input the Unicode character with hexadecimal code 0x2603. You can do so in one of three ways:

"\u2603"           # with \u + 4 hex digits

[1] "☃"

"\U00002603"       # with \U + 8 hex digits

[1] "☃"

intToUtf8(0x2603)  # from an integer

[1] "☃"

For characters above 0xffff, the first method won’t work. On Windows, a bug in the current version of R (fixed in R-devel) prevents using the second method.

When you try to print Unicode in R, the system will first try to determine whether the code is printable or not. Non-printable codes include control codes and unassigned codes. On Mac OS, R uses an outdated function to make this determination, so it is unable to print most emoji. The utf8_print function uses the most recent version (10.0.0) of the Unicode standard, and will print all Unicode characters supported by your system:

print(intToUtf8(0x1f600 + 0:79)) # base R

[1] "\U0001f600\U0001f601\U0001f602\U0001f603\U0001f604\U0001f605\U0001f606\U0001f607\U0001f608\U0001f609\U0001f60a\U0001f60b\U0001f60c\U0001f60d\U0001f60e\U0001f60f\U0001f610\U0001f611\U0001f612\U0001f613\U0001f614\U0001f615\U0001f616\U0001f617\U0001f618\U0001f619\U0001f61a\U0001f61b\U0001f61c\U0001f61d\U0001f61e\U0001f61f\U0001f620\U0001f621\U0001f622\U0001f623\U0001f624\U0001f625\U0001f626\U0001f627\U0001f628\U0001f629\U0001f62a\U0001f62b\U0001f62c\U0001f62d\U0001f62e\U0001f62f\U0001f630\U0001f631\U0001f632\U0001f633\U0001f634\U0001f635\U0001f636\U0001f637\U0001f638\U0001f639\U0001f63a\U0001f63b\U0001f63c\U0001f63d\U0001f63e\U0001f63f\U0001f640\U0001f641\U0001f642\U0001f643\U0001f644\U0001f645\U0001f646\U0001f647\U0001f648\U0001f649\U0001f64a\U0001f64b\U0001f64c\U0001f64d\U0001f64e\U0001f64f"

utf8_print(intToUtf8(0x1f600 + 0:79)) # truncates to line width

[1] "😀​😁​😂​😃​😄​😅​😆​😇​😈​😉​😊​😋​😌​😍​😎​😏​😐​😑​😒​😓​😔​😕​😖​😗​😘​😙​😚​😛​😜​😝​😞​😟​😠​😡​😢​😣​…"

utf8_print(intToUtf8(0x1f600 + 0:79), chars = 500) # increase character limit

[1] "😀​😁​😂​😃​😄​😅​😆​😇​😈​😉​😊​😋​😌​😍​😎​😏​😐​😑​😒​😓​😔​😕​😖​😗​😘​😙​😚​😛​😜​😝​😞​😟​😠​😡​😢​😣​😤​😥​😦​😧​😨​😩​😪​😫​😬​😭​😮​😯​😰​😱​😲​😳​😴​😵​😶​😷​😸​😹​😺​😻​😼​😽​😾​😿​🙀​🙁​🙂​🙃​🙄​🙅​🙆​🙇​🙈​🙉​🙊​🙋​🙌​🙍​🙎​🙏​"

(Characters with codes above 0xffff, including most emoji, are not supported on Windows.)

The utf8 package provides the following utilities for validating, formatting, and printing UTF-8 characters:

• as_utf8() attempts to convert character data to UTF-8, throwing an error if the data is invalid;

• utf8_valid() tests whether character data is valid according to its declared encoding;

• utf8_normalize() converts text to Unicode composed normal form (NFC), optionally applying case-folding and compatibility maps;

• utf8_encode() encodes a character string, escaping all control characters, so that it can be safely printed to the screen;

• utf8_format() formats a character vector by truncating to a specified character width limit or by left, right, or center justifying;

• utf8_print() prints UTF-8 character data to the screen;

• utf8_width() measures the display with of UTF-8 character strings (many emoji and East Asian characters are twice as wide as other characters).

The package does not provide a method to translate from another encoding to UTF-8 as the iconv() function from base R already serves this purpose.