1. Unicode: A Grand Tour Character Encodings & Unicode

2. This presentation and its associated materials licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 License. You may use these materials without obtaining permission from the author. Any materials used or redistributed must contain this notice. [Derivative works may be permitted with permission of the author.] This work is copyright © 2008 Addison P. Phillips

3. Addison Phillips Globalization Architect, Lab126 This Presentation “Internationalization and Unicode Conference” Tutorial

4. Globalization Architect, Lab126 (Yes, you can touch my Kindle) Chair, W3C Internationalization WG Editor, IETF LTRU-WG (BCP 47) Unicode

5. the design and development of a product that is enabled for target audiences that vary in culture, region, or language. [W3C] a fundamental architectural approach to software development

6. Opinions differ on capitalization (C12N); choose from:  i18N  I18n  I18N Very geeky; not very internationalized ( I19G ?) I N T E R N A T I O N A L I Z A T I O N I 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 N I18N Localization=L10N Globalization=G11N Canonicalization=C14N

7. The basics of text processing in software.

8. “Character encodings consume more than 80% of my work day. They are the source of more mis- information and confusion than any other single thing. And developers aren’t getting any better educated.” ~Glen Perkins Globalization Architect

9. Real Jargon Multibyte Variable width Wide character Character encoding Coded character set Bidi or bidirectional Glyph, character, code unit Unicode Potentially Bogus Jargon kanji double-byte language extended ASCII ANSI, OEM encoding agnostic

10. “ bits ” : 010000010101101101101000 “ byte ” or “ octet ” : 01000001 (0x41) code unit: a unit of physical storage and information interchange represent numbersrepresent numbers come in various sizes (e.g. 7, 8, 16, 32, 64 bits)come in various sizes (e.g. 7, 8, 16, 32, 64 bits) how do we map text to the numbers used by computers?

11. Glyphs A “ glyph ” is screen unit of text: it ’ s a picture of what users think of as a character. A “ grapheme ” is a single visual unit of text. Characters A “ character ” is a single logical unit of text. A “character set” is a set of characters. A “ code point ” is a number assigned to a character in a character set. A “ coded character set ” is a character set where each character has a code point. Bytes A “character encoding” maps a sequence of code points ( “ characters ” ) to a sequence of code units (such as bytes). A “ code unit ” is a single logical unit of storage. … 0xC3 0x80 … U+00C0 À

12. Collection ( repertoire ) of characters, that is: a set. Organized so that each character has a unique numeric (typically integer) value ( code point ). Examples: Unicode ASCII (ANSI X3.4) ISO 646 JIS X 208 Latin-1 (ISO 8859-1)

13. Maps a sequence of code points (characters) to a sequence of code units (e.g. bytes). Some encodings use another unit instead of the byte. For example, some encodings use a 16-bit, 32-bit, or 64- bit code unit. U+00C00xC3 0x80

14. All texthas a character encoding All text has a character encoding When things go wrong, start by asking what the encoding is, what encoding you expected it to be, and whether the bytes match the encoding. In memory, on disk, on the network, etc.

15. Tofu hollow boxes Mojibake garbage characters Question Marks (conversion not supported)

17. Can appear as either hollow boxes (empty glyph) or as question marks (Firefox, for example) Not usually a bug: it’s a display problem Can mask or masquerade as character corruption.

18. When Good Characters Go Bad

19. View text using the wrong encoding Apply a transfer encoding and forget to remove it Convert to an encoding twice Convert to or from the wrong encoding Overzealous escaping Conversion to entities ( “ entitization ” ) Multiple conversions

22. 7 bits = 2 7 = 128 characters Enough for “ U.S. English ”

23. ASCII for characters 0x00 through 0x7F Accented letters and other symbols 0x80 through 0xFF

24. charCp1252Cp437Cp850 È0xC8?0xD4

25. Windows ’ s encodings (called “ code pages ” ) are generally based on standard encodings — plus some additional characters. Example: CP 1252 is based on ISO 8859-1, but includes 27 “ extra ” characters in the C1 control range (0x80- 0x9F)

26. Originally an IBM character encoding term. IBM numbered their character sets with “ CCSIDs ” (coded character set ids) and numbered the corresponding character encodings as “ code pages ”. Microsoft borrowed code pages to create PC-DOS. Microsoft defines two kinds of code pages: “ ANSI ” code pages are the ones used by Windows GUI programs. “ OEM ” code pages are the ones used by command shell/command line programs. Neither “ ANSI ” nor “ OEM ” refer to a particular encoding standard or standards body in this context. Avoid the use of ANSI and OEM when referring to encodings.

27. So far we ’ ve been looking at single-byte encodings:  one byte per character  1 byte = 1 character (= 1 glyph?)  256 character maximum  Good enough for most alphabetic languages Some languages need more characters. What about the “double-byte” languages? Don’t those take two bytes per character? 丏丣並 À

28. Escape sequences to select another character set Example: ISO 2022 uses escape sequences to select various encodings Use a larger code unit ( “ wide ” character encoding) Example: IBM DBCS code pages or Unicode UTF-16 2 16 = 64K characters 2 32 = 4.2 billion characters Use a variable-width encoding Variable width encodings use different numbers of code units to represent different types of characters within the same encoding

29. One or more bytes per character 1 byte != 1 character May use 1, 2, 3, or 4 bytes per character May use shift or escape sequences May encode more than one character set In fact, single-byte encodings are a special case of multibyte! Multibyte Encoding: Any “variable-width” encoding that uses the byte as its code unit.

30. JIS X 213  11,233 characters  (2) 94x94 character planes

31. Specific byte ranges encoding characters that take more than one byte. A “ lead byte ” One or more “ trailing bytes ” Code point != code unit あ 1-4-1 (code point) 0x82 0xA0 A 1-3-33 (code point) 0x41

32. In order to reach more characters, Shift_JIS characters start with a limited range of “lead bytes” These can be followed by a larger range of byte values (“trail byte”)

34. Lead bytes can be trail byte values Trail bytes include ASCII values Trail bytes include special values such as 0x5C ( “ \ ” ) int pos = strchr(mybuf, ‘@’);

35. Stateful Encodings ex. IBM “ MBCS ” code pages [SI/SO shift between 1- byte and 2-byte characters] ISO 2022 [escape sequence changes character set being encoded]

37. A transfer encoding syntax is a reversible transform of encoded data which may (or may not) include textual data represented in one or more character encoding schemes. Email headers URIs IDN (domain names) Abc ソース =?UTF-8?B?QWJj44K 944O844K5?= Abc ソース

38. Document formats often require a single character encoding be used for all parts of the document. Process Output (HTML, XML, etc.) Templates ISO 8859-1 Content UTF-8 Data Shift_JIS When data is merged, the encodings must be merged also (or some of the data will be “mojibake”). Common Encoding Conversion Tools and Libraries iconv (Unix) ICU (C, C++, Java) perl Encode Java (native2ascii, IO/NIO) (etc.)

39. Encoding conversion acts as a “ filter ” Replacement characters ( “ question marks ” ) replace characters from the source character set that are not present in the target character set. ISO 8859-1 ÀàС£ ?????? »èç????? ???? UTF-8 ÀàС£ ?????? »èç????? ???? ISO 8859-1 ÀàС£ UTF-8 детски »èçينس 文字 Shift_JIS 文字化け ? (0x3F) is the replacement character for ISO 8859-1

40. Need for more converters and conversion maps Difficulty of passing, storing, and processing data in multiple encodings Too many character sets… … leads to what we call “ code page hell ”

41. A

42. Fights mojibake because: characters are from the common repertoire; characters are encoded according to one of the encoding forms; characters are interpreted with Unicode semantics; unknown characters are not corrupted Basic Principles Universal repertoire Logical order Efficiency Unification Characters, not glyphs Dynamic composition Semantics Stability Plain Text Convertibility

43. Unicode is a character set that supports all of the world’s languages and writing systems.  Code space of up to 0x10FFFF characters (about 1.1 million)  Unicode and ISO 10646 are maintained in sync.  Unicode is maintained by an industry consortium.  ISO 10646 is maintained by the ISO.

44.  Divide Unicode in equal sized regions of code points.  17 planes (0 through 0x10), each with 65,535 characters.  Plane 0 is called the Basic Multilingual Plane (BMP).  > 99% of text in the wild lives in the BMP  Planes 1 through 0x10 are called supplementary planes.

45. An organized collection of characters. Each character has a code point aka Unicode Scalar Value (USV) U+0041 <= hex notation

46. code point name character class combining level bidi class case mappings canonical decomposition mirroring default grapheme clustering ӑ (U+04D1) CYRILLIC SMALL LETTER A WITH BREVE  letter  non-combining  left-to-right  decomposes to U+0430 U+0306  Ӑ U+04D0 is uppercase (and titlecase)

47. Many characters were included in Unicode for round-trip conversion compatibility with legacy encodings: ①②③４５Ⅵ ¾ Lj ¼ Nj ½ dž ︴︷︻︽﹁﹄ ｦｨｩｫｪｭﾞ ﺲ ﺳ ﻫ ﺽ ﵬ ﷺ fi fl ffi ffl ſt ﬔ Compatibility Characters includes presentation forms legacy encoding: a term for non- Unicode character encodings.

48. U+FEFF Used to indicate the “ byte-order ” of UTF-16 code units 0xFE FF; 0xFF FE Also used as a Unicode signature by some software (Windows ’ s Notepad editor, for example) for UTF-8 0xEF BB BF Appears as a character or renders as junk in some formats or on some systems. For example, older browsers render it as three bytes of mojibake.

49. U+FFFD Indicates a bad byte sequence or a character that could not be converted. Equivalent to “ question marks ” in legacy encoding conversions � there was a character here, but it is gone now

50. Composition can create “new” characters Base + non-spacing (“combining”) characters A + ˚ = Å U+0041 + U+030A = U+00C5 a + ˆ +. = ậ U+0061 + U+0302 + U+0323 = U+1EAD a +. + ˆ = ậ U+0061 + U+0323 + U+0302 = U+1EAD

51. ญัตติที่เสนอได้ผ่านที่ประชุมด้วยมติเอกฉันท ญั = ญ + ั glyph = consonant + vowel ญัตติที่เสนอได้ผ่านที่ประชุมด้วยมติเอกฉันท (word boundaries)

52. What is Unicode? यूनिकोड क्या है ? यू नि को ड न + ि = नि

53. கொ ‘ko’ U+0B95 U+0BBE U+0BC6 Combining mark drawn to the left of the base character KaAaE

55. UTF-32 Uses 32-bit code units. All characters are the same width. UTF-16 Uses 16-bit code units. BMP characters use one 16-bit code unit. Supplementary characters use two special 16-bit code units: a “surrogate pair”. UTF-8 Uses 8-bit code units (bytes!) It ’ s a multi-byte encoding! Characters use between 1 and 4 bytes. ASCII is ASCII in UTF-8

56. (U+1251) UTF-32:0x00001251 UTF-16:0x1251 UTF-8:0xE1 0x89 0x91 (U+10338) 0x00010338 0xD800 0xDF38 0xF0 0x90 0x8C 0xB8 A (U+0041) UTF-32:0x0000041 UTF-16:0x0041 UTF-8:0x41 À (U+00C0) UTF-32:0x000000C0 UTF-16:0x00C0 UTF-8:0xC2 0x80

57. Uses 32-bit code units (instead of the more-familiar 8- bit code unit, aka the “ byte ” ) Each character takes exactly one code unit. U+1251 0x00001251 U+10338 0x00010338

58. Easy to process each logical character takes one code unit can use pointer arithmetic Not commonly used Not efficient for storage 11 bits are never used BMP characters are the most common — 16 bits wasted for each of these Affected by processor architecture (Big-Endian vs. Little-Endian)

59. Uses 16-bit code units (instead of the more-familiar 8- bit code unit, aka the “ byte ” ) BMP characters use one unit Supplementary characters use a “ surrogate pair ”, special code points that don ’ t do anything else. 0x1251U+1251 0x1251U+1251 0xD800 0xDF38U+10338 0xD800 0xDF38U+10338 High Surrogate Low Surrogate 0xD800-DBFF0xDC00-DFFF Unique Ranges!

60. Most common languages and scripts are encoded in the BMP. Less wasteful than UTF-32 Simpler to process (excepting surrogates) Commonly supported in major operating environments, programming languages, and libraries May not be suitable for all applications Affected by processor architecture (Big-Endian vs. Little-Endian) Requires more storage, on average, for Western European scripts, ASCII, HTML/XML markup.

61. 7-bit ASCII is itself All other characters take 2, 3, or 4 bytes each lead bytes have a special pattern trailing bytes range from 0x80 -> 0xBF 0xxxxxxx 0xxxxxxx 110xxxxx 10xxxxxx 110xxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx 1110xxxx 10xxxxxx 10xxxxxx 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx Lead Bytes Trail Bytes < 0x80 < 0x800 < 0x10000 Supplementary Code Points

62. ASCII-compatible Default or recommended encoding for many Internet standards Bit pattern highly detectable (over longer runs) Non-endian Streaming C char* friendly Easy to navigate Multibyte encoding requires additional processing awareness Non-shortest form checking needed Less efficient than UTF-16 for large runs of Asian text

63. Set Web server to declare UTF-8 in HTTP Content-Type header Declare UTF-8 in META tag header Actually use UTF-8 as the encoding!! <?php header("Content-type: text/html; charset=UTF-8"); ?> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8” /> Fight 文字化け !

64. It’s more than just a character set and some encodings…

65. Unicode provides additional information:  Character name  Character class  “ctype” information, such as if it’s a digit, number, alphabetic, etc.  Directionality (LTR, RTL, etc.) and the Bidi Algorithm  Case mappings (UPPER, lower, and Titlecase)  Default Collation and the Unicode Collation Algorithm (UCA)  Identifier names  Regular Expression syntaxes  Normalization  Compatibility information Many of these items are in the form of Unicode Technical Reports http://www.unicode.org/reports

66. Abc ABC abc abC aBc abc Unicode Normalization has to deal with more issues: single or multiple combining markssingle or multiple combining marks compatibility characterscompatibility characters presentation formspresentation forms Ǻ U+01FA U+00C5 U+0301 U+00C1 U+030A U+212B U+0301 U+0041 U+0301 U+030A U+0041 U+030A U+0301

67. Ǻ Form D canonical decomposition Form C canonical decomposition followed by composition Form KD kompatibility decomposition Form KC kompatibility decomposition followed by composition ways to represent: U+01FA U+00C5 U+0301 U+00C1 U+030A U+212B U+0301 U+0041 U+0301 U+030A U+0041 U+030A U+0301

68. OriginalForm CForm DForm KCForm KD U+01FA U+0041 U+0301 U+030A U+01FAU+0041 U+0301 U+030A U+00C5 U+0301U+01FAU+0041 U+0301 U+030A U+01FAU+0041 U+0301 U+030A U+00C1 U+030AU+01FAU+0041 U+0301 U+030A U+01FAU+0041 U+0301 U+030A U+212B U+0301 U+01FAU+0041 U+0301 U+030A U+01FAU+0041 U+0301 U+030A U+01FAU+0041 U+0301 U+030A U+0041 U+030A U+0301 U+01FAU+0041 U+0301 U+030A U+01FAU+0041 U+0301 U+030A Ǻ

69. Not all compatibility characters have a compatibility decomposition. Not all characters that look alike or have similar semantics have a compatibility decomposition. For example, there are many ‘dots’ used as a period. Not all character variations are handled by normalization. For example, upper, title, and lowercase variations. Normalization can remove meaning

70. Some languages are written predominantly from left- to-right (LTR). Some languages are written predominantly from right- to-left (RTL). (A few can be written top- to-bottom or using other schemes) Unicode defines character “ directionality ” and a “ Bidi ” algorithm for rendering text. Uses logical, not visual, order. Uses levels of “ embedding ”. Requires markup changes in some HTML for full support.

71. Characters are encoded in logical order. Visual order is determined by the layout. Override and bidi control characters “Indeterminate” characters

72. Paste in Arabic

73. Defines default collation algorithm and sequences (UTS#10) Must be tailored by language and “locale” (culture) and other variations. Language Swedish:z < ö German:ö < z Usage German Dictionary: öf < of German Telephone: of < öf Customizations Upper-firstA < a Lower-Firsta < A

74. Find grapheme, word, and line-break boundaries in text. Tailored by language Provides good basic default handling

76. Remember “all text has an encoding”? user input via forms email data feeds existing, legacy data database instances uploads Use UTF-8 for HTML and Web forms Use UTF-8 in your APIs Check that data really is UTF- 8 Control encoding via code; avoid hard-coding the encoding Watch out for legacy encodings Convert to Unicode as soon as practical. Convert from Unicode as late as possible. Wrap Unicode-unfriendly technologies

77. Your System Map Your System APIs  use Unicode encoding  hide internal storage encoding Data Stores, Local I/O  use Unicode encoding  consider an encoding conversion plan Front Ends  use Unicode encoding Back Ends, External Data  Uses Unicode?  If not, what encoding?  Store the encoding! API Unicode Legacy Encoding Detect / Convert Capture Encoding Detect / Convert Unicode Cloud Unicode Interface Convert to Legacy Input

78. Be aware of whether you need to count glyphs, characters, or bytes: Is the limit “ screen positions ”, “ characters ”, or “ bytes of storage ” ? Should you be using a different limit? Which one are you actually counting? यूनिकोड (4 glyphs) य ू न ि क ो ड (7 characters) E0-A4-AF E0-A5-82 E0-A4-A8 E0-A4-BF E0-A4-95 E0-A5-8B E0-A4-A1 (21 bytes) varchar(110)

79. Code unit Code point Character Glyph Multibyte encoding Tofu Mojibake Question Marks “All text has an encoding”

80. 17 planes of goodness 1.1 million potential code points 150,000 assigned code points 3 encodings UTF-32 UTF-16 UTF-8 Normalize Bidi Collation Case folding … and so much more

81. “ Would you please write the code for I18N on the whiteboard before you go? ” #import i18n.h #define UNICODE Would you write the code for I18N on the whiteboard before you go? #define UNICODE #import I18N.h