UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode

Character Sets Character Sets - a complete group of characters for one or more writing system. Coded character set – a mapping from a set of abstract characters to a set of integers. Character encoding scheme- a mapping from a coded character set to a set of octets.

Standards International standards – - RFC 2130, - ISO RFC standards come out of the Internet Engineer Task Force(IETF), it stands for Request For Comments. ISO is a worldwide federation of national standards bodies such as ANSI, JISC, KISI, CSA(GB} and ECMA.

Coded character sets (1) ASCII ( American Standard Code for Information Interchange) or ISO 646 defined by 7-bit -Latin alphabet; -Arabic numerals; -Punctuation marks; -Some computer control codes;

Coded character sets (2) ISO-8859 Standards- ISO to 15 ISO (Latin1) - the most widely used standard, it contains the characters necessary for writing in Western European and Scandinavian languages. It is also an ANSI Standard and known as the ANSI character set. The first 128 positions are identical to ASCII All use 8-bit byte Windows use ISO as its character set.

Coded character sets (3) Asian Language character sets Traditional Chinese(Taiwan) -BIG5: 8-16 bit, 94x157 matrix :13002 characters; -CNS X5012: 16 bit, characters. Simplified Chinese(China) -GB : 8-16 bit, 94x94 matrix: 6763 characters. -GBK (Extended GB ): 8-16 bit, 94x94 matrix, characters.

Coded character sets (3) Japanese(Japan) -JIS x0208: 16 bit, 94x94 matrix, 6879 characters. -JIS x0212: 16 bit, 5801 supplemental kanji. -JIS x0201: 7 bit, JIS-Roman plus half-width katakana. Korean(Korea) - KSX-1001(KSC 5601): 8-16 bit, 94x94 matrix, 4888 hanja. -KSC-5636: 7 bit Korean version of ASCII.

Coded character sets (4) EUC (Extended UNIX Code); ISO 2022 Escape sequence Windows Code pages: 8-16 bit, - cp1252 English is ISO or Latin1or Windows ANSI; - cp1200 Unicode code page; - cp932 Japanese is shift-JIS; - cp936 Simplified Chinese is close to GB ; - cp949 Korean is in KSC order; - cp950 Traditional Chinese is the same as Big 5.

Han Unification The Han standards consist of :JIS X0208(6349 Kanji), GB (6763 Hanzi), CNS (13951 Hanzi) and KSC 5601 (4888 Hanji). All characters in these standards must be included in Without unification, more than 100,000 characters are separately encoded Characters from these standards should be unified in ISO that is identical characters from two or more of these standards may have the same code point in The unification of Han characters allows for approximately 40,000 characters.

Coded character sets (5) Unicode: 16 bit, most languages are covered, developed by the Unicode Consortium. Unicode 1.0  Unicode 2.0 contains 38,885 characters. It will be Unicode 3.0 in this year. Unicode is similar to ISO 10646, the UCS(Universal Character Set) encoding, - UCS-2: i.e.Unicode, 16 bit, a fixed with 2 byte format scheme; - UCS-4: also called ISO 10646, 32 bit, 4 byte format, 32,000 planes each with 65,000 characters capacities, for tall 2,080 million characters. The 1 st plane is in use(that’s Unicode) - UTF-7: 7-40 bit, UCS Transformation Format, a Unicode character encoding scheme using a 7 bit. - UTF-8: 7-48 bit, UCS Transformation Format, a Unicode character encoding scheme using a 8 bit. - UTF-16(UCS-2E): bit. Windows NT, use Unicode, has a single character set.

UCS Transformation Format; A variable-width or multi-byte encoding format; In UTF-8, the standard ASCII characters occupy only one byte, other Unicode characters occupy two or three bytes. Start CharacterEnd CharacterRequired Data Bits Binary Byte Sequence (x = data bits) \u0000\u007F7 0xxxxxxx \u0080\u07FF xxxxx 10xxxxxx \u0800\uFFFF161110xxxx 10xxxxxx 10xxxxxx Table The UTF-8 Encoding UTF-8

Unicode ISO (UCS2) 16 bits 8 bits EUC PC CPs Shift-JIS ISO2022 ISO8859 EBCDIC Apple CPs 7 bits ASCII GB JIS KSC Encoding Evolution

Programming Unicode How does Win32 support it How to write apps for Unicode How to be backward compatible

Win32 Server Windows NT Base Win32 App (Unicode) Win32 App Non-Unicode ANSI to Unicode Conversion Win32 Client Client-Server Boundary System fully Unicode internally Unicode in Win32

Separate Unicode Datatype - Wide character: Unicode - 8-bit char:ANSI, DBCS Parallel Unicode and ANSI APIs - Unicode and ANSI windows classes - Implicit code conversion Resources always in Unicode

Programming Unicode Basic techniques - How to migrate existing code base Special topics - Interaction with non-Unicode apps - Untyped file system Foundation for script/language specific functionality Migration example: Win32

Windows Unicode Programming Code Conversion Data Exchange Common Source Win32 EXE Non-Unicode Win32 EXE Unicode

Generic data types TCHAR LPTSTR wchar_t wchar_t * char char * Explicit data types WCHARCHARLPWSTR LPSTR wchar_tcharwchar_t *char * Generic data types in C

Macros and literals String literals TEXT(“hello”); “hello”;L”hello”; numeric equivalence ‘A’ = 0x41 0x0041 = L’A’

Dual function prototypes SetWindowText (HWND, LPTSTR) ; SetWindowTextA (HWND, LPSTR) ; SetWindowTextW (HWND, LPWSTR) ; #ifdef UNICODE Generic API prototypes Resolved to explicit prototypes

Basic conversion steps Use generic data types: TCHAR, LPTSTR for Text Use explicit types for BYTE pointers (data buffers) Use TEXT() macro for literal constants Adjust pointer arithmetic Use generic function prototypes

Conversion metrics About 10% of source lines need to be modified by global replace LPSTR  LPTSTR strstr()  wcswcs() About 2-5% need small modification lstrlen(s)  lstrlen(s)/sizeof(TCHAR) Less than 1% need revised algorithm OpenFile()  SearchPath(); CreateFile()

Summary: Migrating Windows- based Program to Unicode 1.Modify your code to use generic data types 2.Modify your code to use generic function prototypes 3.Surround any character or string literal with the TEXT macro. 4.Create generic version of your data structures 5.Change your make process. 6.Adjust pointer arithmetic. 7.Check for any code that assumes a character is always 1 byte long. 8.Add Unicode-specific code if necessary. 9.Add code to support special Unicode characters. 10.Determine how using Unicode will affect file I/O. 11.Double check the way in which you retrieve command line arguments. 12.Debug your port by enabling your compiler’s type-checking