1. UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode

2. Character Sets Character Sets - a complete group of characters for one or more writing system. Coded character set – a mapping from a set of abstract characters to a set of integers. Character encoding scheme- a mapping from a coded character set to a set of octets.

3. Standards International standards – - RFC 2130, - ISO RFC standards come out of the Internet Engineer Task Force(IETF), it stands for Request For Comments. ISO is a worldwide federation of national standards bodies such as ANSI, JISC, KISI, CSA(GB} and ECMA.

4. Coded character sets (1) ASCII ( American Standard Code for Information Interchange) or ISO 646 defined by 7-bit -Latin alphabet; -Arabic numerals; -Punctuation marks; -Some computer control codes;

5. Coded character sets (2) ISO-8859 Standards- ISO-8859-1 to 15 ISO-8859-1(Latin1) - the most widely used standard, it contains the characters necessary for writing in Western European and Scandinavian languages. It is also an ANSI Standard and known as the ANSI character set. The first 128 positions are identical to ASCII All use 8-bit byte Windows use ISO-8859-1 as its character set.

6. Coded character sets (3) Asian Language character sets Traditional Chinese(Taiwan) -BIG5: 8-16 bit, 94x157 matrix :13002 characters; -CNS 11643 X5012: 16 bit, 48000 characters. Simplified Chinese(China) -GB2312-80: 8-16 bit, 94x94 matrix: 6763 characters. -GBK (Extended GB2312-80): 8-16 bit, 94x94 matrix, 21003 characters.

7. Coded character sets (3) Japanese(Japan) -JIS x0208: 16 bit, 94x94 matrix, 6879 characters. -JIS x0212: 16 bit, 5801 supplemental kanji. -JIS x0201: 7 bit, JIS-Roman plus half-width katakana. Korean(Korea) - KSX-1001(KSC 5601): 8-16 bit, 94x94 matrix, 4888 hanja. -KSC-5636: 7 bit Korean version of ASCII.

8. Coded character sets (4) EUC (Extended UNIX Code); ISO 2022 Escape sequence Windows Code pages: 8-16 bit, - cp1252 English is ISO8859-1 or Latin1or Windows ANSI; - cp1200 Unicode code page; - cp932 Japanese is shift-JIS; - cp936 Simplified Chinese is close to GB2312- 80; - cp949 Korean is in KSC 5601-1992 order; - cp950 Traditional Chinese is the same as Big 5.

9. Han Unification The Han standards consist of :JIS X0208(6349 Kanji), GB 2312-80 (6763 Hanzi), CNS 11543 (13951 Hanzi) and KSC 5601 (4888 Hanji). All characters in these standards must be included in 10646 Without unification, more than 100,000 characters are separately encoded Characters from these standards should be unified in ISO 10646 that is identical characters from two or more of these standards may have the same code point in 10646. The unification of Han characters allows for approximately 40,000 characters.

10. Coded character sets (5) Unicode: 16 bit, most languages are covered, developed by the Unicode Consortium. Unicode 1.0  Unicode 2.0 contains 38,885 characters. It will be Unicode 3.0 in this year. Unicode is similar to ISO 10646, the UCS(Universal Character Set) encoding, - UCS-2: i.e.Unicode, 16 bit, a fixed with 2 byte format scheme; - UCS-4: also called ISO 10646, 32 bit, 4 byte format, 32,000 planes each with 65,000 characters capacities, for tall 2,080 million characters. The 1 st plane is in use(that’s Unicode) - UTF-7: 7-40 bit, UCS Transformation Format, a Unicode character encoding scheme using a 7 bit. - UTF-8: 7-48 bit, UCS Transformation Format, a Unicode character encoding scheme using a 8 bit. - UTF-16(UCS-2E): 16-32 bit. Windows NT, use Unicode, has a single character set.

11. UCS Transformation Format; A variable-width or multi-byte encoding format; In UTF-8, the standard ASCII characters occupy only one byte, other Unicode characters occupy two or three bytes. Start CharacterEnd CharacterRequired Data Bits Binary Byte Sequence (x = data bits) \u0000\u007F7 0xxxxxxx \u0080\u07FF 11 110xxxxx 10xxxxxx \u0800\uFFFF161110xxxx 10xxxxxx 10xxxxxx Table The UTF-8 Encoding UTF-8

12. Unicode ISO 10646 (UCS2) 16 bits 8 bits EUC PC CPs Shift-JIS ISO2022 ISO8859 EBCDIC Apple CPs 7 bits ASCII GB JIS KSC Encoding Evolution

13. Programming Unicode How does Win32 support it How to write apps for Unicode How to be backward compatible

14. Win32 Server Windows NT Base Win32 App (Unicode) Win32 App Non-Unicode ANSI to Unicode Conversion Win32 Client Client-Server Boundary System fully Unicode internally Unicode in Win32

15. Separate Unicode Datatype - Wide character: Unicode - 8-bit char:ANSI, DBCS Parallel Unicode and ANSI APIs - Unicode and ANSI windows classes - Implicit code conversion Resources always in Unicode

16. Programming Unicode Basic techniques - How to migrate existing code base Special topics - Interaction with non-Unicode apps - Untyped file system Foundation for script/language specific functionality Migration example: Win32

17. Windows Unicode Programming Code Conversion Data Exchange Common Source Win32 EXE Non-Unicode Win32 EXE Unicode

18. Generic data types TCHAR LPTSTR wchar_t wchar_t * char char * Explicit data types WCHARCHARLPWSTR LPSTR wchar_tcharwchar_t *char * Generic data types in C

19. Macros and literals String literals TEXT(“hello”); “hello”;L”hello”; numeric equivalence ‘A’ = 0x41 0x0041 = L’A’

20. Dual function prototypes SetWindowText (HWND, LPTSTR) ; SetWindowTextA (HWND, LPSTR) ; SetWindowTextW (HWND, LPWSTR) ; #ifdef UNICODE Generic API prototypes Resolved to explicit prototypes

21. Basic conversion steps Use generic data types: TCHAR, LPTSTR for Text Use explicit types for BYTE pointers (data buffers) Use TEXT() macro for literal constants Adjust pointer arithmetic Use generic function prototypes

22. Conversion metrics About 10% of source lines need to be modified by global replace LPSTR  LPTSTR strstr()  wcswcs() About 2-5% need small modification lstrlen(s)  lstrlen(s)/sizeof(TCHAR) Less than 1% need revised algorithm OpenFile()  SearchPath(); CreateFile()

23. Summary: Migrating Windows- based Program to Unicode 1.Modify your code to use generic data types 2.Modify your code to use generic function prototypes 3.Surround any character or string literal with the TEXT macro. 4.Create generic version of your data structures 5.Change your make process. 6.Adjust pointer arithmetic. 7.Check for any code that assumes a character is always 1 byte long. 8.Add Unicode-specific code if necessary. 9.Add code to support special Unicode characters. 10.Determine how using Unicode will affect file I/O. 11.Double check the way in which you retrieve command line arguments. 12.Debug your port by enabling your compiler’s type-checking