Pen Research Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation

TDNN Time-Delayed Neural Network Ink is cut into segments via simple algorithm For Latin script, we just cut at bottoms For Arabic or Hindi, we may need a different algorithm Features are computed per segment Chebychev coefficients Variety of size metrics: width, height, offsets to neighbors, etc. Each supported letter or character has an output node Many have 2 outputs, for beginning of character and continuation of character Very large training sets are collected from tens of thousands of native writers Time-Delayed Neural Network Ink is cut into segments via simple algorithm For Latin script, we just cut at bottoms For Arabic or Hindi, we may need a different algorithm Features are computed per segment Chebychev coefficients Variety of size metrics: width, height, offsets to neighbors, etc. Each supported letter or character has an output node Many have 2 outputs, for beginning of character and continuation of character Very large training sets are collected from tens of thousands of native writers

Beam Search Very similar to speech recognition systems System lexicon Simple list of words (like a spellchecker) Stored as a trie Plus regular expressions for numbers, dates, times, currency, phone numbers, etc. Low-scoring sequences are “culled” from the trie as we go Converts character recognizer into word recognizer We can recognize a word even if you mangle one of the letters Supports sloppier (and therefore faster) handwriting Very similar to speech recognition systems System lexicon Simple list of words (like a spellchecker) Stored as a trie Plus regular expressions for numbers, dates, times, currency, phone numbers, etc. Low-scoring sequences are “culled” from the trie as we go Converts character recognizer into word recognizer We can recognize a word even if you mangle one of the letters Supports sloppier (and therefore faster) handwriting

… … … Output Matrix dog68 clog57 dug51 doom42 divvy37 ooze35 cloy34 doxy29 client22 dozy13 Ink Segments Top 10 List d 92 a 88 b 14 c 86 o 77 a 73 l 76 t 5 g 68 t 8 b 6 o 65 g 57 t 12 TDNN Space TDNN a b d o g a b t t c l o g t Lexicon e a … … … … … Beam Search a* a d* d g* g o* o TDNN + Beam Search

Personalization New In Vista Shape adaptation Collect samples from you Simple idea: continue same training we do at Microsoft, but only on your samples Implicit Explicit Text adaptation Collect your personal words from Word and outgoing s New In Vista Shape adaptation Collect samples from you Simple idea: continue same training we do at Microsoft, but only on your samples Implicit Explicit Text adaptation Collect your personal words from Word and outgoing s Capital I? dev RTM KKOMO dogfooding Qi trie featurize Herry

Open System Recognizer API is published Any recognizer may support this API A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform Non-Microsoft recognizers available now in Japan, China, and Russia Compete with my group Or cover languages we don’t cover Or cover non-text (music, math, chemistry) Sorry: recognition result API is very text oriented Strings in Unicode Perhaps your text might be XML Recognizer API is published Any recognizer may support this API A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform Non-Microsoft recognizers available now in Japan, China, and Russia Compete with my group Or cover languages we don’t cover Or cover non-text (music, math, chemistry) Sorry: recognition result API is very text oriented Strings in Unicode Perhaps your text might be XML Microsoft Tablet PC / Tablet PC Platform / Programming the Tablet PC / Creating a Recognizer Start / All Programs / Microsoft Tablet PC Platform SDK / Microsoft Tablet PC Platform SDK Documentation

Latin Orthography XP has U.S. English, U.K. English, German, French, Spanish, and Italian Vista adds Dutch and Brazilian Portuguese Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian No ship estimates are available Serbian is written in both Latin and Cyrillic Making plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD World’s largest orthography Largest count of languages Largest combined count of literates XP has U.S. English, U.K. English, German, French, Spanish, and Italian Vista adds Dutch and Brazilian Portuguese Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian No ship estimates are available Serbian is written in both Latin and Cyrillic Making plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD World’s largest orthography Largest count of languages Largest combined count of literates

East Asian Orthographies Ideographic Orthographies Completely different code base Focus on large count of characters XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean Vista adds personalization, and improves cursive recognition Completely different code base Focus on large count of characters XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean Vista adds personalization, and improves cursive recognition

Cyrillic and Greek Orthographies Same TDNN + beam search technology works equally well in Greek and Cyrillic But we must collect new training sets Cyrillic has more upper/lowercase confusion Russian Collection completed Currently working on recognizer No estimate on shipping date Some work also underway in Serbian Cyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh Other former-Soviet Turkic republics are in a state of transition back to Latin scripts Serbian is written in both Latin and Cyrillic Same TDNN + beam search technology works equally well in Greek and Cyrillic But we must collect new training sets Cyrillic has more upper/lowercase confusion Russian Collection completed Currently working on recognizer No estimate on shipping date Some work also underway in Serbian Cyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh Other former-Soviet Turkic republics are in a state of transition back to Latin scripts Serbian is written in both Latin and Cyrillic

Greek ΕλληνικήέδωσανμουγλώσσαΤη Ink by Dr. John Drakopoulos I was given a language that is Hellenic

Russian Ink by Vladimir Smirnov Кремль. Кремль, Ото слышал всех я про Kremlin, Kremlin. I’ve heard from everyone about [it]

Bi-Directional Orthographies Arabic and Hebrew Text written right-to-left, but numbers written left-to-right Arabic has its own digits Hebrew uses “western” digits No uppercase / lowercase distinction Arabic is cursive-only Up to 4 forms per letter (initial, medial, final, isolated) Hebrew is print-only 5 letters have a separate final form Abjabs (consonant alphabets) We have collections underway Some initial experimentation in both languages No estimate on shipping date Arabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan ) Text written right-to-left, but numbers written left-to-right Arabic has its own digits Hebrew uses “western” digits No uppercase / lowercase distinction Arabic is cursive-only Up to 4 forms per letter (initial, medial, final, isolated) Hebrew is print-only 5 letters have a separate final form Abjabs (consonant alphabets) We have collections underway Some initial experimentation in both languages No estimate on shipping date Arabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan )

Bi-Directional Example Arabic ١٢٣ 123 عمره [his] age عاماً Year[s] Ink by Ahmed Kamal

Bi-Directional Example Hebrew מבבל. Plimpton 322 [is] from Babylonia. 322 פלימפטון Ink by Ethan Zoller

Brahmic Orthographies Left to right, no uppercase / lowercase Abugidas (syllabic alphabets) Default vowel (short “Ə” or “uh” sound) is not written All other vowels require a vowel sign added to the consonant This includes a “no-vowel sign” Called halant (“choked”) in Hindi, virama in other Indic languages Vowel sound follows consonant sound There are also independent vowels Hindi Devanagari collection in progress Some initial experimentation No estimate on shipping date Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India) Left to right, no uppercase / lowercase Abugidas (syllabic alphabets) Default vowel (short “Ə” or “uh” sound) is not written All other vowels require a vowel sign added to the consonant This includes a “no-vowel sign” Called halant (“choked”) in Hindi, virama in other Indic languages Vowel sound follows consonant sound There are also independent vowels Hindi Devanagari collection in progress Some initial experimentation No estimate on shipping date Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India)

Vowel Signs “KAA” “KU” “KUU” “KO” “KOW” “KE” “KEE” “KA” “KAI” “KAY” “KR” “K”

Consonant Clusters ST halant RING Unicode buffer: A transliteration of the English word “string”

Brahmic Comparison K Kh G Gh C Ch Devanagari: क ख ग घ च छ Bengali: ক খ গ ঘ চ ছ Gurmukhi: ਕ ਖ ਗ ਘ ਚ ਛ Gujarati: ક ખ ગ ઘ ચ છ Tamil: க ச Telugu: క ఖ గ ఘ చ ఛ Kannada: ಕ ಖ ಗ ಘ ಚ ಛ K Kh G Gh C Ch Devanagari: क ख ग घ च छ Bengali: ক খ গ ঘ চ ছ Gurmukhi: ਕ ਖ ਗ ਘ ਚ ਛ Gujarati: ક ખ ગ ઘ ચ છ Tamil: க ச Telugu: క ఖ గ ఘ చ ఛ Kannada: ಕ ಖ ಗ ಘ ಚ ಛ

Demo Title Name Title Group Name Title Group

Early Feedback As for Vista, I am in love with the B2 Tablet implementation! I can see where the effort is going! The TIP is awesome and the handwriting recognition is vastly improved. There is hardly a time where I have to correct my input - no matter how sloppy I am. Unsolicited feedback from a Vista Beta II user whose identity I do not know

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Backup Slides

Devanaguri Example प “PA” [R sign] period [abbreviation] headline ल “LA” period [abbreviation] headline म “MA” ि [vowel sign] “I” ा [vowel sign] “AA” ि [vowel sign] “I” Pr. Ltd. [Private Limited]