1. Multimodal Apps: Tablet PC & Speech Development in.NET casey chesnut brains-N-brawn.com Wisconsin.NET June 2005

2. Source Code The associated source can be found here: –http://www.brains-n-brawn.com/artifacts/ugTabletSpeech.ziphttp://www.brains-n-brawn.com/artifacts/ugTabletSpeech.zip

3. Seamless Computing Advanced Web Services (MVP05) Compact Framework (MVP04) MapPoint Tablet PC (MVP03) Speech Artificial Intelligence Direct3D Media Center

4. Questions How many programmers? –Tablet PC –Speech –Media Center

5. Outline Tablet PC Speech –Speech API (SAPI) –Speech Application SDK (SASDK) –Speech Server Demo –Tablet and Speech –Media Center and Speech

6. Outline : Tablet PC Development environment How it works Working with Ink Opinion Future

7. Development Environment Windows XP Pro (non Tablet edition) Visual Studio.NET 1.1 Tablet PC SDK 1.7 –http://www.microsoft.com/downloads/details.aspx?familyid=b46d 4b83-a821-40bc-aa85-c9ee3d6e9699&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?familyid=b46d 4b83-a821-40bc-aa85-c9ee3d6e9699&displaylang=en Recognizer Pack –http://www.microsoft.com/downloads/details.aspx?FamilyId=080 184DD-5E92-4464-B907-10762E9F918B&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?FamilyId=080 184DD-5E92-4464-B907-10762E9F918B&displaylang=en Digitizer Board –http://www.wacom.com/productinfo/index.cfmhttp://www.wacom.com/productinfo/index.cfm Tablet PC

8. How Ink works Digitizer collects stroke information Strokes are broken up into characters / words / drawings Character / word stroke info is transformed into some feature set Feature set is run through some sort of pre-trained AI Output is mapped to a dictionary or words

9. Demo Digitizer collects stroke information Tablet PC Inspector –http://codebetter.com/blogs/peter.van.ooijen/archive/0001/01/01/56161.aspxhttp://codebetter.com/blogs/peter.van.ooijen/archive/0001/01/01/56161.aspx

10. Demo Strokes are broken up into characters / words / drawings InkDivider –Tablet PC SDK Sample

11. Demo Character / word stroke info is transformed into some feature set Feature set is run through some sort of pre- trained AI Demo –/aiTabletOcr Article –http:// www.brains-N-brawn.com/aiTabletOcr /http:// www.brains-N-brawn.com/aiTabletOcr /

12. Demo Output is mapped to a dictionary or words Dictionary Tool –http://blogs.msdn.com/omars/archive/2004/04/15/113597.aspxhttp://blogs.msdn.com/omars/archive/2004/04/15/113597.aspx Article –http:// www.brains-N-brawn.com/tabletDic /http:// www.brains-N-brawn.com/tabletDic /

13. Working with Ink InkControls InkOverlay –Collection –Recognition RealTimeStylus Ink on the web

14. Ink Controls InkEdit InkPicture Code from scratch

15. InkOverlay Collection Recognition Demo apps

16. RealTimeStylus RealTimeStylusPlugin –Tablet PC SDK Sample

17. Ink on the Web IE only InkBlogWeb –Tablet PC SDK Sample Article –http://www.brains-N-brawn.com/tabletWeb/http://www.brains-N-brawn.com/tabletWeb/

18. Opinion Green Light –Tablet PC Edition 2005 improved recognition and usability dramatically –Recognition Pack made development more accessible –Language Support Chinese (Traditional and Simplified),U.S. English, U.K. English, French, German, Italian, Japanese, Korean, Spanish

19. Possible Future VS.NET 2005? Avalon? Will IE7 have tighter integration with ink? Longhorn – baked in Possiblity for training ink recognition

20. What about Pocket PCs Handwriting Recognition Form factors

21. Outline : Speech How does it work? –Synthesis (TTS) –Recognition (SR) Development –Speech API (SAPI) –Speech Application SDK (SASDK) –Speech Server (MSS)

22. How Synthesis Works Text is converted to phonemes Phonemes are appended together Audio is played back Demo –/ttSpeech app Article –http://www.brains-N-brawn.com/ttSpeech/http://www.brains-N-brawn.com/ttSpeech/

23. How Recognition Works Audio wav is transformed to some meaningful form Phonemes are found in audio signals Phonemes are mapped to a dictionary or words Demo –wavReader app Article –http://www.brains-N-brawn.com/noReco/http://www.brains-N-brawn.com/noReco/

24. Speech API (SAPI) Old school COM Windows applications Can do dictation Demo –SAPI app

25. Opinion Yellow light –It works, but is aging –Has to be trained for dictation –Limited language support Green light for Tablet PCs –Tablet PC has recognition and synthesis engines installed –Some Tablets have microphone arrays built in

26. Future System.Speech –Simple API –Reflection capabilities –Standards support (SSML, SRGS) –Engines should be improved from all the Speech Server work

27. What about Pocket PCs OEMs can add VoiceCommand WindowsMobile has the SAPI API, but no engines PlatformBuilder is supposed to have engines There are 3 rd party engines for purchase

28. Speech Application SDK VS.NET 1.1 integration For web based apps –Voice-only telephony –Multimodal browser Demo –Code voice-only from scratch Article –http://www.brains-N-brawn.com/noHands/http://www.brains-N-brawn.com/noHands/

29. SASDK Speech Synthesis –Inline –Code behind –Prompt functions –Prompt databases Speech Recognition –Inline –Static Grammar –Dynamic Grammar –DTMF

30. Speech Server Runs SASDK applications Primarily for Voice-only apps Also for Multimodal PocketPC apps Speech Language Packs –North American Spanish –Canadian French Article –http://www.brains-N-brawn.com/speechMulti/http://www.brains-N-brawn.com/speechMulti/

31. Deployment

32. Opinion Green light for Voice-Only –Great tool support –Cheap hardware –Language support Red light for Multimodal –Standards battle with VoiceXml –IE Speech Add-Ins are not accessible –Pocket IE Speech Add-In not updated for R2 release, nor does it support Smartphone

33. Possible Future VS.NET 2005? XAML? Will IE7 have voice browsing built-in? Other browsers to add SALT support? Pocket IE Professional?

34. Combo Demos Ink and Speech (WinForm) –InkCollection app –http://www.brains-N-brawn.com/tabletStrator/http://www.brains-N-brawn.com/tabletStrator/ Ink and Speech (WebForm) –Video –http://www.brains-N-brawn.com/tabletWeb/http://www.brains-N-brawn.com/tabletWeb/ Remote and Speech (AddIn) –http://www.brains-N-brawn.com/mceSAPI/http://www.brains-N-brawn.com/mceSAPI/ Remote and Speech (HostedHTML) –http://www.brains-N-brawn.com/mceSALT/http://www.brains-N-brawn.com/mceSALT/

35. Questions