1 Keyboard, Speech, and Pen Input in Your Controls Kevin Gjerstad Lead Program Manager Kevin Gjerstad Lead Program Manager Session Code: CLI351 Robert Chambers Development Manager

2 Agenda Why Input Matters To Your App Avalon Input Goals and Architecture Events, Routing, Handling Code Demos Feature Demos Customizing Speech in Your App Summary Why Input Matters To Your App Avalon Input Goals and Architecture Events, Routing, Handling Code Demos Feature Demos Customizing Speech in Your App Summary

3 Tools Client Application Model AvalonWindows Forms Web & Service Application Model ASP.NET / Indigo Win FS Compact Framework Yukon Mobile PC Optimized System.Help System.Drawing System.NaturalLanguageServices Data Systems Application Model Presentation Data Mobile PC & Devices Application Model Communication Command Line NT Service DataSet Mapping ObjectSpaces ObjectSpace Query Schema Item Relationship Media Audio Video Images System.Messaging System. Discovery System.DirectoryServices System.Remoting System.Runtime.Remoting Active Directory Uddi System.Web.Services Web.Service Description Discovery Protocols System.MessageBus Transport Port Channel Service Queue PubSub Router System.Timers System.Globalization System.Serialization System.Threading System.Text System.Design Base & Application Services Fundamentals System.ComponentModel System.CodeDom System.Reflection System.EnterpriseServices System.Transactions Security System.Windows. TrustManagement System.Web. Security System.Message Bus.Security Authorization AccessControl Credentials Cryptography System.Web.Configuration System.MessageBus.Configuration System.Configuration System.Resources System.Management System.Deployment System.Diagnostics ConfigurationDeployment/Management System.Windows System.Windows.Forms System.Console System.ServiceProcess System.Windows.Forms System.Web System.Storage System.Data.SqlServer Animation Controls Control Design Panel Controls Dialogs SideBar Notification System.Windows Documents Text Element Shapes Shape Ink UI Element Explorer Media System.Windows.Forms Forms Control Print Dialog Design System.Web.UI Page Control HtmlControls MobileControls WebControls Adaptors Design Ports InteropServices System.Runtime System.IO System.Collections Generic System.Search Annotations Monitoring Logging Relevance System.Data SqlClient SqlTypes SqlXML OdbcClient OleDbClient OracleClient Core Contact Location Message Document Event System.Storage System.Web Personalization Caching SessionState System.Xml Schema Serialization Xpath Query Permissions Policy Principal Token System.Security System.Collaboration RealTimeEndpoint TransientDataSession SignalingSession Media Activities HttpWebRequest FtpWebListener SslClientStream WebClient System.Net NetworkInformation Sockets Cache System.Web Administration Management Navigation Peer Group Policy Serialization CompilerServices Recognition System.Speech Synthesis System.Windows.Input WinFx

4 No Longer Just a Desktop World Keyboard, Mouse not enough anymore Innovation is accelerating Form Factors Input Devices Natural Language Technologies International requirements are great Asian markets are fastest growing World markets continuing to globalize Keyboard, Mouse not enough anymore Innovation is accelerating Form Factors Input Devices Natural Language Technologies International requirements are great Asian markets are fastest growing World markets continuing to globalize

5 New Forms, Devices, Scenarios Media Center PC Tablet PC Smart Phone Smart Display Pocket PC

6 New Input Methods Tablet and Ink Shipping 3 rd generation of Tablet technologies Speech Built-in dictation and commanding support. Managed Speech API for extensibility and customization Keyboard A suite of keyboard services exploiting new input framework New input devices Including new navigation hardware, mice, keyboards, microphones, etc. Tablet and Ink Shipping 3 rd generation of Tablet technologies Speech Built-in dictation and commanding support. Managed Speech API for extensibility and customization Keyboard A suite of keyboard services exploiting new input framework New input devices Including new navigation hardware, mice, keyboards, microphones, etc.

7 Last ten years have seen a great deal of input innovation on Windows but…

8 MouseKeyboard DOS WinNLS PenWin Win3x C J IMM32SAPI AIMM TSFSAPI5 CUAS Ink SDK Win95 Win98 Win2k XP Tablet MSAA Single binary mouse wheel MouseKeyboard WinNLS PenWin C J K IMM32SAPI Windows key HID MSAA AIMM mouse wheel Single binary TSFSAPI5 WISP CUAS Ink SDK Windows Input API History

9 Our Goals for Avalon Promote Integration and Ease of Use Key functionality automatically enabled Fewer scenarios requiring device-specific code Promote Consistency Across Devices and Languages Promote rich devices, innovation Provide unprecedented language support Give developers (you) flexibility by making it extensible, pluggable Promote Integration and Ease of Use Key functionality automatically enabled Fewer scenarios requiring device-specific code Promote Consistency Across Devices and Languages Promote rich devices, innovation Provide unprecedented language support Give developers (you) flexibility by making it extensible, pluggable

10 Input Architecture Hwnd Source Raw Input stream Staging Area Filters - Mouse - Keyboard - Stylus - User-defined Monitors - Mouse - Keyboard - Stylus - User-defined Event Routing Win32 Queue Devices - Mouse - Keyboard - Stylus - User Defined

11 Event routing Tunneling First notify the grandparent, then the parent, then the child Bubbling Opposite of Tunneling - First notify the child, then the parent, then parent’s parent, etc. Typical pattern PreviewFoo event (tunnel) followed by Foo event (bubble) We often talk about these pairs as a single event Tunneling First notify the grandparent, then the parent, then the child Bubbling Opposite of Tunneling - First notify the child, then the parent, then parent’s parent, etc. Typical pattern PreviewFoo event (tunnel) followed by Foo event (bubble) We often talk about these pairs as a single event

12 Events – Routing Examples Spacebar Tab in button Tab in edit Alt-F Unrecognized key Spacebar Tab in button Tab in edit Alt-F Unrecognized key Window Handled Not Handled Panel Button Edit

13 Event Promotion Route low-level events first, higher level events next Retain ability for developers who want to get to the device events, keep it simple for those who don’t Route low-level events first, higher level events next Retain ability for developers who want to get to the device events, keep it simple for those who don’t

14 Basic Event Handlers

15 Tunnel Handler

16 Stylus & Mouse Handlers

17 Text Input - Integrated TSF Support TSF is an API particularly for input methods requiring access to the document (for context, composition, conversion and correction purposes) such as IMEs, Speech, Handwriting and others. Avalon TSF Support consists of TextBox implements the text store APIs A TextInput event for controls that don't implement their own text store TSF is an API particularly for input methods requiring access to the document (for context, composition, conversion and correction purposes) such as IMEs, Speech, Handwriting and others. Avalon TSF Support consists of TextBox implements the text store APIs A TextInput event for controls that don't implement their own text store

18 TextInput Event Like other input events in Avalon, Text input events follow the low-level event  promoted event pattern Avalon will first route the appropriate KeyDown/KeyUp events, and if those are not handled, then a TextInput event.  KeyDown Event KeyEventArgs.Key == Key.TextInput KeyEventArgs.Key == Key.ImeProcessed  TextInput Event TextInput event is device and language independent Like other input events in Avalon, Text input events follow the low-level event  promoted event pattern Avalon will first route the appropriate KeyDown/KeyUp events, and if those are not handled, then a TextInput event.  KeyDown Event KeyEventArgs.Key == Key.TextInput KeyEventArgs.Key == Key.ImeProcessed  TextInput Event TextInput event is device and language independent

19 Integrated Text Input

20 The Open Command

21 Commanding Cut, Copy, Paste, Open – these are commands A command is a form of input The “Invoke” event Command Invoke is often the result of other input Eg, ctrl-C invokes Copy command “ctrl-C = Copy” is an input binding Can also call RaiseCommand(…) directly Cut, Copy, Paste, Open – these are commands A command is a form of input The “Invoke” event Command Invoke is often the result of other input Eg, ctrl-C invokes Copy command “ctrl-C = Copy” is an input binding Can also call RaiseCommand(…) directly

22 Benefits of Commands Centralized handling logic Manage command status (eg, disabling menu items) Leverage existing commands Handle more input devices Command routing Finding someone to handle the command invoked Centralized handling logic Manage command status (eg, disabling menu items) Leverage existing commands Handle more input devices Command routing Finding someone to handle the command invoked

23 Command Library A default set of common commands like Cut, Copy, Paste, Open, etc Or define your own command Each standard command includes: Key bindings Icon Pen gesture Speech info Future devices A default set of common commands like Cut, Copy, Paste, Open, etc Or define your own command Each standard command includes: Key bindings Icon Pen gesture Speech info Future devices

24 Speech Functionality Built In Speech Recognition engine User configuration, Training Microphone control + Feedback UI Command and Control For all on screen elements (buttons, links etc.) For starting or switching applications Commanding Control Pattern Dictation Dictate into any text field Dictation commands to correct/edit text Standard Correction User Experience Speech Recognition engine User configuration, Training Microphone control + Feedback UI Command and Control For all on screen elements (buttons, links etc.) For starting or switching applications Commanding Control Pattern Dictation Dictate into any text field Dictation commands to correct/edit text Standard Correction User Experience

25 Extending Speech Support Strengths of speech as an input modality Can refer to things that are not on the screen “Start Microsoft Excell” Can be parameterized “Schedule a meeting tomorrow at 2” Efficient “Schedule an hour long meeting with my staff and Joe tomorrow afternoon at 2 in the main conference room” Strengths of speech as an input modality Can refer to things that are not on the screen “Start Microsoft Excell” Can be parameterized “Schedule a meeting tomorrow at 2” Efficient “Schedule an hour long meeting with my staff and Joe tomorrow afternoon at 2 in the main conference room”

26 System.Speech namespace Desktop-related functionality of COM API (SAPI) exposed in managed code Object-centric rather than interface-centric Additions so application can easily integrate into the system-wide speech user experience Dynamic grammar generation using W3C’s SRGS-derived object model SDK sample that demonstrates main functionality including semantic properties and recognition alternates Desktop-related functionality of COM API (SAPI) exposed in managed code Object-centric rather than interface-centric Additions so application can easily integrate into the system-wide speech user experience Dynamic grammar generation using W3C’s SRGS-derived object model SDK sample that demonstrates main functionality including semantic properties and recognition alternates

27 Extending Speech

28 New Device (“XEEL”) View full specifications at

29 XEEL Navigation

30 Summary Avalon Input Simple Consistent Extensible Avalon Input Simple Consistent Extensible

31 Community Resources Get Your Questions Answered! Client Lounge: middle of the Exhibit Hall Connect with Microsoft client product teams, and PDC 2003 Speakers Ask The Experts Tuesday 7pm – 9pm in Hall G, H Newsgroups: Microsoft.public.windows.developer.winfx.avalon PDC Weblogs Longhorn DevCenter: Now! Questions? Comments? Client Lounge: middle of the Exhibit Hall Connect with Microsoft client product teams, and PDC 2003 Speakers Ask The Experts Tuesday 7pm – 9pm in Hall G, H Newsgroups: Microsoft.public.windows.developer.winfx.avalon PDC Weblogs Longhorn DevCenter: Now! Questions? Comments?

32 © Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.