1. “Avalon”: Using UI Automation to Improve Application Testability Robert Sinclair Group Manager Microsoft Corporation robsi@microsoft.com Robert Sinclair Group Manager Microsoft Corporation robsi@microsoft.com Session Code: CLI307

2. Agenda What is UI Automation? Demo: Windows UI Automation in action A comparison of approaches Overview of Windows UI Automation Sample Code Security Considerations Key Take-Aways What is UI Automation? Demo: Windows UI Automation in action A comparison of approaches Overview of Windows UI Automation Sample Code Security Considerations Key Take-Aways

3. Tools Client Application Model AvalonWindows Forms Web & Service Application Model ASP.NET / Indigo Win FS Compact Framework Yukon Mobile PC Optimized Data Systems Application Model Presentation Data Mobile PC & Devices Application Model Communication Command Line NT Service DataSet Mapping ObjectSpaces ObjectSpace Query Schema Item Relationship Media Audio Video Images System.Messaging System. Discovery System.DirectoryServices System.Remoting System.Runtime.Remoting Active Directory Uddi System.Web.Services Web.Service Description Discovery Protocols System.MessageBus Transport Port Channel Service Queue PubSub Router Fundamentals System.ComponentModel System.CodeDom System.Reflection System.EnterpriseServices System.Transactions Security System.Windows. TrustManagement System.Web. Security System.Message Bus.Security Authorization AccessControl Credentials Cryptography System.Web.Configuration System.MessageBus.Configuration System.Configuration System.Resources System.Management System.Deployment System.Diagnostics ConfigurationDeployment/Management System.Windows System.Windows.Forms System.Console System.ServiceProcess System.Windows.Forms System.Web System.Storage System.Data.SqlServer Animation Controls Control Design Panel Controls Dialogs SideBar Notification System.Windows Documents Text Element Shapes Shape Ink UI Element Explorer Media System.Windows.Forms Forms Control Print Dialog Design System.Web.UI Page Control HtmlControls MobileControls WebControls Adaptors Design System.Collections Generic System.Search Annotations Monitoring Logging Relevance System.Data SqlClient SqlTypes SqlXML OdbcClient OleDbClient OracleClient Core Contact Location Message Document Event System.Storage System.Web Personalization Caching SessionState System.Xml Schema Serialization Xpath Query Permissions Policy Principal Token System.Security System.Collaboration RealTimeEndpoint TransientDataSession SignalingSession Media Activities HttpWebRequest FtpWebListener SslClientStream WebClient System.Net NetworkInformation Sockets Cache System.Web Administration Management Navigation Peer Group Policy Recognition System.Speech Synthesis UI Automation System.Windows.Automation System.Timers System.Globalization System.Serialization System.Threading System.Text System.Design Ports InteropServices System.Runtime System.IO Serialization CompilerServices Base & Application Services System.Windows.Automation Provider InteropProvider

4. “UI Automation” defined Gather information about the UI Dynamically discover UI structure Extract property information Receive event notifications when UI changes Query an element for its behavior Interact with UI elements Click a button, scroll a list, move a window, etc. Inject keystrokes and mouse input Gather information about the UI Dynamically discover UI structure Extract property information Receive event notifications when UI changes Query an element for its behavior Interact with UI elements Click a button, scroll a list, move a window, etc. Inject keystrokes and mouse input – code that programmatically drives another application’s UI to yield a desired result

5. Windows UI Automation in Action Goals: Traverse the UI tree Query a control for its functionality Interact with the control Goals: Traverse the UI tree Query a control for its functionality Interact with the control

6. Application #2Application #1 Today’s Approach to Automation Custom 1 WinFormsSDMWin32 Custom 2 Text PanelWin32 Automation Utility Win32 Messages & APIs Win32 Messages & APIs SDM Object Model SDM Object Model Custom 1 APIs Custom 1 APIs … … Text Object Model Text Object Model Custom 2 APIs Custom 2 APIs WinForms Methods WinForms Methods

7. Interacting with a Control Comparison of Approaches Today – Invest in each control library Use each library’s proprietary APIs to: Traverse its tree Collect UI information Manipulate the controls At runtime: Test the control’s classname Call your library or control-specific routines Longhorn – Use Windows UI Automation Traverse UI structure using a single tree Base your app logic on control behaviors Obtain UI Automation security permissions Today – Invest in each control library Use each library’s proprietary APIs to: Traverse its tree Collect UI information Manipulate the controls At runtime: Test the control’s classname Call your library or control-specific routines Longhorn – Use Windows UI Automation Traverse UI structure using a single tree Base your app logic on control behaviors Obtain UI Automation security permissions

8. UI Automation Clients UI Automation Providers Windows UI Automation Core Longhorn Model: UI Automation Assistive Technology Products Scripting Utilities Testing Framework “Avalon” Implementation Control Proxy Custom Implementation …InteropProvider…Provider MSAvalon.Windows.Automation Legacy Control ISV Object Model Atuomated Test …InteropProvider

9. Windows UI Automation Automation framework built into Longhorn Platform-level support for automating all UI elements Avalon, WinForms, Win32, Visual Basic, etc. Exposes a consistent object model for all controls 3 rd party controls easily integrate into model Security Model – client must be trusted Locale, machine, and resolution independent Creates new opportunities for innovation in: Automated UI Testing Assistive Technology Products Command-and-Control Utilities Automation framework built into Longhorn Platform-level support for automating all UI elements Avalon, WinForms, Win32, Visual Basic, etc. Exposes a consistent object model for all controls 3 rd party controls easily integrate into model Security Model – client must be trusted Locale, machine, and resolution independent Creates new opportunities for innovation in: Automated UI Testing Assistive Technology Products Command-and-Control Utilities

10. UI Automation Overview Logical Tree – structure of the UI Stitches all UI trees into one coherent structure Eliminates unnecessary elements Resembles the structure perceived by an end user Properties – important UI information Name, Bounding Rectangle, Persistent ID, etc. Events – notification of UI changes Window creation, change in focus or selection, etc Control Patterns – control behavior Scroll, Selection, Window, ExpandCollapse, etc. Input – simple mouse and keyboard input Logical Tree – structure of the UI Stitches all UI trees into one coherent structure Eliminates unnecessary elements Resembles the structure perceived by an end user Properties – important UI information Name, Bounding Rectangle, Persistent ID, etc. Events – notification of UI changes Window creation, change in focus or selection, etc Control Patterns – control behavior Scroll, Selection, Window, ExpandCollapse, etc. Input – simple mouse and keyboard input

11. UI Automation Control Patterns Mutually exclusive classes of control behavior Control developers (providers) expose these patterns for new or existing controls Automation developers (clients) use patterns to Discover what functionality a control offers Gather pattern-specific property information Receive pattern-specific events Programmatically manipulate the control Examples: Button: Invoke ListBox: Scroll, Selection ComboBox: Scroll, Selection, ExpandCollapse Mutually exclusive classes of control behavior Control developers (providers) expose these patterns for new or existing controls Automation developers (clients) use patterns to Discover what functionality a control offers Gather pattern-specific property information Receive pattern-specific events Programmatically manipulate the control Examples: Button: Invoke ListBox: Scroll, Selection ComboBox: Scroll, Selection, ExpandCollapse

12. Expand Collapse Pattern Goals: Traverse Logical Element tree Query for supported patterns Find an element that supports ExpandCollapse Manipulate the control Goals: Traverse Logical Element tree Query for supported patterns Find an element that supports ExpandCollapse Manipulate the control

13. Security Model No default automation permissions UI Automation functionality is protected according to the following permissions: Read – Navigate tree, get properties, receive events Write – Call control pattern methods Input – Call methods in the Input class Access to Rights Managed requires additional permissions No default automation permissions UI Automation functionality is protected according to the following permissions: Read – Navigate tree, get properties, receive events Write – Call control pattern methods Input – Call methods in the Input class Access to Rights Managed requires additional permissions

14. 3 Key Take-aways  Platform-level support for automating UI  Control independence Simplifies your client logic Means less code to write and maintain  New opportunities Integrate additional control libraries Expose custom object models, properties, events, etc. Extend the UI Automation framework  Platform-level support for automating UI  Control independence Simplifies your client logic Means less code to write and maintain  New opportunities Integrate additional control libraries Expose custom object models, properties, events, etc. Extend the UI Automation framework

15. Call to Action Next 30 days Study System.Windows.Automation.* Spend a day playing with UI Automation Submit feedback on client or provider side Longhorn Beta 1 Providers: Implement UI Automation for your custom controls Clients: Use UI Automation to develop your automation clients Longhorn RTM Ship your fully-enabled UI Automation providers and clients!! Next 30 days Study System.Windows.Automation.* Spend a day playing with UI Automation Submit feedback on client or provider side Longhorn Beta 1 Providers: Implement UI Automation for your custom controls Clients: Use UI Automation to develop your automation clients Longhorn RTM Ship your fully-enabled UI Automation providers and clients!!

16. Question & Answer Join me for further discussion in the Client Track Lounge Please fill out an on-line session evaluation on COMMNET Thank You for Coming! Join me for further discussion in the Client Track Lounge Please fill out an on-line session evaluation on COMMNET Thank You for Coming!

17. © 2003-2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.