1 Developing for Test Automation and Accessibility Using Programmatic Access to the UI Thomas Logan FUN307 Program Manager Microsoft Corporation

2 Key Take-Aways UI Automation Specifies a Universal Object Model for the GUI Implementing UI Automation Provides the Dual Benefit of Test Automation and Accessibility Control Patterns Fully Expose the Functionality of Complex UI

3 Programmatic Access Programmatic access enables developers to write code that; Navigates between UI elements Gathers information about the UI Interacts with UI elements Receives notifications when the UI changes

4 UI Automation (UIA) Programmatic access component of the new accessibility model for Windows Design and development based upon test automation and assistive technology vendor requirements. Creates a universal object model for accessing UI cross-process Successor to Microsoft Active Accessibility (MSAA) Supported on Windows Vista, XP and Server 2003

5 Architecture Client-side API (.NET Managed API) (.NET Managed API) ClientProvider UIA Infrastructure UIA Provider-side interface definitions (.NET Managed API or C++) Test Automation and Assistive Technology Manufacturers Application and Control Developers

6 Key Components of UIA Automation tree – Provides navigation of the UI Automation Element – Represents any piece of the UI Properties – Important UI information Control patterns – Allows interaction with UI elements Allows interaction with UI elements Events – Notifications of changes in UI

7 demo Automating the UI

8 Control Complexity Button Dial List HTML 1 Level N Level

9 Automation Tree Provides unified navigation across UI framework trees of elements. Root node is desktop HWND tree is building block of Automation tree Control View - default view of automation tree

10 Automation Elements An automation element represents a piece of UI, regardless of underlying implementation (Win32, WinForms, Windows Presentation Foundation) Can be obtained Using screen pixel location From focused element From an Hwnd Through navigating the UI

11 Properties Allow clients to gather information about a UI element that is important for programmatic access. UIA Control Type UI Framework Framework Specific Property UI Automation Property Button Windows Presentation Foundation ContentNameProperty ButtonWin32CaptionNameProperty ImageTrident/HTMLALTNameProperty Table: Mapping one standard UIA property to multiple property names in other UI frameworks.

12 Three Key Properties AutomationIdProperty String that uniquely identifies an element among its siblings in the tree ControlTypeProperty Primary identifier for what type of control the automation element represents. NameProperty String of text that a user would use to explain which control was being referred to

13 Steps For Automating The UI Add references to client automation dlls Use System.Windows.Automation namespace Create Condition to match specific element in tree Issue Find command to get reference to element in tree Request Control Pattern from element in tree to perform operation Use information contained on Control Pattern object to control UI

14 Provider Interfaces IRawElementProviderSimple (Patterns and Properties) IRawElementProviderFragment (… adds Structure) IRawElementProviderFragmentRoot (… adds Focus and Hit Testing)

15 Tree Integration Create proxy/adaptor for control to expose control’s underlying object model Handle WM_GETOBJECT Desktop WM_GETOBJECT get_HostRawElementProvider UIA Treats both providers as a single node Merges properties Aggregates children

16 Making a Custom Win32 Control Programmatically Accessible using UIA demo

17 Steps For Provider Creation Include UIAutomationCore.h and UIAutomationCoreAPI.h Create proxy/adaptor class to hold automation/accessibility specific code for control Initialize Automation Identifiers Implement IRawElementProviderSimple interface get_ProviderOptionsget_HostRawElementProviderGetPropertyValue(…)GetPatternProvider(…) Return COM Object in Query Interface Handle WM_GETOBJECT message

18 Control Patterns Describe a particular aspect of a control’s functionality. (Defined to consist of one or more of the following components.) Properties – associated with the pattern’s functionality associated with the pattern’s functionality Methods – allow the control to be manipulated allow the control to be manipulated Events – associated with the pattern’s functionality Structure – parent/child and sibling relationships parent/child and sibling relationships

19 Patterns Associated with ControlType ControlType Associated Control Pattern(s) ButtonInvoke List Selection, Scroll ComboBox Selection, ExpandCollapse, Value TreeItem SelectionItem, ExpandCollapse Edit Text, Value

20 Control Patterns FunctionalityControlPattern Ability to be checked/unchecked Toggle Ability to have numeric value set RangeValue Ability to have text value set Value Ability to be moved / resized Transform Ability to show or hide information ExpandCollapse Implement the appropriate interface based upon the actions a control provides through mouse/keyboard

21 Control Pattern Implementation Implement IRangeValueProvider on proxy class get_Minimumget_Maximumget_LargeChangeget_SmallChangeget_IsReadOnlyget_ValueSetValue(…) Return COM Object in QueryInterface Respond to PatternId request in GetPatternProvider

22 Events Enable clients to be notified when changes occur in the UI Where applicable events are always associated with a reference to automation element where change occurred Clients can filter what parts of automation tree to listen for events on Providers raise events after the UI has been updated

23 Events demo

24 Recap UI Automation Specifies a Universal Object Model for GUI Implementing UI Automation Provides the Dual Benefit of Test Automation and Accessibility Control Patterns Fully Expose the Functionality of Complex UI

25 Community Resources WinFX Software Development Kit: Microsoft Accessibility home page: Please fill out an Evaluation at the Kiosks outside.

26 Appendix

27 Control Patterns DockPatternExpandCollapsePatternGridPatternGridItemPatternInvokePatternMultipleViewPatternRangeValuePatternScrollPatternScrollItemPattern SelectionPatternSelectionItemPatternTablePatternTableItemPatternTextPatternTogglePatternTransformPatternValuePatternWindowPattern