1 Graphical User Interfaces Graphical User Interfaces (GUIs) have become an important and accepted way of interacting with today's software. Although they make software easy to use from a user's perspective, they complicate the software development process. In particular, testing GUIs is more complex than testing conventional software, for not only does the underlying software have to be tested but the GUI itself must be exercised and tested to check whether it confirms to the GUI's specifications.

2 Correctness of a GUI Testing the correctness of a GUI is difficult for a number of reasons. First of all, the space of possible interactions with a GUI is enormous, in that each sequence of GUI commands can result in a different state and a GUI command may need to be evaluated in all of these states. The large number of possible states results in a large number of input permutations requiring extensive testing, e.g., Microsoft released almost 400,000 beta copies of Windows95 targeted at finding program failures

3 Problem of coverage For conventional software, coverage is measured using the amount and type of underlying code exercised. These measures do not work well for GUI testing, because what matters is not only how much of the code is tested, but in how many different possible states of the software each piece of code is tested. An important aspect of GUI testing is verification of its state at each step of test case execution. An incorrect GUI state can lead to an unexpected screen, making further execution of the test case useless since events in the test case may not match the correspond- ing GUI components on the screen.

4 GUI Underlying Code Research Focus Interactions between the GUI and the Underlying Code

5 GUIs are Event Driven Individual User Events –NOT ENOUGH ! –Sequences of User Events lead to Different States Test Case: Sequence of User Events How to Generate Test Cases ? Use Planning to Select Likely Test Cases Why Planning for GUI Testing

6 Infinitely Many Randomly Choose Sequences Expert Chooses Sequences Automatically Generate Events for COMMONLY USED TASKS Selecting Test Sequences Initial State Goal State Multiple Event Sequences This is the text.

7 Generation of test cases An important component of testing is the generation of test cases. Manual creation of test cases and their evaluation, and conformance to coverage criteria are very time consuming. Thus, some automation is necessary when testing GUIs. We present a new technique to automatically generate test cases for GUI systems. The key idea is that the test designer is likely to have a good idea of the possible goals of a GUI user and it is simpler and more effective to specify these goals than to specify sequences of events that the user might employ to achieve them.

8 A Plan for a GUI Task SelectText(“This”) SelectText(“text”) SetFontSize(18) MouseClick( U ) Initial State This is the text. Goal State This is the text. is the This text. is the This text.

9 Outline Using Planning for Test Case Generation –Overall Approach –Exploiting GUI Structure –Generating Alternative Test Cases Experimental Results Related Research Concluding Remarks

10 Straightforward Approach Define One Operator for each User Action Operator :: CUT Preconditions: isCurrent(Menu2). Effects: FORALL Obj in Objects Selected(Obj)  ADD inClipboard(Obj) DEL onScreen(Obj) DEL Selected(Obj) ADD isCurrent(Menu1) DEL isCurrent(Menu2). Menu2 Menu1 File Edit View Ins Cut

11 Exploit the GUI’s Structure Reduce the Number of Operators –System more Efficient –Easier for the Test Designer

12 Opening Modal Windows English (United States) OK Cancel Default... Set Language

13 Opening Menus File Send To Mail Recipient

14 Interacting with the Underlying Software Underlying Software Edit Copy

15 Create Hierarchical Operators Two Types of Abstractions –Combine Buttons  Create System- Interaction Operators –Decompose GUI Hierarchically  Create Abstract Operators

16 Translation A high-level plan must be translated into primitive GUI events. The translation process makes use of the operator- event mappings stored during the modeling process.

17 Create System-Interaction Operators File_SendTo_MailRecipient = Sys-Interaction Operator: File Send To Mail Recipient

18 Create Abstract Operators SelectFromList() Default OK Cancel Language Window’s Operator Set Main GUI’s Operator Set … Set Language SelectFromList() Default OK Cancel... English (United States) OK Cancel Default... Straightforward Approach Set Language … Set Language... Main GUI’s Operator Set Using Abstraction

19 Create Abstract Operators SetLanguage() SelectFromList (“English(US)”) OK Abstract Operator Planner SelectFromList() Default OK Cancel Language Window’s Operator Set SetLanguage()... High Level Plan Sub Plan Define Abstraction

20 Effects of Exploiting the GUI’s Structure Reduction in Planning Operators –325 operators  32 operators –Ratio 10:1 for MS WordPad –20:1 for MS Word System Automatically Determines the System-interaction and Abstract Operators

21 Initial State Goal State This is the text.

22 Test Case SelectText (“This”) FormatFont (“This”, 18pt) FormatFont (“text”, Underline) SelectText (“text”) Primitive Operator Primitive Operator Abstract Operator Abstract Operator INITIALINITIAL GOALGOAL

23 FormatFont18 OK FormatFontUnderlineOK Planner

24 FormatFont Format Font Mapping SelectText (“This”) Format Font18 OK SelectText (“text”) FormatFont Underline OK

25 Different from HTN Planning SelectText (“This”) FormatFont (“This”, 18pt) FormatFont (“text”, Underline) SelectText (“text”) Primitive Operator Primitive Operator Abstract Operator Abstract Operator INITIALINITIAL GOALGOAL FormatFont18 OK FormatFontUnderlineOK Planner No Interactions

26 Alternative Test Case SelectText (“This”) SelectFromList (18) FormatFont (“text”, Underline) SelectText (“text”) Primitive Operator Primitive Operator Primitive Operator Abstract Operator SelectText (“This”) SelectFromList (18) SelectText (“text”) Format Font Underline OK SelectText (“This”) FormatFont (“This”, 18pt) FormatFont (“text”, Underline) SelectText (“text”) Primitive Operator Primitive Operator Abstract Operator Abstract Operator INITIALINITIAL GOALGOAL

27 Methods to Generate Alternative Test Cases Different Results from Planner Abstract Operator Decompositions Linearizations of the Partial-order Plan

28 Feasibility Study Purpose –To Determine whether Planning is a Feasible Approach for GUI Test Case Generation Execution Time Human Effort Experimental Design –GUI: MS WordPad –Planner: IPP –Hardware Platform: 300 MHz Pentium based Machine, 200 MB RAM, Linux OS –8 Tasks, Multiple Test Cases for each Task [Koehler et al. ‘97]

29 Concluding Remarks Automatic Planning is a Feasible Approach for GUI Test Case Generation Automatic Generation of Preconditions and Effects from GUI Specifications Generate Expected Output (Automated Verification)

30 Coverage Criteria Two purposes –Test data selection criteria Rules used to select test cases –Test data adequacy criteria Rules used to determine how much testing has been done Common Examples for Conventional Software –Statement coverage –Branch coverage –Path coverage Structural Representation of the Code

31 Coverage Criteria for GUIs Cannot use code-based coverage –Source code not always available –Event-based input Different level of abstraction Our Contribution –Hierarchical structure of the GUI in terms of events –Coverage criteria based on events

32 GUI Definition Hierarchical Graphical Front-end Accepts User-generated and System- generated events Fixed sets of events Deterministic Output State of the GUI is the set of Objects and their Properties

33 GUI Representation Motivation –GUI testing needs a “Unit of Testing” Manageable Test the unit comprehensively Test interactions among units –GUIs are created using library elements Need to test these elements before packaging them for reuse –Certain level of confidence that the element has been adequately tested User of these elements should be able to test the element in its context of use

34 Model GUI Hierarchically Hierarchy –GUIs are decomposed into a hierarchy of components –Hierarchical decomposition makes testing intuitive and efficient –Several hierarchical views of GUIs –We examine Modal Dialogs to create the hierarchical model

35 Modal Windows in GUIs Main

36 Modal Windows in GUIs Main Print invokes

37 Modal Windows in GUIs Main Print Properties Components

38 Integration Tree Main Properties FileNewFileOpenPrintFormatFontFileSavePageSetupViewOptions Definition: Integration tree is a triple Definition: Integration tree is a triple N is the set of components in the GUIN is the set of components in the GUI R є N is a designated component called the Main componentR є N is a designated component called the Main component B is the set of directed edges showing the invokes relation between components, i.e., (C x, C y ) є B iff C x invokes C y.B is the set of directed edges showing the invokes relation between components, i.e., (C x, C y ) є B iff C x invokes C y.

39 Open … Save Representing a Component File EditHelp Open … Save Cut Copy Paste About … Contents … Event-flow Graph follows Definition: Event e x follows e y iff e x can be performed immediately after e y.

40 Event-flow Graph File Edit Help OpenSave CutCopyPaste AboutContents To File, Edit and Help To File, Edit and Help Definition: Event-flow graph is a 4-tuple Definition: Event-flow graph is a 4-tuple V is the set of vertices, representing events,V is the set of vertices, representing events, E is the set of directed edges, showing the follows relationship,E is the set of directed edges, showing the follows relationship, B is the set of events first available (shown in red),B is the set of events first available (shown in red), I is the set of events that invoke other components ().I is the set of events that invoke other components (dotted lines).

41 Classifying Events Opening modal windows –Restricted-focus events Closing modal windows –Termination events Opening modeless windows –Unrestricted-focus events Opening menus –Menu-open events Interacting with underlying software –System-interaction events ClassificationClassification –A new classification of events aids in creating the hierarchical model of the GUI

42 Coverage Criteria Intuitively –Each component is a unit of testing –Test events within each component Intra-component coverage criteria –Test events across components Inter-component coverage criteria

43 Coverage Criteria Intra-component Coverage –Event coverage Individual events Each node in the event-flow graph –Event-interaction coverage Each pair of events Each edge in the event-flow graph –Length-n event sequence coverage Sequences of events Bounded by length –Length-1 event sequences –Length-2, length-6 event sequences Paths in the event-flow graph

44 Coverage Criteria Inter-component Coverage –Invocation coverage Invoke each component Each restricted-focus event –Invocation-termination coverage Invoke each component and terminate it Restricted-focus event followed by a termination event –Inter-component length-n coverage Longer sequences from one component to another Bounded by length

45 Case Study Purpose –To determine: How many test cases do we need to test WordPad Correlation between event and code-based coverage Experimental design –GUI: our version of MS WordPad (36 modal windows, 362 events) –Hardware platform: 350 MHz Pentium based machine, 256 MB RAM

46 Test Cases for WordPad Results

47 Correlation between Event-based & Code-based Coverage Code Instrumentation Generated all event sequences up to length 3. Total test cases: 21,659 Executed all 21,659 cases and obtained execution traces Statement coverage

48 Correlation between Event-based & Code-based Coverage Results

49 Automated Execution Expected State Test Executor GUI Under Test Execution Monitor ACTUAL STATE: (isCurrent ROOT) (Contains ROOT D Verifier Test Cases

50 Comparing Actual and Expected States Verifier Three Levels of Testing –Changed Property Set (Operators) –GUI Relevant Property Set (Specifications) –Complete Property Set (Toolkit/Language) Hybrid Approach –Use all 3

51 Deriving Expected State Total CPU time (test case and expected state) sec.

52 Execution Relevant-properties verification Total running time < 10 minutes