Hierarchical GUI Test Case Generation Using Automated Planning Atif M. Memon, Student Member, IEEE, Martha E. Pollack, and Mary Lou Soffa, Member, IEEE

Introduction Graphical User Interfaces – important in interacting with today’s software Makes software easier to use Makes software testing more difficult than conventional software - must find a PATH from some initial state to a solution

Consists of labels, buttons, menus, and pop-up lists User interacts with components to create events What is a GUI? GUI Window Object Window Microsoft WordPad

Challenges in Testing GUI Applications GUI’s becoming more complex  Complexity in testing correctness and underlying software Enormous amounts of possible interaction Determining coverage of Test Cases Regression testing Goal: Automatic generation of test cases Solution: P lanning A ssisted T ester for grap H ical U ser interface S ystems

State of the Art A new technique both scalable and partially automated that reduces dependence on user expertise ProsCons Mimic novice users Not scalable Scalable Semi-automated Test case quality depends on user expertise Automated test case generation Finite state machine models

35,000 foot View of the Mountain Performs hierarchical structure of GUI  create hierarchical operators Test designers describe operators(preconditions, effects)‏ Takes goals as input and generates multiple sequences of events (“plans”) automatically Plans becomes the test cases for GUI PATHS… …easier to specify goals than all the possible PATHS toward the goal

PATHS: AI Planning with Operators Operator (preconditions, effects)‏ Given to the AI planning algorithm Guaranteed to result in the goal state when executed in the initial state Initial StateGoal State

The PATHS Test Case Generation Process Setup Phase – PATHS creates a hierarchical model of the GUI and returns list of operators from the model to test designer Design preconditions and effects in simple, planning system language Plan Generation Phase – Test designer describes scenarios by defining a set of initial and goal states for test case generation PATHS generates a test suite for test case generation

Phase 1: Setup PATHS partitions GUI into several classes using structural properties. Menu-open Events – expand set of GUI events available to user Unrestricted-focus Events – open GUI windows that do not restrict user’s focus, expand set of GUI events available to user (ex. Toolbar) Restricted-focus Events – open GUI windows that have special property once invoked, monopolize GUI interaction, restrict user view of GUI to specific range until window is terminate (ex. Pop-up window) System-interaction Events – interact with underlying software to perform some action

Phase 1: Operator Types System-Interaction Operators – derived from those GUI events that generate interactions with the underlying software Ex. Prefix_Suffix  File_Open = Abstract Operators – represents a sequence of GUI events that invoke a window which monopolizes the GUI interaction Treat this restricted-focus window as a separate planning problem and decompose

Phase 1: Setup Example GUI Events File + Edit + New Open* Save Save As* Cut Copy Paste Open.Up Open.Select Open.Cancel Open.Open SaveAs.Up SaveAs.Select SaveAs.Cancel SaveAs.Save Planning Operators File_ File_ File_ File_ Edit_ Edit_ Edit_ New Open* Save SaveAs* Cut Copy Paste

Phase 1: Operator Event Mapping Transforms GUI menu events into planning operators These results Returned as Planning operators to the Designer …who then specifies preconditions and effects for each planning operator

Phase 2: Test Case Generation Test case designer inputs defined operators into PATHS and identifies a task (initial state, goal state)‏ Planning Operators PATHS Defined Task Tests Cases Achieving Goal

Abstract Operator Decomposition Operators from the higher level need to be decomposed into lower level operations by Inserting expansion from operator-event mappings, or Making an additional call to the planner Hierarchical mechanism aids in regression testing since changes made to one component to do not necessarily invalidate all test cases.

Plan Generation: AI Planning Problem Planning Problem – P( Λ, D, I G ) P(Operator Set, Finite Set of Objects, Initial State, Goal State)‏ Solution – S i < S j – Step i must occur before Step j (not necessarily right before)‏ Causal Links -- Ensure no step threatens a required link

Partial Order Plans… …aids in deriving total order plan Add ordering constraints i.e. Linearization of partial-order plan … are solutions to a planning problem iff every consistent linearization of the partial-order plan meets the solution conditions Want to learn more about how the Authors implemented AI Planning…? We invite you to read about it in the paper. (IPP and HTN)‏

Evaluation: Empirical Approach A PATHS prototype was developed based Microsoft WordPad 325 GUI events Reduced to 32 system-interaction and abstract operators Evaluation: Empirical Approach 10-fold reduction of operators streamline plan generation (phase 2 of PATHS)‏

Evaluation: Run Two Experiments Two experiments were run using the PATHS prototype Experiment 1: Generate test cases for multiple tasks Is PATHS practical and efficient? Experiment 2: Compare hierarchical vs. single- level test case generation Is the hierarchical modeling approach more efficient than single-level?

Experiment 1: Generating Test Cases for Multiple Tasks Use PATHS prototype to generate a suite of test cases for 9 WordPad tasks Deploy experiment on Pentium-based computer running Linux on 200Mb RAM Track times required to generate high-level plans and subplans PATHS is practical and efficient

Experiment 2: Hierarchical vs. Single-Level Test Case Generation Use PATHS prototype to generate test cases for 6 WordPad tasks Deploy experiment on same computer (Pentium-based running Linux on 200Mb RAM)‏ Track plan lengths Track times required to generate a single-level and a hierarchical test case Hierarchical test case generation is more efficient

Conclusions Applying AI planning to the problem is a good match Hierarchy of the model is efficient in generating test cases 4- to 6-fold saving in plan length Test case generated in seconds rather than hours New technique is scalable when software releases new version, operators can be recycled New technique is highly automated

PATHS: Contributions Use well-known and used AI planning technique capable of solving large-space problems Exploits structural features present in GUIs to reduce model size & complexity, and efficiently generate test cases Hierarchical Planning makes regression testing easier Increased portability – platform specific details incorporated last in test case generation Reuse of commonly-appearing operator definitions

Open Questions PATHS is practical, but is it useful? Preliminary exploration achieves 20-fold reduction in GUI events How does one measure usefulness? How well does PATHS scale? Manual analysis of operators will vary from application to application How good are the test cases generated by PATHS? Coverage Negative test cases

Related / Future Work Use library of operators to minimize human intervention How to automatically generate preconditions and effects of operators