GUI Testing Phan Thế Vinh Nguyễn Quang Nhật –

Outline What is GUI? -> GUI Testing? GUI Testing types GUI Testing approaches GUI Testing oracles GUI Testing tools 2

G.U.I – Graphical User Interface How programs interacts with user How the user actions are performed to access application's features 3

G.U.I – Graphical User Interface Is a hierarchical, graphical front end to a software system During the execution  The GUI state: {… (w, p, v), …} w: widgets (graphical object) p: set of properties v: discrete value 4

5 GUI Testing Whether application responds in functionally correct manner Input Program GUI Correct? Expected = Obtained?

6 GUI Testing  Non-GUI testing: suites are composed of test cases that invoke methods of the system and catch the return values;  GUI testing: suites are composed of test cases that are:  able to identify the components of a GUI;  able to exercise GUI events (mouse clicks,…);  able to provide inputs to the GUI components (filling text fields,…);  able to test the functionality underlying a GUI set of components;  able to check the GUI representations  often, strongly dependent on the used technology;

7 GUI Testing : Type System testing  Test the whole system Acceptance testing  Accept the system Regression testing  Test the system changes

8 Acceptance Testing Acceptance Tests are specified by the customer and analyst to test that the overall system is functioning as required (Do developers build the right system?). Manual Acceptance testing: User exercises manually Acceptance testing with “GUI Test Drivers” (at the GUI level). Tools help the developer do acceptance testing through a user interface Table-based acceptance testing. Customer enters in a table the expectations of the program’s behaviour Black-Box (aka functional) approaches Define test specification then executed manually, by means of the GUI or by table-based testing.

9 Approaches for GUI Testing Manual Based on the domain and application knowledge of the tester Capture and Replay Based on capture and replay of user sessions Model testing Based on the execution of user sessions selected from a model of the GUI  Which type of model to use?  Event-based model  State-based model  Domain model  How do obtain the model to be used?  Specification-based model  Model recovered from existing software systems  Log-based model

10 Event-based Model Model the space of GUI event interactions as a graph Given a GUI: 1.create a graph model of all the possible sequences that a user can execute 2.use the model to generate event sequences

11 Event-based Model File NewOpenSave … Edit Cut CopyPasteReplaceUndo … Editbox0 Editbox1 Match case Find nextReplace Replace all Cancel Top level TC: Oracle: & !CRASH “Event-flow graph”

12 Event-based Model Model Type:  Complete event-model  Partial event-model Event types:  Structural events (Edit, Replace)  Termination events (Ok, cancel)  System interaction events (Editbox0, Find next) Coverage criteria  Event coverage  Event coverage according the exercised functionality  Coverage of semantically interactive events  2-way, 3-way coverage  ….

13 State-based Model Model the space of GUI event interactions as a state model, e.g., by using a finite state machine (FSM): States are screenshot/representation of the GUI Transitions are GUI events that change the GUI state Given a GUI: 1.create a FSM of the possible sequences that a user can execute, considering the GUI state 2.use the FSM to generate event sequences e1 e2 e3 e4 e5

14 State-based Model FSM: - State: GUI screenshot -Transition: event that changes the GUI state

15..example Problem: state explosion!  Use of a state abstraction function It maps concrete states into abstract states (i.e., sets of concrete states)

16 Log-based Recovered Model How do obtain the model?  starting from system specification or requirements  starting from the system (i.e., reverse engineering) 1. trace some system executions (at method calls level) 2. infer a model 3. refine it manually, if needed

17 Test oracles for GUI-based testing It could be difficult to detect faults looking the GUI  Crash testing is often used; In a GUI test case, an incorrect GUI states can take the user to an unexpected/wrong interface screen or it can make the user unable to do a specific action;  e.g., after the click of a button, we try to click the button again but we fail since the button no longer exists, after the first click. A GUI state can be “represented” by the components expected to be part of the GUI in a give time and their state/value e.g., window position, GUI objects, GUI title, GUI content, GUI screenshots,

18 GUI errors: examples Incorrect functioning Missing commands (e.g., GUI events) Incorrect GUI screenshots/states The absence of mandatory UI components (e.g., text fields and buttons) Incorrect default values for fields or UI objects Data validation errors Incorrect messages to the user, after errors Wrong UI construction ….

19 GUI-based Testing: process 1. Identify the testing objective by defining a coverage criteria 2. Generate test cases from GUI structure, specification, model  Generate sequences of GUI events  Complete them with inputs and expected oracles 3. Define executable test cases 4. Run them and check the results

20 GUI-based Regression Testing GUI-based testing means to execute the GUI of a system exercising its GUI components; A small changes in the GUI layout can make the GUI test cases old and useless; Hence, GUI-based test suite need to be maintained and often chagned  supporting tools are welcomed Often, GUIs are realized by means of rapid prototyping or automatic framework. This requires an efficient approach to generate and maintain GUI test suite  supporting tools are welcomed

21 Capture and Replay A capture and replay testing tool captures user sessions (user inputs and events) and store them in scripts (one per session) suitable to be used to replay the user session. An ad-hoc infrastructure is needed to intercept GUI events, GUI states, thus storing user sessions and also to be able to replay them. - they can work at application or VM level

22 Recorded information Inputs, outputs, and other information needed to replay a user session need to be recorded during the capture process. Examples:  General information: date/time of recording, etc.  System start-up information  Events from test tool to system  Point of control, event  Events from system to test tool  Checkpoints / expected outputs  Time stamps

23 Capture and Replay: the process 1. The tester interacts with the system GUI to run the system, thus generating sessions of sequence of mouse clicks, UI and keyboard events; 2. The tool captures and stores the user events and the GUI screenshots; a script is produced per each user session 3. The tester can automatically replay the execution by running the script the script can be also changed by the tester the script can be enriched with expected output, checkpoints the script can be replicated to generate many variants (e.g., changing the input values) 4. In case of GUI changes, the script must be updated

24 Marathon Abbot Guitar HtmlUnit, HttpUnit,JWebUnit HtmlFixture Selenium …. Tools for GUI-based testing Java Web

25 Calc_1 : Logic mixed to GUI GUI realized by using Swing Calc_2 : Logic mixed to GUI GUI realized by using AWT Calc_3 : Logic separated from the GUI GUI realized by using AWT +Swing Running Example: Calculator These apps are from the net hence please refer to the author’s copyright

26  Is a tool that helps writing System/GUI tests for Java/Swing applications  Consists of a recorder, player, and an editor  Records tests script in Python It is possible to use it without knowing Python …  Allows to insert assertions in the script easily Using a specific GUI  Shows testing results using JUnit’s control bar Red/green  download Marathon and its user manual Marathon

27 Marathon example (1) A calculator application was developed (Swing) We want to write tests for the calculator  Ex. add two values Steps: 1. create a new Marathon project 2. create a new empty test (a test is a Python script file) 3. build the test (recording)

28 Marathon example (2) Record the script click on the record button in the toolbar – the red circle enter “ =”, the result is displayed (16) press control + right-click in the text area, to open the contextual menu select Assert Text stop recording (the button with a red square), the script now looks like: save the script

29 Marathon example (3) Run the script Select the “JUnit” tab Click the test view, you should see your new created test Click on the “Run all tests” button, in the JUnit’s control bar and … get the green bar …

30 Abbot  Is a tool that helps writing System/GUI tests for Java AWT/Swing applications  Consists of a recorder, player, and an editor (via Costello, built on top of Abbot)  Records tests script in Java  Allows to write test cases directly from Java code (named programmatic GUI testing)  Allows to insert assertions in the script easily  Shows testing results using JUnit’s control bar  Red/green  Two main building blocks:  ComponentReferences to get a handle on a GUI component  Robot: to perform user-level actions on various GUI components  download Marathon and its user manual

31 Abbot example (1) A calculator application was developed (AWT/Swing) We want to write tests for the calculator  Ex. add two values Steps: 1.create a new empty Abbot test 2.Initialize the test with the application under test information 3. build the test (recording)

32 Abbot example (2) Recording 1. Start recording of all actions 2. Execute the software 3. Add assertions 1. select the GUI element with SHIFT+F1 2. select the property (e.g., (“Text”) 3. set the expected value 4. Stop recording

33 Guitar A testing framework implementing the “Event- flow graph” approach Four main components: The GUIRipper: extract GUI information from a program The GUIStructure2Graph: build a traversable graph representation of the GUI elements The TestCaseGenerator: create an extensive set of test cases based on the graph The GUIReplayer: run the program as instructed by these tests  It is a research tool…  It requires ANT to be executed

34 Guitar (1) Four main components: (1) GUIRipper: GUI information extraction >ant -Dproperties=jfcripper.properties -f jfcripper.xml (2) GUIStructure2Graph : Event-flow graph inference >ant -v -f GUIStructure2GraphConvert.xml

35 (3) TestCaseGenerator : Test cases generation traversing the graph >ant -v -f TestCaseGenerator.xml (4) GUIReplayer : Run the program GUI according to the generated test cases >ant -Dproperties=jfcreplayer.properties -f jfcreplayer.xml Guitar (2)

36 HtmlUnit, HttpUnit, JWebUnit HtmlFixture Selenium …. Tools for other GUI-based testing of Java

37 XUnit Web tools HtmlUnit, HTTPUnit, JWebUnit, etc. - They are inspired by JUnit - They let us to verify the functionalities of Web sites - They emulate the relevant portions of browser behavior form submission, JavaScript, http authentication, cookies, automatic page redirection, etc. - They allow Java test code to examine returned pages in terms of text, DOM, or containers of forms, tables, and links - They make it easy to write Junit tests that verify HTTP requests and responses

38 … HttpUnit example import junit.framework.TestCase; import com.meterware.httpunit.WebResponse; import com.meterware.httpunit.WebConversation; import com.meterware.httpunit.WebForm; import com.meterware.httpunit.WebRequest; public class SearchExample extends TestCase { public void testSearch() throws Exception { WebConversation wc = new WebConversation(); WebResponse resp = wc.getResponse(" WebForm form = resp.getForms()[0]; form.setParameter("q", "HttpUnit"); WebRequest req = form.getRequest("btnG"); resp = wc.getResponse(req); assertNotNull(resp.getLinkWith("HttpUnit")); resp = resp.getLinkWith("HttpUnit").click(); assertEquals(resp.getTitle(), "HttpUnit"); assertNotNull(resp.getLinkWith("User's Manual")); } Set the google page Fill the google form get the google response Send the request verify the response

39 HtmlFixture - It is used to exercise and test web pages. - It allows us make assertions about the structure of a page and to navigate between pages. - We can also fire java script, submit forms, "click" links, etc. - It is based on a fixture that lets us navigate to any element of the page DOM. To use this we need some knowledge about the DOM (Document Object Model).

40 … example

41 Capture and Replay for Web Selenium Selenium IDE: a Firefox add- on that will do simple record- and-playback of interactions with the browser Selenium WebDriver: a collection of language specific bindings to drive a browser -- the way it is meant to be driven.