Lab 8: GUI testing Software Testing LTAT.05.006

Install & Introduction This lab requires virtual machine VirtualBox is preferred Import the .ova file to your VM This file contains Windows 10 with Testar In the Desktop, you find Testar folder, which contains Testar and SUTs for the lab and homework task Make sure you have enough free space (>= 20 GB)

Capture & Replay and Visual testing tools Selenium, Sikuli Simple and easy to use Maintenance problem if user interface changes

‘Scriptless’ testing tool Testar Scripts are generated automatically No maintenance issues with regards to scripts Main purpose is robustness testing Robustness is the ability of a computer system to cope with errors during execution and cope with erroneous input.

Idea of Testar tool (1) GUI contains graphical objects called widgets Widgets form a hierarchy called widget tree All widgets have properties (title, etc.) Widget tree and properies of each widget form a GUI state GUI State changes, if user executes some action (click, write, etc) Oracles can be defined to check whether a state has been reached that is suspicious to be incorrect

Idea of Testar tool (2) Check oracles Sequence leading to failure Failure detected No failure detected Start SUT Scan GUI state Check stop criteria Stop SUT Yes No Execute action Derive (+filter) actions Select action

Idea of Testar tool (3) Some parameters have to be set before running Testar (-> stop criteria) Check oracles Sequence leading to failure Failure detected No failure detected Start SUT Scan GUI state Check stop criteria Stop SUT Yes No Regular expressions are used to configure oracles and filters Execute action Derive (+filter) actions Select action

Idea of Testar tool (4) – Stop criteria Number of Sequences (=Iterations) Number of Actions per Sequence Identify when there is enough actions done for one test

Idea of Testar tool (5) - Oracles Testar cannot determine if the SUT functions correctly For example, it cannot determine that 2+2 does not equal to 5 Testar can detect failures such as crashes and hangings (without oracles) In addition, Testar can detect failures if widget titles matches oracles Example: user specifies „fail“ as an oracle If word „fail“ appears in some state, then Testar reports it as a failure (test = fail) If word “fail” doesn’t appear in a sequence, then by default the test passes

Idea of Testar tool (6) - Filters By default, Testar will execute actions on all the widgets it can find Including buttons on the title bar (close, minimize, maximize, system) User can define that specific widgets cannot be executed For example, forbid clicking on ‘close’ button To do this, an action filter can be configured Filtering is done by matching the widget title to filters For example, user can specify „.*close.*“ as a filter and Testar will not execute actions on widgets, which title contains „close“

Testar cannot determine Testar can determine

Introduction to folder structure Bin Testar.bat File to run Testar Output Sequences_ suspicioustitle Sequences_ unexpectedclose Graphs Testar Sequences that matched oracles Sequences where the SUT closed unexpectedly Contains raw graph files and a converter See TA’s lab instructions for explanation. SUTs Note: This slide only shows the most important folders Calculator ATM For task in lab For homework task

Interpreting output (1) User must determine if the reported failure is… false positive – reported as failure, but is not really true positive – an actual failure That’s why the folders are called „unexpected…“ and „suspicious…“ User must look through all the sequences, even though many are duplicates, because there is no way to automatically analyse them

Interpreting output (2) Testar generates files, which can be converted to visual graphs „offline_graph_conversion.bat“ is automatically provided in each graph folder All the graphs are converted automatically by the .bat file We use the screenshotted graph to understand what happend in the sequence Looking through the screenshotted graphs should be quite simple and easy to understand, however… It can happen that there are some layout issues and not all the screenshots are connected

Like in this example, where you can see only one state before failure Which was previous state?

To find the previous state, use the action number inside the brakets

Search for lower number to find previous state Ctrl + f

It will show you the previous action This way you can move through the graph if some parts are not connected

Hints (1) Use a reasonable number of sequences (say, something between 20 and 100) If the number is too small, you might not trigger any failure If the number is too high, the execution might take too long and the (manual) analysis of the output files might take too long Use a reasonable number of actions in each sequence It will keep graphs smaller and therefore easier to read About 20 should be fine, but it depends how you use the filters

Hints (2) If you keep finding the same failure many times, you should consider filtering out the path leading to that failure It will save you time while inspecting graphs Also, make sure, that you don’t apply filter too early, otherwise you might miss some of the failures Information messages are not to be considered failures if they match those listed in the appendixes A (in-lab task) and C (homework task) Don’t write oracles matching these messages Check out Testar manual for more information about the tool