1 Event Driven Automated Testing for GUI-based Java Programs Presented by Yanhong Sun Thesis Advisor: Dr. Edward L. Jones April 28, 2003

2 Outline Introduction Related Work  Test strategy / Test plan  Test case generation  Test execution  Test verification Goals of My Thesis  Implementing the initial AGUIJ  The AGUIJ Features  Case study  The future work References

3 Introduction To grade GUI-based Java program by hand is a redundant, boring and time intensive work The Automated grader for GUI-based Java programs (AGUIJ) is a fast and efficient technique for checking programming assignments submitted by numerous students

4 Introduction (cont.) All programs have interfaces  Text interface  Graphical user interface The applications with text interfaces are driven by data  Automated grader can be readily accomplished with the help of I/O redirection Currently, I am trying to extend the automated grading service for GUI-based Java programs

5 Introduction (Cont.) GUIs are event-driven  The type of input is more varied button clicks, mouse movements, etc.  It is not so easy to reassign the input stream with the help of I/O redirection  Special tool is needed to simulate the input and capture the output

6 Related Work Testing life cycle: Figure 1 Testing life cycle Design Implementation Evaluation Analysis Execution Test Strategy/ Plan Test Script, Data, Driver Test Cases Test Results Problem Report Specification

7 Related Work (Cont.) Test strategy / test plan  Most well-known coverage criteria include statement coverage, branch coverage, and path coverage  Since the input to a GUI consists of a sequence of events, such criteria can not apply for GUI testing  Memon, Soffa and Pollack [6] presented new coverage criteria. These coverage criteria use events and event sequences to specify a measure of test adequacy for GUI testing

8 Related Work (Cont.) Test case generation  One of the main difficulties in testing GUI-based applications lies in describing the input and output.  Memon, Pollack and Soffa[8] present PATHS, Planning Assisted Tester for graphical user interface Systems, a new technique to automatically generate test cases for GUIs. Given: a set of operators, an initial state, and a goal state Produces: a sequence of the operators that will transform the initial state to the goal state

9 Related Work (Cont.)  Chen and Subramamiam [7] present a prototype VESP, a Visual Environment for manipulating test Specification of GUI-based application in Java. Testers can easily modify the test specification  Ostrand, Anodide, Foster and Goradia [9] have implemented an experimental test development environment (TDE) which replaces the low-level scripting language with a high-level scenario language and provide a visual model for modifying and creating variations of recorded sequences

10 Related Work (Cont.) Test Execution  Almost all of the testing tools for GUI-based applications are based on capture/replay technique  The capture/replay technique is good for regression testing  There exist various commercial capture/replay tools XRunner [10] -- an automated tool for X windows application JavaStar [11] – SunTest ’ s GUI testing tool, accesses the Java AWT components Marathon [12] – allows the tester to play and record scripts against a Java swing UI

11 Related Work (Cont.) Verification  The traditional verification tool is a test oracle, a separate program that generate expected results for a test case and compares them with actual results.  Most existing automation tools usually cannot efficiently compare graphic objects.  Takahashi [13] develops a technique that aids automatic behavior verification for verifying the graphic objects

12 Expected goals of my thesis In summary, the anticipated goals of my thesis are:  Implementing the initial AGUIJ to seek the feasibility of the automated testing for GUI- based Java programs  Making AGUIJ work for multiple frames  Implementing the Test Generator Engine which generates the Jemmy test engine source file automatically  Implementing the format of test specification (grading plan) which can be read by Test Generator Engine

13 Implementing the initial AGUIJ At the very beginning, a bright idea was assigned to me by Dr. Jones to construct an automated grader for GUI-based Java program By study and research, I found out that almost all of the testing tools for GUI-based applications are based on capture/replay technique pros: techniques are available and free to use cons: i) needs oracle support ii) can ’ t start automated test until the tester manually tests a perfect program

14 Implementing the initial AGUIJ (cont.) Suddenly, another technique called Jemmy caught my attention.  Jemmy [1] is a Java TM library that is used to create automated tests for Java GUI applications  Jemmy contains methods to reproduce all user actions that can be performed on Swing/AWT components (i.e. button pushing, text typing).  Pros: i) can perform test execution and test verification in one program ii) can start automated test right away

15 Implementing the initial AGUIJ (Cont.) Figure 1 Automated Grading Service Student Programs Specification Design Test Cases Compile Jemmy test engine (Java) class file(s) Jemmy test engine (class) Not compile Grading Log files compile

16 AGUIJ Features Test modes (position based or object based)  In position-based mode, the selection of an item on the screen is determined by the program based on the position of the item within the window  In object-based mode, the test software sends the event not to a given position in the window, but to a specific object.  AGUIJ is object-based test mode

17 AGUIJ Features (Cont.) Output checking  Bitmap comparison compares graphical images bit-by bit  Content-based comparison ignores the presentation, but compares the underlying data itself  The AGUIJ use content-based comparison checker to verify whether the program runs correctly

18 Case study I used the initial AGUIJ to grade the Java GUI programs, the outcome was optimistic Table 1 Time consumed in grading The ratio of correctness: 92.4% Automated Grading Service Manual Grading Service Compiling 7 mins Running 19 mins Total About 30 mins About 4-5 hours without any interruption

19 Future Works Although the initial AGUIJ tends to make the GUI test automated, there are still pitfalls  To implement Jemmy test engine source file is not a fully automated step  The service produces lower productivity  The service can deal only with one single frames The future work should involve in AGUIJ generalization to achieve more automateion

20 Future Works (Cont.) Working for multiple frames  I will extend the automated grading service to work for GUI-based Java programs with multiple frames Implementing the Test Generator Engine  I will develop the Test Generator Engine, which can transform a formatted grading plan to a Jemmy test engine source file.

21 Future Works (Cont.) Implementing the format of the grading plan  The grading plan must meet the following criteria: Easy to write Easy to understand Amenable to automated generation of the Jemmy test engine source file  The information are needed to generate the Jemmy test engine source file: The name of class file to be tested The title of frames The components in the GUI The test cases.

22 Future Works (Cont.) I intend to use the format like HTML to implement grading plan. Table 2 A sample of formatted grading plan (HTML style) FAMUCurrencyExchange.class type = JComboBox; order = 1 type = JComboBox; order = 2 type = JTextField; order = 1 type = JTextField; order = 2 type = JButton; name = Convert number = 1 type = JComboBox; order = 1; name = Currency Type 1; action = select item 2; item name = U.S.Dollar type = JTextField; order = 1; action = type type = JComboBox; order = 2; name = Currency Type 2; action = select item 4; item name = Japanese Yen type = JButton; action = push type = JTextField; order = 2; content = or deduction = 25 The class name The components in the GUI The test cases User inputs The expect outputs deductions

23 Future Works (Cont.) The automated grading service will be updated to the following: Figure 3 The updated Automated Grading Service System Student Programs Specification Write grading plan Compile Test Engine Generator Jemmy test engine (Java) Not compile Grading Log files Jemmy test engine (class) class file compile

24 References [1] Jemmy Module. See [2] Alan Walworth. Java GUI Testing. Dr. Dobb ’ s Journal February 1997 [3] Tessella Support Services PLC. Automated GUI Testing. Tessella Scientific software solutions January 1999, issue V1.R1.M1 [4] J. Skrivanek and A. Sotona. Tesing ForteTM for JAVATM [5] Cay S. Horstmann, Gary Cornell. The core of JAVA, second edition. [6] A. M. Memon, M. L. Soffa, and M. E. Pollack. Coverage Criteria for GUI Testing. [7] J. Chen, and S. Subramaniam. A GUI Environment to Manipulate FSMs for Testing GUI-Based Applications in Java. In proceedings of the 34th Hawaii international Conference on System Sciences, 2001

25 References (Cont.) [8] A. M. Memon, M. E. Pollack, and M. L. Soffa. Hierarchical GUI test case generation using automated planning. IEEE Transactions on Software Engineering, 27(2): , Feb.2001 [9] T. Ostrand, A. Aanodide, H. Foster, and T. Goradia. A Visual Test Development environment for GUI Systems. In proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis(ISSTA-98), pages 82-92, New York, Mar ACM Press. [10] XRunner. See /xrunner6http:// /xrunner6 [11] JavaStar. See [12] Marathon. See [13] J. Takahashi. An Automated Oracle for Verifying GUI Objects. Software Engineering Notes, vol. 26 no 4: 83-88, July 2001 [14] Fewster, Mark. Software Test Automation, Addison Wesley, New York, 1999

26