1 Event Driven Automated Testing for GUI-based Java Programs Presented by Yanhong Sun Thesis Advisor: Dr. Edward L. Jones July 17, 2003

2 Outline Problem Statement Background Motivation of Research Jemmy Module AGUIJ Design  Constructions of AGUIJ Design AGUIJ Prototype Test Specification language Jemmy Test Engine Generator (JTEG)  Demonstration  Limitations of AGUIJ The future works References

3 Event Driven Automated Testing for GUI-based Java Programs Problem Statement  The amount of testing Java program increases  Testing GUI-based Java programs by hand is a redundant, boring and time intensive work.  AGUIJ --- Automated grader for GUI-based Java programs

4 Event Driven Automated Testing for GUI-based Java Programs Background  All programs have interfaces Text interface Graphical user interface  The applications with text interfaces are driven by data Automated grader uses I/O redirection Julia and I have been working on this for one year

5 Event Driven Automated Testing for GUI-based Java Programs  GUIs are event-driven The type of input is more varied button clicks, mouse movements, etc. It is not so easy to reassign the input stream with the help of I/O redirection Special tool is needed to simulate the input and capture the output

6 Event Driven Automated Testing for GUI-based Java Programs Motivation of Research  Testing life cycle

7 Event Driven Automated Testing for GUI-based Java Programs  Testing life cycle Figure 1 Testing life cycle Design Implementation Evaluation Analysis Execution Test Strategy/ Plan Test Script, Data, Driver Test Cases Test Results Problem Report Specification

8 Event Driven Automated Testing for GUI-based Java Programs Motivation of Research  Testing life cycle?  What activities the automated GUI testing process tends to apply to?

9 Event Driven Automated Testing for GUI-based Java Programs WWhat activities the automated GUI testing tends to apply to? Test case generation Test execution Test verification (Test oracle)

10 Event Driven Automated Testing for GUI-based Java Programs Motivation of Research  Testing life cycle?  What activities the automated testing process tends to apply to?  What tools are available for automated GUI testing?

11 Event Driven Automated Testing for GUI-based Java Programs  What tools are available for automated GUI testing? Almost all of the testing tools for GUI-based applications are based on capture/replay technique (see Figure 2) There exist various commercial capture/replay tools XRunner [10] JavaStar [11] Marathon [12]

12 Event Driven Automated Testing for GUI-based Java Programs Figure 2 Standard commercial capture/replay tool interactions Tester Test script Capture/Replay tool Software under test script editing replay record

13 Event Driven Automated Testing for GUI-based Java Programs The shortcomings of using capture/replay tool Needs external oracle support. Manual testing is necessary for capturing events sequence. May not be able to replay until the tester has manually tested a perfect program

14 Event Driven Automated Testing for GUI-based Java Programs Jemmy module  Jemmy is a Java TM library  Can perform test execution & test verification in one program --- no external oracle support  No need for recorder --- no any manual testing, start automated test right away

15 Event Driven Automated Testing for GUI-based Java Programs AGUIJ Design generate run Student Programs Specificationtest specification Compile JemmyTest Engine Generator (JTEG) Not compile Grading Log files Jemmytest engine (class) class file Student Programs Specification test specification Compile JemmyTest Engine Generator (JTEG) Jemmytest engine (Java) Not cmpile Jemmytest engine (class) class file(s) compile generate run

16 Event Driven Automated Testing for GUI-based Java Programs  Constructions of AGUIJ design The initial AGUIJ implementation Test Specification language design. Jemmy Test Engine Generator (JTEG) design

17 Event Driven Automated Testing for GUI-based Java Programs The initial AGUIJ Figure 3 The initial AGUIJ Student Programs Specification Compile Jemmytest engine (Java) classfile(s) Jemmy test engine (class) Not compile Grading Log files compile Student Programs Specification Compile Jemmy test engine source file (Java) classfile(s) Jemmy test engine (class) compile run Grading plan

18 Event Driven Automated Testing for GUI-based Java Programs Case Study Table 1 Time consumed in grading (total 66 students)  The ratio of consistency: 92.4% Automated Grading Service Manual Grading Service Compiling 7 mins Running 19 mins Total About 30 mins About 4-5 hours without any interruption

19 Event Driven Automated Testing for GUI-based Java Programs Limitations of the initial AGUIJ --- To implement Jemmy test engine source file is not a fully automated step  The tester needs to be familiar with Jemmy API --- Lower productivity  To implement Jemmy test engine source file start-to- finish for each program assignment  Have similar structure

20 Event Driven Automated Testing for GUI-based Java Programs Test Specification language design The criteria must be met  Easy to write  Easy to understand  Amenable to automated generation of the Jemmy test engine (JTE) source file

21 Test casesExpected data Deduction 1Type “Florida\n” in State field “Warm” is displayed in Climate field 25 2Type “Alaska\n” in State field “ Cold ” is displayed in Climate field 25 grading plan

22 climate type = JTextField; order = 1 type= JTextField; order = 2 number = 1 type = JTextField; order = 1; name = "state"; action = type “ Florida\n" type = JTextField; order = 2; name = "temperature"; content = equals "Warm"; deduction = 25 number = 2 type = JTextField; order = 1; name = "state"; action = type “ Alaska\n" type = JTextField; order = 2; name = "temperature"; content = equals “ Cold"; deduction = 25 Test casesExpected dataDeduction 1 Type “Florida\n” in State field “Warm” is displayed in Climate field 25 2 Type “Alaska\n” in State field “ Cold ” is displayed in Climate field 25

23 Event Driven Automated Testing for GUI-based Java Programs JTEG Design Figure 4 The function of JTEG Test Specification JTEG Jemmy Test Engine(JTE) source file read generate

24 Event Driven Automated Testing for GUI-based Java Programs  Demonstration

25 Event Driven Automated Testing for GUI-based Java Programs  Limitations of AGUIJ Only support Java GUI with one single frame Access most Swing components, not all of them The syntax check of the test specification is not powerful enough

26 Event Driven Automated Testing for GUI-based Java Programs Future Works  Extending AGUIJ to support multiple frames Naming problem Application state --- hierarchical relationship --- parallel relationship  Using XML to describe test cases in test specification  Generating test specification automatically

27 Event Driven Automated Testing for GUI-based Java Programs References [1] Jemmy Module. See [2] Alan Walworth. Java GUI Testing. Dr. Dobb ’ s Journal February 1997 [3] Tessella Support Services PLC. Automated GUI Testing. Tessella Scientific software solutions January 1999, issue V1.R1.M1 [4] J. Skrivanek and A. Sotona. Tesing ForteTM for JAVATM [5] Cay S. Horstmann, Gary Cornell. The core of JAVA, second edition. [6] A. M. Memon, M. L. Soffa, and M. E. Pollack. Coverage Criteria for GUI Testing. [7] J. Chen, and S. Subramaniam. A GUI Environment to Manipulate FSMs for Testing GUI-Based Applications in Java. In proceedings of the 34th Hawaii international Conference on System Sciences, 2001

28 Event Driven Automated Testing for GUI-based Java Programs [8] A. M. Memon, M. E. Pollack, and M. L. Soffa. Hierarchical GUI test case generation using automated planning. IEEE Transactions on Software Engineering, 27(2): , Feb.2001 [9] T. Ostrand, A. Aanodide, H. Foster, and T. Goradia. A Visual Test Development environment for GUI Systems. In proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis(ISSTA-98), pages 82-92, New York, Mar ACM Press. [10] XRunner. See /xrunner6http:// /xrunner6 [11] JavaStar. See [12] Marathon. See [13] J. Takahashi. An Automated Oracle for Verifying GUI Objects. Software Engineering Notes, vol. 26 no 4: 83-88, July 2001 [14] Fewster, Mark. Software Test Automation, Addison Wesley, New York, 1999 [15] XMLSPY 5. See

29 Event Driven Automated Testing for GUI-based Java Programs Acknowledgement CIS department CREST Dr. Edward Jones Dr. Clement Allen Dr. YiLi Zeng Dr. Roselyn Williams Mr. Aubrey Rembert Mr. Hongtao Chen

30