Automation of Software Test Eunyoung Hwang

Agenda What is software test automation? pros and cons Return on investment (ROI) Test automation tools Demo for Selenium IDE Data we have been collect

Motivation! 143,529,861 Links Before we start, let’s take a look a couple of examples regarding test. There is a Korea knowledge portal web site which provides search functionality to find information in each government organization’s data. You want to test if the search result is correct and the link on each organization works well. Then you hire some part time students and give sample key word and let them input and click the links. What about there are this enourmous amount of links. Then what would you do? Here is a campus map and we are installing the terminals in each building which detect the sutdent ID card and open the gate, buy foods or check for attendance and so on. To test it, you need to run all over the campus. If you do enough test before you install terminals then you might need to just check if the terminal is working or not when you deploy it. It can reduce the manpower and also provide stable system.

Test Automation To automate a manual process by using testing tool The use of software to control the execution of tests The comparison of actual outcomes to predicted outcomes The setting up of test preconditions, other test control , test reporting functions Using the right tool for the project can reduce time and costs the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions.[1] Commonly, test automation involves automating a manual process already in place that uses a formalized testing process. One of criteria to define good programmer and bad programmer can be that the good programmer uses tests to detect their mistakes as soon as possible. The sooner you test for a mistake the greater your chance of finding it and the less it will cost to find and fix

Pros. And Cons. Pros.(Cases need automated tests) Cons.(Cases need manual tests) a set of same test cases repeatedly costs more to automate regression tests with frequent changes in timely manner visual setting (eg. Right color and font have been populated on screen or not) a large test matrix (running different languages on different OS platforms at the same time) => reduce the number of manual testers => Test with abnormal random data Automation testing is adopted in scenarios where monotonous test cases have to be executed over and over again. Automation testing is also suitable when all the code paths have to be checked before the final release of the project build. Pros of Automation 1. If you have to run a set of tests repeatedly, automation is a huge win for you. 2. It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner. 3. It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner. 4. Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially. Cons of Automation 1. It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually. 2. Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.

Return on Investment gains costs By company, Keane Numerator/ denominator = fraction Keane is testing framework solution company Value has been accumulated 500 test cases per test cycle costs By company, Keane

Return on Investment By company, Keane as the test cycle per year increase, the return on invest is bigger That means for more often test and test automation is absolutely effective Even the first year there is gains and savings As evident from the graph above, ROI for test automation increases with increasing test cycle frequency. In addition, test execution yields positive ROI earlier in the testing program as the frequency of test cycles increases. By company, Keane

Two General Approaches Code-driven testing Uses testing frameworks such as the xUnits (eg. Junit, PHPUnit) Executes unit tests to verify that the program runs as expected Key feature of Agile software development for Test-driven development Graphical user interface testing Allows users to interactively record and playback any number of times, comparing actual results to those expected Pros - requires little or no software development on any GUI application Cons - relabeling a button or moving it to another part of the window may require re-recording -A growing trend in software development is the use of testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) that allow the execution of unit tests to determine whether various sections of the code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected. - determine whether various sections of the code are acting as expected under various circumstances Test-driven development (TDD) in Agie software development is a software development process that relies on the repetition of a very short development cycle. Unit tests are written to define the functionality before the code is written. This way is more relable than just test once at the end of a waterfall development cycle <GUI testing> -Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. -The advantage of this approach is that it requires little or no software developmen This approach can be applied to any application that has a graphical user interface.

Tools Selenium (IDE, RC) Watir QTP SoapUI Company Open source HP Quick Test Professional Open Source Features IDE: Firefox add-on provide record and playback Also, generate code to run the tests with Selenium RC RC(Remote Control): a client/server system allows you to control web browsers locally or on other computers Web Application Testing in Ruby is a functional testing tool for web applications Modular (Record & Playback) driven, Data driven (Data tables), Keyword driven (Keyword view, function libraries etc) framework Strong in reporting and database application Web service testing tool for service-oriented architectures (SOA), both SOAP and Rest cons Have to learn a vendorscript(Selenese) Have to learn Ruby Use only Vbscript Expensive licence cost Free of charge $14,000 per licence Trial version available $349/yr SoapUI Pro. Trial 14days seleniumhq.org/ watir.com/ http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172957#tab=TAB1 soapui.org IDE : Selenium IDE (Integrated Development Environment) support for several languages, OS and almost every browser(RC) Firefox add-on that makes it easy to record and playback tests in Firefox 2+. Also, generate code to run the tests with Selenium RC Selenium is easy to use because all you have to do is click. For Watir, you need to know ruby to be able to build testing senario Selenese straightforward API) http://stackoverflow.com/questions/606550/watir-vs-selenium-vs-sahi QTP http://www.qtphelp.com/2011/12/14-good-reasons-for-choosing-qtp-for.html Modular (Record & Playback) driven framework, Data driven (Data tables) Framework (Wow! I love the built in data driver functions and the data driving capabilities of QTP, simply amazing!!!), Keyword driven (Keyword view, function libraries etc) framework or any other custom framework.

Simple example in PHPUnit (1) Regular PHP code Simple Automated Testing Code firefox>webdeveloper>selenium ide Open and explain about it Open our website http://greenbay.usc.edu/csci577/spring2012/projects/team12/Symfony/web/app_dev.php/Scheduler/CourseProfileList/ Search for english 9a Start recording Finish and replay it Try break point Save test case as well 6. Selenium IDE is an integrated development environment for Selenium scripts.

Simple example in PHPUnit (2) Testing array operations with PHPUnit firefox>webdeveloper>selenium ide Open and explain about it Open our website http://greenbay.usc.edu/csci577/spring2012/projects/team12/Symfony/web/app_dev.php/Scheduler/CourseProfileList/ Search for english 9a Start recording Finish and replay it Try break point Save test case as well 6. Selenium IDE is an integrated development environment for Selenium scripts.

Selenium RC and PHPUnit Selenium RC is a test tool that allows you to write automated user-interface tests for web applications in any programming language Result if fails firefox>webdeveloper>selenium ide Open and explain about it Open our website http://greenbay.usc.edu/csci577/spring2012/projects/team12/Symfony/web/app_dev.php/Scheduler/CourseProfileList/ Search for english 9a Start recording Finish and replay it Try break point Save test case as well 6. Selenium IDE is an integrated development environment for Selenium scripts.

Selenium IDE firefox>webdeveloper>selenium ide Open and explain about it Open our website http://greenbay.usc.edu/csci577/spring2012/projects/team12/Symfony/web/app_dev.php/Scheduler/CourseProfileList/ Search for english 9a Start recording Finish and replay it Try break point Save test case as well 6. Selenium IDE is an integrated development environment for Selenium scripts.

Summary Test automation pros and cons Return on investment (ROI) Tools PHPUnit Selenium IDE

References [1] http://www.opensourcetesting.org [2] http://www.nickjenkins.net/prose/testingPrimer.pdf [3] http://www.softwareqatest.com/qatweb1.html [4] http://www.keane.com/resources%2Fpdf%2FWhitePapers%2FWP_ROIforTestAutomation.pdf [5] Jussi Kasurinen, Ossi Taipale, and Kari Smolander, “Software Test Automation in Practice: Empirical Observations,” Advances in Software Engineering, vol. 2010, Article ID 620836, 18 pages, 2010. doi:10.1155/2010/620836 [6] http://www.csd.uwo.ca/courses/CS614b/DiLoreto1.pdf [7] http://en.wikipedia.org/wiki/Test_automation

Backup Slides

Return on Investment By company, Keane 3 as the test cycle per year increase, the return on invest is bigger That means for more often test and test automation is absolutely effective Even the first year there is gains and savings By company, Keane 3