Ananya Das Commercial Test Automation Tools and their real benefits

An Automation Fable Once upon a time in the peaceful land of Testing, the wizard proposes, “ Let there be automation..” and the King agrees “Yes.. that is the way to be” Enter Chandan Nilekani “delegate” Reviews commercial tools, selects, buys, costs …. Assigns to Narad Murthy “overworked “ Narad has several other responsibilities, tool is complicated, tries customer care Finally gets an expert – works and works – several days pass … but no automation Enter Nazim Premji “short timer” Fresh graduate, new ideas, trainings Works full time … library, gets some parts working After 6 months … gets a great offer ….. Enter Shankar Nadar “hardluck” Tries to execute scripts, no documentation Incomplete scripts, poor error messages, gets some bugs … king happy Next version of product...Changes …. Changes scripts … long hours … gets working Product shipped Customer very unhappy … lots of bugs … why ?? Automation Programming error had dismissed these bugs … King – “Off with their Heads !!!!!!!!!“

Some Commercial test Automation tools QTP, RFT, TestPartner Silktest IBM Rational Robot Functional Testing Load Runner QA Load Performance Testing Quality Center Test Director Test Management

Evaluating a commercial tool How easy is it to use? Learning Curve? Continuous Integration (CI) friendly? Call & Log without manual intervention? Work with existing systems ? How does it interact? IE (client side)? HTTP layer? What types of tests does it support? Web? WinForms? Performance? Framework for Integration Test (FIT)? SQL? Does it support recording? Does it support scripting? Languages? Easy creation of scripts ? Object Oriented Scripting Support? Interactive Debugging Tools? Easy integration with External libraries? Ability to identify objects using multiple parameters? Ability to run multiple scripts consistently in a Batch mode? Ability to store test application's objects so that scripts are well maintainable? Reusability of Scripts and Tests? Reporting mechanism? Object parametrization? Ajax? Popup handling? Keyword driven? Automatic Exception Handling? Need to support IE7? How easy is it to transfer tests to/from platform?

Evaluating a commercial tool How well is the tool supported? Commercial? Community? Support for mocking + stubbing? Test case organization? Organizing tests similar to structured manual test plans? Support cross frame script calls? Web Objects? Cost/Licensing? Integrated with other test components? VPN / Remote Desktop friendly? Active Community of Users? Forums? Blogs?

Advantages of Commercial Tools Many of the open source tools come and go with little to no support Most commercial tools are constantly being updated as technologies change Customer support Most commercial tools usually have more functionality (QTP can test various GUI applications: Web,.Net, Java, VB, C/C++ etc) Commercial tools are integrated with Test Management tools which makes reporting and execution much simpler. Functionality Commercial tools usually have a large community of users, which translates into better availability of qualified resources Commercial tools require less advanced programming Resources More test automation frameworks are available for commercial tools Framework

Truth or Myth ? Commercial test tools are expensive ? Myth Under the influence of this myth some companies, especially the small ones: oTry to develop their own test automation tools oUse scripting lang. like Perl and Ruby,use shareware test tools oDo not consider test automation at all Reality It has been calculated that if implemented properly, the cost of this tool is just 3% of the person who uses it, but productivity gain can be very significant

Truth or Myth ? Recorders make test automation easier Myth The Truth: They make recording easy … But they also make it easy to create automation that is difficult and expensive to maintain A recorder can serve as a training device to help you learn a scripting language

Truth or Myth ? You can’t develop test automation for an application until it is stable Myth The Truth: You should develop test plans as soon as the Requirements are documented, You can also develop automated tests from wire- frames, mock-ups and prototypes In the case of wire-frames, you can even test some of your methods If tests are structured properly, only a small portion of your work will have to be maintained when GUI changes are made to an application that has not yet stabilized.

Truth or Myth ? Test automation is most effective for regression testing Part Truth: Test automation is very effective for regression testing It is also effective for first pass testing when a data driven approach is used Automated tests should be designed to find bugs not just verify that regression tests don’t break

Truth or Myth ? It is difficult to automate a dynamic application where data changes constantly Myth The Truth: Data can be separated from test code so that only the data needs to be changed Better yet, in some cases input data can be captured at runtime to meet the criteria required for a test Or a database query can be executed to get data at runtime Verification data can also be captured at runtime so that tests remain flexible in a dynamic environment

Truth or Myth ? Test automation is a software development task Truth: Automated tests should be designed, developed and tested You need to have some kind of a programming background to implement test automation. Test Automation is not as complex as C++/C#/Java development. Automated test components are assets that should be treated like application source code Test automation standards should be developed Test automation should be subject to peer reviews

Truth or Myth ? Productivity gains will be realized from test tools right away Myth: Test automation is an investment Quality gains will be realized right away Productivity gains will be realized over time as skills develop and manual testing tasks are offset by completed automation

Truth or Myth ? Automated tests are difficult to maintain Myth The Truth: Separate business rules, test methods and data to promote maintainability If properly structured, automated tests can be maintained across many releases of a target application It’s important to run automated tests on every build and make incremental changes as needed

Why is Automation great Speed up testing to accelerate releases Allow testing to happen more frequently Reduce costs of testing by reducing manual labor Improve test coverage Ensure consistency Improve the reliability of testing Allow testing to be done by staff with less skill Define the testing process and reduce dependence on the few who know it Make testing more interesting Develop programming skills BUT ………….

Then … why failure ? About 63 % of the Automation projects fail …. Why ? Misconceptions Testing tools don’t work and are difficult to use This application is too complex and the schedules are too tight Senior Management will never support it Automation will eliminate the need for all manual testing Manual testing must have taken place before automating a feature

Critical Success Factors Clearly define Automation Objectives Get management support for the project Clearly demonstrate the short-term & well as long-term financial benefits to be accrued by investing quality resources and time in the project. Ensure that all stakeholders are in agreement on the end objective of the project. Top Management, the IT team, the developers, the tester, and the automation team--may have different requirements from the project. However, success will be elusive unless they all agree to a common goal. Treat Automation as Software development Study the requirements Planning and tracking Set realistic goals, assess existing skill sets and resources Evaluate the testing tools; perform automation feasibility; Identify reusable functions

Critical Success Factors Design and develop a framework Design and document extensible automation architecture, which clearly outlines the foundation and rules for the project. Incorporate flexibility and scalability into the architecture to be able to cope with growth in the requirements Also, provide the testing team with a proof-of-concept test suite and make test case maintenance a top priority Evaluate testing tools Validate the tools and approach at the earliest and evaluate its compatibility with the organization’s requirement and the AUT Check the availability of trained/experienced resources

Critical Success factors` Decide what needs to be automated Include short and simple transactions and tests that are executed regularly. Execution with multiple datasets Do not automate where it is difficult to predict results or are not going to be repeated. Avoid automating long or complex transactions, cross multiple applications Spend time on Automation Strategy and Plan List of activities Organization test objective Reusable and Modular test script identification and planning Run Automation as a full time project Look at it as a long term investment Co-operation with functional testers

GM Regression Automation

ROI ROI = BENEFIT/COST Automation Cost = Price of Tool + Dev Cost + Maint. Cost + Execution Cost Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost ROI = (Manual Testing Cost - Automation Cost)/Automation Cost Looks right, Doesn’t it? NO ….. You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT. You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.

Automation ROI People available to work on other projects Higher market share Higher margins Reduce Time to Market Defects found early More testing gets done faster Test Efficiency Effective regression tests Accuracy Test Effectiveness

Conclusion … Long Live the King Commercial Automation tools are useful if implemented and managed correctly ROI should be managed CSFs should be considered seriously And then... the King said – “ Let there be automation ….” And there were profits, productivity, growth and success all thru the Kingdom THE END