Teleconference Evaluating Functional Testing Solutions: The Forrester Wave ™ Q Carey Schwaber Analyst Forrester Research June 27, Call in at 12:55 pm Eastern Time

2 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Agenda The what and why of functional testing solutions How Forrester evaluated functional testing solutions The findings of Forrester’s evaluation Recommendations and WIM

3 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Agenda The what and why of functional testing solutions How Forrester evaluated functional testing solutions The findings of Forrester’s evaluation Recommendations and WIM

4 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Definitions Functional tests: ►Tests to verify that application functionality conforms to predefined specifications

5 Entire contents © 2006 Forrester Research, Inc. All rights reserved. What are the key ingredients of a functional testing solution? In scope: »Manual testing »Functional test automation »Test management Out of scope: »Unit testing »Static analysis »Performance testing »Application monitoring

6 Entire contents © 2006 Forrester Research, Inc. All rights reserved. “Does your shop conduct any automation of functional test scripts?” Base: 74 IT decision-makers at North American and European companies

7 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Pros and cons of manual testing Pros Cost of test case design is minimal »No required tools or tool expertise »No automation necessary »No need for lead time prior to test execution Can be scripted, exploratory, or both »Simultaneous learning, test design, and test execution (Bach) »Useful before, during, and after design and execution of standard manual test scripts Cons Cost of test execution is high »Cost = execution time X labor rate »Execution time is high »No efficiencies when execution is repeated Scripted test execution is tedious »No creativity required All forms are highly error-prone »Quality depends on tester’s attention to detail over time »Documentation of test results is another potential source of errors

8 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Specialized tool support can improve efficiency of scripted manual testing Tool support helps manual testers: »Store test plans, test scripts, and test results in a single, secure location »Share test components (e.g., “login”) across test cases »Automate data entry and data verification

9 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Pros and cons of test automation Pros Frees testers for more intelligent types of testing (e.g., exploratory testing) Drives down time, cost of test execution Permits shops to expand scope of testing efforts Cons Increases up-front investment in test design Easy to waste time automating the “wrong” tests — or the right tests in the wrong way Requires more technical expertise, specialized tools support than manual testing

10 Entire contents © 2006 Forrester Research, Inc. All rights reserved. A simplified view of the economics of test automation If a test script will be run every week for the next two years, automate the test if the cost of automation is less than the cost of manually executing the test 104 times. Cost of automation Cost of manually executing the test as many times as the automated test will be executed Choose to automate when: Cost of test automationCost of tool(s) Labor costs of script creation Labor costs of script maintenance To calculate the cost of automating a test script:

11 Entire contents © 2006 Forrester Research, Inc. All rights reserved. “Why does your company not perform any test automation?” Base: 38 North American and European IT decision-makers at companies that don’t perform any test automation

12 Entire contents © 2006 Forrester Research, Inc. All rights reserved. The right balance varies by test effort Testing team composition »Programming skills vs. subject matter expertise »Division of labor that leverages each team members’ strengths »Assessment of development team’s own testing efforts Nature of the application under test »Application technology »Application stability Timeline »Time available for creation of automated test scripts »Expected lifetime of the application

13 Entire contents © 2006 Forrester Research, Inc. All rights reserved. The benefits of an integrated test management solution for manual and automated testing Common interface for planning and monitoring all test activities Change management for manual and automated test assets Submission of defects from manual testing and test automation tool directly to test management tool Incremental automation of portions of the test suite

14 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Agenda The what and why of functional testing solutions How Forrester evaluated functional testing solutions The findings of Forrester’s evaluation Recommendations and WIM

15 Entire contents © 2006 Forrester Research, Inc. All rights reserved. How did we decide which vendors to evaluate? Criteria for inclusion: $10M in annual revenues Support for manual testing, test automation, and test management Some excluded vendors: RadView and Seapine »Both included in last year’s Wave of test automation tools Worksoft, SDT, and LogiGear »Focus on keyword-driven test automation Agitar and Parasoft »Focus on developer testing

16 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Evaluated vendors and their product offerings Borland Software SilkTest, SilkCentral Test Manager, and SilkCentral Issue Manager Compuware QACenter Enterprise Edition: TestPartner, QARun, QADirector, TrackRecord, Reconcile Empirixe-Tester, e-Manager Enterprise IBM Rational Functional Tester, Rational Robot, Rational Manual Tester, Rational ClearQuest Mercury Interactive QuickTest Professional, WinRunner, Quality Center

17 Entire contents © 2006 Forrester Research, Inc. All rights reserved. The Forrester Wave ™ evaluation process Evaluation conducted between February and May of 2006 »Based on product capabilities generally available by June 1, 2006 Open process to select 87 evaluation criteria: »Interviewed vendors, experts, outsourcers, and users Vendor self-evaluations »Evaluation relies in part on data provided by vendors Interviews on vendor strategy »Conversations with executives to determine how vendors will enhance their offerings in the future Product demonstrations »Validate our understanding of product capabilities Extensive fact-checking with customer references »Determine how vendor offerings work in practice as well as in theory

18 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Evaluation criteria Forrester evaluated these five vendors’ solutions across 87 criteria These criteria fell into three categories (and 19 subcategories): »Current offerings »Strategy »Market presence

19 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Current offerings criteria Solution architecture What is the architecture of the vendor’s functional testing solution? (Operating systems, databases, directory servers) Life-cycle integration With what life-cycle tools do the products in the vendor’s functional testing solution integrate? (Requirements management, issue management, SCM, performance testing, SOA management, integration interfaces) Manual testing How strong are the manual testing capabilities of the vendor’s functional testing solution? (Test case creation, attachments, reuse, impact analysis) Test automation How strong are the test automation capabilities of the vendor’s functional testing solution? (Testable applications, object recognition, script creation) Test management How strong are the test management capabilities of the vendor’s functional testing solution? (Scalability, supported test assets, test planning, test execution, workflow, reporting and analytics, collaboration)

20 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Strategy criteria Product direction How well does the vendor’s product strategy position it for market leadership? (Product vision, product roadmap) Focus What percentage of the vendor’s resources are focused on the products in its functional testing solution? R&D How significantly is the vendor investing in R&D for the products in its functional testing solution? (R&D resources, R&D expense ratio) Price What is the price of the vendor’s functional testing solution? (Test automation price per user, manual testing price per user, test management price per user, solution price for a typical deployment, maintenance price, pricing structure) Financial resources to execute on strategy What is the ratio of the vendor’s current assets to its current liabilities?

21 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Market presence criteria Installed base How large is the vendor's installed base in terms of unique companies? (Total installed base, functional testing installed base, new functional testing customers, functional testing reference customers) Financial strength How financially healthy is the vendor? (Revenues, revenue growth) EmployeesHow many employees does the vendor have? Training What training is available for the products in the vendor’s functional testing solution? Who are the vendor's partners for its functional testing solution? (Technology partners, systems integrators, channel partners) Partnerships

22 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Agenda The what and why of functional testing solutions How Forrester evaluated functional testing solutions The findings of Forrester’s evaluation Recommendations and WIM

23 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Findings Empirix2.11 Vendor Mercury IBM Compuware Borland Ranking Overall rankings (50% current offering, 50% strategy) One leader Three strong performers One contender

24 Entire contents © 2006 Forrester Research, Inc. All rights reserved. How to create a custom ranking Determine how much each of the evaluation criteria matters to you. Weight the evaluation criteria accordingly. Read the score explanation text to familiarize yourself with these tools and vendors. Follow up with demos, trials, and pilots.

© 2006, Forrester Research, Inc. Reproduction Prohibited Forrester Wave™: Functional Testing Solutions, Q2 ’06 May 2006, Tech Choices “The Forrester Wave™: Functional Testing Solutions, Q2 2006”

26 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Vendor profile: Borland Strengths: »Life-cycle integration »Reporting and analytics Weaknesses: »Manual testing »Automated test script creation »Environment support Best for: »Shops with testers who have programming skills. »Shops that use other Borland life-cycle management products

27 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Vendor profile: Compuware Strengths: »Overall breadth — though not depth — of capabilities »Built-in support for risk-based testing Weaknesses: »Weak support for hand-coding and graphical modification of test scripts »Core test management capabilities available only to CARS customers »Too many disparate interfaces Best for: »Project-level testing efforts »Shops that use other Compuware products

28 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Vendor profile: Empirix Strengths: »Strong support for Web environments »Specialized support for Web services »XML-based APIs Weaknesses: »e-Tester has extremely limited environment support »e-Tester does not serve technical or nontechnical testers well »e-Manager Enterprise has minimal support for manual testing »e-Manager Enterprise offers only basic capabilities when it comes to test management »The solution as a whole falls down on life-cycle integration Best for: »Project-level testing efforts »Web applications and services testing efforts

29 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Vendor profile: IBM Strengths: »Support for manual testing »Support for custom coding of test scripts »A platform for test management Weaknesses: »Nonprogrammers don't get much help on test automation »Environment support is still limited, although it is improving »Test execution capabilities are primitive »The functional testing solution itself is in need of better integration Best for: »Using other IBM Rational tools »Doing a great deal of manual testing »Having testers with programming experience

30 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Vendor profile: Mercury Strengths: »Enhanced user productivity through ease of use »Top-notch environment support »Proven scalability across multiple dimensions Weaknesses: »Weak scripting language and scripting environment »Limited management of changes to reused manual test components »Corporate instability Best for: »Centralized testing organizations »Companies that use other Mercury products

31 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Agenda The what and why of functional testing solutions How Forrester evaluated functional testing solutions The findings of Forrester’s evaluation Recommendations and WIM

32 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Factors to consider when selecting a functional testing solution Application technologies in use »Legacy 4GL, Web services, ERP/CRM, custom controls Skill sets »Strong knowledge of the business, programming experience and/or aptitude Organizational structure »Centralized test organization, testers on development teams, offshore testing Development life-cycle tools in use »Tools for developer testing, requirements definition and management, issue management, software configuration management IT operations tools in use »Tools for deployment, performance monitoring, SOA management IT management tools in use

33 Entire contents © 2006 Forrester Research, Inc. All rights reserved. How will vendors improve their offerings? Better enable incremental automation of manual test cases Provide better facilities for graphical creation and modification of test cases Improve support for testing in an SOA environment Do more to facilitate geographically distributed testing efforts Improve integrations with development, operations, and management tools Continue to explore open standards

© 2006, Forrester Research, Inc. Reproduction Prohibited The next area of innovation: SOA testing July 2006 (Upcoming) Trends “SOA Raises The Stakes For Software Quality”

© 2006, Forrester Research, Inc. Reproduction Prohibited Manual and automated functional testing are good candidates for offshore outsourcing March 2006, Trends “How To Benefit From Offshore Testing Services”

36 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Carey Schwaber / Thank you

37 Entire contents © 2006 Forrester Research, Inc. All rights reserved. Selected bibliography May 31, 2006, Tech Choices “The Forrester Wave, Q2 2006: Functional Testing Solutions” March 8, 2006, Trends “How To Benefit From Offshore Testing Services” September 15, 2005, Quick Take “Take Careful Inventory Before Adopting Standalone Code Quality Tools” May 16, 2005, Best Practices “Software Quality Is Everybody’s Business” February 3, 2005, Tech Choices “Evaluating Automated Functional Testing Tools”