ForrTel: Selecting An Automated Functional Testing Tool Carey Schwaber Analyst Forrester Research February 8, Call in at 12:55 p.m. Eastern Time

Theme There is no single best functional testing tool; rather, different tools are more or less appropriate in different environments.

Agenda Test automation in context Features and functions — today and tomorrow How seven leading tools stack up How to pick the right tool for the job

Major categories of application quality tools Construction Design Testing DeploymentAnalysis Functional testing Performance testing Developer testing Application monitoring

Definitions Functional tests: ►Tests to verify that application functionality conforms to predefined specifications. Automated functional testing tools: ►Tools to build and automatically execute suites of functional tests scripts.

How to calculate the cost of test automation: A simplified view of the economics of test automation If a test script will be run every week for the next two years, automate the test if the cost of automation is less than the cost of manually executing the test 104 times. Cost of automation Cost of manually executing the test as many times as the automated test will be executed Cost of test automationCost of tool(s) Labor costs of script creation Labor costs of script maintenance Automate if:

Strategies for effective test automation Automation as part of a well-defined testing process Different processes for different products and teams Construction of test scripts out of modular, reusable script components Division of labor between testing specialists and automation specialists Highly selective automation of test scripts

Where is automation the most valuable? Start out by automating test scripts that: Verify the most critical functionality Are the most likely to expose defects Are expensive or impossible to perform manually Use the first suites you build for: Smoke testing Regression testing

Agenda Test automation in context Features and functions — today and tomorrow How seven leading tools stack up How to pick the right tool for the job

Tool features and functions: Today Script creation and enhancement »Record and playback »Hand coding »Data-driven »Keyword-driven Script maintenance »Object property management »Support for custom objects Integration »Integration with other testing tools »Integration with other life-cycle tools

Highlight: Keyword-driven testing How it works: »Automation engineers build modular test components. »Test designers assemble these components into test scripts. What it means: »Test design and test automation become discrete activities. »Team members’ respective strengths are better leveraged. How to get it »Build it »Buy it (LogiGear, Mercury, SDT, or Worksoft) »Download it (Emos, FIT/FitNesse, SAFS)

Tool features and functions: Tomorrow Adopt and leverage open standards Improve integration with other life-cycle tools Reach out to new user populations Improve support for distributed teams and organizations Expect testing tool vendors to:

Agenda Test automation in context Features and functions — today and tomorrow How seven leading tools stack up How to pick the right tool for the job

The Wave process User survey: »Online survey of 34 testing tool users to find out what they like and don’t like about their primary testing tools Open process to select 80+ evaluation criteria: »Interviewed vendors, experts, and SIs Vendor self-evaluations and vendor interviews Extensive fact-checking »Spoke with 14 customer references to ensure accuracy of vendor claims

Vendors and tools we evaluated ToolVendorName of testing suite or companion tools TestPartnerCompuwareQACenter Enterprise Edition+ QuickTest Professional MercuryQuality Center e-TesterEmpirix e-TEST suite Functional TesterIBM Rational Test Manager, Manual Tester, Performance Tester WebFTRadViewTestView Suite SilkTestSegueSilkCentral, SilkPerformer QA WizardSeapineTestTrack Pro

Vendor evaluation — Forrester Wave™ Risky bets Contenders Current offerings StrategyWeak Strong Leaders Strong performers IBM Empirix Seapine Compuware Mercury RadView Segue

What’s the story behind this graphic? Mercury is currently the most capable overall. But IBM Rational has the best strategy. Compuware comes in third; it is many things to many audiences, but not everything to any single audience. Segue and Empirix, which have very different strengths, average out to about the same position. RadView did well in spite of the firm’s financial troubles. Seapine, an emerging challenger, isn’t ready for the big leagues yet.

Evaluation criteria: Strategy Corporate strategy »R&D spending »Size of developer staff »Size of support staff and quality of support Product strategy »Product road map »Breadth of target audience Price »License cost of product as evaluated »Licensing flexibility Sales strategy »Global reach »Channel partnerships

Evaluation criteria: Market presence Installed base »Number of users of this product »Number of customers of all products Services Employees Financials »Annual revenue »Year-over-year revenue growth

How you can use the Wave spreadsheet Determine how much each of the evaluation criteria matters to you. Set your own weights. Read the score explanation text to familiarize yourself with these tools and vendors. Follow up with demos, trials, and pilots.

How to get to the Wave spreadsheet Click here

How to weight the criteria Enter your weightings on the “Scores and weightings” tab of the Wave spreadsheet. View the resultant custom Wave on the “Custom Forrester Wave” tab.

Agenda Test automation in context Features and functions — today and tomorrow How seven leading tools stack up How to pick the right tool for the job

The three most important decision drivers 1. Support for the applications under test 2. Suitability for your team’s skill level(s) » Technical skill level » Skill homogeneity 3. Integration with other in-house tools » Development tools » Requirements management tools » SCM tools » Performance testing and monitoring tools

Different tools suit teams testing different kinds of apps Centralized testing groups often need to test a wide range of apps. Testers working with development teams often need to test only a few kinds of apps. Firms dedicated to a particular technology (e.g., J2EE or.NET) won’t require a tool with very broad range. Mercury QuickTest Pro Segue SilkTest Compuware TestPartner IBM Rational Functional Tester Seapine QA Wizard Empirix e-Tester RadView WebFT Broader range of supported application technologies Narrower range of supported application technologies Web only

Different tools suit users with different skill levels Technical usersNontechnical users Technical and nontechnical users Mercury QuickTest Pro Compuware TestPartner Empirix e-Tester Seapine QA Wizard IBM Rational Functional Tester Segue SilkTest RadView WebFT

Different tools suit firms with other in-house tools »Compuware TestPartner integrates with Compuware’s developer testing tools and application monitoring tools. »Empirix e-Tester integrates with Empirix’s performance testing and monitoring tools. »IBM Rational Functional Tester integrates with IBM’s development tools — and Microsoft’s. »Mercury QuickTest Pro integrates with Mercury’s performance testing and monitoring tools. Each tool integrates with some third-party tools, but a few integrate very well with other tools in the vendor’s lineup and are more attractive to firms that use these other tools:

Takeaways Firms should determine their approach to test automation before they begin shopping for tools. Three factors contribute to the cost of test automation; tool choice impacts all three. While there is no “best” functional testing tool, there are tools that are more or less appropriate given: »The range of applications under test. »The level and mix of user skill levels. »Other in-house life-cycle tools.

Carey Schwaber Thank you! Entire contents © 2005 Forrester Research, Inc. All rights reserved.

Selected bibliography February 3, 2005, Tech Choices “Evaluating Automated Functional Testing Tools” February 3, 2005, Tech Choices “Automated Functional Testing Vendor Profile: Mercury Interactive” February 3, 2005, Tech Choices “Automated Functional Testing Vendor Profile: IBM Rational Functional Tester” February 3, 2005, Tech Choices “Automated Functional Testing Vendor Profile: Compuware TestPartner” October 8, 2004, Quick Take “What Hyades Means For IBM, Its Customers, And Its Competitors”

Client Choice ballots for Application Development and Infrastructure are open through February 11. Make your opinion count and vote today. To view Client Choice current ballots, click here: