1. Webinar Achieve Continuous Quality With Service Virtualization And Testing Tools Diego Lo Giudice, Vice President, Principal Analyst April 11, 2014. Call in at 10:55 a.m. Eastern time

2. © 2014 Forrester Research, Inc. Reproduction Prohibited2 Agenda › Old testing problems made worse by the age of the customer › What benefits clients are achieving with SVT › Why CA Technologies and IBM lead, Parasoft and HP follow, and SmartBear Software contends

3. © 2014 Forrester Research, Inc. Reproduction Prohibited3 Agenda › Old testing problems made worse by the age of the customer › What benefits clients are achieving with SVT › Why CA Technologies and IBM lead, Parasoft and HP follow, and SmartBear Software contends

4. © 2014 Forrester Research, Inc. Reproduction Prohibited4 Transform the customer experience. Embrace the mobile mind shift. Become a digital disruptor. Age of the customer Turn big data into business insights. Build and run new systems of engagement with deep customer insights for superior CX. Create and deliver mobile apps that support customer engagement anywhere and anytime. Develop and deliver apps that leverage more data and better analytics for decision-makers. Support your companies’ digital disruption strategy with the right technologies. The age of the customer’s four imperatives: implications for AD&Ds

5. In the age of the customer, modern applications will rule the world.

6. © 2014 Forrester Research, Inc. Reproduction Prohibited6 Social media and search Systems of engagement (SOE) In-store, point-of-sale systems Systems of record (SOR) Cloud-based services provided by third parties › Mobile, Web, and smart devices › Frequent changes › Delight clients › Delivered frequently › Mobile, Web, and smart devices › Frequent changes › Delight clients › Delivered frequently › Stable requirements › Highly transactional › Less change › Delivered infrequently › Stable requirements › Highly transactional › Less change › Delivered infrequently Traditional Cloud But what are modern applications?

7. © 2014 Forrester Research, Inc. Reproduction Prohibited7 Base: 96 QA and IT leaders; Source: A commissioned study conducted by Forrester Consulting on behalf of a large system vendor, September 2013 Testing is still an afterthought. Long testing cycles, too many handoffs, and developers versus testers Test environments are an issue in many ways. › 71% claim test environments are not consistently available. › 56% claim specific issues with integration testing It’s hard to achieve high levels of automation. › 18% automate less than 20% of their tests. › 57% automate between 20% and 50% of their tests. Old testing problems are still here

8. © 2014 Forrester Research, Inc. Reproduction Prohibited8 Base: 96 QA and IT leaders; Source: A commissioned study conducted by Forrester Consulting on behalf of a large system vendor, September 2013 Old testing challenges are still here Integration testing just gets harder. › Systems of engagement increase SI challenge. › Parallel or Agile development is seriously challenged. Performance testing presents many struggles. › 64% struggle to measure performance when interfacing external systems. › 56% struggle to measure extreme conditions. Creating and provisioning test data are ad hoc or not done. › 63% claim it is hard to virtualize different data scenarios for testing. Developers develop ad hoc and nonreusable stubs that don’t necessarily mimic production precision.

9. © 2014 Forrester Research, Inc. Reproduction Prohibited9 Base: 96 QA and IT leaders; Source: A commissioned study conducted by Forrester Consulting on behalf of a large system vendor, September 2013 “Going forward, how do you see your integration needs changing?” Modern apps, cloud, mobile, social, and big data will increase integration complexity over the next 12 to 18 months Testing challenges can only get worse!

10. Modern applications are more than ever all about continuous quality. › Continuous feedback › Continuous development › Continuous testing › Continuous delivery › {Continuous deployment}

11. © 2014 Forrester Research, Inc. Reproduction Prohibited11 Agenda › Old testing problems made worse by the age of the customer › What benefits clients are achieving with SVT › Why CA Technologies and IBM lead, Parasoft and HP follow, and SmartBear Software contends

12. © 2014 Forrester Research, Inc. Reproduction Prohibited12 Testing tools that capture, model, and simulate application behavior to test automatically the functional and nonfunctional behavior of applications in a simulated production environment Forrester’s definition of SVT

13. © 2014 Forrester Research, Inc. Reproduction Prohibited13 Source: January 27, 2014, “The Forrester Wave™: Service Virtualization And Testing Solutions, Q1 2014” Forrester report

14. © 2014 Forrester Research, Inc. Reproduction Prohibited14 With SVT, you can create virtualized services to simulate production and also test various simulated scenarios Production Applications under test Service virtualization Rest Application services SWIFT Online cloud third-party services LDAP Security server Mainframe MQ JDBC DB SAP/legacy SAP Web services SOAP REST JDBC MQ RFC CICS Data Perf Config Simulations Web browser Mobile apps Tablet Internet apps Composite applications

15. © 2014 Forrester Research, Inc. Reproduction Prohibited15 BenefitsHow Quality — less production defects More stable test environments across all stages Test environment more aligned to production Better test coverage Left shift of testing Lower costs/increase productivity Production environment simulation versus duplication Less time spent in fixing production bugs Sharing/reuse of virtualized assets (stubs) Speed/time-to-market Reduce wait time for testing — test lab provisioning Increasing functional and nonfunctional test automation Incremental and Agile development

16. © 2014 Forrester Research, Inc. Reproduction Prohibited16 Base: 54 QA and IT leaders; Source: A commissioned study conducted by Forrester Consulting on behalf of a large system vendor, September 2013 “What percentage improvement have you achieved using service virtualization for each of the following categories?” Those that adopted SVT achieved 50% improvement in coverage, wait time, availability, and time-to-market and less defects in production More than 50% improvement: › 26% test coverage › 24% wait times › 23% service availability › 16% production defects › 15% time-to-market

17. © 2014 Forrester Research, Inc. Reproduction Prohibited17 What clients are doing with SVT What clients didHow they did it › Large manufacturing — reduced test cycle time on a constrained service › What: built a virtual asset (VA) of the constrained service that had a window of availability of only 30 minutes every three weeks › Benefit: The VA was available always and instantly for many projects. › Large global bank — 1) automated testing over proprietary connectivity of Internet banking channel and 2) time-to-market of SOA design and testing of expensive core banking services. › What: 1) virtualized a bespoke protocol and automated functional and nonfunctional testing and 2) created VAs around ESB and WISDL to test business processes virtually › Benefits: 1) automation of regression tests not possible before and 2) avoid access for testing expensive core banking services running on mainframe. › Large telecom — transforming to Agile and continuous delivery › What: 1) integrated SVT in the ALM process and started testing earlier by generating VAs from WISDL and 2) integrated with CI to automate tests › Benefits: 1) implemented in small batches and 2) increased automation › Large online bank — testing constrained third-party mainframe expensive services and getting data that previously was not always accessible. › What: created VAs for third-party services › Benefits: reduced blocked dates for testing by 39% (147 man days in a month), reduced by 50% regression test time on CRM, more tests done with same resources, and less bugs in production How to get started (from clients real cases): › Start small and then grow. › Find an easily quantifiable financial justification. › Sponsors are either EAs or QMs. › Involve developers early on.

18. © 2014 Forrester Research, Inc. Reproduction Prohibited18 Service virtualization and testing enable the transition to continuous testing and improved DevOps Source: January 27, 2014, “The Forrester Wave™: Service Virtualization And Testing Solutions, Q1 2014” Forrester report

19. © 2014 Forrester Research, Inc. Reproduction Prohibited19 Agenda › Old testing problems made worse by the age of the customer › What benefits clients are achieving with SVT › Why CA Technologies and IBM lead, Parasoft and HP follow, and SmartBear Software contends

20. © 2014 Forrester Research, Inc. Reproduction Prohibited20 Microsoft “testing in production” The expanded market for SVT includes other types of complementary tools

21. © 2014 Forrester Research, Inc. Reproduction Prohibited21 Source: January 27, 2014, “The Forrester Wave™: Service Virtualization And Testing Solutions, Q1 2014” Forrester report Evaluated vendors: product information and selection criteria

22. © 2014 Forrester Research, Inc. Reproduction Prohibited22 Source: January 27, 2014, “The Forrester Wave: Service Virtualization And Testing Solutions, Q1 2014” Forrester report The Forrester Wave™: Service Virtualization And Testing Solutions, Q1 2014

23. © 2014 Forrester Research, Inc. Reproduction Prohibited23 SVT is an emerging market with strong momentum › SVT technology is not new! › Few vendors but some of high caliber and others with long testing tenure › Strong market momentum 2013 to 2014 Vendors added many new logos in the past 12 months — mostly double digit. Many SVT projects are deploying. Modern application delivery trends are increasing the need for SVT.

24. © 2014 Forrester Research, Inc. Reproduction Prohibited24 Why the Leaders excel › CA Technologies and IBM lead the pack with rich features, comprehensive strategy, and strong strategic enablement. › Both have a very strong ALM and DevOps tie. IBM integration with its UrbanCode family but with “traditional” prevailing focus on “developers” CA Technologies integration with Nolio family but with more of an “operations” (ITSM) heritage › CA Technologies leads overall in features richness while IBM leads in strategy, but both came close.

25. © 2014 Forrester Research, Inc. Reproduction Prohibited25 Current offeringCA Technologies 4.35IBM 4.10 Platform and protocol support Testing scenarios Simulation accuracy Scalability Integration with ALM and testing tools StrategyCA Technologies 4.40IBM 4.70 Licensing flexibility, scalability, simplicity, and transparency Market presenceCA Technologies 4.25IBM 4.25 Global presence and installed base Market momentum Where CA Technologies and IBM differ

26. © 2014 Forrester Research, Inc. Reproduction Prohibited26 Why HP and Parasoft are Strong Performers › Parasoft: Comes in very strong from a feature and functionality standpoint (4.10 as IBM!) But is weaker on global strategy and presence › HP: Solution is not as rich as others, but being built ground up, it offers great and easy-to-execute use cases with QC, UFT, and LoadRunner. Weakest DevOps tool integration (overall strategy) and weaker market momentum. Will it catch up? Product road map is aggressive.

27. © 2014 Forrester Research, Inc. Reproduction Prohibited27 Current offeringHP 2.80Parasoft 4.10 Dev/tester tools and productivity Platform and protocol support Simulation accuracy Integration with DevOps tools Integration with ALM and testing tools StrategyHP 4.00Parasoft 3.00 Product road map Ability to execute Market presenceHP 2.25Parasoft 2.50 Global presence and installed Market momentum Partnerships Where HP and Parasoft differ

28. © 2014 Forrester Research, Inc. Reproduction Prohibited28 SmartBear Software contends but has a different strategy › SmartBear Software is a monoprotocol SVT with minimum functionality focusing on SOAP. › Mostly an API testing tool with SV as a feature Dominates the SOAP API testing market Strongest in global presence, installed base, and market momentum. Its entry-level product is an open source option (SoapUI). Might expand its SVT capabilities but in a lightweight “developers” oriented approach.

29. © 2014 Forrester Research, Inc. Reproduction Prohibited29 SVT: What’s next › All vendors lack a strong modeling simulation environment to simplify the development and consumption of virtual assets. Improving ease of use › Platform and protocols: REST › Deeper integration with DevOps and SDLC › Improved cloud (hybrid, private, public, IaaS, PaaS, and SaaS) › Self-service

30. © 2014 Forrester Research, Inc. Reproduction Prohibited30 Selected Forrester Research › March 4, 2014, “Modern Application Delivery Drives Digital Business Success” › February 5, 2014, “How Can You Scale Your Agile Adoption?” › January 27, 2014, “The Forrester Wave™: Service Virtualization And Testing Solutions, Q1 2014” › January 15, 2013, “Consistent Performance In Agile Teams Must Include Testing”

31. Thank you Diego Lo Giudice +39 340.771.0775 dlogiudice@forrester.com http://blogs.forrester.com/diego_lo_giudice Twitter: @dlogiudice

32. © 2014 Forrester Research, Inc. Reproduction Prohibited32 Our Agile and Lean playbook has a wealth of additional information on Agile And soon coming modern application delivery playbook: Agile, ALM, continuous delivery. Stay tuned!

33. © 2014 Forrester Research, Inc. Reproduction Prohibited33 Register online for Forrester’s forum for technology management leaders UNLEASH YOUR DIGITAL BUSINESS › May 5 to May 6 in Orlando, FL › June 12 to June 13 in London “This forum will provide technology management leaders the direction they need to help their firms transform into thriving digital businesses that embrace change as a matter of routine.”