1 © 2011 Infosys Ltd. ERP Testing Challenges– Need for a Stable Test Environment Presented At STEP AUTO 2011 – ERP testing Conference – Dec 8, 2011 by A. Raghunath, Senior Project Manager – Package testing Pinaki Sen Sarma, Project Manager – Package testing

2 Table of Contents Nuances of ERP Testing & related discussion Typically how QA environment is provided Environment transition activity Types of environment issues Environment Issue Impacts Recommended activities for improved environment stability Conclusion

3 Nuances of ERP Testing Integration of multiple modules But are these enough? What are the critical success factors other than skill set? Do we take anything for granted? There may be quite a few contributing factors, but for our discussion, we will focus on THE NEED FOR A STABLE TEST ENVIRONMENT The bigger picture – End to end perspective Niche Skill set

4 Typically how QA environment is provided The instances for QA environment (Stage environment) could be created in multiple ways: Copy of past release environment Copy of production Hardware of the QA environment is almost never at par with the production environment because of the cost factor involved!

5 Environment transition activity Environment related verifications done during transition phases are: i.Pre-flow activity ii.IT Verification iii.Smoke Test iv.Ramp-up

6 Types of environment issues and associated impacts a.Concurrent Program (CP) Issue b.TIBCO Issue c.Data-sync Issue d.Performance Issue e.Unplanned Environment Downtimes Teams IssueQADevelopmentInfraDBAs Concurrent Program IssueYes Yes (Occasionally) TIBCO IssueYes No Performance IssueYes NoYes (Occasionally) Data-sync IssueYes NoYes Environment DowntimeYes

7 Environment Issue Impact (Contd..) Environment issues account for the most number of defects in a majority of the releases we have handled this far. Following is the Pareto analysis of one such release:

8 Environment Issue Impact (Contd..) There can be a wide range of issues which fall under the environment issue category :  CDB & Concurrent Program issues constitute 26% of the environment related defects  Many of the CDB issues were pertaining to replication of data from DB1 -> DB2  Less frequent sub root-causes (like Cross Functional Issue, Invalid Trigger, SABRIX Issue etc ) have been clubbed under Misc category.

9 Environment Issue Impact (Contd..) It’s not just the environment issues per-se that affects testing, but also the frequency of recurrence of the same. While some of them are one-time issues, there are quite a few which recur very frequently, resulting in more effort wastage than originally anticipated:

10 Environment Issue Impact (Contd..) We had captured data for effort loss across 2 consecutive releases and following were our findings: Converting the above effort loss into dollar values shows the significant impact of an unstable environment on the overall cost. Quarterly ReleaseTotal Available HrsActual Impacted Hrs Release 1 – Core Projects Release 1 – Project B Release 1 – Regression Total Release Total % Effort Lost for Release 1 15% Release 2 – Core Projects Release 2 – Project B Release 2 – Regression Total Release Total % Effort Lost for Release 2 19% ReleaseTotal QA Cost Actual Loss (%) Actual Loss (‘000s of $$) Release 1700 K15 %105 K Release 2400 K19 %76 K We need a stable test environment!

11 Recommended activities for improved environment stability Strengthening the environment transition process Data-sync verification during instance copy Dedicated Infra/DBA support to monitor environment issues More hardware support for testing instances Robust resolution mechanism for QA issues (w.r.t environment) Creation of focused teams for environment monitoring Separate performance test environment Looking at new paradigms like Cloud computing / Virtualization as options for creating the test instance

12 Conclusion Initial drivers for cost reduction – Offshoring Looking for additional efficiencies – one key area is a stable QA environment Adequate attention, proactive monitoring and other factors can reduce cost between 15 – 20% Newer Paradigms like Cloud computing / virtualization of QA environment will help in driving additional efficiencies.

THANK YOU! © 2011 Infosys Technologies Ltd. 13