10 Point Success Strategy in Crowd Testing Sree Ram Prasad, Manager Nilesh Kumar, Sr. Consultant Deloitte Consulting

Abstract Agenda Crowd Testing – Brief Refresher With Increasing digitization in every aspect of life, breakneck pace of development, ever shortening shelf-life of mobile apps, and more focus on user experience, it is more pertinent than even to look at Crowd Testing as our best ally in Quality Control. To set the context, we’ll briefly touch upon Crowd testing in general, benefits of crowd testing and types of testing which can be targeted through Crowd testing, and some of the major challenges. We intend to presents a 10 point agenda to overcome all the major challenges, and introduce innovative techniques to reap definite success from Crowd Testing projects. Agenda Crowd Testing – Brief Refresher Benefits of Crowd Testing 5 Drivers for Crowd Testing Targeted Testing Types Major Challenges Success Strategy Caution with Crowd Testing References & Appendix Author Biography

Crowd Testing – Brief Refresher Collective Cognizance Real World Testing Diversity Focus on consumer experience Localization Cost Effective Easy to scale up or down Shorter time to market

5 Drivers for Crowd Testing

Diversified Tester Base Collective Cognizance Benefits of Crowd Testing Diversified Tester Base Extensive Coverage Cost Effective Faster Turnover Collective Cognizance Feedback

Targeted Testing Types Localization Testing Performance Testing Beta Testing Security Testing Web/Mobile Testing Usability and Functional Testing

Major Challenges in Crowd Testing

10 Point Success Strategy Train the Tester Recorded Demos Mock Screens in Design Document Micro-learning 1 Focus on Coverage Crowd Re-orientation Real Time Coverage Report Crowd Feedback to Drive 2

10 Point Success Strategy cont.… Communicate to Succeed Chat-bots to Answer Generic Queries Busting Language Barriers Multiple Channels to Disseminate Information Maintaining Confidentiality & Building Trust Non-Disclosure Agreements Meetups Four Colour Ethics Meter 3 4

10 Point Success Strategy cont.… Motivated Testers Interactive and Engaging Test Management Reward Bonus Points for Active Users Crowd Contests 5 Address Technological Challenges Version Control Test Management Confidentiality and security Feature Toggle, App vs APIs, Web Hosted App 6

10 Point Success Strategy cont.… Mute False alarms Crowd Awareness Surveys Crowd Evaluation Matrices Correct Test Data Sample Space Defect Logs and Diagnostic report Mandatory Execution Evidence Deriving value from Bugs Defect Guidelines Pro-active Exception Handling Bug Revalidation Value Proposition of Bug Analytics Based Defect Assessment 7 8

10 Point Success Strategy cont.… Intelligent Crowd Sampling Qualifier Round for Crowd Analytics Based Crowd Sampling Crowd Skills Survey Device Verification 9 Cohesive Team Inclusion Brand Loyalty Long Term Relations Crowd Satisfaction Survey 10

Caution with Crowd Testing Crowd Testing is not an ultimate solution to all the testing needs. Engagement in Crowd Testing should be exercised with utmost caution, or be rather avoided when Dealing with highly sensitive and confidential data Possibility of security compromise and intellectual property theft Possibility of Exposure of critical and core business logic Unrealistic expectations from Crowd Testing Automation has better ROI than Crowd Testing

References & Appendix https://hbr.org/2013/04/using-the-crowd-as-an-innovation-partner https://opensignal.com/reports/2015/08/android-fragmentation/ http://blog.harbinger-systems.com/2016/02/crowdsourced-testing-an-overview/ http://www.qualitestgroup.com https://www.utest.com/ https://www.techgig.com/testingcontest https://www2.deloitte.com/

Author Biography Sree Ram Prasad Bhatiprolu Nilesh Kumar Sree Ram Prasad Bhatiprolu (sbhatiprolu@deloitte.com) Manager, Deloitte USI Sree Ram Prasad Bhatiprolu is Manager with Deloitte Consulting USI, Hyderabad, India. Sree Ram is certified scrum master with 14 years of IT experience, worked across LSHC (Life sciences and Health care), TMT (Technology, Media & Telecommunications) and FSI (finance, securities and insurance) industries. Sree Ram holds Masters degree in software engineering from BITS Pilani. He has about 14 years of experience in leading, managing and delivering testing engagements. He has extensive experience in Automation, Integration Testing, Functional Testing and Performance and Testing. Sree Ram Prasad Bhatiprolu Nilesh Kumar (nilkumar@deloitte.com) Sr. Consultant, Deloitte USI Nilesh Kumar is a Sr. Consultant with Deloitte Consulting USI, Hyderabad, India. He has over 7 years of experience in Software Quality and Assurance. Nilesh has worked on various automation testing tools like QTP, Selenium, RFT and various test management tools. He is ISTQB and PAHM certified professional. Nilesh Kumar

Thank You!!!