1. Challenges Faced by QA Team in DevOps Testing
Shanthala Bhat & Manager
Manjunath A Padmanabha & Senior Test Lead
PwC

2. Abstract
The DevOps is a concept for software development and delivery process that highlights collaboration between software development and operation professionals. The popularity of DevOps has grown in recent years as it helps in releasing a software rapidly, more frequently and reliably. The DevOps practice is being more often adopted for Microservice based architecture application as each service is small, easily tested and deployable through continuous integration.
As per the survey published in January 2017 by RightScale, DevOps adoption has increased from 66% in 2015 to 74% in As we move on its important to know the different challenges faced while implementing this concept from testing point of view. Hence this whitepaper focuses on providing information on different challenges faced while testing Microservice architecture based application implemented with DevOps concepts. The testing community and test architects would find this paper very handy.

3. Business Case
This business case is related to one of the world leader in financial technology solutions which provides payment solutions for credit cards, debit cards, electronic payments, and check-related services. Their current system was outdated, not scalable and increased with operational expenses. The team was looking for setting up new system fulfilling following goals of business expansion.
Faster time to market
Allow easy, agile expansion for other countries, and for future acquisitions
Reusable services supporting other business workloads
Better deployment frequency
Improved end user satisfaction
Increased ability to build the right product by fast experimentation

4. Business Solution
In order to achieve the above goals, the team decided to go with a concept of agile methodology, DevOps, Microservice architecture style and BDD way. The architecture goal was to build an application enabled through use of cloud native capabilities, Microservices and Continuous Delivery. The application was comprised of a number of Microservices to avoid the problems of traditional monolithic applications as they are modelled around business domains.
This implementation achieved the desired goals around availability and maintainability, Microservices exhibited below traits:
discoverable using a service registry
software load balancers and circuit breakers to protected from external dependency failure be independently deployable with provision for zero downtime
auto scaled up to meet demand and to scale back when demand recedes to optimize cost
Be exposed as a RESTful API over HTTPS
The communication between each Microservice happened either through HTTPs between services or Message Queuing between components

5. Testing Approach for Business Case

6. Quality Engineering Approach

7. Cultural Shift Skilled Resources Testing Mindset CI Readiness
Challenges Faced - Generic
Cultural Shift
Training
RASCI
Focus on delivery of Business Value
Skilled Resources
SME (DevTest)
Training (All-rounder)
Involving Developers
Testing Mindset
Exploratory Tests
Testing along with BA Team
Testing across SCRUM teams
CI Readiness
Framework Readiness
Tools integration with CI
Dashboards for Result analysis
Code Quality
Frequent design updates
Coding guidelines
Test Coverage through unit tests

8. Automation Testing Performance Testing Security Testing
Challenges Faced – Testing Types
Automation Testing
Endpoint URI – Repository outside Codebase
Junit/Mockito – Repository within Codebase
Nightly script runs
Performance Testing
Simple Performance Test Environment with Quick Tests
Debugging CI Jobs through tools
Result analysis through integrated tools
Security Testing
Identify and adopt SAST tool at sprint level
Plan for DAST at early stage
Automation for DAST

9. Conclusion
While implementing DevOps concept for one of the payment application, the testing team faced lot of challenges and as a team we were able to come up with appropriate solution. The same has been tried to outline in the paper. The solution are very generic in nature. One may have different challenges and solutions as well considering nature of application, different Dev/Operations/QA team they are working with. We hope this whitepaper would be helpful for the testing team who are yet to start working on testing in DevOps concept.

10. References & Appendix
Section
Reference Link
Security Testing Challenges
ops-testing/?utm_source=online- advertising&utm_medium=q317-glossary-devops- testing- google&utm_source=google&utm_medium=cpc&g clid=EAIaIQobChMIkbmfzq6G1gIV0EsNCh1W- gLGEAAYBCAAEgJSdfD_BwE
Performance Testing Challenges
challenges-when-performance-testing-continuous- delivery-pipelines

11. References & Appendix
Terminology
Definition QA
Quality Assurance
SAST
Static Application Security Testing
DAST
Dynamic Application Security Testing
POC
Proof Of Concept CI
Continuous Integration IT
Information Technology
SME
Subject Matter Expert
CISQ
Consortium for IT Software Quality
OMG
Object Management Group
CDN
Content Delivery Network
KPI
Key Performance Indicator
API
Application Programming Interface

12. Author Biography
Shanthala Bhat is working for PwC Limited from last 5 years and currently designated as “Manager”. She holds a degree in E&E engineering from UVCE.
Has experience of 12+ years in software testing that includes Testing web applications (server-client network), Web services, Performing Multilingual Testing, Product Testing, Test Management and Test Automation. As a Manager has an expertise in Test Management, development of Test Strategy, test cases, Defect Reports, scope management, risk management and different Test Metrics for V&V activities. Also have knowledge on test automation tools like QTP, VSTS 2012 Ultimate, Selenium, LoadRunner and Telerik.
She has written white papers on subjects like Agile Testing, Web Trust & Safety and Framework and Process driven Outsourcing which was selected in one of the STeP-IN Summit and participated in QAI STC-2016, STeP-IN Test Autothoon 2017 Contest. She is fun loving person and hobbies include reading magazines, solving cross words, puzzles and listening to music.
You can reach her at –

13. Co-Author Biography
Manjunath Ananthapadmanabha is a Senior Test Lead with Pricewaterhouse Coopers (PwC) SDC, Bangalore. He has about 11 years of software testing experience handling projects in specialized testing areas such as Performance, Database, and DW/ETL across numerous industry sectors, including investment banking, healthcare, insurance and telecom. His areas of specialization include Performance testing, continuous integration and test data management.
Manjunath has an engineering degree in electronics and communication from Visvesvaraya Technological University, India. Manjunath can be reached at

14. Thank You!!!