Continuous Validation – An Approach to production Assurance in DevOps Shailja Sehgal, Sr Principal QE Engineer, Karanam Ramadevi, Sr Principal QE Engineer Manhattan Associates.

Abstract IT industry is going through an “Inflection point” where majority of companies are looking for ways to bring innovation faster to their customers and provide them the competitive advantage. One of the most popular trends that is evolving is re-architecting their large monolithic enterprise applications (esp. in Retail & eCommerce) and adopting “micro services” architecture in order to enable “Continuous delivery, and faster time to market”. This architecture along with Cloud/SaaS based offering helps achieve high availability, easy upgradability and deployability. One recent research shows 66% of these companies is adopting DevOps model and there is fundamental change in testing paradigm where Developer/SDETs are merged into testing cycle right at the beginning to implement concepts like Shift left, TDD, early and continuous automation etc. Some of the testing practices evolving in this space are: Unit Level Contract based Integration within containers While there are many benefits of going in this direction but at the same time it is posing basic challenges like End to end product view can be minimized Balance between “test as specified” vis-à-vis “test as expected” Balance between speed of delivery vis-à-vis quality To achieve “faster time to market”, the traditional way of doing mammoth system testing cycle approach to stabilize a product will not work anymore. Hence, in addition, to continuous delivery and continuous integration practices, organizations need to adopt a more leaner and flexible “continuous validation” approach. Also, this approach needs to bring in customer perspective fairly early in the cycle. In US, 40% of top retailers are Manhattan Associate’s customer. Hence it is imperative that our “continuous validation” approach has our customer at the center. And that is what the paper talks about how we simulate/mimic “A Live automated Model company by running a 24X7 in production like environment where the target is to “Use” the product like our customer does. Focus is on mimicking real end to end key customer operations covering customer data, profile, volume, geographies, demographics etc. In parallel, we have a live monitoring system focusing on outages, leakages, system anomalies, deadlocks etc. The clear focus is we need to move from “reactive” mode of response to a “proactive” mode to ensure our customers uptime, scalability and availability is adhered to.

Micro service based Architecture –Why! Need of continuous delivery Enables faster time to market Enables high availability, clean code Shortening deliveries Scalable, Reliable Independently deployable Micro services based architecture is emerging as popular choice for many Enterprise applications especially cloud based models

Poses a need of Continuous Validation! A challenge for Quality in DevOps! End to end product view can be minimized Balance between “test as specified” vis-à-vis “test as expected” Balance between speed of delivery vis-à-vis quality Focus is on making component work Silo-ness happening - Testing a service in isolation If no testing, wrong gets visible very quickly Traditional Approach may not work ! Above all a need, “Shifting Right into production” Poses a need of Continuous Validation!

Shift Right to "Assure Production” Design a “24X7 Live Model Company” by simulating real customer business scenarios with key focus in mind to become the “first customer” of our product thereby Use the product(s) as a customer does Adopt testing strategies to a continuous delivery model Automated configuration(s) with robust ‘production like’ monitoring

Our Approach to “Assuring Production” Our “24X7 Live Model Company” covered Day Begin, Day IN and Day end operations completely automated (as shown below) similar to what our customers will do – given below is a simplistic representation of the customer landscape.

How did we do it The testing methodologies adopted was multi-fold: “Understanding customer’s business and their primary flows” Designing business scenarios, to cover 80% of our top 20% customer’s daily business operations. Built a robust automated system with end to end business operations covering end to end business scenarios in a “Day in life of an retail customer” with focus on: Continuous system running 24x7 and should simulate a production like environment Simulate/Integrate 3rd Party /(and/or) Manhattan systems to the model Inject incremental load into the model periodically Bring in randomization logic to build unpredictability Time slicing to schedule jobs to simulate daily business operations right from Inventory operations to order capturing through shipping(as shown below) that happens in one day Live Monitoring Systems – While the business operations are in progress, in a SaaS model, there is a mandatory requirement to monitor the systems at different levels. In addition to running the live system our solution focusses on this aspect to a great extent. Some critical monitors are

Measurable impact Some of key impacts that our tool has brought in: The 24X7 simulated systems acted as a constant Quality enablers as it helped us in achieving: Continuous uptime validation of business operations covering 70% of daily business operations Simulated outage scenarios which helped us in ensuring uptime is 99% Improvement in upgradability from 50-90% Actionable Analytics: Through constant monitoring, we started measuring $ value impact internally Enhances Customer Experience –helped us to assess our systems in terms of “Production Readiness” Helped in achieving scalability by designing scenarios and volume data for Peak season covering Cyber Sale Helped identifying issues like performance, failure tolerance, and user experience, new and unexpected usage scenarios

Author Biography Shailja Sehgal is a Sr Princicpal QE in the R&D organization at Manhattan Associates India With 14+ years of strong and proven expertise in Mobility, Supply Chain Retail, OMNI channel, Automation (Jmeter, API Integration, SeeTest for mobility). Active member of Mobile CoE and partnering with other organizations for sharing best practices on Mobility. Expertise in QA testing for various supply chain products. Good understanding and exposure of JMeter, REST Api, Selenium, SeeTest, TestNG, and RDBMS and worked on different databases like Oracle, DB2, SQL Server.Worked on integration and functional, user interface, regression testing and smoke testing. Expertise in defect tracking and bug reporting tools. Actively contributing to external forums by presenting white papers and formal presentations. Karanam Ramadevi is Sr Princicpal QE in the R&D organization at Manhattan Associates India With 13+ years of strong and proven expertise in Manual and Automation Testing on Supply Chain Retail, OMNI channel (Cloud platform), Automation(Selenium, Java & JMeter and API Integration). Expertise in QA testing for various supply chain products. Good understanding and exposure of Java, JMeter, REST Api, Selenium, TestNG, Jenkins and RDBMS and worked on different databases like Oracle, MySQL, SQL Server. Worked on integration and functional, user interface, regression testing and smoke testing. Expertise in defect tracking and bug reporting tools. Actively contributing to external forums by presenting white papers. Strong expertise on Product Quality Assurance, product testing and Process Engineering

Thank You!!!