Breaching Barriers to Continuous Delivery with Automated Test Gates SOFTWARE QUALITY CONFERENCE PACIFIC NW Breaching Barriers to Continuous Delivery with Automated Test Gates Chris STRUBLE OctOBEr 18, 2016 PNSQC ™

Key points What is Continuous Delivery and why is it important? SOFTWARE QUALITY CONFERENCE Key points PACIFIC NW What is Continuous Delivery and why is it important? Barriers to Continuous Delivery How we breached the barriers at Vertafore Helpful delivery patterns we learned that anyone can use PNSQC ™

What is Continuous Delivery? SOFTWARE QUALITY CONFERENCE PACIFIC NW A set of principles and practices to reduce the cost, time, and risk of delivering incremental changes to users -- Jez Humble PNSQC ™

You might be Continuously Delivering if: SOFTWARE QUALITY CONFERENCE PACIFIC NW You automated your infrastructure You automated your deployments You have a visible delivery pipeline You run automated tests in all environments (including Production) You COULD release your code with quality at any time (even if you don’t) PNSQC ™

Drivers to Continuous Delivery SOFTWARE QUALITY CONFERENCE Drivers to Continuous Delivery PACIFIC NW Many companies deliver multiple times a day Google, Facebook, Amazon, NetFlix Everyone has mobile devices that get updates frequently and automatically Companies that can release more quickly and reliably with quality have a competitive advantage. PNSQC ™

Barriers to Continuous Delivery SOFTWARE QUALITY CONFERENCE Barriers to Continuous Delivery PACIFIC NW Requires multiple skill sets Development Infrastructure automation System administration Difficult to find people with all the skills Few end-to-end solutions available Vendor-specific or build in the cloud Business context PNSQC ™

Continuous Delivery At Vertafore Case Study SOFTWARE QUALITY CONFERENCE PACIFIC NW Continuous Delivery At Vertafore PNSQC ™

About Vertafore SOFTWARE QUALITY CONFERENCE PACIFIC NW Vertafore is an intermediary in the insurance industry, providing information solutions for insurance agencies, carriers, managing general agents, and state agencies. The company’s products touch every point in the distribution channel. Our engineers support dozens of products that combine to make the broadest and most complete portfolio of software in the insurance industry. http://www.vertafore.com Headquarters in Bothell, Washington. PNSQC ™

About me SOFTWARE QUALITY CONFERENCE PACIFIC NW Chris Struble Senior Software Engineer in Test with over 20 years experience Now working in build, deployment, and infrastructure automation Taught internal classes on automation with Selenium, Jenkins and Chef I live in Renton, Washington with my wife, two teenagers and three cats I play computer strategy games, reading history and historical fiction. PNSQC ™

Favorite TV Series/Books (The Last Kingdom) SOFTWARE QUALITY CONFERENCE PACIFIC NW “Like Game of Thrones, but real.” PNSQC ™

About the project Assembled a team of 6-8 people Started in 2013 SOFTWARE QUALITY CONFERENCE PACIFIC NW Assembled a team of 6-8 people Started in 2013 SDETs, QAs, System Admins Tasked to implement a continuous delivery system for over 20 products Dedicated team, not responsible for product development or testing Motivation: Releases were painful and had a high failure rate PNSQC ™

Barriers to the project SOFTWARE QUALITY CONFERENCE Barriers to the project PACIFIC NW Government regulations require us to limit who has access to Production (e.g HIPAA) Organization walls between Development and Operations are rigorously maintained Customer agreements limit when we can release updates to Production Changes to Production must be approved by a change review board No product source code in the cloud PNSQC ™

How we implemented Continuous Delivery SOFTWARE QUALITY CONFERENCE How we implemented Continuous Delivery PACIFIC NW Jenkins – continuous integration server Build jobs – create a binary artifact (zip, iso, tgz) Deploy jobs – deploy artifact to environment Test jobs – run tests (integration, UI) Promotion jobs – promote an artifact if all the automated tests for it have passed PNSQC ™

How we implemented Continuous Delivery SOFTWARE QUALITY CONFERENCE How we implemented Continuous Delivery PACIFIC NW Artifactory – binary artifact repository Development repository – to store built artifacts Release repository – to store promoted artifacts Only Promoted artifacts can ever be released PNSQC ™

How we implemented Continuous Delivery SOFTWARE QUALITY CONFERENCE How we implemented Continuous Delivery PACIFIC NW Chef – infrastructure automation Manages configuration of servers Jenkins Artifactory Standardized machine types PNSQC ™

SOFTWARE QUALITY CONFERENCE Automated test gates PACIFIC NW Criteria that must be met for an artifact to be promoted Example 1 : Gate Type: Unit Gate Pass Score (percentage): 100 Gate Results Type: JUnit Example 2: Gate Type: UI Gate Pass Score (percentage): 95 Gate Results Type: MSTest PNSQC ™

Why automated test gates are awesome SOFTWARE QUALITY CONFERENCE Why automated test gates are awesome PACIFIC NW Evaluate automatically in seconds (Jenkins job) Development teams can decide which gates to use and the criteria Teams very quickly required a pass score of 100% for almost all their gates Manual gates where it is preferred for a person to make the final decision PNSQC ™

And then we encountered a barrier… SOFTWARE QUALITY CONFERENCE PACIFIC NW PNSQC ™

We could not deploy directly to Production… SOFTWARE QUALITY CONFERENCE We could not deploy directly to Production… PACIFIC NW Vertafore has a Staging environment to practice releases to Production Production and Staging are both network isolated Web requests initiated from outside these environments are restricted Jenkins was not allowed to deploy directly into these environments PNSQC ™

…so we found a way to deploy indirectly SOFTWARE QUALITY CONFERENCE …so we found a way to deploy indirectly PACIFIC NW We placed Artifactory cache servers inside Staging and Production Requests for promoted artifacts are sent out every 10 minutes Only promoted artifacts in the Release repository are replicated We also placed Jenkins servers inside Staging and Production Release engineers use Jenkins build job name and build number Manual run during releases only, no automatic triggering PNSQC ™

How our delivery pipeline works SOFTWARE QUALITY CONFERENCE How our delivery pipeline works PACIFIC NW PNSQC ™

SOFTWARE QUALITY CONFERENCE The Results PACIFIC NW Over 200 people are using our pipeline, with over 800 Jenkins jobs Currently building over 8000 artifacts a month. Our pipeline has delivered at least one release a month for over 2 years, with an average of 10 products each release, with 100% successful releases. Deployment failures on release nights have declined enormously. An emergency release can be delivery to Production under 30 minutes. PNSQC ™

Patterns for continuous delivery SOFTWARE QUALITY CONFERENCE Patterns for continuous delivery PACIFIC NW Separate build and deployment code Standardize machine types Separate and promote artifacts Automate promotion decisions (e.g. automated test gates) PNSQC ™

Patterns for continuous delivery SOFTWARE QUALITY CONFERENCE Patterns for continuous delivery PACIFIC NW Automate artifact cleanup Train your users continuously Automate rule compliance PNSQC ™

SOFTWARE QUALITY CONFERENCE Final thoughts PACIFIC NW Delivering software to customers is not an optional feature. You have to do it. You might as well do it well. Don’t let barriers stop you from transforming the parts of your delivery pipeline that you can transform today. Tomorrow, when better tools will be available, the knowledge you gained will help you understand better how to deliver in the future. PNSQC ™

Acknowledgements SOFTWARE QUALITY CONFERENCE PACIFIC NW Thanks to colleagues and former colleagues who contributed to this work: Raul Alvarez, Cody Dockens, Saratha Prabu, Rajani Tadanki, David Echols, Rasheed Usman, Lowell Young, Cameron Straka, Ebencilin Chandradhas, Venkat Chilakala, Denzil Dwelle, Tom Sawin, Rita McCann, George Tsang PNSQC ™

Contact Me Chris Struble Senior Software Development Engineer in Test SOFTWARE QUALITY CONFERENCE PACIFIC NW Chris Struble Senior Software Development Engineer in Test Vertafore Email: cstruble@vertafore.com LinkedIn: https://www.linkedin.com/in/chrisstruble Blog: http://opentesting.blogspot.com/ Author Page: http://www.pnsqc.org/chris-struble/ PNSQC ™