© Equal Experts UK Ltd and lyndsayp simple software solutions to big business problems. Making Software. Better. Smoothing the continuous delivery path A tale of two teams

© Equal Experts UK Ltd and lyndsayp What’s this all about? 2 Continuous Delivery Done Well

© Equal Experts UK Ltd and lyndsayp What’s this (Continuous Delivery) all about? 3 “... producing valuable software in short cycles and ensure that the software can be reliably released at any time ” -

© Equal Experts UK Ltd and lyndsayp Continuous doughnuts 4 Fast feedback Risk Reduction T: 0:00:000T: 2:00:000

© Equal Experts UK Ltd and lyndsayp There’s no (CD) silver bullet 5 1 (php) monolith 50 deploys / day 1.5 billion page views / month >600 java microservices >100 deploys / day ~2 billion requests per day

© Equal Experts UK Ltd and lyndsayp Prefer Continuous Improvement over Continuous Delivery ▪.Net monolith ▪ 26 -> 24 -> 34 releases per year ▪ 6

© Equal Experts UK Ltd and lyndsayp 7 People first, practices/process second, products last

© Equal Experts UK Ltd and lyndsayp monolith team - how they roll 8 ▪ Private sector ▪ Significant legacy code ▪ High daily traffic (~100 million visits) ▪ ~8 product teams ▪ Weekly deployments: – Pick release candidate – Regression test – Dog food – Deploy

© Equal Experts UK Ltd and lyndsayp Scala microservices team - how they roll 9 ▪.Gov - HMRC Digital ▪ 2 years old (greenfield) ▪ Calendar spikes (Visits peak is ~4 million) ▪ >30 product teams ▪ Multiple deployments per day – Teams “own” their own services – “You build it, you run it” -

© Equal Experts UK Ltd and lyndsayp Continuous Delivery Done Well Patterns for successful practice 10

© Equal Experts UK Ltd and lyndsayp Best practice #1 - Healthy CI - mind & body 11

© Equal Experts UK Ltd and lyndsayp Best practice #1 - Healthy CI - mind & body 12 Almost green!

© Equal Experts UK Ltd and lyndsayp Best practice #2 - Testing as an activity, not a phase 13 Product Owner Define Dev Build QA Test Web Ops Deploy

© Equal Experts UK Ltd and lyndsayp Best practice #2 - Testing as an activity, not a phase 14 Product Owner Define Dev Build QA Test Web Ops Deploy Test

© Equal Experts UK Ltd and lyndsayp Best practice #3 - Healthy automated tests 15 Unit Tests API UI

© Equal Experts UK Ltd and lyndsayp Best practice #3 - (Un)healthy automated tests 16 API UI Unit Tests The “Hour Glass”

© Equal Experts UK Ltd and lyndsayp UI Best practice #3 - (Un)healthy automated tests 17 API The “Ice Cream Cone” Unit Test s

© Equal Experts UK Ltd and lyndsayp Best practice #3 - Healthy automated tests 18 Unit Tests API UI The “Tear Drop”

© Equal Experts UK Ltd and lyndsayp Best practice #4 - Low (manual) cost deployment (and rollback) 19 VS.

© Equal Experts UK Ltd and lyndsayp Best practice #5 - Metrics, monitoring and alerting 20 “Big data” Social media User behaviour Business metrics Service performance Service errors Disk and Network Monitor all things!

© Equal Experts UK Ltd and lyndsayp Metrics, monitoring and alerting - getting it right 21 Detect Alert Respond Display Analyse Time & Pain increasing!

© Equal Experts UK Ltd and lyndsayp Metrics, monitoring and alerting - getting it wrong 22 Detect Respond Display Analyse Time & Pain increasing! Alert

© Equal Experts UK Ltd and lyndsayp Metrics, monitoring and alerting - where and how? 23 Detect Alert Respond Display Analyse

© Equal Experts UK Ltd and lyndsayp Metrics, monitoring and alerting - stories from the trenches 24 Detect Alert Respond Display Analyse

© Equal Experts UK Ltd and lyndsayp Best practice summary 1. Healthy CI (mind and body) 2. Testing as an activty, not a phase 3. Tear drop shape automated tests (focus on behaviour, with loose coupling) 4. Low (manual) cost deployment and rollback 5. Metrics, monitoring and reporting 25 Product Owner Define Dev Build QA Test Web Ops Deplo y Test Unit Tests API UIUI Detect Alert Respond Display Analyse

© Equal Experts UK Ltd and lyndsayp Shared Pain 26 Continuous Delivery Done Well

© Equal Experts UK Ltd and lyndsayp Shared pain #1 - Zero down-time releases 27

© Equal Experts UK Ltd and lyndsayp Shared pain #2 - Manual deployment steps 28 VS.

© Equal Experts UK Ltd and lyndsayp Shared pain summary 1. Zero down time releases 2. Manual deployment steps 29 VS.

© Equal Experts UK Ltd and lyndsayp Team specific accelerators 30 Continuous Delivery Done Well

© Equal Experts UK Ltd and lyndsayp Accelerators - 5 why’s 31 ● Create a timeline. ● Gather those involved. ● Walk through the timeline. ● Choose an entry point event: ○ Event ○ Why? ○ Process improvement ● Repeat x5 (ish)

© Equal Experts UK Ltd and lyndsayp Accelerators - 5 why’s - example 32 EventWhy?Improvement Error threshold alert triggered. Calls to service X were timing out. Implement failover / circuit breaker. Calls to service X were timing out. Queries run by Service X were taking too long. Alert on long running queries. Queries run by Service X were taking too long. Index was missing for a new field....

© Equal Experts UK Ltd and lyndsayp Accelerators - automated release notes - the problem #1 33 Much confusion and pain

© Equal Experts UK Ltd and lyndsayp Accelerators - automated release notes - the problem #2 34 Agile Project Management Tool Source control system Features Stories Defects Login Reporting Payments API’s

© Equal Experts UK Ltd and lyndsayp Accelerators - automated release notes - the solution 35 Agile Project Management Tool Source control system Features Stories Defects Login 1. Identify source changes 2. Identify stories/defects 3. Identify areas 4. Group by story/defect 5. For each, summarise areas and authors 6. Group by: a. What’s done b. What’s in progress c. Untracked work Reporting Payments API’s

© Equal Experts UK Ltd and lyndsayp Team specific accelerators 36 Continuous Delivery Done Well

© Equal Experts UK Ltd and lyndsayp Accelerators - teams “own” their own microservices 37 ▪ “You build it, you run it” ▪ Prefer service over library dependencies

© Equal Experts UK Ltd and lyndsayp Accelerators - backward compatible API’s (principle of least surprise) 38 Production Staging QA Dev Everyone shares (and cares)

© Equal Experts UK Ltd and lyndsayp Team specific accelerators summary Monolith: why’s 2. Automated release note Microservices: 1. Microservice dependencies, not library dependencies 2. Backward compatible API’s 39 Agile Project Management Tool Source control system Features Stories Defects Login Reporting Payments API’s

© Equal Experts UK Ltd and lyndsayp References and resources 40 ▪ Build Quality In (promo code - LP Continuous Delivery Implementing Lean Software Development Release It

© Equal Experts UK Ltd and lyndsayp simple software solutions to big business problems. Making Software. Better. Thank LinkedIn linkedin.com/company/equal-experts uk.linkedin.com/in/lyndsp UNITED KINGDOM Equal Experts UK Ltd 30 Brock Street London NW1 3FG INDIA Equal Experts India Private Ltd Office No. 4-C Cerebrum IT Park No. B3 Kumar City, Kalyani Nagar Pune, Web CANADA Equal Experts Devices Inc Midpark way S.E. T2X 1M2 Calgary, Alberta PORTUGAL + Equal Experts Portugal Rua Tomás da Fonseca - Torres de Lisboa Torre G, 5º Andar Lisboa Thank You