1. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing the continuous delivery path A tale of two teams

2. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp What’s this all about? 2 Continuous Delivery Done Well

3. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp What’s this (Continuous Delivery) all about? 3 “... producing valuable software in short cycles and ensure that the software can be reliably released at any time ” - https://en.wikipedia.org/wiki/Continuous_delivery

4. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Continuous doughnuts 4 Fast feedback Risk Reduction T: 0:00:000T: 2:00:000

5. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp There’s no (CD) silver bullet 5 1 (php) monolith 50 deploys / day 1.5 billion page views / month http://goo.gl/XIB29i >600 java microservices >100 deploys / day ~2 billion requests per day http://goo.gl/ZJBiK6

6. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Prefer Continuous Improvement over Continuous Delivery ▪.Net monolith ▪ 26 -> 24 -> 34 releases per year ▪ https://goo.gl/KF9xXw 6

7. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp 7 People first, practices/process second, products last

8. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp.Net monolith team - how they roll 8 ▪ Private sector ▪ Significant legacy code ▪ High daily traffic (~100 million visits) ▪ ~8 product teams ▪ Weekly deployments: – Pick release candidate – Regression test – Dog food – Deploy

9. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Scala microservices team - how they roll 9 ▪.Gov - HMRC Digital ▪ 2 years old (greenfield) ▪ Calendar spikes (Visits peak is ~4 million) ▪ >30 product teams ▪ Multiple deployments per day – Teams “own” their own services – “You build it, you run it” - https://goo.gl/DyfeOf

10. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Continuous Delivery Done Well Patterns for successful practice 10

11. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #1 - Healthy CI - mind & body 11

12. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #1 - Healthy CI - mind & body 12 Almost green!

13. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #2 - Testing as an activity, not a phase 13 Product Owner Define Dev Build QA Test Web Ops Deploy

14. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #2 - Testing as an activity, not a phase 14 Product Owner Define Dev Build QA Test Web Ops Deploy Test

15. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #3 - Healthy automated tests 15 Unit Tests API UI

16. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #3 - (Un)healthy automated tests 16 API UI Unit Tests The “Hour Glass”

17. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp UI Best practice #3 - (Un)healthy automated tests 17 API The “Ice Cream Cone” Unit Test s

18. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #3 - Healthy automated tests 18 Unit Tests API UI The “Tear Drop”

19. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #4 - Low (manual) cost deployment (and rollback) 19 VS.

20. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice #5 - Metrics, monitoring and alerting 20 http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318 “Big data” Social media User behaviour Business metrics Service performance Service errors Disk and Network Monitor all things!

21. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Metrics, monitoring and alerting - getting it right 21 Detect Alert Respond Display Analyse Time & Pain increasing!

22. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Metrics, monitoring and alerting - getting it wrong 22 Detect Respond Display Analyse Time & Pain increasing! Alert

23. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Metrics, monitoring and alerting - where and how? 23 Detect Alert Respond Display Analyse

24. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Metrics, monitoring and alerting - stories from the trenches 24 Detect Alert Respond Display Analyse

25. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Best practice summary 1. Healthy CI (mind and body) 2. Testing as an activty, not a phase 3. Tear drop shape automated tests (focus on behaviour, with loose coupling) 4. Low (manual) cost deployment and rollback 5. Metrics, monitoring and reporting 25 Product Owner Define Dev Build QA Test Web Ops Deplo y Test Unit Tests API UIUI Detect Alert Respond Display Analyse

26. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Shared Pain 26 Continuous Delivery Done Well

27. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Shared pain #1 - Zero down-time releases 27 https://en.wikipedia.org/wiki/VO2_max

28. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Shared pain #2 - Manual deployment steps 28 https://en.wikipedia.org/wiki/Tailorhttps://en.wikipedia.org/wiki/Zara_(retailer) VS.

29. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Shared pain summary 1. Zero down time releases 2. Manual deployment steps 29 VS.

30. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Team specific accelerators 30 Continuous Delivery Done Well

31. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Accelerators - 5 why’s 31 ● Create a timeline. ● Gather those involved. ● Walk through the timeline. ● Choose an entry point event: ○ Event ○ Why? ○ Process improvement ● Repeat x5 (ish)

32. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Accelerators - 5 why’s - example 32 EventWhy?Improvement Error threshold alert triggered. Calls to service X were timing out. Implement failover / circuit breaker. Calls to service X were timing out. Queries run by Service X were taking too long. Alert on long running queries. Queries run by Service X were taking too long. Index was missing for a new field....

33. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Accelerators - automated release notes - the problem #1 33 Much confusion and pain

34. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Accelerators - automated release notes - the problem #2 34 Agile Project Management Tool Source control system Features Stories Defects Login Reporting Payments API’s

35. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Accelerators - automated release notes - the solution 35 Agile Project Management Tool Source control system Features Stories Defects Login 1. Identify source changes 2. Identify stories/defects 3. Identify areas 4. Group by story/defect 5. For each, summarise areas and authors 6. Group by: a. What’s done b. What’s in progress c. Untracked work Reporting Payments API’s

36. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Team specific accelerators 36 Continuous Delivery Done Well

37. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Accelerators - teams “own” their own microservices 37 ▪ “You build it, you run it” - https://goo.gl/DyfeOf https://goo.gl/DyfeOf ▪ Prefer service over library dependencies

38. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Accelerators - backward compatible API’s (principle of least surprise) 38 https://en.wikipedia.org/wiki/VO2_max Production Staging QA Dev Everyone shares (and cares)

39. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp Team specific accelerators summary Monolith: 1. 5 -why’s 2. Automated release note Microservices: 1. Microservice dependencies, not library dependencies 2. Backward compatible API’s 39 Agile Project Management Tool Source control system Features Stories Defects Login Reporting Payments API’s

40. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp References and resources 40 ▪ Build Quality In (promo code - LP2015 - https://goo.gl/ZRuQKw) Continuous Delivery Implementing Lean Software Development Release It

41. © Equal Experts UK Ltd and lyndsayp ltd 2015 @EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Thank You Twitter @EqualExperts @lyndsp LinkedIn linkedin.com/company/equal-experts uk.linkedin.com/in/lyndsp UNITED KINGDOM +44 203 603 7830 helloUK@equalexperts.com Equal Experts UK Ltd 30 Brock Street London NW1 3FG INDIA +91 20 6607 7763 helloIndia@equalexperts.com Equal Experts India Private Ltd Office No. 4-C Cerebrum IT Park No. B3 Kumar City, Kalyani Nagar Pune, 411006 Web www.equalexperts.com www.lyndsayp.com CANADA +1 403 775 4861 helloCanada@equalexperts.com Equal Experts Devices Inc 205 - 279 Midpark way S.E. T2X 1M2 Calgary, Alberta PORTUGAL + helloPortugal@equalexperts.com Equal Experts Portugal Rua Tomás da Fonseca - Torres de Lisboa Torre G, 5º Andar 1600-209 Lisboa Thank You