1. Thinking Inside the Container: A Continuous Delivery Story Maxfield Stewart Engineering Manager: Riot Games (NOTE: PASTE IN PORTRAIT AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

2. Who’s This Guy?

3. Thinking Inside the Container

4. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

5. A Quick Note on Riot Games

6. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) The Scale of League MORE THAN 67 MILLION MONTHLY ACTIVE PLAYERS MORE THAN 27 MILLION DAILY ACTIVE PLAYERS 7.5 MILLION PEAK CONCURREN T PLAYERS

7. 1.25 Million Builds a Year

8. 10,000 - 14,000 Containers A Week

9. 120 Jobs An Hour

10. A Containerized Build Farm JENKINSSWARM DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker 1.10.3 4 Core/32GB RAM/120GB LVS DRYDOCK BUILD HOSTS SSH Docker API

11. STORY TIME! (aka. Presenting the Problem)

12. 3500+ Unique Build Jobs

13. 650+ Builds an Hour

14. 90+ Build Slaves

15. Expecting 100 More!

16. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Story Time.... Ticket EngineersBuild Team

17. Story Time.... Ticket EngineersBuild Team

18. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) What Did We Want? TEAMS NEEDED TO MOVE FAST PRODUCTS HAD TO OWN THEIR STACK CONFIGURA TION AS CODE

19. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Maybe We Want...

20. “Along Came a Whale” (More Story Time!)

21. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

22. Oh Look! Another Way to Deploy!

23. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Docker For Newbz

25. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer

26. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB”

27. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA”

28. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA” Build Job Queue

29. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA” Build Job Queue

30. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) What Did We Need? CONTAINER RUNNING AS A BUILD SLAVE HOW TO SCALE IT WAY TO HAVE ENGINEERS PROVIDE DOCKERFILES PROFIT?

31. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) A Build Slave Container

32. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Add a Bit of Secret Sauce... Base Slave Image Universal Riot Tools Product/App Specific Build Stuff

33. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) A Real Example

34. Provisioning and Plugins += ?

35. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Of Whales and Plugins MESOS PLUGIN KUBERNETES PLUGIN DOCKER PLUGIN

36. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Of Whales and Plugins MESOS PLUGIN KUBERNETES PLUGIN DOCKER PLUGIN

37. Quick Look

38. Provisioning and Plugins Image Name Label

39. Groovy To the Rescue https://github.com/maxfields2000/dockerjenkins_tu torial/tree/master/jenkinsscripts

40. We Created A Monster

41. We Need to Inspect Our Containers

42. ./harbormaster --registry_user "$REGISTRY_USER" --registry_token "$REGISTRY_TOKEN" --jenkins_user $USER -- jenkins_pass $PASSWORD --jenkins --dockerhost $TEST_DOCKER_HOST "$IMAGENAME" "$CLOUDNAME" "$LABELNAME"

43. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Are We Done?

44. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) But It Needs to Scale!

45. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) If Only...

46. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) If Only...

47. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) But It Needs to Scale! JENKINSSWARM DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker 1.10.3 4 Core/32GB RAM/120GB LVS BUILD HOSTS SSH Docker API

48. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY Image

49. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER Image Name + Jenkins Label

50. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER Runs Tests

51. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER SWARM DOCKERHOST Image Pulled to Swarm

52. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER SWARM DOCKERHOST Groovy API Called Jenkins

53. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Build Job Quick Look

54. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Dockerception

55. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Where to Build Containers? JENKINSSWARM DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker 1.10.3 4 Core/32GB RAM/120GB LVS DRYDOCK BUILD HOSTS SSH Docker API

56. Mai Tai’s On the Beach Build Engineer

57. How Do You Actually Build It?

58. (engineering.riotgames.com)

59. https://github.com/maxfields2000/dockerjenkins_tutorial

60. You Make This Sound Simple.

61. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 1 - Docker Isn’t “Simple” DOCKERFILES REQUIRE DECENT SYSTEMS ADMIN KNOWLEDGE DOCKER IMAGES THAT BUILD DOCKER IMAGES IS DOCKERCEPTION DOCKER “VOODOO AND BLACK MAGIC” QUESTIONS

62. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 2 - Containers != VM’s CANNOT MOUNT REMOTE FILE SYSTEMS HAVE RULES FOR DOCKERFILE VS RUN TIME EPHEMERAL CONTAINERS DON’T MAINTAIN STATE

63. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 3 - Garbage Collection DOCKER BUILD DOCKER PULL DOCKER RUN VOLUME

64. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 4 - Maintenance/Failure PULL HOSTS ON/OFFLINE UPDATE ALL IMAGES ROLLING RESTARTS

65. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 5 - How Will You Upgrade?

66. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 7 - Credentials & Security WHAT SHOULD YOU PUT IN YOUR BASE IMAGE? PASSWORDS IN YOUR SOURCE CODE? SSH KEYS IN YOUR REGISTRY? PASSING EVERYTHING IN AS A ENVIRONMENT VARIABLE?

67. (NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

68. Over 1200 New Build Jobs Created

69. 30% of All Environments Are Containers

70. Environment Change/Create/Fix Tickets Dissappeared

71. We No Longer Have “Just” A Build Team

72. QUESTIONS?