Thinking Inside the Container: A Continuous Delivery Story Maxfield Stewart Engineering Manager: Riot Games (NOTE: PASTE IN PORTRAIT AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

Who’s This Guy?

Thinking Inside the Container

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

A Quick Note on Riot Games

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) The Scale of League MORE THAN 67 MILLION MONTHLY ACTIVE PLAYERS MORE THAN 27 MILLION DAILY ACTIVE PLAYERS 7.5 MILLION PEAK CONCURREN T PLAYERS

1.25 Million Builds a Year

10, ,000 Containers A Week

120 Jobs An Hour

A Containerized Build Farm JENKINSSWARM DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker Core/32GB RAM/120GB LVS DRYDOCK BUILD HOSTS SSH Docker API

STORY TIME! (aka. Presenting the Problem)

3500+ Unique Build Jobs

650+ Builds an Hour

90+ Build Slaves

Expecting 100 More!

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Story Time.... Ticket EngineersBuild Team

Story Time.... Ticket EngineersBuild Team

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) What Did We Want? TEAMS NEEDED TO MOVE FAST PRODUCTS HAD TO OWN THEIR STACK CONFIGURA TION AS CODE

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Maybe We Want...

“Along Came a Whale” (More Story Time!)

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

Oh Look! Another Way to Deploy!

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Docker For Newbz

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB”

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA”

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA” Build Job Queue

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” MasterSlaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA” Build Job Queue

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) What Did We Need? CONTAINER RUNNING AS A BUILD SLAVE HOW TO SCALE IT WAY TO HAVE ENGINEERS PROVIDE DOCKERFILES PROFIT?

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) A Build Slave Container

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Add a Bit of Secret Sauce... Base Slave Image Universal Riot Tools Product/App Specific Build Stuff

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) A Real Example

Provisioning and Plugins += ?

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Of Whales and Plugins MESOS PLUGIN KUBERNETES PLUGIN DOCKER PLUGIN

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Of Whales and Plugins MESOS PLUGIN KUBERNETES PLUGIN DOCKER PLUGIN

Quick Look

Provisioning and Plugins Image Name Label

Groovy To the Rescue torial/tree/master/jenkinsscripts

We Created A Monster

We Need to Inspect Our Containers

./harbormaster --registry_user "$REGISTRY_USER" --registry_token "$REGISTRY_TOKEN" --jenkins_user $USER -- jenkins_pass $PASSWORD --jenkins --dockerhost $TEST_DOCKER_HOST "$IMAGENAME" "$CLOUDNAME" "$LABELNAME"

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Are We Done?

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) But It Needs to Scale!

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) If Only...

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) If Only...

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) But It Needs to Scale! JENKINSSWARM DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker Core/32GB RAM/120GB LVS BUILD HOSTS SSH Docker API

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY Image

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER Image Name + Jenkins Label

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER Runs Tests

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER SWARM DOCKERHOST Image Pulled to Swarm

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Putting It All Together Engineer REGISTRY HARBOR MASTER SWARM DOCKERHOST Groovy API Called Jenkins

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Build Job Quick Look

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Dockerception

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Where to Build Containers? JENKINSSWARM DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker Core/32GB RAM/120GB LVS DRYDOCK BUILD HOSTS SSH Docker API

Mai Tai’s On the Beach Build Engineer

How Do You Actually Build It?

(engineering.riotgames.com)

You Make This Sound Simple.

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 1 - Docker Isn’t “Simple” DOCKERFILES REQUIRE DECENT SYSTEMS ADMIN KNOWLEDGE DOCKER IMAGES THAT BUILD DOCKER IMAGES IS DOCKERCEPTION DOCKER “VOODOO AND BLACK MAGIC” QUESTIONS

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 2 - Containers != VM’s CANNOT MOUNT REMOTE FILE SYSTEMS HAVE RULES FOR DOCKERFILE VS RUN TIME EPHEMERAL CONTAINERS DON’T MAINTAIN STATE

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 3 - Garbage Collection DOCKER BUILD DOCKER PULL DOCKER RUN VOLUME

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 4 - Maintenance/Failure PULL HOSTS ON/OFFLINE UPDATE ALL IMAGES ROLLING RESTARTS

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 5 - How Will You Upgrade?

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP) Lesson 7 - Credentials & Security WHAT SHOULD YOU PUT IN YOUR BASE IMAGE? PASSWORDS IN YOUR SOURCE CODE? SSH KEYS IN YOUR REGISTRY? PASSING EVERYTHING IN AS A ENVIRONMENT VARIABLE?

(NOTE: PASTE IN PHOTO AND SEND BEHIND FOREGROUND GRAPHIC FOR CROP)

Over 1200 New Build Jobs Created

30% of All Environments Are Containers

Environment Change/Create/Fix Tickets Dissappeared

We No Longer Have “Just” A Build Team

QUESTIONS?