Delivering eBay's CI Solution with Apache Mesos & Docker Mohit Soni Ashish Hunnargikar Software Engineer Software Engineer

Why we did it.... Thousands of VMs dedicated to run Jenkins! Utilization is less than 5%  VMs idle most of time (no one runs CI builds frequently) VM sprawl and poor resource utilization Polyglot application stacks (Java, C++, Node.js, Python, Scala) Different OS flavors (Ubuntu & RHEL) Software version conflicts Special hardware requirements Build slave management and maintenance is a nightmare

And here’s what we did.... Mesos for cluster management Marathon to launch Jenkins masters Jenkins Mesos-plugin to launch build slaves Build slaves Dockerized as well as regular

Why use Docker containers? Managing varied slave flavors in a large cluster is hard! Docker effectively isolates dependencies inside the container Docker allows host OS to remain independent of workload dependencies Why build Docker-in-Docker? Run Docker build/push/pull inside a Docker container Eliminates redundant CI build dependency downloads Helps to enforce a heterogeneous Mesos slave cluster Allow different OS flavors Prevent software version conflicts Application dependencies Hardware independence

Basic Use Case Build C++ & Java applications on both RHEL & Ubuntu Mesos slave host server is running Ubuntu Jenkins job is running builds in a RHEL or Ubuntu container Application dependencies can be preinstalled in base images OR can be downloaded during build time

Advanced Use Case Build RHEL & Ubuntu Docker images Mesos slave host server is running Ubuntu Jenkins job is running inside a Docker container Jenkins job is building a RHEL/Ubuntu Docker image from an application Dockerfile

Enabling the Docker CI workflow using Jenkins & Mesos

Initial Build Strategy Individual Docker-in-Docker Setup Complete isolation Docker daemon running inside the Docker container requires privileged mode I/O overhead of downloading and duplicating all the AUFS layers

Optimized Build Strategy Shared Docker-in-Docker Setup Docker daemon not required inside the container so no privileged mode Single reusable AUFS image layer cache Redundant network and disk I/O eliminated for layer downloads Lack of isolation for layers

For more information visit: Q&A www.ebaytechblog.com http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/ http://www.ebaytechblog.com/2014/05/12/delivering-ebays-ci-solution-with-apache-mesos-part-ii/ For more information visit: Apache Mesos: http://mesos.apache.org Jenkins Mesos Plugin: https://github.com/jenkinsci/mesos-plugin Docker: https://www.docker.io Marathon Framework: https://github.com/mesosphere/marathon