Openshift for Continuous Integration Alex Lossent – IT/PES/IS AI meeting 1-Oct-2015 Openshift for Continuous Integration1

Outline Continuous Integration: context and history Platform-as-a-Service concept Docker/Kubernetes/Openshift How Openshift compares to Puppet, Heat Plans and outlook AI meeting 1-Oct-2015 Openshift for Continuous Integration2

Why a Continuous Integration platform? Concept: improve code quality by integrating code frequently and applying automatic quality control (tests etc.) Awareness of multiple CI setups for all large software projects at CERN In IT, PH, GS, EN… Using Bamboo, Jenkins, TeamCity… First experience with Jenkins in PES: AIQA Project to provide a Jenkins “platform as a service” to make it easier for developers to deploy CI AI meeting 1-Oct-2015 Openshift for Continuous Integration3

CI: current offer Since end 2014, we provide a CI “platform” Using well-established, open-source Jenkins A Java web application Request an instance from Documentation Clear interest from developers >40 instances as of September 2015 forum on September A “platform” rather than a “service” We provide the infrastructure (Puppet/Openstack) Out of scope: support for Jenkins configuration itself, integration of existing CI setups Full details presented at ITTF 5-Dec-2014ITTF 5-Dec-2014 AI meeting 1-Oct-2015 Openshift for Continuous Integration4

Significant challenges Need more flexibility Every project has specific needs No standard development platform at CERN Frequent requests for custom RPMs, libraries, OSes, plugins, integration scenarios etc. Need more automation Many changes trigger a support ticket Too much manual work for both users and ourselves Need better scaling 1..n VM per project with CI Security Team project to provide static code analysis AI meeting 1-Oct-2015 Openshift for Continuous Integration5

Towards a PaaS solution We want to provide the infrastructure for CI While giving maximum freedom to Jenkins users With minimum cost of entry and maintenance effort for users I.e. self-service, fully automated, self-repairing A good candidate for a PaaS solution AI meeting 1-Oct-2015 Openshift for Continuous Integration6

Options to deploy a web app… AI meeting 1-Oct-2015 Openshift for Continuous Integration7 Less effort Less flexibility Specific help More effort Best flexibility Unspecific help Web hosting: AFS/DFS website Middleware-on- demand SaaS: Drupal SharePoint IaaS: Openstack Puppet DNSLB PaaS: Good flexibility Limited effort “to develop, run, and manage Web applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app” [Wikipedia]Wikipedia

PaaS: useful technologies Containers Operating System-level virtualization: A host system runs several isolated user spaces on top of a single kernel Provides app isolation with less overhead than VMs Containers can be provisioned and started in seconds Allow to separate maintenance of the host OS from that of the application itself Like VMs can stop worrying about physical hardware Docker A popular container solution An application can be packaged with all its environment into an image Includes all libraries, app framework, code, configuration… Portable: can be deployed and instantiated with little constraints on a variety of hosts AI meeting 1-Oct-2015 Openshift for Continuous Integration8

PaaS: useful technologies Kubernetes Docker container orchestration system by Google Software-defined network (SDN) to provide advanced services Recent but gets a lot of traction RedHat Openshift Builds a PaaS solution on top of Kubernetes Adds user management, UI, web proxy, image registry, application templates etc. And an application lifecycle (Continuous Deployment) AI meeting 1-Oct-2015 Openshift for Continuous Integration9

What can we expect from using Openshift? Flexibility Delegate full control on the Jenkins app to users Self-service and automation (by design) Scaling Better resource-efficiency with containers Advanced level of service (from Kubernetes) Persistent storage via Ceph (RBD) Fast recovery (virtual IPs, replication controller) Good availability (container migration) AI meeting 1-Oct-2015 Openshift for Continuous Integration10

How to apply this to CI? We provide and maintain a reference Jenkins Docker image With relevant pre-configuration and integration And an Openshift application template for it Users create instances of Jenkins on our Openshift cluster (self-service via CLI/GUI) Allow user-provided Docker images for slaves Each project can have its own build environment No need to deploy and maintain dedicated VMs Fresh container for each job run AI meeting 1-Oct-2015 Openshift for Continuous Integration11

PaaS general use case Openshift interesting for general web application hosting ~1400 HTTP-related openings in the external firewall Applications currently hosted outside CERN More flexible than web hosting/MWoD Wide choice of frameworks, languages… Significantly less effort than IaaS No host OS to manage Configuration management tailored for web apps Scalability, redundancy, good availability are almost free Fills a gap in providing tools to facilitate web app deployment and upgrade AI meeting 1-Oct-2015 Openshift for Continuous Integration12

Case study: GitLab IaaS-based deployment Many VMs (dev/prod, redundancy with DNSLB) NFS filer, DBoD (single points of failure) Complex configuration management with Puppet Puppet module + tricks, custom Lemon sensors Significant work with each new version Complex, manual upgrade procedure 30 minute downtime (even for possibly online upgrades) Not doable with Puppet (lack of synchronization), even less so with migration of Puppet repos to GitLab Slow recovery in case of failure DNS update delay Can be much longer if DBoD or NFS filer fail (if possible) AI meeting 1-Oct-2015 Openshift for Continuous Integration13

Case study: what Openshift could improve Better resource efficiency: a lot less VMs Direct use of Ceph (RBD) volume for storage Single GitLab instance (“pod”) at any given time Faster recovery Takes seconds to reschedule a container, reconnect to storage, reassign virtual IP SPoF (and dependency loop) removal: NFS filer possibility to host database in Openshift as well Considerably simplified management and procedures Apps are described in terms of how to generate a Docker image from source code and a deployment configuration Openshift designed to facilitate Continuous Delivery Automation of building Docker images, deploying new app versions (and rollback) Can leverage standard RoR migration/upgrade procedure AI meeting 1-Oct-2015 Openshift for Continuous Integration14

Caveat: security updates App owners responsible for maintaining their apps… and all their dependencies Security fixes! Openshift provides an (optional) workflow whereby a base image can be updated and app rebuilt on top of it Mostly appropriate for apps with a Continuous Deployment model Frequent integration, updates and redeployment AI meeting 1-Oct-2015 Openshift for Continuous Integration15

An interesting complement to existing web app hosting offer AI meeting 1-Oct-2015 Openshift for Continuous Integration16 Less effort Less flexibility Specific help More effort Best flexibility Unspecific help Web hosting: AFS/DFS website Middleware-on- demand SaaS: Drupal SharePoint IaaS: Openstack Puppet DNSLB PaaS: Good flexibility Limited effort

What about Heat? Openstack Heat was also considered as a way to address current challenges with CI Heat = orchestration for Openstack Can be used to provide a Jenkins “application template” Improvements regarding automation, self-service, flexibility This option would have been fantastic if available end 2014 when CI project started But now suffers from availability of a solution like Openshift Non-trivial work required: integration of Heat with Puppet to maintain long-lived VMs complete refactor of how Jenkins is managed by Puppet Resource efficiency “level of service” (recovery etc.) AI meeting 1-Oct-2015 Openshift for Continuous Integration17

Where are we now? Prototype Openshift cluster was relatively easy to deploy Hosts managed with Puppet (+Ansible) Polishing to do but no blocker in sight Work on a preconfigured Jenkins image in progress With SSO, GitLab integration, etc. Production status before end of the year is still realistic AI meeting 1-Oct-2015 Openshift for Continuous Integration18

Collaboration opportunities? Docker has been lurking around for some time, but no well-defined strategy yet In particular, no official SLC5/6 or CC7 images A new type of resource to manage Reassign when users leave CERN, etc. Integration with existing services? New technologies to master Docker, Kubernetes are definitely technologies that will play a major role in the coming years AI meeting 1-Oct-2015 Openshift for Continuous Integration19

Do you want to know more? Openshift 3 Roadshow: free workshop in Geneva on October 22ndfree workshop in Geneva on October 22nd AI meeting 1-Oct-2015 Openshift for Continuous Integration20

Case study: GitLab IaaS-based deployment Cluster of 2 large VMs for web frontend Cluster of 2 small VMs for state (redis) DNSLB aliases for cluster management (active/standby in both case) NFS filer for storage (single point of failure) Postgres DBoD (some redundancy) Complex configuration management with Puppet Custom Lemon sensors for DNSLB selection Upstream Puppet module to deploy GitLab “from source”, many contributions from CERN Significant work with each new version Various tricks to integrate upstream module with Teigi secrets and other CERN specificities AI meeting 1-Oct-2015 Openshift for Continuous Integration22

Case study: GitLab (2) Complex, manual upgrade procedure Virtually impossible to automate upgrade via Puppet (lack of coordination) Ballet with puppet agent, services, git repos around the core RoR upgrade process 30 minute downtime (even for possibly online upgrades) Getting worse as repos hosting the Puppet code for GitLab configuration move… to GitLab Slow recovery in case of failure DNS update delay if it can be detected by custom Lemon sensors Possibly much longer if DBoD or NFS filer fail With Puppet code hosted on GitLab (so no change possible while GitLab down), can they even be restored? AI meeting 1-Oct-2015 Openshift for Continuous Integration23