Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Red Hat on Jenkins: By the numbers Nam Duong Red Hat, Inc.

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Agenda Products currently leveraging Jenkins Deployment size: # of Masters/Slaves/Jobs in use across Jboss Middleware QE Top 10 list of interesting Plugins Code contributions Experiences/Pitfalls/Lessons learned Best Practices Demo: Production/Staging server (sheer numbers) OpenShift on Jenkins EC2 and GIT plugin demo

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Red Hat products Products currently leveraging Jenkins JBoss offerings: Enterprise Application Platform JBoss Developer Studio SOA JBoss Portal JBoss OSGI Many, many others OpenShift – OpenShift is our Platform as a Service (PaaS) offering. Our Jenkins instances are running in the cloud, hosted on OpenShift Flex and used to build and test OpenShift

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Deployment Size JBoss Middleware Team: – 1 Master IBM System x3550 M3, 2x E5640 CPU, 24GB RAM, 2x SAS 73GB 15K-RPM in RAID1 – Slaves connected 113 (various OS: linux, Win, Solaris, HP, AIX) – ~4700 Jobs configured (~1000 running/month) – ~95% of Middleware test automation is called/executed via Jenkins OpenShift Flex Team: – 1 Master 7.5 GB, 4CPU, 10GB EBS (RHEL 5.6) – 10 Slaves Building EC2 Images and 5 slaves running tests – 20 jobs (RHEL 6.1) – ~90% of our automation is executed via Jenkins

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Interesting plug-ins used Management (Groovy*, Disk Usage*, Build Publisher*, Workspace Cleanup*) Cloud Control (EC2 plugin, DeltaCloud*) Enterprise focused plugins (SmartFrog) Code Coverage (cobertura, emma) Source control (GIT/SVN/CVS) *Represents code contributions

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Code Contributions Memory leaks: Involved in diagnosis, resolution In the Hudson timeframe, we contributed to stability through reporting of bugs/instability (large deployment) First ones to run into these types of problems (pure volume) LTS (long term support - Jenkins stable branch) Development and maintenance of plugins (see previous) reporting bug + bug fixes

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Experiences/Pitfalls/Lessons learned 1 master may not handle all jobs at once - potentially provide failover functionality on Jenkins Jobs getting stuck due to polling console output through ssh. Need to limit jobs from writing to console. Disk Utilization issues - Overcome through quotas set on jobs Logging was too verbose, Jenkins instance stalled due to low/no disk space OOM issues when trying to view the giant logs - fix in progress

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Best Practices Set limits on memory usage (quota) of slave Set limits (quota) on Disk usage for jobs Tools on shared NFS drive – for maintenance and replication/recovery Memory – lots of memory to buffer incremental growth of Master Jenkins master and all slaves under CFEngine control (config management/recovery) Like all reliable services, use Dev/Stg/Prod increments. Can scale vertically, but has anyone tried scaling horizontally?

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Demo Production instance within corporate firewall: Production: Staging our plugins Staging: Jenkins running on OpenShift Flex; testing OpenShift Flex EC2 and GIT plugin demo Used to create a client testbed Creates an EC2 instance, prints the IP address, pulls down all code bits

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Links/Contact We appreciate your feedback of your user experience

Jenkins User Conference Jenkins User Conference San Francisco, Oct 2 nd 2011 #jenkinsconf Thank You To Our Sponsors Platinum Sponsor Gold Sponsor Silver Sponsor Bronze Sponsors Coming Soon: The CloudBees Newsletter for Jenkins Please complete the Jenkins survey to help us better serve the community (bonus: a chance to win an Apple TV!)