SPI NIGHTLIES Alex Hodgkins

SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all the results from the previous nights build (number of build/test errors) and historical data.  Place all of the files onto AFS and make the log files accessible from the nightlies page.

Our build configuration  Slots  An ordered list of projects to build and their configuration (version, where to build, etc).  Can contain any number of projects to build.  Contains a unique list of platforms that belong to the slot.  Platforms  Dictate a specific machine configuration: The architecture Operating system Compiler to use Build mode (e.g. debug)  Also contain a numbered priority.

Current Infrastructure  We use LCGCMT as our ‘configuration management tool’ to manage the checkout, build, install and test of each software project.  The infrastructure is responsible for:  Automating the usage of LCGCMT to perform all build steps  Putting results into our database  Moving builds to AFS (for Mac and Windows)  Deleting old build data

Current Infrastructure (cont.)

Current infrastructure (cont.)  Run once nightly through crontab after server has been restarted  Continually requests platforms from server  Stops running when the server has no jobs left  Runs on Linux, Mac and Windows Client Server  Restarted every night through crontab  Acts as a queue (based on priority assigned to each job) for all loaded platforms  Performs all database connections  Runs on Linux, so writes to AFS for Mac and Windows clients.  Uses XMLRPC to receive requests from the clients.

Problems with infrastructure  Queue  No way of viewing the queue.  No way to add/remove a platform to be built to the running queue – the entire queue must be reset.  Client  No automated way to kill a started build.  If a platform is manually re-built on the same day it will overwrite the previous build completely.  Builds occasionally hang, and stay hung silently until they are manually killed.  Scheduling  Clients must be launched manually or the crontab edited to schedule a client to run later.  The moving of builds to AFS is currently scheduled by crontab, instead of when a build finishes.  Reporting  Once a build has been requested from the server it is marked as completed – the server doesn’t know who is building it or what stage it’s at.  The infrastructure sends up to 20 s per project per night.

Basic Infrastructure Requirements  We must be able to supply multiple configuration options for a job; architecture, operating system, compiler, mode and project tag.  It must perform all build steps for all requested projects in order.  All data generated (build logs, number of errors, etc.) must be viewable from a web interface.  Historical data is also required so we can look at trends – a database of some description is required.

Available build automation software  There are many different build automation tools available, all of which carry out the same core set of tasks:  Checkout specified code.  Run user specified build steps (python script, maven script, etc).  Collect test/build logs.  There are differences between the various tools, but they are largely irrelevant to us as it is things that we won’t use like different supported builders (CMake, Ant, QMake, MSBuild, etc.) or IDE integration.

Benefits of other build systems  The main benefit is that all the build systems have useful functionality already implemented that we do not have:  Maintain a list of connected clients – when a job is added to the queue the most appropriate client is selected to do the job.  Continuous integration.  The ability to kill a job and any processes it created via the web interface.  Display the output of a running job in real time in the web interface – immediate notification upon failure.  Estimated build times.  RSS feed for the build history.  Used by many people, so there is documentation, plugins etc.

Drawbacks of other build systems  Due to complexity of the nightlies, if we used another build automation system we would still require the following:  Scripts to set up the environment and perform all the build stages.  A web summary page (like we have now) with its own database – something like the BuildBot waterfall page will not contain all of the information we need.  Scripts to place all build data into a database.  Manually defined slots and platforms – we would likely have to add all configurations to the web interface, so editing would become more complicated.

Drawbacks of other build systems (cont.)  Complexity of nightlies mean a lot of time must be spent integrating the current working infrastructure.  No direct support for our current build tools – typically it is much easier to integrate with something like a Maven project, for which there are many plugins/tutorials.  It would not be as easy to extend for the other users of the current system (e.g. LHCb).  We would also be reliant on a build system we don’t have full control over.  We chose Hudson to look into as it is easily extendable, simple to use and is being actively developed.

Hudson  I attempted to use Hudson to build an entire slot:  To have multiple configurations for a single job a ‘matrix project’ that creates all possible combinations from given lists must be used.  This would generate 162+ possibilities (of which we run 20) which can only be filtered using a single groovy statement.  This leaves separate jobs for everything, so we would potentially have to:  Create a job for each project (ROOT, RELAX, CORAL, etc).  Create a job for each slot/platform combination so that they can be called individually or as a group.  Create a slot job that would set all the options for each project in environment variables for the build scripts to use, and lists all the projects to build.  The only other option is to leave the configuration inside the current infrastructure, and use Hudson as a tool just for client/server interaction.  There is also the minor issue of Hudson being forked to Jenkins over a trademark dispute with Oracle in January.

Future of our infrastructure  We are slowly moving towards implementations of the following core set of features:  The client/server interaction will be like the other build automation software – the server will keep track of connected clients, and distribute jobs to them.  Ability to add jobs to the queue via a web interface, as well as kill them or remove from the queue.  Ability for a job to timeout.  Estimated completion times.  Ability to run the same job more than once in a day without overwriting the previous build (going towards continuous integration).  But various changes have had to be made to the infrastructure to make these possible.