EMI is partially funded by the European Commission under Grant Agreement RI Build and Test Services of the EMI project: Lessons Learned and Perspectives Andres Abad Rodriguez, CERN Duarte Bacelar de Begonha De Meneses, CERN Manchester,

EMI INFSO-RI Situation Pre-EMI Challenges EMI Build and Test service ETICS infrastructure EMI metrics and reports Situation Post-EMI Benefits and lessons learned Conclusions Outline 10/04/2013EMI Tools2

EMI INFSO-RI Tools are needed for success climbing 10/04/2013EMI Tools3 EMI 1 Kebnekaise EMI 2 Matterhorn EMI 3 Monte Bianco Proper tools needed for success!

EMI INFSO-RI Situation Pre-EMI EMI Tools4 10/04/2013

EMI INFSO-RI Challenges EMI Tools510/04/2013 Different cultures, environments, requirements and priorities led to different tools difficult to merge Uniform the build environment and policies to provide a common software engineering workflow Obtain packages compliant with the main distributions guidelines (Fedora EPEL and Debian) Provide a comprehensive infrastructure able to satisfy the totality of needs Introduce QA metrics generation and reporting in the same tool-chain Support everybody on a new system

EMI INFSO-RI Situation in EMI EMI Tools6 10/04/ Pbuilder Mock + Pbuilder Mock + Pbuilder Mock

EMI INFSO-RI Main EMI Build and Test service milestones 10/04/2013EMI Tools7 First year All Product Teams migrated to use the same set of tools and using the same platforms Metrics gathering and QA reports generation Second year Improve the system by adapting it to the new platforms and extending the infrastructure to use standard tools (Mock and Pbuilder) Provide more comprehensive QA reports Third year Consolidate the work done, specially with Mock and Pbuilder, and improve the general reliability and performance of the full system Build only from source packages

EMI INFSO-RI ETICS: EMI Build and Test Service EMI Tools810/04/2013 Worker Nodes

EMI INFSO-RI How it works All the jobs are executed in virtual nodes using an elastic infrastructure The nodes are reverted to the initial clean state after the jobs are finished Started on demand to manage high load in specific platforms Benefits Better use of hardware resources Standardization of platforms as all users are building and testing using the same images ETICS virtual node pool 9EMI Tools10/04/2013

EMI INFSO-RI First year SL5 (32 and 64 bits) Static pool of nodes (Fix number of nodes) 50 VMs, jobs/month Second year SL5 and SL6 (32 and 64 bits) and Debian 6 (64 bits) Dynamic pool of nodes (Satisfy the demand with the same hardware) 290 VMs, jobs/month Third year SL5 and SL6 (32 and 64 bits) and Debian 6 (64 bits) Dynamic pool of nodes (Improved load balancing) 310 VMs, jobs/month Platforms and pool evolution 10/04/2013EMI Tools10 Pool of Nodes Jobs

EMI INFSO-RI EMI metrics and reports 10/04/2013EMI Tools11

EMI INFSO-RI First year Policies definition New QA plugins: SLOCCount, FindBugs, PMD, Checkstyle, PyLint, RPMLint, CPPCheck, etc... Development of the QA reporting tool Second year New plugins: RPMlint, CPPcheck, PyLint and IPv6 New reports: RPM compliance level Improved and more comprehensive QA reports Third year New plugin: Lintian New reports: DEB compliance level EMI metrics and reports evolution 10/04/2013EMI Tools12

EMI INFSO-RI Standardized tools and procedures Building from source (Mock and Pbuilder) A central build and test infrastructure is not needed anymore Some PTs have already selected their future tools. Among others: Situation Post-EMI 10/04/2013EMI Tools13

EMI INFSO-RI The system evolved together with the project to provide required functionality on time. Standardized software lifecycle process for all the PTs using standard tools. The products have been extended to new platforms. Generation of packages compliant with main distribution guidelines. Benefits and lessons learned 14 EMI Tools 10/04/2013

EMI INFSO-RI Questions? 10/04/2013EMI Tools15